Before I get into much, I would suggest posting this to the Image Acquisition (IMAQ) Developer Exchange pages. They would surely have more experience with this and might be able to help you more. It's difficult to tell what would cause the problem from my machine since I don't have the correct hardware, but I would suggest contacting whoever developed the AVI write function. It sounds, from what you've said, that this is where the issue is stemming from and they might be familiar with it.
Is the problem that it freezes your machine? Try watching your system performance indicators to see if the processor is being overwhelmed. That could be why it's not writing anymore: no more resources. This may give you a start.
J.R. Allen