First, I'd recommend that you try using VISA for serial I/O. It's more robust.
It sounds like you might just be having problems with hardware flow control. Do you have it enabled on your instrument? If not, I would try enabling it, and enabling it with VISA in your application.
If you are getting messages that are terminated by end-of-line characters (or some other message that's not part of the data), then I would investigate the solution that Dennis recommended.
I hope this helps.
Brian