Using viQuery on a serial port can prove difficult, as you have already figured out. Its main purpose is parsing ASCII data, so having some non-ASCII termination character is the best way to do this. If your serial instrument has an alternate 'end' indicator, call viSetAttribute to change either the termchar value or the end mode for input (VI_ATTR_ASRL_END_IN).
Perhaps a better way, if you know how much data is coming, is to call viWrite and then viRead yourself to get the full quantity of binary data from the serial port. Then call viSScanf to have VISA interpret the binary data. Note that VISA will interpret the data as a string, so any binary value of 0 will likely cause that to not work as expected.
Hope this helps,
Dan Mondrik
Senior Softwar
e Engineer, NI-VISA
National Instruments