I just figured it out. I was trying to set the wire mode on the RS485 to two-wire auto, and the RS232 to RS232-auto. The problem was that I had a 'WIRE_MODE=xxx' in my NI-RT.INI file, this disabled the ability to set the wire modes with the serial property node. In addition, the documentation has changed: The pre-VISA 3 instructions were to let (page 6-5 NI Serial Hardware and Software for Windows User Manual)
RS485 4wire WIRE_MODE=128
RS485 2wire DTR w/echo =129
RS485 2wire DTR cont =130
RS485 2wire auto =131
but the new VISA serial has these as 0,1,2,3 respectively, with 128, 129 and 130 being RS232 wire modes. 131 is undefined now.
So it is all working as of this afternoon. Was obtuse to find these errors though, and neither the NI-RT.INI file
or the new VISA wire mode setting ability are documented correctly anywhere that I could find. Thanks for the offer of help!