In helping me understand what could be happening with you application, it would be useful to get the output of NI-Spy while your application is executing - particularly when you see the error. NI-Spy will show exactly how your application calls NI-DMM. Launch NI-Spy from Start >> Programs >> National Instruments >> NI-Spy. You can use NI-Spy Help to find out more about this utility.
NI-DMM allows you to go between scalar and waveform measurements, and the best way to do that is to explicitly use niDMM Configure Measurement.vi and niDMM Configure Waveform Acquisition.vi when you need to change the mode. If you are programming the attributes through LabVIEW property nodes, you will need to configure more of the attributes explicitly - only changing Operation Mode is not enough; you would need to change Operation Mode, Function, Range (and Resolution for scalar mode) if you are going from scalar to waveform modes and back. Again, the best way to do it is to use the "configure" vis.
Regrads,
Sead Suskic
National Instruments