Data acquisition freezes (RS-232 and USB) and spectrometer fails to connect after restart (USB)

[Smog]

In our lab, we have been struggling with failures in simple data acquisition during long experiments (1-3 days, for example). We run two top-level VIs simultaneously. One records experimental conditions (temperature, pressure, relative humidity, etc.; RS-232) in a smog chamber, while the other records spectra from an Ocean Optics spectrometer (HR-2000+; USB) and fits for gas concentrations in the chamber. At random intervals, data acquisition "freezes", in the sense that the instruments are still supposed to be connected, but readings are no longer being refreshed. We have observed these failures anytime from one minute to eight hours or more after starting data logging, and in both top-level VIs, which are linked only in their use of some shared global variables.

Due to these random intervals, our diagnosis of the problem has been hampered. We have designed a few work-arounds in the mean-time; we have each VI monitor changes in global variables written by the other VI. If the variables (measurements) are constant, we know the data acquisition is frozen, and the system is forced to restart, and another VI recovers data logging. However, after the restart, the spectrometer fails to connect. We are aware that failures in USB connections are common enough in LabVIEW, but we are not sure why they really only occur in our setup after a restart. Of course, we then have to unplug and reconnect the spectrometer manually, a solution which is impractical if the data acquisition freezes at 3:00am. So a whole experiment can be spoiled.

We have two problems then: the data acquisition freezing, and the spectrometer failing to connect after restart. True, we can introduce a relay to simulate unplugging the spectrometer, but it would be better to solve the first problem, which would likely resolve our issues altogether. Some other background: We save our data in TDMS files, and these can get very large and fragmented during a long experiment. But defragmenting the files at regular intervals didn't solve the issue. We also tried running the VIs independently, on different machines, but still we experienced freezing. Most infuriatingly, both VIs froze alternately, so we have not been able to further pin-point the problem to a particular VI.

I realize that it is difficult to propose solutions to such a broad problem without more of the details, so if anyone can even pose some more specific questions regarding our setup, I would be thankful.

[Mark_Ramsdale]

Hi,

Do you have any error codes on the instruments after they freeze?  Is is possible to set up a log file to read errors after each instrument call to try and trap the errors?

How are the instruments controlled, using VISA calls? Perhaps the spectrum analyser thinks it is still connected, or the COM port is still connected even though the instrument has stopped communicating.

What is the symptom of freezing, is there a never ending loop, failure to update the front panel, or does labview stop al together?

Are you running in the development environment or as .exe programs?

Do you have a routine that evaluates replies from the instrument, perhaps making sure messages are not being sent when the instrument is still trying to reply?

Well, that is my shot in the dark.  Generally I find it helpful to try and trap errors in the instrument communications or log them, in order to get some picture of what it is doing.

Do you think the instrument is failing, causing the VI to freeze, or is the VI freezing, and leaving all the com ports open, etc.....   

[Smog]

Hi Mark,

Thanks for your reply and suggestions. I will need more time to consider all your questions (and save an error log), but I wanted to pass on some of the meagre details that I can right now.

>How are the instruments controlled, using VISA calls? Perhaps the spectrum analyser thinks it is still connected, or the COM port is still connected even though the instrument has stopped communicating.

The instruments are all controlled by VISA calls. In each instance, the connection is configured, data is requested and received, and then the connection is closed, for a single reading.

>What is the symptom of freezing, is there a never ending loop, failure to update the front panel, or does labview stop al together?

Typically, after communication freezes, the front panel continues to update, but the new data points are all the same value. Besides this, though, the VI continues to run normally.

>Are you running in the development environment or as .exe programs?

We are running in the development environment. Is there an advantage to running the VIs as .exe programs, instead?

>Do you have a routine that evaluates replies from the instrument, perhaps making sure messages are not being sent when the instrument is still trying to reply?

We wait for termination characters from the instruments.  

>Do you think the instrument is failing, causing the VI to freeze, or is the VI freezing, and leaving all the com ports open, etc.....  

We have noticed that communications to several otherwise unrelated instruments fail at the same time; for example, temperature readings from our thermocouples and relative humidity readings from our Omega probe become constant simultaneously, though they are connected to the computer separately and not through a hub.

Thanks again for your reply to my vague post. Hopefully, after saving an error log, I will be able to be more specific. Best regards,

Elijah

[Mark_Ramsdale]

Hi,

In regards to the freezing, if you are waiting for termination characters, do they occur, or do you start timing out?  This seems like a good point in the code to set a log, do some error trapping, and such to see if the instruments just stop responding.  Generally speaking you can cycle at a read if you don't get a termination character or you time out, but this all depends on the instrument,etc .. To know if it is worth waiting longer.

Also, are there some  'clear or reset or re-init VI's' you can use if the communication stops. Perhaps you need a error trapping routine, based on errors from the VISA read or based on your logic for interpreting time outs, partial data, etc ...

Are you using any kid of interrupts or other "operation complete" functions, are these available in the instrument.  They can help with waiting on reads or when to read, etc..  Not sure if this would be overkill, without any details of the freeze.

***

With other instruments failing at the same time I would be inclined to think the OS might have something going on. I think this is probably the biggest tell as to what is going on.

With error handling it is possible that one instrument routine sending an error will cascade to effectively bypass or turn off other instrument reading routines, hard to know with no code.

***

Not sure why your reading would remain constant.  If the instruments are no longer communicating I would expect some zeros, NaN, and +/- INF, depending on code structure.