High Speed Streaming Data RTOS

[MgDAQ]

I need to read in an analog input value and quickly save the data to file with a time stamp.  I use the Cont Acq Voltage – To File (Compacted).vi from the Data Compaction for High-Speed Streaming to Disk Examples to save the data to a binary file.

To view the data I transfer the file to the local PC and I use the code from Graph Acquired Compacted Data.vi (also from the Data Compaction for High-Speed Streaming to Disk Examples). To put the data in a format one can import into excel I added to this code the Write To Spreadsheet File VI. 

The two vi’s work very well, however since the data is RAW 8U, I do not get the time stamp I need.  Naturally I changed the data type to waveform 1D. This is where the problems occur.  I suspect this is due to the data compaction code portion.  The data viewed is noisy, and the spreadsheet file is also messy – no timestamp.

I initially thought to use the Cont Acq&Graph Voltage – Write Data to Spreadsheet File.vi, however the File Dialog function is no longer supported, and I am told to use the File Dialog Express VI instead. Also, I am concerned about using this vi anyways because I don’t know if it is the most efficient way to stream the data to disk quickly. The application is time critical, so the goal was to stream the data to disk, then upload and view the data after the experiment in a more user friendly format (excel).

Are there any examples that show how to stream a waveform datatype to disk quickly and efficiently using the real time environment that does not use unsupported functions? Yes, I need the timestamp. I really prefer to use the waveform data type and not get the timestamp from math by writing code to use the start time, end time and change in time (dt).

Please do not tell me to use Express VI’s as National Instrument’s own documentation clearly state not to use Express VI’s in the real time environment as their performance are not predictable.

Currently using:

LabVIEW 8.6.1, RTOS

PXI-1050

Your direction is most appreciated.

Thank you.

[rex1030]

Well to write data quickly you are smart to stick to the binary write and read. This is because you will be writing the least amount of bits necessary to accurately represent the numbers you are saving. However, as you have discovered, the binary data write makes it impossible to write a different data type. I suggest using TDMS. TDMS format is also binary data but allows you to write other descriptive things about the data that you want to write, for instance, start time, data rate, end time. It is unfortunate that you do not have LV2009 because improvements were made to the behavior of the TDMS write on RT systems, but it should still work for what you are trying to do.

Check out the example VIs for tdms and make sure that you are creating and writing the attributes of the TDMS file before you begin recording data (not in your record loop). 

How many channels and at what rate are you trying to record?

edit: Be sure to use the distributed system manager to monitor the cpu and memory of your program during testing. You want to look for bad things like the CPU spiking to 99% for a few seconds or the memory steadily filling up and see how your code changes affect cpu and memory load.

Message Edited by rex1030 on 12-10-2009 01:31 PM

[MgDAQ]

Thank you for your response.

Right now I am trying to get one channel working. Eventually there will be 12 channels reading different measurements.  I am trying to build the program using a solid foundation by using the hardware efficiently rather than throw something together and worry about the time critical aspects later.

The time length to record the data will be 200ms. We are hoping to record at 1000 Hz.

Are there any high speed streaming examples with TDMS for the waveform data type?  The TDMS Timestamp producer-consumer example creates the time stamp, and does not use the waveform datatype.

[rex1030]

I was not suggesting you should "throw something together and worry about the time critical aspects later", but recording at that rate will tend to show you the limits of your system and monitoring the system as you test different aspects of your program will help identify whether or not the method you chose is going to work, rather than just looking at the data file and noticing the data is noisy.

The best way to do this is read from the channels and write a set of data to a queue in one loop and then read from the queue and record a decent chunk of data from the queue to the tdms file in another loop, as I am sure you have seen in the example VIs you mentioned. Be sure not to mess with the loop priorities. Monitoring ram and cpu will help identify if queue overflow or underflow is occuring with this method, meaning you are reading too slow or too fast from the queue. 

Since you are not recording at 1000 Hz for more than 200 data points per channel (200ms at 1000hz =200 samples), you are more likely to not run into memory leaks as easily as if you were trying to record for an hour.

Feel free to post your code if you run into any problems.

[AndrewMc]

The easiest and fastest way to write your DAQmx data is using DAQmx 9.0 feature for streaming to disk.

If you just call the "DAQmx Configure Logging" VI in your application, all of your data will be logged to the specified TDMS file.  Just specify a file path on that VI.  Note that on RT, C: correlates with the ftp root; therefore, C:\test.tdms would save a file called test.tdms to the ftp root.

To get that support, just download DAQmx 9.0.2 if you haven't already: http://joule.ni.com/nidu/cds/view/p/id/1444/lang/en