High level design questions (high speed buffered ADC, data logging, FIFO, etc.)

[mschmit]

Hardware: PXI-8187 with RTOS, PXI-6132 S-Series DAQ, PXI-6259 M-Series DAQ all in a PXI chassis connected to LAN, LV8.2 with a host PC on LAN

Goal: A real-time program that acquires data on 3 channels using the 6132 at a speed of 1MSamples/sec, acquires data on 6 other channels at around 50kSamples/sec (using the 6259).  I would like the acquisition to be continuous and buffered, such that there is no loss of data at any time during the operation.  I would also like the ability to turn on plotting to screen of various results (like waveforms, Spectral results, etc.)  This would only be for small periods of time when the user requests it.

I would like the ability save the information when I see fit, basically based on a user operated switch, or some analysis task that triggers data storage on the 8187 local hard disk.

Implementation: I will be running this entire project on the RT hardware.  The UI will be served via http server such that the program is accesible from many different locations.

Two Timed Loops: One loop is deterministic and contains the DAQmx read portion of the program.  I will have two different DAQmx tasks, one for the 6132 high speed task and one for the slower speed 6259 task.  The other loop is non-deterministic.  It contains the code for data analysis and data storage.  With these things in mind, are my target adquisition rates possible to achieve? Furthermore, what should my loop timings and buffer sizes be?  Should the non-deterministic loop run at the same speed, or faster?  I am timing the loop such that the ADQ will fill the entire buffer during the loop cycle.  I assume this is proper.

Data Buffers Between Loops: As far as I can tell I have two options for buffering the data between the loops.  1) Use RT FIFOs for each channel (1D arrays).  I can't seem to get an RT FIFO to pass a 2D array properly (3 channels x length of ADQ buffer).  If there is code that shows how to do this, let me know.   Total of 9 RT FIFOs.  2) Use shared variables to pass the data.  This uses less space on my block diagram, but I'm not sure it's better.  Furthermore, I have to change the size of the FIFO manually in the Project Explorer, which is kind of a pain.  Also, I've read that shared variables are somewhat slower than RT FIFOs when passing large arrays.  Finally, there is no way to pass 2D arrays using shared variables as far as I can tell (correct me if I'm wrong).  What is the best method for speed and without losing any data?  Any recommendations on FIFO buffer sizes?  I was using the same size buffer for ACQ as FIFO (something like 10,000).

Writing to Disk: I would like to write the files in text format as a tab delimitted spreadsheet so they are easy to handle for post-processing (MATLAB).  I know that binary files are faster and can be opened in MATLAB, but I prefre text files.  I'm writing them in blocks as they come in from the buffer.  Any tips on this portion are appreciated.

Screenshot of the guts of the program (using shared variables) is attached.  Thanks!

[mschmit]

Forgot to attach picture.

[Greg G.]

One minor note, which doesn't really address your question, but may be useful:

If you want to send arrays that have >1 dimension, you can "cheat" by using the reshape array primitive to reshape your 3-d array into a 1-d array that is 3x longer before writing into the RT FIFO, and then reshape it back into 3-d after you read it.  This may execute faster than your current code.

greg

[mschmit]

Greg,

Thanks.  I hadn't thought about that at all.  It's not "cheating", just a "workaround"...

[mschmit]

I've implemented a lot of the systems in a single VI.  I took the advice about concatinating the data so as to pass it in a single RT FIFO.  Seems to work well.

But, I seem to bumping up to the speed limit of my hardware and I think there are ways to eek out more performace from the system.   I can't seem to increase the aquisition speed above 700kS/sec without losing some data.  The image attached should give you an idea on what I'm talking about.  If I want to be able to record any portion of the acquired signals, this is unacceptable to have breaks in the data stream.  It doesn't appear that I'm filling up my FIFO either as I've implented a speed up algoritm for the non-deterministic loop (FIFO read and plot to screen loop).

How should I go about increasing the speed of my acquisition without losing data?

[Peron]

Hi there!

Brief notes:

As far as I know, RT FIFO`s performance is better than shared variables’.

If you do not need to perform the analysis and data storage in a deterministic way, I think the non-deterministic part should have a larger period, even you could use a plain while loop (which leads to a lower overhead than timed loops) and use to time it Wait until next millisecond with a larger time (depending on what you want). Is it possible for you to use a VI architecture based on state machine template; maybe you already know it: it means using a state machine as normal priority vi and calling from one of this states ("running" state) the time critical loop. Maybe is too simple for your requirements? I'm not experience in http server.

Is it possible for you to write the files in binary format in the real time target and convert them to .mat files or to spreadsheet by means of a postprocessing VI?

Hope something provides you an idea!

[mschmit]

Thanks for the feedback.  I like the idea of using a longer while loop, assuming that I can clear the RT FIFO in time so that I don't overfill it.  I'll play with this, and especially using a standard while loop rather than a timed loop (non deterministic that is).

Also, the binary write and then post process is a pretty good idea.  Any clue on how much faster a binary write is compared to an ASCII text write?

I'm not all that informed about state machines.  Are there any good examples that show hos a state machine architecure works?  I'll poke around to see if there are any good examples of this, but like I said, there will be minimal interaction with the host, and no File I/O operations, only non-deterministic plotting to screen and settign what type of analysis is done.  Nothing at a high throughput rate.  The speed crucial tasks are data adquisition and logging to file.

Thanks for the ideas.

[Peron]

Hi,

You can find the information about binary, ascii and so on in http://zone.ni.com/devzone/cda/tut/p/id/3746.

I’ve had a look at LabView 8.0 examples and I think that you could fit the project example Command Based Architecture.lvproj. This is not based on state machine but I think it can be as well a good option for your needs (state machine is one of the design patterns provided in LabView but are less developed than Command Based Architecture.lvproj). The RT VI of this example uses three loops: one for sending data to the host, one for receiving data from the host and one timed loop for the deterministic operations.

Hope it helps!