Parallel DAQ and Post-Processing of Data from DAQ

[Alex Wu]

Problem: I need acquire a lot of data (~2 M) from a DAQ board and post process them, then repeat the above again and again in a loop. The problem here is that the time consumed by the data-post-processing (interpolation) is quite long, but it still less than the time consumed for acquiring those data, for instance, 5 s for data acquisiton and 4 s for data-processing. As the data acquisition and processing is repeated a lot (~1k times). I wonder if I can parallel them like the following way: process n-1th set of data during the acquisition of nth set of data. If so, I needn't waste time for the data post-processing. How should I do? Any suggestions are really appreciated. Thanks

BTW: the computer I used has a dual core processors, and the LV version is 8.0.

Message Edited by Alex Wu on 06-09-2007 12:56 AM

[mikeporter]

Yes you can make this parallel.

• You create one loop that does nothing but acquire the data and stuff it into a LV queue.
• A second loop running in parallel to the first one dequeues the data and processes it.

The queue would be created outside both loops and the reference passed into them. The function for dequeueing an item has a timeout. If you set this to -1 the analysis loop will not consume any system resources until the data arrives. If you need more details on how to implement this let me know and I can put together a quick example for you. Also note that given the length of time that a complete test run will consume (17 minutes at a minimum) it would be advisable to save the results of each analysis operation to disk or a database as it is completed. Otherwise a [crash/power failure/coop tripping over a power cord] could result in the loss of a lot of data.

The only thing that might cause this setup problems is that if the data processing is so CPU-intensive that it begins to interfere with the acquisition process. But this would rarely be a show-stopper since there are a variety of ways to throttle analysis operations so they take longer, but require fewer resources.

Mike...

Message Edited by mikeporter on 06-09-2007 11:45 AM

[Alex Wu]

Mike

Could you please paste an example how it should be handled at your earliest convenience? Thanks a lot.

Alex

[mikeporter]

This shows the basic technique. The top loop acquires the data, the bottom loop processes it. When you run this note that the indicator wired to the output of the random number generators continues changing even while the analysis process is working. Note also that the graph updates every 5 seconds and the average indicator updates 3 seconds after the graph.

I also implemented a shutdown mechanism that consists of posting an item to the queue that contains an empty array.

Mike...

Message Edited by mikeporter on 06-09-2007 03:35 PM

[winterishere8]

Mike
Very good example. I implemented this and my acquisition and processing works good. One question though. How and where you close or release the reference to the queue ? Do you do it in both the while loops ?

[NIquist]

This thread is old so Mike may not answer but if you look in the first loop there is code to end the second loop.  Inside the second loop there is code to release the queue.

[winterishere8]

For some reason, when I implemented this, it doesn seem to do a very good job. It almost has the same performance as data acquisition->processing in a loop. I've attached a snapshot. I've blacked some items that's kinda internal. Any thoughts guys ?

[NIquist]

I assume you're using the #PPIterations global just to keep a count of the tests so i don't think that's an issue (globals are always trouble suspects ).  Your queue structure looks OK to me so i would suspect something in your 'tube data' subVI is the slow point.

BTW, you can just wire the error out of the de-queue elements function to the stop terminal in the second loop (it will automatically change into "stop on error").  Then, releasing the queue in the top loop will stop the second loop too.  That way you don't have to monitor for an empty array.  It's a bit more elegant is all.  Either way will work.

[winterishere8]

am attaching snapshot of the tubedata ... the vi that acquires the data from the DAQ device ....
thanks for your input nlquist

[NIquist]

OK, you're doing all your DAQ task configurations and file maintenance in every iteration of that subVI.  You should open your files and configure the DAQ task at the beginning of your code and then pass references into the subVI.  The subVI should only have operations to read the DAQ data and then write to an already open file.  You then close the DAQ task and file reference only once at the end of your main VI.