Design Suggestions for Multiple DaqMX Task Streaming App?

[wired]

I'm working on a LabVIEW application in which I'm streaming high-speed data to disk from multiple PXI devices simultaneously.  Each device has its own DaqMX task, and all tasks stream to the same file.  The PXI device configuration (which devices are in the chassis, which slots they're in, and which channels to read from each device) is determined at runtime.

Does anyone have a suggestion for a design model for this?  To make matters worse, I'd like to be able to specify a channel to monitor its data during the streaming.  I'm thinking the Producer-Consumer model is the basic approach, and I'm at the point where I have an array of DaqMX tasks, one for each device.  I could probably extend that array to be an array of clusters containing:

   1) DAQ Command (e.g. Initialize, Start, Stop, Acquire, etc)
   1) Task ID
   2) Control reference to 2-D array (where each DaqMX read can be stored)
   3) Array of channel names (to allow selection of channel to be monitored

This could be passed as notifier data, to a data collection subVI, but the part I'm struggling with is finding the best way to run X number of tasks in parallel, where X is not known until runtime.

Any suggestions would be appreciated.

[DuffmanSays]

Hello wired,

This sounds like an interesting application that you are building. In regard to your post, you were asking about suggestions for the design model. I think you are on the right track with the producer / consumer architecture. You may chose among several different ways for the presentation of your data, but the TDMS file format is the one which I would suggest for high-speed streaming to disk. This format logs data at high speeds along with meta data that you can write for each channel.

You are currently struggling with the way to run an undetermined number of tasks in parallel. What I might suggest would be to use a reentrant sub VI for these tasks. The idea behind a reentrant VI is that it will create a new copy in memory for each time it is called. You might use this VI in a for loop and pass different parameters to this VI such that it will be configured differently for each of the tasks. In this scenario, the different runs of the for loop effectively kick off the different tasks each configured with different parameters. I hope this information is useful for you. Please post back and let us know how your development is coming. Thanks,

Mike D.
National Instruments
Applications Engineer

[Herbert Engels]

The TDM Streaming API allows for concurrent reading and writing access to the same file, so you might be able to go with a simpler architecture (see example "TDMS - Concurrent File Access"). This feature takes some additional disc performance though. If that turns out to be a problem, I would agree that a producer / consumer pattern that keeps the monitoring data in memory would be beneficial.

Depending on the data throughput you are trying to handle, you might also want to consider storing unscaled integer data along with a set of scaling factors and later expand them into floating point values. That will significantly reduce the disc footprint of your files and hence improve performance. See examples "Cont Acq&Graph Voltage - Write Unscaled Data to File (TDMS).vi" and "Cont Acq&Graph Voltage - Graph Unscaled Data from File (TDMS).vi".

Hope that helps,
Herbert

[wired]

Thanks for the input.  With regards to the file format, the decision has been made by my superiors not to use TDMS - unfortunate, but NI hasn't provided the information to write a MatLab file reader, which is a requirement.  So, I've created a custom file format tailored to the needs of my application, but generic enough to be used for other apps. With it, I've been able to stream 8 channels at 800KHz (4 channels each from 2 PXI-6120s) without breaking a sweat.  However, the performance varies greatly depending on which slots the cards are in (but that's a whole different discussion - see the PXI forum for that one).  Once NI solves that one, I'll feel a lot more comfortable.

I have already made reentrant subVIs that can perform a specific DAQ task.  The problem with a for loop is that the VI sits and wait for a start trigger, then acquires the streaming data.  I can't start the next VI because I'm in the first one.  I thought about creating a data collection VI, and this VI would start up to 6 other VIs in parallel, based on how many and which cards were present.  It's a bit messy, because each slot can contain one of two devices, so I'd need to check which type it was before calling it.  I'm thinking I'll have to create the task list and the references to the data in the main GUI loop, and then pass this using a notifier or queue to the data collection loop. 

[Herbert Engels]

Did you try the DLL we provide for TDMS in MatLab?

You can also choose to support TDMS based on its binary layout, but using the DLL will save you a lot of work.

Herbert

[DuffmanSays]

I can definitely see your point with using reentrant VIs; though you can run multiple instances, the for loop will not iterate until each VI is complete. As you mentioned, you may have to determine which cards are present in order to determine how many VIs to start from your "data collection VI." I cannot suggest a better method than passing the task list and references to your acquisition loop. Please post back if we can help at all as you continue to develop this architecture. Thanks,

Mike D.
National Instruments
Applications Engineer