Parallelism with file read and DAQmx write

[Djaunl]

Hi all,

I'm trying to come up with a good design to implement, if possible, reading waveform data from a file and writing it with the DAQmx drivers in parallel to the file read. I'd be very interested to know if this is possible and if so, what the best way to do it is. 

Basically, I want to continuously read in data from a TDMS file and continuously write it out to N channels on a couple boards using the DAQmx drivers; these channels will all be synchronized. The file is gigantic so I cannot read it all in at once. I must read it in in small buffers. However, if I read the data in and output it in series there will be a delay as the next buffer of data is read in, as the DAQmx write VI blocks the main thread. 

I'm unsure of how to implement a better architecture. I want the file reader to output a certain buffer size of data to the write VI every so often, and then, as the DAQmx write VI is writing the data to the hardware, I want the file reader to be reading in the next buffer of data. This way there will not be delays. 

One idea that I keep seeing when searching Google and these forums is the producer/consumer architecture. However, I am unsure if this will result in any delays or not when feeding the data to the consumer loop via the queue. I'm outputting a continuous waveform that needs high time resolution, so any delay must be near instantaneous (< 1 microsecond). 

I'm sorry if I've missed something obvious, but I've been trying to figure this out for quite some time now and haven't been able to come up with a solution. I greatly appreciate any advice. 

Thanks!

[RavensFan]

Producer/consumer sounds like the way to go.  I would call the DAQmx write loop the consumer that reads items from a queue.  The file read would be the producer that writes to the queue.

The first thing you need to do is figure out how to do the DAQmx write loop so that the new data can be written into the write buffer without delays.  There are numerous DAQmx examples and one of the must show how to continually feed new data into the Write buffer.

That new data would be read from the queue before the DAQmx loop needs to refill the write buffer.  The trick now is to have a throttling mechanism so that the consumer can communicate with the producer loop to put another element in the queue.

There might be 2 ways to do this (probably even more ways.)

1.  Have the producer loop queue up 2 elements (so that one can be read and another is next in line "on deck" to be read).  The producer loop will continually poll the number of elements in the queue.  Only when the queue size drops less than 2 (meaning the consumer just read 1 element out) it will read the next part of the file and queue that up.)

2.  Like #1, but instead of polling, have the consumer loop trigger a notifier right after it dequeues an element so that producer loop know when to read the next part of the file and queue that data.

With these methods, the DAQmx write loop will handle keeping the DAQ buffer full.  The queue handles buffering between the file read loop and the DAQmx write loop.  Having 2 elements in the queue provides a little extra buffer compared to just one element, but more elements could be buffered.  The only limitation would be how large the elements are and the limits of computer memory.

[Djaunl]

Thanks a lot for the reply. I've been implementing the producer/consumer design pattern today, as well as working on some synchronization stuff (I have a question on that, but I'll put it in another thread for better organization). I haven't yet finished, so I'll post back with the completed design, or any questions, when I'm at that point.

As for continuously feeding the DAQmx Write VI new data I just put the Write VI in a loop, with DAQmx Start Task after the Write on First Call, and disabled regeneration. I haven't tested it yet, but it seems like it should work.  DAQmx Timing is configured for continuous samples; the producer/consumer loops will end when the TDMS read reaches end of file. 

Thanks again. 

[MatINEODefense]

Hello!

I found this thread and it happens I have exactly the same need, I'd like to know if you manage do your parallelisation. If you did, could you share how you did it please?

(DAQmx configuration and if possible your VI).

Thanks a lot

[Alisha_P]

Hi,

I recommend looking at this community example - it shows how to do a basic read in parallel to a write to file.

https://decibel.ni.com/content/docs/DOC-2431