Processing large amounts of Data from DAQ

[SLOwDown]

So I have an application that gathers a lot of data from instruments (20 MS/s on three channels). I have been trying to figure out how to use producer consumer loops to process this data. The data must be separated into chunks of approximately 1.5 million samples to be processed. The location where the data needs to be separated is determined by a dll that I wrote which monitors two of the channels and finds where to split the data. My initial thought was to use a shift register to store the data and then have my dll figure out where to take the chunks, and then place the data in a queue for the consumer loop to process one chunk of data at a time. However, I am not sure is this is the best way to realize this.

Basically I have:

• Data coming in from three channels (about 65000 samples at a time)
• That data needs to be combined so that my dll can find where to split it into chunks
• another loop should receive the chunks for further processing.

If anyone could point me in the right direction that would be awesome. I have been looking into circular buffers, the queue system, and shift registers; however, I am not sure about how to do this.

[RavensFan]

When I read your message, I thought it was a repeat of this other message thread I saw earlier today.

Collect data in chunks from continuous data stream

I'm hoping that is not you using a different username.  You might want to read that one since it sounded like they wanted to do the same thing you are asking for here.

[SLOwDown]

Definitely a different person! But that is similar and that solution helps me a little. The biggest issue I'm running into is how to break the data into chunks without knowing the exact length of the chunk, as it depends on one of the signals. My thought was that my dll could return a index on some sort of buffer and the data before that could be removed from the buffer and put in a queue, without disturbing data after. (As it would be part of the next chunk). 

[Yamaeda]

---

@SLOwDown wrote:

The biggest issue I'm running into is how to break the data into chunks without knowing the exact length of the chunk, as it depends on one of the signals. My thought was that my dll could return a index on some sort of buffer and the data before that could be removed from the buffer and put in a queue, without disturbing data after. (As it would be part of the next chunk). 

---

That sounds like a strange design, but yes, you'd have to analyze the signal and cut accordingly. Store the remains in a shift register in the consumer and when you get new data, append the data and redo the analyze+cut, it shouldn't be too hard.

/Y

[Kevin_Price]

This sounds fairly challenging to me, due to the combo of high data rate (20 MS/s on three channels) and the need to break it into chunks of unknown & variable size.

You'll want to handle this data in ways where you aren't allocating and releasing memory on the fly.  You'll want to minimize data copying too.

Offhand, I'd question why you've chosen to retrieve data from DAQ in sets of ~65k samples at a time.  That means your data processing work has to cycle every 3 msec.  I'd probably want to aim for something more like at least 300k-500k samples at a time.   Each read has overhead, each accumulation of retrieved data takes time, each search for split criteria takes time, each assembly of ~1.5M samples takes time -- it adds up.

    And maybe it'd prove beneficial to retrieve *more* than 1.5 million samples at a time.  That'd reduce the impact of read overhead and accumulation time. 

Just some thoughts.  Also, do you know that the dll can operate on the data without making copies or doing memory allocation?

You may benefit from structuring your code in a way that lets you do your processing with a parallel worker pool.

-Kevin P

[SLOwDown]

So the 65 000 points at a time is out of my control as that's the rate that the DAQ outputs data while in streaming mode. In block mode it could do larger chunks, but then a substantial amount of data is lost between blocks which is not acceptable for our purposes.

The dll was written with that in mind so it does not make any copies or use any unnecessary memory. I've also tested the dlls using a c++ test environment and they perform more than fast enough. 

I'll look into the parallel worker pool stuff. Thanks!

[Ben]

You may also want to consider a "bucket-brigade" approach where break up the work into fast tiny steps and use queues to pass data between loops running in parallel each doing part of the work and passing data to the next loop.

I was able to keep up with 200MHz of 32-bits digital data stream live and keep up with it using that approach. In that case 1 bit was the clock and another bit was the data that was left channel on the rising clock edge and right channel on the falling clock edge.

Ben