Collect data in chunks from continuous data stream

[coolhandLV7]

I'm trying to capture data in chunks, say 20 arrays of data at a time, from a continuous 1D array data stream.  I'm sure someone has done this i have searched but cannot find info on this topic, probably not wording it correctly in the search..

I'm using produce and consume loops to queue the data, my struggle is with grabbing 0-19 and then 20-49 and so on to create a 2D array (data chunk) for analysis.

[crossrulz · Accepted solution]

On mobile, so I can't look at your code.  But assuming you are queuing up a 1D Array for each scan in the producer, what I do is put the Dequeue in a FOR loop as part of the consumer loop.  You can set the FOR loop to run 20 times and autoindex the dequeued arrays, giving you a 2D array of 20 scans to process.

[coolhandLV7]

Nice I think that worked, thank you.  Pretty simple, I've attached the updated code in case it cold help someone in the future.

[crossrulz]

A couple of things I would change with your code:

1.  The consumer needs to close the queue.  If you let the consumer close the queue, then you will lose data.

2.  You don't really need the second queue to stop the consumer.  What I do in a similar situation is use the queue data to tell the consumer when all of the data is used up by sending an empty array.  So when the consumer reads the empty array, it knows to stop.