accessing a queue in multiple loops?

[deskpilot]

I am writing data to a queue in one loop, and passing that queue into two other parrallel while loops. I want to be sure that both dequeues will have access to all of the data. That is, one dequeue cannot remove a data point from the queue and prevent the other parrallel dequeue from accessing that data point.

In case I am unclear, I take data from a daq read in one loop, in a second loop I perform analysis on each data point to decide certain actions, and in yet another loop, I am saving this data. Is there any possibility of missing the data in either consumer loop?

[smercurio_fc]

The Preview Queue Element will provide the element without removing it from the queue. However, at some point it will need to be removed. In your setup when do you know that it's safe to do so?


Message Edited by smercurio_fc on 05-09-2008 12:02 PM

[Ben]

When analyzing my design and deciding what mechanism I should use interprocess communications, I will look for the pattern "One or many writters but only one reader" as an indication that a queue can be used. Please let me suggest that you just use two queues and save yourself a lot of headaches and work. This will let the queues just be queues and will also allow the two consumers to run at completely different rates.

Ben

BUt maybe we should wait fro Raven Fan to reply to see what he thinks.

Message Edited by Ben on 05-09-2008 12:09 PM

[RavensFan]

It might be better to create 2 queues and let your first loop put a copy of the data into both queues.  Each consumer loop will dequeue from its respective queue.

[deskpilot]

Thanks for the suggestions guys. Things are getting cluttered, so I was hoping the single writer queue would help to streamline the code a little, and some of the discussions I found on here made it sound like I would not have any trouble. But if there is any possibility for a race condition, then I will remove the chance and just use two.  

As far as the preview, it still seems like one loop could remove the element before the other has an opportunity to preview it. It is safe to remove the element only after both have read it. As far as how I know when that has happened .... the evaluation loop is looking for a short burst in a huge subset, if it missed that particular signal, I would know (and it would be very bad), otherwise, I don't know when either of the loops has processed the data until everything is done and I open the files.

Message Edited by deskpilot on 05-12-2008 11:50 AM

[RavensFan]

If you are worried about diagram clutter, it may be best to keep all the queue references together.  Either as an array of queue references or better, a cluster of queue references.  That will bundle up all the queue wires into one.  Then you can unbundle the appropriate wire as necessary in whichever loops you need it.

[johnsold]

You could use an Action Engine with the main data and two markers, one for each reader. The data source loop writes data to the AE. Each reader loop reads a quantity of data and updates its marker in a single call to the AE. From time to time one (or more) of the loops calls the AE with the Action = Remove All Data Read by Both. The AE itself keeps track of the data which has been read and disposes of it when it is no longer needed.

The AE would have Actions: Initialize (if needed), Write Data, Read and update Marker 1, Read and Update Marker 2, Remove All Data Read by Both, and possibly others. The inputs would be Action, Data, Quantity to Read, and Error in. Outputs would be Data, Updated Marker, and Error Out, and possibly, the quantity of unread data (unread by this reader).

This could easily be extended to larger numbers of readers. It could also be operated as a circular buffer with two markers per reader.

Lynn

[Matthew_Kelton]

Another option is to use notifiers.  Since you didn't post your code and I don't know what else you're trying to do in your consumer loops, it may or may not be a good idea.

Basically, your DAQ loop gets a data point/set and sends a notification with the data that the file and analysis loops are waiting for.  Then, both will grab the data and do what you need to do.

The caveat is that you can drop data if your consumer loops cannot keep up with your DAQ loop.  Although, if you have that issue, the queue solution will eventually cause problems as it falls further and further behind.


Message Edited by Matthew Kelton on 05-12-2008 12:11 PM

[Ben]

Lynn wrote ".... It could also be operated as a circular buffer with two markers per reader." so there is no need to do the "remove All"

Which is how I would do it before the days of the polymorphic queue or if the writer is on one machine and the reader is on another.

Ben

Message Edited by Ben on 05-12-2008 12:13 PM

[rpursley8]

You could also register a User Event of your Data type for each of your other loops.  You can then generate your User Event and the data would be passed to both loops.  

If you are using DAQmx, that can also be registered as an Event in your DAQ loop.  See attached.