Help? Looking for better approaches to mutlithreading, parallelism (producer/consumer)

[sid137]

 Hi,

     So II would like to do some parallel processing of data using the multiple cores of my computer, and I'm trying to find a way to do this in labview.  I deally, I would liek to be able to take a given task, put the work into a work queue, have threads from each processor take a task from the work queue and process it in parallel until the work queue is empty, and put the result into a results queue where I could then write the results to a file. So in a text-based languange, the implmentation would be straightforward.. but I'm not sure how to go about this in Labview. 

       I've read about the different designs patterns, and was trying to use a Producer/Consumer approach with queues. What I wanted to do was read a list of files from a directory, stuff the paths into a queue (Work Producer), then have a second loop (Work consumer/Results Producer) dequeue each file, process it, and send the results to a second queue.  Then at the end, a final loop (Results consumer) would read all of the results.

 My question now is..  is there a way to do this more cleanly than what I have on the diagram?  Right now, I have to know in advance how many parallel tasks I want to use, then I have to wire up each task before executing the program.  If I move to a different computer, I need to change the wiring digram.  Is there a way to abstract this to avoid having to make huge changes to the block diagram?  I was considering trying to make the parallel portion fit into a subVI so that I could just copy and paste several sub-vi's to easily add and remove parallel threads..    but its not so simple to have to keep track of all the queue references, and etc..  

Has anyone already done somoething like this? I feel like what i'm trying now is becomming a big mess

Thanks

[sid137]

My Results queue in the first image was misnamed (had changed to do a test).  Here is the correct image of the block diagram

 

[Ben]

You can run clones (instaciate from tempaltes) of the crunching code where each get a ref to the queue based on its name. When they see the queue is empty the terminate and close.

Make sure your queue is full be fore you lauch them.

There is supposed to be new property that tells you how many CPU you have but I can't find it at the moment.

Have fun!

Ben

[F._Schubert]

Place a wait in your top loop ('Check to see if parsing queue is empty')!

Felix