Producer consumer for data acquisition

[DarkMarc86]

Hi everyone,

I have been doing data acquisition for a number of years and I have recently changed my code into the popular producer-consumer architecture. I'm now sometimes getting an error when I stop the data recording to a file. I was hoping somebody could take a look at it and give me some pointers in the right direction. I know it's big and messy, but you can ignore most of it. Ignore the large flat sequence structure in the top left. You can ignore the event structure in the producer loop, except for the 'record button' case. You can ignore the IIR filter and the resample functions inside the producer loop as I have tested these thoroughly.I will try and descrbe the consumer loop. First convert from an array of waveforms to an array of doubles, then I add offsetts and multipliers (this but of the code is well tested), then I take that data and plots a few graphs, and output to a large 2D array of clusters on the front pannel, as well as checking for threshold values and changingbackground colours, The only two things that I would like somebody to have a look at in the consumer loop are the two case structures: the single shot finction and the record function. I am not sure if these are being called as efficiently as possible. I have a feeling that I am not updating variables correctly and it's causing a race condition and rather than do a patch fix, I would rather correct the whole architecture of my program to get this right once and for all.

The hardware I am using is SCXI equipment and I am acquiring data from 150 strain gauges.

I would greatly appreciate any comments. Thanks in advance. (The main VI file and offsett file are attached)

[Basil-B]

Hi DarkMarc86,

Could you elaborate on the error you are getting, is there a number associated with it and what does the text say? Looking at your loop I don't see any immediate cause for concern but perhaps the error will help clarify things. I did however notice a few things that would make your code be more efficient and quite possibly could be the reason for your error. In the recording case structure within the while loop, you never close out the reference that you make to the file. See how there is a teminal out for the blue wire on the write to text file VI? You need to wire that into a close reference VI outside the while loop. Furthermore, rather than local variable referencing the record button you could just put another instance of the record control connecting to the case terminal of the case structure just as you use it in the event structure. I am also curious why you have the event structure record stuff inside the while loop. Couldn't you take the record part outside it as an event structure. That whole while loop will wait until the event structure to either process an event or timeout before it proceeds to the next iteration.

Regards,

Basil Beirouti

Applications Engineering

National Instruments

[DarkMarc86]

Hi Basil,

Thanks for your comments.

The error says "Error 6 occurred at Get File Size in Bombardier Melbourne E class tram carbody testing v5.vi." I believe it's occuring because the recording in the case structure is still continuing for whatever reason, even after I stop recording and because "refnum out" no longer exists, it cannot feed into the "get file size" function.

When you say "put another instance of the record control," do you mean to simply have another wire going from the output of the record button, to the input of the case selector in the case structure?

I'm not exactly sure why the event structure is inside the while loop. I've read i a few articles about producer/consumer that if you have an event structure inside the while loop it ensures that the producer loop always executes before the consumer loop. Source: https://www.ni.com/en/support/documentation/supplemental/21/producer-consumer-architecture-in-labvie... (section 4, synchronisation).

Can I add one more thing. In future I hope to expand this program to be able to record all 160 channels at 1000Hz simultaniously. At the moment, if I run it and disable the downsampling, the best I can get it to record at is about 7-8Hz, and the recording periods aren't consistent either. Am I trying to do something thats impossible, or is my code just inefficient? The DAQ card we have has a bandwidth of 200kHz so I don't believe I'm trying to do something that it's incapable of. Perhaps it's a hard drive limitation?

Thanks in advance.

[Basil-B]

Hi DarkMarc,

First of all I want to commend you for using the producer consumer architecture, which is a very powerful and versatile programming architecture especially suitable for the scenario you are working on. You have properly made the architecture for the P-C loop but there is just so much going on in your while loop that it cannot execute fast enough for the sampling frequency you want. I am also very confused by what you are trying to do. Concerning the event structure, you are right about how it initializes one loop before the other, but do you necessarily require one to be initialized before the other? Either way, the way it is done is different in the page you linked, the event structure you have is waiting on a change in the value of the record button while the one in the example is waiting on a change in the status of the queue (the queue being obtained to begin with). In order to speed up processing time, do all your reading in the producer loop and then enqueue the elements. Do all your recording and processing in the consumer loop by dequeueing the elements over there. That is the reason the functionality is so powerful, it allows you to start taking new measurements before you're done processing and recording your older measurements. Furthermore, try to open references outside your while loop and open the file outside the while loop so that you are just doing the writing inside the loop. Keep in mind everything you have in the loop needs to run every time the loop iterates, so the less inside the loop the better. If you can try to simplify the code it would make it much easier for both of us to diagnose the problem.

[JÞB]

You are thinking pretty straight.  What you have is a race condition caused by an improper implementation of the Architecture.

The Producer (Data) loop should not be handling UI Events. It should be responsible for acquiring data  Use a UI Loop to control those events.

The UI loop should not be opening or closing File References. It should be handling User Events and passing commands to the Data Processing and logging loops.

The data Processing and Logging loops should not be polling the UI (Reading a local Variable and writing an indicator) They should be reading a notifier written in the Record Button Value change event to determine if logging is desired by the user.

Because your loops are overcoupled (Doing things another loop should be responsible for) you wind up using local variables to pass information between loops.  In this case that lead to the race condition when "Record Button changes Value the file reference is closed HOWEVER the other loop had already read the now stale value from the Local (Probably the First thing done each loop iteration)

and so when the data arrives at the case the true case executes and since the file got closed the refnum local returns 0 and you get error 6.

Here is a snipette derived from a similar project template that ships with LabVIEW 2012.  It demonstrates the consept of keeping unrelated actions seperated into individual code groupings.

[DarkMarc86]

Thanks Basil and Jeff for the comments. I'd like to start modifying my VI to incorporate all of your sugegstions. Jeff, I am using DAQmx functions in Labview 8.0 and I'm seeing a lot of functions in your example that I don't recognise. I'm particularly confused about the function which feeds into an 'unbundle by name' with the three outputs; acquisition, logging and UI. What is this function and how can I replicate it in my version?

Marc Listmangof

[tyk007]

---

@DarkMarc86 wrote:

Thanks Basil and Jeff for the comments. I'd like to start modifying my VI to incorporate all of your sugegstions. Jeff, I am using DAQmx functions in Labview 8.0 and I'm seeing a lot of functions in your example that I don't recognise. I'm particularly confused about the function which feeds into an 'unbundle by name' with the three outputs; acquisition, logging and UI. What is this function and how can I replicate it in my version?

Marc Listmangof

---

What Jeff has done is shown the block diagram of the main VI produced using one of the LabVIEW 2012 Templates. The template produces skeleton code that can be filled in for the application. In this case, the VI before the Unbundle is creating 3 x queues (refnums contained in that cluster) for passing messages between the loops as well as sending an Initalize message that is clearly application-dependent. The Template produces other aspects of that framework as well as that bloock diagram you see there. It sounds like you don't have LabVIEW 2012 (you mention 8.0) so unfortunately this template is not available to you in its current form.

[DarkMarc86]

Do you think it's worth upgrading to the latest Labview version so that I'm able to reproduce code like this or can it be done in Labview 8.0. Our main application for Labview is data acquisition and I understand there's not many new features from 8.0 onwards for data acquisition.

[tyk007]

---

@DarkMarc86 wrote:

Do you think it's worth upgrading to the latest Labview version so that I'm able to reproduce code like this or can it be done in Labview 8.0. Our main application for Labview is data acquisition and I understand there's not many new features from 8.0 onwards for data acquisition.

---

I believe TDMS (https://www.ni.com/en/support/documentation/supplemental/06/the-ni-tdms-file-format.html) was introduced since 8.0 and could be of value for you (thats the reason I have been unable to easily down-convert, sorry). However the basic framework here is what you can see - multiple loops with queues between them. It's a standard design architecture and it sounds like you're pretty much on top of the basics. If you wanted to, you could create yourself a basic framework much like this one using the standard methods you have already been using, and then customise that for your applications.

As far as upgrading to a newer version goes - there are obviously many new features since LabVIEW 8.0 - it's down to you really whether any fo the new features will offer you enough value that out-weights the cost. I don't know enough about your application or business to comment on whether these project templates alone warrant spending the money (given they are intended to save engineering time). You could investigate with your local NI sales staff to see whether an upgrade is justified based on this or other features you see.

[DarkMarc86]

Jeff do you happen to know where I can find a similar producer consumer block diagram example like the one you posted, except for labview 8.0? I'v been trying to replicate your architecture but it's very difficult when I don't know what half of the functions are.

I am really puzzled by the UI loop, and the UI data source. I don't understand where on the diagram the actual front pannel 'control' icons reside?