averaging streaming data

[deskpilot]

I am taking analog measurements on an SCXI 1102C through a PXI to LV2009. I am running the DaqMx Read 1d Waveform at 100 Samples/second in a while loop that iterates every 45 - 50 ms. This means I get about 5 data points per while loop iteration. Even though the program is windows based, it has a fair amount of determinism built in. I now need to make a decision based on an average of 15 samples, preferably 25 - 50.

I can't slow the while loop down because their are other channels in the data that need to trigger things at the quickest rate possible in windows. I can live with the 50 - 100 ms latency for these, but not the 500 ms I would get if I slowed the loop down to perform this new task.

The only two options I've come up with are to use five shift registers on the while loop, but I really don't like this idea. Or open the TDMS file while I am writing this data to it and read the .25 second block that I need to average to make a decision. Also seems like a bad idea.

A third option, I think would be to place a subVI that writes this channel to five different functional locals and constantly updates them creating a running buffer. This doesn't seem like a bad idea, I just haven't figured out how to implement it yet.

Any obviously better options that I'm missing?

[Broken_Arrow]

I have used Mean PtByPt inside the loop with good results - it is very fast. I write to file in a parallel loop.

[RobCole]

One of the most common structures is run parallel loops. One loop would handle data acquisition while another loop handles saving to file. This also allows another parallel loop to read the data and make decisions based on that data. It would be reasonable to set up a functional global to use as the buffer for this data.

Another benefit of this type of system is that the loop saving to disk does not have to run at the same rate as the acquisition loop. Makes disk access more efficient if saving data in larger chunks.

     Rob

[deskpilot]

I do write the file in a different loop by using queues. Their is also a parrallel output loop that gets information based on global booleans written to in the read loop.

The Mean ptbypt VI will only work once I have compiled the data from the previous five loop iterations. Looks like a functional global based buffer is the way to do this (I always call them functional locals, forgot that was the wrong name) . Hoping somebody could point me towards an example of some sort to give me a jumpstart.  

[Broken_Arrow]

---

Robert Cole wrote:

.....

Another benefit of this type of system is that the loop saving to disk does not have to run at the same rate as the acquisition loop. Makes disk access more efficient if saving data in larger chunks.

Rob

---

Yep. When creating data fairly quickly like the OP suggested, I like to Queue up 50 or 100 lines of data then write that chunk to disk, with the exception of errors, which get written immediately.

As far as the averaging goes, the speed suggested doesn't seem prohibitive of putting averaging code inside the data loop, IMO.

[deskpilot]

So how do you queue the 100 data points when each iteration only outputs 5. Do you read Queue status and wait until it has 100 data points in it and then dequeue?

[RobCole · Accepted solution]

There are a couple of different methods that can be used. If the write loop is running slower than the acquisition loop, then the functional global has to keep a number of samples in it anyway. This allows for more data to work on when averaging.

The other is to use a running average. I have attached a VI that does that. It is a small functional global on its own.

     Rob

[Broken_Arrow]

---

deskpilot wrote:
So how do you queue the 100 data points when each iteration only outputs 5. Do you read Queue status and wait until it has 100 data points in it and then dequeue?

---

Yes, and you can set the Queue size to 100 / preallocate the queue to better manage memory.

Note: Mean PtbyPt is also a FG. I don't see why you can't use this, or the other posted VI, to get your averages.

Message Edited by Broken Arrow on 12-17-2009 01:04 PM

[deskpilot]

Robert, thanks that is perfect. It took me a few guesses and checks to get it to work with a waveform, but that is simpler than what I was chewing on.

Broken Arrow. I didn't realize how the ptbypt works, but now that I do, you can still only put in one data point at a time, unless I'm missing something. While you can wire a waveform to this input, when you do it is only actually looking at the last data point in that waveform packet.

What I mean is, each iteration of the loop, the Daq Read outputs a certain number of data points. That number is approximately equal to your acquisition rate setting on your DAQ Read, divided by your loop execution rate. In my case, the acquisition rate is 100 Hz while the loop execution rate is about 20 Hz meaning each channel outputs about five points per loop iteration.

While you can wire a waveform to scalar indicators and scalar inputs like the one on the ptbypt VI, if my data points for a single read of a channel are 1.0, 1.1, 1.2, 1.1, 47.5, then the scalar indicator or input only sees 47.5. (Not that my data looks like this, just giving an example)

I need to average five sets of those five points. If I do what you suggest I would only be averaging five out of the twenty five points.

OH, and I swear I'm not giving you a hard time, but in regrads to your other comment this is straight from the help; "max queue size only limits the number of elements in the queue and does not preallocate the queue." You can preallocate to manage memory, but setting setting the queue size on obtain queue is not how you do it.

Thanks for all of the help and suggestions.

Message Edited by deskpilot on 12-17-2009 01:52 PM

[Broken_Arrow]

---

deskpilot wrote:

OH, and I swear I'm not giving you a hard time, but in regrads to your other comment this is straight from the help; "max queue size only limits the number of elements in the queue and does not preallocate the queue." You can preallocate to manage memory, but setting setting the queue size on obtain queue is not how you do it.

Thanks for all of the help and suggestions.

Message Edited by deskpilot on 12-17-2009 01:52 PM

---

That's correct. I did not imply that setting the size of a Queue implicitly preallocates memory.

p.s. Robert's VI takes in a DBL, just like PtbyPt.

Message Edited by Broken Arrow on 12-17-2009 02:04 PM