topic Re: Slow rate calculation in LabVIEW

Slow rate calculation

IMSargon — Thu, 18 Jun 2009 16:35:29 GMT

The attached VI is my attempt to calculate the feed rate of a pump which pumps out of a container on a scale. The rate is very slow, on the scale of about 10Kg/h. The best way I can think of to calculate such a rate is to time how long it takes for the scale to change. I also need some ability to filter out noise (bumping the scale, tubes being hit, breezes, instability between mass changes). It works, more or less, but I feel it could use improvement.

Assume the indicator, "Pump Mass", is constantly being updated with the current scale mass (which measures pounds and converts to kilograms by using a factor of 0.45)

The intention of my code is to first check the mass once per second. If the mass does not change, it increments the timer. When it does change, and if it has been more than 5 seconds (that's my filter), it calculates the current rate and then resets the process.

1. Is there a better way to write my method than how I have written it?

2. Is there a method that is better than my method?

Re: Slow rate calculation

BrianInTheForest — Fri, 19 Jun 2009 15:47:28 GMT

I try to avoid local variables in my programming because they have bitten me more then once by causing race conditions.

I pretty much used your program but instead of local variables I pass the data with shift registers inside the loop.

The GetSetMass.vi is in a parallel loop just as an example but it can be used somewhere else in your program to bring in the updated Pump Mass information.

Re: Slow rate calculation

IMSargon — Fri, 19 Jun 2009 21:14:50 GMT

So your shift registers eliminate not only the need for the locals but also the need for the hidden indicators I was using to store data. The subVI is simply a FGV, is that correct? Is it a common practice to use many SubVIs in this way? Are there other benefits besides avoiding race conditions?

Thank you for your good advice.

Re: Slow rate calculation

GerdW — Fri, 19 Jun 2009 21:23:00 GMT

Hi IMSargon,

"other benefits":

Locals cause data copies, hence increase memory needs (bad when using big arrays). Making data copies almost always means slower execution too...

Re: Slow rate calculation

altenbach — Fri, 19 Jun 2009 21:55:31 GMT

Brians example is somewhat flawed, so you should make a few corrections:

The loop in the lower left spins way too fast, consuming all CPU. It only needs to spin if the value changes.
The sequence structure has no purpose.
For efficieincy, all terminals of the subVI should be outside any structures.
There is no way to stop the program, we can only abort.

Here's a quick version without subVIs. It's just a dumb refactor without locals. I would still watch for a better algorithm. 🙂

Message Edited by altenbach on 06-19-2009 02:55 PM

Re: Slow rate calculation

IMSargon — Fri, 19 Jun 2009 23:09:20 GMT

Loop spins too fast:

This is merely a placeholder for the actual code that reads the scale. You can throw a 100ms wait in there if you want, as that's how often I check the scale.

Sequence structure has no purpose:

Another benefit of using those shift registers instead of my locals. Thanks.

All terminals should be outside any structures:

That's interesting. What about doing that makes it more efficient? It looks like just moving the terminals outside the case structure would change the behavior. How would you change the subVI?

No stop:

As with the first flaw, this is just some demo code copied from a larger program so I could get some help. The larger program has a better stop function.

Your version is good, but would you change it if other loops in the program needed access to Pump Mass? With it in the loop, it seems I'd have to use locals again.

I'm always happy to hear a better algorithm. Using a timer rather than an increment each second would improve precision, but I never could get it working quite right. Actually, my algorithm is discussed on this other forum: http://mathforum.org/kb/message.jspa?messageID=6698550&tstart=0

I tried to implement a simple form of Hagman's first suggestion. If someone knows what he's talking about in his second suggestion, maybe that'd make a good algorithm.

Re: Slow rate calculation

BrianInTheForest — Fri, 19 Jun 2009 23:22:07 GMT

IMSargon wrote:
So your shift registers eliminate not only the need for the locals but also the need for the hidden indicators I was using to store data. The subVI is simply a FGV, is that correct? Is it a common practice to use many SubVIs in this way? Are there other benefits besides avoiding race conditions?

Thank you for your good advice.

Glad to help.

I use functional global variables(FGV) all the time... for passing many types of data amongst VI's.

I do use local variables for initializing controls.

Re: Slow rate calculation

altenbach — Sat, 20 Jun 2009 16:27:45 GMT

IMSargon wrote:
>All terminals should be outside any structures:
That's interesting. What about doing that makes it more efficient? It looks like just moving the terminals outside the case structure would change the behavior. How would you change the subVI?

I was talking about the FGV subVI by Brian. For a detailed discussion, see this post by Darren (read that entire thread for details).

No, it won't change the behavior if you do it right, it could for example look like that:

IMSargon wrote:
Your version is good, but would you change it if other loops in the program needed access to Pump Mass? With it in the loop, it seems I'd have to use locals again.

Not necessarily. (of course there is nothing wrong with a local here and there). Still, if you would use a queue, the loop would only spin when a new value is available. Keep track of the tick count and the code will know when the last significant change happenend.

Message Edited by altenbach on 06-20-2009 09:27 AM

Re: Slow rate calculation

altenbach — Sat, 20 Jun 2009 16:37:34 GMT

IMSargon wrote:
I'm always happy to hear a better algorithm. Using a timer rather than an increment each second would improve precision, but I never could get it working quite right. Actually, my algorithm is discussed on this other forum: http://mathforum.org/kb/message.jspa?messageID=6698550&tstart=0
I tried to implement a simple form of Hagman's first suggestion. If someone knows what he's talking about in his second suggestion, maybe that'd make a good algorithm.

I would just collect raw weight data every second and place a few hundred points (or whatever) into a FIFO buffer, then simply do some low-pass filtering or smoothing operation (e.g. fit to a polynomial of suitable order) to calculate the derivative at the newest edge of the data.

It seems way too convoluted to throw away data based on changes and such.

Do you have some typical raw data, e.g. weight reading every second for an hour or so to play with?

Re: Slow rate calculation

IMSargon — Sat, 20 Jun 2009 17:48:52 GMT

Raw data? Great idea! I've attached two files with example data. In both cases the pump is running at a constant rate. The two files are two different scales with two different pumping rates.

Ideally, we would like to know the rate at which we are pumping in real time. However, the scale isn't sensitive enough to do that. So what's the next best thing?

Message Edited by IMSargon on 06-20-2009 12:48 PM

Re: Slow rate calculation

altenbach — Sat, 20 Jun 2009 20:40:56 GMT

(OK, CSV means comma separated values, however you files are tab delimited. Be careful with extensions! :D)

I think all you need to do is use the pt-by-pt linear fit of the incoming data with a reasonable buffer size (e.g. 3000 points) and things look pretty good.

(The lower FOR loop processes the data one point at a time, calculating the instantaneous slope from the historical data. In your application, it would just read the current weight at constant intervals at each iteration.)

Here's a quick example.

Message Edited by altenbach on 06-20-2009 01:40 PM

Re: Slow rate calculation

IMSargon — Sat, 20 Jun 2009 22:24:44 GMT

@ altenbach

(Interestingly enough, the CSV file format is tab delimited by default in OpenOffice. While technically we should be using a DSV extension for such things, CSV has become the de-facto extension for any delimiter separated value file.)

This is really great. It only takes your software about 2 minutes to give a reasonable rate, while I was having trouble beating 10. Your rate does oscillate by about 10%, though. I need to mix my chemicals with an accuracy of about 1%. That's the tradeoff, of course, between using more or fewer points for the fit.

So here are the points I'm pondering:

- How to display the rate such that the user does not take it for what it's not (the instantaneous rate of the pump)

- How to filter out noise such as that seen in the second example data file (Repeat after me: "The chemical tank is not a table!")

I'll see if I can run this live on Tuesday. It's a good deal better than what I've been using. We'll see how that goes.

Re: Slow rate calculation

altenbach — Sat, 20 Jun 2009 23:40:36 GMT

IMSargon wrote:
(Interestingly enough, the CSV file format is tab delimited by default in OpenOffice. While technically we should be using a DSV extension for such things, CSV has become the de-facto extension for any delimiter separated value file.)

I think the de-facto standard is tab delimited (execel text export, LabVIEW (write/read spreadsheet file, spreadsheet string to array, etc.). If you call a tab delimited file *.csv, excel does not correctly open it because it expects commas. From a global viewpoint, comma delimited is probably one of the worst choices, simply because some countries in Europe use the comma for the decimal point, so doing a CSV would chop the data up beyond recognition. 😉 So if you ever think your company will expand and sell overseas, don't do csv! 😄

IMSargon wrote:
This is really great. It only takes your software about 2 minutes to give a reasonable rate, while I was having trouble beating 10. Your rate does oscillate by about 10%, though. I need to mix my chemicals with an accuracy of about 1%. That's the tradeoff, of course, between using more or fewer points for the fit.

I think the oscillation is real. Look at the data: There are areas that stick much longer to one value than others.

In any case, there is no way to get better info out of your raw data. You have to work with what you have. If the slope is relatively constant between experiments, you could incoporate that knowledge somehow. Right now, there is no way to tell the slope in the early data if all readings are the same.

If your chemical reaction blows up if you're off by 1%, you simply cannot do it this way.

Maybe you could place a massflow controller on the drain or measure hydrostatic pressure at the bottom of the tank for more accurate data?

Instead of a single indicator, you could display it on a chart.

To eliminate the "glitch" in the second example, you need to incorporate knowledge, e.g. ("the weight can only go down") and filter accordingly. This is tricky. What actually happened there? Did some Hippo take a swim in the tank?

Re: Slow rate calculation

IMSargon — Sun, 21 Jun 2009 01:21:25 GMT

To see if the noise is real, I generated some perfect data, where the mass decreases by 1 kg exactly every 112.5 seconds. The results are below. (I also multiplied the slope by -3200 to give me fed Kg/h and autoscaled Y)

The shape of the oscillation does closely resemble that in our previous example data, besides the different frequency.

Edit: I had someone take a look at it. It sounds like what I need to do is run the input data through a lowpass filter. I'll see how that works out.

With just the feed rates, you're right, there just isn't enough information. With these improvements so far though, and coupled with the other inputs from the process (mixer mass, etc) I'm getting closer to that goal than I had previously thought possible. You know, we've been trying to convince my boss to get a mass flow meter or a hydrostatic pressure guage for a year now. Oh well.

"The weight can only go down" is exactly what I'm planning on doing. No time this weekend, but maybe Monday I'll put everything together. I'm not sure exactly what caused that glitch, but maybe I should test a sample for evidence of dissolved hippo 😛

Message Edited by IMSargon on 06-20-2009 08:21 PM

Re: Slow rate calculation

altenbach — Sun, 21 Jun 2009 01:37:46 GMT

IMSargon wrote:
To see if the noise is real, I generated some perfect data, where the mass decreases by 1 kg exactly every 112.5 seconds. The results are below. (I also multiplied the slope by -3200 to give me fed Kg/h and autoscaled Y)
The shape of the oscillation does closely resemble that in our previous example data, besides the different frequency.

This is similar to "spectral leakage", caused because the history buffer is not a multiple of the step lenght.

How is the weight measured? What kind of hardware is this? Since the problem is due to limited digital resolution, I don't think a direct lowpass filter would help.

Another option could be "dithering". Simply add random noise corresponding to about the step size followed by lowpass filtering?

Re: Slow rate calculation

IMSargon — Sun, 21 Jun 2009 02:12:41 GMT

The scale has an increment of 1lb, so I do have a bit better resolution than I have shown in my example data, but ultimately the problem is the same.

The value comes in as ASCII over RS-232

I tried toying with the point-by-point Butterworth filter. It was successful at high sampling frequencies, but then it took a long time to get up to the real data.

> Simply add random noise corresponding to about the step size followed by lowpass filtering

Really? That would be a brilliant solution if it works. I tried a method by which 1 is added to or subtracted from each data point, at random, before passing through a butterworth filter. In my imperfection, it makes it worse. Does it work for you?

Re: Slow rate calculation

altenbach — Sun, 21 Jun 2009 02:22:28 GMT

Dithering needs to be done on the analog circuit, before the ADC. This is not something that works digitally.