reducing noise of incoming data

[sayndesyn]

Yes, unfortunately I am using labview 5.1 as the company I am at does not want to pay for the newer software being that this is the only time they use it.  Understandable if this version still has the functions I need. Images or any sort of tips would be much appreciated as to where in the diagram I need to be tackling this issue.   Like I said before I am a complete Lab View newb, but I do have a technical background.  thanks, Sam

[sayndesyn]

One of the challenges with this project is that I need to smooth out the data, but it has to be in near realtime so that the tool operator is getting a reading and knows which part of the piece has the thick or thin wall as it moves past the ultrasound.  So it is not like I can just get all the data stored and then processed and fit to a function later.  In one of the above posts someone posted about a running average and a link to wikipedia.  That seams like a viable option.  One thing I am thinking about is whether a sinusoidal curve is best for the operator to find the highs and lows or more of a linear line with a definite peak.  Any more comments, ideas or different directions are appreciated.  thanks, Sam

[Cory_K]

I dont think you are going to be fitting the data to a function.

The data may not be best described by any given function if it is wall thickness.

A running average just deals with calculating an average of several close points.

x(n) = [ x(n-3) + x(n-2) + x(n-1) + x(n+1) + x(n+2) + x(n+3)] / 6

Since you will be doing this in real time, you can not do x(n+#) because there are no points ahead of the current point (because those have not happened yet). So you could only do an average of several previous points.

[WCR]

When trying to find the central tendency of spiky data, a median filter performs better than a moving average filter.  It is more robust, i.e. it is less likely to be fooled or pulled off target by the occasional outlier.  I have to estimate the time-varying diameter of a blood vessel determined from ultrasound imaging - maybe some similarity to your application.  In my case, a first VI does image analysis of the ultrasound video, and produces a diameter signal as output.  This diameter signal is spiky and neeeds cleaning up.  For this particular problem, a median filter gives better results than a moving average.

Background:  Median is a robust estimator of central tendency, i.e. it is less sensitive to violation of the assumption that the noise is normally distributed.  The moving average with flat ("boxcar") weighting, or triangular or other more exotic weighting parameters, is a linear finite impulse response (FIR) fiilter.  Butterworth and other classic lowpass filters are also linear.  These linear estimators are optimal estimators if the noise is normally distributed, but spiky noise is notoriously un-normal: for a given standard deviation, distant outliers are considerably more probable than the normal distribution predicts.  This means that the moving average gets pulled off-course more than it should by outliers, and a median filter does not.

Ref: Numerical Recipes in C, 2nd ed., section 15.7 (Robust Estimation)

I was unable to open your VI due to the old version.

Bill Rose

[sayndesyn]

What you are saying definitely makes sense Bill.

Do you have a mathematical example of how this function differs and essentially ignores spikes as compared to a moving average function?  It seems like it is essentially modeled like a high speed damper.  The quicker something changes, the more it resists.  I'm curious to see what function you are using to get this effect without simply being a low pass filter.  

Message Edited by sayndesyn on 06-17-2009 12:51 PM

[WCR]

The median of a group of numbers is found by ranking the values from low to high and taking the "middle" value.  If the group has an even number of elements, the average of the two middle values is used.  Examples:

median(1,2,3,4,5)=3;  med(1,2,3,4,10)=3; med(-1e6,0,0,1,2)=0.

Labview has a median function in the base version: on functions palette, select Mathematics -> Prob & Stat -> Median.  Feed it N points at a time, and it will return the median of those N points.

In version 8 and some earlier versions there is a "point by point median" function in a toolbox: tell it the "width" (i.e number of points to use) in the median calculations, then feed it the array of points, one at a time, and it will return the "moving median", point by point.  On functions palette, select Signal Processing -> Point by Point -> Prob & Stat -> Median PtbyPt.  You may not have this.

[WCR]

A snapshot of the block diagram and front panel of a simple VI is attached (sinc eyou have version 5).  It uses the Median PtbyPt VI which is part of the signal processing toolbox I think.  The simulated data used is also attached.  It is just a text file with a column of 400 numbers which I made in Excel: a sine wave with spiky noise.

Bill R.

[Coq_Rouge]

---

WCR wrote:

A snapshot of the block diagram and front panel of a simple VI is attached (sinc eyou have version 5).  It uses the Median PtbyPt VI which is part of the signal processing toolbox I think.  The simulated data used is also attached.  It is just a text file with a column of 400 numbers which I made in Excel: a sine wave with spiky noise.

Bill R.

---

Remember the PtyByPt concept was not invented in 5.1, show the poor man a snapshot of the PtyByPt also 😉

[WCR]

CoqRouge suggests I post a snapshot of the block diagram of Median PtbyPt, so someone without it can see how it works.  A good idea, but it calls other PtbyPt routines, which call others, so I think that will not be useful.

I was incorrect when I said earlier that I used Median PtbyPt.vi.  I used Median Filter PtbyPt.vi, which calls Median PtbyPt.vi.  The input value "window width" in my posted diagram is actually multiplied by 2 and 1 is added, internally in Median Filter PtbyPt.vi, before it calls Median PtbyPt.vi.  Therefore, if I set "window width"=5, a moving median using 11 points will be computed.  This moving median returned by Median Filter PtbyPt.vi is 5 points delayed relative to the raw waveform in this case.

I suspect that old base Labview has the built in function Median.vi which returns the median of all the points you feed it.  With some cleverness with shift registers and/or array indexing it is possible to step through an array one point at a time, taking tthe median of the last N points each time.  Good luck!

[sayndesyn]

Thank you very much for all the help guys, I know what needs to happen now.  Just need to impliment it.  Small detail, ha.  Time to get my hands dirty.