remove outliers or not just outliers but wrong data

[BigBoi]

Sorry, posted in the wrong board at first. Maybe someone can remove it from Most Active Hardware Boards : Machine Vision. Thx

So, here's the problem. I've got a laser vibrometer which can measure vibration behavior of ultrasonic speakers by sending a laser beam to the surface and measuring the Doppler frequency of the reflected beam. This works pretty fine but the problem is that there is lots and lots of wrong data because the surface of the speakers are not totally flat and reflecting.

If you look at the data you could find a curve that matches the data. I am using a median filter in first place which does a pretty neat job. You can see it in the pictures. In the next step I devide the data arrays and try to make each area look better with polynomial fit. I can't do it for the whole area because I guess best way would be Levenberg Marquardt but I don't have a function for the expected curve. They can look pretty different.

So the main problem I have are the areas where you can't see the curve in the measurement data (hope you see what I mean when you look at the pics). The median filter doesn't help there because there are too few right data values. What would be the best way to filter this data. I tried to build a filter that cuts out areas where the standard deviation exceeds a given limit. This didn't work very good (maybe it was my implementation). Would be happy for any idea of how to fix the problem.

http://img244.imageshack.us/my.php?image=measurementdatalu1.jpg

http://img83.imageshack.us/my.php?image=medianqd6.jpg

http://img244.imageshack.us/my.php?image=polynomialuy2.jpg

I know it doesn't look so bad on the pics but it does a lot of the time. When you need the fault everything works perfect. ^^

Just one more thing. Is there any chance of evaluating the data optically. Like a person could take a pencil and draw a line where the data values make it possible. Didn't find anything that would help here. Maybe someone has the same problem.

[Ray.R]

Do you have LV-vision tools?  It would do what you described at the bottom of your post.

What you described in your attempt was to essentially create a mask to remove the unwanted "noise" in the measurement.  That should work fine as well, however, it may be tricky when getting closer to the desired data (threshold level & intensity).

The approach also depends on what you do with this data?  Will it be used during data acquisition? If so, using the vision tools to "draw a line" over the curve will likely be too slow. If this is done after the acquisition, then it will probably be okay.

Can you provide more details on what you are trying to accomplish (overall)?

Thanks,

RayR

[BigBoi]

Thanks for the quick reply RayR.

Sorry, I don't have LV Vision tools and I'm working with LV7.1.

I just want to filter areas where there's only noise and find a fitting curve for the rest. It's all done after the aquisition so there's time for a slower algorithm.

What would be the best way to remove the noise?

Sascha

[Ray.R]

I have a crazy idea in mind which is a bit hard to describe.  I may have to write an example in LV7.x for you.

Basically, it would create a mean over a number of points to create a virtual line.  The number of points for the mean would match the original samples.

The means would be used to define a range (+ / - values).  You compare the array with the range array (I think this is polymorphic and would do it without a loop).  You will get an array of 1's & 0's. 1's if it's within range.  You then mask the original array with this mask.

Just a crazy untried idea.  If I have time, I'll experiment with code and post it.

Can you post raw data for me to try?

Thanks,

RayR

[BigBoi]

Hi RayR,

here you got some raw data. Hope you can read it. I used the Express-VI to write LV data to a file. Looks weird in notepad and wordpad but better in MS Word. ^^

I guess your idea won't really work. For those areas where there is not so much noise the median filter does the job pretty good. And the mean value of the other areas won't help because its only noise in some areas. Maybe you can see it in the attatched data. It's a pretty good example. But I appreciate any help or idea that could help.

Thx in advance.

Sascha

[BigBoi]

Now I have built a filter that just calculates the diffrence between two neighboring values and compares the absolute difference to a given maximum. And if its bigger the data value will be deleted. So you can decrease the maximum step by step until you cancel out most of the noise. This works good for my application. When there are only a few outliers left, I use a median filter to remove the rest.

For someone with the same problem:

[lmtis]

Hi Sascha,

I could not open your data file in LV.  Could you zip it in a plain text of spreadsheet format?

[DFGray]

You may also want to try a more robust fitting technique on your raw data.  A Lorentz distribution fit using a downhill simplex works well in this type of situation.  For details of this technique, look at the chapter on data fitting in Numerical Recipes in C (older online versions have all the info you need - see chapter 15, section 7).

[DonRoth]

Hi Damien:

Have you coded anything like this up yet in LabVIEW?  If not, I may give it a shot.  I am working with some data right now that when  modeled and curve fit as lorentzian + guassian, subsequently provides the most accurate peak detection.

Sincerely,

Don

[DonRoth]

ps. The .pdf links for numerical recipes on the main page you refer to do not seem to want to open.  However, the express VI on curve fitting looks to be a good place to start developing the model.  So does the guassian curve fit VI if just modeling as guassian....Don