Bad points at beginning and end of filtered data?

[Jeanius]

I'm sending an array of acquired values into a 6th order zero-phase 1Hz lowpass butterworth filter and into another zero-phase rectangular smoothing filter.  The filtered data seems to have "tails" towards the beginning and end, which I'm somewhat familiar with, but unsure as to how to remedy.  I checked my original data, and, noisy as it is, it does not drop as far as this does.  Do I just need to clip off the first and last few points of data, or is there a more elegant solution to this?

Attached is an image of my filtered data and its derivative.

[mikeporter]

It depends on what you want. If the beginning and end points are really there and are part of the data that you are interested in, trimming them of prior to filtering will make your output "prettier" but it won't be showing a complete record of what is happening. On the other hand, if the anomalous start and end points are themselves artifacts of something else, or outside the scope of the data that you are interested in - trim away...

Mike...

[Jeanius]

The beginning and end are definitely there, but don't tail downwards as shown in the image.  The original data is a noisy exponential slope, but after filtering, the tails show up, which I think is a product of the IIR filter.  It's essential that it be a zero-phase lag fitler, timing is important.  Is there another method to apply a zero-phase lag butterworth filter that doesn't create these tails?  If not, is there a predictable way to determine the length of the afflicted region of data so that I can program in a sequence that cuts that data out?

[chilly_charly]

A possible improvement would be to add a n points (n = filter order) at the beginning and end of your data (using the first and last array values...) , proceed the data, then slice off the same number of points. 

[Ben]

I suspect CC will jump all over me about breaking some math rule (like taking the log of a ngative number) but this may help.

WIth a zero-phase shift filter the data at the front and back of the data set suffer from edge effects. To get around this issue I have;

1) Reflect* the data set about the first point and prepend that data to the front end.

2) Reflect the data about the last and post pend that to the back end.

3) Do the zero-phase filter.

4) Toss all of the data that coresponds to the pre and post pended data.

* By "Reflect" I mean manipulate the data set such that it looks like you are holding a mirror up to the end of the plot. At the front end this means reversing the array order and negating and shifting the data relative to the first point. Similarly at the back end.

Ben

[Jeanius]

Ah, hey, that is a good idea.  Thus far I've just been removing 3/dt of the data points from each end of the data sets.  I'll give that a try.  It may be that this method will still cause the data to come out bad, due to false averaging/filtering, but if the error is low enough it'd be nice to maintain the entire data file.

[chilly_charly]

Hi Ben ! 🙂

Although you claim steadily that you are not a math wizard, I must admit that I learned a lot from your last post. 😮

My solution solved the problem caused by the data offset at the edge of the data set, but there was still a discontinuity on the first (and higher order) derivatives at that point.

Your approach is much better since it solves the higher order discontinuities problems. 

Attached is my interpretation of the previous discussion. There is clearly no edge problem at the end... Any comment ?

Message Edité par chilly charly le 08-29-2008 04:58 PM

Message Edité par chilly charly le 08-29-2008 04:59 PM

[chilly_charly]

Some problems with the forums ?

I could not attach the vi to the my previous post. I got a message "Unexpected error" 

And there seems also to be a bug with the front panel image : some controls have been only partially copied. I'll investigate that later. 

[Ben]

Thanks CC!

I'll try to recreate that VI you posted because I expected better results but I never tried it with a sin wave only real data.

If you happen to have that VI handy and can post it (LV 8.5, no 8.6 at home yet) I'd apprecieate it.

Take care,

Ben

[chilly_charly]

Trying again...

Edit : There is obviously a problem with the forums : I could not post the vi (Unexpected error), but I was successfull with the .zip archive.

Message Edité par chilly charly le 08-30-2008 01:54 PM