Best way to find max slope on noisy signal

[Bandit]

Hi,

I have an application that logs a sudden pressure rise. I need to find maximum rate of rise. Point by point methods are unreliable in this instance. Although I wouldn't consider the signal excessively noisy, there is enough noise to cause significant errors in a point to point method. Noise appears to be of no particular frequency, just white noise. The relationship between the level of noise and the general rate of rise means I can get max dx/dt of twice the expected rate at the moment.

I'm seeking advice the best solution for this.

The options I'm considering to obtain a noise free curve are...

1. Find a best fit line through the data.

2. Smooth the data with some form of running average - not so convinced by this second option.

3. Filter the data (unsure how best to do this).

With any of these I would then perform a point to point fit as before.

Any opinions on which is best?  Or another way that I haven't spotted?

Apologies for no illustrative data. I'm at home, and not in the office for a few days, but I had to post as this is bugging me.

Thanks for any help.

Regards,

Bandit

[altenbach]

---

Bandit wrote:

1. Find a best fit line through the data.

---

IF you have a good model (e.g. an exponential), this is definitely the preferred way to do things.

I would be curious on how some typical data actually looks like, so please post it the next time you are in the office.

[Bandit]

Thanks for the reply Altenbach. I'm not the only one looking at the forum on a Sunday then!

Sadly I don't have a good model. The experimental data will rise from a base level, the rate of rise will vary considerably, and it will attain a peak value, which it may or may not hold.

(or if the experimental event fails to occur - just a long flat line! )

I'll try and convert some data to a postable form when I'm in the office. This application is the first where I have used TDMS format for storage - anything previously and I'd have a handy .csv file ready to post.

I could reduce noise by sampling at a faster rate and averaging adjacent points to get back to the required rate.

I could also find the area of the curve where the slope is steepest, and then average a number of slope results in this area. I don't expect to find sudden changes in the steepest part of the curve.

Regards,

Bandit. 

[Bandit]

 Hi Altenbach. Data is attached, with a .vi to view it (LV8.5, WinXP)

I exported the data to a csv file from the tdms file I use for storage. It has 3 columns, with explanitory headers, time, sensor1, sensor 2.

I cut the data file down a lot, the original is a couple of seconds long at 50kHz sampling rate. Here is just the region of interest with the pressure rise, but noise levels are the same throughout the full length file.

This data was captured from pressure sensors in a chamber at ambient pressure, with a second chamber at higher pressure connected via a manually operated valve. Rise occurs when valve between chambers is opened. After pressure peaks, it looks like the pressure wave reflects around the chamber a little. I can't be sure how close this simulation is to the real thing, but during real tests I expect to see a range of pressure rises both faster and slower.

Anyway, if you zoom the chart you can clearly see that point to point dy/dt will vary considerably and not be representative of the slope I'm looking for.

I took a look at the curve fitting .vi's in the mathematics pallette (I'm using LV pro BTW) and I guess your comment about knowing what model to use stems from these.

I hope this example proves useful, and look forward to your observations.

Regards,

Bandit.

[Bandit]

Oops - see attachment

Message Edited by Bandit on 06-25-2008 12:51 PM

[IanW]

If the data you gave is typical you may be OK with a filter and dx/dt. If you look at the FFT you'll see that the noise is at high frequencies and not white noise. Take a look at the modified vi.

[Ben]

[Warning: Ben's comments on math are always questionable]

I have had good results by using a zero-phase shift low pass filter to condition the signal before doing a point-by-point slope and then search for max. If you need to be able to evaluate the the most recent reading ( in zero-phase shift filters you loose info at start and end of data set depending on the order of the filter) I have had succes by reflecting the data set about the final point, then filter, and then toss the data that was added during the reflection. This approach was used in an net-based just in time inventory system to detect when a delivery was made without humans getting involved.

Ben

[johnsold]

Here is a modification of your VI with a "linear fit to a segment of the data" method. It manually finds the slope, but the process could be automated. The subVI was adapted from another fitting problem similar to yours.

1. I changed to the Read From Spreadsheet File rather than reading the file line by line.
2. I put the data from the file into a control and made Current Values Default so I did not need to keep reading the same file. You will want to remove the diagram disable structures to look at the data in other files.
3. This is not optimized for large data sets. With the data you sent, segment lengths of 1000 result in very slow performance. Start with length = 100. That smoothes quite a bit of the noise. Look at index values around 2000, 3000, and 4000.

Modify to meet your needs.

Lynn

[oyester]

Have you checked out Savitsky-Golay filters?

[Bandit]

First - apologies to all. As usuall I'm juggling several projects, but nevertheless I should have replied sooner.

Second - thanks to all. Your advice was very helpful. The fft helped me identify a source of noise and improve wiring to keep it out. The best solution for my application then proved to be the best fit line. I have a solution and I'm very happy with it.

Bandit