How can I remove some parts of a signal?

[Elfia]

Hello!

I just want to ask how to remove some parts of a signal which you can see on the picture below. (black circles).

 In short I have almost 100 tdms files, almost all of them have this five columns pattern and I need to analyze all of the columns separatly by mathematical methods.

Thank you ver much for your help and answers!

[Elfia]

Hello!

My project was slot milling five times in a row (that's one tdms file) while collecting data about the force, sound and acceleration.

Now I have to analyze all the tdms files with mathematical methods.

My questions:

1. How can I delete or remove the signal where the machine was in idle condition? (The parts with black circle.)

I have almost 100 tdms files I'd like to automatize. Almost all of them have this representative pattern. (five columns).

And after that how can I separatly handle all of the columns?

In short I have 100 tdms files, all of them have these five columns and I need to analyze every column (mean, spread, rms, etc.)

Thank you very much for your help and answers!

 

[GerdW]

Hi elfia,

in the end each signal is an array of samples...

You need to find the edges of those "columns", then you can use ArraySubset to get just the regions of interest!

[Kevin_Price]

Initial thoughts off the top of my head:

1. Main goal: identify the transition points where the larger oscillations start and end.

2. Identify how many samples per cycle you have of those seemingly-oscillating signals.  Round it up little bit and call it 'N'.

3. Consider a sliding window on your data with size = N samples.  Establish a characteristic criteria such as maybe (Max - Min).

4.  A backward-looking sliding window will find the transition from smaller to larger oscillations.  A forward-looking sliding window will find the transition from larger to smaller oscillations.   And a forward-looking sliding window starting from sample 'i' is the same as the backward-looking windows starting from (i + N - 1).

5. So now you're looking for where that criteria crosses some threshold so you know you're at or near a transition point.

6. Once you identify those transitions, you can extract the data corresponding to the 5 subsets of interest.

-Kevin P

[Elfia]

Thank you, both of you. 

I'd like to make my question a little bit more precise. How can I automate finding the perfect threshold in the positive and negative section as well? 

I have loads of tdms files and they don't have the same amplitude.

I made the reading from folder automatic. And after that I'd like to make five arrays (for every "column") for every tdms (there are 98 files). And even after that I need to make mathematical methods to analyze them and to find correlation.

I thought about moving average or some threshold VI but I don't know what to do about the changing amplitude.

---

---

[GerdW]

Hi Elfia,

---

@Elfia wrote:

How can I automate finding the perfect threshold in the positive and negative section as well? 

I have loads of tdms files and they don't have the same amplitude.

---

Is the "noise" in the blue plot always below some amplitude?

Are there always those nice spikes in the red plot?

As you know your signals only you can determine such criteria to distinguish between "wanted data" and noise…

[Elfia]

Yes, there is a goodish interval between the value of 0,05 and -0,05.

So is there any option to remove the data which is between this interval by the amplitude? 

Thanks in advance!! 

[Kevin_Price]

Refer back to my msg #3.  My sliding window of (Max - Min) would be pretty good at identifying the boundary between background noise and the events of interest.  A sliding window of RMS would be less sensitive to spurious noise spikes during the "background noise" times, but would also not identify the boundaries as precisely.   A sliding window of Mean would *not* be a good idea because if there are many oscillation cycles in the window, the mean will tend toward 0. 

There's probably a more sophisticated signal processing approach that I'm not familiar with.  These are just some seat-of-the-pants thoughts to help you get started.  Like GerdW said, it's necessary to have a deeper knowledge of your data's key characteristics to be able to design a solid algorithm for your automated processing.  None of us are going to have that knowledge.   The more you can clearly describe here, the more likely someone can help better.

-Kevin P

[wiebe@CARYA]

It's your data, what data do you want to remove?

Something like 'remove everything that's low between everything that's high' or an image doesn't work well for computers (or people trying to help).

Try to make your human recognition discrete.

How would you remove the data? What criteria would you use?

You can actually do this manually for one or two files. Note what you're doing to remove the data. Zoom in into the crossings from signal to no signal. That's where it gets interesting.

You probably end up with a test on the absolute amplitude. This test results in a 'signal' or 'no signal' for each sample. Tricky, because even the signal has samples near 0. So, make each sample that is 'signal' validate itself + and - a few samples. That will effectively classify the data into signal and no signal.

You can make this as fancy as you want. For instance, in stead of adding samples to the 'signal' samples, you can go back and forth until the first zero crossing.

You can automatically determine the amplitude. Or not... It depends on the data if this adds value. You can do this in lots of ways, including a simple average, histograms, normalizations, all kinds of combinations.

We can help with all that, but you'd have to come up with plausible rules. And some data would be required.

[GerdW]

Hi Elfia,

please don't create multiple threads for the very same problem. It only creates noise in the forum…