rejection of elements in 1D array using statistical functions

[beam]

Hello,

It would be very nice if somebody could give me some suggestions how to accomplish the following:

I have a 1D-raw data array with e.g. 10 data elements. First, I need to determine the average (nu) and the standard deviation (sigma) of all array elements. After that, I need to calculate the absolute difference (delta) for each single array element (i.e.: array element - nu = delta(normalized)). For each array element I have now the corresponding absolute difference (delta). Please see attached example.

Now, I have to delete all array elements (out of the 1D-raw data array) which meet the following expression: (array element - delta) > 3x sigma. The result will be an improved 1D-array without the "bad" array elements.

How do I do this kind of data rejection in the fastest and most efficient way? I have attached my first steps in this procedure, but I do not know how to realize the comparison: (array element - delta) > 3x sigma , for the element rejection.

If somebody could help me with this it would be very much appreciated.

Kind regards,

Beam

p.s. I think that using the "remove from array" function would not be the fastest way, because this function will resize the array after an element is rejected. I think, maybe using a loop structure will be faster? My real life 1D-arrays will have ~256-512 elements.

[altenbach]

Beam,

The simplest way would be to just do exactly what you said, but translated into LabVIEW. See attached.

How big are your arrays?

[beam]

Dear Mr Altenbach,

thank you very much for your help!

My array size will be between ~128 to 800 elements (DBL-type with ~6 digits of precision).


Kind regards,

beam

[altenbach]

For a few hundred elements you should be OK. Once your array size reaches a few hundred thousand elements you have to start worrying about code optimization. The attached modification will most likely be quite a bit better for very large arrays. (See my discussion and benchmark results for a very similar problem here..)

When in doubt program a few versions, then race them against each other.

[beam]

Dear Mr Altenbach,

thank you very much for your help and comments! During further VI development the following situation occured:

Situation:
In addition to the "Raw Data Array" a CORRESPONDING array (called "Corresponding Array (Raw Data)") of the same size was recorded. That means: element 0 in the "Raw Data Array" corresponds to element 0 in the "Corresponding Array (Raw Data)" forming a pair of values (and so on for all following array elements).

Now, if an element in the "Raw Data Array" is rejected, I need also to reject the corresponding element in the "Corresponding Array (Raw Data)".

Question:
I hope that I solved this problem (please see attached "dataRejection_corresponding array.vi"), but I am not sure if this is the best way to do it?

It would be very nice if you could give me some comments.


Kind regards,

beam

[altenbach]

Beam,

Yes, this looks OK!

You can simplify it a little bit by using only a single case structure and index shifter, there is no need to do duplicate these operations. See attached (LabVIEW 7.1). 🙂

[altenbach]

OK, I think you can use autoindexing to make it even simpler. (Apparently, LabVIEW does not allocate extra buffers compared to the above version.)

(edit: Also the "sigma factor" control and its multiplication should be done outside the loop ;). There is no reasonable scenario where it should be changed while the loop is still executing)

Message Edited by altenbach on 04-22-2005 06:32 PM

[beam]

Dear Mr. Altenbach,

thank you very much for your help! It works beautifully!


Kind regards,

beam