The modified thompson tau technique

[naisty]

Hi,

 I would like to remove outliers from data with the modified thompson - tau technique,
do somebody create the VI for the modified thompson - tau technique?

I would appreciate it if you would let me know if the VI is in libraries of LabVIEW(ver. 8.6).

Regards
naisty

[battler.]

I am very interested in details about this technique.

The only VI I know about for removing outliers is the quite weak Remove Outliers VI.  This VI only allows you to remove outliers based on an X and/or Y Range you must specify or Indexes; quite pathetic really.

It would be nice for this VI to have at least the in-built capability to remove outliers based on standard deviation..

Message Edited by battler. on 03-12-2010 01:50 AM

[battler.]

Method is explained quite well here:

http://www.mne.psu.edu/ME345/Lectures/Outliers.pdf 

It is not difficult to program.  The hardest part would be calculating the modified Thompson tau value which is calculated from the critical value of the Student's t-test.  You could use the tabulated values for Thompson tau but Ithink it would much more flexible to calculate based on arbitrary n number of samples.

You need the critical students t-value for error = 0.05 and degrees of freedom = n-2.

With the method only one outlier is considered at a time, subsequently removed and Thompson tau recalculated.. and so on.

[naisty]

Thanks battler.

I have made a VI for LabVIEW 8.6 to remove outliers with the technique.

thx,

naisty

[battler.]

Well done naisty.

I'm not too sure about the sequence structure - I think it could be done a bit better.  I tested your VI and it appears correct.

The calculation of the modified Thompson tau is correct against the tabled values given by the reference.  I tested it against the Example for no outliers.

I'd be interested to see the perfomance of this technique for a wider variety of datasets. 

Note: it is only suitable for removing outliers in a sample of a single variable.

Questions for naisty:

- What are you using it for?  Can you post some typical data (re-post the VI and set the input array with your data to Default values) or plots.

- Do you know of a textbook reference for the modified Thompson Tau method?

[naisty]

Thank you for your reply and questions.


I'm trying to figure out a judging algorithm based on the outliers.pdf for a wider variety of datasets.
The VI is still in the make due to the subjective second criterion.
 

> What are you using it for?  Can you post some typical data (re-post the VI and set the input array
> with your data to Default values) or plots.

For my research. ( http://mswebs.naist.jp/LABs/kawai/tsumatori/ )
Specifically, I'd like to obtain the correct average value from acquired signal by the optical system I designed.
 

> Do you know of a textbook reference for the modified Thompson Tau method?

I've found some references about this method.

  Title: Measurement Uncertainty
  ASME Performance Test Codes
  Publisher: The American Society of Mechanical Engineers (ASME)

  Title: Introduction to Engineering Experimentation
  Authers: Anthony J. Wheeler,Ahmad R. Ganji
  Publisher: Prentice Hall
  pp.163-

  Title:  Rapid Quantitive Analysis of Magnesium Stearate
           in Tabulates Using Laser-induced Breakdown Spectoscopy
  Authors:   St-Onge, L., Archambault, J. F., Kwong, E., Sabsabi, M., Vadas, E. B.,
  Publisher: Journal of Pharmaceutical Science,
  vol. 8, pp. 272-288

 
Regards,
naisty
 

[battler.]

Hi naisty,

It would be nice to have implemented the method for pair of (x,y) variables.  Please do post.

I can't ready much of that research page 😉

Thanks for the references.  I will try to find these and others in order to implement a better method for removing outliers.

Battler.

[naisty]

Hello battler,

I've crated the new VIs for removing outliers and testing.

This VI support two variables.

In order to evaluate the subjective second criterion in the outliers.pdf,

I used the correlation coefficient.

the flow chart of the VI is here.

1. single variable?  -- y --> removing outliers by the thompson - tau method  --> exit

2. using the specified curve fitting order, getting the pairs of x,y value of the fitted curve and the correlation coefficient, r.

3. evaluating standarized residual, |e/S|.  (all of |e/S| < 2? --y--> exit)

4.  r >= 0.99? -- y --> exit

    ( this condition may be not necessary. )

5. if r <= 0.95, removing suspected outliers. else (that is, 0.99 > r > 0.95), adding 1 to the fitting order.

6. return to 2.

Could you test these VIs?

(Please save both files to the same directory.)

Regards,

naisty

[battler.]

Hi naisty,

Thanks for posting.  I will test ASAP and let you know.

Battler.

[battler.]

I looked briefly at the VIs.

I think that it is simple enough (and there is little doubt) to show that you have impemented the method described in that paper correctly.  I will get around to that over the weekend.

It is another thing (and what I'm really interested in) to test the behaviour and performance of the method.  I think it needs to be compared with removing outliers using say 2-3 standard deviations.