Numeric Limit Test

[dphan128]

I'm using "greater or equal than and less than or equal then" statement in my sequence. But I still get a fail when my reading is equal to the minimum value. How do I fix that problem ? Please help.
Thanks

[Bob_Y.]

If your measurement and/or limits are floating point numbers, maybe they are not really equal in all bits. Typically you will only show a few digits of precision, but if you showed more digits it would quite possibly be right that the measurement is not in spec.

If you are using integers, I am at a loss for answers.

Hope that this helps,
Bob

[Caribe]

As Bob said, the problem is probably with floating point numbers.

Maybe you are measuring the Value as an integer type, but when passing
it to TestStand, it is converted to a floating point number.

Floating point Numbers are tricky and they are not able to represent
all the real numbers, but only some numbers, separated from precision intervals.
If you want a perfect limit, you schould check which is the nearest
representable number to the limit you want.

Cheers,

- Caribe -

[dphan128]

I'm using real number in my simple calculation (minus the second number from the first number) and then compare with the limits. But the first number has 3 decimal place and the second number has 2 decimal place. So I assume that the result number has 3 decimal place and when it is compared with the limit value which is a real number with 2 decimal place (before it is round up).
Anyhow my question is how do I fix the problem.
Thanks

[John Gush]

It doesn't really matter how many decimal places are visable because every value you enter needs to be converted to it's binary equivalent. It is possible that the decimal number you entered can not be represented exactly. Have a look in TestStand to see exactly what the value is compared to the limits. They may look the same up to the 5 or 6 decimal places that they show but it may differ further along down the line. Remember that floating points are represented by 64 bits so they are very accurate by they still can't represent every decimal number. To 'solve' the problem you may need to account for the error within the your limits.

John