LabVIEW 6.1 DBL precision

[ghenton]

I am using DBL precision float (default control settings). I am setting precision = 3 decimal places. I fill and array or control with numbers like .001, .002, .003 etc. But if you look at them with a precision of 20 decimal places, you will see we have a number like 0.01299999999999999940 instead of the 0.013 that I typed in the cells. Incidentally, the number above I copied directly from the array. I hand typed .013 in the cell.

Of course if you send these numbers around a loop in a shift register and add a value like .001 to it each time it becomes very poluted. So if you set the resolution at 3, you do not see these invisible errors. But your program sure knows about it. So when you test, does .013 => .013, it may actually be c
omparing 0.01299999999999999940 => .013. And of course you do not get an equal result.

What is my solution?

[johnsold]

What you are seeing is the result of the finite precision of the binary representation of the number. If you dig down into the help files you will find specifications on the range and precision of various numeric types. The double uses 64 bits, some of which are expeonent and some are mantissa. Just as 1/3 or 1/7 is an infintely repeating decimal in base 10, many numbers cannot be represented as exact binary fractions. When you enter such a number LV uses the best binary representation availble for the data type selected.

The comparison solution: Never do an exactly equal comparison on floats. (It is OK on integer types). Use >= or <= and a small error bound to avoid these issues.

Lynn

[Cult of Nurse's]

ghenton wrote:
> I am using DBL precision float (default control settings). I am
> setting precision = 3 decimal places. I fill and array or control
> with numbers like .001, .002, .003 etc. But if you look at them with
> a precision of 20 decimal places, you will see we have a number like
> 0.01299999999999999940 instead of the 0.013 that I typed in the cells.
> Incidentally, the number above I copied directly from the array. I
> hand typed .013 in the cell.
>
> Of course if you send these numbers around a loop in a shift register
> and add a value like .001 to it each time it becomes very poluted. So
> if you set the resolution at 3, you do not see these invisible errors.
> But your program sure knows about it. So when you test, does .013 =>
> 013,
it may actually be comparing 0.01299999999999999940 => .013.
> And of course you do not get an equal result.
>
> What is my solution?

Use 'within range' instead of equals.