Error in Less function

[berrg]

When I add 0.1 to zero through a while loop and a case structure until a stop value. When stop is 0.1-0.7 it stops at stop value, but when it's 0.8 it goes to 0.9 in stead of 0.8. Please see attached sample program.

I use Labview 6i (6.0).

[John Lum]

This is happening because floating-point numbers have a limited precision inherent to their digital representation. It's often hazardous to compare one with another and expect them to be exactly equal.

You can see what's happening if you add more digits of precision to your floats by right-clicking and choosing Format & Precision. Crank it up to 16 or so and note that the repeated addition of 0.1 eventually results in a number slightly smaller than 0.8.

Final analysis: no problem with LabVIEW. You'd see the same thing in C or Visual Basic or whatever. I'd advise recoding it to avoid this sort of pitfall.

Regards,
John Lum
National Instruments

[Zvezdana S.]

This is happening because floating-point numbers have a limited precision inherent to their digital representation. Setting the precision to at least 16 is the only way to see all the digits 0.8999 that round up to 0.90 with 2 digits of precision.

Zvezdana S.
National Instruments