Possible Compare Bug

[jspaarg]

The purpose of this function is to round a double with x digits of resolution to a double with 0.1 resolution and then determine the lowest value.
It seems that a compare of numbers with 0.1 resolution does not occur correctly when the lower number is odd.
It works fine when the lower number is even.

Is this a bug, or am I missing something?

Original code in LabView 7.1.

Thanks

[RavensFan]

I don't understand what you are doing with your "Low" number.  Why are you reading a local variable of it before you have written a value to its indicator?  Your Input to Round code seems okay to me.  I don't understand what you are trying to do with the comparison.

[jspaarg]

In the real code the low is in a shift register.
The idea is that each successive input is compared against the lowest value to determine when the input reaches its maximum low.

The actual application is measuring transmissivity of glass using a fire wire camera and the input is the average transmissivity for a given data frame from the camera.

You can use the code that I uploaded and change the value in the input field to 2.62 and notice that the low is properly updated.
If you then change the input to 2.52, low is not properly updated.

[johnsold]

A value with odd multiples of 0.1 cannot be expressed exactly in binary. This is why you comparisons may sometimes fail.

You can multiply by 10, round to an integer, compare integers, change back to double and divide by ten. That will limit resolution and let you perform comparisons.

Another approach is to use the In Range and Coerce function with the limits set to machine epsilon (the smallest double which can be represented on your computer) or some larger value if you may have multiple roundoff errors accumulated. Machine epsilon may not be available in LV 7.1.

Lynn

[jspaarg]

So you are saying that basic math comparison functions can yield invalid results when at 0.1 resolution?
And only when the lower number is odd?

[RavensFan]

When I put in 2.62.  Round and Low became 2.6.  When I put in 2.52, both became 2.5.  I'm using LV 8.5.1

I did it in LV 7.1 and saw the same behavior.  What results do you see?

Message Edited by Ravens Fan on 07-22-2008 03:02 PM

[TCPlomp]

Never trust floats.

The substraction has the following result: 0,0999999999999996447

Which is smaller than 0.1.
Never trust on the representation of numbers. Binary numbers cannot represent floating point numbers correctly.

See here

The reason for the difference between odd and even must be in the official IEEE 754 rounding rules.
These rules state that a number is rounded to the nearest integer. If the number is exactly in the center (.50000000000000000000000000000000000) it will be rounded to the nearest odd integer (13.5 -> 14, 14.5 ->14, 15.5->15), this is called bankers rounding, it is mentioned in the linked wikipedia article.

Ton



Message Edited by TonP on 07-22-2008 09:06 PM

[jspaarg]

Wow, those just worked here also.

However,

Put 2.72 in input and 2.8 in low and see what happens.

Thanks

[jspaarg]

I can verify the subtration results

I have run into issues with floating point numbers in the past, but nothing this blatant.

I used the same rounding technique on the difference from the subtract and got an error at the 18th decimal place.

I guess I will compare against 0.09999 which should cover my needs.

I sure do miss integer math in 8-bit microcontrollers.