Determination of Integer values

[Hunt4u666]

Hi,

I wrote a short program to determine whether there is an integer in a floating point numeric control or not.

This is working fine as long as I use the keyboard to enter the values confirming with the enter button.

If I use the increment/decrement buttons to change the values only the value I entered with the keyboard is detected as an integer despite the values in the equal comparison (block diagram) are the same.

E.g: If I write a 2.0 into the numreic control and press enter the result is true

When I then increment to 3.0 it is false as long as I rewrite it with the keyboard confirming with enter.

What could be wrong?

Thanks for help

[GerdW]

Hi Hunt,

there's nothing wrong - it's the behaviour of floating point numbers. Set the precision of your control to 17+ digits to see what's happening!

There are some other threads on this (or related: look for "quotient & remainder bug") topic: floats cannot hold any arbitrary number. So incrementing in non-integer steps easily leds to rounding errors in floats.



Message Edited by GerdW on 11-13-2007 11:13 AM

[Hunt4u666]

Ok, thanks.

I will look up the other topics.

Regards

[jrpe]

I think the easiest way to deal with floating point numbers in this case would be to check that they are equal within a tolerance. Since I doubt you need 17 digits of precision, you can just subtract one number from the other, take the absolute value and check if the result is less than whatever threshold you want. For example, if you wanted the numbers to be equal within 1E-6 you would do the following

Message Edited by jrpe on 11-13-2007 08:58 AM

[Ben]

Jeff,

The Quotient Remainder if dividing by one will let you test any number as you have illustrated.

The fixed offset can run into problems with very large numbers that have more digits than than the signifigance of the data type (sgl 7 digits?).

But under circumstances were a human will be incrementing by one starting a zero this should not be a problem. (Hint, don't pay your Beta testers by the hour )

Ben

Message Edited by Ben on 11-13-2007 08:27 AM

[jrpe]

haha alright thanks. I will keep that in mind. I did know about the limitations of the fixed offset, but from the problem description, it seemed that a high level of precision was unlikely.

oh well...i tried...and was corrected....and learned something! haha