Equal numbers greater than each other?

[LabVIEWn00b]

Greetings all! Having a bit of difficulty with a very simple comparison inside a while loop. I'm using LV 8.5, trying to calculate a resistance value, then pick from an array of available resistors to meet said value. It's a dirt simple state machine, and it seems to work just fine until the very end when it's about to select the last resistor to meet the calculated value. For some reason, when the two values are equal, the greater than comparison gives a TRUE. I've tried the same comparison on it's own in a different VI, and sure enough, the comparison yields a FALSE. Am I dealing with a bug here?
I can attach the VI if needed.

Thanks!

[smercurio_fc]

Are these floating point values? Can't perform exact comparison with floats.

Comparing Floating Points Does Not Seem to be Working Properly.

Message Edited by smercurio_fc on 06-30-2008 09:19 AM

[Ray.R]

You must be comparing FLOAT.   It is not a bug, but a numeric reality.

[Ben]

See also here where Christian writes

"..."1" cannot be represented exactly in binary..." (which is a typo we will never let him forget. )

Ben

[altenbach]

Hey! 😄

[Ray.R]

LOL!

[LabVIEWn00b]

I'm  not using floating point values, and I've tried changing the precision and the representation. Recall, I said that the same comparison works on it's own in another VI. I don't know if it has something to do with its position in the loop in the IV or what. I've got it working now, had to do some minor conversions. As far as I can see there's something involved somewhere which is introducing FAR more precision than needed, but it's been taken care of.

Thanks all!

[TCPlomp]

Well If you don't use floating point calculation, this would be a very serious bug in LabVIEW. Without the code NI or we can't see if there's a bug, so please post the malfunctioning code.

Ton

[smercurio_fc]

You say you're not using floating points, but you're changing the precision. This makes no sense. Precision is irrelevant and meaningless with integers. Please post your VI.

[LabVIEWn00b]

Well, I'm using DBLs, but I have the display format set to Automatic, with either 3 or 4 digits. Am I confused here? Not that I'd find this at all surprising. I'll post the VI in a bit.