float to IEEE754: accuracy problem

[Slyfer]

I have a problem with this operation.

Example it's simple and gives correct results until I try with *.0 values, like 30.0

The output string is in this case the string for 29.999999

How can I avoid this?

Thanks

[smercurio_fc]

It's 41F00000 on my computer.

It's possible that you could get 29.99999.... Perhaps you should read this.

[Slyfer]

If you manually enter the value 29.99 in the control, then LV round it to 30.0 (display round...), but internally keeps the 29.99 value, and you have incorrect result.

Then if you continue playing back and forth with arrows keys over the control, the bug persists.

I just want to reproduce correctly numbers with 1 decimal cipher precision (x.y). Do I have to round somewhere? Or setting up the numeric control differently?

[smercurio_fc]

You didn't say you were entering 29.99 - you said you were entering 30.0. When I enter 29.99 in your VI the numeric control gets changed to 30.0 and the string indicator displays 41F00008, which corresponds to 30.000015 (according to the babbage web site). This makes perfect sense to me. What string are you getting when you enter 29.99 in the control and run the VI?

[Slyfer]

My control:

3 digit precision, enter the value: 29,990, I get "41EFEB9F"

Babbage webpage:

enter the value 29,990, I get "41EFEB85", that corresponds indeed to 29,990000

is this correct?

If I want to have accuracy only on numbers with 1 digit precision, how do I have to set up my control? I don't want to insert a value like 29.9 and get a "30" as string output, because it'd be wrong! these will be temperature setpoints, cannot be wrong!

Message Edited by Slyfer on 02-23-2009 05:30 PM

[smercurio_fc]

If you change the digits of precision then you also need to change the increment on your Data Range since you are coercing the values. If you change the digits of precision to 3 you will not be able to enter 29.99 unless you change the increment to 0.01. You also need to have your indicator match the digits of precision.

Your original VI was set up for 1 digit of precision. I didn't see anything wrong with it. If I take your original VI and enter 29.9 in the control and run it, the string output says 41EF3333. The babbage web site indicates that his hex string corresponds to a decimal of ... 29.900000. But, be aware, that for 1 digits of precision, and a coercion to the nearest 0.1, then entering 29.99 isn't the same as entering 29.9. It makes perfect sense that 29.99 would in that case be "rounded up". That's what the coercion is doing. However, because floating point number cannot be stored in mathematically absolute values in computers, you won't get 30.0. You instead get 30.000015 when it gets coerced. This is the reality of computers.