Floating Point Errors when using "Coerce to nearest"

[Ben]

---

@Intaris wrote:

What I find interesting is that I can't get a DBL Digital display to set to -1.000000000000000222044605 at all. It only displays its value as -1.00000000000000022000000. This is all without coercing any increment.

Just tried it with a knob, and it also won't.

---

...

Is the difference "epsilon" (see the numeric palette >>> Constants)  ?

Ben

[Intaris]

Ben, that's interesting.

It's pretty darn close, that's for sure.

The value being shown in the OPs original post is - 1 -epsilon.

epsilon on my PC is shown above, OPs original "problem" number shown below.

-0.00000000000000022204460492503130
-1.000000000000000222044605

[Ben]

---

@Intaris wrote:

Ben, that's interesting.

It's pretty darn close, that's for sure.

The value being shown in the OPs original post is - 1 -epsilon.

epsilon on my PC is shown above, OPs original "problem" number shown below.

-0.00000000000000022204460492503130
-1.000000000000000222044605

---

The difference between those two is the "...0492503130" is being rounded to "....05"

Ben

[pogo]

Since you already coded this to create the image, why didn't you actually run it?

If you did, you would see that this single-precision conversion does not in fact replicate the problem created by the coerce node.

Here, I've attached the code in the form you requested ("rounding3").

As you can see for at least this one input case, the output is still correct when "manually" coerced.

For this same input case, the "coerced" knob's result is still incorrect.

So, this single-precision conversion is probably not what's going on inside the knob's code.

But again, it's not my job to reverse-engineer the code for the knob. Trying to reverse-engineer stuff is fun, but not really on-point for this discussion. Since I think I found a bug, my job is just to report the incorrect behavior to NI, so that an NI programmer can fix the actual code. He or she won't have to guess at the original algorithm... they'll be able to see it directly.

It looks like I and some others here are beginning to converge on the idea that in fact the "coerce" property of the controls is indeed NOT working correctly... regardless of why it is not working correctly.

[pogo]

That's really interesting. Is there any chance you're using an Apple? (I'm using a Dell brand PC).

The fact that the decimals agree as far as they do suggest that the floating point representation is the same as mine, but that maybe there's something odd about the display itself.

If you subtract -1.00000000000000022 from your value, do you get zero?

Do you also get zero if you subtract -1.000000000000000222044605 from your value?

-Pogo

[Ben]

---

@pogo wrote:

That's really interesting. Is there any chance you're using an Apple? (I'm using a Dell brand PC).

The fact that the decimals agree as far as they do suggest that the floating point representation is the same as mine, but that maybe there's something odd about the display itself.

If you subtract -1.00000000000000022 from your value, do you get zero?

Do you also get zero if you subtract -1.000000000000000222044605 from your value?

-Pogo

---

Just stopping by because I am busy...

Q:

When you set the "coerce to nearest" to 0.01 what value do YOU think the "0,01" is being interpreted as?

The "0,01" is at best off by Machine Epsilon I would think because that is the best the machine can do.

Ben

[pogo]

You ask:

When you set the "coerce to nearest" to 0.01 what value do YOU think the "0.01" is being interpreted as?

The "0.01" is at best off by Machine Epsilon I would think because that is the best the machine can do.

(my italics added)...

I mostly agree, with a nuanced difference. I think that the 0.01 is converted to the closest version it can be in base 2.

This may of course be an approximation. But if the input value is either an integer power of two (1/2. or 1/4, or 1/8, etc.) or the sum of such values, then the conversion is exact. In other cases, it's worse. So, I would suspect that a value like 0.01 (that isn't the sum of some integer powers of 2) will at worst be offset by machine epsilon.

Then, when I display this value in an indicator, this base 2 approximation will be re-converted back to base 10 for me, the human reader.

What it shows me in base 10 can be an exact representation of the value as it was in base 2, because none of the base 2 bit values are infinitely repeating. In other words, I think that all the rounding happens when the base 10 value is converted to base 2, not the other way around.