double subtract issue

[Himmet]

Hi everyone

0,3-0,2=0,1 ???

Why does Labview turn false result?

[Miha_Vitorovic]

Because precise conversion between decimal fractions (human way) and binary fractions (machine way) is not always possible.

Consider this: in decimal 0,3 = 3/10

                     in binary 0,3 = 1/4 + 1/32 +1/64 + .... and you cannot add it up to exactly 0,3.

So, to summarize, when comparing the results of a double arithmetics to some other double, you must never use '=' to compare to an exact constant. What you need to do, is check if the result falls into a certain range - depending on your needs.

Br, Miha

[Himmet]

Thanks Miha...

[Norbert_B]

You are running into a well-known situations occurring within all programming language. It's all about numeric respresentation on a digital system (so you will have "quantized values").

There are already a dozens of threads about this (e.g. this one) so please use the search function of this forum.

thanks,

Norbert

EDIT: Make it a rule: Never use a simple "equal" comparison on floating point values. Always compare to a range of values!

[Yamaeda]

To understand the problem you'll need to understand how Float works and is represented.

Since all numbers are stored in binary format there's no 0.1(decimal), it gets translated to 1/16+1/32+1/256 and so on (similar to 0.00011001... bimals(?))

So, simply put, 0.2-0.1 in decimal doesn't turn out to be as nice answer when looking at binary, thus the comparison with 0.1 doesn't work.

0.5-0.25 will work fine ...

/Y

[RavensFan]

Someone owes S-mercurio another nickel for his retirement fund.

[smercurio_fc]

---

@Ravens Fan wrote:

Someone owes S-mercurio another nickel for his retirement fund.

---

I was just going to say that!