labview comparision error?

[dhoangz]

could anyone tell me what I'm doing wrong here?  why is it that x not equal to y?

[tbob]

DBLs or any floating point numbers are subject to small errors.  I took your controls and expanded the precision to 16 digits of precision.  This is what I got, which is clearly not equal:

Instead of using an equality comparison, try subtracting the two, taking the absolute value, and then comparing that value to be less than something like 1e-6.

Message Edited by tbob on 10-05-2005 02:03 PM

[chilly_charly]

The answer is simple : your numbers are different ! No bug here. ..

The difference is so small that you can't display it : 4.36557E-11

But of course if you compute the difference between both number you will find it.

That comes probably from the way your numbers were generated. One is probably the result of some calculation. For instance 60000/3

[PaulG.]

I tried to duplicate this ... and I could not. I have my precision set to 16 and I still get 20000.000000000. How did you do that?

[dhoangz]

thanks everybody.  is there a format number in labview to use in calculation like this?

[Barneyho]

I deleted your 0.0 and replaced it with 0.  Of course the .0 came back and it worked.  LabVIEW does tend to keep random sig figs around in it's memory for some reason. 

You could try zeroing this out before you run your application if you were worried about residual data.  Just create a local variable outside your loop/case or whatever and set it to a constant = 0.  You could also make that constant an I32 instead of a DBL to make sure it didn't even include any sig figs.

Good Luck!

[PaulG.]

Never mind. Got it! Interesting problem. I've never seen this before.