subtraction error

[RussellSenior]

Why does Diadem calculate 0.005-0.00499998 to be equal to 1.99999999998812E-08 rather than 2E-08 like it should be?  I think this is the root cause why some of my calculations are off by about 10 nanoseconds.  

[Edna-S]

Take a look at the following example. You can round up using script.

https://decibel.ni.com/content/docs/DOC-25809

[Brad_Turpin]

Hi Russell,

DIAdem is simply showing you the full DBL precision of that number.  You get the same effect in LabVIEW if you set the digits to display high enough (see attachment).  What time issues are you having?

Brad Turpin

DIAdem Product Support Engineer

National Instruments