LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

error in Pi value

Solved!
Go to solution

Extended precision (80 bits) really doesn't get you much.  The math co-processors use it internally in order to guarantee that their results are always accurate to the full precision of doubles (64-bit).  If you use extended precision, you have no guard bits and the truncation errors will start piling up; in the end, you may have a result that's no better than DBL.

"If you weren't supposed to push it, it wouldn't be a button."
0 Kudos
Message 21 of 22
(453 Views)

As I mentioned earlier, even slight reordering of computations can change the result. Here's a illustrative old example. Even though Pi is involved here, we cannot blame it for the differences! 😄

 

0 Kudos
Message 22 of 22
(443 Views)