LabVIEW
Change/add math vi's to prevent rounding errors
one more note:
It's a bad idea to have a "#" in the filename - Firefox always cuts the filename before that character giving truncated names...
do you really think you can trick the IEEE754-FloatingPoint-numbers by rounding? You still get the same "unprecise" behaviour of a floating point:
I changed the formatting of input and output to show 17 sign. digits and not hiding zeros.
Now I gave "1.1" as input and after rounding to 5 digits I get the above result...
You surely can round after each and every operation to the desired number of signif. digits, but you still get floating point results. So it should be easier to program some fixed-point arithmetic using integers like I64...
Message Edited by GerdW on 10-22-2007 03:35 PM
Yes, I believe that I can trick IEEE. 
Try it, you'll see that the last digit is no problem.
That last digit is perfectly well handled by IEEE. Try to multiply 1.1 by 10. What you get is 11.00000000000 and not 11.000000000001 Somehow, IEEE does take care of the quantization error in the last digit.
The problem is with any extra rounding errors that are introduced when doing calculations.
calculating 0.2 via 1.2 / 6 gives: 0.199999999999999983
while 0.2 from a control gives: 0.200000000000000011
That's the kind of rounding errors that you have to get rid of. Rounding the calculated 0.199999999999999983 to n sign. numbers gives 0.200000000000000011 again.
So...
You can't, and also don't have to, remove the fundamental quantization error in representing the IEEE floating point numbers. But can remove the extra rounding errors that are introduced in calculations.
P.S. Interestingly: On my computer, 1.1 reads 1.10000000000000009
After reading you're last remark, I see that something is not clear yet.
I'm absolutely not suggesting that you should round after each and every floating point operation! I only want to round the numbers when doing a floating point comparison operation. Not with any other operation. In the example above, the difference between the two represenations of 0.2 is only 1.4 e-16. Very small indeed. But it does make the difference between QR giving the answer 5 or 6. The boolean aspect of the floor operation, increases that insignificant error by 10^16, and makes it very significant.
That's why I want to round the numbers before doing a comparison. In any other floating point operation, that 1e-16 error might maybe increase to 1e-15, but nothing more. It stays insignificant. In that case, it's better to wait with rounding till the very end.
Message Edited by Anthony de Vries on 10-22-2007 04:18 PM
Very interesting link shoneill. (Especially the webpages behind them) Found a place in my bookmarks.
Last night, I got something really weird. Using EXT, i did the simple division: 1.8 / 0.3 This gives the answer 6.00000000 exact. And still the floor operation rounds that to 5. 
I give up. My head starts to hurt from thinking about floating points. I think I'll only use FXP in the future.
