Double Precision Help

[j.masse]

I have a strange anomaly occurring. I am reading many (%.6f) doubles from a spreadsheet in which several need to be averaged and then create a new datasheet with the new data. The newly averaged values get written to a new spreadsheet with 6 digits of precision.

Here is my problem. When I average the values read in from the spreadsheet file, I may get a 7^th bit of precision. I am dividing by an even number so the new 7^th digit is a 5. Sometimes the digit rounds up and sometimes it doesn’t round. Ex. -4.6008545 will round to -4.600855 but -4.5835165 will remain at -4.583516. Is there a way to get a consistent rounding? And yes, we need to be that accurate.

I have LabView 8.5 and I write the spreadsheets with the Write to Spreadsheet File.vi and read them with the Read Spreadsheet File.vi. Thanks Jeff

[Joseph_Loo]

First question that needs to be asked is what format is being used. Are you writing to a decimal string format? If so, you do realize that the conversion of a decimal string format to a binary number is not perfect. There are all kinds of round up and downs. So do you sanitize the data?

[Photon_Dan]

LabVIEW uses a rounding rule that is designed to "cancel out" rounding errors (I think).  I've heard it called "rounding-half-even" and "Banker's rounding".  The rule states (from the link):

round-half-even, where a number is rounded to the nearest digit, and if it is exactly half-way between two values then it is rounded to the nearest even digit. This method ensures that rounding errors cancel out (on average), and is sometimes called “Banker’s rounding”. For example, 12.345 rounded to four digits is 12.34 and 12.355 rounded to four digits is 12.36. This is also called round to nearest, ties to even.

Dan Press
PrimeTest Automation

[tst]

Here's some code for rounding to the nearest multiple (0.00001 in your case) which uses common rounding. Read that thread before using it.

Please note that this should work if you divide a number input by the user, but might not work if you divide a number where you remove precision for the display (e.g. your number might look like 0.85 when in effect it is 0.849999999243, which means it will always round down).

Ah, looks like you can't download without logging in, so here's the code. Be sure to also read this thread.

[j.masse]

Joseph, what is 'sanitize the data?' Thank you to the others who have has as well. Jeff