LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Truncating and Rounding problems


@dacad wrote:

This is why I state that 1.1049 rounds to 1.1.  Again, perhaps old school but the 9 round the 4 to a five, in turn rounding the 0 to a 1.

 



Actually it should round to 1.105.  When rounding to a nearest number, it is only appropriate to consider the number right next to it, as the other numbers are considered statistically insignificant...  for example, consider rounding 1.149 to the nearest 0.1.  It rounds to 1.1, not 1.2 because you shouldn't consider the thousandths place when rounding to the nearest tenth...

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 11 of 11
(411 Views)