LabVIEW
U32 + "digits of precision" (??)
Solved!
@ rolfk : " No, it's not a bug. It's an inherent attribute of floating point precision. A double has about 16 digits .... / ..... "
Thank you rolfk for your (very good) explanation ... but I know the limitations and restrictions of floating point precision.
RavensFan said : " The numeric controls don't have an integer display format, just a floating point format. "
I didn't know that ! (Once we know this, everything becomes clear.)
rolfk wrote:
80 can not be exactly represented in the binary base 2 system that is inherent to the binary notation used in computers.
Of course it can (assuming we are talking about IEEE-754 Floating Point. Any integer up to (a big number on the order of 2^52) can be exactly represented, particularly if it has factors of 2 (as does 80). The Dbl format has 1 sign bit, 11 exponent bits (stored in "offset" mode, so that 0x400 = 2^1, and 0x3ff = 2^0, with the remaining 52 bits (plus an "always 1" 53rd bit) are the rest of the number, expressed as a fraction.
So write 80 = 5 * 16 = 5 * 2^4. Writing 5 in binary, 101, we need to shift it two places to the right to get it in proper "fraction" form (1.01, in binary), for which we compensate by adding 2 to the exponent, so 80 becomes 1.01 (binary) or 1.25 (decimal) times 2^6, 1.25 * 64 does, in fact, = 80.
Writing this all out in hex, the sign+exponent part (3 Hex digits) becomes 405, and the fraction (with the top "1" bit omitted, as per the standard) becomes 4000000000000. Stringing this all together, Dbl 80 is represented (exactly) as 0x4054000000000000.
It is fairly easy to show the Hex representation of this using LabVIEW.
This snippet converts 80 to a Dbl, typecasts it as an Array of U8, then displays this as 2-digit Hex (with leading 0). One would have to go through a lot of conniptions to turn an integer such as this into a string of fractional 9's. (This is fun -- I haven't worried about the IEEE formats since my PDP-11 days ...)
Bob Schor
