problems with ext to display conversion

[GerdW]

Hi,

I stumbled on a problem when displaying EXT numbers in a numerical indicator.
For some numbers the display is wrong when using big numbers for precision ("significant digits") - see attachment for LV8.21!

For mantissa=0x01 and 0x05 to 0x0f and power=1 (will be 2^16 in the vi) the displayed value is wrong - even when the internal representation is correct as can be seen by the converted U8 array...

Any comments? Is this problem solved in LV8.5?

[johnsold]

What error do you see?

When I ran you VI in 8.2.1 and 8.5 on my Mac (G5, PPC), I saw nothing that looked wrong or different between the two versions. The Mac's implementation of EXT is different from Windows, I think, so that may be part of the difference.

Lynn

[GerdW]

Hi Lynn,

I'm sorry - I forgot the main part of the messages:

When running the example vi with default values you should get 65536 (=1*2^16) as result. Instead of this I get 65535.9999999999999...
The same happens for the mentioned mantissas 0x5 to 0xf: instead of a nice integer you get a fraction in the display BUT the internal representation is still integer!

So it seems (IMHO) to be a problem for those numbers when they are converted to a decimal number for display...

[johnsold]

I definitely do not see that in either 8.2.1 or 8.5. It may be a processor hardware or OS related issue.

Lynn

[GerdW]

Here's a pic of the frontpanel:



Message Edited by GerdW on 12-19-2007 03:21 PM

[AmirG]

Hello Gerd,
would you please test the attachment VI?
Could you reproduce this behaviour using this VI?

Thanks,
Best regards,
AmirG

[GerdW]

Hi AmirG,

yes, I can reproduce the same (wrong) behaviour with your vi (see attached pic).
You only changed the display to show less digits and replaced one EXT indicator by a DBL one?

The point is:
When you want to show more significant digits (let's say 20) then the indicator shows a wrong value (65535.9999...999 instead of 65536.0000...000) even when the internal representation of the EXT/DBL number has NO fraction (all mantissa bits are zero!)... IMHO when the internal representation of an EXT is an integer (=power of 2) then the numerical indicator shouldn't show any other digits than "0" after the decimal point!

[AmirG]

Hello Gerd,
I see. I´ll take care of this issue.

regards,
AmirG

[AmirG]

Hello Gerd,

this is just the nature of floating point numbers.  Everything has finite precision in the world of computers.  If you look at the context help for extended precision it says "precision varies by platform, at least 15 digits". In fact, you are trying to get more precision than the computer can handle.

This phenomenon is known as "loss of significance".

Best regards,

AmirG

[GerdW]

Hi AmirG,

here I can't agree with you.
I want to display 2^16 in a numeric indicator. This is a number with NO digits after the point. It's represented internally as 1*2^16 - just a number for the exponent, the other bits are zero as can be seen from the flattened string. But in the numeric indicator a "65535.9999...999" shows up instead of a "65536"...
So for me it seems to be a problem with conversion to display!