Resolution of extended type

[dragonlordged]

I have a VI where I'm performing various arithmetic calculations to an extended variable of 24 characters. Somehow, however, when I attempt to add a number like 256 to 100,000,000,000,000,000,000,000, it loses it. I've done a little messing around, and it seems to also lose stuff when dealing with 21 character variables, but not with 18 character ones. Does this have to do with the resolution of the extended type, or is it something more mundane? Thanks.

[GerdW]

Hi dragonlord,

you ran into the limits of floating point arithmetics of the CPU...

SGL allows 23 bits mantissa, DBL 52 bits, and EXT 63 bits resolution. This translates to ~7, ~16, and 20 decimal digits, respectively.

If you need more precise arithmetics you can code your own or use one of the available "high precision" software packages

[dragonlordged]

I can break this 24 digit number down into smaller numbers and use those. Was just wondering if there was an easier way. Thanks anyways!