LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Resolution of extended type

I have a VI where I'm performing various arithmetic calculations to an extended variable of 24 characters. Somehow, however, when I attempt to add a number like 256 to 100,000,000,000,000,000,000,000, it loses it. I've done a little messing around, and it seems to also lose stuff when dealing with 21 character variables, but not with 18 character ones. Does this have to do with the resolution of the extended type, or is it something more mundane? Thanks.
Using LabVIEW 2009 on Windows Vista
0 Kudos
Message 1 of 3
(2,417 Views)

Hi dragonlord,

 

you ran into the limits of floating point arithmetics of the CPU...

SGL allows 23 bits mantissa, DBL 52 bits, and EXT 63 bits resolution. This translates to ~7, ~16, and 20 decimal digits, respectively.

If you need more precise arithmetics you can code your own or use one of the available "high precision" software packages

Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
Message 2 of 3
(2,405 Views)

I can break this 24 digit number down into smaller numbers and use those. Was just wondering if there was an easier way. Thanks anyways!

Using LabVIEW 2009 on Windows Vista
0 Kudos
Message 3 of 3
(2,402 Views)