LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Format into string converts some whole numbers wrong


wiebe@CARYA wrote:

That online IEEE-754 converter seems to use singles (hex value is 0x44044000). Although that doesn't seem significant here.


Correct, but if a SGL can do "529" then a DBL should definitely be up to the task.

0 Kudos
Message 11 of 12
(334 Views)

@Intaris wrote:

wiebe@CARYA wrote:

That online IEEE-754 converter seems to use singles (hex value is 0x44044000). Although that doesn't seem significant here.


Correct, but if a SGL can do "529" then a DBL should definitely be up to the task.


Yes, and I tried sgl, dbl and ext, but they all have the same result.

0 Kudos
Message 12 of 12
(329 Views)