LabWindows/CVI

cancel
Showing results for 
Search instead for 
Did you mean: 

Numeric Control giving false input when i use GetCtrlVal

Smiley Sad when i give value of 179.7 as input to numeric control and then use GetCtrlVal to get the value in double or float then i get value = 179.6999999999991 why is it so , i want absolute 179.7 and i have a constraint that i want to allow the numeric control to get only one value after decimal point. plz help . behaviour of numeric control is very strange.

0 Kudos
Message 1 of 2
(3,726 Views)

This is a frequently asked question also in this forum Smiley Wink See, for example, here.

The answer is that any floating point number such as your 179.7 is stored in binary format of say 64 bit, which naturally is an approximation with a finite resolution.

 

0 Kudos
Message 2 of 2
(3,722 Views)