01-08-2016 02:07 AM
when i give value of 179.7 as input to numeric control and then use GetCtrlVal to get the value in double or float then i get value = 179.6999999999991 why is it so , i want absolute 179.7 and i have a constraint that i want to allow the numeric control to get only one value after decimal point. plz help . behaviour of numeric control is very strange.
01-08-2016 02:56 AM
This is a frequently asked question also in this forum See, for example, here.
The answer is that any floating point number such as your 179.7 is stored in binary format of say 64 bit, which naturally is an approximation with a finite resolution.