LabVIEW
strange behavior with greater?
Solved!
@littlesphaeroid wrote:
Another approach might be to multiply the result by ten and convert to an integer before the comparison.
Subset of "multiply by 10^(number of places to the right of the decimal point)". Incidentally, this works if you raise it to a negative number as well. Let's say you want to compare to the nearest MHz. Multiply by 10^-6.
@thols wrote:
NI's short explanation.
NI's longer explanation, read about the machine epsilon there, which can be used when comparing floating-point numbers.
Read about this idea. , in the discussion there are some examples how to compare floating point numbers approximately.
The "Longer explanation" link didn't work for me as it was eaten by a Google search ref, but the original link is in there:
http://download.ni.com/pub/gdc/tut/floating_point_in_labview.pdf
I was always more interested in the practical application, rather than the academic discussion, of the FP issue. I only ever had to do the comparison to a certain spec or tolerance, so I don't really care about machine epsilon and all that stuff. So I just do an approximation like I showed previously. It's probably good enough for the vast majority of FP issues you will run into.
