How do you fix the math errors when converting a string to a double?

[concept25]

Often an instrument will return a measurement through a communications device as an ASCII string. This requires conversion to a numeric value so that the measurement can be tested against numeric limits.
In this example if you were testing to a limit less than or equal to 58.7, the test would fail even though the measurement was passing.
Scan("58.7000", "%f", &number);
number = 58.700000000000003

[Chaos]

Dont know if there is a standard way of doing this or not. A real cheap and dirty way ive used in the past was something like:

number = ((int)number*100000)/100000

IE multiply the number to desired precision, convert to an integer to drop off the rounding errors, and divide back to get original answer. Sorry I dont have a better answer. I havnt had to use something like this in a very long time...

[cosmo]

Hi Concept,
You remember the values stored as floats and doubles are a little 'sloppy', right? So, think about how much precision the max number needs to have.

scan("58.7000", "%f", &number);
numPrecision = 0.000001;
if (number < (58.7 + numPrecision)) {
...
}

You can use the same concept to subtract from a min value and enforce a max-min range (or even a min range by itself).

if ((number < (58.7 + numPrecision)) && (number > (38.125 - numPrecision))) {
...
}


Orlan

[scottrod]

ScanFile (InputString, "%s>%f", &DoubleValue);

Is there not a modifier (or set of modifiers) either on the source string variable and/or the target double variable that will get rid of this problem?

My input char string is always something like -12.5000 or -9.3500 (always four digits after the decimal).

Sometimes it nails it with all zeros, and sometimes I get -12.500000000000001 or -9.350000000000001.

The code is a solution of course (and what I'm doing now), but not very elegant...

[scottrod]

Here's what I ended up with.

Value *= 10000;              // Turns the important part into a integer (up to 4 decimal places).

modf (Value, &Value);     // Gets rid of any remainder.
Value /= 10000;             // Turns Value back into the original number without the conversion error.