Problems with rounding numbers using Fmt()

[alChris]

I am trying to explain why one of the values in a report written using CVI has changed when it is run today compared to the same program run in 2005. There is one difference in the environment that I am aware of: the program will have originally been written in CVI 5.x and today it runs on CVI 8.5.1. The values used in the report are retrieved from a database as strings with 3 decimal place precision. There is then a routine that reduces this to 2 decimal places using Fmt(result, "%s<%f[p2]", draw);. The expectation is that the numbers are rounded. I have included a short piece of code that simulates the conversion and a list of results. It seems that sometimes Fmt rounds up and sometimes rounds down.

Are there any rules for this ( I was taught at school if there is a 5 in the next decimal place round up)?

Is there a better way to code the conversion?

Where there any changes to the CVI Fmt command in the last 5 years that would explain the different results?

 Original Result

 3 d.p        2 d.p.
 20.695   20.70  round up
 20.605   20.61  round up
 20.615   20.61  round down
 20.625   20.62  round down
 20.635   20.64  round up
 20.645   20.64  round down
 20.655   20.66  round up
 20.665   20.66  round down
 20.675   20.68  round up
 20.685   20.68  round down
 20.695   20.70  round up
 20.705   20.70  round down

 The following code needs to be run with the debug window visible.

#include <utility.h>
#include <formatio.h>
#include <ansi_c.h>
#include <cvirte.h>

char raw[7];       //String holding 3 decimal place length value as retrieved from database
double draw;      //Length converted to 2 decimal places - expectation is that the number is rounded 
char result[7];   //Target string to hold length value converted to 2 decimal places

int main (int argc, char *argv[])
{
    if (InitCVIRTE (0, argv, 0) == 0)
        return -1;    /* out of memory */

Fmt(raw, "%s<%s","20.705");
Fmt(&draw,"%f<%s",raw);
Fmt(result, "%s<%f[p2]", draw);
DebugPrintf("%s  %s\n", raw, result);
return 0;
}

Chris 

[msaxon]

Floating point numbers in C (and most other computer languages) are stored in binary (base 2) format rather than decimal (base 10) format. A consequence of this is that a number that can be expressed exactly in base 10  cannot necessarily be expressed exactly as a C floating point number. The effect you are seeing is demonstrates this limitation.

To illlustrate this, consider the number 20.705 (as in your example). This cannot be expressed exactly in floating point representation, so depending on the way that the compiler and/or run-time libraries for the compiler you are using convert strings into floating point numbers, the floating point representation may be slightly more or slightly less than 20.705. By increasing the number of decimal places printed, it can be seen that the floating point representation of 20.705 is actually (in CVI 9) approximately 20.70499999999999829470. So when you come to round to 2 decimal places, the software does what it believes to be the correct thing and rounds down to 20.70.

To take another example, 20.725 is actually represented as (approximately) 20.72500000000000142 so when this is rounded to 2 decimal places, it is rounded up to 20.73.

Whether the floating point representation is slightly above or below the exact decimal representation is determined (in this particular case) by the algorithms used to convert string representation of decimal numbers to floating point format. For any one development tool, such as CVI, these algorithms can change in new releases so one might expect differences of the sort you have seen between CVI 5 and 8.5.1.

Another thing that can catch you out is that the conversion algorithms used in the compiler and the run time libraries may be different, so a compile-time conversion may result in a different representation than a run-time conversion. For example, in the code fragment:

double a = 20.725; // compile-time conversion

double b;

Fmt(&b,"%f<%s","20.725"); // run-time conversion

You cannot guarantee that a and b will contain exactly the same number.

There are many ways to circumvent this problem. In your case, you might want to do some coding to round the number internally to two decimal places before trying to output it, something like:

void FormatTo2DP( double a, char a_at_2dp[])

{

   Fmt( a_at_2dp, "%s<%f[p2]", 0.01 * RoundRealToNearestInteger(0.1 * (RoundRealToNearestInteger( a * 1000.0)));

}

In this code, the double is first multiplied by 1000 and rounded to the nearest integer, then divided by 10 and rounded again, then divided by 100 before being converted to a string value. It assumes, of course, that your data will not be so large that a data value multiplied by 1000 cannot be represented as an integer.

[msaxon]

Possibly a better version of that last function might be:

void FormatTo2DP( double a, char a_at_2dp[])

{

   Fmt( a_at_2dp, "%s<%f[p2]", RoundRealToNearestInteger(RoundRealToNearestInteger(a*1000.0)/10.0)/100.0);

} 

since 10.0 and 100.0 can be exactly represented in floating point, whereas 0.1 and 0.01 cannot.