Excel_RangeSetItem produces values with wrong precision

[the steve]

I am using Excel_RangeSetItem to send some floating point values to excel. Each value is precise to 4 decimal places. When the data is sent to excel, there are a lot of extra numbers tacked on the end.

For example:
0.04902
becomes
0.0490299984812737

The command I am using is:
error = Excel_RangeSetItem (ExcelRangeHandle, &ErrorInfo, CA_VariantInt (x),CA_VariantInt (REPORT_FREQ), CA_VariantFloat(0.04903));

Other setup information:
Windows2k
LabWindowsCVI 5.5.1
Excel 9 (2000)

Any suggestions?

Thanks,
Stephen Hamblin

[JuanCarlos]

Steve,

I did some research on this, seems that the CV_VariantFloat function is doing some data casting that is causing this change in the numbers. If you run this snippet of code you'll see that the value of the float changes even without sending it to Excel.

static VARIANT varFloat;
static VARIANT varDouble;
float numFloat = 0.04903;
double numDouble = 0.04903;

main()
{
varFloat = CA_VariantFloat (numFloat);
varDouble = CA_VariantDouble (numDouble);
CA_VariantGetDouble(&varDouble,&numDouble);
CA_VariantGetFloat (&varFloat, &numFloat);

return 0;
}

I'll file this bug here because changing to variant should not change the value at all. As a workaround I would suggest using double data type for the transfers, CVI will not an
y problems and I don't think Excel will have any problems either.

Let me know if you have any questions on this.

Regards,

Juan Carlos
N.I.

[the steve]

Thanks Juan. When I changed everything from float to double my problems when away.

steve