How can I prevent rounding when writing to spreadsheet file

[shoneill]

I'm a poor old soul and only have LV 6.1. Can someone save the DBl error Vi to 6.1 for me please?

I generated the output by simply placing a DBL control on a front panel, and in a FOR loop converted the DBL to string with varying decimal places (using the i terminal of the for loop to auto-generate the format string for "Format into string"). Then I displayed the resulting string array on the front panel.

That's it.

I simply wanted to see how exactly a DBL with 8.3 can be represented as a string, and I got different values than you're reporting. Mine all went positive slightly, whereas yours go negative.

Shane.

[dpatch]

Shane,

Give this a try. I am not sure that it saved correctly!

dave

[shoneill]

I tried it and see the problem.

I have investigated a little and am attaching an interesting VI.

As long as the functions "Spreadsheet string to array" and "Scan from string" are used as SGL, both arrays are identical.

If both wires are left as DBL, then they're no longer equal. The scan from string seems to give different results than the spreadsheet to array function. Seems almost as though it's calculating with SGL even though it shows DBL output. Wierd.

Can someone have a look and say if I'm just doing something stupid.

Shane.

[NIDave]

OK, I double checked your vi and it works just as you said, theyre identical when they are single but different when theyre double. However if you change the format specifier from an "f" to a "d" to represent a signed decimal integer. It works either way. Replace the %.6f with %.6d and the "SAME" LED illuminates everytime. I think converting to a floating point with fractions was causing the hiccup. Heres the format specifier info so we dont have to track it down again:

Use the following conversion codes for integers:
d—Signed decimal integer.
...

Use the following conversion codes for floating-point numbers:
f—Floating-point number with fractional format (for example, 12.345).
....

And also a copy of the last VI with my changes.

[dpatch]

Will you please save this as LV version 7.0...and Shane, if you would like, I will save it back for you in 6.1.

I really appreciate you guys lokoing at this...I thought I was going crazy!

[NIDave]

Hey I just hope it helps.
-Dave

here it is in 7.0 (I cant cave in 6.i)

[dpatch]

Here is a LV 6.1 version. Shane is now seeing the same thing i was seeing. Now my question is, is the problem a rounding error or a conversion error?

It is very possible that there is a SGL somewhere in the line when it should be a double.