How can I prevent rounding when writing to spreadsheet file

[dpatch]

I am trying to write arrays of Doubles the user enters on teh screen to a spread sheet file. The arrays contain exact frequencies that will be written to an instrument. The doubles are formatted to 6 decimal places. When I write them to the file LabView is rounding them, which is changing the frequency values the instrument will step through. For example, I write 8.3 and in the file I see 1.299999. I am using the Write To Spreadsheet File Vi. How can I prevent this from happening?

[NIDave]

Its problably just that you need to wire "%.6f" to the format terminal on the top of the write spreadsheet file. I made a simple vi that passes an array of doubles to the write spreadsheet file vi and it gets 6 digits of percision. Try that and see if it helps out. if not maybe you can post your vi so that we can figure it out.
Hope it helps.
-Dave

Although the more I think about it I cant figure why an 8.3 will be changed to a 1.299999

[BJD1613]

try converting to a spreadsheet string and using the write characters to file vi. This will allow you to write it as a tab delimited text file that you can open in excel. Try this example. Hope this helps.

[dpatch]

Thanks for the quick response! I am including the %.6f format string and I am getting the the correct percision. THe problem still remains...sometimes...and I do mean sometimes...the number changes as indicated above. It appears that there is rounding going on which is looking at lower order bits in the real number that are not cleared. Is there any option in LabView to say "no rounding" or "truncate bits past the 6th decimal place"?

[shoneill]

Do please bear in mind that floating point numbers are only able to approximate multiples of many numbers. If you take a dbl control and set it to 8.3, and then set 10 or 12 digits visible, you will see that it's not exactly 8.3. This is a limitation of dbl representation, and is common to basically all programming languages.

Check this link for more information.

Shane.

[dpatch]

I agree with you completely. However, I am using a control that is an array of doubles to enter exact frequencies to control an instrument. If I write these values to a datalog file (a binary file) they read and write correctly. The issue here seems to be the "Array To Spreadsheet String" function. When it converts the double to a string, it rounds based on the format sting input (%.6 in this case). If you can not control the rounding, how can anyone ever send an ascii command to an instrument that contains converted doubles? I could use a series of multiplies, integer conversion and divides to solve the problem but this seems extremely wasteful since there is a large number of values to be converted.

[shoneill]

I've just thrown a small VI together to show the effect, and although with increasing decimal places, 8.3 changes to something else, it doesn't change to 8.29999.

Looks like the problem lies somewhere else.

Shane.

[dpatch]

Thanks for the effort!

I am not sure how you generated the output you attached. I created a small vi and included an ASCII data file to read in. Put the files in a directory and run the vi. When you hit the Read button, it will prompt you for a file to read...select the data file. When the data is read, it is displayed on the front panel. Compare what is displayed to what is in the file. You will see the errors.

[Randy Riggs]

I checked your example, dpatch, and I don't see the problem. You're importing this file into single floats... A single float cannot represent 17.800000 and so it reads 17.799999, as an example. Unless there's something else there I'm not seeing.

Did you check out the link that Shane posted earlier?

Randy

[dpatch]

I did see Shane's earlier post, but was not sure how he generated that output. I just put the example vi together fast, when I initially noticed the problem, I had changed the Read (and write) spreadsheet vi's to handle doubles. I agree, there is something I am missing too. This can't be a bug in LV, it has to be something stupid I am doing. It is too basic a thing! If you just want to have fun, open the read spreadsheet vi in the llb and save it to a new location and change it's arrays to doubles.

To be honest, I just got done writing my own vi's to handle the reads and writes (it was too easy). I was just being lazy and was going to use the spreadsheet routines. So please, don't spend any valuable time on it.