LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Why does this .vi round my numbers?

Solved!
Go to solution

I am using a high accuracy DVM with 6 and 1/2 digits of precision for a reason, because I need the accuracy, so why is this .vi rounding off my numbers to 3 decimal points?

 

 

Download All
0 Kudos
Message 1 of 3
(2,329 Views)
Solution
Accepted by topic author LarsUlrich

Write to Spreadsheet defaults to 3 places.  Right click the Format terminal and create a constant.  Change the 3 to desired # of decimal places.

0 Kudos
Message 2 of 3
(2,315 Views)

Turn on context help and hover over that function.

 

You will see there is an input at the top called format, and its default value is %.3f.

 

You need to wire in another format such as %.6f.

Message 3 of 3
(2,311 Views)