06-14-2011 07:56 AM
Hello all,
I was making a list of timestamps, base number and increments are steps of 8millisecond (DBL). When I display this list formatted as relative time with 3 or more digits of precision there seems to be a rounding error in the display. When I format it as floating point numbers the correct value is shown.
I use Labview 8.6 so maybe it is solved in newer versions.
Can anyone verify if this is solved in LV 2010+?
Kind regards,
Mark
06-14-2011 01:21 PM
First, let's keep in mind that the actual number that we are dealing with is 3574.05599999998503000000...
According to the LabVIEW Help on the Format Specifier Syntax:
.Precision—When you use it with floating-point notation, this element specifies the number of digits to the right of the decimal point. If . is not present, LabVIEW uses a precision of six digits. If . is 0, LabVIEW does not insert a precision. When you use it with string parameters, .Precision specifies the maximum width of the scanned field. LabVIEW truncates strings longer than this length.
_Significant Digits—Displays the data by rounding to the number of digits you specify. LabVIEW rounds the data only for display purposes, which does not affect the original data. .Precision affects only the digits to the right of the decimal point, and _Significant Digits includes all non-spacing digits.
Note that with Precision it doesn't say anything about rounding. When using %.4f ("Precision") you're getting 3574.0560, which indicates it's rounding. Also according to the LabVIEW docs, <%<digit>u> is "fractional seconds with <digit> precision". If "Precision" is not supposed to round, then %3u should give .055. If "Precision" does round, then it should give .056.
It appears that with %3u LabVIEW is simply chopping off at the third decimal place without doing anything to the number (like rounding). If the words "<digit> precision" for the help on %<digit>u is supposed to be the same as "Precision" then I'd say that its behavior is incorrect. If not, then the documentation is in error or incomplete.