LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

determine number of digits of precision from a number

Its monday after thanksgiving and I have a brain freeze. 

 

 

I want to convert a double floating point number to a fractional string, but in cases when the number has no fractional part (Digits to the right of decimal) I simply do not want to have lots of zeros.  The number ot fractional string requires a precision input but I need to be able to look at teh number, calcualte the precision needed then feed it into this standard VI.

 

How do I calculate the digits of precision from a number?

0 Kudos
Message 1 of 5
(8,373 Views)

This is tricky.  Some numbers, such as LabVIEW's representation of pi (or if you compute, say, the square root of 2) have an infinite decimal part (don't worry -- LabVIEW only approximates the infinite decimal).  So the first question to answer is "How many significant digits to the right of the decimal place do you want?".  Suppose this number is N.  To take any Double and transform it to have at most N digits to the right of the decimal point, (a) multiply by 10^N, (b) round, (c) divide by 10^N.

 

Now you can figure out what precision you need.  First, get all the significant digits to the right of the decimal point.  You can (almost) do this in a While loop with N in a Shift register -- multiply the number by 10 (shifting a decimal from the right to the left of the decimal point), and compare the (original) number with its rounded version, stopping the loop if this is true.  Note that this will shift N one too many places (but saves a Case Statement).  Now you need to "count the digits" (and subtract 1).  Again, While is your Friend -- start dividing by 10, rounding to discard the remainder, and stopping when you get to 0.  The index of this second While Loop, when it exits, will be the Precision you want (again, almost).

 

There are two "gotchas".  One is handling negative numbers -- the truncation step can "mis-fire" in the second part of the loop, but you can simply do the computation with the absolute value of N.  The second is the case N = 0, which will return a Precision of 0 instead of 1.  Simply wrap the second While loop in a Case statement, test (once) for =0 and return 1 if true, else do the While computation.

 

Suggestion -- make this a sub-VI (call it something clever like "Precision") so you can use it over and over.

 

BS

0 Kudos
Message 2 of 5
(8,353 Views)

Just use Format Into String.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
Message 3 of 5
(8,349 Views)

"Digits of precision" does not mean much except in the context of controls and indicators. The data on the wire always has the full precision supported by the data type. Because of the way data is represented in binary in computers, decimal digits of precision is a largely meaningless term.  For example 0.1 (decimal) = 1/10 is an infinitely repeating fraction in binary. Adding 0.1 to itself ten times produces sum = 0.99999999999999988900 or an error of 1 - sum = 1.1102230246251565000E-16.

 

There is a one without lots of zeros!

 

What will you be doing with the string?  If you tell us what you are trying to accomplish, someone may be able to offer useful suggestions.

 

Lynn

 

 

0 Kudos
Message 4 of 5
(8,348 Views)

@id wrote:

 

I want to convert a double floating point number to a fractional string, but in cases when the number has no fractional part (Digits to the right of decimal) I simply do not want to have lots of zeros...


You may use the format value function (edit: or the format into string function) with a "#" to remove trailing zeros.  For example, the below will convert the DBL to a fractional string with up to 6 digits of precision and will remove the trailing zeros:

 

Untitled 1 Block Diagram _2014-12-01_13-48-32.png

 

 

Best Regards,

John Passiak
Message 5 of 5
(8,337 Views)