string to number by "scan value" with format string "%.3f" doesn'work

[weitong]

I got a small problem.

To convert a string to number, I use the function "scan value" with "%.3f" for the" format string" connector . I wish to get number with 3 digits. But it dosen't work with the setting of "%.3f".

Thanks.

[pincpanter]

Those kind of specifiers are used to format numbers into strings, not the opposite. If you need to cut digits in excess, you have to do it by code, reading the input as a string.

[tst]

To expand on that, there's no such thing as a floating point number with three digits. The number will almost always be close to the fraction you want, but not exact. The format code controls how the number is converted for a string, usually for display. Here are some options:

1. If you want to get the 3 digits after the period, you could do what PP suggested (parse the string to find the period, then add 3 and use String Subset to cut the string).
2. You could also take the entire number, but round it to the nearest multiple of 0.001. You do that by dividing the number by N (0.001, in this case), rounding the result and then multiplying by N. If you want to see the exact string, you need to always round towards zero.
3. You could take the entire string as the number, but change the display format of the numeric indicator to show only three digits.

[weitong]

But if  I put "%5.f" instead of "%.3f" with the" format string" connector, it works,i.e. cuts the number into 5 numbers including period ("256.34567"->256.3).

[pincpanter · Accepted solution]

Please check the Format Specifier Syntax in the LV help. There is a table describing the purpose of each specifier. When you read in the description "when you use a formatting function", the specifier applies to number->string, when you read "when you use a scanning function", the specifier applies to string->number. The Width syntax element applies to both situations, the .Precision syntax element does not.

[weitong]

Thanks.