Fract/Exp String to Numbe Display Format

[H P]

I have a Numeric String which has a large decimal value, with the Digits of precision ranging upto 16 digits. I want to convert this String to a Numeric value and I have used a 'Fract/Exp String to Number'. In my VI I hae set:

Data Type: EXT (because it allows Number of Decimal Digits from 15 to 20)
Data Format: Floting Point
Digits: 16
Precision Type: Precision Digits

The following is my question: If the input string has Precision Digits = 15 Eg: 5.369607712106161
Then I am getting output = 5.3696077121061609, which is about the same, but not the same number I sent.
or , String 1.145152879691825 gives me number = 1.1451528796918251
String 5.818356760119754 gives me number = 5.8183567601197543

Please suggest how I can make the output more precise. 

Thank you.

Regards,

H.

[Darin.K · Accepted solution]

Wire a constant to the default input of your string to number function and set the type to EXT, it defaults to DBL (notice the red dot on your indicator).

[GGT]

Hello H P,

If I understand your issue correctly, you have simply reached the limits of your PC's (and most everybody elses) ability to repesent floating point numbers accurately. Your machine, being binary and with a limited resolution, simply cannot do it. Simply paste your string into a numeric control set to display 16 digits of precision, hit enter or click outside the control and you see exactly the same thing.

If you're doing calculations with your data, the errors are most likely to be negligable. However, a problem may occur if you are testing such numbers for equality. Have a look at,

http://support.microsoft.com/kb/69333

which I think is quite good. Another avenue may be to factor up your data to integers, which PCs can represent accurately, do your processing, then factor back down at the end. Alas, there will always be minute errors working with precisions such as you are. I'm sure others will offer better guidance than I can.

Regards.

[H P]

Hello Darin

Thank you for the post. It worked!

I do not understnd one thing: My  output number is EXT and the default value is also EXT with precision value set to 0. How does wiring the default value itself works, while not wiring it didnt work before!

Thanks,

H

Message Edited by H P on 12-17-2009 03:45 PM

[Darin.K]

The value of '0' is meaningless (assuming a valid input string), that constant is only there to determine the type.  If you don't connect anything, it assumes DBL regardless of what indicator you connect.

Message Edited by Darin.K on 12-17-2009 01:41 PM

[H P]

Please correct me if I am wrong, 

The figure at http://zone.ni.com/reference/en-XX/help/371361E-01/glang/fract_exp_string_to_number/ indicates that the default is DBL, however the explanation says the default to be extend precision. Am I misunderstanding any thing here.

Thanks.

H

[GGT]

Well, you learn something new about LabVIEW every day, don't you!  Thanks Darin.k. Knew some one would come up with something better.

GGT

[Darin.K]

---

H P wrote:

Please correct me if I am wrong, 

The figure at http://zone.ni.com/reference/en-XX/help/371361E-01/glang/fract_exp_string_to_number/ indicates that the default is DBL, however the explanation says the default to be extend precision. Am I misunderstanding any thing here.

Thanks.

H

---

Words?  You read the words?  I have never noticed that, I tend to look at the control types and the hint in the label.  Looks like a documentation blunder, not the first and not the last.  I guess you got to take this one for the team.

Message Edited by Darin.K on 12-17-2009 01:50 PM

[Darin.K]

---

GGT wrote:

Well, you learn something new about LabVIEW every day, don't you!  Thanks Darin.k. Knew some one would come up with something better.

GGT

---

I'll admit when I first read the post a little voice inside my head went "floating point problem".  Sounds like you have that voice as well, it will serve you very well.