11-14-2011 01:31 PM
So the question is, what is rounding my numbers? Or, more generally, what have I done wrong here?
The main program this sub.vi runs for needs these numbers as strings, and I think I have either picked the wrong number to string conversion .vi or missed a setting somewhere, becasue the program takes a perfectly good number like -0.340316 and rounds it to -0 (negative zero? that's right, negative zero).
Negativity of the zero aside, the number to decimal string .vi is rounding my numbers to integers.
Solved! Go to Solution.
11-14-2011 01:34 PM
The function is supposed to convert to an integer. Simply go to Help>Show Context Help and read the description. The function you want is called Number to Fractional String.
11-14-2011 01:36 PM
Haha thanks, I had read the first sentence of the context help "Converts a number to a string of decimal digits at least width characters wide" and that just sounded like what I needed, the "decimal string" name is a little misleading to noobs like me.
11-14-2011 02:08 PM
Yes. It has always seemed misleading. In this case it means decimal as opposed to hexadecimal, octal, or binary, rather than decimal where most people think of numbers that have a decimal point.