I believe you are refering to the number of displayeddigits. Right click on the control, select properties, go to format... Then you can define the number of significant digits, or the number of digits after the separator...
You should be posting questions about TestStand to that forum but I think what you want is done in the sequence editor. when configuring the numeric limit, click the Numeric Format button, change Type to Real and set Number of Fractional Digits to whatever you want.