how to control a sigificant of digits?

[labmaster]

Dear all,

I am writing some voltage sweeping code with VISA-GPIB communication.

First, the all voltages (ext. type) are calculated with full digits in advance and then saved into an array.

But my instrument want to get 3 digits only (0.001) and I would like to record the real voltage in the instrument.

How to control the significant of digits (for the input connector of the VISA writing) in array or line connection?

Thank you in advance

[GerdW]

Hello labmaster,

you probably want to send those voltage values to your voltage supply as text formatted (GPIB-communication).
To do so you should use "Format into string" with format string like "%.3f" to get 3 digits after the decimal separator.
Btw extended format is kind of overkill if you only need 3 digits. DBL will half your memory needs and SGL will do too (if there is no extensive mathematics to do).

[labmaster]

Very thank you for the reply.

Can you tell me other numerical way not using a string function?

Espeically, I would like to control the digits just before the connection to the VISA.

[GerdW]

Hello labmaster,

the standard way of rounding will help you: multiply by 1000 (=3 digits after separator), round to nearest, divide by 1000.
But be aware: using floating point numbers can give "unwanted" results as not all fractions can be represented by them! Example: comparision of floating point numbers can give unexpected results...

Other solution:
If applicable you can use Integer numbers and treat them as fixed point numbers. The last three digits represent the fractional part, all upper digits the integer part.
Example: 12345 will be interpreted as 12.345, 9.876 will be represented by 9876. But you need your own math routines (add/sub will work anyway, mult/div needs correctional factors and so on).