Home Community Discussion Forums Most Active Software Boards LabVIEW Topic

LabVIEW

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Is it possible to set the precision for the subtraction operator?

Re: Re : Is it possible to set the precision for the subtraction operator?

‎07-25-2008 01:24 AM

LabVIEW 8.5 (8.2?) has fixed points datatypes. Never touched them, but they might work.

Ton
Free Code Capture Tool! Version 2.1.3 with comments, web-upload, back-save and snippets!
Nederlandse LabVIEW user groep www.lvug.nl
My LabVIEW Ideas

LabVIEW, programming like it should be!
0 Kudos
Message 11 of 12
(441 Views)
Reply

Re: Re : Is it possible to set the precision for the subtraction operator?

‎07-25-2008 12:58 PM - edited ‎07-25-2008 12:59 PM

Hi TonP & TonH,

FXP only handles upto 64 bits, thus giving a valid range of upto 1.8E+19 (and an accuracy of 1).
You only get 64 bits with no exponent component at all!

Either you handle IEEE specifications correctly or you have to use some other arbitrary precision format (as mentioned before) for this kind of calculation...


Message Edited by GerdW on 07-25-2008 07:59 PM
Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
0 Kudos
Message 12 of 12
(421 Views)
Reply