03-09-2017 11:47 AM
@AnthonyMN wrote:
Thank you everyone for the help! Just as Bob suspected and everyone else confirmed, it had to do with a super small error in rounding, how "1" wasn't equal to 1. While I realize it's maybe not the correct way to fix it, I added a subtract 0.0001 (one order smaller than the smallest precision.
A better "fix" is to implement your own "Dbl Equality" function, taking into account the precision you want. Here's one way to do it:
I would recommend making this into a Sub-VI with three inputs, one output. I would also code it as a VIG (do you know about those?) so you can get it to "remember" any greater-than-zero "Smallest Delta" input so you only need to use this terminal the first time you call it. I put quotation marks around the name of the Output Boolean -- I could also have called it "X is very close to Y" ...
Bob Schor
03-09-2017 03:36 PM - edited 03-09-2017 03:41 PM
@Bob_Schor wrote:
@AnthonyMN wrote:
Thank you everyone for the help! Just as Bob suspected and everyone else confirmed, it had to do with a super small error in rounding, how "1" wasn't equal to 1. While I realize it's maybe not the correct way to fix it, I added a subtract 0.0001 (one order smaller than the smallest precision.
A better "fix" is to implement your own "Dbl Equality" function, taking into account the precision you want. Here's one way to do it:
I would recommend making this into a Sub-VI with three inputs, one output. I would also code it as a VIG (do you know about those?) so you can get it to "remember" any greater-than-zero "Smallest Delta" input so you only need to use this terminal the first time you call it. I put quotation marks around the name of the Output Boolean -- I could also have called it "X is very close to Y" ...
Bob Schor
I've implemented similar VIs that take into account how many decimal places to carry out the comparison, but I never thought to make one that actually specifies the difference as a number. So now you can say "to the nearest +/- 0.005".