Re: # variables in SQR

[Andrew Barnett <abarnett@ISD.HIH.COM.AU>]

>We've noticed some strange behavior in SQR when using # variables in SQR.
snip
>Sometimes they're equal, sometimes they're not.  It sure make things
>difficult when trying to compare these numbers and the result is not
>what it seems it should be!  Does this mean I should round # variables
>at all times as a general rule?
>
>Thanks,
>Tim Harris
>Systems Analyst
>Oregon State University

I've struck a problem in the past that is possibly related. I had a simple
loop with a counter variable. I wanted to display something every 10 loops
so I checked for MOD(#COUNTER, 10) = 0. Sometimes it didn't work when it
should have, say when #COUNTER = 20. When I PRINTed or DISPLAYed #COUNTER to
as many decimal places as possible it showed as exactly 20. If I subtracted
20 from it I got what looked like exactly zero. However, if I used

   if #COUNTER = 20

or

   if (#COUNTER - 20) = 0

they both returned false. #COUNTER had suffered some sort of a rounding
error which was so small as to be undisplayable but was there nonetheless.
What makes it worse is that I was only incrementing it by exactly 1, no
fractions. Rounding did fix the problem.

--------------------------------------------------------
 Andrew Barnett                 abarnett@isd.hih.com.au
 - Wizzard
--------------------------------------------------------