Re: Numeric Var Comparisons

[gadiy@sun4.miti.com]

     Kay,
     
     The problem that you're outlining is due to the fact that SQR uses 
     machine floating point (C double) for numbers. Certain calculation 
     result in 45.320999998 instead of 45.321. Truncating makes the problem 
     worse. Instead you should round:
     
     if round(#database,3) != round(#calc,3)
     
     The next release of SQR (Version 3.5 due later this year) will offer 
     exact match using high precision decimal numbers.
     
     Gadi Yedwab
     MITI
     


______________________________ Reply Separator _________________________________
Subject: Numeric Var Comparisons
Author:  usa.net!sqr-users@netcomsv.netcom.com at Internet
Date:    6/8/95 2:26 PM


  We are running SQR 2.5.7 on Oracle 7.0.16.4.0 on a VAX.  When trying
to compare some calculated values against stored values in our database 
and flag the ones that were different, SQR was flagging values that were 
the same.  SQR was not consistent in doing this though.  For example, 
sometimes it would flag #calc = 345.89 as different from #database = 
345.89, and other times it would say it was the same.  
It did however always catch the incorrect values.
     
  Below is a little sample program that I just created to see if it
was SQR or me that was messing up.  When the program is run it 
flags them as NOT EQUAL.  If I put a TRUNC(#val,3) around each
variable in the IF statement then it catches them as EQUAL.  But this 
is not the case in the actual code that runs against ORACLE.
     
  What's up with this?  Any help or suggestions on where to go would
be greatly appreciated.
     
  Thanks,
     Kay
     
     
BEGIN-REPORT
     
 let #database = 45.321
     
  do q
     
END-REPORT
     
begin-procedure q
 let #calc = 145.321 - 100.000
 display 'database ' noline
 show #database   edit 99999.9999999
 display 'cacl ' noline
 show #calc  edit 99999.9999999
     
  if trunc(#database,3) != trunc(#calc,3)
     display 'NOT EQUAL'
  else
     display 'EQUAL'
  end-if
     
end-procedure