Re: numeric compare problem

[Albert Howard <starbukk@HOTMAIL.COM>]

Its not a bug in sqr, it is a flaw in the way real numbers are
processed, that gets worse as the precision levels get deeper.  And you
will NEVER see it by trying to display the number.  Basically, after
calculating your number, the value stored in the registers might be
0.0000299999999999999999->.  But you will see this as 0.000030. Try
this..determine to what precision you want to test and move the values
to formatted alpha fields.

  move #price_po to $price_po 9.99999999
  move #new_price to $new_price 9.999999999
  if $new_price <> $price_po
    .
    .
 Another approach is to multiply 10 to the power of precision you want
and truncate.  I.e trunc(#new_price * 100000).

>Date:         Wed, 27 May 1998 20:07:54 -0700
>Reply-To: SQR-USERS@USA.NET
>From: Steve Bollinger <c_sbolli@QUALCOMM.COM>
>Subject:      numeric compare problem
>To: Multiple recipients of list SQR-USERS <SQR-USERS@list.iex.net>
>
>I am running into a problem where a numeric compare operation in SQR
does
>not appear to be operating as it should.  I am on Oracle 7.2.2 and have
>seen this same problem on Sybase.  The code is:
>
>  display ' #new_price: ' noline
>  display #new_price
>  display ' #PRICE_PO: ' noline
>  display #PRICE_PO
>  if #new_price <> #PRICE_PO
>        display 'adjust-schedule-price'
>        do adjust-schedule-price
>  end-if
>
>This code was executed 3 times in a single run of the SQR.  In each
case
>the 2 amounts were equal.  The first two times 'adjust-schedule-price'
was
>not executed.  The third time is was.  Here was the display for that
third
>time:
>
> #new_price: 0.000030
> #PRICE_PO: 0.000030
>adjust-schedule-price
>
>I believe this to be an SQR problem with numeric compares under certain
>conditions, but I don't know how to cure this or exactly what the
>conditions are.  When I change the code:
>
>  if #new_price <> #PRICE_PO
>
>to:
>
>  if to_char(#new_price) <> to_char(#PRICE_PO)
>
>this does cure it.  I just don't want to do this to all my numeric
>compares. I shouldn't have to!
>
>I have already thought it was a rounding error but it don't believe
that is
>the  problem.  #PRICE_PO is loaded directly from a table column that
allows
>5 places.  #new_price is the addition of 2 table columns that both
allow 5
>places.  Part of the problem is that it does not consistently fail.
The
>first two executions of this code test equal correctly.
>
>Any known bugs in this area of SQR.
>


______________________________________________________
Get Your Private, Free Email at http://www.hotmail.com