Re: rounding error in SQR let command?

[Arun Kumar <arunkumar13@YAHOO.COM>]

Rich,
Did you try any logical operators?
some thing like this

let #amount = &amount

if #amount <> &amount
   Show ' The amount after let statement is different'
end-if

SQR has a precision of 32 digits. When you dealing
with high precision number the sqr looks very
deceptive.

HTH

--- "Banks, Richard" <Richard.Banks@CBA.COM.AU> wrote:
> Ivan,
>
> Still getting my rounding error even when using
> round function
>
>   let #c = ROUND(&AMOUNT,2)
>   #debugB show ' SQL: Found Amount ' &amount ' for '
> $source ' via '
> $txn_type ' / ' #c
>
> SQL: Found Amount 199999999999999999.98 for AG1 via
> DR /
> 200000000000000000.000000
>
> Thanks anyway.
>
> > -----Original Message-----
> > From: Turner, Ivan [SMTP:Ivan.Turner@QWEST.COM]
> > Sent: Friday, 15 March 2002 1:49
> > To:   SQR-USERS@list.iex.net
> > Subject:      Re: rounding error in SQR let
> command?
> >
> > a,b,and c should all have the same value.
> >
> > let #a = ROUND(#amount,2)
> > let #b = ROUND(to_char($amount),2)
> > let #c = ROUND(&AMOUNT,2)
> >
> >
> > -----Original Message-----
> > From: Banks, Richard
> [mailto:Richard.Banks@CBA.COM.AU]
> > Sent: Tuesday, March 12, 2002 9:00 PM
> > To: SQR-USERS@list.iex.net
> > Subject: rounding error in SQR let command?
> >
> >
> > I'm retrieving a value from the database like so;
> >
> > begin-select
> > SUM(LINE.MONETARY_AMOUNT) &AMOUNT
> >
> >   let #amount = &AMOUNT
> >   let $amount = &AMOUNT
> >
> >   #debugB show ' SQL: Found Amount ' &amount ' / '
> #amount ' / ' $amount
> >
> > FROM [etc]
> >
> > -----
> >
> > My log has the following output;
> >
> > SQL: Found Amount 199999999999999999.98 /
> 200000000000000000.000000 /
> >
>
199999999999999999.9800000000000000000000000000000000
> >
> > ie the let #amount command is changing the value
> (albiet a tiny percentage
> > of the number in question).  I haven't noticed
> this with any other values
> > before not that I ever see numbers so large.
> >
> > If I try let #amount2 = to_number($amount) I still
> get the
> > rounding/conversion error for SQR
> >
> > or even simpler;
> >
> >   let $rich = '199999999999999999.98'
> >   let #rich = to_number($rich)
> >   show $rich ' / ' #rich
> >
> > produces the same problem.
> >
> > Any suggestions?  Just ignore?  Blame SQR or the
> O/S?
> >
> > FYI the number formatted is
> 200,000,000,000,000,000.00 and I can't even
> > pronounce it!  200 thousand (British not US)
> billions?
> >
> > Using SQR For PeopleSoft 8.16 (NT) on DB2
> >
> >
> > **************   IMPORTANT MESSAGE  **************
> > This e-mail message is intended only for the
> addressee(s) and contains
> > information which may be confidential. If you are
> not the intended
> > recipient
> > please advise the sender by return email, do not
> use or disclose the
> > contents, and delete the message and any
> attachments from your system.
> > Unless specifically indicated, this email does not
> constitute formal
> > advice
> > or commitment by the sender or the Commonwealth
> Bank of Australia (ABN 48
> > 123 123 124) or its subsidiaries.
> > **************************************************


__________________________________________________
Do You Yahoo!?
Yahoo! Sports - live college hoops coverage
http://sports.yahoo.com/