Re: rounding error in SQR let command?

[James Womeldorf <jwomeldo@FASTENAL.COM>]

Richard,

In VB I believe a 'double precision' number has about 15 or 16 significant
digits, which means that any number which contains more than 16 (or so)
digits will probably not be exactly what you would expect.

The following indicates that SQR is probably based upon the same precision:

let #n = 100000000000000000000
let #nn = #n / 3
show #nn

In the log file we find:
33333333333333300000.000000

So I think your problem is not a rounding problem but a precision problem.

Jim

Jim Womeldorf
Programmer/Analyst
Fastenal Company
jwomeldo@fastenal.com
(507) 453-8250


-----Original Message-----
From: Banks, Richard [mailto:Richard.Banks@CBA.COM.AU]
Sent: Sunday, March 17, 2002 4:56 PM
To: SQR-USERS@list.iex.net
Subject: Re: rounding error in SQR let command?


Ivan,

Still getting my rounding error even when using round function

  let #c = ROUND(&AMOUNT,2)
  #debugB show ' SQL: Found Amount ' &amount ' for ' $source ' via '
$txn_type ' / ' #c

SQL: Found Amount 199999999999999999.98 for AG1 via DR /
200000000000000000.000000

Thanks anyway.

> -----Original Message-----
> From: Turner, Ivan [SMTP:Ivan.Turner@QWEST.COM]
> Sent: Friday, 15 March 2002 1:49
> To:   SQR-USERS@list.iex.net
> Subject:      Re: rounding error in SQR let command?
>
> a,b,and c should all have the same value.
>
> let #a = ROUND(#amount,2)
> let #b = ROUND(to_char($amount),2)
> let #c = ROUND(&AMOUNT,2)
>
>
> -----Original Message-----
> From: Banks, Richard [mailto:Richard.Banks@CBA.COM.AU]
> Sent: Tuesday, March 12, 2002 9:00 PM
> To: SQR-USERS@list.iex.net
> Subject: rounding error in SQR let command?
>
>
> I'm retrieving a value from the database like so;
>
> begin-select
> SUM(LINE.MONETARY_AMOUNT) &AMOUNT
>
>   let #amount = &AMOUNT
>   let $amount = &AMOUNT
>
>   #debugB show ' SQL: Found Amount ' &amount ' / ' #amount ' / ' $amount
>
> FROM [etc]
>
> -----
>
> My log has the following output;
>
> SQL: Found Amount 199999999999999999.98 / 200000000000000000.000000 /
> 199999999999999999.9800000000000000000000000000000000
>
> ie the let #amount command is changing the value (albiet a tiny percentage
> of the number in question).  I haven't noticed this with any other values
> before not that I ever see numbers so large.
>
> If I try let #amount2 = to_number($amount) I still get the
> rounding/conversion error for SQR
>
> or even simpler;
>
>   let $rich = '199999999999999999.98'
>   let #rich = to_number($rich)
>   show $rich ' / ' #rich
>
> produces the same problem.
>
> Any suggestions?  Just ignore?  Blame SQR or the O/S?
>
> FYI the number formatted is 200,000,000,000,000,000.00 and I can't even
> pronounce it!  200 thousand (British not US) billions?
>
> Using SQR For PeopleSoft 8.16 (NT) on DB2
>
>
> **************   IMPORTANT MESSAGE  **************
> This e-mail message is intended only for the addressee(s) and contains
> information which may be confidential. If you are not the intended
> recipient
> please advise the sender by return email, do not use or disclose the
> contents, and delete the message and any attachments from your system.
> Unless specifically indicated, this email does not constitute formal
> advice
> or commitment by the sender or the Commonwealth Bank of Australia (ABN 48
> 123 123 124) or its subsidiaries.
> **************************************************