Re: SQR Floating Point problem

[Aya Fall <afall@NS.SYMPATICO.CA>]

Hi Clayton:

You should convert your variables to character variables.
Example, if you are storing the 13.45 in a numeric variable #Total1, you
should convert the #Total1 to a character variable:
  let $Total1 = edit (to_char((#Total1),'99.99')
and you print $a. It should do it.

Hope this helps.

Aya

Clayton Moore wrote:
>
> Hello!
>
> For auditing purposes, we are accumulating the total amount of debits and
> credits, and the total number of records processed.  We then compare those
> accumulated amounts to a trailer record that has control totals.  We are
> processing over 500,000 records and over $1 billion (US dollars) worth of
> debits and credits.
>
> We are finding that the accumulated totals have amounts past the second
> significant digit.  For example, when we thought the total should be
> $13.45, SQR was saying the total was 13.450002, or even 13.449998.
> Although the amount of the difference is small, over a half million records
> it can add up to differences in the accumulated and control totals.
>
> PeopleSoft has a fix in the number.sqc file:  PS_SQR_ROUND and
> PS_SQR_TRUNC.  We are currently using these routines, but found they are
> not 100% reliable, as the problem is still occuring.
>
> Has anyone else experienced this problem?  If so, how were you able to fix
> it?
>
> Thanks in advance for your time and help!