Re: Reading a flat file

[Wayne Ivory <wivory@WSL.COM.AU>]

(Resent because it hadn't appeared on the list after 24 hours)

Parijat,

I'm happy you got a working solution from Tony.  Just a comment about my
suggestion of reading the entire file to which you replied "What if the file
was a huge 5MB?!!".  My response would be "So what?"  Many computers could
quite happily cope with this amount of data in memory.

Those of us who've been in the industry awhile tend to have warnings ringing
in the back of our heads that we picked up from our early days of
programming regarding the foolhardiness of using such "huge" amounts of
memory in a program.  But the reality is that systems these days are
light-years ahead of what we were using back then and it's no longer the
issue it used to be.  Of course it would still be possible to "cross the
line", but before discarding such avenues I would try them out and see if
they can run without impacting performance.

As I said I'm glad you found your solution.  I'm just trying to encourage
open minds.

Regards

Wayne Ivory
Information Services
Westralian Sands Limited


-----Original Message-----
From: Parijat Sahai [mailto:SQR-USERS@USA.NET]
Sent: Thursday, 25 February 1999 2:59
To: parijatsahai@HOTMAIL.COM; SQR-USERS@list.iex.net
Subject: Re: Reading a flat file


Hi Tony,

Thanks for the solution. This was exactly what I was looking for.

Just a comment for some other solutions suggested. One was to read all
data in a string variable and then use "substr". What if the file was a
huge 5MB?!! Another was to read 12 chars in a filler or junk variable
and discard it. Probably it was being assumed that EACH record contained
first 12 chars as junk, which wasn't the case. Another was to read the
file char by char (!!) and create a new file skipping the first 12
chars, and then use this new file for further processing. Terrible, as
far as performance is concerned. Imagine a 5MB file being copied char by
char!

Anyway, thanks everyone for your replies. This mail was an effort to
summarize the various solutions suggested and to highlight the most
efficient and elegant one.

Just a quick note, Tony. In your solution, you don't need to have the
last piece of code after "end-while". The last record (12 chars long)is
processed inside the while loop itself. After the last record (12 chars
long) is read and processed, #end-file is set to 1 when SQR issues
another "Read" command and then it exits the loop. Now $out contains
either end-of-file char or linefeed char depending upon the last char in
flat file which should be ignored.

Also, "$out <> ''" should be replaced by "not isblank($out)" coz
presence of linefeed char or end-of-file char in $out gives
unpredictable (or incorrect) result in the former case. I checked that
on my 95 machine.

Thanks.

Parijat.


>Date:         Tue, 23 Feb 1999 17:52:46 -0500
>Reply-To: SQR-USERS@usa.net
>From: Tony DeLia <tdelia@EROLS.COM>
>Subject:      Re: Reading a flat file
>To: Multiple recipients of list SQR-USERS <SQR-USERS@list.iex.net>
>
>Hi Parijit,
>
>   I understand your problem... "After the first 12 bytes" you need to
>read fixed length records of 150 characters... and there are NO
>line-feed characters... In effect this means...
>
> Record 1 =  12 bytes
> Record 2 = 150 bytes
> Record 3 = 150 bytes
> Record n = 150 bytes
>
>I don't know of a way to accomplish this directly with the OPEN
>statement...
>BUT off the top of my head I would try this approach...
>
>**********************************************************************
>
>open $ifile as 1 for-reading record=150:fixed_nolf
>
>let $out = ''
>
>while 1 = 1
>
>   read 1 into $rec:150
>
>   if #end-file = 1
>      break
>   end-if
>
>   let $rec_1 = substr($rec,1,12)
>   let $rec_2 = substr($rec,13,138)
>
>   if $out <> ''
>      let $out = $out || $rec_1
>      do Process-Full-Record
>   end-if
>
>   let $out = $rec_2
>
>end-while
>
>!  Process Last Record
>if $out <> ''
>   let $out = $out || $rec_1
>   do Process-Full-Record
>end-if
>
>**********************************************************************
>
>I would double-check the last record read since it won't be 150
>characters... (Last chunk should be 12 bytes)... This is just one way
to
>handle this...
>
>                                              -Tony DeLia
>
>Venu Gopal wrote:
>>
>> Parijat
>>
>> You read entire record into one varible and then use extract you can
>> extract from a particular position
>>
>> Venu
>>
>> AlphaSoft Services Corp.
>
>--
>Tony DeLia
>AnswerThink Consulting Group
>PeopleSoft Solutions Practice - Delphi Partners
>tdelia@erols.com


______________________________________________________
Get Your Private, Free Email at http://www.hotmail.com