Register
It is currently Mon Dec 22, 2014 9:55 am

awk for breaking a file into smaller files


All times are UTC - 6 hours


Post new topic Reply to topic  [ 2 posts ] 
Author Message
 PostPosted: Fri Sep 26, 2008 9:20 am   

Joined: Fri Sep 26, 2008 7:58 am
Posts: 10
Hi all this is my first post and I am having a little trouble solving this. I need to break a file up into many (many...many...many) smaller files based on the occurrence of the number "3" at the end of a line.

As an example of what I'm trying to accomplish:

a b c d 3
a b f g
t h g
j r i 0 3
a f t w

I wanted to end up outputting this:
1st file:
a b c d 3
a b f g
t h g

2nd file:
j r i 0 3
a f t w

At first I tried this script:

Code:
awk '/'3$'$/{d++}{print > d"extract.txt"}' $1


It worked perfectly, however the original input text file contained just over 960000 lines. As my first attempt did not contain a "close" statement Linux informed me that:

Quote:
awk: cannot open "1020extract.txt" for output (Too many open files)


However when I put the "close" in:

Code:
awk '/'3$'$/{d++}{print > d"extract.txt"}{close ( d"extract.txt" )}' $1


I only get the line preceding each occurrence of a line ending with "3"...I suppose it is closing the file before anything more can be written to it? Any ideas?


Top
 Profile  
 PostPosted: Mon Sep 29, 2008 8:17 am   

Joined: Fri Sep 26, 2008 7:58 am
Posts: 10
Not to worry all the problem has been solved.

Used grep instead.

Code:
stopword='3$'

cat $1 |while read line; do
   echo $line |grep $stopword && filecounter=$(( filecounter +1 ))
   echo $line >> newfile.$filecounter
done


Thanks all the same


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 2 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP