Register
It is currently Mon Dec 22, 2014 3:35 pm

Need help with script loop


All times are UTC - 6 hours


Post new topic Reply to topic  [ 20 posts ] 
Author Message
 PostPosted: Tue Jan 19, 2010 9:19 pm   

Joined: Tue Jan 19, 2010 9:10 pm
Posts: 12
I would like to know how to make a script that I can run a bunch of commands using a variable then have it rerun with the next variable from a list of uid's one per line

This is so I can track who leaves Facebook or defriends me. I would like it to also track the same for any of my friends who want to know also. Hence a uid for each friend who wants my help
Something like

#/bin/bash

#list of uids

346823923
343242863
974924732

#block of commands to run from each uid

curl --user-agent "Firefox" -b cookies.txt http://www.facebook.com/ajax/typeahead_friends.php/\?u\=$uid\&__a\=1 -o $uid.new
sed -i s/\{\"t\"\:/'\n'\{\"t\"\:/g $uid.new
sed -i s/\]\,\"viewer/\,'\n'\]\,\"viewer/g $uid.new
diff base/$uid.base $uid.new &> log/$uid.log
cat log/$uid.log
mv log/$uid.log log/$uid."`date`".log
cp $uid.new old/$uid."`date`"
mv $uid.new base/$uid.base


Top
 Profile  
 PostPosted: Wed Jan 20, 2010 4:48 am   

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221
Use a for-loop (or a while read, i prefer for :P).

If you have the UID's in a text file named uids.txt you can do something like this. (just a space or carriage return seperated list.)
Code:
for uid in $(cat uids.txt);do
   curl --user-agent "Firefox" -b cookies.txt http://www.facebook.com/ajax/typeahead_friends.php/\?u\=$uid\&__a\=1 -o $uid.new
   sed -i s/\{\"t\"\:/'\n'\{\"t\"\:/g $uid.new
   sed -i s/\]\,\"viewer/\,'\n'\]\,\"viewer/g $uid.new
   diff base/$uid.base $uid.new &> log/$uid.log
   cat log/$uid.log
   mv log/$uid.log log/$uid."`date`".log
   cp $uid.new old/$uid."`date`"
   mv $uid.new base/$uid.base
done


There's probably a better ways of checking for it but this should do the trick :)

Best regards
Fredrik Eriksson


Top
 Profile  
 PostPosted: Wed Jan 20, 2010 8:56 am   

Joined: Mon Mar 02, 2009 3:03 am
Posts: 579
no mercy! this is all so messy.

where does the 'uid list' come from? is it an array, a file? else? (from space?)

you should comment your script so we/you would know what it does, 'cos right now it sems to do unnecessary things.

please, provide some of curl's output sample...

__
it's a bad day.


Top
 Profile  
 PostPosted: Wed Jan 20, 2010 11:29 am   

Joined: Tue Jan 19, 2010 9:10 pm
Posts: 12
Thanks for the help I got it to work. And just if anyone is interested in using it here is my more complete version with a lot of comments. I know some could be better but it works. The uids are kept one per line in a seperate text file. The only thing stopping this from being cronable is that I have to make sure I have a current Facebook session logged in to get lists from people who hide theirs.



Code:
#!/bin/bash

#Copies the cookies database to get around it being locked if anyone uses it must be changed to your file location
cp ~/.mozilla/firefox/gwatdpoo.SecondaryProfile/cookies.sqlite .

#Extracts the Facebook cookies to text using a seperate shell script then deletes the database file
sh extractcookies cookies.sqlite | grep facebook > cookies.txt
rm cookies.sqlite

#Loop to run for every uid
for uid in `cat uids`
do
#Downloads friend list
curl --user-agent "Firefox" -b cookies.txt http://www.facebook.com/ajax/typeahead_friends.php/\?u\=`echo $uid`\&__a\=1 -o $uid.new

#Formats it so it can be diff'd
#Makes each person have their own line since the default is one long line
sed -i s/\{\"t\"\:/'\n'\{\"t\"\:/g $uid.new
#Moves a little bit at the end to its own line to help the diff if something changes and they are not the end of the list anymore
sed -i s/\]\,\"viewer/\,'\n'\]\,\"viewer/g $uid.new
#Nothing to do with the script it just cleans up the urls so they can be copy and pasted to a browser
sed -i 's/\\\//\//g' $uid.new
#Adds a new line to the end of the file just to stop diff from complaining
echo >> $uid.new

#Finds differences against old version and logs it
diff base/$uid.base $uid.new > log/$uid.log
#I want to see what is happening as it happens so this shows the current uid then shows the saved log
echo $uid
cat log/$uid.log
#Moves the log into storage
mv log/$uid.log log/$uid."`date`".log

#Keeps copy then moves new version to the base that will be used next time
cp $uid.new old/$uid."`date`"
mv $uid.new base/$uid.base
done


The extractcookies script for no reason other then someone might like to use this method to track their friends list
Code:
#!/bin/bash
sqlite3 "$1" <<EOF
.mode tabs
.header off 
select host as domain, 
  case substr(host,1,1)='.' when 0 then 'FALSE' else 'TRUE' end as flag,
  path, 
  case isSecure when 0 then 'FALSE' else 'TRUE' end as secure,
  expiry as expiration, name, value
from moz_cookies;   
EOF


Top
 Profile  
 PostPosted: Fri Jan 22, 2010 10:29 pm   
Site Admin
User avatar

Joined: Sun May 15, 2005 9:36 pm
Posts: 673
Location: Des Moines, Iowa
That's a pretty interesting script actually.


Top
 Profile WWW  
 PostPosted: Sat Jan 23, 2010 12:35 am   

Joined: Tue Jan 19, 2010 9:10 pm
Posts: 12
Ok since I want to track a few people it is getting annoying to only refer to them by uid so if I was to modify the uids file to be some thing like this.

327457234 "John Doe"
326832238 "Jane Doe"

How would I modify the "for uid in `cat uids`" to only use the uid and maybe have a separate $name variable with everything between the same lines "s so that I can echo it instead of the $uid when outputting the changes to the terminal and maybe depending on my mood use $name as part of the file names.


Top
 Profile  
 PostPosted: Sat Jan 23, 2010 8:37 pm   

Joined: Mon Mar 02, 2009 3:03 am
Posts: 579
You choose a separator for your uids file
Code:
327457234;John Doe;
326832238;Jane Doe;
then you change the for-loop for a while-loop, defining Internal Field Separator environmental variable for the read comand to be the separator of uids file
Code:
while IFS=';' read uid name
do #for example
   echo "_${name}_ has uid #$uid"
done < uids
which will output
Code:
_John Doe_ has uid #327457234
_Jane Doe_ has uid #326832238


while-loop is the best way to parse a file, because you can
read the_whole_line
or
read first_element second_element rest_of_the_line


Last edited by Watael on Sat Jan 23, 2010 8:59 pm, edited 1 time in total.

Top
 Profile  
 PostPosted: Sat Jan 23, 2010 8:54 pm   

Joined: Tue Jan 19, 2010 9:10 pm
Posts: 12
Thank you. I got that to work and it made the output easy to read.


Top
 Profile  
 PostPosted: Sun Jan 24, 2010 1:47 pm   

Joined: Tue Jan 19, 2010 9:10 pm
Posts: 12
Well every time I get finished I eventually get bored and tweak it a little. and find new stuff I don't know how to do. Although this boredom has made it almost something that could be released and people could figure out easily. I have used more pipes to reduce file saves to the hard drive and cleaned up the friend list output.

My question now is when I use if to check for a directory or that the log has contents how can I check for not existing or if the the log is zero sized so that I can just use a then and not a else. Which leaves me with random crap in the then. This is worded badly but will be apparent if you look at the script.

Also any other suggestions to make this more professional looking.


Code:
#!/bin/bash


#Couple variables to edit
COOKIEDIR=~/.mozilla/firefox/gwatdpoo.SecondaryProfile
VERBOSE=yes
#Default is numeric by uid
SORTALPHA=no




#Checks if needed directories exist and creates them if needed
if [[ -d base ]]; then nouse="justneededsomethingforthethen" ; else mkdir base ; fi
if [[ -d old ]]; then nouse="justneededsomethingforthethen" ; else mkdir old ; fi
if [[ -d log ]]; then nouse="justneededsomethingforthethen" ; else mkdir log ; fi


#Note a valid session cookie is only needed to get friend lists for people who don't show it to everyone
#Confirms it can make a fresh cookie
echo "Have you made sure you are logged into Facebook"
echo "Press enter to continue... "
read

#Copies the cookies database to get around it being locked and attempts to find it if not properly told the location
if [[ -e $COOKIEDIR/cookies.sqlite ]] ; then cp $COOKIEDIR/cookies.sqlite . ; else cp `locate cookies.sqlite | head -n 1` . ; fi

#Extracts the Facebook cookies to text then deletes the database file
sqlite3 cookies.sqlite <<EOF
.output cookies.txt
.mode tabs
.header off 
select host as domain, 
  case substr(host,1,1)='.' when 0 then 'FALSE' else 'TRUE' end as flag,
  path, 
  case isSecure when 0 then 'FALSE' else 'TRUE' end as secure,
  expiry as expiration, name, value
from moz_cookies;   
EOF

rm cookies.sqlite

#Saves the date and time in a variable
DATE=`date`

#Loop to run for every uid
while IFS=';' read uid name
do
#Downloads friend list
curl --silent --user-agent "Firefox" --cookie cookies.txt http://www.facebook.com/ajax/typeahead_friends.php/\?u\=`echo $uid`\&__a\=1 | \

#Formats it so it can be diff'd and easily read
#Makes each person have their own line since the default is one long line
sed s/\{\"t\"\:/'\n'/g | \
#Nothing to do with the script it just cleans up the urls so they can be copy and pasted to a browser
sed 's/\\\//\//g' | \
#Cleanup
sed s/\,\"i\"\:/'\t''\t'/g | \
#More cleanup
sed s/\,\"u\"\:\"/'\t'/g | \
#Even more cleanup
sed s/\"\,\"n\".*$// | \
#Creates new file that only has people and not random info at beginning of file
grep facebook > "$name".new

#Sorts list alphabetically if wanted
if [[ $SORTALPHA == yes ]]
then
   mv "$name".new "$name".new2
   sort "$name".new2 > "$name".new
   rm "$name".new2
fi


#Finds differences against old version and logs it
diff base/"$name".base "$name".new > log/"$name"."$DATE".log

#I want to see what is happening as it happens so this shows log followed by some space to easily see where different people start
if [[ $VERBOSE == yes ]]
then
   echo $name
   cat log/"$name"."$DATE".log
   echo ""
   echo ""
   echo ""
   echo ""
   echo ""
fi

#Moves the log into storage if it has contents
if [[ -s log/"$name"."$DATE".log ]]
then
   nouse="justneededsomethingforthethen"
else
   rm log/"$name"."$DATE".log
fi


#Keeps copy then moves new version to the base that will be used next time
cp "$name".new old/"$name"."$DATE"
mv "$name".new base/"$name".base
done < uids


Top
 Profile  
 PostPosted: Sun Jan 24, 2010 2:28 pm   

Joined: Mon Mar 02, 2009 3:03 am
Posts: 579
exclamation mark (!) can be used for negation
Code:
for directory in base old log; do
   if [ ! -d $directory ]; then mkdir $directory; fi
done
or shorter without exclamation mark but a OR (||)
Code:
for directory in base old log; do [ -d $directory ] || mkdir $directory; done


Top
 Profile  
 PostPosted: Mon Feb 01, 2010 11:58 pm   

Joined: Tue Jan 19, 2010 9:10 pm
Posts: 12
Ok I have been using this for awhile and need a workaround for some annoyances. If I leave the list in default uid order it randomly gets sent from Facebook out of order which cause false inserts and deletions when comparing with a list in the right order. If I sort it alphabetically it constantly changes because people constantly change their names for some reason. (respell their name, add or remove middle name or I have seen some young girls who just for fun hyphenate their last name with a celebrity's last name.)

How can I take my list which is in this format *note the names have " in the actual file
"John Doe" (2 tabs) uid (1 tab) url
and rearrange it to
uid (tab) "John Doe" (tab) url

Then I can sort based on uid which never changes and atleast get rid of the false deletions and insertions and have them properly show up as changes


Top
 Profile  
 PostPosted: Tue Feb 02, 2010 2:03 am   

Joined: Mon Mar 02, 2009 3:03 am
Posts: 579
You could use sed (GNU sed):
Code:
cat your_list.in
"John Doe"              uid     url
sed -i 's/\([^  ]*\)            \([^    ]*\)    \(.*\)/\2       \1      \3/' your_list.in
# or
#sed -i 's/\([^\t]*\)\t\t\([^\t]*\)\t\(.*\)/\2\t\1\t\3/' your_list.in
cat your_list.in
uid   "John Doe"   url


Top
 Profile  
 PostPosted: Tue Feb 02, 2010 3:11 pm   

Joined: Tue Jan 19, 2010 9:10 pm
Posts: 12
Thank you


Top
 Profile  
 PostPosted: Wed Feb 03, 2010 1:56 pm   
User avatar

Joined: Mon Jan 18, 2010 8:10 pm
Posts: 40
darkshadow, would you please explain this line to me? How does it work?
Quote:
Code:
#Downloads friend list
curl --silent --user-agent "Firefox" --cookie cookies.txt http://www.facebook.com/ajax/typeahead_friends.php/\?u\=`echo $uid`\&__a\=1 | \

I have tried 'curl' before, but this line seems to involve more than simply reading the 'html'.
Where do you get the 'facebook' '$uid'?
How do these "--user-agent" and "--cookie" options work?
Your scripts sound exciting, and I want to try them, but I get nothing from this line.
:-/


Top
 Profile  
 PostPosted: Wed Feb 03, 2010 6:52 pm   

Joined: Tue Jan 19, 2010 9:10 pm
Posts: 12
The --user-agent is to fake being a web browser even though I don't use a full agent. Facebook requires one to prevent automatic downloading of files (even though it only fools the newest of users)

The --cookies loads a authenticated sessions credentials and is only needed for some people if they have their friend list hidden from non friends. Although it is impossible to block people from seeing your list if they know this url.

To find a users uid go to their profile and it will be in the url if they don't have a vanity url.
example http://www.facebook.com/profile.php?id=3826483428
If they use a vanity url I get their uid by viewing the filename of their profile picture it will be everything after the letter and before the _
example n3826483428_5393.jpg

you save the uid and name in a external "uids" file using the following format note the ;'s
2386423842;John Doe;
8329164832;Jane Doe;


Here is my most up to date version of the script
Code:
#!/bin/bash


#Couple variables to edit
COOKIEDIR=~/.mozilla/firefox/gwatdpoo.SecondaryProfile
#Shows logs on terminal, recommended or you have to open each log individually.
VERBOSE=yes




#Checks if needed directories exist and creates them if needed
for directory in base old log; do
   if [ ! -d $directory ]; then mkdir $directory; fi
done


#Copies the cookies database to get around it being locked and attempts to find it if not properly told the location
if [[ -e $COOKIEDIR/cookies.sqlite ]] ; then cp $COOKIEDIR/cookies.sqlite . ; else cp `locate cookies.sqlite | head -n 1` . ; fi

#Extracts the Facebook cookies to text then deletes the database file
sqlite3 cookies.sqlite <<EOF
.output cookies.txt
.mode tabs
.header off 
select host as domain, 
  case substr(host,1,1)='.' when 0 then 'FALSE' else 'TRUE' end as flag,
  path, 
  case isSecure when 0 then 'FALSE' else 'TRUE' end as secure,
  expiry as expiration, name, value
from moz_cookies;   
EOF

rm cookies.sqlite

#Saves the date and time in a variable
DATE=`date`

#Loop to run for every uid
while IFS=';' read uid name
do
#Downloads friend list
curl --silent --user-agent "Firefox" --cookie cookies.txt http://www.facebook.com/ajax/typeahead_friends.php/\?u\=`echo $uid`\&__a\=1 -o "$name".fl

#If download is empty exits and tells you to login to Facebook
grep facebook "$name".fl > "$name".test
if [[ ! -s "$name".test ]]
then
        rm "$name".fl "$name".test
        echo "Please login to Facebook to generate a valid session cookie then retry"
        exit 1
else
   rm "$name".test
fi

#Formats it so it can be diff'd and easily read
#Makes each person have their own line since the default is one long line
sed s/\{\"t\"\:/'\n'/g "$name".fl | \
#Nothing to do with the script it just cleans up the urls so they can be copy and pasted to a browser
sed 's/\\\//\//g' | \
#Cleanup
sed s/\,\"i\"\:/'\t''\t'/g | \
#More cleanup
sed s/\,\"u\"\:\"/'\t'/g | \
#Even more cleanup
sed s/\"\,\"n\".*$// | \
#Creates new file that only has people and not random info at beginning of file
grep facebook | \
#Rearranges list for sorting incase Facebook sent any in the wrong order which prevents extra output from diff
sed 's/\([^\t]*\)\t\t\([^\t]*\)\t\(.*\)/\2\t\1\t\3/' | \
#Sorts the list
sort -n > "$name".new

#Removes unformatted friend list
rm "$name".fl

#Finds differences against old version and logs it
diff base/"$name".base "$name".new > log/"$name"."$DATE".log

#I want to see what is happening as it happens so this shows log followed by some space to easily see where different people start
if [[ $VERBOSE == yes ]]
then
   echo $name
   cat log/"$name"."$DATE".log
   echo ""
   echo ""
   echo ""
   echo ""
   echo ""
fi

#Removes the log if it is empty
if [[ ! -s log/"$name"."$DATE".log ]]
then
   rm log/"$name"."$DATE".log
fi


#Keeps copy then moves new version to the base that will be used next time
cp "$name".new old/"$name"."$DATE"
mv "$name".new base/"$name".base
done < uids


Top
 Profile  
 PostPosted: Thu Feb 04, 2010 7:49 am   
User avatar

Joined: Mon Jan 18, 2010 8:10 pm
Posts: 40
Okay, this is looking better...

I found the correct folder for COOKIEDIR, where my '.mozilla/firefox/[MyFolderName]/cookies.sqlite' file is located. And I installed 'sqlite' and 'sqlite3'.

Now, by line 41, "rm cookies.sqlite", I have a text file './cookies.txt' with all the information from 'cookies.sqlite' copied to it.

It looks like I am set up to run the line, "curl --silent --user-agent "Firefox" --cookie cookies.txt http://www.facebook.com/ajax/typeahead_friends.php/\?u\=`echo $uid`\&__a\=1 | \".

Good stuff, darkshadow!
I'm lovin' it!

g o =:)

Nope! I will have to create a dummy 'uid.txt' file first. I will try using, as a '$uid', the value of my facebook cookie, 'c_user', which is a ten digit number like the examples you listed.


Top
 Profile  
 PostPosted: Thu Feb 04, 2010 8:04 am   
User avatar

Joined: Mon Jan 18, 2010 8:10 pm
Posts: 40
Success!

When I use my own facebook uid in the curl command, before all the sed cleanup, I get returned a massive string with all my facebook friends listed.
Listed along with the friends' names are the fields, '"i":iiiiiiiii', where "i" is given as a nine-digit number, presumably their facebook uids.

Thanks, darkshadow!

P.S. (to anyone who doesn't already know their facebook uid number), here is how to find it in the firefox (iceweasel) web-browser:

Main toolbar > Edit > Preferences > Privacy
"Show Cookies" button
In the directory tree, open the folder "facebook.com", and highlight the item with cookie name, "c_user".
This is the cookie containing your facebook uid, the contents will be displayed in the GUI window below.
The content of this cookie is your facebook.com uid number ("c_user" number).
(From here, if you follow along with darkshadow's script, you should be able to retrieve all the uids for the people on your facebook friends list.)

:)


Last edited by gofree on Thu Feb 04, 2010 9:13 am, edited 1 time in total.

Top
 Profile  
 PostPosted: Thu Feb 04, 2010 8:24 am   
User avatar

Joined: Mon Jan 18, 2010 8:10 pm
Posts: 40
gofree wrote:
Listed along with the friends' names are the fields, '"i":iiiiiiiii', where "i" is given as a nine-digit number, presumably their facebook uids.


Actually there are fields like this: '"u":"http:\/\/www.facebook.com\/profile.php?id=uuuuuuuuu"', where "u" is a nine- (or ten-) digit number, the same as "i" (above)...

(Unless they have a "vanity url", as darkshadow said, where the value for "u" will be a personalized url, perhaps with their name, instead of the standard, "profile.php?id=".)

Exactly as darkshadow described:
Quote:
To find a users uid go to their profile and it will be in the url if they don't have a vanity url.
example http://www.facebook.com/profile.php?id=3826483428


Top
 Profile  
 PostPosted: Thu Feb 04, 2010 9:11 am   
User avatar

Joined: Mon Jan 18, 2010 8:10 pm
Posts: 40
Now that I have all my friends uids, if I try one of them in the line:
Quote:
Code:
curl --silent --user-agent "Firefox" --cookie cookies.txt http://www.facebook.com/ajax/typeahead_friends.php/\?u\=`echo $uid`\&__a\=1
...
I get another massive string, similar to the one with my facebook id number, except with the friends list of the person whose uid I used.
Works like a charm!

B-)

Thanks, darkshadow!

:)


Top
 Profile  
 PostPosted: Thu Feb 04, 2010 6:16 pm   

Joined: Tue Jan 19, 2010 9:10 pm
Posts: 12
just note the uid/names filename is "uids" not "uid.txt" if you want to change it just modify the last line in the script since it is "done < FILENAME"


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 20 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP