Register
It is currently Fri Sep 19, 2014 11:52 pm

Searching / hard linking / removing duplicate files


All times are UTC - 6 hours


Post new topic Reply to topic  [ 1 post ] 
Author Message
 PostPosted: Tue Mar 18, 2008 3:22 pm   

Joined: Fri Mar 07, 2008 4:50 pm
Posts: 1
This is a script for searching and hard linking duplicate files or removing / showing them with little changes. The script searches for files with the same size which are not already linked and calculates md5sum for them. Files with the same md5 are the same and can be linked or deleted or whatever you prefer. For hard linking the file system must support inodes of course. The script doesn't change anything but produces a script for replacing duplicates with hard links. If you wish to remove duplicates instead of hard linking them then just replace the line with "echo": echo rm \'$f\'
Pros: Doesn't touch already linked files and doesn't calculate md5sum for them.
Cons: It might be not optimal solution for a lot of huge files with the same size.

#!/bin/bash
# This script is intended for hard linking duplicate files or removing / showing them with little changes.
# The script doesn't change anything but finds duplicates and produces a script for replacing duplicates with
# hard links.
find "$@" -type f -printf "%20s %20i %p\n" \
| sort | uniq -D -w 21 \
| sort -k 2 | uniq -f 1 -w 21 \
| sort | uniq -D -w 21 \
| cut -b 43- | sort | tr \\n \\000 \
| xargs -0 md5sum | sort | uniq -D -w 32 \
| {
while read h f; do
if test "${f:0:1}" = '*' ; then f="${f:1}" ; fi
if test "$h" != "$ph" ; then ph="$h"; pf="$f"
else echo rm \'$f\' \&\& ln \'$pf\' \'$f\' \|\| exit 3; fi
done
} | sort


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 1 post ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP