msgbartop
A chronological documentation test project, nothing serious, really!
msgbarbottom

20 May 2007 Finding the 50 largest directories

du -S / | sort -nr | head -n50

Tags: ,

Posted by

15 May 2007 Shell script for removing duplicate files

The following shell script finds duplicate (2 or more identical) files and outputs a new shell script containing commented-out rm statements for deleting them.
You then have to edit the file to select which files to keep – the script can’t safely do it automatically!

OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -print0 |
  xargs -0 -n1 md5sum |
    sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
    sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF;
chmod a+x $OUTF; ls -l $OUTF

Example output (rem-duplicates.sh)

#! /bin/sh
#rm ./gdc2001/113-1303_IMG.JPG
#rm ./reppulilta/gdc2001/113-1303_IMG.JPG

#rm ./lissabon/01-01-2001/108-0883_IMG.JPG
#rm ./kuvat\ reppulilta/lissabon/01-01-2001/108-0883_IMG.JPG

#rm ./gdc2001/113-1328_IMG.JPG
#rm ./kuvat\ reppulilta/gdc2001/113-1328_IMG.JPG

Explanation

  1. write output script header
  2. list all files recursively under current directory
  3. escape all the potentially dangerous characters with xargs
  4. calculate MD5 sums
  5. find duplicate sums
  6. strip off MD5 sums and leave only file names
  7. escape strange characters from the filenames
  8. write out commented-out delete commands
  9. make the output script writable and ls -l it

Tags: , , , , , , , ,

Posted by

07 May 2007 Monitoring /var/log/secure for break-in attempts

date >> ssh-intruders.log ; cat /var/log/secure | grep -i "sshd.*authentication failure" | sort | awk '{FS="rhost="; print $2}' | awk '{FS="user="; print $1}' | grep ".*\..*\." | grep -v "knownhost.com" | grep -v "knownhost2.com" | sort | uniq | while read i; do counter=`grep -i "$i" /var/log/secure | wc -l` ; echo "$counter attempts by $i"; done >> ssh-intruders.log ; cat ssh-intruders.log

Tags: , , ,

Posted by

07 May 2007 Liste opp alle prosesser som kjører på systemet, uten duplikater

# ps ahx --format=%c | sort -u

Tags:

Posted by

07 May 2007 Find duplicate usernames in /etc/passwd

# cat /etc/passwd | awk -F: '{print $1}' | sort | uniq -c | grep -v 1
# cat /etc/shadow | awk -F: '{print $1}' | sort | uniq -c | grep -v 1
# awk -F: '{ print $1, $5}' /etc/passwd

Tags: , ,

Posted by