msgbartop
A chronological documentation test project, nothing serious, really!
msgbarbottom

18 May 2007 Delete files older than x days

find /data/folder -type f -ctime +60 -exec /bin/rm {} \;

(more…)

Tags:

Posted by

17 May 2007 Courier-IMAP to Dovecot Migration Script

I’ve just migrated a couple of servers from Courier-IMAP to Dovecot, and am very happy with the latter so far. I thought I’d share the courier2dovecot shell script I whipped together (based on the instructions I found in the migration how-to), for converting Courier-IMAP maildirs to Dovecot format.

While the script is rather simple, it can save a fair bit of typing when migrating some hundreds of mail accounts, especially since Courier stores some of its own state files recursively in each folder of the maildir hierarchy, making it a real pain to otherwise manually rename or remove all of them.

Here’s a plain-English summary of what the script currently does, when given a Courier maildir path as an argument:

  • Check that the given path is indeed a valid Courier maildir, and exit if not.
  • Rename the file ‘courierimapsubscribed’ to ‘subscriptions’, and strip the ‘INBOX.’ prefix from its contents.
  • Recursively rename all of files of the name ‘courierimapuiddb’ to ‘dovecot-uidlist’.
  • Recursively delete all of files of the name ‘courierimaphieracl’, ‘courierimapacl’ and ‘courierimapkeywords’.

The script will also verbosely print out each action it performs, handy for redirecting the output to a log file for a large migration.

Hopefully people about to jump ship from Courier to Dovecot find this useful. If anyone comes up with improvements to the script, please send them my way.

#!/bin/sh
#
# courier2dovecot -- Converts a Courier maildir to Dovecot format.
# Copyright (c) 2005 Arto Bendiken. Released under the GNU GPL.
# Newest version available from http://bendiken.net/scripts/
#
# 2005-10-21 initial version for Dovecot 1.0.
#
dir="$1"
if [ -z "$dir" ] || [ "$dir" = "-?" ] || [ "$dir" = "-h" ] || [ "$dir" = "--help" ]; then
  echo "Usage: $0 maildirpath"
  exit 1
fi
if [ ! -d "$dir" ] || [ ! -e "$dir/courierimapsubscribed" ]; then
  echo "$dir is not a path to a Courier maildir"
  exit 1
fi
find $dir -name courierimapsubscribed -print0 | xargs -0r rename -v 's/courierimapsubscribed/subscriptions/'
find $dir -name subscriptions -print0 | xargs -0r sed -i 's/INBOX\.//'
find $dir -name courierimapuiddb -print0 | xargs -0r rename -v 's/courierimapuiddb/dovecot-uidlist/'
find $dir -name courierimaphieracl -print0 | xargs -0r rm -vrf
find $dir -name courierimapacl -print0 | xargs -0r rm -vf
find $dir -name courierimapkeywords -print0 | xargs -0r rm -vrf

Comment to the script:
Hey Arto & everyone using this script:

Be careful when trying to run this on Red Hat Enterprise Linux or CentOS 4 systems. These operating systems ship with a different rename command than the Debian systems. You can also tell them apart based on the output when running ‘rename’ with no arguments.

This is the perl version which works:
Usage: rename [-v] [-n] [-f] perlexpr [filenames]

This is the RHEL/CentOS version that doesn’t:
call: rename.orig from to files…

Hope this helps someone out there, and thanks for the script Arto!

Source: http://bendiken.net/2005/11/03/courier-imap-to-dovecot-migration-script

Tags: , , , , , , , , , ,

Posted by

15 May 2007 Shell scripts for archiving digital photos in directories by month

This is my version of the shell script “Shell scripts for archiving digital photos in directories by date“. It utilizes  the exiftags command available in most linux distributions like Gentoo and debian Linux. Though the metacam program is also usefull since it can read Nikon NEF-files.Here’s an example of a directory tree they create:

2006  2006-10       IMG_48324.JPG       IMG_48325.JPG       IMG_48326.JPG       IMG_48331.JPG       IMG_48333.JPG       IMG_48334.JPG       IMG_48337.JPG  2006-11       IMG_48338.JPG  ...etc...

move-digiphotosThis bash script (move-digifotos) scans EXIF tags from .JPG files in current directory with metacam, creates necessary directories under $BASEDIR and moves the files in them:

#!/bin/bash# Reads EXIF creation date from all .JPG files in the# current directory and moves them carefully under##   $BASEDIR/YYYY/YYYY-MM/YYYY-MM-DD/## ...where 'carefully' means that it does not overwrite# differing files if they already exist and will not delete# the original file if copying fails for some reason.## It DOES overwrite identical files in the destination directory# with the ones in current, however.## This script was originally written and put into# Public Domain by Jarno Elonen ; in June 2003.# Feel free to do whatever you like with it.BASEDIR=/home/backup/testfind -maxdepth 1 -name "*.JPG" | while read x; doDATE=`exiftags "$x" | \   egrep "^[ t]*Image Created:" | \   sed -r "s/Image Created: ([0-9:]*).*/\1/"`if [ ! -z "$DATE" ];then    YEAR=`echo $DATE | sed -r "s/([0-9]*):([0-9]*):([0-9]*)/\\1/"`    MONTH=`echo $DATE | sed -r "s/([0-9]*):([0-9]*):([0-9]*)/\\2/"`    if [ "$YEAR" -gt 0 ] & [ "$MONTH" -gt 0 ]    then    INSTDIR=${BASEDIR}/${YEAR}/${YEAR}-${MONTH}    install -d "$INSTDIR"    INSTFILE="$INSTDIR/$x"    if [ -e "$INSTFILE" ] && ! cmp -s "$x" "$INSTFILE"    then        echo "WARNING: '$INSTFILE' exists already and is different from '$x'."    else        echo "Moving '$x'"        cp -ax "$x" "$INSTFILE"        if ! cmp -s "$x" "$INSTFILE"        then        echo "WARNING: copying failed somehow, will not delete original '$x'"        else        rm -f "$x"        fi    fi    else    echo "WARNING: '$x' doesn't contain date."    fielse    echo "WARNING: '$x' doesn't contain date."fidone

Tags: , , , , , , , , , ,

Posted by

15 May 2007 Shell scripts for archiving digital photos in directories by date

The fact that most digital cameras store the shooting dates inside JPEG files in the EXIF format makes it possible to automatically archive the photos by date right after they are downloaded, which is exactly what the scripts below do.

Here’s an example of a directory tree they create:

1999
  1999-07
     1999-07-14
       IMG_48324.JPG
       IMG_48325.JPG
       IMG_48326.JPG
     1999-07-17
       IMG_48331.JPG
       IMG_48333.JPG
       IMG_48334.JPG
       IMG_48337.JPG
  ...etc...

move-digiphotos
This bash script (move-digifotos) scans EXIF tags from .JPG files in current directory with metacam, creates necessary directories under $BASEDIR and moves the files in them:

#!/bin/bash

# Reads EXIF creation date from all .JPG files in the
# current direcotry and moves them carefully under
#
#   $BASEDIR/YYYY/YYYY-MM/YYYY-MM-DD/
#
# ...where 'carefully' means that it does not overwrite
# differing files if they already exist and will not delete
# the original file if copying fails for some reason.
#
# It DOES overwrite identical files in the destination directory
# with the ones in current, however.
#
# This script was originally written and put into
# Public Domain by Jarno Elonen  in June 2003.
# Feel free to do whatever you like with it.

  BASEDIR=/home/jarno/gfx

find -maxdepth 1 -name "*.JPG" | while read x; do
  DATE=`metacam "$x" | \
    egrep "^[ \t]*Image Capture Date:" | \
    sed -r "s/Image Capture Date: ([0-9:]*).*/\1/"`
  if [ ! -z "$DATE" ];
  then
    YEAR=`echo $DATE | sed -r "s/([0-9]*):([0-9]*):([0-9]*)/\\1/"`
    MONTH=`echo $DATE | sed -r "s/([0-9]*):([0-9]*):([0-9]*)/\\2/"`
    DAY=`echo $DATE | sed -r "s/([0-9]*):([0-9]*):([0-9]*)/\\3/"`
    if [ "$YEAR" -gt 0 ] & [ "$MONTH" -gt 0 ] & [ "$DAY" -gt 0 ]
    then
      INSTDIR=${BASEDIR}/${YEAR}/${YEAR}-${MONTH}/${YEAR}-${MONTH}-${DAY}
      install -d "$INSTDIR"
      INSTFILE="$INSTDIR/$x"
      if [ -e "$INSTFILE" ] && ! cmp -s "$x" "$INSTFILE"
      then
        echo "WARNING: '$INSTFILE' exists already and is different from '$x'."
      else
        echo "Moving '$x'"
        cp "$x" "$INSTFILE"
        if ! cmp -s "$x" "$INSTFILE"
        then
          echo "WARNING: copying failed somehow, will not delete original '$x'"
        else
          rm -f "$x"
        fi
      fi
    else
      echo "WARNING: '$x' doesn't contain date."
    fi
  else
    echo "WARNING: '$x' doesn't contain date."
  fi
done

fetch-digiphotos
The fetch-digiphotos script fetches pictures from a camera through gphoto2 into a temporary directory under $TMPDIR, moves them with move-digiphotos and finally offers to delete them from the camera:

#!/bin/bash

# Downloads pictures from a digital camera using gphoto2,
# moves them with move-digiphotos.sh and optionally deletes
# them from the camera's memory.
#
# This script was originally written and put into
# Public Domain by Jarno Elonen  in June 2003.
# Feel free to do whatever you like with it.

TMPDIR=/home/jarno/gfx/tmp

if [ ! -d "$TMPDIR" ]; then
echo "*** Error: download directory '$TMPDIR' does not exist, aborting."
exit 1
fi
pushd "$TMPDIR" > /dev/null

gphoto2 -P
if [ $? != 0 ]; then
echo "*** error executing gphoto2, aborting."
popd > /dev/null
exit 1
fi

move-digiphotos

echo
echo -n "Delete pictures from camera? [y/N] "
read x
if [ $x == "y" ]; then
gphoto2 -D
fi
if [ $? != 0 ]; then
echo "Warning: error executing gphoto2."
popd > /dev/null
exit 1
fi

echo "Done."
popd > /dev/null

Kilde: Jarno Elonen, 2003-06-17

Tags: , , , , , ,

Posted by

15 May 2007 Shell script for removing duplicate files

The following shell script finds duplicate (2 or more identical) files and outputs a new shell script containing commented-out rm statements for deleting them.
You then have to edit the file to select which files to keep – the script can’t safely do it automatically!

OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -print0 |
  xargs -0 -n1 md5sum |
    sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
    sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF;
chmod a+x $OUTF; ls -l $OUTF

Example output (rem-duplicates.sh)

#! /bin/sh
#rm ./gdc2001/113-1303_IMG.JPG
#rm ./reppulilta/gdc2001/113-1303_IMG.JPG

#rm ./lissabon/01-01-2001/108-0883_IMG.JPG
#rm ./kuvat\ reppulilta/lissabon/01-01-2001/108-0883_IMG.JPG

#rm ./gdc2001/113-1328_IMG.JPG
#rm ./kuvat\ reppulilta/gdc2001/113-1328_IMG.JPG

Explanation

  1. write output script header
  2. list all files recursively under current directory
  3. escape all the potentially dangerous characters with xargs
  4. calculate MD5 sums
  5. find duplicate sums
  6. strip off MD5 sums and leave only file names
  7. escape strange characters from the filenames
  8. write out commented-out delete commands
  9. make the output script writable and ls -l it

Tags: , , , , , , , ,

Posted by