msgbartop
A chronological documentation test project, nothing serious, really!
msgbarbottom

11 Mar 2021 Dovecot backup over SSH using doveadm

Dovecot backup using doveadm over SSHThis is just a short post about how to do  Dovecot Maildir backup using the command doveadm backup initiated from your home server that is not on the Internet, to your Internet facing mail server using SSH as a secure transport medium.

The post is not based on any linux distribution and can be used without any modifications as long as you have access to bash. My particular config is based on Ubuntu 20.04 and Centos 8 in my home lab.

The servers have been named host-A, host-B and host-C to better understand the configuration used.

I have installed Dovecot with a similar config as my Internet facing installation so that all email accounts can be backed up in a safe manner. The home lab is behind NAT and a firewall and is not accessible from the Internet by choice.

The Dovecot mail server on the Internet is placed behind a reverse proxy (HAproxy) in a secure manner and is not accessible directly from the Internet. SSH access directly to the mail server is not allowed, but you can access it by SSH jumping through the Bastion host. To make this as simple and automated as possible I have modified my .ssh/config file with the needed configuration to allow doveadm access the Dovecot server without any problems.

SSH config

To allow my home lab server (host-A) access the Bastion host (host-B) over SSH I have created a custom .ssh/config file with SSH-keys. Config of SSH-keys is not being described here.

Host A – .ssh/config

Host host-B
User username
HostName b.example.com
IdentityFile ~/.ssh/id_rsa
Host host-C
User username
HostName <address of host C>
IdentityFile ~/.ssh/id_rsa
ProxyJump host-B

Host B – .ssh/config

Host host-C
Hostname <address of host C>
IdentityFile ~/.ssh/id_rsa

To verify that our SSH connection is working we start a SSH session fro host A with the command

$ ssh host-C

And if everything is working as expected you are now logged into the mail server over SSH.
This was made possible by the ProxyJump directive in .ssh/config file defined on host-A.

Doveadm backup

The doveadm command is versatile and can be used to perform many tasks, but I am planning it to solve my Dovecot Maildir backup needs. doveadm backup performs one-way synchronization. If there are any changes in the destination they will be deleted, so the destination will look exactly like the source.

You can also use doveadm sync to performs two-way synchronization. It merges all changes without losing anything. Both the mailboxes will end up looking identical after the synchronization is finished.

Backup of Dovecot

We are now ready to do the actual backup of Dovecot using the doveadm backup command. Usually the doveadm command is being run from the source and towards the target host, but in my case I reverse it because my home lab is not accessible from the Internet.

The command to initiate backup of a single user account using doveadm over SSH

# doveadm backup -R -u username@example.com ssh Host-B doveadm dsync-server -u username@example.com

When the backup command is running you will see the following process running on the source host-C

doveadm dsync-server -u username@example.com dsync-server

Similarily you will see the following three processes on the target host, host-A in my home lab

doveadm -v backup -R -u username@example.com ssh host-C doveadm dsync-server -u username@example.com
ssh host-C doveadm dsync-server -u username@example.com dsync-server
ssh -W [IP-address of host-C]:port host-B

To automate things and backup all user emails I use a simple bash script to query Dovecot about all users and perform backup of all accounts, one by one using doveadm backup over SSH.

List all Dovecot users

# doveadm user *@*
user1@example.com
user2@example.com
user3@example.com

The script to backup mail from all users accounts

#!/bin/bash
doveadm user *@* | while read user; do
doveadm -v backup -R -u $user ssh host-C doveadm dsync-server -u $user
done

Options

-v option lets doveadm be verbose
-R option allows us to perform a Reverse backup, ie initiated from target host

If you do not have the same mailbox format in both ends, you can perform a conversion from the source to the target. I am using Maildir on both servers so a conversion is not necessary.

The doveadm backup command can be a little bit tricky if you abort the initial sync of email accounts before it finishes. If this happens you just delete the target directory and start the backup operation again.
To keep your backup updated regularly create a cron job with your doveadm backup command and you are all set.

Tags: , , , , ,

Posted by

15 Jul 2015 Virtualize physical Ubuntu linux server

This post describes how to create a bit for bit copy of a Ubuntu 14.04 LTS server using tools like gddrescue and qemu from a Ubuntu Live-CD. This procedure can actually be used to create a copy of any operating system, not just Ubuntu.

This could probably have been done more easily and faster using VMware vCenter Converter Standalone Client but I have experienced that not all linux flavours can be converted easily.

To perform such an operation you need several things.

  • Extra harddrive – same size or bigger than used disk space
  • Access to the linux packages gddrescue, qemu-imgThe first thing you should do to reduce the time ddrescue uses to copy the data is to remove unneeded files and folders as well as removing harddrives that you do not want to copy.

Creating the disk image

# ddrescue --nosplit /dev/sda imagefile.img imagefile.log

The –nosplit option copies the disk without retrying or splitting the file and is also “fast”.
Remember to place the imagefile.img on another harddisk than you are imageing to avoid filling your disk.

Convert the img file to a VWware VMDK disk file

# qemu-img -p convert -f raw sda.img -O vmdk sda.vmdk

Options used
-p gives you a nice progress indication of the conversion
-f raw tells us that we a trying to convert a raw disk image
-O vmdk describes the output format of the new disk image, in this example a VMware VMDK-file

qemu-img (1.5.0) supports a wide range of formats like vvfat vpc vmdk vhdx vdi sheepdog sheepdog sheepdog rbd raw host_cdrom host_floppy host_device file qed qcow2 qcow parallels nbd nbd nbd dmg tftp ftps ftp https http cow cloop bochs blkverify blkdebug.

Tags: , , , , , ,

Posted by

08 Oct 2014 Set og change default “Open With” application in Ubuntu

This post describes how to associate a file ending like PAR2-files with the pypar2 application in Ubuntu Unity.
You can also use the mimeopen command to change associated file type to other applications.

$ mimeopen -d unknown-file.par2
Please choose a default application for files of type application/x-par2

	1) Other...

use application #1
use command: /usr/bin/pypar2
Opening "unknown-file.par2" with usr  (application/x-par2)

pypar2 will now open the files automatically when you open it using Nautilus filemanager.

This has been tested on Ubuntu 13.10 and 14.04.

Tags: , , , ,

Posted by

19 Nov 2013 Installing build-essentials in CentOS or Ubuntu

This post desrcibes how to install “build-essentials” on CentOS/RedHat og Ubuntu/Debian. build-essentials provides the packages you need to build or compile source code (make and gcc) and create binary files. The build-essentials package is a quick way to install the packages needed to compile VMware Tools on a VMware guest.

In Ubuntu

$ sudo aptitude install build-essentials

In CentOS

# yum groupinstall "Development Tools"

You might also need kernel sources to compile some software

# yum install kernel-devel kernel-headers

Tags: , , ,

Posted by

09 Jul 2013 Add new harddisk to linux without reboot

This is a short post on how you can add a new SATA og SCSI harddisk in your running/live linux machine without a reboot.
The BIOS of the machine is not aware of new hardware being added, but you can ask the disk controllers to rescan and become aware of new harddisks.

This is how you force a rescan of the SCSI controller in your linux machine
$ sudo echo "- - -" > /sys/class/scsi_host/host3/scan
I used host3 because the disk was connected to that controller, but you should replace it with the controller you use.

To see the result of the rescan you can issue the dmesg command
$ sudo dmesg
[ 3117.041231] ata3: hard resetting link
[ 3117.371445] ata3: SATA link down (SStatus 0 SControl 300)
[ 3117.371464] ata3: EH complete
[ 3128.224802] ata4: exception Emask 0x10 SAct 0x0 SErr 0x4050000 action 0xf
[ 3128.224809] ata4: SError: { PHYRdyChg CommWake DevExch }
[ 3128.224818] ata4: hard resetting link
[ 3129.100450] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 3129.108545] ata4.00: ATA-8: ST31500341AS, HP23, max UDMA/100
[ 3129.108550] ata4.00: 2930277168 sectors, multi 0: LBA48 NCQ (depth 0/32)
[ 3129.140500] ata4.00: configured for UDMA/100
[ 3129.151034] ata4: EH complete
[ 3129.151159] scsi 3:0:0:0: Direct-Access ATA ST31500341AS HP23 PQ: 0 ANSI: 5
[ 3129.151491] sd 3:0:0:0: Attached scsi generic sg1 type 0
[ 3129.151568] sd 3:0:0:0: [sdb] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
[ 3129.151895] sd 3:0:0:0: [sdb] Write Protect is off
[ 3129.151900] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 3129.152052] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 3129.176718] sdb: sdb1
[ 3129.178535] sd 3:0:0:0: [sdb] Attached SCSI disk
[ 3571.306535] ata4: hard resetting link
[ 3571.781238] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 3571.821270] ata4.00: configured for UDMA/100
[ 3571.831850] ata4: EH complete
[ 3756.103000] ata4: hard resetting link
[ 3756.574619] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 3756.614674] ata4.00: configured for UDMA/100
[ 3756.625209] ata4: EH complete

This example has been tested on Ubuntu 12.04, but should work on any other linux flavour available.
This post was highly inspired by http://www.cyberciti.biz/tips/vmware-add-a-new-hard-disk-without-rebooting-guest.html that also describes how you remove devices etc withour reboot. This is a great way to add new harddisk to a virtualized linux server that cannot be rebooted.

Tags: , , , ,

Posted by