msgbartop
A chronological documentation test project, nothing serious, really!
msgbarbottom

11 Mar 2021 Dovecot backup over SSH using doveadm

Dovecot backup using doveadm over SSHThis is just a short post about how to do  Dovecot Maildir backup using the command doveadm backup initiated from your home server that is not on the Internet, to your Internet facing mail server using SSH as a secure transport medium.

The post is not based on any linux distribution and can be used without any modifications as long as you have access to bash. My particular config is based on Ubuntu 20.04 and Centos 8 in my home lab.

The servers have been named host-A, host-B and host-C to better understand the configuration used.

I have installed Dovecot with a similar config as my Internet facing installation so that all email accounts can be backed up in a safe manner. The home lab is behind NAT and a firewall and is not accessible from the Internet by choice.

The Dovecot mail server on the Internet is placed behind a reverse proxy (HAproxy) in a secure manner and is not accessible directly from the Internet. SSH access directly to the mail server is not allowed, but you can access it by SSH jumping through the Bastion host. To make this as simple and automated as possible I have modified my .ssh/config file with the needed configuration to allow doveadm access the Dovecot server without any problems.

SSH config

To allow my home lab server (host-A) access the Bastion host (host-B) over SSH I have created a custom .ssh/config file with SSH-keys. Config of SSH-keys is not being described here.

Host A – .ssh/config

Host host-B
User username
HostName b.example.com
IdentityFile ~/.ssh/id_rsa
Host host-C
User username
HostName <address of host C>
IdentityFile ~/.ssh/id_rsa
ProxyJump host-B

Host B – .ssh/config

Host host-C
Hostname <address of host C>
IdentityFile ~/.ssh/id_rsa

To verify that our SSH connection is working we start a SSH session fro host A with the command

$ ssh host-C

And if everything is working as expected you are now logged into the mail server over SSH.
This was made possible by the ProxyJump directive in .ssh/config file defined on host-A.

Doveadm backup

The doveadm command is versatile and can be used to perform many tasks, but I am planning it to solve my Dovecot Maildir backup needs. doveadm backup performs one-way synchronization. If there are any changes in the destination they will be deleted, so the destination will look exactly like the source.

You can also use doveadm sync to performs two-way synchronization. It merges all changes without losing anything. Both the mailboxes will end up looking identical after the synchronization is finished.

Backup of Dovecot

We are now ready to do the actual backup of Dovecot using the doveadm backup command. Usually the doveadm command is being run from the source and towards the target host, but in my case I reverse it because my home lab is not accessible from the Internet.

The command to initiate backup of a single user account using doveadm over SSH

# doveadm backup -R -u username@example.com ssh Host-B doveadm dsync-server -u username@example.com

When the backup command is running you will see the following process running on the source host-C

doveadm dsync-server -u username@example.com dsync-server

Similarily you will see the following three processes on the target host, host-A in my home lab

doveadm -v backup -R -u username@example.com ssh host-C doveadm dsync-server -u username@example.com
ssh host-C doveadm dsync-server -u username@example.com dsync-server
ssh -W [IP-address of host-C]:port host-B

To automate things and backup all user emails I use a simple bash script to query Dovecot about all users and perform backup of all accounts, one by one using doveadm backup over SSH.

List all Dovecot users

# doveadm user *@*
user1@example.com
user2@example.com
user3@example.com

The script to backup mail from all users accounts

#!/bin/bash
doveadm user *@* | while read user; do
doveadm -v backup -R -u $user ssh host-C doveadm dsync-server -u $user
done

Options

-v option lets doveadm be verbose
-R option allows us to perform a Reverse backup, ie initiated from target host

If you do not have the same mailbox format in both ends, you can perform a conversion from the source to the target. I am using Maildir on both servers so a conversion is not necessary.

The doveadm backup command can be a little bit tricky if you abort the initial sync of email accounts before it finishes. If this happens you just delete the target directory and start the backup operation again.
To keep your backup updated regularly create a cron job with your doveadm backup command and you are all set.

Tags: , , , , ,

Posted by

09 Feb 2020 Modify Rspamd throughput (RRD) graph

This short post describes how to remove data from Rspamd throughput RRD-graph, usually stored in the rspamd.rrd file. The location depends on the linux distribution, but it is located /var/lib/rspamd/rspamd.rrd on Ubuntu 18.04.

This procedure can most likely be used on all types of RRD-files and is not exclusive for Rspamd.

It is recommended to stop the Rspamd daemon and make a backup of your rspamd.rrd file before you continue.

Backup

$ sudo systemctl stop rspamd.service
$ sudo cp -ax /var/lib/rspamd/rspamd.rrd /var/lib/rspamd/rspamd.rrd-$(date -I)

We have now created a backup file of our RRD-file.

Dump RRD-file

Next we need to create a dump of the RRD-file to a XML-file before we can do any modifications on the data.

$ sudo rrdtool dump /var/lib/rspamd/rspamd.rrd /tmp/rspamd.rrd.xml

Structure of the RRD-file

The Rspamd file is the basis for the graphs and are ordered in archives based on the datasets By day, By week, By Month and By year and you will find the same structure in the rrd-file if you search for 60, 300 600 or 3600 seconds.

    <!-- Round Robin Archives -->
    <rra>
            <cf>AVERAGE</cf>
            <pdp_per_row>60</pdp_per_row> <!-- 60 seconds -->
            <cf>AVERAGE</cf>
            <pdp_per_row>300</pdp_per_row> <!-- 300 seconds -->
            <cf>AVERAGE</cf>
            <pdp_per_row>600</pdp_per_row> <!-- 600 seconds -->
            <cf>AVERAGE</cf>
            <pdp_per_row>3600</pdp_per_row> <!-- 3600 seconds -->

Editing

The XML-file of the RRD-file is now stored in /tmp/rspamd.rrd-xml and can be edited with your preferred editor.

I removed several months of empty data points by searching them up and deleting the lines I did not want.
The values I deleted were inside the <database> tags like this

            <database>
                    <!-- 2019-02-08 23:00:00 CET / 1549663200 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                    <!-- 2019-02-09 00:00:00 CET / 1549666800 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                    <!-- 2019-02-09 01:00:00 CET / 1549670400 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
                    <!-- 2019-02-09 02:00:00 CET / 1549674000 --> <row><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v><v>NaN</v></row>
...

Restore RRD-file

Restore the modified RRD-file and start rspamd

$ sudo rm -f /var/lib/rspamd/rspamd.rrd && \
rrdtool restore -f /tmp/rspamd.rrd.xml /var/lib/rspamd/rspamd.rrd && \
chown _rspamd:_rspamd /var/lib/rspamd/rspamd.rrd
$ sudo systemctl start rspamd.service

Check if Rspamd complains

$ sudo tail -f /var/log/rspamd/rspamd.log 

You either have wrong file permission og done something wrong with the RRD-file uf you see the following error

...
2020-02-08 22:51:24 #15878(controller) ; csession; rspamd_controller_handle_graph: no rrd configured
...

Restore backup

This is the procedure if you made a mistanke and want to restore your backup (you did remember to create a backup before you started?).

Stop rspamd daemon and restore your RRD-backup. I assume you are doing this the same day you created a backup file.

$ sudo systemctl stop rspamd.service
$ sudo cp -ax /var/lib/rspamd/rspamd.rrd-$(date -I) /var/lib/rspamd/rspamd.rrd
$ sudo systemctl start rspamd

Check you rspamd.log and see if you have any error messages

$ sudo systemctl status rspamd.service
$ sudo tail -f /var/log/rspamd/rspamd.log

And thats all.

Tags: , , , , , , ,

Posted by

05 Sep 2014 Backup VMware ESXi using BazaarVCB

This post describes how you can backup your VMware ESXi home installation with free license using BazaarVCB if you do not have a vCenter Server available. Bazaarvcb is the fastest backup solution I have used on the free VMware hypervisor.
Download the latest version from the download page.

The backup script is run by crontab every night and looks like this

#!/bin/bash
bazaarvcbPath="/media/backup/bazaarvcb-0.9.7b-linux-i386/bazaarvcb"
backupsPath="/media/backup"
hostname="192.168.0.222"
username="root"
password="password"
rollOut="30"
vmNames="vm-guest1 vm-guest2 vm-guest3"

for VM in $vmNames; do
        `$bazaarvcbPath backup -H $hostname -u $username -p $password --roll-out $rollOut $VM $backupsPath/$VM`
done

The backups are full so make sure you have enough disk space available.

bazaarvcb options

$ bazaarvcb -h
usage: bazaarvcb [-h]  ...

optional arguments:
  -h, --help   show this help message and exit

valid commands:
    checkhash  check .hsh files integrity in one directory
    listvm     list registered VMs on the ESXi host
    queryvm    display VMs informations
    listbackup
               search for backups in local and remote directories
    querybackup
               display report file of one particular
    backup     backup a VM
    restore    restore a backup

Open TCP port 31031 in your firewall to ensure that you have a high transfer rate, otherwise the backup will be transferred over SSH protocol and will be capped in speed to about 7MB/s (on the free hypervisor).
Bazaarvcb cannot backup a VM with snapshot(s).

Warning!
I have not tested this on a host connected to a vCenter server and can not confirm that it will work or not.

Tags: , , ,

Posted by

02 Aug 2012 Backup of Zimbra MailBox using zmmailbox

This is a short script I use to backup the Zimbra mailbox content for my users.
This has been used on a Zimbra Collaboration Server (ZCS Open Source Edition) 7.2 installation, but should work on earlier versions as well.
I use another script to backup the whole Zimbra installation, but that might be another blog post.


#!/bin/bash -x
# Backup of Zimbra mailboxes using zmmailbox
# Restore of mailbox should be performed using:
# /opt/zimbra/bin/zmmailbox -z -m user@host postRestURL -u https://HOST "//?fmt=tgz&resolve=skip" mailbox-name-date.tgz
BackupFolder="/backup/zimbra"
MailBox="user1 user2 user3 userN"
DateToday=`date -I`
for name in $MailBox
do
sudo -u zimbra /opt/zimbra/bin/zmmailbox -z -m $name@pario.no getRestURL "//?fmt=tgz" > mailbox-$name-$DateToday.tgz
done

The backup files are named mailbox-user1-20120802.tgz mailbox-user2-20120802.tgz …

Tags: , , , , , ,

Posted by

02 Nov 2009 Howto backup mysql stored functions and stored procedures

This is how you can backup you MySQL database(s) and stored procedures

# mysqldump --routines <dbname>

Or you can backup only the stored procedures

# mysqldump --no-create-db --no-create-info --no-data --routines <dbname>

Tags: , , ,

Posted by