Category: Linux

Duplicity backups

I’m running Buffalo Linkstation as an home storage for some files. I have previously rooted (to be able to install my own apps) that and installed some opt packages. I also have service from ISP where I can store several Gigabytes of data e.g. for backup purposes.

Recently I finally completed the Duplicity backups,  here is how it basically went.

Prerequisites for me was:
– Linux server where you store your files to be backed up (I have Buffalo Linkstation)
– Remote place where you store (I have Kapsi ISP)

First install the duplicity

For me as I’m using rooted Buffalo Linkstation LS-QVL 8TB I’m using commands like

ipkg install py25-duplicity
ipkg install py25-boto
ipkg install py25-crypto
ipkg install py25-paramiko

I had to install quite many packages but paramiko is the one that I’m actually using (ssh, sftp). This process wasn’t that straight forward and first tries didn’t end up in success, thats why implementing this took so long.

Also prerequisites for duplicity there is librsync GnuPG NcFTP Boto and Python 2.4 or later. Those you will have to install also.

Then I created the following scripts to do the backup.

fullbackup.sh

#!/bin/sh
# duplicity full backup to kapsi

#Email address of recipient. Adjust to suit.
export Email="where_to@send_the_reports.com"

# timestamp for email
date=`date +"%d.%m.%Y %k:%M:%S"`

# Export the PASSPHRASE variable
export PASSPHRASE=Your_secret_password!
export TMPDIR=/opt/tmp/duplicity

# Get the date
repDate=`date "+%Y%m%d_%H%M%S"`

# Cleanup first (anything older than 60 days)
/opt/bin/duplicity-py2.5 remove-older-than 60D -v9 --force scp://account@server.com:22//home/users/account/siilo/backup/duplicity >>/var/log/fullbackup_$repDate.log

# Now do the backup
/opt/bin/duplicity-py2.5 full /mnt/array1/path --asynchronous-upload --volsize 100 --include-globbing-filelist=/mnt/array1/path/cfg/backupaccount.txt scp://account@server.com:22//home/users/account/siilo/backup/duplicity >>/var/log/fullbackup_$repDate.log

# Get the disk space
echo "Availble Disk Space on Server" >> >>/var/log/fullbackup_$repDate.log
echo >>/var/log/fullbackup_$repDate.log
/opt/bin/ssh -p22 account@server.com quota -v -s >> >>/var/log/fullbackup_$repDate.log
/opt/bin/ssh -p22 account@server.com du -h ///home/users/account/siilo/backup/duplicity >>/var/log/fullbackup_$repDate.log

# Mail me the results
cat /var/log/fullbackup_$repDate.log | nail -s "Backup Full [$date]" $Email

–volsize parameter changes the default 25MB filesize to 100MB and –asynchronous-upload starts sending the package immediately once it’s ready and packs the second package. This increases the temporary space needed but makes transfers faster.

I have similar script with incremental option to use in incremental backups.

Cron

00 4 1 * * /opt/local/bin/fullbackup.sh
00 4 2-31 * * /opt/local/bin/incrbackup.sh

backupaccount.txt

/mnt/array1/path/Dir1
/mnt/array1/path/Dir2
/mnt/array1/path/Dir3
- /mnt/array1/path/Dir4/misc
- /mnt/array1/path/Dir4/not_important
/mnt/array1/path/Dir4
- /mnt/array1/path/trashbox
- **

On backupaccount.txt file I just say the directories under /mnt/array1/path what I want to be included and excluded. So the /mnt/array1/path/Dir4 gets copied but it excludes few directories. Also in the end there is to exclude trashbox and ** for rest from /mnt/array1/path.

This is basically how the things gets backed up.

Benefits

– Crypted backups, so you can send them where ever you want without fear of getting your files to wrong hands
– Space and Bandwith efficient. Incrementals are really clever and fast to do even for big file repositories.
– Adaptivity, it supports many different protocols and file format is standard. You can probably have it running same way in the future services also.

I think for my purposes duplicity backups is really good way to do backups to remote location safely.