[article_overview]
Debian backup server and domains
What will be backed up:
Server configuration (/etc/, /var/log/, etc)
Web files (php, html, js, css, png, jpg, etc)
Mail (postfix, mailbox, maildir, etc)
Database data (mysql)
[/article_overview]
Quick overview of the backup process:
A bash (command line) script which can be automated to run daily via cron.
The files of interest will be combined into a single file via tar.
The tar file will then be compressed and archived on the server.
The tar file can then be synchronized to your local pc for off-site backup using rsync
Backup server configuration
Create the file structure to store the backups
/home/backup/ - where backup scripts are kept
/home/backup/server - where rolling backups of server configuration are kept
/home/backup/domains - where the latest uncompressed backup of domains are kept for later off-site rsync
/home/domains/<domain>.com/www/backup - where rolling backups of domain web/email/database are kept
1
2
3
4
|
mkdir /home/backup/
mkdir /home/backup/server
mkdir /home/backup/domains
mkdir /home/domains/<domain>.com/www/backup
|
Bash script to back up server configuration
1
|
vi /home/backup/server.sh
|
Add
Note: When copying scripts with long lines, make sure when pasting that the line is not created as two lines
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
|
#!/bin/bash
echo $(date +"%Y-%m-%d %H:%M:%S")" Backup server configuration..."
# get list of installed packages
cd /root
dpkg -l > dpkg_installed.txt
# required! cd to root to quite the tar warning about removing leading /
# and makes recovery easier since have whole dir structure
cd /
SITE='server'
CURRENT_DATE=$(date +%Y%m%d)
CURRENT_BACKUP=home/backup/server/$SITE"_"$CURRENT_DATE".tar"
MYSQL_DUMP=home/backup/server/$SITE"_dbs.sql"
# directories & files to backup
# include any global server data required to restore or reinstall the server
# but exclude any domain specific data
FILES="
${MYSQL_DUMP}
etc
root
home/mysql
home/users
home/backup/server.sh
home/backup/domains.sh
var/log
var/spool"
# directories and files to ignore
EXCLUDE=" --exclude=var/spool/postfix/private --exclude=var/spool/postfix/public "
# dump system databases
# include any global mysql dbs
/usr/bin/mysqldump --add-drop-table --databases mysql mailserver ilohamail \
--lock-all-tables -q -K -Q -C -f -h localhost -uroot -p<password> > $MYSQL_DUMP
# combine all directories and files and db into one file
tar -cpf $CURRENT_BACKUP $EXCLUDE $FILES
# update current backup for rsync
rsync -a $CURRENT_BACKUP home/backup/server/"$SITE".tar
# compress
gzip -f $CURRENT_BACKUP
# keep backup for 7 days on server
OLD_FILES=$(find home/backup/server/*.gz -mtime +7)
rm -f $OLD_FILES
# stats
SIZE=$(du -h ${CURRENT_BACKUP}.gz | awk '{print $1}')
echo $(date +"%Y-%m-%d %H:%M:%S")" Server configuration backed up "$SIZE"."
exit 0;
|
Make it executable, and run it
1
2
3
|
cd /home/backup/
chmod 755 server.sh
./server.sh
|
You should have backups in /home/backup/server
Bash script to backup domains
1
|
vi /home/backup/domains.sh
|
Add
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
#!/bin/bash
# required! cd to root to quite the tar warning about removing leading /
# and makes recovery easier since have whole dir structure
cd /
# array of domains to backup
SITES=( '<doamin1.com>' '<doamin2.com>' '<doamin3.com>')
# list of corresponding mysql users/password; all databases which they have access too will be backed upo
MYSQL_USERS=( '<user1>' '<user2>' '<user3>')
MYSQL_PWDS=( '<password1>' '<password2>' '<password3>')
NBR_SITES=${#SITES[@]}
for (( index = 0; index < NBR_SITES; index++ ))
do
SITE=${SITES[$index]}
MYSQL_USER=${MYSQL_USERS[$index]}
MYSQL_PWD=${MYSQL_PWDS[$index]}
echo $(date +"%Y-%m-%d %H:%M:%S")" Backing up "$SITE"..."
SITE_DIR=home/domains/$SITE
CURRENT_DATE=$(date +%Y%m%d)
# label backup by date
CURRENT_BACKUP=${SITE_DIR}/www/backup/${SITE}_${CURRENT_DATE}.tar
# current mysql dump
MYSQL_DUMP=${SITE_DIR}/www/backup/${SITE}_dbs.sql
# directories and files to include
FILES="
${MYSQL_DUMP}
${SITE_DIR}"
# directories and files to ignore
# dont backup the backups!
EXCLUDE=" --exclude=${SITE_DIR}/www/backup "
# dump all databases users has access too
/usr/bin/mysqldump --add-drop-table --all-databases \
--lock-tables -q -K -Q -C -f -h localhost -u${MYSQL_USER} -p${MYSQL_PWD} > $MYSQL_DUMP
# combine all directories and files into one file
tar -cpf $CURRENT_BACKUP $EXCLUDE $FILES
# make copy in domains for rsync later
# cp -f $CURRENT_BACKUP home/backup/domains/${SITE}.tar
rsync -a $CURRENT_BACKUP home/backup/domains/${SITE}.tar
# compress
gzip -f $CURRENT_BACKUP
# only keep X backups in site dir
OLD_FILES=$(find ${SITE_DIR}/www/backup/*.gz -mtime +7)
rm -f $OLD_FILES
# stats
SIZE=$(du -h ${CURRENT_BACKUP}.gz | awk '{print $1}')
echo $(date +"%Y-%m-%d %H:%M:%S")" "$SITE" backed up "$SIZE"."
done
exit 0;
|
Make it executable, and run it
1
2
3
|
cd /home/backup/
chmod 755 domains.sh
./domains.sh
|
You should have backups in
/home/backup/domains - to be rsynced later
and
/home/domains/<domain#.com>/www/backup - for the end user to access; this should not be a web accessible directory but just sftp accessible to the end user
Automate the backup process via cron
1
|
vi /etc/cron.daily/backup.cron
|
Add
1
2
3
4
5
6
7
8
9
10
11
|
#!/bin/bash
START=$(date +%s)
echo $(date +"%Y-%m-%d %H:%M:%S")" Start daily backup..."
/home/backup/domains.sh
/home/backup/server.sh
END=$(date +%s)
echo $(date +"%Y-%m-%d %H:%M:%S")" End daily backup."
DIFF=$[$END-$START]
DIFF=$[$DIFF/60]
echo "Processing time: "$DIFF" minutes"
echo
|
Make it executable
1
|
chmod 755 /etc/cron.daily/backup.cron
|
Test it
1
|
/etc/cron.daily/backup.cron
|
You should have backups in
/home/backup/server
/home/backup/domains
/home/domains/<domain#.com>/www/backu
Using rsync over ssh, copy the backup to a remote server (or your local pc using
cygwin)
rsync is an bandwidth efficient method of synchronizing data. rsync only transfers the differences between files thus resulting in less bandwidth being used. Transferring over ssh allows the connection to be encrypted and you do not have to open another port just for rsync. However,
rync does not work well against compressed files hence the reason for keeping the uncompressed tar backup in addition to the compressed backup.
Note:
This is a pull and not a push of the backup; a pull is useful when the remote server is not always available (like you local pc).
Local server references where your backups are created; your actual server
Remote server references where you are off-site archiving your backups, such as your local pc
Create the backup user on your local server and secure the connection between your local server and the remote server
1
|
adduser --shell /usr/bin/rssh <remote user>
|
Install a restricted shell to limit what the backup user can do
rssh - Restricted shell allowing only scp, sftp, cvs, rsync
Configure rssh
Change
1
2
3
4
5
6
|
allowscp
allowsftp
#allowcvs
#allowrdist
allowrsync
user="<remote user>:011:10001:" # remote backup; allow rsync and scp
|
On the remote server, create the same user and change to the directory of the user who will be pulling the backup
1
2
|
adduser <remote user>
cd /home/<remote user>
|
On the remote server, create ssh keys to allow login without a password just for the backup user
When asked for a password, just press enter (no password)
1
|
ssh-keygen -t rsa -f .ssh/id_rsa
|
Copy the contents of <remote user>/.ssh/id_rsa.pub on the remote server to
<remote user>/.ssh/authorized_keys on your local server
use copy/paste, vi, scp, ftp, whichever to copy the contents over from the remote host to your server
Example using scp, from the remote server
1
|
scp .ssh/id_rsa.pub <remote user>@<your local server>:/home/<remote user>/.ssh/authorized_keys
|
Example using copy/paste from your local server
1
2
3
|
cd /home/users/<remote user>
mkdir .ssh
vi .ssh/authorized_keys
|
Paste in the contents of <remote user>/.ssh/id_rsa.pub
Note: when you paste in the key, make sure it is all on one line
often, pasting will create multiple lines instead of line wrapping
go to the first line, and press shift-J in vi usually helps
Now, change permissions of the .ssh dir and authorized_keys file
1
2
|
cd /home/users/<remote user>
chown -R <remote user>:<remote user> .ssh
|
Test that you can log in securely without a password
On the remote server
1
2
|
cd /home/<remote user>
scp .ssh/id_rsa <remote user>@<your local server>:/home/users/<remote user>/some_test_file
|
If this is the first time you have connected to your local server from the remote server, you will be prompted to add your local server to the remote servers list of know hosts.
Type yes and press enter
1
2
3
4
|
The authenticity of host '<your local server> (<your local server>)' can't be established.
RSA key fingerprint is 11:11:11:11:11:11:11:11:11:11:11:11:11.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '<your local server>' (RSA) to the list of known hosts.
|
Give the backup usr read permissions to the backups
1
2
|
chgrp <user> /home/backup/server
chgrp <user> /home/backup/domains
|
Bash script to pull backups from your local server to the remote server
1
|
vi /home/<remote user>/pull_server_backup.sh
|
Add
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
#!/bin/bash
echo "rsync my local server backup to local PC.."
echo
# your local server info
SERVER="<your local server ip or dns>"
USER="<remote user>"
SRC1="/home/backup/server/server.tar"
SRC2="/home/backup/domains/*.tar"
# remote server
#DEST="/cygdrive/c/cygwin/server_backup/"
DEST="/home/<remote user>/server_backup/"
CURR_DIR="${DEST}current/"
INCR_DIR="${DEST}"`date +%Y%m%d`"/"
if [ ! -d $CURR_DIR ]
then
mkdir $CURR_DIR
fi
if [ ! -d $INCR_DIR ]
then
mkdir $INCR_DIR
fi
# avz - archive verbose compress
OPTS="-avz --force --ignore-errors --progress --stats --human-readable"
OPTS="$OPTS --timeout=999"
# OPTS="$OPTS -b --backup-dir=$INCR_DIR"
OPTS="$OPTS --log-file=rsync.log"
OPTS="$OPTS --list-only" # do not actually transfer files; for testing
# OPTS="$OPTS --bwlimit=100" # kbps
# OPTS="$OPTS --exclude-from 'rsync.exclude'"
# OPTS="$OPTS --files-from 'rsync.files'"
OPTS="$OPTS -e ssh"
rsync $OPTS $USER@$SERVER:$SRC1 :$SRC2 $CURR_DIR
echo
echo "rsync $OPTS $USER@$SERVER:$SRC1 :$SRC2 $CURR_DIR"
echo
# only keep X backups
OLD_FILES=$(find ${DEST} -maxdepth 1 -mindepth 1 -type d -mtime +14)
if [ -n "$OLD_FILES" ]; then
echo "Deleted "$OLD_FILES
rm -rf $OLD_FILES
fi
# compress incr files
echo "Compressing and archiving backups.."
INCR_FILES=$(find ${CURR_DIR} -type f -name '*.tar')
for FILE in $INCR_FILES; do
OLDSIZE=$(du -hs ${FILE} | awk '{print $1}')
FILENAME=${FILE#$CURR_DIR}
gzip -f -c $FILE > ${INCR_DIR}${FILENAME}.gz
NEWSIZE=$(du -hs ${INCR_DIR}${FILENAME}.gz | awk '{print $1}')
echo "Compressed from "$OLDSIZE" to "$NEWSIZE" : "$FILENAME
done
# stats
echo
SIZE=$(du -hs ${CURR_DIR} | awk '{print $1}')
echo $(date +"%Y-%m-%d %H:%M:%S")" "$CURR_DIR" backed up "$SIZE"."
SIZE=$(du -hs ${INCR_DIR} | awk '{print $1}')
echo $(date +"%Y-%m-%d %H:%M:%S")" "$INCR_DIR" backed up "$SIZE"."
echo
|
Make it executable
1
|
chmod 755 /home/<remote user>/pull_server_backup.sh
|
Test it
1
|
/home/<remote user>/pull_server_backup.sh
|
You should get a list of files rsync would transfer
To actually transfer the files, comment out
1
|
#OPTS="$OPTS --list-only" # do not actually transfer files; for testing
|
Te first run will be slow as rsync has nothing to diff against; the results will be similar to sftp-ing the backup files. Subsequent runs will be much faster. Change some of the data files being backed up and re-run the script - try it.
If the remote server is your local windows pc, install
cygwin,
Form the cygwin setup, select and install ssh and rsync.
Create a windows bat script for cygwin
1
2
3
4
5
6
7
8
|
@echo off
C:
chdir C:\cygwin\bin
bash -l pull_server_backup.sh
pause
|
References:
http://www.debianhelp.co.uk/rsync.htm
http://www.samba.org/ftp/rsync/rsync.html
8
