Incremental Backup using Rsync and Hard links

Introduction :

Rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated.
The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.
Some of the additional features of rsync are:

* support for copying links, devices, owners, groups, and permissions
* exclude and exclude-from options similar to GNU tar
* a CVS exclude mode for ignoring the same files that CVS would ignore
* can use any transparent remote shell, including ssh or rsh
* does not require root privileges
* pipelining of file transfers to minimize latency costs
* support for anonymous or authenticated rsync daemons (ideal for mirroring)

Hard Link Concepts:

Hard links are similar to symlinks. They are normally created using the ln command but without the -s switch. A hard link is when 2 file entries point to the same inode and disk blocks. Unlike symlinks there isn't a file and a symlink but rather 2 links to the same file. If you delete either entry the other will remain and will still contain the data. Here is an example of both:

------- Symbolic Link Demo -------

% echo foo > x
% ln -s x y

% ls -li
38062 -rw-r--r-- 1 kmk users 4 Jul 25 14:28 x
38066 lrwxrwxrwx 1 kmk users 1 Jul 25 14:28 y -> x

% less y
foo


% rm x
% ls -li
38066 lrwxrwxrwx 1 kmk users 1 Jul 25 14:28 y -> x

% less y
y: No such file or directory



------- Hard Link Demo -------
% echo foo > x
% ln x y

% ls -li
38062 -rw-r--r-- 1 kmk users 4 Jul 25 14:28 x
38066 lrwxrwxrwx 1 kmk users 1 Jul 25 14:28 y

% less y
foo

% rm x
% ls -li
38066 lrwxrwxrwx 1 kmk users 1 Jul 25 14:28 y

% less y
foo


Recovering files from backups:

Because rsync doesn't put the backed up files into any kind of archive this is as simple as copying a file. Just find the file you need on the backup server and copy it to where you need it to be. If you are restoring it to another server just use scp to get it there.

Rotating backups

In order to get incremental backups we will use a special feature of the cp command that will copy an entire directory tree by making hard links of each file instead of actually duplicating them. First you count up how many existing backups there are and decide if there are too many. If so delete the oldest . Then you use cp -al to duplicate the most recent backup into a new directory with the current time stamp.

Monthly incremental for Repository using Rsync and Hard links.

#source directory for backup
source=/home/docs/

#target for backup
target=/inc_bak/document/monthly

#bakup directory name assignment
bakdir=$(date +%b_monthly_bak)

#rsync between source and target
rsync -arq --delete $source $target/base

#echo "Rsync Completed Sucessfully."

#If backup dir exist remove.
if [ -d $target/$bakdir ]; then
#echo "file exist so removed."
rm -fr $target/$bakdir ;
fi;

#Create hard link for base.
cp -alfr $target/base/* $target/$bakdir/

#echo "Hard Link Completed Successfully."

Description:

rsync -arq --delete $source $target/base

-a, --archive archive mode
-r, --recursive recurse into directories
--delete delete files that don't exist on sender
-q, --quiet suppress non-error messages

cp -alfr $target/base/* $target/$bakdir/

-a archive mode
-l only copy the link
-f force
-r recursively

Reference :

Home site for Rsync: http://rsync.samba.org/

Article for Rsync with Hard link :

http://www.sanitarium.net/golug/rsync_backups.html

technorati tags:backup, incremental, rsync, hardlink, linux