Add option to set exact backup times
Change-Id: Ic43b0f4c77c5d25551ea32e0cac3d9958beb2b8d
diff --git a/README.rst b/README.rst
index 1829485..a7e0b72 100644
--- a/README.rst
+++ b/README.rst
@@ -73,6 +73,46 @@
default location, same on both client and server side.
+Backup client at exact times:
+
+..code-block:: yaml
+
+ xtrabackup:
+ client:
+ enabled: true
+ full_backups_to_keep: 3
+ incr_before_full: 3
+ backup_dir: /var/backups/mysql/xtrabackup
+ backup_times:
+ dayOfWeek: 0
+ hour: 4
+ minute: 52
+ compression: true
+ compression_threads: 2
+ database:
+ user: user
+ password: password
+ target:
+ host: host01
+
+ .. note:: Parameters in ``backup_times`` section can be used to set up exact
+ time the cron job should be executed. In this example, the backup job
+ would be executed every Sunday at 4:52 AM. If any of the individual
+ ``backup_times`` parameters is not defined, the defalut ``*`` value will be
+ used. For example, if minute parameter is ``*``, it will run the backup every minute,
+ which is ususally not desired.
+ Available parameters are ``dayOfWeek``, ``dayOfMonth``, ``month``, ``hour`` and ``minute``.
+ Please see the crontab reference for further info on how to set these parameters.
+
+ .. note:: Please be aware that only ``backup_times`` section OR
+ ``hours_before_full(incr)`` can be defined. If both are defined, The
+ ``backup_times`` section will be peferred.
+
+ .. note:: New parameter ``incr_before_full`` needs to be defined. This
+ number sets number of incremental backups to be run, before a full backup
+ is performed.
+
+
Backup server rsync and non-default backup directory:
.. code-block:: yaml
@@ -102,6 +142,42 @@
server:
restrict_clients: false
+
+Backup server at exact times:
+
+..code-block:: yaml
+
+ xtrabackup:
+ server:
+ enabled: true
+ full_backups_to_keep: 3
+ incr_before_full: 3
+ backup_dir: /srv/backup
+ backup_times:
+ dayOfWeek: 0
+ hour: 4
+ minute: 52
+ key:
+ xtrabackup_pub_key:
+ enabled: true
+ key: key
+
+ .. note:: Parameters in ``backup_times`` section can be used to set up exact
+ time the cron job should be executed. In this example, the backup job
+ would be executed every Sunday at 4:52 AM. If any of the individual
+ ``backup_times`` parameters is not defined, the defalut ``*`` value will be
+ used. For example, if minute parameter is ``*``, it will run the backup every minute,
+ which is ususally not desired.
+ Please see the crontab reference for further info on how to set these parameters.
+
+ .. note:: Please be aware that only ``backup_times`` section OR
+ ``hours_before_full(incr)`` can be defined. If both are defined, The
+ ``backup_times`` section will be peferred.
+
+ .. note:: New parameter ``incr_before_full`` needs to be defined. This
+ number sets number of incremental backups to be run, before a full backup
+ is performed.
+
Client restore from local backups:
.. code-block:: yaml
diff --git a/tests/pillar/client_backup_times.sls b/tests/pillar/client_backup_times.sls
new file mode 100644
index 0000000..2e83129
--- /dev/null
+++ b/tests/pillar/client_backup_times.sls
@@ -0,0 +1,22 @@
+xtrabackup:
+ client:
+ enabled: true
+ full_backups_to_keep: 3
+ incr_before_full: 3
+ backup_dir: /var/backups/mysql/xtrabackup
+ backup_times:
+ dayOfWeek: 0
+# month: *
+# dayOfMonth: *
+ hour: 4
+ minute: 52
+ compression: true
+ compression_threads: 2
+ database:
+ user: user
+ password: password
+ target:
+ host: host01
+ qpress:
+ source: tar
+ name: url
\ No newline at end of file
diff --git a/tests/pillar/server_backup_times.sls b/tests/pillar/server_backup_times.sls
new file mode 100644
index 0000000..44a141b
--- /dev/null
+++ b/tests/pillar/server_backup_times.sls
@@ -0,0 +1,16 @@
+xtrabackup:
+ server:
+ enabled: true
+ full_backups_to_keep: 3
+ incr_before_full: 3
+ backup_dir: /srv/backup
+ backup_times:
+ dayOfWeek: 0
+# month: *
+# dayOfMonth: *
+ hour: 4
+ minute: 52
+ key:
+ xtrabackup_pub_key:
+ enabled: true
+ key: key
\ No newline at end of file
diff --git a/xtrabackup/client.sls b/xtrabackup/client.sls
index 8a246b7..fff7cc8 100644
--- a/xtrabackup/client.sls
+++ b/xtrabackup/client.sls
@@ -35,8 +35,23 @@
{%- if not client.cron %}
- commented: True
{%- endif %}
- - minute: 0
-{%- if client.hours_before_incr is defined %}
+{%- if client.backup_times is defined %}
+{%- if client.backup_times.dayOfWeek is defined %}
+ - dayweek: {{ client.backup_times.dayOfWeek }}
+{%- endif -%}
+{%- if client.backup_times.month is defined %}
+ - month: {{ client.backup_times.month }}
+{%- endif %}
+{%- if client.backup_times.dayOfMonth is defined %}
+ - daymonth: {{ client.backup_times.dayOfMonth }}
+{%- endif %}
+{%- if client.backup_times.hour is defined %}
+ - hour: {{ client.backup_times.hour }}
+{%- endif %}
+{%- if client.backup_times.minute is defined %}
+ - minute: {{ client.backup_times.minute }}
+{%- endif %}
+{%- elif client.hours_before_incr is defined %}
{%- if client.hours_before_incr <= 23 and client.hours_before_incr > 1 %}
- hour: '*/{{ client.hours_before_incr }}'
{%- elif not client.hours_before_incr <= 1 %}
diff --git a/xtrabackup/files/innobackupex-client-runner.sh b/xtrabackup/files/innobackupex-client-runner.sh
index cafa81e..868e4bb 100644
--- a/xtrabackup/files/innobackupex-client-runner.sh
+++ b/xtrabackup/files/innobackupex-client-runner.sh
@@ -1,13 +1,29 @@
{%- from "xtrabackup/map.jinja" import client with context %}
{%- from "xtrabackup/map.jinja" import server with context %}
#!/bin/sh
-#
+#
# Script to create full and incremental backups (for all databases on server) using innobackupex from Percona.
# http://www.percona.com/doc/percona-xtrabackup/innobackupex/innobackupex_script.html
#
# Every time it runs will generate an incremental backup except for the first time (full backup).
# FULLBACKUPLIFE variable will define your full backups schedule.
+SKIPCLEANUP=false
+while getopts ":skip-cleanup" opt; do
+ case $opt in
+ skip-cleanup)
+ echo "Cleanup will be skipped" >&2
+ SKIPCLEANUP=true
+ ;;
+ force-full)
+ echo "Full backup will be force triggered"
+ FORCEFULL=true
+ ;;
+ \?)
+ echo "Invalid option: -$OPTARG" >&2
+ ;;
+ esac
+done
USEROPTIONS="--user={{ client.database.user }} --password={{ client.database.password }} --socket=/var/run/mysqld/mysqld.sock"
#TMPFILE="/var/log/backups/innobackupex-runner.$$.tmp"
LOGDIR=/var/log/backups
@@ -19,9 +35,14 @@
SERVERBACKUPDIR={{ server.backup_dir }} # Server side backups base directory
FULLBACKUPDIR=$BACKUPDIR/full # Full backups directory
INCRBACKUPDIR=$BACKUPDIR/incr # Incremental backups directory
+KEEP={{ client.full_backups_to_keep }} # Number of full backups (and its incrementals) to keep
+{%- if client.backup_times is defined %}
+INCRBEFOREFULL={{ client.incr_before_full }}
+{%- else %}
HOURSFULLBACKUPLIFE={{ client.hours_before_full }} # Lifetime of the latest full backup in hours
FULLBACKUPLIFE=$(( $HOURSFULLBACKUPLIFE * 60 * 60 ))
-KEEP={{ client.full_backups_to_keep }} # Number of full backups (and its incrementals) to keep
+{%- endif %}
+
rsyncLog=/var/log/backups/innobackupex-rsync.log
{%- if client.compression is defined %}
@@ -91,6 +112,7 @@
compression_threads=
fi
+{%- if client.backup_times is not defined %}
# Run an incremental backup if latest full is still valid. Otherwise, run a new full one.
if [ "$LATEST_FULL" -a `expr $LATEST_FULL_CREATED_AT + $FULLBACKUPLIFE + 5` -ge $STARTED_AT ] ; then
# Create incremental backups dir if not exists.
@@ -113,6 +135,38 @@
echo "Running new full backup."
innobackupex --defaults-file=$MYCNF $USEROPTIONS $compress $compression_threads $FULLBACKUPDIR > $TMPFILE 2>&1
fi
+{%- else %}
+# Get number of full and incremental backups
+NUMBER_OF_FULL=`find $FULLBACKUPDIR -maxdepth 1 -mindepth 1 -type d -print| wc -l`
+NUMBER_OF_INCR=`find $INCRBACKUPDIR -maxdepth 2 -mindepth 2 -type d -print| wc -l`
+echo "Number of Full backups stored: " $NUMBER_OF_FULL
+echo "Number of Incremental backups stored: " $NUMBER_OF_INCR
+echo "----------------------------"
+#If number of incremental mod number of full backups to keep equals 1, run full backup, otherwise run incremental
+if [ $(( ($NUMBER_OF_INCR + $NUMBER_OF_FULL) % ($INCRBEFOREFULL + 1) )) -eq 0 || FORCEFULL=true ] ; then
+ echo "Running new full backup."
+ innobackupex --defaults-file=$MYCNF $USEROPTIONS $compress $compression_threads $FULLBACKUPDIR > $TMPFILE 2>&1
+else
+ # Create incremental backups dir if not exists.
+ TMPINCRDIR=$INCRBACKUPDIR/$LATEST_FULL
+ mkdir -p $TMPINCRDIR
+
+ # Find latest incremental backup.
+ LATEST_INCR=`find $TMPINCRDIR -mindepth 1 -maxdepth 1 -type d | sort -nr | head -1`
+
+ # If this is the first incremental, use the full as base. Otherwise, use the latest incremental as base.
+ if [ ! $LATEST_INCR ] ; then
+ INCRBASEDIR=$FULLBACKUPDIR/$LATEST_FULL
+ else
+ INCRBASEDIR=$LATEST_INCR
+ fi
+
+ echo "Running new incremental backup using $INCRBASEDIR as base."
+ innobackupex --defaults-file=$MYCNF $USEROPTIONS $compress $compression_threads --incremental $TMPINCRDIR --incremental-basedir $INCRBASEDIR > $TMPFILE 2>&1
+fi
+{%- endif %}
+
+
if [ -z "`tail -1 $TMPFILE | grep 'completed OK!'`" ] ; then
echo "$INNOBACKUPEX failed:"; echo
@@ -147,10 +201,32 @@
# Cleanup
-echo "Cleanup. Keeping only $KEEP full backups and its incrementals."
-AGE=$(($FULLBACKUPLIFE * $KEEP / 60))
-find $FULLBACKUPDIR -maxdepth 1 -type d -mmin +$AGE -execdir echo "removing: "$FULLBACKUPDIR/{} \; -execdir rm -rf $FULLBACKUPDIR/{} \; -execdir echo "removing: "$INCRBACKUPDIR/{} \; -execdir rm -rf $INCRBACKUPDIR/{} \;
+if [ $SKIPCLEANUP=false ] ; then
+ {%- if client.backup_times is not defined %}
+ echo "----------------------------"
+ echo "Cleanup. Keeping only $KEEP full backups and its incrementals."
+ AGE=$(($FULLBACKUPLIFE * $KEEP / 60))
+ find $FULLBACKUPDIR -maxdepth 1 -type d -mmin +$AGE -execdir echo "removing: "$FULLBACKUPDIR/{} \; -execdir rm -rf $FULLBACKUPDIR/{} \; -execdir echo "removing: "$INCRBACKUPDIR/{} \; -execdir rm -rf $INCRBACKUPDIR/{} \;
-echo
-echo "completed: `date`"
-exit 0
+ echo
+ echo "completed: `date`"
+ exit 0
+ {%- else %}
+ echo "----------------------------"
+ echo "Cleanup. Keeping only $KEEP full backups and its incrementals."
+ NUMBER_OF_FULL=$(( `find $FULLBACKUPDIR -maxdepth 1 -type d -print| wc -l` - 1))
+ FULL_TO_DELETE=$(( $NUMBER_OF_FULL - $KEEP ))
+ echo "Found $NUMBER_OF_FULL full backups and $KEEP should be kept. Thus $FULL_TO_DELETE will be deleted"
+ if [ $FULL_TO_DELETE -gt 0 ] ; then
+ cd $INCRBACKUPDIR
+ ls -t $FULLBACKUPDIR | tail -n -$FULL_TO_DELETE | xargs -d '\n' rm -rf
+ cd $FULLBACKUPDIR
+ ls -t | tail -n -$FULL_TO_DELETE | xargs -d '\n' rm -rf
+ else
+ echo "There are less full backups than required, not deleting anything."
+ fi
+ {%- endif %}
+else
+ echo "----------------------------"
+ echo "--skip-cleanup parameter passed. Cleanup was not triggered"
+fi
diff --git a/xtrabackup/files/innobackupex-server-runner.sh b/xtrabackup/files/innobackupex-server-runner.sh
index 3acff6e..3ff538a 100644
--- a/xtrabackup/files/innobackupex-server-runner.sh
+++ b/xtrabackup/files/innobackupex-server-runner.sh
@@ -6,11 +6,42 @@
BACKUPDIR={{ server.backup_dir }} # Backups base directory
FULLBACKUPDIR=$BACKUPDIR/full # Full backups directory
INCRBACKUPDIR=$BACKUPDIR/incr # Incremental backups directory
+KEEP={{ server.full_backups_to_keep }} # Number of full backups (and its incrementals) to keep
+{%- if server.backup_times is defined %}
+INCRBEFOREFULL={{ server.incr_before_full }}
+KEEPINCR=$(( $INCRBEFOREFULL * KEEP ))
+{%- else %}
HOURSFULLBACKUPLIFE={{ server.hours_before_full }} # Lifetime of the latest full backup in hours
FULLBACKUPLIFE=$(( $HOURSFULLBACKUPLIFE * 60 * 60 ))
-KEEP={{ server.full_backups_to_keep }} # Number of full backups (and its incrementals) to keep
+{%- endif %}
# Cleanup
+{%- if server.backup_times is not defined %}
echo "Cleanup. Keeping only $KEEP full backups and its incrementals."
AGE=$(($FULLBACKUPLIFE * $KEEP / 60))
find $FULLBACKUPDIR -maxdepth 1 -type d -mmin +$AGE -execdir echo "removing: "$FULLBACKUPDIR/{} \; -execdir rm -rf $FULLBACKUPDIR/{} \; -execdir echo "removing: "$INCRBACKUPDIR/{} \; -execdir rm -rf $INCRBACKUPDIR/{} \;
+
+echo
+echo "completed: `date`"
+exit 0
+{%- else %}
+echo "Cleanup. Keeping only $KEEP full backups and its incrementals."
+NUMBER_OF_FULL=$(( `find $FULLBACKUPDIR -maxdepth 1 -type d -print| wc -l` - 1))
+NUMBER_OF_INCR=$(( `find $INCRBACKUPDIR -maxdepth 2 -type d -print| wc -l` - 1))
+FULL_TO_DELETE=$(( $NUMBER_OF_FULL - $KEEP ))
+INCR_TO_DELETE=$(( $NUMBER_OF_INCR - $KEEPINCR ))
+echo "Found $NUMBER_OF_FULL full backups and $KEEP should be kept. Thus $FULL_TO_DELETE will be deleted"
+echo "Found $NUMBER_OF_INCR full backups and $KEEPINCR should be kept. Thus $INCR_TO_DELETE will be deleted"
+if [ $FULL_TO_DELETE -gt 0 ] ; then
+cd $FULLBACKUPDIR
+ls -t | tail -n -$FULL_TO_DELETE | xargs -d '\n' rm -rf
+else
+echo "There are less full backups than required, not deleting anything."
+fi
+if [ $INCR_TO_DELETE -gt 0 ] ; then
+cd $INCRBACKUPDIR
+ls -t | tail -n -$INCR_TO_DELETE | xargs -d '\n' rm -rf
+else
+echo "There are less incremental backups than required, not deleting anything."
+fi
+{%- endif %}
\ No newline at end of file