Add option to set exact backup times
Change-Id: I6681daa581978d281e0d60e97da3ecab9e89ac39
diff --git a/README.rst b/README.rst
index 3f60542..1e095b8 100644
--- a/README.rst
+++ b/README.rst
@@ -64,6 +64,46 @@
.. note:: full_backups_to_keep param states how many backup will be stored locally on zookeeper client
+Backup client at exact times:
+
+..code-block:: yaml
+
+ zookeeper:
+ backup:
+ client:
+ enabled: true
+ full_backups_to_keep: 3
+ incr_before_full: 3
+ backup_times:
+ dayOfWeek: 0
+ hour: 4
+ minute: 52
+ compression: true
+ compression_threads: 2
+ database:
+ user: user
+ password: password
+ target:
+ host: host01
+
+ .. note:: Parameters in ``backup_times`` section can be used to set up exact
+ time the cron job should be executed. In this example, the backup job
+ would be executed every Sunday at 4:52 AM. If any of the individual
+ ``backup_times`` parameters is not defined, the defalut ``*`` value will be
+ used. For example, if minute parameter is ``*``, it will run the backup every minute,
+ which is ususally not desired.
+ Available parameters are ``dayOfWeek``, ``dayOfMonth``, ``month``, ``hour`` and ``minute``.
+ Please see the crontab reference for further info on how to set these parameters.
+
+ .. note:: Please be aware that only ``backup_times`` section OR
+ ``hours_before_full(incr)`` can be defined. If both are defined,
+ the ``backup_times`` section will be peferred.
+
+ .. note:: New parameter ``incr_before_full`` needs to be defined. This
+ number sets number of incremental backups to be run, before a full backup
+ is performed.
+
+
Backup server rsync
.. code-block:: yaml
@@ -87,6 +127,44 @@
backup:
restrict_clients: false
+
+Backup server at exact times:
+
+..code-block:: yaml
+
+ zookeeper:
+ backup:
+ server:
+ enabled: true
+ full_backups_to_keep: 3
+ incr_before_full: 3
+ backup_dir: /srv/backup
+ backup_times:
+ dayOfWeek: 0
+ hour: 4
+ minute: 52
+ key:
+ zookeeper_pub_key:
+ enabled: true
+ key: key
+
+ .. note:: Parameters in ``backup_times`` section can be used to set up exact
+ time the cron job should be executed. In this example, the backup job
+ would be executed every Sunday at 4:52 AM. If any of the individual
+ ``backup_times`` parameters is not defined, the defalut ``*`` value will be
+ used. For example, if minute parameter is ``*``, it will run the backup every minute,
+ which is ususally not desired.
+ Available parameters are ``dayOfWeek``, ``dayOfMonth``, ``month``, ``hour`` and ``minute``.
+ Please see the crontab reference for further info on how to set these parameters.
+
+ .. note:: Please be aware that only ``backup_times`` section OR
+ ``hours_before_full(incr)`` can be defined. If both are defined, The
+ ``backup_times`` section will be peferred.
+
+ .. note:: New parameter ``incr_before_full`` needs to be defined. This
+ number sets number of incremental backups to be run, before a full backup
+ is performed.
+
Client restore from local backup:
.. code-block:: yaml
diff --git a/tests/pillar/backup_client_backup_times.sls b/tests/pillar/backup_client_backup_times.sls
new file mode 100644
index 0000000..d9952c6
--- /dev/null
+++ b/tests/pillar/backup_client_backup_times.sls
@@ -0,0 +1,16 @@
+zookeeper:
+ backup:
+ client:
+ enabled: true
+ full_backups_to_keep: 3
+ incr_before_full: 3
+ backup_times:
+ dayOfWeek: 0
+# month: *
+# dayOfMonth: *
+ hour: 4
+ minute: 52
+ target:
+ host: cfg01
+ restore_latest: 1
+ restore_from: local
\ No newline at end of file
diff --git a/tests/pillar/backup_server_backup_times.sls b/tests/pillar/backup_server_backup_times.sls
new file mode 100644
index 0000000..6076503
--- /dev/null
+++ b/tests/pillar/backup_server_backup_times.sls
@@ -0,0 +1,17 @@
+zookeeper:
+ backup:
+ server:
+ enabled: true
+ full_backups_to_keep: 3
+ incr_before_full: 3
+ backup_dir: /srv/backup
+ backup_times:
+ dayOfWeek: 0
+# month: *
+# dayOfMonth: *
+ hour: 4
+ minute: 52
+ key:
+ zookeeper_pub_key:
+ enabled: true
+ key: key
\ No newline at end of file
diff --git a/zookeeper/backup.sls b/zookeeper/backup.sls
index f060dd4..7c64ce7 100644
--- a/zookeeper/backup.sls
+++ b/zookeeper/backup.sls
@@ -31,8 +31,24 @@
{%- if not backup.cron %}
- commented: True
{%- endif %}
+{%- if backup.client.backup_times is defined %}
+{%- if backup.client.backup_times.dayOfWeek is defined %}
+ - dayweek: {{ backup.client.backup_times.dayOfWeek }}
+{%- endif -%}
+{%- if backup.client.backup_times.month is defined %}
+ - month: {{ backup.client.backup_times.month }}
+{%- endif %}
+{%- if backup.client.backup_times.dayOfMonth is defined %}
+ - daymonth: {{ backup.client.backup_times.dayOfMonth }}
+{%- endif %}
+{%- if backup.client.backup_times.hour is defined %}
+ - hour: {{ backup.client.backup_times.hour }}
+{%- endif %}
+{%- if backup.client.backup_times.minute is defined %}
+ - minute: {{ backup.client.backup_times.minute }}
+{%- endif %}
+{%- elif backup.client.hours_before_incr is defined %}
- minute: 0
-{%- if backup.client.hours_before_full is defined %}
{%- if backup.client.hours_before_full <= 23 and backup.client.hours_before_full > 1 %}
- hour: '*/{{ backup.client.hours_before_full }}'
{%- elif not backup.client.hours_before_full <= 1 %}
diff --git a/zookeeper/files/backup/zookeeper-backup-client-runner.sh b/zookeeper/files/backup/zookeeper-backup-client-runner.sh
index d18dc78..24e2b51 100644
--- a/zookeeper/files/backup/zookeeper-backup-client-runner.sh
+++ b/zookeeper/files/backup/zookeeper-backup-client-runner.sh
@@ -2,6 +2,19 @@
#!/bin/bash
# Script to backup zookeeper schema and create snapshot of keyspaces
+SKIPCLEANUP=false
+while getopts "sf" opt; do
+ case $opt in
+ s)
+ echo "Cleanup will be skipped" >&2
+ SKIPCLEANUP=true
+ ;;
+ \?)
+ echo "Invalid option: -$OPTARG" >&2
+ ;;
+ esac
+done
+
# Configuration
# -------------
BACKUPDIR="{{ backup.backup_dir }}/full"
@@ -9,17 +22,20 @@
TMPDIR="$( pwd )/${PROGNAME}.tmp${RANDOM}"
TMPLOG="zookeeper-tmplog.log"
ZOOKEEPERDIR="/var/lib/zookeeper/version-2/"
- KEEP={{ backup.client.full_backups_to_keep }}
- HOURSFULLBACKUPLIFE={{ backup.client.hours_before_full }} # Lifetime of the latest full backup in seconds
- LOGDIR="/var/log/backups"
- RSYNCLOG="/var/log/backups/zookeeper-rsync.log"
-
-
+ {%- if backup.client.backup_times is not defined %}
+ HOURSFULLBACKUPLIFE={{ backup.client.hours_before_full }} # Lifetime of the latest full backup in hours
if [ $HOURSFULLBACKUPLIFE -gt 24 ]; then
FULLBACKUPLIFE=$(( 24 * 60 * 60 ))
else
FULLBACKUPLIFE=$(( $HOURSFULLBACKUPLIFE * 60 * 60 ))
fi
+ {%- endif %}
+ KEEP={{ backup.client.full_backups_to_keep }}
+ LOGDIR="/var/log/backups"
+ RSYNCLOG="/var/log/backups/zookeeper-rsync.log"
+
+
+
# Create backup directory.
# ------------------------
@@ -93,8 +109,26 @@
# Cleanup
# ---------
+if [ $SKIPCLEANUP = false ] ; then
+ {%- if backup.client.backup_times is not defined %}
+ echo "----------------------------"
echo "Cleanup. Keeping only $KEEP full backups"
AGE=$(($FULLBACKUPLIFE * $KEEP / 60))
find $BACKUPDIR -maxdepth 1 -type d -mmin +$AGE -execdir echo "removing: "$BACKUPDIR/{} \; -execdir rm -rf $BACKUPDIR/{} \;
-
+ {%- else %}
+ echo "----------------------------"
+ echo "Cleanup. Keeping only $KEEP full backups"
+ NUMBER_OF_FULL=`find $BACKUPDIR -maxdepth 1 -mindepth 1 -type d -print| wc -l`
+ FULL_TO_DELETE=$(( $NUMBER_OF_FULL - $KEEP ))
+ if [ $FULL_TO_DELETE -gt 0 ] ; then
+ cd $BACKUPDIR
+ ls -t | tail -n -$FULL_TO_DELETE | xargs -d '\n' rm -rf
+ else
+ echo "There are less full backups than required, not deleting anything."
+ fi
+ {%- endif %}
+else
+ echo "----------------------------"
+ echo "-s parameter passed. Cleanup was not triggered"
+fi
# Fin.
diff --git a/zookeeper/files/backup/zookeeper-backup-server-runner.sh b/zookeeper/files/backup/zookeeper-backup-server-runner.sh
index 59e9225..003712a 100644
--- a/zookeeper/files/backup/zookeeper-backup-server-runner.sh
+++ b/zookeeper/files/backup/zookeeper-backup-server-runner.sh
@@ -6,16 +6,31 @@
BACKUPDIR="{{ backup.backup_dir }}/full"
KEEP={{ backup.server.full_backups_to_keep }}
- HOURSFULLBACKUPLIFE={{ backup.server.hours_before_full }} # Lifetime of the latest full backup in seconds
-
+ {%- if backup.server.backup_times is not defined %}
+ HOURSFULLBACKUPLIFE={{ backup.server.hours_before_full }} # Lifetime of the latest full backup in hours
if [ $HOURSFULLBACKUPLIFE -gt 24 ]; then
FULLBACKUPLIFE=$(( 24 * 60 * 60 ))
else
FULLBACKUPLIFE=$(( $HOURSFULLBACKUPLIFE * 60 * 60 ))
fi
+ {%- endif %}
# Cleanup
# ---------
- echo "Cleanup. Keeping only $KEEP full backups"
- AGE=$(($FULLBACKUPLIFE * $KEEP / 60))
- find $BACKUPDIR -maxdepth 1 -type d -mmin +$AGE -execdir echo "removing: "$BACKUPDIR/{} \; -execdir rm -rf $BACKUPDIR/{} \;
+{%- if backup.server.backup_times is not defined %}
+echo "----------------------------"
+echo "Cleanup. Keeping only $KEEP full backups"
+AGE=$(($FULLBACKUPLIFE * $KEEP / 60))
+find $BACKUPDIR -maxdepth 1 -type d -mmin +$AGE -execdir echo "removing: "$BACKUPDIR/{} \; -execdir rm -rf $BACKUPDIR/{} \;
+{%- else %}
+echo "----------------------------"
+echo "Cleanup. Keeping only $KEEP full backups"
+NUMBER_OF_FULL=`find $BACKUPDIR -maxdepth 1 -mindepth 1 -type d -print| wc -l`
+FULL_TO_DELETE=$(( $NUMBER_OF_FULL - $KEEP ))
+if [ $FULL_TO_DELETE -gt 0 ] ; then
+ cd $BACKUPDIR
+ ls -t | tail -n -$FULL_TO_DELETE | xargs -d '\n' rm -rf
+else
+ echo "There are less full backups than required, not deleting anything."
+fi
+{%- endif %}