Howto create a Backup Cloud Product with Puppet and BackupPC for openQRM Cloud

This HowTo is about how to create a Cloud Product for an Automatic-Backup solution on the Cloud systems (VMs and Bare-Metal) with Puppet and BackupPC on an openQRM Cloud environment.

Requirements

  • One (or more) physical Server
  • at least 1 GB of Memory
  • at least 100 GB of Diskspace
  • VT (Virtualization Technology) enabled in the Systems BIOS so that the openQRM Server can run Virtual Machines later

Install openQRM on Debian

Install a minimal Debian on a physical Server. Then install and initialize openQRM

Please notice!
A detailed Howto about the above initial starting point is available at Install openQRM on Debian, Virtualization with KVM and openQRM on Debian and Cloud Computing with openQRM on Debian

For this howto we assume you have successfully made it through Cloud Computing with openQRM on Debian.

Goal

The Goal for this Howto is to create a "Backup" Cloud Product which the following properties:

  • It needs to be fully automated without regular manual tasks for the system administrator(s)
  • It needs to "selectable" as a regular Cloud Product in the openQRM Cloud Portal and in openQRM Enterprise Cloud Zones.
  • It needs to provide an Cloud user UI to schedule the backup and to restore data (probably also in another location/system)

Characteristic of BackupPC

BackupPC is a open-source backup solution licensed under the GPL which supports backup and restore for Unix and Windows based systmems. It provides several different transport mechanisms (rsync over ssh, rsyncd, tar, SMB etc.) and support deduplication to reduce the overall storage consumption. BackupPC is written in Perl and includes user-friendly configuration files plus a web based UI.

Please check http://backuppc.sourceforge.net/info.html for more informations about BackupPC.

Setup

Since the "Backup" Cloud Product should be "purchasable" by the openQRM Cloud customer as any other Cloud Application Product it must be defined in the openQRM Cloud Product configuration. The best options to integrate the automated Nagios configuration is with either Puppet or Ansible. In this Howto we are going to use Puppet.

The first step of this HowTo describes the installation of a seperated BackupPC server. The further steps of this Howto assume the BackupPC server is running on a seperated dedicated system. Only the UI of the BackupPC server is displayed on the openQRM server.

Please notice!
In the following configuration commands and Puppet classes please replace

BACKUPPC_HOSTNAME with the real hostname of your BackupPC server.

OPENQRM_SERVER_HOSTNAME with your openQRM Servers hostname

OPENQRM_DOMAIN_NAME with your actual configured Domainname in the DNS-Plugin configuration

SSH_SYSTEM_USER with your actual unix system account name

SECRET_DB_PASSWORD with your password for the openQRM Database.

Installation of the dedicated BackupPC server

- Please make sure the following Perl dependencies are installed: Compress::Zlib, Archive::Zip, XML::RSS and File::RsyncP. Those dependencies can be checked by e.g.

perldoc Compress::Zlib

To install those dependencies please run e.g.

cpan install Compress::Zlib

Create a system user and group "backuppc"

Download BackupPC and unpack it in a temporary directory and run the installer by:

perl configure.pl

The following configuration section is self-explaining.

Edit the

/etc/BackupPC/config.pl

as following:

$Conf{EMailFromUserName} = 'ADMINISTRATOR_EMAIL@OPENQRM_DOMAIN_NAME';
$Conf{EMailAdminUserName} = 'ADMINISTRATOR_EMAIL@OPENQRM_DOMAIN_NAME';
$Conf{Language} = 'de';
$Conf{BackupFilesOnly} = \['/app', '/var/app'\];
$Conf{XferMethod} = 'rsync';
$Conf{RsyncShareName} = \['/app', '/var/app'\];

# One task is the Hostkey which needs to be accepted by the new system.
# Another task is that ip are going to be recycled. There we added the following 2 configuration lines

$Conf{RsyncClientCmd} = '$sshPath -q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -x -l root $host $rsyncPath $argList+';
$Conf{RsyncClientRestoreCmd} = '$sshPath -q -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -x -l root $host $rsyncPath $argList+';

Create and enable the BackupPC init script located in the temporary installation directory

Create a rsa SSH Key. This is for the automatic authentication of the clients for the "rsync over ssh" transport mechanism.

ssh-keygen -t rsa

The

/home/SSH_SYSTEM_USER/.ssh/id_rsa.pub

file is later distributed to all Cloud servers via Puppet.

Creating the Puppet recipe for the Backup product

Currently the standard resource for BackupPC are not included in the Puppet version on the mainstream linux distribution for now. Therefore we are using a mechanism which just reports that a Cloud user bought the backup product. A cron job on the central openQRM server then automatically configures BackupPC.

Please notice!
The requirement for that mechanism is to have "stored configuration" enabled and enabled in Puppet. How to archive this is explained in detail in the previous howto about "Howto create a Monitoring Cloud Product with Puppet and Nagios for openQRM Cloud 5.1"

Here the Puppet class "g_backup":

class c_backup {
  ssh_authorized_key {'backuppc@BACKUPPC_HOSTNAME':
  ensure => 'present',
  user => 'root',
  key => '[content-of-the-previously-created-.ssh/id_rsa.pub-key-file]'
  type => 'ssh-rsa',
}
@@file {"/app/openqrm/backup/clients/$hostname":
  content => "$fqdn $ipaddress",
  tag => 'backup',
  }
}

To include this puppet class into openQRM create another recipe in the "groups" directory of the openQRM Puppet configuration with the following content:

class g_backup {
  include c_backup
}

After adding this class to Puppet the "g_backup" application can be added as Cloud product in openQRM.

Here the Puppet class for the openQRM server itself. It cleans-up a local directory for the client data nd runs the

gen_backuppc_hosts_config

shell script which then generates the BackupPC configuration from the Puppet DB informations of the Cloud system.

class backup {
  exec { 'housekeeping':
    command => 'rm -rf /app/openqrm/backup/clients/*',
    cwd => '/app/openqrm/backup/clients',
    path => '/sbin:/usr/sbin:/usr/bin:/bin',
  }
  File <<| tag == "backup" |>> {
  require => Exec ['housekeeping'],
  }
  # get the contact and email for the cloud system from the openQRM DB
  exec { 'gen_backuppc_hosts_config':
    command => '/app/openqrm/tools/gen_backuppc_hosts_config',
    cwd => '/app/openqrm/tools',
    path => '/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/global/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
  }
}
node 'OPENQRM_SERVER_HOSTNAME.OPENQRM_DOMAIN_NAME' {
  include backup
}


The following script

gen_backuppc_hosts_config

gets the owner of the Cloud system from the openQRM database and generates the BackupPC hosts.pl configuration. The resulting generated configuration is then copied locally (for the UI) and also remotely to the BackupPC server.


#!/bin/bash
#
# gen_backuppc_hosts_config
#
# generates the BackupPC hosts config file
#
# host  dhcp  user  moreUsers  #
# SSH_SYSTEM_USER.de 980 5957
#
BACKUP_CLIENTS="/app/openqrm/backup/clients"
MYSQL="/usr/bin/mysql -uroot -pSECRET_DB_PASSWORD openqrm -s -N"
TMP_DIR="/tmp/$$"
[ -d ${TMP_DIR} ] && rm -rf ${TMP_DIR}
mkdir ${TMP_DIR}
TMP_FILE="/${TMP_DIR}/gen_backuppc_hosts_config.$$"
# Header ausgeben
echo "#
# The first non-comment non-empty line gives the field names and should
# not be edited!!
#
host  dhcp  user  moreUsers  # <--- do not edit this line
#farside  0  craig  jill,jeff  # <--- example static IP host entry
#larson  1  bill  # <--- example DHCP host entry
OPENQRM_SERVER_HOSTNAME.OPENQRM_DOMAIN_NAME 0  SSH_SYSTEM_USER" >${TMP_FILE}

for DATEI in ${BACKUP_CLIENTS}/*
do
  HOSTNAME="`basename ${DATEI}`"
  USERNAME="`echo "select u.cu_name from cloud_requests r INNER JOIN cloud_users u ON r.cr_cu_id = u.cu_id where cr_status='3' AND cr_appliance_hostname = '${HOSTNAME}';"|${MYSQL}`"
  EMAIL="`echo "select u.cu_email from cloud_requests r INNER JOIN cloud_users u ON r.cr_cu_id = u.cu_id where cr_status='3' AND cr_appliance_hostname = '${HOSTNAME}';"|${MYSQL}`"
  IP="`cat ${DATEI}|head -1 | awk '{ print $2 }'`"
  FQDN="`cat ${DATEI}|head -1 | awk '{ print $1 }'`"
  if [ "X${USERNAME}" == "X" ]
  then
    rm ${DATEI}
    continue
  fi
  if [ "X${EMAIL}" == "X" ]
  then
    rm ${DATEI}
    continue
  fi
  if [ "X${IP}" == "X" ]
  then
    rm ${DATEI}
    continue
  fi
  echo "${FQDN} 0 ${USERNAME}" >> ${TMP_FILE}
done
scp ${TMP_FILE} BACKUPPC_HOSTNAME:/etc/BackupPC/hosts
ssh BACKUPPC_HOSTNAME "/etc/init.d/backuppc reload"
rm ${TMP_FILE}
[ -d ${TMP_DIR} ] && rm -rf ${TMP_DIR}

The only thing left to do is to define the "g_backup" application as Backup Cloud product in openQRM.

Conclusion

Another Howto showing how easy it is to add "your own ideas" to openQRM!

Links

openQRM Community - http://www.openqrm.com

openQRM Project at sourceforge - http://sourceforge.net/projects/openqrm

openQRM Enterprise - http://www.openqrm-enterprise.com

openQRM at Twitter - https://twitter.com/openQRM

openQRM at Facebook - https://www.facebook.com/openQRM

Puppet - http://puppetlabs.com

BackupPC - http://backuppc.sourceforge.net

Another openQRM Howto sponsored by openQRM Enterprise