My Account

Backing Up Hundreds of Sites on WP Engine

Effectively backing up hundreds/thousands of WordPress sites can be tricky. Most solutions like VaultPress become crazy expensive as they charge per site. For example 100 sites X $165/year for VaultPress Basic = $16,500/year. Other options like CodeGuard and Mover, while significantly cheaper then VaultPress, are still multiple thousands of dollars a year. Then there are management tools like InfiniteWP which lack incremental backup capabilities and do not scale very well. So I set out and wrote my own backup script.  This script is optimized to backup hundreds of websites from WP Engine to Dropbox incrementally on a daily basis. Since it only relies on FTP access to the server, it’s quite robust and scalable solution.

Overview of setup

Setting up the server

Install LFTP. LFTP will handle the incremental sync from your websites to the local server. The version of LFTP bundled with Centos 6.7 had some bugs which cause it to crash when running long processes. For that reason, I’ve uninstalled the bundle version and installed from source the latest version of LFTP.

yum remove lftp
mkdir ~/Downloads
cd Downloads
wget http://lftp.yar.ru/ftp/lftp-4.6.4.tar.gz
tar -xvf lftp-4.6.4.tar.gz
cd lftp-4.6.4
yum groupinstall "Development tools"
yum install readline-devel
yum install gnutls-devel
yum install zlib-devel
./configure
make
sudo make install
ln -s /usr/local/bin/lftp /usr/bin/lftp
lftp -v    ### verify it's working

Next, install Rclone. Rclone will handle syncing from the local backup server to Dropbox. Rclone also supports many other cloud options like Google Drive, Amazon S3, Rackspace cloud and more.

mkdir ~/Downloads
cd Downloads
wget http://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip

Create a new Dropbox developer app here: https://www.dropbox.com/developers/apps and select “Create App”. Fill out like so using a unique name.

This will generate an App key and App secret which Rcloud will use. Now configure Rcloud with Dropbox.

~/Downloads/rclone-v1.19-linux-amd64/rclone config
n
Dropbox
2
Fill in your app_key
Fill in your app_secret
Copy and paste the URL from command line to complete the verification.
y
q

The backup script

Create the following folders. Since my VPS is only handling backups, I store everything under my home folder.

mkdir ~/Backup
mkdir ~/Logs
mkdir ~/Scripts
mkdir ~/Tmp

My script is broken up into multiple files. You can grab a starter files here github.com/anchorhost/Anchor-Backup-Script. Modify the following files to fit your setup.

  • anchor_backup_config.sh – This stores email address, paths and the WP Engine site names
  • anchor_backup_logins.sh  – This stores FTP info for each site defined in the main config

To manually run a one time backup:

~/Scripts/backup_to_server.sh && ~/Scripts/backup_to_dropbox.sh

The backup is broken into to main files. The first file backup_to_server.sh will connect to each site and download a copy locally. Since WP Engine automatically outputs it’s database to the file server, there is no steps required to capture the database. The second file backup_to_dropbox.sh will upload the files to Dropbox. If you use a Dropbox for Business account, that means all previous version of the files will be stored within Dropbox’s revisions system.

Scheduling cron job

I scheduled this to run daily at 1am. I like to use nano to modify my crontab.

export VISUAL=nano; crontab -e

Then added the following line and exit and save using control + X then Y.

5 1 * * * /root/Scripts/backup_to_server.sh && /root/Scripts/backup_to_dropbox.sh

Email Notifications

I have the script send two emails each times it runs. One reports on the local copy download. The other reports on the upload to Dropbox.

Conclusion

While there is plenty of improvement that could be scripted, this has been a solid solution for the last few months. Adding new installs is as simply as a few lines of configurations. If my server IP changes, a simple find and replace on the logins file will get things rolling again.