Harald's Random Stuff

Since moving Docker containers is easy, and Amazon keeps on patching and rebooting my little AWS server, time to put the theory into practice and move to some other place.

Moving was 4 parts:

• Copy the data volumes over
• Change DNS
• Configure the load balancer
• Start everything in the new place

The first one needed some scripting since moving data containers in Docker is not a built-in function. And that needs a corresponding restore function of course. Here the scripts, first backup:

#!/bin/bash
# Does a backup from one or many Docker volumes
# Harald Kubota 2016-07-09

if [[ $# -eq 0 ]] ; then
 echo "Usage: $0 docker_volume_1 [docker_volume_2 ...]"
 echo "will create tar.bz2 files of the docker volumes listed"
 exit 10
fi

today=`date +%Y-%m-%d`

for vol in "$@" ; do

 backup_file=$vol-${today}.tar.bz2
 echo "Backup up docker volume $i into $backup_file"
 docker run --rm -v $vol:/voldata -v $(pwd):/backup debian:8 \
 tar cf /backup/$vol.tar.tmp /voldata
 bzip2 <$vol.tar.tmp >$backup_file && rm -f $vol.tar.tmp
done

and here restore:

#!/bin/bash
# Restore a tar dump back into a Docker data volume
# tar file can be .tar or .tar.bz2
# Name is always VOLUME-NAME-20YY-MM-DD.tar[.bz2]
# Harald Kubota 2016-07-09

if [[ $# -ne 1 ]] ; then
 echo "Usage: $0 tarfile"
 echo "will create a new volume derived from the tarfile name and restore the tarfile data into it"
 exit 10
fi

today=`date +%Y-%m-%d`

for i in "$@" ; do

 volumename=$(echo $i | sed 's/\(.*\)-20[0-9][0-9]-[0-9][0-9]-[0-9][0-9]\.tar.*/\1/')
 docker volume create --name $volumename

 # if .bz2 then decompress first
 length=${#i}
 last4=${i:length-4}
 tar_name=$i
 delete_tar=0

 echo "Restoring tar file $i into volume $volumename"

 if [[ ${last4} == ".bz2" ]] ; then
 tar_name=$(basename $i .bz2)
 bunzip2 <$i >$tar_name
 delete_tar=1
 fi
 #echo "tar_name=$tar_name, delete_tar=$delete_tar"

 docker run --rm -v $volumename:/voldata -v $(pwd):/backup debian:8 \
 tar xfv /backup/$tar_name
 if [[ $delete_tar -eq 1 ]] ; then
 rm $tar_name
 fi
done

With this, moving Docker volumes is a piece of cake, and it doubles as a universal backup mechanism too.

Updating DNS is trivial, as is the Load Balancer part which maps the popular port 80 to the correct Docker instance. Starting is a simple “docker-compose up”.

My WordPress/MySQL Docker instances on AWS are unhappy: after some time, they stop. No idea why. All I see is

Failed Units: 1
 polkit.service

when I log in after WordPress seems to have died. And no Docker process is running. While I can restart them easily, it’s annoying to do so manually. At this point I have 2 possibilities: find the root cause, or work around it via monitoring and automatic restarting when needed.

And a load balancer to make the service available even when AWS does kill my Docker containers would be nice. Actually a load balancer has to do monitoring anyway. Since this topic came up at work too, it’s double useful. So a load balancer it is!

A nice article, although a bit dated, is here. He knows what he’s talking about: he made up elastic binary trees and wrote haproxy and made it saturate a 10G link back in 2009.

nginx is a popular http load balancer (amond other things it can do), but since haproxy can do TCP in general, it’s more universally suitable for me, so that’s what it’ll be.

Step 1 is to have something to load balance. Because this is blog runs on WordPress with its MySQL instance, doubling this is non-trivial: WordPress keeps files in its web directory, and the MySQL instanced would need a multi-master replication setup.

Instead I’ll have a static copy of the web page…and then use a load balancer to switch to the static page when the primary is dead.

PS: To be fair AWS might not be at fault here. Truth is: I don’t know what’s causing the problem. On Google Cloud I had no such issue, but that’s about all I can say. I don’t expect problems in the Docker containers as they are quite literally identical. Even the version of CoreOS is identical.

Anyway, this all sounds like a good excuse to set up some load balancing and monitoring.

I changed slightly the way to start my containers. Now they always contain the backup volume too:

docker run --name blog-mysql -v blog-backup:/backup \
-v blog-mysql-data:/var/lib/mysql -d mysql/mysql-server:5.7

docker run --name blog-app -v blog-backup:/backup \
-e WORDPRESS_DB_PASSWORD=DBPASSWORD -e WORDPRESS_DB_USER=WPDBACCOUNT \
--link blog-mysql:mysql -v wordpress-data:/var/www/html -p 80:80 \
-w /var/www/html/wordpress -d wordpress:latest

The reason is that I can run a backup job inside the container. Important for the DB backup as now I can use msqldump. Before it was: stop mysql, tar up the DB files, start MySQL again.

Making a backup in each container:

tar -C /var/www/html -z -f /backup/blog-wp-`data +%Y-%m-%d.tar.gz .

resp.

mysqldump -pROOTMYSQLASSWORD --all-databases | gzip >/backup/blog-db-`date +%Y-%m-%d`.tar.gz

Now the problem is how to get those files out and to where…

Dropbox/Google Cloud don’t offer access via ftp/scp/etc. Time to look into the various storage offerings from Google/Amazon/Azure.

When you read this, this blog moved do a (currently) Google Cloud VM Instance using Docker.

There’s a lot of examples how to set up containers for this combination of WordPress and MySQL, but the migration from an existing installation was nowhere mentioned. Thus here my notes from the end to the start:

Run this blog in Docker

1. To start I need
   1. a server which can run Docker
   2. Docker 1.9.x as I need the volume command
   3. a MySQL dump of the cold copied DB files: mysql-backup.dump
   4. a tar archive from the wordpress directory: wordpress-backup.tar.gz
   5. a DNS entry to point to the Docker host
   6. Port 80 on the Docker host exposed to a public IP
2. On the Docker host, create 3 empty data volumes which will host the persistent DB data, wordpress directory and backups:

   docker volume create --name blog-mysql-data
   docker volume create --name wordpress-data
   docker volume create --name blog-backup

3. Populate blog-backup with the backup files:

   docker run -it -v blog-backup:/backup -v /home/harald:/data debian:8 bash
   cp /data/mysql-backup.dump /data/wordpress-backup.tar.gz /backup
   exit

4. blog-backup is a volume which contains backups in /backup of the wordpress directory as well as the (cold) mysql DB. Extract like this:

   docker run -it -v blog-backup:/backup \
   -v blog-mysql-data:/var/lib/mysql \
   -v wordpress-data:/var/www/html debian:8 bash
   cd /var/lib
   tar xfv /backup/mysql-backup.dump
   cd /var/www/html
   tar xfv /backup/wordpress-backup.tar.gz
   exit

5. Start MySQL Docker container first

   docker run --name blog-mysql \
   -v blog-mysql-data:/var/lib/mysql \
   -d \
   mysql/mysql-server:5.7

6. Now start WordPress (and replace the passwords and account of course)

   docker run --name blog-app \
   -e WORDPRESS_DB_PASSWORD=MY_PASSWORD \
   -e WORDPRESS_DB_USER=MY_ACCOUNT \
   --link blog-mysql:mysql \
   -v wordpress-data:/var/www/html \
   -p 80:80 \
   -w /var/www/html/wordpress \
   -d \
   wordpress:latest

Google Cloud Configuration

When creating the VM which runs Docker, make sure you get Docker 1.9 or newer as the docker volume command does not exist until then. For now (December 2015) that means to choose the beta CoreOS instance for your Google Cloud VM.

Be able to copy files to the VM.

Beside this, make http and https traffic externally visible and remember the IP assigned to the VM.

DNS

My DNS is on linode.com, so I have to change the blog DNS entry there. TTL is now 5min (instead of 1h default) to make testing a bit faster.

Alternative during testing it’s sufficient to make the machine which runs the browser to point the FQDN to the current IP. The Docker host or container does not care about its external IP.

Initial Polulation of the DB Filesystem

The first population of the MySQL data was a bit tedious and manual due to the unexpected upgrade needed.

1. Have a dump from the SQL DB:

    mysqldump -pMY_ADMINPW --all-databases > mysql-backup.dump

2. Run mysql with the dump being available. Mount the directory which has the dump in /data2 and the (empty) blog-mysql-data under /var/lib/mysql and set an initial DB root password:

   docker run -it \
   -v /home/harald/dockerstuff/mysql-data:/data2 \
   -v blog-mysql-data:/var/lib/mysql \
   -e MYSQL_ROOT_PASSWORD=mypw -P mysql/mysql-server:5.7

3. Since we did not name this container, we have to find its name or container ID:

   CONTAINERID=`docker ps --format "{{.ID}} {{.Image}}" | grep mysql-server | awk '{print $1}'`
   docker exec -it $CONTAINERID bash

4. Inside the mysql container now run the import:

   mysql -u root -pmypw </data2/mysql-backup.dump

5. and do a DB upgrade and stop mysql:

   mysql_upgrade -u root -p mypw
   pkill -SIGTERM mysqld

6. By now the DB root password is change to the one from the dump. And blog-mysql-data now has a working MySQL DB with the last dump we took.

Initial Population of the WordPress Filesystem

I tried initially to use a plan vanilla WordPress Docker as unmodified as possible, but since I needed to add plugins and themes, I tried to find a programmatic way to add them. While thinking about future updates of WordPress and its plugins, it made me realize that a separate data volume for the wordpress directory is in order. The alternative would have been to rewriting /entrypoint.sh in the original WordPress Docker container.

1. Start the WordPress Docker container with no modifications but let it connect to the MySQL container:

   docker volume create --name wordpress-data

   docker run -it --name blog-app \
   -e WORDPRESS_DB_PASSWORD=MY_PASSWORD \
   -e WORDPRESS_DB_USER=MY_ACCOUNT \
   --link blog-mysql:mysql \
   -v wordpress-data:/var/www/html \
   -p 80:80 \
   -w /var/www/html/wordpress wordpress:latest

2. The /entrypoint.sh script will populate /var/www/html/ with a complete WordPress instance. Changing the WORKDIR to /var/www/html/wordpress puts those files to where I need them as that’s where the files are on the old server.
3. Now you can stop the WordPress container. The data files are kept.
4. I had to put a lot of my uploaded images back:

   docker run -it -v /home/harald:/data2 \
   -v wordpress-data:/var/www/html debian:8 bash

5. Inside the Debian container copy the files to wordpress/wp-content/uploads

   cp -pr /data2/wordpress/wp-content/uploads/* /var/www/html/wordpress/wp-content/uploads/

6. MySQL container was running at this time. Starting the WordPress container now again:

   docker run -it --name blog-app \
   -e WORDPRESS_DB_PASSWORD=MY_PASSWORD \
   -e WORDPRESS_DB_USER=MY_ACCOUNT \
   --link blog-mysql:mysql \
   -v wordpress-data:/var/www/html \
   -p 80:80 \
   -w /var/www/html/wordpress wordpress:latest

7. For testing, edit /etc/hosts of the machine with the browser to make the FQDN of the blog point to the IP of the Docker host.
8. Now in a browser I was able to see everything from my blog, log in, update Askismet, install YAPB and install the theme Suffusion.
9. Stop the container, mount the data volume as before, and create a tar dump of /var/www/html/wordpress as wordpress-backup.tar.gz

   docker stop blog-app
   docker run -it -v wordpress-data:/var/www/html \
   -v /home/harald:/data2 debian:8 bash
   cd /var/www/html
   tar cfvz /data2/wordpress-backup.tar.gz wordpress
   exit

At this point wordpress-data is the complete wordpress directory and I have a tar archive of it.

Outstanding Items

Backup

MySQL

MySQL is easy enough manually:

docker exec -it blog-mysql bash

Inside run a mysqldump. Then transfer the dump to a place off-site. Automate by not using bash but instead call the script to make the backup. Or run mysqldump from another server but i think that causes more network traffic. I’d need to expose the mysql ports for the latter though.

WordPress

WordPress directory is equally easy:

docker exec -it blog-app bash

Inside again run a tar and/or rsync to a remote site.

Potential Issues

• MySQL and WordPress currently must run on the same Docker host. To have them on separate hosts a network needs to be created to connect those.
  However I would have been ok to have MySQL and WordPress in one container as I plan not to scale. Right now I use the micro-instance of Google Cloud and I’m fine with this.
• Disk space on the Docker host. It’s limited. 10G Google gives me (resp. I assigned myself).
  volumes use all disk space they can get, so the backup volume WILL increase if I do dayly dumps inside with no expiration. I plan to move them off-size though, so I can delete old backups quickly.
• If the Docker hosts fails/restarts, I have to manually restart my 2 containers.
• CPU/RAM of the f1-micro instance (1 shared CPU, 0.6GB RAM): it’s enough, but memory is used up:

  total       used       free     shared    buffers     cached 
  Mem:        607700     580788      26912      33404      12284     122960 
  -/+ buffers/cache:     445544     162156 
  Swap:            0          0          0

Comments

• Note that the debian:8 image contains neither bzip2 nor xz.