Linode started to offer US$5 VMs and they are available in Tokyo (9ms ping as opposed to 122ms ping to California), so I could not resist to get another one and use it for some experimenting which I simply don’t dare to do on this very blog page (and my wife is using it for work too).
Goal
- 2 web servers serving web server stuff (a simple static index.html serves here)
- 3 virtual web addresses:
- first web server
- second web server
- round-robin web server which serves data from the first and second web server
- Everything with SSL without any warning from web browsers
Ingredients
- 1 VM with a public IP address
- 3 FQDNs pointing all to above IP address
- www1.qw2.org
- www3.qw2.org
- www4.qw2.org
- 2 web servers (www1, www3)
- 1 load-balancer (www4)
- 3 certificates for above FQDNs
Let’s Encrypt Certificates
I use acme.sh for my few certificate needs. This is how to get a new certificate issued:
./acme.sh --issue --dns dns_linode --dnssleep 1200 -d www4.qw2.org
This is using the DNS API from Linode (who hosts my DNS records). See more details here. It creates the required TXT record and removes it later again. I found that 1200 seconds wait time works. 900 does not always. I end up using 10 seconds, suspend the acme.sh command (shell ^Z), and use “dig -t TXT _acme-challenge.www4.qw2.org” until it returns some TXT record. Then continue the suspended acme.sh command.
You should then have a new directory www4.qw2.org in your acme.sh directory with those files:
harald@blue:~/.acme.sh$ ls -la www4.qw2.org/ total 36 drwxr-xr-x 2 harald users 4096 Feb 19 10:16 . drwx------ 17 harald users 4096 Feb 19 00:26 .. -rw-r--r-- 1 harald users 1647 Feb 19 10:16 ca.cer -rw-r--r-- 1 harald users 3436 Feb 19 10:16 fullchain.cer -rw-r--r-- 1 harald users 1789 Feb 19 10:16 www4.qw2.org.cer -rw-r--r-- 1 harald users 517 Feb 19 10:16 www4.qw2.org.conf -rw-r--r-- 1 harald users 936 Feb 19 00:26 www4.qw2.org.csr -rw-r--r-- 1 harald users 175 Feb 19 00:26 www4.qw2.org.csr.conf -rw-r--r-- 1 harald users 1675 Feb 19 00:26 www4.qw2.org.key
You’ll need the fullchain.cer and the private key www4.qw2.org.key later.
Repeat for www1 and www3 too.
Note that the secret key is world readable. the .acme.sh directory is therefore secured with 0700 permissions.
Setting up the Web Servers
Using lighttpd herte. The full directory structure:
harald@lintok1:~$ tree lighttpd lighttpd ├── 33100 │ ├── etc │ │ ├── lighttpd.conf │ │ ├── mime-types.conf │ │ ├── mod_cgi.conf │ │ ├── mod_fastcgi.conf │ │ ├── mod_fastcgi_fpm.conf │ │ └── www1.qw2.org │ │ ├── combined.pem │ │ └── fullchain.cer │ └── htdocs │ └── index.html ├── 33102 │ ├── etc │ │ ├── lighttpd.conf │ │ ├── mime-types.conf │ │ ├── mod_cgi.conf │ │ ├── mod_fastcgi.conf │ │ ├── mod_fastcgi_fpm.conf │ │ └── www3.qw2.org │ │ ├── combined.pem │ │ └── fullchain.cer │ └── htdocs │ └── index.html └── docker-compose.yml
Using the lighttpd.conf is simple and can be done in 5 or 10 minutes. The part for enabling https is this:
$SERVER["socket"] == ":443" {
ssl.engine = "enable"
ssl.pemfile = "/etc/lighttpd/www1.qw2.org/combined.pem"
ssl.ca-file = "/etc/lighttpd/www1.qw2.org/fullchain.cer"
}
fullchain.cer is the one you get from the Let’s Encrypt run. “combined.pem” is created via
cat fullchain.cer www1.qw2.org.key > combined.pem
Here the content of docker-compose.yml:
lighttpd-33100: image: sebp/lighttpd volumes: - /home/harald/lighttpd/33100/htdocs:/var/www/localhost/htdocs - /home/harald/lighttpd/33100/etc:/etc/lighttpd ports: - 33100:80 - 33101:443 restart: always lighttpd-33102: image: sebp/lighttpd volumes: - /home/harald/lighttpd/33102/htdocs:/var/www/localhost/htdocs - /home/harald/lighttpd/33102/etc:/etc/lighttpd ports: - 33102:80 - 33103:443 restart: always
To start those 2 web servers, use docker-compose:
docker-compose up
If you want to have a reboot automatically restart the service, then use do “docker-compose start” afterwards which installs a service.
To test, access: http://www1.qw2.org:33100, https://www1.qw2.org:33101, http://www3.qw2.org:33102, https://www3.qw2.org:33103
They all should work, and the https pages should find a proper security status (valid certificate, no name mismatch etc.).
Adding HAProxy
HAProxy (1.7.2 as of the time of writing) can be the SSL termination and forwarded traffic between the web server and HAProxy is unencrypted (resp. can be encrypted via another method), or HAProxy can simply forward traffic. Which one is preferred depends on the application. In my case it makes most sense to let HAProxy handle SSL.
First the full directory structure:
haproxy ├── docker-compose.yml └── etc ├── errors │ ├── 400.http │ ├── 403.http │ ├── 408.http │ ├── 500.http │ ├── 502.http │ ├── 503.http │ ├── 504.http │ └── README ├── haproxy.cfg └── ssl └── private ├── www1.qw2.org.pem ├── www3.qw2.org.pem └── www4.qw2.org.pem
The www{1,3}.qw2.org.pem were copied from the lighttpd files.
haproxy.cfg:
harald@lintok1:~/haproxy/etc$ cat haproxy.cfg global user nobody group users #daemon # Admin socket stats socket /var/run/haproxy.sock mode 600 level admin stats timeout 2m # Default SSL material locations #ca-base /usr/local/etc/haproxy/ssl/certs #crt-base /usr/local/etc/haproxy/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). tune.ssl.default-dh-param 2048 ssl-default-bind-options no-sslv3 no-tls-tickets ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA -AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256 :kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES1 28-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA :DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256 -SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:A ES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CB C3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl-default-server-options no-sslv3 no-tls-tickets ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-R SA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA2 56:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AE S128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-S HA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES2 56-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA :AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES- CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA defaults log global mode http option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /usr/local/etc/haproxy/errors/400.http errorfile 403 /usr/local/etc/haproxy/errors/403.http errorfile 408 /usr/local/etc/haproxy/errors/408.http errorfile 500 /usr/local/etc/haproxy/errors/500.http errorfile 502 /usr/local/etc/haproxy/errors/502.http errorfile 503 /usr/local/etc/haproxy/errors/503.http errorfile 504 /usr/local/etc/haproxy/errors/504.http stats enable stats uri /stats stats realm Haproxy\ Statistics stats auth admin:SOME_PASSWORD frontend http-in bind *:80 acl is_www1 hdr_end(host) -i www1.qw2.org acl is_www3 hdr_end(host) -i www3.qw2.org acl is_www4 hdr_end(host) -i www4.qw2.org use_backend www1 if is_www1 use_backend www3 if is_www3 use_backend www4 if is_www4 frontend https-in bind *:443 ssl crt /usr/local/etc/haproxy/ssl/private/ reqadd X-Forward-Proto:\ https acl is_www1 hdr_end(host) -i www1.qw2.org acl is_www3 hdr_end(host) -i www3.qw2.org acl is_www4 hdr_end(host) -i www4.qw2.org use_backend www1 if is_www1 use_backend www3 if is_www3 use_backend www4 if is_www4 backend www1 balance roundrobin option httpclose option forwardfor server s1 www1.qw2.org:33100 maxconn 32 backend www3 balance roundrobin option httpclose option forwardfor server s3 www3.qw2.org:33102 maxconn 32 backend www4 balance roundrobin option httpclose option forwardfor server s4-1 www1.qw2.org:33100 maxconn 32 server s4-3 www3.qw2.org:33102 maxconn 32 listen admin bind *:1936 stats enable stats admin if TRUE
Replace “SOME_PASSWORD” with an admin password for the admin user who can stop/start backends via the Web UI.
Here the docker-compose.yml file to start HAProxy:
harald@lintok1:~/haproxy$ cat docker-compose.yml haproxy: image: haproxy:1.7 volumes: - /home/harald/haproxy/etc:/usr/local/etc/haproxy ports: - 80:80 - 443:443 - 1936:1936 restart: always
To start haproxy, do:
docker-compose up
The Result
Now http://www1.qw2.org as well as https://www1.qw2.org works. No need for specific ports like 33100 or 33101 anymore. Same for www3.qw2.org. www4.qw2.org is a round-robin of www1 and www3, but it’s using the www4 certificate when using https. In all cases HAProxy terminates the SSL connections and it’s presenting the correct certificates.
Related: on http://www4.qw2.org:1936/haproxy?stats you can see the statistics of HAProxy.
Connecting it all all
Running 2 web servers plus the load-balancer with all of them internally connected and only the load-balancer visible on port 80 resp. 443 needs a new docker-compose.yml (changed to version 3 syntax) and a slight matching change haproxy.conf file:
harald@lintok1:~/three$ cat docker-compose.yml version: '3' services: lighttpd-33100: image: sebp/lighttpd volumes: - /home/harald/lighttpd/33100/htdocs:/var/www/localhost/htdocs - /home/harald/lighttpd/33100/etc:/etc/lighttpd expose: - 80 restart: always lighttpd-33102: image: sebp/lighttpd volumes: - /home/harald/lighttpd/33102/htdocs:/var/www/localhost/htdocs - /home/harald/lighttpd/33102/etc:/etc/lighttpd expose: - 80 restart: always haproxy: image: haproxy:1.7 volumes: - /home/harald/three/haproxy/etc:/usr/local/etc/haproxy ports: - 80:80 - 443:443 - 1936:1936 restart: always
No need for lighttpd to handle SSL anymore (no more port 443 needed to be exposed at all). Only the HAProxy is visible from outside. Small changes are needed on haproxy.conf, but only in the backend section:
[...] backend www1 balance roundrobin option httpclose option forwardfor server s1 lighttpd-33100:80 maxconn 32 backend www3 balance roundrobin option httpclose option forwardfor server s3 lighttpd-33102:80 maxconn 32 backend www4 balance roundrobin option httpclose option forwardfor server s4-1 lighttpd-33100:80 maxconn 32 server s4-3 lighttpd-33102:80 maxconn 32 [...]
And with “docker ps” we can see what’s happening under the hood of docker-compose:
harald@lintok1:~/three$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 742a2e5388f2 sebp/lighttpd "lighttpd -D -f /e..." 4 minutes ago Up 3 minutes 80/tcp three_lighttpd-33100_1 9d4c61e6c162 sebp/lighttpd "lighttpd -D -f /e..." 4 minutes ago Up 3 minutes 80/tcp three_lighttpd-33102_1 2e41dfa26ac9 haproxy:1.7 "/docker-entrypoin..." 4 minutes ago Up 3 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:1936->1936/tcp three_haproxy_1