Head In The Clouds

Provider-agnostic cloud provisioning and Docker orchestration

By Andreas Jansson (@ndreasa)

What is it?

  • Fabric tasks for managing cloud (and non-cloud) servers
    • Create, inspect, delete, SSH, etc.
  • Fabric tasks for doing Docker things
    • Create, inspect, delete, SSH, etc.
  • A simple Docker orchestration tool
  • Pretty shit compared to Mesos, CoreOS, etc.
    • Made for small-ish projects
    • ≤1 maintainer
  • Built for two fairly different use cases, managing servers for both This Is My Jam and for my part-time PhD

Modular, use what you need

# fabfile.py

from headintheclouds.tasks import *
from headintheclouds import ensemble
from headintheclouds import ec2
from headintheclouds import digitalocean
from headintheclouds import docker

Cloud provisioning use cases

Create a server

$ fab create:provider=ec2,name=myserver,size=m1.small
Waiting for instance to start [pending]
Waiting for instance to start [pending]
Waiting for instance to start [pending]
Waiting for instance to start [pending]
Waiting for instance to become accessible
Waiting for instance to become accessible
Waiting for instance to become accessible
Waiting for instance to become accessible

Done.

List your servers

$ fab nodes
[54.205.54.241] Executing task 'nodes'
ec2
name      size      ip             internal_ip    state    created                  
myserver  m1.small  54.205.54.241  10.211.67.146  running  2014-04-29 22:48:36-04:00

digitalocean
name  size  ip  state


Done.

SSH into your servers

$ fab -R myserver ssh
[54.205.54.241] Executing task 'ssh'
[localhost] local: ssh -o StrictHostKeyChecking=no -i "/home/andreas/.ssh/ec2.pem" ubuntu@54.205.54.241 ""
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

[snip]

Last login: Wed Apr 30 02:57:09 2014 from 209-6-206-91.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com
ubuntu@domU-12-31-39-0A-40-64:~$ 

Check current pricing

$ fab pricing
[54.205.54.241] Executing task 'pricing'
ec2
size         memory  cores  storage      gpu  recent  median  stddev  max    cost 
t1.micro      0.615      1  ebsonly           0.003   0.003   0.0     0.003   0.02
m1.small        1.7      1  1 x 160           0.007   0.007   0.0     0.007  0.044
m3.medium      3.75      1  1 x 4 SSD         0.008   0.008   0.0     0.008   0.07
c3.large       3.75      2  2 x 16 SSD        0.016   0.016   0.0     0.016  0.105
m3.large        7.5      2  1 x 32 SSD        0.016   0.016   0.002   0.02    0.14
r3.large       15.0      2  1 x 32 SSD                                       0.175
c3.xlarge       7.5      4  2 x 40 SSD        0.032   0.032   0.0     0.032   0.21
m3.xlarge      15.0      4  2 x 40 SSD        0.032   0.032   0.0     0.032   0.28
r3.xlarge      30.5      4  1 x 80 SSD                                        0.35
c3.2xlarge     15.0      8  2 x 80 SSD        0.064   0.064   0.0     0.065   0.42
m3.2xlarge     30.0      8  2 x 80 SSD        0.066   0.067   0.001   0.069   0.56
g2.2xlarge     15.0      8  60 SSD       gpu                                  0.65
r3.2xlarge     61.0      8  1 x 160 SSD                                        0.7
c3.4xlarge     30.0     16  2 x 160 SSD       4.8     0.195   0.969   4.8     0.84
i2.xlarge      30.5      4  1 x 800 SSD                                      0.853
r3.4xlarge    122.0     16  1 x 320 SSD                                        1.4
c3.8xlarge     60.0     32  2 x 320 SSD       0.279   0.298   0.176   1.0     1.68
i2.2xlarge     61.0      8  2 x 800 SSD                                      1.705
r3.8xlarge    244.0     32  2 x 320 SSD                                        2.8
i2.4xlarge    122.0     16  4 x 800 SSD                                       3.41
hs1.8xlarge   117.0     16  24 x 2048                                          4.6
i2.8xlarge    244.0     32  8 x 800 SSD                                       6.82

digitalocean
size   memory  cores  disk  transfer  cost 
512MB     0.5      1    20       1TB  0.007
  1GB       1      1    30       2TB  0.015
  2GB       2      2    40       3TB   0.03
  4GB       4      2    60       4TB   0.06
  8GB       8      4    80       5TB  0.119
 16GB      16      8   160       6TB  0.238
 32GB      32     12   320       7TB  0.476
 48GB      48     16   480       8TB  0.705
 64GB      64     20   640       9TB  0.941
 96GB      96     24   960      10TB  1.411


Done.

Terminate servers

$ fab -R myserver terminate
[54.205.54.241] Executing task 'terminate'
Sleeping for ten seconds so you can change your mind if you want to!!!
Terminating EC2 instance i-040ec554

Done.

Docker use cases

Run a container

automatically installs Docker if it's not there already

$ fab -R myserver docker.run:image=andreasjansson/graphite,name=graphite,ports=80
[54.198.87.139] Executing task 'docker.run'

[snip - installing docker and pulling image]

[54.198.87.139] out: 3891a87d3bed: Download complete 
[54.198.87.139] out: 0fef1dcfa1fe84db43d9d0f03b4d89b203744f559add27e181bf8775f26b24e3
[54.198.87.139] out: 

[54.198.87.139] sudo: iptables -t nat -A DOCKER -p tcp --dport 80 -j DNAT --to-destination 172.17.0.2:80

Done.

List running containers

$ fab docker.ps
[54.198.87.139] Executing task 'docker.ps'
name      ip          ports                                            created              image                  
graphite  172.17.0.2  80:80, 2003:None, 22:None, 7002:None, 2004:None  2014-04-30 03:18:21  andreasjansson/graphite

Done.
Disconnecting from ubuntu@54.198.87.139... done.

Bind ports

you can bind and unbind while the container is running

$ fab -R myserver docker.bind:graphite,2003
[54.198.87.139] Executing task 'docker.bind'
[54.198.87.139] sudo: iptables -t nat -A DOCKER -p tcp --dport 2003 -j DNAT --to-destination 172.17.0.2:2003

Done.

SSH into a container

only works if you have an SSH server running inside the container, default user/pass is root/root

$ fab -R myserver docker.ssh:graphite
[54.198.87.139] Executing task 'docker.ssh'
[localhost] local: ssh -A -t -o StrictHostKeyChecking=no -i "/home/andreas/.ssh/ec2.pem" ubuntu@54.198.87.139 sshpass -p 'root' ssh -A -t -o StrictHostKeyChecking=no 'root'@172.17.0.2 
Warning: Permanently added '172.17.0.2' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.13.0-24-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@0fef1dcfa1fe:~#

Kill a container

$ fab -R myserver docker.kill:graphite
[54.198.87.139] Executing task 'docker.kill'
[54.198.87.139] sudo: iptables -t nat -D DOCKER -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.17.0.2:80
[54.198.87.139] sudo: iptables -t nat -D DOCKER -p tcp -m tcp --dport 2003 -j DNAT --to-destination 172.17.0.2:2003
[54.198.87.139] sudo: docker kill graphite
[54.198.87.139] out: graphite
[54.198.87.139] out: 

[54.198.87.139] sudo: docker rm graphite
[54.198.87.139] out: graphite
[54.198.87.139] out: 


Done.

Docker orchestration

  • Reads YAML config file
    • Syntax based on Orchard's Fig
    • Designed to be easy to both write and read
  • Your entire infrastructure in a single file
    • If it doesn't fit, you probably shouldn't use HITC...
  • Manages dependencies between servers and containers
  • iptables-based firewalls

Basic example config file

# my-environment.yml

wordpress:
  provider: digitalocean
  size: 512MB
  containers:
    wordpress:
      image: jbfink/docker-wordpress
      ports:
        - 80
  firewall:
    80: "*"

Run it!

$ fab ensemble.up:my-environment
[54.198.87.139] Executing task 'ensemble.up'
Calculating changes...
The following servers will be created:
wordpress
The following containers will be created:
wordpress (wordpress)
The following servers will get new firewalls:
wordpress (1 rules total)
Do you wish to continue? [Y/n] y
>>>>>>>>>>>>>>>>>>>>>>>>> starting <Server: wordpress>
Creating 1 Digital Ocean 512MB droplets
Waiting for droplet to start [pending: 1, running: 0]

[snip - creating server, installing docker, setting up firewall, etc]

Run it again! (idempotent)

$ fab ensemble.up:my-environment
[54.198.87.139] Executing task 'ensemble.up'
Calculating changes.....

Done.

Longer example

Dependencies, templates, etc.

# wordpress.yml

wordpress:
  provider: digitalocean
  size: 512MB
  containers:
    wordpress:
      image: jbfink/docker-wordpress
      ports:
        - 80
    serverstats:
      image: andreasjansson/collectd-write-graphite
      environment:
        HOST_NAME: ${host.name}
        GRAPHITE_HOST: ${graphite.ip}
        # hack to make sure the graphite.containers.graphite container
        # is up before we start streaming data to it
        _DEPENDS: ${graphite.containers.graphite.ip}
  firewall:
    template: default_firewall

graphite:
  provider: digitalocean
  size: 512MB
  containers:
    graphite:
      image: andreasjansson/graphite
      ports:
        - 80
        - 2003
  firewall:
    template: default_firewall

templates:
  default_firewall:
    22: "*"
    80: "*"
    "*/*": $internal_ips

Dependencies

are turned into a DAG, which is traversed twice. Once when figuring out what needs to change to meet the YAML file. And then when actually making those changes.

State

isn't stored anywhere. The state of the cluster is defined by what is actually in the cluster.

Makes it less error prone (but also a little slower, since you have to interrogate that state every time you ensemble.up)

That's about it