Use LVM physical block device instead of loopback file for storing Docker containers

Today I will show you how to setup Docker to use physical LVM volumes for storing container images and metadata. This post is conceptually similar to my previous post about setting up Cinder to use a physical block device instead of a loopback file.

You can find many guides that discuss the performance downsides of mounting files as loopback devices to use as virtual disks (see References below). Docker and Cinder default to using loopback devices for their storage backends out of convenience; developers just want to get started without mucking around with creating LVM partitions and doing sysadmin work.

Fortunately, setting up Docker to use a real (as opposed to virtual) LVM block device is not hard.

First stop the docker daemon and rm -rf /var/lib/docker from your disk.

Second, create an LVM physical volume that will store your Volume Group which will in turn contain your Logical Volumes for Docker data and metadata. For a step-by-step example of creating a new LVM-type partition using gdisk, creating the PV and VG refer to my previous post about setting up Cinder. In the case of Docker, however, you will also need to create 2 Logical Volumes with the lvcreate command.

Assuming your Volume Group is named vg-docker, you can create logical volumes data and metadata as follows:

sudo lvcreate -L xG -n data vg-docker
sudo lvcreate -L yG -n metadata vg-docker

Where:
-L specifies the size of the volume
xG, yG denote the size in Gigabytes for the Logical Volumes
-n specifies the name of the Logical volume
vg-docker is the name of a pre-existing Volume Group

Note that you SHOULD NOT create a file system on the new LV's with mkfs or some other tool. Docker will manage the containers without the overhead of a file system.

Now you need to configure the Docker daemon to use /dev/vg-docker/data and /dev/vg-docker/metadata for storing containers and their metadata.

For Archlinux:

$ sudo cp /usr/lib/systemd/system/docker.service
/etc/systemd/system/

Edit /etc/systemd/system/docker.service so that it contains the following:

ExecStart=/usr/bin/docker daemon -H fd:// --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata

Since the docker.service systemd unit file has changed you must run systemctl daemon-reload to apply the config changes. Then restart the docker daemon with systemctl restart docker


For Fedora 23:

The process is a little bit different for f23; instead of editing a systemd service file, you have to edit /etc/sysconfig/docker-storage and specify the LV to use for container data and metadata (note that on F23 I named the Volume Group vgdocker instead of vg-docker):

DOCKER_STORAGE_OPTIONS= --storage-opt dm.metadatadev=/dev/vgdocker/metadata --storage-opt dm.datadev=/dev/vgdocker/data

Finally restart the docker daemon:

[fedjun@u36jfed23 sysconfig]$ systemctl restart docker
[fedjun@u36jfed23 sysconfig]$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-04-07 13:50:41 KST; 4s ago
     Docs: http://docs.docker.com
 Main PID: 8420 (sh)
   CGroup: /system.slice/docker.service
           ├─8420 /bin/sh -c /usr/bin/docker daemon            $OPTIONS           ...
           ├─8425 /usr/bin/docker daemon --selinux-enabled --log-driver=journald -...
           └─8426 /usr/bin/forward-journald -tag docker

Apr 07 13:50:39 u36jfed23 forward-journal[8426]: Forwarding stdin to journald usi...r
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.071256..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.218922..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.661117..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]:
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.661407..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.661936..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.661978...1
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.662365..."
Apr 07 13:50:41 u36jfed23 systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
[fedjun@u36jfed23 sysconfig]$ sudo docker info
[sudo] password for fedjun:
Containers: 0
Images: 0
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-253:9-131604-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/vgdocker/data
 Metadata file: /dev/vgdocker/metadata
 Data Space Used: 53.74 MB
 Data Space Total: 96.64 GB
 Data Space Available: 96.58 GB
 Metadata Space Used: 1.09 MB
 Metadata Space Total: 10.73 GB
 Metadata Space Available: 10.73 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.109 (2015-09-22)
Execution Driver: native-0.2
Logging Driver: journald
Kernel Version: 4.4.6-300.fc23.x86_64
Operating System: Fedora 23 (Twenty Three)
CPUs: 4
Total Memory: 7.6 GiB
Name: u36jfed23
ID: 45U7:CXDD:LUT3:NBHX:7L6X:VELC:JBZG:XFY5:SQ6L:6LHX:LBAD:PMG5

You can see in docker info that docker is now using a real block device for data and metadata. If docker was still using a loopback device, docker info would contain the lines, Data loop file and Metadata loop file.

References:

http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/

https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

https://docs.docker.com/engine/admin/systemd/

https://www-01.ibm.com/support/knowledgecenter/linuxonibm/liaat/liaatbpstorage.htm

Enabling X11 in Docker containers

Docker containers generally don't come with Xorg installed because of the unnecessary bloat and the fact that most containers run applications that don't need X windows. However, it is possible for docker containers to use the host machine's X11!

Jessica Frazelle from Docker has lots of images on Dockerhub that support running GUI applications from a container. For testing docker + X11, I used the image jess/gparted.

In the container's dockerfile, which is on github, the command for launching the container is as follows:

docker run -v /tmp/.X11-unix:/tmp/.X11-unix \
  --device=/dev/sda:/dev/sda \
  -e DISPLAY=unix$DISPLAY gparted

For some reason the above invocation kept giving me the error:

Unable to find image 'gparted:latest' locally
Pulling repository docker.io/library/gparted
Error: image library/gparted:latest not found

I was able to launch the gparted container with the following commands:

[archjun@pinkS310 ~]$ docker run -ti --rm -e DISPLAY=$DISPLAY --device=/dev/sda:/dev/sda --device=/dev/sdb:/dev/sdb -v /tmp/.X11-unix:/tmp/.X11-unix jess/gparted:latest

Note that I have included the --rm flag so that the container will be deleted on exit instead of hanging around under /var/...

The order of the arguments is also different (which is probably unimportant). You will also notice that I specified two devices (/dev/sda and /dev/sdb) because my laptop has two drives. The crucial difference is that I explicitly specified the docker container name including the tag 'latest'.

For some reason, if I don't specify the tag, docker complains that it can't find the docker image.

Using Pipework to enable communication between docker containers and the host

The docker daemon (I currently have version 1.9.1 installed on Archlinux) creates a bridge interface named docker0 on startup and when containers are spawned, they are all connected to to this interface by default. For security reasons, containers have access to the Internet through NAT, but are not otherwise externally visible. This is problematic if we would like to ping containers from our host or have containers communicate with machines on our network.

There is a great tool called Pipework (https://github.com/jpetazzo/pipework) that incorporates commands for docker, iproute2, and Linux bridge. It's written in Bash shell.

It automates the process of creating network bridges and virtual ethernet ifaces (veth) on the host as well as additional network interfaces within containers (LXC or Docker).

First let's see what docker images I have available from which I can launch some containers:

[archjun@pinkS310 ~]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
jess/chromium       latest              c0aed183c970        4 days ago          567.5 MB
jess/gparted        latest              6d1bee229713        7 days ago          212.4 MB
l3iggs/archlinux    latest              0ac34c50f830        10 days ago         365.8 MB
busybox             latest              c51f86c28340        5 weeks ago         1.109 MB

These were all downloaded from Dockerhub using docker pull repoName. I will launch two container instances from the busybox image:

[archjun@pinkS310 ~]$ docker run -ti --rm busybox /bin/sh

An explanation of the option flags (from man docker run):

-i or --interactive
Keep STDIN open even if not attached (i.e. connected to the container).

-t or --tty
Allocate a pseudo-TTY (pty)

--rm
remove the container with docker rm when you exit the container

/bin/sh
Finally you must give a command to run inside the container. The busybox image does not contain bash, but it does have sh.

Let's take a look at the network ifaces inside the busybox container:

/ # ip a
1: lo: mtu 65536 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
7: eth0@if8: mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link 
       valid_lft forever preferred_lft forever

Only two ifaces exist, loopback and eth0@if8 which is connected to the bridge iface docker0. Note that the IP is in the range 172.17.x, which is in the default docker settings. Through the docker bridge, containers can communicate with each other. But my local machine has an IP in the range 192.168.10.x, so direct communication with docker containers through docker0 is not yet possible.

 I will launch one more busybox container:

[archjun@pinkS310 ~]$ docker run -ti --rm busybox:latest /bin/sh

The second container also has only two network ifaces, one of which is mapped to bridge iface docker0.

/ # ip a
1: lo: mtu 65536 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
9: eth0@if10: mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.3/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:3/64 scope link 
       valid_lft forever preferred_lft forever

From a terminal on my host, let's take a look at the running containers:

[archjun@pinkS310 ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
859be599f53d        busybox:latest      "/bin/sh"           2 hours ago         Up 2 hours                              drunk_perlman
5140cd8079d4        busybox             "/bin/sh"           2 hours ago         Up 2 hours                              stoic_mcnulty

Two busybox containers are running, drunk_perlman and stoic_mcnulty.

The following network ifaces are active on the host:

[archjun@pinkS310 ~]$ ip a show up
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether f8:a9:63:3c:23:64 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.97/24 brd 192.168.10.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::faa9:63ff:fe3c:2364/64 scope link 
       valid_lft forever preferred_lft forever
3: wlp2s0: mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b8:ee:65:d8:fd:f7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.40.195/24 brd 192.168.40.255 scope global wlp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::baee:65ff:fed8:fdf7/64 scope link 
       valid_lft forever preferred_lft forever
4: virbr0: mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 52:54:00:59:95:09 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
6: docker0: mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:3e:81:64:5d brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:3eff:fe81:645d/64 scope link 
       valid_lft forever preferred_lft forever
8: vetha4b3346@if7: mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether 96:ce:70:36:f9:5e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::94ce:70ff:fe36:f95e/64 scope link 
       valid_lft forever preferred_lft forever
10: veth04edbf0@if9: mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether 92:01:7a:5b:0b:06 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::9001:7aff:fe5b:b06/64 scope link 
       valid_lft forever preferred_lft forever

docker0 is the bridge interface created by the docker daemon/systemd service. virbr0 is the bridge iface for use by hypervisors like KVM or Virtualbox. My ethernet interface, enp1s0, has the IP 192.168.10.97/24

Finally, the virtual eth ifaces veth... correspond to the single ports within each of the busybox containers connected to docker0.

Now, using the pipework script run as root, I will create a new bridge interface called br1. Each of the containers will be connected to it through new network ifaces within the two containers.

[archjun@pinkS310 ~]$ sudo pipework br1 drunk_perlman 192.168.10.101/24
[archjun@pinkS310 ~]$ sudo pipework br1 stoic_mcnulty 192.168.10.102/24

Now on the host there are three new ifaces:

[archjun@pinkS310 ~]$ ip a show up
...
11: br1: mtu 1500 qdisc noqueue state UP group default 
    link/ether 6e:7c:fc:f6:04:f5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::10c3:eff:fec7:8d89/64 scope link 
       valid_lft forever preferred_lft forever
13: veth1pl26713@if12: mtu 1500 qdisc fq_codel master br1 state UP group default qlen 1000
    link/ether 6e:7c:fc:f6:04:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::6c7c:fcff:fef6:4f5/64 scope link 
       valid_lft forever preferred_lft forever
15: veth1pl23625@if14: mtu 1500 qdisc fq_codel master br1 state UP group default qlen 1000
    link/ether aa:4f:6d:9c:b9:bc brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::a84f:6dff:fe9c:b9bc/64 scope link 
       valid_lft forever preferred_lft forever

You can see that the two new veth ifaces have br1 as their master.

Inside each of the containers you can see one new interface with one of the IP addresses specified above (in the IP range 192.168.10.x) 

/ # ip a show up

1: lo: mtu 65536 qdisc noqueue 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

7: eth0@if8: mtu 1500 qdisc noqueue 

    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff

    inet 172.17.0.2/16 scope global eth0

       valid_lft forever preferred_lft forever

    inet6 fe80::42:acff:fe11:2/64 scope link 

       valid_lft forever preferred_lft forever

14: eth1@if15: mtu 1500 qdisc fq_codel qlen 1000

    link/ether 66:5c:ae:2b:26:3a brd ff:ff:ff:ff:ff:ff

    inet 192.168.10.102/24 scope global eth1

       valid_lft forever preferred_lft forever

    inet6 fe80::645c:aeff:fe2b:263a/64 scope link 

       valid_lft forever preferred_lft forever

The new iface in container stoic_mcnulty is eth1..., which is connected to bridge br1

Inside container drunk_perlman, you can see a new iface eth1@if13:

/ # ip a show up

1: lo: mtu 65536 qdisc noqueue 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

9: eth0@if10: mtu 1500 qdisc noqueue 

    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff

    inet 172.17.0.3/16 scope global eth0

       valid_lft forever preferred_lft forever

    inet6 fe80::42:acff:fe11:3/64 scope link 

       valid_lft forever preferred_lft forever

12: eth1@if13: mtu 1500 qdisc fq_codel qlen 1000

    link/ether f2:08:19:49:64:4c brd ff:ff:ff:ff:ff:ff

    inet 192.168.10.101/24 scope global eth1

       valid_lft forever preferred_lft forever

    inet6 fe80::f008:19ff:fe49:644c/64 scope link 

       valid_lft forever preferred_lft forever

The new iface was created by pipework and has an IP on the same subnet as localhost.

So far, so good. With just this setup, however, I will be unable to ping the docker containers from my host. Now I must make my Ethernet port enp1s0 into a slave of bridge br1.

[archjun@pinkS310 ~]$ sudo ip l set enp1s0 master br1

[archjun@pinkS310 ~]$ bridge link

2: enp1s0 state UP : mtu 1500 master br1 state forwarding priority 32 cost 19 

5: virbr0-nic state DOWN : mtu 1500 master virbr0 state disabled priority 32 cost 100 

8: vetha4b3346 state UP @(null): mtu 1500 master docker0 state forwarding priority 32 cost 2 

10: veth04edbf0 state UP @(null): mtu 1500 master docker0 state forwarding priority 32 cost 2 

13: veth1pl26713 state UP @(null): mtu 1500 master br1 state forwarding priority 32 cost 2 

15: veth1pl23625 state UP @(null): mtu 1500 master br1 state forwarding priority 32 cost 2

bridge link (show) displays the current port config and flags for linux bridges. I am not sure why docker0 does not show up in the output, as it is also a bridge iface (running bridge link as root makes no difference). You can see that enp1s0 now has master br1

Since br1 is the master for Ethernet port enp1s0, I have to clear the IP address from enp1s0 and assign it to br1 instead:

[archjun@pinkS310 ~]$ sudo ip a flush enp1s0

[archjun@pinkS310 ~]$ sudo ip a add 192.168.10.97/24 dev br1

Now pinging the containers at 192.168.10.101 and ...102 from the host machine works. The host's wired IP is 192.168.10.97 (the address for br1, which is the master iface for enp1s0).

[archjun@pinkS310 ~]$ ping 192.168.10.101

PING 192.168.10.101 (192.168.10.101) 56(84) bytes of data.

64 bytes from 192.168.10.101: icmp_seq=1 ttl=64 time=0.093 ms

64 bytes from 192.168.10.101: icmp_seq=2 ttl=64 time=0.059 ms

64 bytes from 192.168.10.101: icmp_seq=3 ttl=64 time=0.051 ms

^C

--- 192.168.10.101 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1998ms

rtt min/avg/max/mdev = 0.051/0.067/0.093/0.020 ms

[archjun@pinkS310 ~]$ ping 192.168.10.102

PING 192.168.10.102 (192.168.10.102) 56(84) bytes of data.

64 bytes from 192.168.10.102: icmp_seq=1 ttl=64 time=0.107 ms

64 bytes from 192.168.10.102: icmp_seq=2 ttl=64 time=0.047 ms

^C

--- 192.168.10.102 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 999ms

rtt min/avg/max/mdev = 0.047/0.077/0.107/0.030 ms

Pinging the containers at 101 and 102 from my host machine works. Pinging other machines on the local network also works fine:

[archjun@pinkS310 ~]$ ping 192.168.10.58

PING 192.168.10.58 (192.168.10.58) 56(84) bytes of data.

64 bytes from 192.168.10.58: icmp_seq=1 ttl=64 time=0.817 ms

64 bytes from 192.168.10.58: icmp_seq=2 ttl=64 time=0.448 ms

64 bytes from 192.168.10.58: icmp_seq=3 ttl=64 time=0.483 ms

64 bytes from 192.168.10.58: icmp_seq=4 ttl=64 time=0.447 ms

^C

--- 192.168.10.58 ping statistics ---

4 packets transmitted, 4 received, 0% packet loss, time 2999ms

rtt min/avg/max/mdev = 0.447/0.548/0.817/0.158 ms

Now let's see if the containers (I will only show the terminal for one container, since they look identical on the CLI as they don't have unique hostnames) can ping localhost as well as other hosts on the LAN:

/ # ping 192.168.10.97

PING 192.168.10.97 (192.168.10.97): 56 data bytes

64 bytes from 192.168.10.97: seq=0 ttl=64 time=0.104 ms

64 bytes from 192.168.10.97: seq=1 ttl=64 time=0.088 ms

64 bytes from 192.168.10.97: seq=2 ttl=64 time=0.107 ms

^C

--- 192.168.10.97 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.088/0.099/0.107 ms

/ # ping 192.168.10.58

PING 192.168.10.58 (192.168.10.58): 56 data bytes

64 bytes from 192.168.10.58: seq=0 ttl=64 time=0.607 ms

64 bytes from 192.168.10.58: seq=1 ttl=64 time=0.585 ms

64 bytes from 192.168.10.58: seq=2 ttl=64 time=0.543 ms

^C

--- 192.168.10.58 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.543/0.578/0.607 ms

Great! You can see the containers along with other machines on my LAN in zenmap (the GUI for nmap) after a ping scan:

101 and 102 are the busybox docker containers, 97 is the the linux bridge br1 which connects enp1s0 and the containers, and 58 is another host on the LAN.