Use LVM physical block device instead of loopback file for storing Docker containers

Today I will show you how to setup Docker to use physical LVM volumes for storing container images and metadata. This post is conceptually similar to my previous post about setting up Cinder to use a physical block device instead of a loopback file.

You can find many guides that discuss the performance downsides of mounting files as loopback devices to use as virtual disks (see References below). Docker and Cinder default to using loopback devices for their storage backends out of convenience; developers just want to get started without mucking around with creating LVM partitions and doing sysadmin work.

Fortunately, setting up Docker to use a real (as opposed to virtual) LVM block device is not hard.

First stop the docker daemon and rm -rf /var/lib/docker from your disk.

Second, create an LVM physical volume that will store your Volume Group which will in turn contain your Logical Volumes for Docker data and metadata. For a step-by-step example of creating a new LVM-type partition using gdisk, creating the PV and VG refer to my previous post about setting up Cinder. In the case of Docker, however, you will also need to create 2 Logical Volumes with the lvcreate command.

Assuming your Volume Group is named vg-docker, you can create logical volumes data and metadata as follows:

sudo lvcreate -L xG -n data vg-docker
sudo lvcreate -L yG -n metadata vg-docker

Where:
-L specifies the size of the volume
xG, yG denote the size in Gigabytes for the Logical Volumes
-n specifies the name of the Logical volume
vg-docker is the name of a pre-existing Volume Group

Note that you SHOULD NOT create a file system on the new LV's with mkfs or some other tool. Docker will manage the containers without the overhead of a file system.

Now you need to configure the Docker daemon to use /dev/vg-docker/data and /dev/vg-docker/metadata for storing containers and their metadata.

For Archlinux:

$ sudo cp /usr/lib/systemd/system/docker.service
/etc/systemd/system/

Edit /etc/systemd/system/docker.service so that it contains the following:

ExecStart=/usr/bin/docker daemon -H fd:// --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata

Since the docker.service systemd unit file has changed you must run systemctl daemon-reload to apply the config changes. Then restart the docker daemon with systemctl restart docker


For Fedora 23:

The process is a little bit different for f23; instead of editing a systemd service file, you have to edit /etc/sysconfig/docker-storage and specify the LV to use for container data and metadata (note that on F23 I named the Volume Group vgdocker instead of vg-docker):

DOCKER_STORAGE_OPTIONS= --storage-opt dm.metadatadev=/dev/vgdocker/metadata --storage-opt dm.datadev=/dev/vgdocker/data

Finally restart the docker daemon:

[fedjun@u36jfed23 sysconfig]$ systemctl restart docker
[fedjun@u36jfed23 sysconfig]$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-04-07 13:50:41 KST; 4s ago
     Docs: http://docs.docker.com
 Main PID: 8420 (sh)
   CGroup: /system.slice/docker.service
           ├─8420 /bin/sh -c /usr/bin/docker daemon            $OPTIONS           ...
           ├─8425 /usr/bin/docker daemon --selinux-enabled --log-driver=journald -...
           └─8426 /usr/bin/forward-journald -tag docker

Apr 07 13:50:39 u36jfed23 forward-journal[8426]: Forwarding stdin to journald usi...r
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.071256..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.218922..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.661117..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]:
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.661407..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.661936..."
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.661978...1
Apr 07 13:50:41 u36jfed23 forward-journal[8426]: time="2016-04-07T13:50:41.662365..."
Apr 07 13:50:41 u36jfed23 systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
[fedjun@u36jfed23 sysconfig]$ sudo docker info
[sudo] password for fedjun:
Containers: 0
Images: 0
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-253:9-131604-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/vgdocker/data
 Metadata file: /dev/vgdocker/metadata
 Data Space Used: 53.74 MB
 Data Space Total: 96.64 GB
 Data Space Available: 96.58 GB
 Metadata Space Used: 1.09 MB
 Metadata Space Total: 10.73 GB
 Metadata Space Available: 10.73 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.109 (2015-09-22)
Execution Driver: native-0.2
Logging Driver: journald
Kernel Version: 4.4.6-300.fc23.x86_64
Operating System: Fedora 23 (Twenty Three)
CPUs: 4
Total Memory: 7.6 GiB
Name: u36jfed23
ID: 45U7:CXDD:LUT3:NBHX:7L6X:VELC:JBZG:XFY5:SQ6L:6LHX:LBAD:PMG5

You can see in docker info that docker is now using a real block device for data and metadata. If docker was still using a loopback device, docker info would contain the lines, Data loop file and Metadata loop file.

References:

http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/

https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

https://docs.docker.com/engine/admin/systemd/

https://www-01.ibm.com/support/knowledgecenter/linuxonibm/liaat/liaatbpstorage.htm