Popular Posts

Docker Volumes Deep Dive

Different --mount Options
This tutorial helps you to under different mount options, storage drivers, volume drivers(plugins) and troubleshooting issues with Storage Driver(Overlay2).
— Thiru

Volumes are the preferred mechanism to persist container data. bind mounts are dependant on host machine directory structure but volumes are completely managed by docker. more on volumes…​

You can use --mount or --volume to create volume , --mount is recommended for newer docker(>17.06).

Different --mount Options

The --mount option allows us to create different kinds of volumes, below is syntax to use --mount option.

docker run --mount
                volume-opt k=v,
                volume-opt k=v,
  • type : This attribute supports three values, if not specified it is volume.

    1. bind

    2. tmpfs

    3. volume (default)

bind mount

  • bind: Bind mounts have limited functionality compared to volumes. When you use a bind mount, a file or directory on the host machine is mounted into a container.

make sure source directory must exists on host machine , before executing bind mounts.
docker run --rm -it --mount type=bind,source="$(pwd)",target=/app nginx bash

# bind mount with readOnly
docker run --rm -it --mount type=bind,source="$(pwd)",target=/app,readOnly nginx bash

tmpfs mount

  • tmpfs: When you create a container with a tmpfs mount, the container can create files outside the container’s writable layer. more…​

# direct tmpfs (standalone containers)
tvajjala$ docker run -it --rm --mount type=tmpfs,destination=/app nginx bash

# using mount we can specify filesystem, size and permission-mode
tvajjala$ docker run --rm -it --mount type=tmpfs,destination=/mycache,tmpfs-size=1.1M,tmpfs-mode=777 nginx bash
root@fa83e775ce51:/# df -kh /mycache/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.2M     0  1.2M   0% /mycache

volume mount

  • volume: Volumes are the preferred mechanism for persisting data generated by and used by Docker containers.

if source volume doesn’t exist while running container it will create it for you.

# using --mount option (recommended)
tvajjala-mac:~ tvajjala$ docker run --rm -it --mount source=thiruVol,destination=/thiruDir nginx bash
root@297e72a94113:/# df -kh thiruDir/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        59G  2.7G   53G   5% /thiruDir

# using -v/--volume option with `ro` option
docker run --rm -it -v myvolume1:/usr/share/nginx/html:ro nginx bash
you can create named volume before execute above command.
docker volume create mySecretVolume

Running docker container same user as host

Below command demonstrates running docker container same user as your host and mount volume(bind type) to see host files inside container.

docker run --rm -it -v $(pwd):/ws --user `id -u`:`id -g` busybox sh

docker run --rm -it -v $(pwd):/ws --user `id -u tvajjala`:`id -g tvajjala` busybox sh

Bind Propagation

for given bind-mount /named volume can be propagated to replicas of that mount. for a mount point /mnt on /tmp directory, the propagation settings controls /tmp/a would be available on /mnt/a.

tvajjala-mac:bindPropgation tvajjala$ docker run --rm -it --mount type=bind,source=$(pwd),target=/bpmnt nginx bash

tvajjala-mac:~ tvajjala$ docker container inspect 55faf5da0652 --format '{{json .Mounts}}' | python -m json.tool
        "Destination": "/bpmnt",
        "Mode": "",
        "Propagation": "rprivate", (1)
        "RW": true,
        "Source": "/Users/tvajjala/Documents/Personal/bindPropgation",
        "Type": "bind"
  1. default Propagation is rprivate

bind-propagation with private mode

docker run --rm -it --mount type=bind,source="$(pwd)"/src,target=/bpmnt,bind-propagation=private \
                    --mount type=bind,source="$(pwd)"/src,target=/bpmnt2,bind-propagation=private \
                    nginx bash

SELinux label

if you use selinux you can add z or Z option to modify the selinux label of the host file.

  • The z option indicates that the bind mount content is shared among multiple containers.

  • The Z option indicates that bind mount content is private and unshared

docker run --rm -it -v "$(pwd)"/src:/app:z nginx bash

Consistency Option (MacOS)

on macOS every time write happens on host or through mount in a container, the changes are flushed to disk. latest docker version on macOS introduced new flag for mount

docker run --rm -it --mount type=bind,source="$(pwd)",destination=/app,consistency=cached nginx bash
  • consistent/default : fully consistent

  • delegated: delay in container changes visible on host

  • cached: delay in host changes visible on container

Directory Type

The overlay storage driver relies on a technology called "directory entry type" (d_type) and is used to describe information of a directory on the filesystem This can cause some issues with chowning and chmoding, deleting and creating files and directories because the overlay storage driver can’t find directory type (d_type) entries in the Linux kernel.

You can check if your existing XFS filesystem has d_type enabled by running the xfs_info. more…​

[root@storfactory-3 docker]# xfs_info /
meta-data=/dev/sda3              isize=256    agcount=4, agsize=2515200 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0 rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=10060800, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1 (1)
log      =internal               bsize=4096   blocks=4912, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
  1. When ftype is 0, d_type support is disabled. When it is 1, d_type support is enabled and you’re safe to use the overlay(2) storage driver with Docker on an XFS filesystem.

mkfs.xfs -n ftype=1 /mount-point
it isn’t possible to enable d_type support on an existing filesystem

Container Size on disk

To view approximate size of running container, run below command

docker ps -s
A           1 MB (virtual 10 MB)
B           1 MB (virtual 10 MB)
  • Size: amount of data(on disk) that is used for writable layer of each container.

  • virtual size: amount of data used for the read-only image data used by the container plus size

If A, B containers created from same image, Total Disk space used by containers (A+B) = SUM(Size(A)+ Size(B)) + (Virtual Size of (A or B) - Size(A | B))

Ex: (1 MB + 1 MB ) + (10 MB -1 MB) = 2 MB + 9 MB = 11 MB on disk

tvajjala-mac:.docker tvajjala$ docker ps -as
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES               SIZE
becb5a407977        nginx               "/docker-entrypoint.…"   16 minutes ago      Up 16 minutes>80/tcp   trusting_gould      1.12kB (virtual 133MB)

tvajjala-mac:.docker tvajjala$ docker images
REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
nginx                    latest              4bb46517cac3        13 days ago         133MB
nginx image size is equals to virtual size of container.

CoW Strategy

If a file or directory exists in a lower layer within the image, and another layer (including the writable layer) needs read access to it, it just uses the existing file. The first time another layer needs to modify the file (when building the image or running the container), the file is copied into that layer and modified.

Docker volumes not accessible directly on MacOS as It’s hidden inside the xhyve virtual machine.
$ docker volume ls

Storage Driver Vs Volume Plugins

Docker uses storage drivers to manage the contents of the image layers and the writable container layer. Each storage driver handles the implementation differently, but all drivers use stackable image layers and the copy-on-write (CoW) strategy. more…​

Storage driver controls how images and containers are stored and managed on your docker host. whereas volume driver helps to manage volume mounts. (i.e to create volume on remote host/cloud encrypt volume data)

Storage Driver

Storage Drivers

  • overlay2 is the preferred (default) storage driver that supports all linux distributions.

  • aufs is the preferred storage driver for docker 18.06 and older.

  • devicemapper is recommended storage driver for CentOS and RHEL , as the old kernal version did not support overlay2. requires direct-lvm for production environments, because loopback-lvm has very poor performance.

  • btrfs/zfs used for the host file system is btrfs/zfs.

  • vfs storage driver intended for testing purposes, performance of this driver is poor not recommended for production.

How to change Storage Driver

You can change storage driver in different ways as shown below. more…​

  • You can pass docker option to dockerd while running docker engine. more…​

 dockerd --storage-driver=devicemapper
  • You can pass driver under OPTIONS in /etc/sysconfig/docker file.

  • Add entry in /etc/docker/daemon.json

  "storage-driver": "devicemapper"
container writable layers store at /var/lib/docker/<storage-driver>/ location.

Volume Plugins(Drivers)

Volume plugins enable Engine deployments to be integrated with external storage systems such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Docker host.

How to change Volume Plugins

if you want to use different volume plugin(driver), you need to install driver plugin as show below. more…​

you can explore available volume plugins in docker hub
# install your driver plugin
docker plugin install store/hypergrid/hypercloud:1.5 --alias hypercloud

# create volume with driver
docker volume create -d hypercloud --name vol-100


Inspect Volumes

When you run inspect command on container you can find StorageDriver(Overlay2) and Volume Driver(local) and corresponding layers on host

        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/333/diff",
                "MergedDir": "/var/lib/docker/overlay2/333/merged",
                "UpperDir": "/var/lib/docker/overlay2/333/diff",
                "WorkDir": "/var/lib/docker/overlay2/333/work"
            "Name": "overlay2" (1)

        "Mounts": [
                "Type": "volume",
                "Name": "9d8aa8ef5c8d26a4491697b1ea8959bd41bb3b083ac342a5a578a82f13f93470",
                "Source": "/var/lib/docker/volumes/9d8aa8ef5c8d26a4491697b1ea8959bd41bb3b083ac342a5a578a82f13f93470/_data",
                "Destination": "/CERTS",
                "Driver": "local", (2)
                "Mode": "",
                "RW": true,
                "Propagation": ""
  1. is the storage driver name which is specific docker engine(daemon)

  2. is the volume driver(plugin) that is specific to volume mounted to the container.

Changing Volume Default Location

Default location where docker stores these drivers is /var/lib/docker on linux.

  • you change this location in /etc/sysconfig/docker file and restart docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS='-g /new/path/docker'
  • You can also change in /lib/systemd/system/docker.service

$ cat /lib/systemd/system/docker.service

# for containers run by docker
#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecStart=/usr/bin/dockerd  -g /new/path/docker -H fd:// --containerd=/run/containerd/containerd.sock

Docker Deamon Flow

Overlay2 Storage Driver

By default, newer versions of Docker will prefer overlay2 if it is available and has the necessary filesystem to support it. Technically overlay2 works on both ext4 and xfs filesystems. Using xfs is preferred over ext4 but the main thing you need to know about your xfs filesystem would be that it must be formatted correctly with ftype=1.

For example, if I had a partition I was going to use for /var/lib/docker named sdb1, I should have formatted it with:

mkfs.xfs -n ftype=1 /dev/sdb1
mkfs.ext4 -n ftype=1 /dev/mapper/volgroup01-scratch

Then just mount /dev/sdb1 as you normally would in your /etc/fstab (or whatever syntax tickles your fancy):

UUID=173d6966-4df5-4417-9a55-db9f8cac8cde /var/lib/docker xfs defaults 0 0 I can use a basic daemon.json to set overlay2 as the storage driver:

    "storage-driver": "overlay2"
The biggest difference between overlay2 and devicemapper is that overlay2 uses a formatted and mounted filesystem vs using a block device.

When you run systemctl status docker first loads content /usr/lib/systemd/system/docker.service

then loads drops-In content from /etc/systemd/system/docker.service.d directory

[root@lcm-0005 docker.service.d]# systemctl status docker -l
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─docker-sysconfig.conf, http-proxy.conf
   Active: active (running) since Thu 2020-08-27 17:47:34 PDT; 6min ago
     Docs: https://docs.docker.com
 Main PID: 21947 (dockerd)
    Tasks: 9
   Memory: 37.2M
   CGroup: /system.slice/docker.service
           └─21947 /usr/bin/dockerd -g /var/lib/docker

level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc000512340, READY" module=grpc
level=info msg="[graphdriver] using prior storage driver: overlay2"
level=info msg="Loading containers: start."
level=info msg="Default bridge (docker0) is assigned with an IP address Daemon option --bip can be used to set a preferred IP address"
level=info msg="Loading containers: done."
level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2
level=info msg="Docker daemon" commit=ead9442 graphdriver(s)=overlay2 version=19.03.1-ol
level=info msg="Daemon has completed initialization"
level=info msg="API listen on /var/run/docker.sock"
systemd[1]: Started Docker Application Container Engine.

Troubleshooting Overlay2

When you run docker on latest linux kernels with Overlay2 storage driver you will see below warning message

Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled

To fix this follow these steps

# make directory if not exists
sudo mkdir -p /etc/modprobe.d

#create file with name overlay-redirect-dir.conf

# add below content into it
echo 'options overlay redirect_dir=off' > overlay-redirect-dir.conf

#Unload and reload the overlay module with the command
sudo modprobe -r overlay && sudo modprobe overlay

#reboot system
sudo reboot

devicemapper Storage Driver

Refer section to change devicemapper more…​

    "storage-driver": "devicemapper",
    "storage-opts": [
  • STEP-1: change storage-driver to devicemapper in /etc/sysconfig/docker or daemon.json

  • STEP-2: docker will create new volume you check with blkid sth like /dev/mapper/docker-252:0-52953143-pool

[tvajjala]# blkid
/dev/mapper/docker-252:0-52953143-pool: UUID="ffffe1ae-ea67-4912-913b-6bce1b779ed0" TYPE="xfs"

Execution order

You can specify docker runtime parameter in different ways.

Below docker-sysconfig.conf file has execution order defined.

[root@lcm-0005 docker.service.d]# cat /etc/systemd/system/docker.service.d/docker-sysconfig.conf
ExecStart=/usr/bin/dockerd \

You can create file /etc/sysconfig/docker-storage and specify storage options as show below.

DOCKER_STORAGE_OPTIONS=-s devicemapper --storage-opt dm.datadev=/dev/my-vg/docker-data --storage-opt dm.metadatadev=/dev/my-vg/docker-metadata

1 comment: