CentOS as a Docker host causes different container behavior compared to any other host OS

[nessi]

Hi,
I am working with Docker on different Hosts : RHEL7 , SELS12 and CentOS7,
And i found a different behavior in containers which running on CentOS7 as Docker Host, comparing to containers which running on SLES12 or RHEL7 as Docker hosts.

The different behavior is related to those common issues in Docker containers:
https://github.com/docker/docker/issues/7147
https://github.com/docker/docker/issues/6800

In container with CentOS7 as Docker host:
I have a permission to resolve a symbolic in the path: /proc/1
command:

Code: Select all

ls -la /proc/1

Start command of the container:

Code: Select all

docker run -it --name=nessi_centos_test centos:latest bash

But in containers with SLES12 or RHEL7 as Docker hosts:
I get permission denied with the same command, as you can see in the above links.
command:

Code: Select all

ls -la /proc/1

Additional information:
Relied on https://docs.docker.com/engine/security/security/ the containers by default start with restricted set of Linux kernel capabilities.

One of those capabilities is: CAP_SYS_PTARCE
this capability exists by default in any Linux host machine:
Example in Linux machine:


But in all containers it is missing by default (unless you start the container with --cap-add=sys_ptrace)
Example in Container:


you can see here that the container have a restricted set of capabilities that not include the sys_ptrace capability.


So if I start the containers with --cap-add=sys_ptrace in RHEL or SLES as Docker hosts, I get the same behavior as I get in the CentOS 7 as a Docker host.

Example:
Docker host: RHEL7
Docker image: centos:latest (same as before )
Strat command:

Code: Select all

docker run -it --name=nessi_centos_test5 --cap-add=sys_ptrace centos:latest bash

as you can see here, for getting the same behavior as I get in CentOS 7 I need to start my containers with additional sys_ptrace capability .

Technical Information:
1. The different CentOS 7 behavior:
Run command in container:

Code: Select all

ls -la /proc/1

results: no errors
2. The regular behavior of other hosts (RHEL7 and SLES12)
Run command in container:

Code: Select all

ls -la /proc/1

results:

Code: Select all

	ls: cannot read symbolic link /proc/1/cwd: Permission denied
	ls: cannot read symbolic link /proc/1/root: Permission denied
	ls: cannot read symbolic link /proc/1/exe: Permission denied
	

1. The different CentOS 7 behavior reproduced in:

[root@localhost Desktop]# docker info
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 1.11.2
Storage Driver: devicemapper
Pool Name: docker-253:0-136686025-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 324.3 MB
Data Space Total: 107.4 GB
Data Space Available: 35.43 GB
Metadata Space Used: 847.9 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.3 MiB
Name: localhost.localdomain
ID: BPVJ:YDPR:4VUO:WNBN:DVZH:7MEH:TPMP:Y3MP:GMN7:UT36:LQ74:GJ4N
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled


[root@localhost Desktop]# uname -a
Linux localhost.localdomain 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Docker images:
centos:latest
ubuntu:14.04

Also tested on:
Docker daemon version : 1.10.2


2. The regular behavior of other hosts (RHEL7 and SLES12)
RHEL7 as Docker host:

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Mon May 5 11:16:57 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# docker info
Containers: 14
Running: 6
Paused: 0
Stopped: 8
Images: 22
Server Version: 1.11.2
Storage Driver: devicemapper
Pool Name: docker-253:0-67168288-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 9.66 GB
Data Space Total: 107.4 GB
Data Space Available: 16.27 GB
Metadata Space Used: 7.68 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.14 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-12-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: Red Hat Enterprise Linux
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.798 GiB
Name: localhost.localdomain
ID: VL2V:RUOZ:U55X:OCEQ:MAS6:MXYV:CKUY:WJQY:3KH3:LWPW:LUYH:E3MM
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Docker images:
centos:latest
centos:7
ubuntu:14.04
ubuntu:latest
rhel:latest
suse/sles12:latest (the image build on SLES machine and copied to RHEL)

Also tested on:
Docker daemon version : 1.10.3. 1.9

SLES12 as Docker host:

linux-ojix:~ # uname -a
Linux linux-ojix 3.12.28-4-default #1 SMP Thu Sep 25 17:02:34 UTC 2014 (9879bd4) x86_64 x86_64 x86_64 GNU/Linux


linux-ojix:~ # docker info
Containers: 6
Running: 3
Paused: 0
Stopped: 3
Images: 10
Server Version: 1.10.3
Storage Driver: btrfs
Build Version: Btrfs v3.18.2+20150430
Library Version: 101
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.12.28-4-default
Operating System: SUSE Linux Enterprise Server 12
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.853 GiB
Name: linux-ojix
ID: NU4F:MOFR:RTUA:F2OM:4G67:NMGV:76S6:BONN:ASD5:XGHF:KVJQ:N242
WARNING: No swap limit support

Docker images:
centos:latest
centos:7
ubuntu:14.04
ubuntu:latest
suse/sles12:latest


Does anyone understand why CentOS as a Docker host causes different container behavior (ls –la /proc/1 in the container - no error) compared to any other host OS (ls –la /proc/1 in the container - with permission denied errors)?

[nessi]

Hi,

The problem is solved, Thanks a lot

The steps to fix the issue (related to RHEL as docker host machine):
1. I installed Docker version: docker-1.10.3-44 from rhel-7-server-extras-rpms
2. Upgraded my kernel from 3.10.0-229.el7.x86_64 to kernel-3.10.0-327.18.2.el7.x86_64.
3. reboot my machine, and now i can resolve the symbolic link under /proc/1 inside the containers.