[jira] [Created] (MESOS-5839) Mesos docker image can't be started by docker-py

["Chen Zhiwei (JIRA)" <ji...@apache.org>]

Chen Zhiwei created MESOS-5839:
----------------------------------

             Summary: Mesos docker image can't be started by docker-py
                 Key: MESOS-5839
                 URL: https://issues.apache.org/jira/browse/MESOS-5839
             Project: Mesos
          Issue Type: Bug
            Reporter: Chen Zhiwei


I can use `docker run` command to start a Mesos Agent container, but can't use docker-py to start.

When I use docker-py to start the Mesos Agent container, the mesos agent error message:
{quote}
I0713 02:07:55.161175 24808 logging.cpp:193] INFO level logging started!
I0713 02:07:55.162096 24808 main.cpp:264] Build: 2016-06-22 05:16:01 by root
I0713 02:07:55.162303 24808 main.cpp:266] Version: 1.0.0
I0713 02:07:55.162593 24808 main.cpp:269] Git tag: 1.0.0-rc1
I0713 02:07:55.162782 24808 main.cpp:273] Git SHA: dfe62665df67162e4c1064f524d6c0180100a9d2
I0713 02:07:55.169150 24808 systemd.cpp:237] systemd version `229` detected
I0713 02:07:55.277576 24808 containerizer.cpp:198] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
I0713 02:07:55.298914 24808 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0713 02:07:55.302297 24808 main.cpp:406] Starting Mesos agent
I0713 02:07:55.302889 24820 slave.cpp:203] Agent started on 1)@9.21.60.205:5051
I0713 02:07:55.303081 24820 slave.cpp:204] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="false" --authenticatee="crammd5" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="docker,mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO" --master="9.21.60.90:5050" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="false" --systemd_enable_support="false" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/var/lib/mesos"
..........
W0713 02:07:55.481570 24808 logging.cpp:91] {color:blue}*RAW: Received signal SIGTERM from process 19221 of user 0; exiting*{color}
{quote}

The process 19221 is the docker-containerd daemon. I am not sure if this issue if related to this fix: https://issues.apache.org/jira/browse/MESOS-4279 , but mesosphere/mesos:0.28.2 has no this issue.

Following is the Docker daemon log:

{quote}{code}
Jul 12 22:07:54 ubuntu5 docker[19214]: time="2016-07-12T22:07:54.954452857-04:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead."
Jul 12 22:07:54 ubuntu5 kernel: [141068.247724] aufs au_opts_verify:1597:docker[26535]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 12 22:07:54 ubuntu5 kernel: [141068.269016] aufs au_opts_verify:1597:docker[26535]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 12 22:07:54 ubuntu5 kernel: [141068.287598] aufs au_opts_verify:1597:docker[19714]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 12 22:07:55 ubuntu5 docker[19214]: time="2016-07-12T22:07:55.509756825-04:00" level=info msg="Container ea0f9128d489f56f3b9f64a24926a53e499b4b6e5243301a01c69e171d4f054b failed to exit within 0 seconds of signal 15 - using the force"
Jul 12 22:07:55 ubuntu5 docker[19214]: time="2016-07-12T22:07:55.568161690-04:00" level=warning msg="container ea0f9128d489f56f3b9f64a24926a53e499b4b6e5243301a01c69e171d4f054b restart canceled"
{code}{quote}

h2. Docker run commands:
{code}
docker run -d --net=host --pid=host --privileged -e MESOS_MASTER=9.21.60.192:5050 -e MESOS_SWITCH_USER=0 -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_LOG_DIR=/var/log/mesos -e MESOS_WORK_DIR=/var/lib/mesos -v /var/log/mesos:/var/log/mesos -v /var/lib/mesos:/var/lib/mesos -v /var/run/docker.sock:/var/run/docker.sock -v /sys:/sys -v /var/lib/docker:/var/lib/docker -v /cgroup:/cgroup -v /dev:/dev chenzhiwei/mesos:1.0.0-rc1 mesos-slave --no-systemd_enable_support
{code}

h2. docker-py code:
{code}
from docker import Client

cli = Client(base_url='unix://var/run/docker.sock')

host_config=cli.create_host_config(privileged=True, network_mode="host", pid_mode="host", binds=["/dev:/dev", "/sys:/sys", "/cgroup:/cgroup", "/var/lib/mesos:/var/lib/mesos", "/var/log/mesos:/var/log/mesos", "/var/lib/docker:/var/lib/docker", "/var/run/docker.sock:/var/run/docker.sock"])

cli.create_container(image='chenzhiwei/mesos:1.0.0-rc1', command="mesos-slave --no-systemd_enable_support", environment={"MESOS_LOG_DIR": "/var/log/mesos", "MESOS_WORK_DIR": "/var/lib/mesos", "MESOS_MASTER": "9.21.60.192:5050", "MESOS_SWITCH_USER": 0, "MESOS_CONTAINERIZERS": "docker,mesos"}, volumes=["/dev", "/sys", "/cgroup", "/var/lib/mesos", "/var/log/mesos", "/var/lib/docker", "/var/run/docker.sock"], host_config=host_config, name="mesos-agent")

cli.start(container="xxxxx")
{code}

I tested with docker create and docker start, still success. I am not sure if this issue belongs to mesos or docker-py.

I also tried to use mesosphere official docker image(mesosphere/mesos:1.0.0-rc1) but still without luck. The docker run command works all right, but docker-py failed with same error message.

*NOTE*: My Docker host is Ubuntu 16.04 and Docker image is chenzhiwei/mesos:1.0.0-rc1, but you can also use the mesosphere/mesos-1.0.0-rc1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)