[jira] [Commented] (AURORA-1902) Docker containers with not newest OS fails to run

["Zameer Manji (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AURORA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901918#comment-15901918 ] 

Zameer Manji commented on AURORA-1902:
--------------------------------------

This is a known flaw/limitation with Mesos and the DockerConainerizer. Mesos will copy/mount the executor into the docker filesystem, meaning that the filesystem needs to be capable of launching the executor. In our case it needs to have Python 2.7 and the dependencies for libmesos.

Tasks launched with the MesosContainerizer do not suffer from this limitation.

> Docker containers with not newest OS fails to run
> -------------------------------------------------
>
>                 Key: AURORA-1902
>                 URL: https://issues.apache.org/jira/browse/AURORA-1902
>             Project: Aurora
>          Issue Type: Bug
>          Components: Docker, Executor
>    Affects Versions: 0.17.0
>         Environment: Ubuntu: 16.04
> Mesos: 1.1.0
> Aurora: 0.17.0
> Dockerengine: 1.13.1
>            Reporter: Mikhail Lesyk
>
> When trying to launch Docker containers, got an error:
> {code}
> I0308 21:47:56.695737  3888 fetcher.cpp:498] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/7cbc133f-24ac-4937-aa28-09e8c81b647b-S7","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"\/usr\/share\/aurora\/bin\/thermos_executor.pex"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/7cbc133f-24ac-4937-aa28-09e8c81b647b-S7\/frameworks\/47934424-623f-4fcb-9326-bf668149fc77-0000\/executors\/thermos-root-prod-test-0-a2c19f58-aa6c-45d8-a47f-8cf57dc0c261\/runs\/788d2f72-a6eb-4f3e-999c-17158e473661"}
> I0308 21:47:56.701079  3888 fetcher.cpp:409] Fetching URI '/usr/share/aurora/bin/thermos_executor.pex'
> I0308 21:47:56.701162  3888 fetcher.cpp:250] Fetching directly into the sandbox directory
> I0308 21:47:56.701225  3888 fetcher.cpp:187] Fetching URI '/usr/share/aurora/bin/thermos_executor.pex'
> I0308 21:47:56.701282  3888 fetcher.cpp:167] Copying resource with command:cp '/usr/share/aurora/bin/thermos_executor.pex' '/var/lib/mesos/slaves/7cbc133f-24ac-4937-aa28-09e8c81b647b-S7/frameworks/47934424-623f-4fcb-9326-bf668149fc77-0000/executors/thermos-root-prod-test-0-a2c19f58-aa6c-45d8-a47f-8cf57dc0c261/runs/788d2f72-a6eb-4f3e-999c-17158e473661/thermos_executor.pex'
> I0308 21:47:56.730024  3888 fetcher.cpp:547] Fetched '/usr/share/aurora/bin/thermos_executor.pex' to '/var/lib/mesos/slaves/7cbc133f-24ac-4937-aa28-09e8c81b647b-S7/frameworks/47934424-623f-4fcb-9326-bf668149fc77-0000/executors/thermos-root-prod-test-0-a2c19f58-aa6c-45d8-a47f-8cf57dc0c261/runs/788d2f72-a6eb-4f3e-999c-17158e473661/thermos_executor.pex'
> WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
> Traceback (most recent call last):
>   File "apache/aurora/executor/bin/thermos_executor_main.py", line 45, in <module>
>     from mesos.executor import MesosExecutorDriver
>   File "/root/.pex/install/mesos.executor-1.1.0-py2.7-linux-x86_64.egg.47fa022c99c11c7faddf379cbfc46a25c5f215be/mesos.executor-1.1.0-py2.7-linux-x86_64.egg/mesos/executor/__init__.py", line 17, in <module>
>     from ._executor import MesosExecutorDriverImpl as MesosExecutorDriver
> ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /root/.pex/install/mesos.executor-1.1.0-py2.7-linux-x86_64.egg.47fa022c99c11c7faddf379cbfc46a25c5f215be/mesos.executor-1.1.0-py2.7-linux-x86_64.egg/mesos/executor/_executor.so)
> twitter.common.app debug: Initializing: twitter.common.log (Logging subsystem.)
> Writing log files to disk in /mnt/mesos/sandbox
> thermos_executor.pex: error: Could not load MesosExecutorDriver!
> twitter.common.app debug: main sys.exited
> twitter.common.app debug: Shutting application down.
> twitter.common.app debug: Running exit function for twitter.common.log (Logging subsystem.)
> twitter.common.app debug: Finishing up module teardown.
> twitter.common.app debug:   Active thread: <_MainThread(MainThread, started 140218447816512)>
> twitter.common.app debug: Exiting cleanly.
> {code}
> Tested affected systems(with absent of GLIBCXX_3.4.20, GLIBCXX_3.4.21):
> Debian 8
> Ubuntu 14.04
> How to reproduce:
> 1) Prepare Docker image with Python 2.7. Example of Dockerfile:
> {code}
> FROM ubuntu:14.04
> RUN apt-get -y update && apt-get -y install python2.7
> {code}
> 2) build and push image to some repo, example:
> {code}
> docker build -t mlesyk/ubuntu:14.04 . && docker push mlesyk/ubuntu:14.04
> {code}
> 3) create some job with Docker container with any command to run, for example, 
> {code}
> sleep 60
> {code}
> and appropriate container parameter, for example:
> {code}
>     container = Docker(image='mlesyk/ubuntu:14.04')
> {code}
> 4) Run this job in Aurora and observe error from beginning of ticket



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)