You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by tommy xiao <xi...@gmail.com> on 2015/12/17 18:07:49 UTC
Re: Mesos fetcher in dockerized slave
docker in docker mode?
2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
> Dear all,
>
> I'm testing the URIs fetching mechanism for both Marathon applications and
> Chronos jobs and I have found that if the slave is running inside a docker
> container (using *docker_mesos_image* startup flag) and you submit the
> deployment of a dockerized application or job the fetcher step is not
> performed. On the other hand, if I request the deployment of a
> non-dockerized application, the URIs are correctly fetched. Moreover, if I
> don’t provide the docker_mesos_image flag, the fetcher works fine again for
> both dockerized and non-dockerized applications.
>
> Therefore, it seems that the information about the URIs gets lost when the
> dockerized mesos slave spawns the executor docker container that in turn
> launches the application docker container…Has anyone seen this problem
> before? I would like to know if there is a workaround or a fixing.
>
> Thanks a lot in advance for you help
> Best Regards,
> Marica
>
>
> ----------------------------------------------------------
>
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
Thank you, I did not receive the notification (don’t know why…maybe I missed it)
Thanks to all!
Best,
Marica
Il giorno 02/mar/2016, alle ore 09:38, haosdent <ha...@gmail.com> ha scritto:
> I saw MESOS-4249 have already fixed in 0.28?
>
> On Wed, Mar 2, 2016 at 4:31 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
> Dear Shuai and Tim,
>
> any news about the patch for MESOS-4743?
> Can I ask you when it will be officially released?
>
> Thank you in advance
> Best regards,
> Marica
>
> Il giorno 24/feb/2016, alle ore 11:23, Shuai Lin <li...@gmail.com> ha scritto:
>
>> ping @Tim, I think this bug also affects https://issues.apache.org/jira/browse/MESOS-4743 .
>>
>>
>>
>> On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin <li...@gmail.com> wrote:
>> The testing of this case requires to build a docker image for mesos-slave, so it seems not practical to add a test case for it in the mesos tests.
>>
>> Anyway, here is the scripts I use for testing this issue: https://gist.github.com/lins05/14455e92f37e91fd46ff
>>
>> On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin <li...@gmail.com> wrote:
>> Hi Tim,
>>
>> The review is here: https://reviews.apache.org/r/42390/ , would you please take a look?
>>
>> Regards,
>> Shuai
>>
>> On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin <li...@gmail.com> wrote:
>> Hi Maria and Tim,
>>
>> I'm setting up a test case for this scenario that would fail, after which I'll begin on fixing it.
>>
>> Is it feasible to include the fixing in the new release?
>>
>> I'm not sure, does 0.27 have an estimated release date now?
>>
>>
>> Regards,
>> Shuai
>>
>> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <ti...@mesosphere.io> wrote:
>> I can shepherd no problem.
>>
>> Tim
>>
>> On Dec 25, 2015, at 4:32 PM, Shuai Lin <li...@gmail.com> wrote:
>>
>>> I'll work on it. @Tim could you shepherd it?
>>>
>>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>> Hi Tim and Shuai,
>>>
>>> thank you very much for your reply. I have opened a JIRA issue on this: https://issues.apache.org/jira/browse/MESOS-4249
>>> I hope it will be patched soon :)
>>>
>>> Best regards,
>>> Marica
>>>
>>>
>>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <ti...@mesosphere.io> ha scritto:
>>>
>>>> Hi Marica/Shuai,
>>>>
>>>> Sorry haven't been able to spend the time to repro, but looks like Shuai confirmed it.
>>>>
>>>> Can one of you file a JIRA?
>>>>
>>>> Thanks!
>>>>
>>>> Tim
>>>>
>>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com> wrote:
>>>> Hi Marica,
>>>>
>>>> I can reproduce the problem exactly as you described in the first email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set, the fetcher works just fine; With it, the fetcher steps seems skipped. This looks like a bug to me.
>>>>
>>>> Regards,
>>>> Shuai
>>>>
>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>>> Dear all,
>>>>
>>>> I have not solved this issue yet. Please, can anyone run the same test and let me know if the fetcher is correctly invoked?
>>>> The test is really simple, just try to start a dockerized app (see json definition file below) through marathon on a mesos slave running in a docker container started with the option —docker_mesos_image=<mesos slave image>.
>>>> I would appreciate very much any feedback.
>>>>
>>>> Sample Marathon app:
>>>> {
>>>> "id": "test-app",
>>>> "container": {
>>>> "type": "DOCKER",
>>>> "docker": {
>>>> "image": "libmesos/ubuntu"
>>>> }
>>>> },
>>>> "cpus": 1,
>>>> "mem": 512,
>>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>>> }
>>>>
>>>> Docker run command to start dockerized mesos slave:
>>>>
>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>>>>
>>>> Thank you very much in advance!
>>>> Best regards,
>>>> Marica
>>>>
>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <ma...@ba.infn.it> ha scritto:
>>>>
>>>>> Dear Tim,
>>>>>
>>>>> I have collected some information from my test environment, starting the slave container with and without the —docker_mesos_image startup flag. Please let me know if you need further input. Thank you very much for your support!
>>>>>
>>>>> Using the flag —docker_mesos_image:
>>>>>
>>>>> root@mesos-slave:~# docker ps
>>>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 minutes ago Up 3 minutes slave
>>>>>
>>>>> root@mesos-slave:~# docker logs slave
>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user 'root'
>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update is identical to existing resources
>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@90.147.170.246:39381
>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@90.147.170.246:39381
>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>
>>>>> root@mesos-slave:~# docker logs mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
>>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>> Registered docker executor on 90.147.170.246
>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>
>>>>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor bash
>>>>> root@mesos-slave:/# ls -R /tmp/
>>>>> hsperfdata_root/ mesos/
>>>>> root@mesos-slave:/# ls -R /tmp/mesos/
>>>>> /tmp/mesos/:
>>>>> slaves
>>>>>
>>>>> /tmp/mesos/slaves:
>>>>> db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>>>>> frameworks
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>>>>> executors
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>>>>> runs
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>>>>> d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>>>>> stderr stdout
>>>>>
>>>>>
>>>>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d bash
>>>>> root@mesos-slave:/# env
>>>>> HOSTNAME=mesos-slave
>>>>> HOST=90.147.170.246
>>>>> PORT0=31220
>>>>> PORT_10000=31220
>>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>>> PWD=/
>>>>> PORTS=31220
>>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>> SHLVL=1
>>>>> HOME=/
>>>>> MARATHON_APP_ID=/test-app
>>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>>>>> PORT=31220
>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>> _=/usr/bin/env
>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>>>>> /mnt/mesos/sandbox:
>>>>> stderr stdout
>>>>>
>>>>>
>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>> /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>>>>
>>>>> /var/log/mesos:
>>>>>
>>>>>
>>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is created, the file is downloaded in the sandbox:
>>>>>
>>>>> root@mesos-slave:~# docker exec -it slave bash
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>> /var/log/mesos-fetcher.INFO /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157 /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>>>>
>>>>> /var/log/mesos:
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>>>>> Log file created at: 2015/12/19 18:16:21
>>>>> Running on machine: mesos-slave
>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started!
>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the sandbox directory
>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>> root@mesos-slave:/#
>>>>>
>>>>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 bash
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>>>>> snoqualmie.csv stderr stdout
>>>>> root@mesos-slave:/#
>>>>>
>>>>>
>>>>> Best Regards,
>>>>> Marica
>>>>>
>>>>>
>>>>>
>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io> ha scritto:
>>>>>
>>>>>> HI Marica,
>>>>>>
>>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't seem possible we don't pass down the URI flags and if the fetcher failed the container launch should have failed too.
>>>>>>
>>>>>> Also another possible situation is that web UI not really showing the exact content of the sandbox, can you actually go into the directory and see if the files are there?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>>>>> Hi Tim,
>>>>>>
>>>>>> looking at the sandbox I can see only the stderr and stout file (see the attached screenshot). If I remove —docker_mesos_image (and therefore the executor is run inside the slave container) the file specified in the URI field is shown in the sandbox.
>>>>>> Did you verify that the fetcher is called when using the —docker_mesos_image flag ?
>>>>>>
>>>>>> Thanks a lot for your feedback
>>>>>> Marica
>>>>>>
>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>
>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io> ha scritto:
>>>>>>
>>>>>>> Hi Marica,
>>>>>>>
>>>>>>> It should work as we fetch all the files before we launch the executor and place them in the sandbox, and we mount the sandbox into that container as well.
>>>>>>>
>>>>>>> How did you verify that the file is not downloaded?
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>>>>>> Hi Grzegorz,
>>>>>>>
>>>>>>> I’m using this command line for docker run
>>>>>>>
>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>>>>>>>
>>>>>>> where mesos-slave is the image built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>>>>
>>>>>>> I have tested successfully the deployment of dockerized applications through Marathon and dockerized jobs through Chronos and also the recovery seems to work fine with the flag docker_mesos_image. What is not working for me is the fetcher: it seems that when the executor is launched as separate container (thanks to the flag docker_mesos_image) the information about the URIs to be downloaded is lost…I hope someone can help to understand if this a bug or I’ m missing something.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gr...@gmail.com> ha scritto:
>>>>>>>
>>>>>>>> I've tried to use this flag, but cannot really run any container when this flag is set.
>>>>>>>> I've raised this issue here: https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and here: https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but sadly no one was able to help me...
>>>>>>>>
>>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <ma...@ba.infn.it> napisał:
>>>>>>>> OK, the problem I spotted is related to the usage of the flag —docker_mesos_image that allows the executor to
>>>>>>>>
>>>>>>>>
>>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this mesos slave instance. If an image is specified, the docker containerizer assumes the slave is running in a docker container, and launches executors with docker containers in order to recover them when the slave restarts and recovers.
>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>>>>
>>>>>>>> Thank you
>>>>>>>> Marica
>>>>>>>>
>>>>>>>>
>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>
>>>>>>>>> no docker_mesos_image flag in my docker run, and the docker image is build by myself.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>
>>>>>>>>> Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
>>>>>>>>>
>>>>>>>>> Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
>>>>>>>>>
>>>>>>>>> Thank you very much
>>>>>>>>
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>>
>>>>>>>>>> Hi Marica,
>>>>>>>>>>
>>>>>>>>>> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>>>>>>>>>>
>>>>>>>>>> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>>>>>>>>>>
>>>>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>>>>
>>>>>>>>>> the folder in app container is default value:
>>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>>>>
>>>>>>>>>> but in real world, the sandbox is in slave container, not in app container.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>>> Thank you very much,
>>>>>>>>>>
>>>>>>>>>> I’m using a sample application definition file, just for testing purpose:
>>>>>>>>>>
>>>>>>>>>> {
>>>>>>>>>> "id": "test-app",
>>>>>>>>>> "container": {
>>>>>>>>>> "type": "DOCKER",
>>>>>>>>>> "docker": {
>>>>>>>>>> "image": "libmesos/ubuntu"
>>>>>>>>>> }
>>>>>>>>>> },
>>>>>>>>>> "cpus": 1,
>>>>>>>>>> "mem": 512,
>>>>>>>>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Here is the docker run command line:
>>>>>>>>>>
>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>>>>>>>>>>
>>>>>>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>>>>>>> Best regards,
>>>>>>>>>> Marica
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>
>>>>>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>>>>>
>>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>>>> No, using the socket:
>>>>>>>>>>>
>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>>> docker in docker mode?
>>>>>>>>>>>>
>>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>>>>>>>>>>
>>>>>>>>>>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Marica
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>
>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>
>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>> Via Orabona 4
>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Deshi Xiao
>>>>>>>>>> Twitter: xds2000
>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Deshi Xiao
>>>>>>>>> Twitter: xds2000
>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ----------------------------------------------------------
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ----------------------------------------------------------
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Best Regards,
> Haosdent Huang
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by haosdent <ha...@gmail.com>.
I saw MESOS-4249 have already fixed in 0.28?
On Wed, Mar 2, 2016 at 4:31 PM, Marica Antonacci <
marica.antonacci@ba.infn.it> wrote:
> Dear Shuai and Tim,
>
> any news about the patch for MESOS-4743?
> Can I ask you when it will be officially released?
>
> Thank you in advance
> Best regards,
> Marica
>
> Il giorno 24/feb/2016, alle ore 11:23, Shuai Lin <li...@gmail.com>
> ha scritto:
>
> ping @Tim, I think this bug also affects
> https://issues.apache.org/jira/browse/MESOS-4743 .
>
>
>
> On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin <li...@gmail.com>
> wrote:
>
>> The testing of this case requires to build a docker image for
>> mesos-slave, so it seems not practical to add a test case for it in the
>> mesos tests.
>>
>> Anyway, here is the scripts I use for testing this issue:
>> https://gist.github.com/lins05/14455e92f37e91fd46ff
>>
>> On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin <li...@gmail.com>
>> wrote:
>>
>>> Hi Tim,
>>>
>>> The review is here: https://reviews.apache.org/r/42390/ , would you
>>> please take a look?
>>>
>>> Regards,
>>> Shuai
>>>
>>> On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin <li...@gmail.com>
>>> wrote:
>>>
>>>> Hi Maria and Tim,
>>>>
>>>> I'm setting up a test case for this scenario that would fail, after
>>>> which I'll begin on fixing it.
>>>>
>>>> Is it feasible to include the fixing in the new release?
>>>>
>>>>
>>>> I'm not sure, does 0.27 have an estimated release date now?
>>>>
>>>>
>>>> Regards,
>>>> Shuai
>>>>
>>>> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <ti...@mesosphere.io> wrote:
>>>>
>>>>> I can shepherd no problem.
>>>>>
>>>>> Tim
>>>>>
>>>>> On Dec 25, 2015, at 4:32 PM, Shuai Lin <li...@gmail.com> wrote:
>>>>>
>>>>> I'll work on it. @Tim could you shepherd it?
>>>>>
>>>>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>
>>>>>> Hi Tim and Shuai,
>>>>>>
>>>>>> thank you very much for your reply. I have opened a JIRA issue on
>>>>>> this: https://issues.apache.org/jira/browse/MESOS-4249
>>>>>> I hope it will be patched soon :)
>>>>>>
>>>>>> Best regards,
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <ti...@mesosphere.io>
>>>>>> ha scritto:
>>>>>>
>>>>>> Hi Marica/Shuai,
>>>>>>
>>>>>> Sorry haven't been able to spend the time to repro, but looks like
>>>>>> Shuai confirmed it.
>>>>>>
>>>>>> Can one of you file a JIRA?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Marica,
>>>>>>>
>>>>>>> I can reproduce the problem exactly as you described in the first
>>>>>>> email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment
>>>>>>> variable set, the fetcher works just fine; With it, the fetcher steps seems
>>>>>>> skipped. This looks like a bug to me.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Shuai
>>>>>>>
>>>>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <
>>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>>
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> I have not solved this issue yet. Please, can anyone run the same
>>>>>>>> test and let me know if the fetcher is correctly invoked?
>>>>>>>> The test is really simple, just try to start a dockerized app (see
>>>>>>>> json definition file below) through marathon on a mesos slave running in a
>>>>>>>> docker container started with the option —docker_mesos_image=<mesos slave
>>>>>>>> image>.
>>>>>>>> I would appreciate very much any feedback.
>>>>>>>>
>>>>>>>> Sample Marathon app:
>>>>>>>> {
>>>>>>>> "id": "test-app",
>>>>>>>> "container": {
>>>>>>>> "type": "DOCKER",
>>>>>>>> "docker": {
>>>>>>>> "image": "libmesos/ubuntu"
>>>>>>>> }
>>>>>>>> },
>>>>>>>> "cpus": 1,
>>>>>>>> "mem": 512,
>>>>>>>> "uris": [ "
>>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"
>>>>>>>> ],
>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u
>>>>>>>> +%T; done"
>>>>>>>> }
>>>>>>>>
>>>>>>>> Docker run command to start dockerized mesos slave:
>>>>>>>>
>>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e
>>>>>>>> MESOS_CONTAINERIZERS=docker,mesos
>>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos
>>>>>>>> --name slave --net host --privileged --pid host mesos-slave
>>>>>>>>
>>>>>>>> Thank you very much in advance!
>>>>>>>> Best regards,
>>>>>>>> Marica
>>>>>>>>
>>>>>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <
>>>>>>>> marica.antonacci@ba.infn.it> ha scritto:
>>>>>>>>
>>>>>>>> Dear Tim,
>>>>>>>>
>>>>>>>> I have collected some information from my test environment,
>>>>>>>> starting the slave container with and without the —docker_mesos_image
>>>>>>>> startup flag. Please let me know if you need further input. Thank you very
>>>>>>>> much for your support!
>>>>>>>>
>>>>>>>> Using the flag —docker_mesos_image:
>>>>>>>>
>>>>>>>> root@mesos-slave:~# docker ps
>>>>>>>> CONTAINER ID IMAGE COMMAND
>>>>>>>> CREATED STATUS PORTS NAMES
>>>>>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2
>>>>>>>> minutes ago Up 2 minutes
>>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2
>>>>>>>> minutes ago Up 2 minutes
>>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3
>>>>>>>> minutes ago Up 3 minutes slave
>>>>>>>>
>>>>>>>> *root@mesos-slave:~# docker logs slave*
>>>>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task
>>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task
>>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown
>>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>>>>> to user 'root'
>>>>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor
>>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1;
>>>>>>>> mem(*):32 in work directory
>>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task
>>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor
>>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container
>>>>>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task
>>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor
>>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework
>>>>>>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>>>>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590
>>>>>>>> to
>>>>>>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>>>>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for
>>>>>>>> executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>>>>> 90.147.170.246:39381
>>>>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating
>>>>>>>> container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to
>>>>>>>> update is identical to existing resources
>>>>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task
>>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor
>>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@
>>>>>>>> 90.147.170.246:39381
>>>>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update
>>>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>>>>> 90.147.170.246:39381
>>>>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received
>>>>>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>>>>>>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826]
>>>>>>>> Checkpointing UPDATE for status update TASK_RUNNING (UUID:
>>>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update
>>>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209
>>>>>>>> :5050
>>>>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement
>>>>>>>> for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@
>>>>>>>> 90.147.170.246:39381
>>>>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received
>>>>>>>> status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826]
>>>>>>>> Checkpointing ACK for status update TASK_RUNNING (UUID:
>>>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>>>
>>>>>>>> root@mesos-slave:~# *docker logs
>>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor*
>>>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>>> --stop_timeout="0ns"
>>>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>>> --stop_timeout="0ns"
>>>>>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>>>>>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on
>>>>>>>> slave db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>>>>> Registered docker executor on 90.147.170.246
>>>>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>>>>
>>>>>>>> root@mesos-slave:~# *docker exec -it
>>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>>>> bash*
>>>>>>>> root@mesos-slave:/# ls -R /tmp/
>>>>>>>> hsperfdata_root/ mesos/
>>>>>>>> root@mesos-slave:/# ls -R /tmp/mesos/
>>>>>>>> /tmp/mesos/:
>>>>>>>> *slaves*
>>>>>>>>
>>>>>>>> /tmp/mesos/slaves:
>>>>>>>> *db70e09f-f39d-491c-8480-73d9858c140b-S0*
>>>>>>>>
>>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>>>>>>>> *frameworks*
>>>>>>>>
>>>>>>>>
>>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>>>>>>>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000*
>>>>>>>>
>>>>>>>>
>>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>>>>>>>> *executors*
>>>>>>>>
>>>>>>>>
>>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>>>>>>>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0*
>>>>>>>>
>>>>>>>>
>>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>>>>>>>> *runs*
>>>>>>>>
>>>>>>>>
>>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>>>>>>>> *d965f59b-cc1a-4081-95d2-f3370214c84d*
>>>>>>>>
>>>>>>>>
>>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>>>>>>>> stderr stdout
>>>>>>>>
>>>>>>>>
>>>>>>>> root@mesos-slave:~# *docker exec -it
>>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>>> bash*
>>>>>>>> root@mesos-slave:/# env
>>>>>>>> HOSTNAME=mesos-slave
>>>>>>>> HOST=90.147.170.246
>>>>>>>> PORT0=31220
>>>>>>>> PORT_10000=31220
>>>>>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>>>>>> PWD=/
>>>>>>>> PORTS=31220
>>>>>>>>
>>>>>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>>> SHLVL=1
>>>>>>>> HOME=/
>>>>>>>> MARATHON_APP_ID=/test-app
>>>>>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>>>>>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>>>>>>>> PORT=31220
>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>> _=/usr/bin/env
>>>>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>>>>>>>> /mnt/mesos/sandbox:
>>>>>>>> stderr stdout
>>>>>>>>
>>>>>>>>
>>>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>>>> */var/log/mesos-slave.INFO*
>>>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>>>>>>>
>>>>>>>> /var/log/mesos:
>>>>>>>>
>>>>>>>>
>>>>>>>> Disabling the flag —docker_mesos_image the fetcher is called, log
>>>>>>>> is created, the file is downloaded in the sandbox:
>>>>>>>>
>>>>>>>> root@mesos-slave:~# docker exec -it slave bash
>>>>>>>> root@mesos-slave:/#
>>>>>>>> root@mesos-slave:/#
>>>>>>>> root@mesos-slave:/#
>>>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>>>> */var/log/mesos-fetcher.INFO*
>>>>>>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157
>>>>>>>> */var/log/mesos-slave.INFO*
>>>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>>>>>>>
>>>>>>>> /var/log/mesos:
>>>>>>>> root@mesos-slave:/#
>>>>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>>>>>>>> Log file created at: 2015/12/19 18:16:21
>>>>>>>> Running on machine: mesos-slave
>>>>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging
>>>>>>>> started!
>>>>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info:
>>>>>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/
>>>>>>>> www.stat.cmu.edu
>>>>>>>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>>>>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI '
>>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into
>>>>>>>> the sandbox directory
>>>>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI '
>>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource
>>>>>>>> from '
>>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>>> to
>>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched '
>>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>>> to
>>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>>>> root@mesos-slave:/#
>>>>>>>>
>>>>>>>> root@mesos-slave:~# docker exec -it
>>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4
>>>>>>>> bash
>>>>>>>> root@mesos-slave:/#
>>>>>>>> root@mesos-slave:/#
>>>>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>>>>>>>> snoqualmie.csv stderr stdout
>>>>>>>> root@mesos-slave:/#
>>>>>>>>
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Marica
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io>
>>>>>>>> ha scritto:
>>>>>>>>
>>>>>>>> HI Marica,
>>>>>>>>
>>>>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't
>>>>>>>> seem possible we don't pass down the URI flags and if the fetcher failed
>>>>>>>> the container launch should have failed too.
>>>>>>>>
>>>>>>>> Also another possible situation is that web UI not really showing
>>>>>>>> the exact content of the sandbox, can you actually go into the directory
>>>>>>>> and see if the files are there?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <
>>>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>>>
>>>>>>>>> Hi Tim,
>>>>>>>>>
>>>>>>>>> looking at the sandbox I can see only the stderr and stout file
>>>>>>>>> (see the attached screenshot). If I remove —docker_mesos_image (and
>>>>>>>>> therefore the executor is run inside the slave container) the file
>>>>>>>>> specified in the URI field is shown in the sandbox.
>>>>>>>>> Did you verify that the fetcher is called when using the
>>>>>>>>> —docker_mesos_image flag ?
>>>>>>>>>
>>>>>>>>> Thanks a lot for your feedback
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>>
>>>>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io>
>>>>>>>>> ha scritto:
>>>>>>>>>
>>>>>>>>> Hi Marica,
>>>>>>>>>
>>>>>>>>> It should work as we fetch all the files before we launch the
>>>>>>>>> executor and place them in the sandbox, and we mount the sandbox into that
>>>>>>>>> container as well.
>>>>>>>>>
>>>>>>>>> How did you verify that the file is not downloaded?
>>>>>>>>>
>>>>>>>>> Tim
>>>>>>>>>
>>>>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
>>>>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Grzegorz,
>>>>>>>>>>
>>>>>>>>>> I’m using this command line for docker run
>>>>>>>>>>
>>>>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP>
>>>>>>>>>> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos -e
>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
>>>>>>>>>> slave --net host --privileged --pid host mesos-slave
>>>>>>>>>>
>>>>>>>>>> where mesos-slave is the image built from the docker file in this
>>>>>>>>>> repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>>>>>>>
>>>>>>>>>> I have tested successfully the deployment of dockerized
>>>>>>>>>> applications through Marathon and dockerized jobs through Chronos and also
>>>>>>>>>> the recovery seems to work fine with the flag docker_mesos_image. What is
>>>>>>>>>> not working for me is the fetcher: it seems that when the executor is
>>>>>>>>>> launched as separate container (thanks to the flag docker_mesos_image) the
>>>>>>>>>> information about the URIs to be downloaded is lost…I hope someone can help
>>>>>>>>>> to understand if this a bug or I’ m missing something.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Marica
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
>>>>>>>>>> gregory90@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>> I've tried to use this flag, but cannot really run any container
>>>>>>>>>> when this flag is set.
>>>>>>>>>> I've raised this issue here:
>>>>>>>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>>>>>>>>>> here:
>>>>>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
>>>>>>>>>> sadly no one was able to help me...
>>>>>>>>>>
>>>>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>>>>>>>>>> marica.antonacci@ba.infn.it> napisał:
>>>>>>>>>>
>>>>>>>>>>> OK, the problem I spotted is related to the usage of the
>>>>>>>>>>> flag —docker_mesos_image that allows the executor to
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this
>>>>>>>>>>> mesos slave instance. If an image is specified, the docker containerizer
>>>>>>>>>>> assumes the slave is running in a docker container, and launches executors
>>>>>>>>>>> with docker containers in order to recover them when the slave restarts and
>>>>>>>>>>> recovers.
>>>>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>>>>>>>
>>>>>>>>>>> Thank you
>>>>>>>>>>> Marica
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <
>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>> no docker_mesos_image flag in my docker run, and the docker
>>>>>>>>>>> image is build by myself.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <
>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>
>>>>>>>>>>> Yes, I did check inside the container and the csv file was not
>>>>>>>>>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>>>>>>>>>
>>>>>>>>>>>> Are you running your slave with the --docker_mesos_image flag?
>>>>>>>>>>>> Can you please provide me the docker run command you are using to run your
>>>>>>>>>>>> dockerized slave?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you very much
>>>>>>>>>>>>
>>>>>>>>>>> Marica
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <
>>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>>
>>>>>>>>>>>> use your test-app json, i can run it correctly, the csv is
>>>>>>>>>>>> truely download by mesos slave. please check mesos-master:5050 to check the
>>>>>>>>>>>> task detail download files.
>>>>>>>>>>>>
>>>>>>>>>>>> you describe the app container why not found the csv, because
>>>>>>>>>>>> the csv is download in slave container's folder, not in app container. so
>>>>>>>>>>>> if you run
>>>>>>>>>>>>
>>>>>>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>>>>>>
>>>>>>>>>>>> the folder in app container is default value:
>>>>>>>>>>>>
>>>>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>>>>>> but in real world, the sandbox is in slave container, not in
>>>>>>>>>>>> app container.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you very much,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I’m using a sample application definition file, just for
>>>>>>>>>>>>> testing purpose:
>>>>>>>>>>>>>
>>>>>>>>>>>>> {
>>>>>>>>>>>>> "id": "test-app",
>>>>>>>>>>>>> "container": {
>>>>>>>>>>>>> "type": "DOCKER",
>>>>>>>>>>>>> "docker": {
>>>>>>>>>>>>> "image": "libmesos/ubuntu"
>>>>>>>>>>>>> }
>>>>>>>>>>>>> },
>>>>>>>>>>>>> "cpus": 1,
>>>>>>>>>>>>> "mem": 512,
>>>>>>>>>>>>> *"uris": [
>>>>>>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>>>>>>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>>>>>>>>>> ],*
>>>>>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date
>>>>>>>>>>>>> -u +%T; done"
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here is the docker run command line:
>>>>>>>>>>>>>
>>>>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e
>>>>>>>>>>>>> MESOS_IP=<SLAVE_IP> -e
>>>>>>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>>>>>>>> -e
>>>>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>>>>>>> MESOS_docker_mesos_image=mesos-slave
>>>>>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>>>>>>>>>> --privileged --pid host mesos-slave
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> As already mentioned, if I remove the environment variable
>>>>>>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>>>>>>>>>> snoqualmie.csv inside the sandbox.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you again! I’m looking forward to hearing about your
>>>>>>>>>>>>> outcomes.
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Marica
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <
>>>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you please give a sample marathon json, i can test it
>>>>>>>>>>>>> asap.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> No, using the socket:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <
>>>>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> docker in docker mode?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>>>>>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>>>>>>>>>>> inside a docker container (using *docker_mesos_image*
>>>>>>>>>>>>>>> startup flag) and you submit the deployment of a dockerized application or
>>>>>>>>>>>>>>> job the fetcher step is not performed. On the other hand, if I request the
>>>>>>>>>>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>>>>>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>>>>>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Therefore, it seems that the information about the URIs gets
>>>>>>>>>>>>>>> lost when the dockerized mesos slave spawns the executor docker container
>>>>>>>>>>>>>>> that in turn launches the application docker container…Has anyone seen this
>>>>>>>>>>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Marica
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>>>>>>>>>> 2015-12-18 下午4.55.38.png>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>> Via Orabona 4
>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>>
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>>
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> ----------------------------------------------------------
>
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
--
Best Regards,
Haosdent Huang
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
Dear Shuai and Tim,
any news about the patch for MESOS-4743?
Can I ask you when it will be officially released?
Thank you in advance
Best regards,
Marica
Il giorno 24/feb/2016, alle ore 11:23, Shuai Lin <li...@gmail.com> ha scritto:
> ping @Tim, I think this bug also affects https://issues.apache.org/jira/browse/MESOS-4743 .
>
>
>
> On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin <li...@gmail.com> wrote:
> The testing of this case requires to build a docker image for mesos-slave, so it seems not practical to add a test case for it in the mesos tests.
>
> Anyway, here is the scripts I use for testing this issue: https://gist.github.com/lins05/14455e92f37e91fd46ff
>
> On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin <li...@gmail.com> wrote:
> Hi Tim,
>
> The review is here: https://reviews.apache.org/r/42390/ , would you please take a look?
>
> Regards,
> Shuai
>
> On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin <li...@gmail.com> wrote:
> Hi Maria and Tim,
>
> I'm setting up a test case for this scenario that would fail, after which I'll begin on fixing it.
>
> Is it feasible to include the fixing in the new release?
>
> I'm not sure, does 0.27 have an estimated release date now?
>
>
> Regards,
> Shuai
>
> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <ti...@mesosphere.io> wrote:
> I can shepherd no problem.
>
> Tim
>
> On Dec 25, 2015, at 4:32 PM, Shuai Lin <li...@gmail.com> wrote:
>
>> I'll work on it. @Tim could you shepherd it?
>>
>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
>> Hi Tim and Shuai,
>>
>> thank you very much for your reply. I have opened a JIRA issue on this: https://issues.apache.org/jira/browse/MESOS-4249
>> I hope it will be patched soon :)
>>
>> Best regards,
>> Marica
>>
>>
>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <ti...@mesosphere.io> ha scritto:
>>
>>> Hi Marica/Shuai,
>>>
>>> Sorry haven't been able to spend the time to repro, but looks like Shuai confirmed it.
>>>
>>> Can one of you file a JIRA?
>>>
>>> Thanks!
>>>
>>> Tim
>>>
>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com> wrote:
>>> Hi Marica,
>>>
>>> I can reproduce the problem exactly as you described in the first email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set, the fetcher works just fine; With it, the fetcher steps seems skipped. This looks like a bug to me.
>>>
>>> Regards,
>>> Shuai
>>>
>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>> Dear all,
>>>
>>> I have not solved this issue yet. Please, can anyone run the same test and let me know if the fetcher is correctly invoked?
>>> The test is really simple, just try to start a dockerized app (see json definition file below) through marathon on a mesos slave running in a docker container started with the option —docker_mesos_image=<mesos slave image>.
>>> I would appreciate very much any feedback.
>>>
>>> Sample Marathon app:
>>> {
>>> "id": "test-app",
>>> "container": {
>>> "type": "DOCKER",
>>> "docker": {
>>> "image": "libmesos/ubuntu"
>>> }
>>> },
>>> "cpus": 1,
>>> "mem": 512,
>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>> }
>>>
>>> Docker run command to start dockerized mesos slave:
>>>
>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>>>
>>> Thank you very much in advance!
>>> Best regards,
>>> Marica
>>>
>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <ma...@ba.infn.it> ha scritto:
>>>
>>>> Dear Tim,
>>>>
>>>> I have collected some information from my test environment, starting the slave container with and without the —docker_mesos_image startup flag. Please let me know if you need further input. Thank you very much for your support!
>>>>
>>>> Using the flag —docker_mesos_image:
>>>>
>>>> root@mesos-slave:~# docker ps
>>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 minutes ago Up 3 minutes slave
>>>>
>>>> root@mesos-slave:~# docker logs slave
>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user 'root'
>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update is identical to existing resources
>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@90.147.170.246:39381
>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@90.147.170.246:39381
>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>
>>>> root@mesos-slave:~# docker logs mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>> Registered docker executor on 90.147.170.246
>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>
>>>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor bash
>>>> root@mesos-slave:/# ls -R /tmp/
>>>> hsperfdata_root/ mesos/
>>>> root@mesos-slave:/# ls -R /tmp/mesos/
>>>> /tmp/mesos/:
>>>> slaves
>>>>
>>>> /tmp/mesos/slaves:
>>>> db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>>>> frameworks
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>>>> executors
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>>>> runs
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>>>> d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>>>> stderr stdout
>>>>
>>>>
>>>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d bash
>>>> root@mesos-slave:/# env
>>>> HOSTNAME=mesos-slave
>>>> HOST=90.147.170.246
>>>> PORT0=31220
>>>> PORT_10000=31220
>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>> PWD=/
>>>> PORTS=31220
>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>> SHLVL=1
>>>> HOME=/
>>>> MARATHON_APP_ID=/test-app
>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>>>> PORT=31220
>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>> _=/usr/bin/env
>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>>>> /mnt/mesos/sandbox:
>>>> stderr stdout
>>>>
>>>>
>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>> /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>>>
>>>> /var/log/mesos:
>>>>
>>>>
>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is created, the file is downloaded in the sandbox:
>>>>
>>>> root@mesos-slave:~# docker exec -it slave bash
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>> /var/log/mesos-fetcher.INFO /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157 /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>>>
>>>> /var/log/mesos:
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>>>> Log file created at: 2015/12/19 18:16:21
>>>> Running on machine: mesos-slave
>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started!
>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the sandbox directory
>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>> root@mesos-slave:/#
>>>>
>>>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 bash
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>>>> snoqualmie.csv stderr stdout
>>>> root@mesos-slave:/#
>>>>
>>>>
>>>> Best Regards,
>>>> Marica
>>>>
>>>>
>>>>
>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io> ha scritto:
>>>>
>>>>> HI Marica,
>>>>>
>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't seem possible we don't pass down the URI flags and if the fetcher failed the container launch should have failed too.
>>>>>
>>>>> Also another possible situation is that web UI not really showing the exact content of the sandbox, can you actually go into the directory and see if the files are there?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Tim
>>>>>
>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>>>> Hi Tim,
>>>>>
>>>>> looking at the sandbox I can see only the stderr and stout file (see the attached screenshot). If I remove —docker_mesos_image (and therefore the executor is run inside the slave container) the file specified in the URI field is shown in the sandbox.
>>>>> Did you verify that the fetcher is called when using the —docker_mesos_image flag ?
>>>>>
>>>>> Thanks a lot for your feedback
>>>>> Marica
>>>>>
>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>
>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io> ha scritto:
>>>>>
>>>>>> Hi Marica,
>>>>>>
>>>>>> It should work as we fetch all the files before we launch the executor and place them in the sandbox, and we mount the sandbox into that container as well.
>>>>>>
>>>>>> How did you verify that the file is not downloaded?
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>>>>> Hi Grzegorz,
>>>>>>
>>>>>> I’m using this command line for docker run
>>>>>>
>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>>>>>>
>>>>>> where mesos-slave is the image built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>>>
>>>>>> I have tested successfully the deployment of dockerized applications through Marathon and dockerized jobs through Chronos and also the recovery seems to work fine with the flag docker_mesos_image. What is not working for me is the fetcher: it seems that when the executor is launched as separate container (thanks to the flag docker_mesos_image) the information about the URIs to be downloaded is lost…I hope someone can help to understand if this a bug or I’ m missing something.
>>>>>>
>>>>>> Cheers,
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>>
>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gr...@gmail.com> ha scritto:
>>>>>>
>>>>>>> I've tried to use this flag, but cannot really run any container when this flag is set.
>>>>>>> I've raised this issue here: https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and here: https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but sadly no one was able to help me...
>>>>>>>
>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <ma...@ba.infn.it> napisał:
>>>>>>> OK, the problem I spotted is related to the usage of the flag —docker_mesos_image that allows the executor to
>>>>>>>
>>>>>>>
>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this mesos slave instance. If an image is specified, the docker containerizer assumes the slave is running in a docker container, and launches executors with docker containers in order to recover them when the slave restarts and recovers.
>>>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>>>
>>>>>>> Thank you
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>
>>>>>>>> no docker_mesos_image flag in my docker run, and the docker image is build by myself.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>
>>>>>>>> Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
>>>>>>>>
>>>>>>>> Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
>>>>>>>>
>>>>>>>> Thank you very much
>>>>>>>
>>>>>>>> Marica
>>>>>>>>
>>>>>>>>
>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>
>>>>>>>>> Hi Marica,
>>>>>>>>>
>>>>>>>>> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>>>>>>>>>
>>>>>>>>> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>>>>>>>>>
>>>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>>>
>>>>>>>>> the folder in app container is default value:
>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>>>
>>>>>>>>> but in real world, the sandbox is in slave container, not in app container.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>> Thank you very much,
>>>>>>>>>
>>>>>>>>> I’m using a sample application definition file, just for testing purpose:
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> "id": "test-app",
>>>>>>>>> "container": {
>>>>>>>>> "type": "DOCKER",
>>>>>>>>> "docker": {
>>>>>>>>> "image": "libmesos/ubuntu"
>>>>>>>>> }
>>>>>>>>> },
>>>>>>>>> "cpus": 1,
>>>>>>>>> "mem": 512,
>>>>>>>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Here is the docker run command line:
>>>>>>>>>
>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>>>>>>>>>
>>>>>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>>>>>> Best regards,
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>>
>>>>>>>>>> Hi Marica,
>>>>>>>>>>
>>>>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>>>>
>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>>> No, using the socket:
>>>>>>>>>>
>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>>> docker in docker mode?
>>>>>>>>>>>
>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>>>> Dear all,
>>>>>>>>>>>
>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>>>>>>>>>
>>>>>>>>>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Marica
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>
>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>> Via Orabona 4
>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Deshi Xiao
>>>>>>>>>> Twitter: xds2000
>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Deshi Xiao
>>>>>>>>> Twitter: xds2000
>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Deshi Xiao
>>>>>>>> Twitter: xds2000
>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ----------------------------------------------------------
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ----------------------------------------------------------
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
>
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by Shuai Lin <li...@gmail.com>.
ping @Tim, I think this bug also affects
https://issues.apache.org/jira/browse/MESOS-4743 .
On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin <li...@gmail.com> wrote:
> The testing of this case requires to build a docker image for mesos-slave,
> so it seems not practical to add a test case for it in the mesos tests.
>
> Anyway, here is the scripts I use for testing this issue:
> https://gist.github.com/lins05/14455e92f37e91fd46ff
>
> On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin <li...@gmail.com>
> wrote:
>
>> Hi Tim,
>>
>> The review is here: https://reviews.apache.org/r/42390/ , would you
>> please take a look?
>>
>> Regards,
>> Shuai
>>
>> On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin <li...@gmail.com> wrote:
>>
>>> Hi Maria and Tim,
>>>
>>> I'm setting up a test case for this scenario that would fail, after
>>> which I'll begin on fixing it.
>>>
>>> Is it feasible to include the fixing in the new release?
>>>
>>>
>>> I'm not sure, does 0.27 have an estimated release date now?
>>>
>>>
>>> Regards,
>>> Shuai
>>>
>>> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <ti...@mesosphere.io> wrote:
>>>
>>>> I can shepherd no problem.
>>>>
>>>> Tim
>>>>
>>>> On Dec 25, 2015, at 4:32 PM, Shuai Lin <li...@gmail.com> wrote:
>>>>
>>>> I'll work on it. @Tim could you shepherd it?
>>>>
>>>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci <
>>>> marica.antonacci@ba.infn.it> wrote:
>>>>
>>>>> Hi Tim and Shuai,
>>>>>
>>>>> thank you very much for your reply. I have opened a JIRA issue on
>>>>> this: https://issues.apache.org/jira/browse/MESOS-4249
>>>>> I hope it will be patched soon :)
>>>>>
>>>>> Best regards,
>>>>> Marica
>>>>>
>>>>>
>>>>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <ti...@mesosphere.io>
>>>>> ha scritto:
>>>>>
>>>>> Hi Marica/Shuai,
>>>>>
>>>>> Sorry haven't been able to spend the time to repro, but looks like
>>>>> Shuai confirmed it.
>>>>>
>>>>> Can one of you file a JIRA?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Tim
>>>>>
>>>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Marica,
>>>>>>
>>>>>> I can reproduce the problem exactly as you described in the first
>>>>>> email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment
>>>>>> variable set, the fetcher works just fine; With it, the fetcher steps seems
>>>>>> skipped. This looks like a bug to me.
>>>>>>
>>>>>> Regards,
>>>>>> Shuai
>>>>>>
>>>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <
>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I have not solved this issue yet. Please, can anyone run the same
>>>>>>> test and let me know if the fetcher is correctly invoked?
>>>>>>> The test is really simple, just try to start a dockerized app (see
>>>>>>> json definition file below) through marathon on a mesos slave running in a
>>>>>>> docker container started with the option —docker_mesos_image=<mesos slave
>>>>>>> image>.
>>>>>>> I would appreciate very much any feedback.
>>>>>>>
>>>>>>> Sample Marathon app:
>>>>>>> {
>>>>>>> "id": "test-app",
>>>>>>> "container": {
>>>>>>> "type": "DOCKER",
>>>>>>> "docker": {
>>>>>>> "image": "libmesos/ubuntu"
>>>>>>> }
>>>>>>> },
>>>>>>> "cpus": 1,
>>>>>>> "mem": 512,
>>>>>>> "uris": [ "
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"
>>>>>>> ],
>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u
>>>>>>> +%T; done"
>>>>>>> }
>>>>>>>
>>>>>>> Docker run command to start dockerized mesos slave:
>>>>>>>
>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e
>>>>>>> MESOS_CONTAINERIZERS=docker,mesos
>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos
>>>>>>> --name slave --net host --privileged --pid host mesos-slave
>>>>>>>
>>>>>>> Thank you very much in advance!
>>>>>>> Best regards,
>>>>>>> Marica
>>>>>>>
>>>>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <
>>>>>>> marica.antonacci@ba.infn.it> ha scritto:
>>>>>>>
>>>>>>> Dear Tim,
>>>>>>>
>>>>>>> I have collected some information from my test environment, starting
>>>>>>> the slave container with and without the —docker_mesos_image startup flag.
>>>>>>> Please let me know if you need further input. Thank you very much for your
>>>>>>> support!
>>>>>>>
>>>>>>> Using the flag —docker_mesos_image:
>>>>>>>
>>>>>>> root@mesos-slave:~# docker ps
>>>>>>> CONTAINER ID IMAGE COMMAND
>>>>>>> CREATED STATUS PORTS NAMES
>>>>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2
>>>>>>> minutes ago Up 2 minutes
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2
>>>>>>> minutes ago Up 2 minutes
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3
>>>>>>> minutes ago Up 3 minutes slave
>>>>>>>
>>>>>>> *root@mesos-slave:~# docker logs slave*
>>>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown
>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>>>> to user 'root'
>>>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1;
>>>>>>> mem(*):32 in work directory
>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container
>>>>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework
>>>>>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>>>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590
>>>>>>> to
>>>>>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>>>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for
>>>>>>> executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>>>> 90.147.170.246:39381
>>>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating
>>>>>>> container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to
>>>>>>> update is identical to existing resources
>>>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@
>>>>>>> 90.147.170.246:39381
>>>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update
>>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>>>> 90.147.170.246:39381
>>>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received
>>>>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>>>>>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826]
>>>>>>> Checkpointing UPDATE for status update TASK_RUNNING (UUID:
>>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update
>>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209
>>>>>>> :5050
>>>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement
>>>>>>> for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@
>>>>>>> 90.147.170.246:39381
>>>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received
>>>>>>> status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826]
>>>>>>> Checkpointing ACK for status update TASK_RUNNING (UUID:
>>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>>
>>>>>>> root@mesos-slave:~# *docker logs
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor*
>>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>> --stop_timeout="0ns"
>>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>> --stop_timeout="0ns"
>>>>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>>>>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on
>>>>>>> slave db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>>>> Registered docker executor on 90.147.170.246
>>>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>>>
>>>>>>> root@mesos-slave:~# *docker exec -it
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>>> bash*
>>>>>>> root@mesos-slave:/# ls -R /tmp/
>>>>>>> hsperfdata_root/ mesos/
>>>>>>> root@mesos-slave:/# ls -R /tmp/mesos/
>>>>>>> /tmp/mesos/:
>>>>>>> *slaves*
>>>>>>>
>>>>>>> /tmp/mesos/slaves:
>>>>>>> *db70e09f-f39d-491c-8480-73d9858c140b-S0*
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>>>>>>> *frameworks*
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>>>>>>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>>>>>>> *executors*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>>>>>>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>>>>>>> *runs*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>>>>>>> *d965f59b-cc1a-4081-95d2-f3370214c84d*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>>>>>>> stderr stdout
>>>>>>>
>>>>>>>
>>>>>>> root@mesos-slave:~# *docker exec -it
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>> bash*
>>>>>>> root@mesos-slave:/# env
>>>>>>> HOSTNAME=mesos-slave
>>>>>>> HOST=90.147.170.246
>>>>>>> PORT0=31220
>>>>>>> PORT_10000=31220
>>>>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>>>>> PWD=/
>>>>>>> PORTS=31220
>>>>>>>
>>>>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>> SHLVL=1
>>>>>>> HOME=/
>>>>>>> MARATHON_APP_ID=/test-app
>>>>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>>>>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>>>>>>> PORT=31220
>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>> _=/usr/bin/env
>>>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>>>>>>> /mnt/mesos/sandbox:
>>>>>>> stderr stdout
>>>>>>>
>>>>>>>
>>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>>> */var/log/mesos-slave.INFO*
>>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>>>>>>
>>>>>>> /var/log/mesos:
>>>>>>>
>>>>>>>
>>>>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is
>>>>>>> created, the file is downloaded in the sandbox:
>>>>>>>
>>>>>>> root@mesos-slave:~# docker exec -it slave bash
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>>> */var/log/mesos-fetcher.INFO*
>>>>>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157
>>>>>>> */var/log/mesos-slave.INFO*
>>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>>>>>>
>>>>>>> /var/log/mesos:
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>>>>>>> Log file created at: 2015/12/19 18:16:21
>>>>>>> Running on machine: mesos-slave
>>>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging
>>>>>>> started!
>>>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info:
>>>>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/
>>>>>>> www.stat.cmu.edu
>>>>>>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>>>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI '
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into
>>>>>>> the sandbox directory
>>>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI '
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource
>>>>>>> from '
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>> to
>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched '
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>> to
>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>>> root@mesos-slave:/#
>>>>>>>
>>>>>>> root@mesos-slave:~# docker exec -it
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4
>>>>>>> bash
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>>>>>>> snoqualmie.csv stderr stdout
>>>>>>> root@mesos-slave:/#
>>>>>>>
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io>
>>>>>>> ha scritto:
>>>>>>>
>>>>>>> HI Marica,
>>>>>>>
>>>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't
>>>>>>> seem possible we don't pass down the URI flags and if the fetcher failed
>>>>>>> the container launch should have failed too.
>>>>>>>
>>>>>>> Also another possible situation is that web UI not really showing
>>>>>>> the exact content of the sandbox, can you actually go into the directory
>>>>>>> and see if the files are there?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <
>>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>>
>>>>>>>> Hi Tim,
>>>>>>>>
>>>>>>>> looking at the sandbox I can see only the stderr and stout file
>>>>>>>> (see the attached screenshot). If I remove —docker_mesos_image (and
>>>>>>>> therefore the executor is run inside the slave container) the file
>>>>>>>> specified in the URI field is shown in the sandbox.
>>>>>>>> Did you verify that the fetcher is called when using the
>>>>>>>> —docker_mesos_image flag ?
>>>>>>>>
>>>>>>>> Thanks a lot for your feedback
>>>>>>>> Marica
>>>>>>>>
>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>
>>>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io>
>>>>>>>> ha scritto:
>>>>>>>>
>>>>>>>> Hi Marica,
>>>>>>>>
>>>>>>>> It should work as we fetch all the files before we launch the
>>>>>>>> executor and place them in the sandbox, and we mount the sandbox into that
>>>>>>>> container as well.
>>>>>>>>
>>>>>>>> How did you verify that the file is not downloaded?
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
>>>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>>>
>>>>>>>>> Hi Grzegorz,
>>>>>>>>>
>>>>>>>>> I’m using this command line for docker run
>>>>>>>>>
>>>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP>
>>>>>>>>> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos -e
>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
>>>>>>>>> slave --net host --privileged --pid host mesos-slave
>>>>>>>>>
>>>>>>>>> where mesos-slave is the image built from the docker file in this
>>>>>>>>> repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>>>>>>
>>>>>>>>> I have tested successfully the deployment of dockerized
>>>>>>>>> applications through Marathon and dockerized jobs through Chronos and also
>>>>>>>>> the recovery seems to work fine with the flag docker_mesos_image. What is
>>>>>>>>> not working for me is the fetcher: it seems that when the executor is
>>>>>>>>> launched as separate container (thanks to the flag docker_mesos_image) the
>>>>>>>>> information about the URIs to be downloaded is lost…I hope someone can help
>>>>>>>>> to understand if this a bug or I’ m missing something.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
>>>>>>>>> gregory90@gmail.com> ha scritto:
>>>>>>>>>
>>>>>>>>> I've tried to use this flag, but cannot really run any container
>>>>>>>>> when this flag is set.
>>>>>>>>> I've raised this issue here:
>>>>>>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>>>>>>>>> here:
>>>>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
>>>>>>>>> sadly no one was able to help me...
>>>>>>>>>
>>>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>>>>>>>>> marica.antonacci@ba.infn.it> napisał:
>>>>>>>>>
>>>>>>>>>> OK, the problem I spotted is related to the usage of the
>>>>>>>>>> flag —docker_mesos_image that allows the executor to
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this
>>>>>>>>>> mesos slave instance. If an image is specified, the docker containerizer
>>>>>>>>>> assumes the slave is running in a docker container, and launches executors
>>>>>>>>>> with docker containers in order to recover them when the slave restarts and
>>>>>>>>>> recovers.
>>>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>>>>>>
>>>>>>>>>> Thank you
>>>>>>>>>> Marica
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <
>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>> no docker_mesos_image flag in my docker run, and the docker
>>>>>>>>>> image is build by myself.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <
>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>
>>>>>>>>>> Yes, I did check inside the container and the csv file was not
>>>>>>>>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>>>>>>>>
>>>>>>>>>>> Are you running your slave with the --docker_mesos_image flag?
>>>>>>>>>>> Can you please provide me the docker run command you are using to run your
>>>>>>>>>>> dockerized slave?
>>>>>>>>>>>
>>>>>>>>>>> Thank you very much
>>>>>>>>>>>
>>>>>>>>>> Marica
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <
>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>
>>>>>>>>>>> use your test-app json, i can run it correctly, the csv is
>>>>>>>>>>> truely download by mesos slave. please check mesos-master:5050 to check the
>>>>>>>>>>> task detail download files.
>>>>>>>>>>>
>>>>>>>>>>> you describe the app container why not found the csv, because
>>>>>>>>>>> the csv is download in slave container's folder, not in app container. so
>>>>>>>>>>> if you run
>>>>>>>>>>>
>>>>>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>>>>>
>>>>>>>>>>> the folder in app container is default value:
>>>>>>>>>>>
>>>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>>>>> but in real world, the sandbox is in slave container, not in app
>>>>>>>>>>> container.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you very much,
>>>>>>>>>>>>
>>>>>>>>>>>> I’m using a sample application definition file, just for
>>>>>>>>>>>> testing purpose:
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>> "id": "test-app",
>>>>>>>>>>>> "container": {
>>>>>>>>>>>> "type": "DOCKER",
>>>>>>>>>>>> "docker": {
>>>>>>>>>>>> "image": "libmesos/ubuntu"
>>>>>>>>>>>> }
>>>>>>>>>>>> },
>>>>>>>>>>>> "cpus": 1,
>>>>>>>>>>>> "mem": 512,
>>>>>>>>>>>> *"uris": [
>>>>>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>>>>>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>>>>>>>>> ],*
>>>>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date
>>>>>>>>>>>> -u +%T; done"
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> Here is the docker run command line:
>>>>>>>>>>>>
>>>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e
>>>>>>>>>>>> MESOS_IP=<SLAVE_IP> -e
>>>>>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>>>>>>> -e
>>>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>>>>>> MESOS_docker_mesos_image=mesos-slave
>>>>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>>>>>>>>> --privileged --pid host mesos-slave
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> As already mentioned, if I remove the environment variable
>>>>>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>>>>>>>>> snoqualmie.csv inside the sandbox.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you again! I’m looking forward to hearing about your
>>>>>>>>>>>> outcomes.
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Marica
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <
>>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>>
>>>>>>>>>>>> Could you please give a sample marathon json, i can test it
>>>>>>>>>>>> asap.
>>>>>>>>>>>>
>>>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>>
>>>>>>>>>>>>> No, using the socket:
>>>>>>>>>>>>>
>>>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <
>>>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>>>
>>>>>>>>>>>>> docker in docker mode?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>>>>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>>>>>>>>>> inside a docker container (using *docker_mesos_image*
>>>>>>>>>>>>>> startup flag) and you submit the deployment of a dockerized application or
>>>>>>>>>>>>>> job the fetcher step is not performed. On the other hand, if I request the
>>>>>>>>>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>>>>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>>>>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Therefore, it seems that the information about the URIs gets
>>>>>>>>>>>>>> lost when the dockerized mesos slave spawns the executor docker container
>>>>>>>>>>>>>> that in turn launches the application docker container…Has anyone seen this
>>>>>>>>>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Marica
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>>>>>>>>> 2015-12-18 下午4.55.38.png>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Deshi Xiao
>>>>>>>>>> Twitter: xds2000
>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>> Via Orabona 4
>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>>
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Mesos fetcher in dockerized slave
Posted by Shuai Lin <li...@gmail.com>.
The testing of this case requires to build a docker image for mesos-slave,
so it seems not practical to add a test case for it in the mesos tests.
Anyway, here is the scripts I use for testing this issue:
https://gist.github.com/lins05/14455e92f37e91fd46ff
On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin <li...@gmail.com> wrote:
> Hi Tim,
>
> The review is here: https://reviews.apache.org/r/42390/ , would you
> please take a look?
>
> Regards,
> Shuai
>
> On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin <li...@gmail.com> wrote:
>
>> Hi Maria and Tim,
>>
>> I'm setting up a test case for this scenario that would fail, after which
>> I'll begin on fixing it.
>>
>> Is it feasible to include the fixing in the new release?
>>
>>
>> I'm not sure, does 0.27 have an estimated release date now?
>>
>>
>> Regards,
>> Shuai
>>
>> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <ti...@mesosphere.io> wrote:
>>
>>> I can shepherd no problem.
>>>
>>> Tim
>>>
>>> On Dec 25, 2015, at 4:32 PM, Shuai Lin <li...@gmail.com> wrote:
>>>
>>> I'll work on it. @Tim could you shepherd it?
>>>
>>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci <
>>> marica.antonacci@ba.infn.it> wrote:
>>>
>>>> Hi Tim and Shuai,
>>>>
>>>> thank you very much for your reply. I have opened a JIRA issue on this:
>>>> https://issues.apache.org/jira/browse/MESOS-4249
>>>> I hope it will be patched soon :)
>>>>
>>>> Best regards,
>>>> Marica
>>>>
>>>>
>>>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <ti...@mesosphere.io> ha
>>>> scritto:
>>>>
>>>> Hi Marica/Shuai,
>>>>
>>>> Sorry haven't been able to spend the time to repro, but looks like
>>>> Shuai confirmed it.
>>>>
>>>> Can one of you file a JIRA?
>>>>
>>>> Thanks!
>>>>
>>>> Tim
>>>>
>>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Marica,
>>>>>
>>>>> I can reproduce the problem exactly as you described in the first
>>>>> email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment
>>>>> variable set, the fetcher works just fine; With it, the fetcher steps seems
>>>>> skipped. This looks like a bug to me.
>>>>>
>>>>> Regards,
>>>>> Shuai
>>>>>
>>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I have not solved this issue yet. Please, can anyone run the same
>>>>>> test and let me know if the fetcher is correctly invoked?
>>>>>> The test is really simple, just try to start a dockerized app (see
>>>>>> json definition file below) through marathon on a mesos slave running in a
>>>>>> docker container started with the option —docker_mesos_image=<mesos slave
>>>>>> image>.
>>>>>> I would appreciate very much any feedback.
>>>>>>
>>>>>> Sample Marathon app:
>>>>>> {
>>>>>> "id": "test-app",
>>>>>> "container": {
>>>>>> "type": "DOCKER",
>>>>>> "docker": {
>>>>>> "image": "libmesos/ubuntu"
>>>>>> }
>>>>>> },
>>>>>> "cpus": 1,
>>>>>> "mem": 512,
>>>>>> "uris": [ "
>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"
>>>>>> ],
>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>>>>>> done"
>>>>>> }
>>>>>>
>>>>>> Docker run command to start dockerized mesos slave:
>>>>>>
>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e
>>>>>> MESOS_CONTAINERIZERS=docker,mesos
>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos
>>>>>> --name slave --net host --privileged --pid host mesos-slave
>>>>>>
>>>>>> Thank you very much in advance!
>>>>>> Best regards,
>>>>>> Marica
>>>>>>
>>>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <
>>>>>> marica.antonacci@ba.infn.it> ha scritto:
>>>>>>
>>>>>> Dear Tim,
>>>>>>
>>>>>> I have collected some information from my test environment, starting
>>>>>> the slave container with and without the —docker_mesos_image startup flag.
>>>>>> Please let me know if you need further input. Thank you very much for your
>>>>>> support!
>>>>>>
>>>>>> Using the flag —docker_mesos_image:
>>>>>>
>>>>>> root@mesos-slave:~# docker ps
>>>>>> CONTAINER ID IMAGE COMMAND
>>>>>> CREATED STATUS PORTS NAMES
>>>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2
>>>>>> minutes ago Up 2 minutes
>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2
>>>>>> minutes ago Up 2 minutes
>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3
>>>>>> minutes ago Up 3 minutes slave
>>>>>>
>>>>>> *root@mesos-slave:~# docker logs slave*
>>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task
>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task
>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown
>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>>> to user 'root'
>>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor
>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1;
>>>>>> mem(*):32 in work directory
>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task
>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor
>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container
>>>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task
>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor
>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework
>>>>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590
>>>>>> to
>>>>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for
>>>>>> executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>>> 90.147.170.246:39381
>>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating
>>>>>> container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to
>>>>>> update is identical to existing resources
>>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task
>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor
>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@
>>>>>> 90.147.170.246:39381
>>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update
>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>>> 90.147.170.246:39381
>>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received
>>>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>>>>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826]
>>>>>> Checkpointing UPDATE for status update TASK_RUNNING (UUID:
>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update
>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209
>>>>>> :5050
>>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement
>>>>>> for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@
>>>>>> 90.147.170.246:39381
>>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received
>>>>>> status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826]
>>>>>> Checkpointing ACK for status update TASK_RUNNING (UUID:
>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>
>>>>>> root@mesos-slave:~# *docker logs
>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor*
>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>> --stop_timeout="0ns"
>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>> --stop_timeout="0ns"
>>>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>>>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on
>>>>>> slave db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>>> Registered docker executor on 90.147.170.246
>>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>>
>>>>>> root@mesos-slave:~# *docker exec -it
>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>> bash*
>>>>>> root@mesos-slave:/# ls -R /tmp/
>>>>>> hsperfdata_root/ mesos/
>>>>>> root@mesos-slave:/# ls -R /tmp/mesos/
>>>>>> /tmp/mesos/:
>>>>>> *slaves*
>>>>>>
>>>>>> /tmp/mesos/slaves:
>>>>>> *db70e09f-f39d-491c-8480-73d9858c140b-S0*
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>>>>>> *frameworks*
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>>>>>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000*
>>>>>>
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>>>>>> *executors*
>>>>>>
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>>>>>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0*
>>>>>>
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>>>>>> *runs*
>>>>>>
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>>>>>> *d965f59b-cc1a-4081-95d2-f3370214c84d*
>>>>>>
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>>>>>> stderr stdout
>>>>>>
>>>>>>
>>>>>> root@mesos-slave:~# *docker exec -it
>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>> bash*
>>>>>> root@mesos-slave:/# env
>>>>>> HOSTNAME=mesos-slave
>>>>>> HOST=90.147.170.246
>>>>>> PORT0=31220
>>>>>> PORT_10000=31220
>>>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>>>> PWD=/
>>>>>> PORTS=31220
>>>>>>
>>>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>> SHLVL=1
>>>>>> HOME=/
>>>>>> MARATHON_APP_ID=/test-app
>>>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>>>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>>>>>> PORT=31220
>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>> _=/usr/bin/env
>>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>>>>>> /mnt/mesos/sandbox:
>>>>>> stderr stdout
>>>>>>
>>>>>>
>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>> */var/log/mesos-slave.INFO*
>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>>>>>
>>>>>> /var/log/mesos:
>>>>>>
>>>>>>
>>>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is
>>>>>> created, the file is downloaded in the sandbox:
>>>>>>
>>>>>> root@mesos-slave:~# docker exec -it slave bash
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>> */var/log/mesos-fetcher.INFO*
>>>>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157
>>>>>> */var/log/mesos-slave.INFO*
>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>>>>>
>>>>>> /var/log/mesos:
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>>>>>> Log file created at: 2015/12/19 18:16:21
>>>>>> Running on machine: mesos-slave
>>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging
>>>>>> started!
>>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info:
>>>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/
>>>>>> www.stat.cmu.edu
>>>>>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI '
>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into
>>>>>> the sandbox directory
>>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI '
>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource
>>>>>> from '
>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>> to
>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched '
>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>> to
>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>> root@mesos-slave:/#
>>>>>>
>>>>>> root@mesos-slave:~# docker exec -it
>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4
>>>>>> bash
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>>>>>> snoqualmie.csv stderr stdout
>>>>>> root@mesos-slave:/#
>>>>>>
>>>>>>
>>>>>> Best Regards,
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>>
>>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io>
>>>>>> ha scritto:
>>>>>>
>>>>>> HI Marica,
>>>>>>
>>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't
>>>>>> seem possible we don't pass down the URI flags and if the fetcher failed
>>>>>> the container launch should have failed too.
>>>>>>
>>>>>> Also another possible situation is that web UI not really showing the
>>>>>> exact content of the sandbox, can you actually go into the directory and
>>>>>> see if the files are there?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <
>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>
>>>>>>> Hi Tim,
>>>>>>>
>>>>>>> looking at the sandbox I can see only the stderr and stout file (see
>>>>>>> the attached screenshot). If I remove —docker_mesos_image (and therefore
>>>>>>> the executor is run inside the slave container) the file specified in the
>>>>>>> URI field is shown in the sandbox.
>>>>>>> Did you verify that the fetcher is called when using the
>>>>>>> —docker_mesos_image flag ?
>>>>>>>
>>>>>>> Thanks a lot for your feedback
>>>>>>> Marica
>>>>>>>
>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>
>>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io>
>>>>>>> ha scritto:
>>>>>>>
>>>>>>> Hi Marica,
>>>>>>>
>>>>>>> It should work as we fetch all the files before we launch the
>>>>>>> executor and place them in the sandbox, and we mount the sandbox into that
>>>>>>> container as well.
>>>>>>>
>>>>>>> How did you verify that the file is not downloaded?
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
>>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>>
>>>>>>>> Hi Grzegorz,
>>>>>>>>
>>>>>>>> I’m using this command line for docker run
>>>>>>>>
>>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos -e
>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
>>>>>>>> slave --net host --privileged --pid host mesos-slave
>>>>>>>>
>>>>>>>> where mesos-slave is the image built from the docker file in this
>>>>>>>> repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>>>>>
>>>>>>>> I have tested successfully the deployment of dockerized
>>>>>>>> applications through Marathon and dockerized jobs through Chronos and also
>>>>>>>> the recovery seems to work fine with the flag docker_mesos_image. What is
>>>>>>>> not working for me is the fetcher: it seems that when the executor is
>>>>>>>> launched as separate container (thanks to the flag docker_mesos_image) the
>>>>>>>> information about the URIs to be downloaded is lost…I hope someone can help
>>>>>>>> to understand if this a bug or I’ m missing something.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Marica
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
>>>>>>>> gregory90@gmail.com> ha scritto:
>>>>>>>>
>>>>>>>> I've tried to use this flag, but cannot really run any container
>>>>>>>> when this flag is set.
>>>>>>>> I've raised this issue here:
>>>>>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>>>>>>>> here:
>>>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
>>>>>>>> sadly no one was able to help me...
>>>>>>>>
>>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>>>>>>>> marica.antonacci@ba.infn.it> napisał:
>>>>>>>>
>>>>>>>>> OK, the problem I spotted is related to the usage of the
>>>>>>>>> flag —docker_mesos_image that allows the executor to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this
>>>>>>>>> mesos slave instance. If an image is specified, the docker containerizer
>>>>>>>>> assumes the slave is running in a docker container, and launches executors
>>>>>>>>> with docker containers in order to recover them when the slave restarts and
>>>>>>>>> recovers.
>>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>>>>>
>>>>>>>>> Thank you
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <
>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>
>>>>>>>>> no docker_mesos_image flag in my docker run, and the docker
>>>>>>>>> image is build by myself.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <
>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>
>>>>>>>>> Yes, I did check inside the container and the csv file was not
>>>>>>>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>>>>>>>
>>>>>>>>>> Are you running your slave with the --docker_mesos_image flag?
>>>>>>>>>> Can you please provide me the docker run command you are using to run your
>>>>>>>>>> dockerized slave?
>>>>>>>>>>
>>>>>>>>>> Thank you very much
>>>>>>>>>>
>>>>>>>>> Marica
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <
>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>> Hi Marica,
>>>>>>>>>>
>>>>>>>>>> use your test-app json, i can run it correctly, the csv is truely
>>>>>>>>>> download by mesos slave. please check mesos-master:5050 to check the task
>>>>>>>>>> detail download files.
>>>>>>>>>>
>>>>>>>>>> you describe the app container why not found the csv, because the
>>>>>>>>>> csv is download in slave container's folder, not in app container. so if
>>>>>>>>>> you run
>>>>>>>>>>
>>>>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>>>>
>>>>>>>>>> the folder in app container is default value:
>>>>>>>>>>
>>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>>>> but in real world, the sandbox is in slave container, not in app
>>>>>>>>>> container.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>
>>>>>>>>>>> Thank you very much,
>>>>>>>>>>>
>>>>>>>>>>> I’m using a sample application definition file, just for testing
>>>>>>>>>>> purpose:
>>>>>>>>>>>
>>>>>>>>>>> {
>>>>>>>>>>> "id": "test-app",
>>>>>>>>>>> "container": {
>>>>>>>>>>> "type": "DOCKER",
>>>>>>>>>>> "docker": {
>>>>>>>>>>> "image": "libmesos/ubuntu"
>>>>>>>>>>> }
>>>>>>>>>>> },
>>>>>>>>>>> "cpus": 1,
>>>>>>>>>>> "mem": 512,
>>>>>>>>>>> *"uris": [
>>>>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>>>>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>>>>>>>> ],*
>>>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u
>>>>>>>>>>> +%T; done"
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> Here is the docker run command line:
>>>>>>>>>>>
>>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e
>>>>>>>>>>> MESOS_IP=<SLAVE_IP> -e
>>>>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>>>>>> -e
>>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>>>>> MESOS_docker_mesos_image=mesos-slave
>>>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>>>>>>>> --privileged --pid host mesos-slave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> As already mentioned, if I remove the environment variable
>>>>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>>>>>>>> snoqualmie.csv inside the sandbox.
>>>>>>>>>>>
>>>>>>>>>>> Thank you again! I’m looking forward to hearing about your
>>>>>>>>>>> outcomes.
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Marica
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <
>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>
>>>>>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>>>>>
>>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>
>>>>>>>>>>>> No, using the socket:
>>>>>>>>>>>>
>>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <
>>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>>
>>>>>>>>>>>> docker in docker mode?
>>>>>>>>>>>>
>>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>>>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>>>>>>>>> inside a docker container (using *docker_mesos_image* startup
>>>>>>>>>>>>> flag) and you submit the deployment of a dockerized application or job the
>>>>>>>>>>>>> fetcher step is not performed. On the other hand, if I request the
>>>>>>>>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>>>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>>>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Therefore, it seems that the information about the URIs gets
>>>>>>>>>>>>> lost when the dockerized mesos slave spawns the executor docker container
>>>>>>>>>>>>> that in turn launches the application docker container…Has anyone seen this
>>>>>>>>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Marica
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Deshi Xiao
>>>>>>>>>> Twitter: xds2000
>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>>>>>>>> 2015-12-18 下午4.55.38.png>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>> Via Orabona 4
>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Deshi Xiao
>>>>>>>>> Twitter: xds2000
>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>>
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Mesos fetcher in dockerized slave
Posted by Shuai Lin <li...@gmail.com>.
Hi Tim,
The review is here: https://reviews.apache.org/r/42390/ , would you please
take a look?
Regards,
Shuai
On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin <li...@gmail.com> wrote:
> Hi Maria and Tim,
>
> I'm setting up a test case for this scenario that would fail, after which
> I'll begin on fixing it.
>
> Is it feasible to include the fixing in the new release?
>
>
> I'm not sure, does 0.27 have an estimated release date now?
>
>
> Regards,
> Shuai
>
> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <ti...@mesosphere.io> wrote:
>
>> I can shepherd no problem.
>>
>> Tim
>>
>> On Dec 25, 2015, at 4:32 PM, Shuai Lin <li...@gmail.com> wrote:
>>
>> I'll work on it. @Tim could you shepherd it?
>>
>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci <
>> marica.antonacci@ba.infn.it> wrote:
>>
>>> Hi Tim and Shuai,
>>>
>>> thank you very much for your reply. I have opened a JIRA issue on this:
>>> https://issues.apache.org/jira/browse/MESOS-4249
>>> I hope it will be patched soon :)
>>>
>>> Best regards,
>>> Marica
>>>
>>>
>>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <ti...@mesosphere.io> ha
>>> scritto:
>>>
>>> Hi Marica/Shuai,
>>>
>>> Sorry haven't been able to spend the time to repro, but looks like Shuai
>>> confirmed it.
>>>
>>> Can one of you file a JIRA?
>>>
>>> Thanks!
>>>
>>> Tim
>>>
>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com>
>>> wrote:
>>>
>>>> Hi Marica,
>>>>
>>>> I can reproduce the problem exactly as you described in the first email
>>>> of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable
>>>> set, the fetcher works just fine; With it, the fetcher steps seems skipped.
>>>> This looks like a bug to me.
>>>>
>>>> Regards,
>>>> Shuai
>>>>
>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <
>>>> marica.antonacci@ba.infn.it> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I have not solved this issue yet. Please, can anyone run the same test
>>>>> and let me know if the fetcher is correctly invoked?
>>>>> The test is really simple, just try to start a dockerized app (see
>>>>> json definition file below) through marathon on a mesos slave running in a
>>>>> docker container started with the option —docker_mesos_image=<mesos slave
>>>>> image>.
>>>>> I would appreciate very much any feedback.
>>>>>
>>>>> Sample Marathon app:
>>>>> {
>>>>> "id": "test-app",
>>>>> "container": {
>>>>> "type": "DOCKER",
>>>>> "docker": {
>>>>> "image": "libmesos/ubuntu"
>>>>> }
>>>>> },
>>>>> "cpus": 1,
>>>>> "mem": 512,
>>>>> "uris": [ "
>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"
>>>>> ],
>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>>>>> done"
>>>>> }
>>>>>
>>>>> Docker run command to start dockerized mesos slave:
>>>>>
>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e
>>>>> MESOS_CONTAINERIZERS=docker,mesos
>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos
>>>>> --name slave --net host --privileged --pid host mesos-slave
>>>>>
>>>>> Thank you very much in advance!
>>>>> Best regards,
>>>>> Marica
>>>>>
>>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it> ha scritto:
>>>>>
>>>>> Dear Tim,
>>>>>
>>>>> I have collected some information from my test environment, starting
>>>>> the slave container with and without the —docker_mesos_image startup flag.
>>>>> Please let me know if you need further input. Thank you very much for your
>>>>> support!
>>>>>
>>>>> Using the flag —docker_mesos_image:
>>>>>
>>>>> root@mesos-slave:~# docker ps
>>>>> CONTAINER ID IMAGE COMMAND
>>>>> CREATED STATUS PORTS NAMES
>>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2
>>>>> minutes ago Up 2 minutes
>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2
>>>>> minutes ago Up 2 minutes
>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3
>>>>> minutes ago Up 3 minutes slave
>>>>>
>>>>> *root@mesos-slave:~# docker logs slave*
>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task
>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task
>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown
>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>> to user 'root'
>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor
>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1;
>>>>> mem(*):32 in work directory
>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task
>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor
>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container
>>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task
>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor
>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework
>>>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to
>>>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for
>>>>> executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>> 90.147.170.246:39381
>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating
>>>>> container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to
>>>>> update is identical to existing resources
>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task
>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor
>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@
>>>>> 90.147.170.246:39381
>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update
>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>> 90.147.170.246:39381
>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received
>>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>>>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826]
>>>>> Checkpointing UPDATE for status update TASK_RUNNING (UUID:
>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update
>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209
>>>>> :5050
>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement
>>>>> for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@
>>>>> 90.147.170.246:39381
>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received
>>>>> status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826]
>>>>> Checkpointing ACK for status update TASK_RUNNING (UUID:
>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>
>>>>> root@mesos-slave:~# *docker logs
>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor*
>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>> --stop_timeout="0ns"
>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>> --stop_timeout="0ns"
>>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave
>>>>> db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>> Registered docker executor on 90.147.170.246
>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>
>>>>> root@mesos-slave:~# *docker exec -it
>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>> bash*
>>>>> root@mesos-slave:/# ls -R /tmp/
>>>>> hsperfdata_root/ mesos/
>>>>> root@mesos-slave:/# ls -R /tmp/mesos/
>>>>> /tmp/mesos/:
>>>>> *slaves*
>>>>>
>>>>> /tmp/mesos/slaves:
>>>>> *db70e09f-f39d-491c-8480-73d9858c140b-S0*
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>>>>> *frameworks*
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>>>>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000*
>>>>>
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>>>>> *executors*
>>>>>
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>>>>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0*
>>>>>
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>>>>> *runs*
>>>>>
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>>>>> *d965f59b-cc1a-4081-95d2-f3370214c84d*
>>>>>
>>>>>
>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>>>>> stderr stdout
>>>>>
>>>>>
>>>>> root@mesos-slave:~# *docker exec -it
>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>> bash*
>>>>> root@mesos-slave:/# env
>>>>> HOSTNAME=mesos-slave
>>>>> HOST=90.147.170.246
>>>>> PORT0=31220
>>>>> PORT_10000=31220
>>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>>> PWD=/
>>>>> PORTS=31220
>>>>>
>>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>> SHLVL=1
>>>>> HOME=/
>>>>> MARATHON_APP_ID=/test-app
>>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>>>>> PORT=31220
>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>> _=/usr/bin/env
>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>>>>> /mnt/mesos/sandbox:
>>>>> stderr stdout
>>>>>
>>>>>
>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>> */var/log/mesos-slave.INFO*
>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>>>>
>>>>> /var/log/mesos:
>>>>>
>>>>>
>>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is
>>>>> created, the file is downloaded in the sandbox:
>>>>>
>>>>> root@mesos-slave:~# docker exec -it slave bash
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>> */var/log/mesos-fetcher.INFO*
>>>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157
>>>>> */var/log/mesos-slave.INFO*
>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>>>>
>>>>> /var/log/mesos:
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>>>>> Log file created at: 2015/12/19 18:16:21
>>>>> Running on machine: mesos-slave
>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging
>>>>> started!
>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info:
>>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/
>>>>> www.stat.cmu.edu
>>>>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI '
>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into
>>>>> the sandbox directory
>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI '
>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from
>>>>> '
>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>> to
>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched '
>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>> to
>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>> root@mesos-slave:/#
>>>>>
>>>>> root@mesos-slave:~# docker exec -it
>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4
>>>>> bash
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/#
>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>>>>> snoqualmie.csv stderr stdout
>>>>> root@mesos-slave:/#
>>>>>
>>>>>
>>>>> Best Regards,
>>>>> Marica
>>>>>
>>>>>
>>>>>
>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io>
>>>>> ha scritto:
>>>>>
>>>>> HI Marica,
>>>>>
>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't
>>>>> seem possible we don't pass down the URI flags and if the fetcher failed
>>>>> the container launch should have failed too.
>>>>>
>>>>> Also another possible situation is that web UI not really showing the
>>>>> exact content of the sandbox, can you actually go into the directory and
>>>>> see if the files are there?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Tim
>>>>>
>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>
>>>>>> Hi Tim,
>>>>>>
>>>>>> looking at the sandbox I can see only the stderr and stout file (see
>>>>>> the attached screenshot). If I remove —docker_mesos_image (and therefore
>>>>>> the executor is run inside the slave container) the file specified in the
>>>>>> URI field is shown in the sandbox.
>>>>>> Did you verify that the fetcher is called when using the
>>>>>> —docker_mesos_image flag ?
>>>>>>
>>>>>> Thanks a lot for your feedback
>>>>>> Marica
>>>>>>
>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>
>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io>
>>>>>> ha scritto:
>>>>>>
>>>>>> Hi Marica,
>>>>>>
>>>>>> It should work as we fetch all the files before we launch the
>>>>>> executor and place them in the sandbox, and we mount the sandbox into that
>>>>>> container as well.
>>>>>>
>>>>>> How did you verify that the file is not downloaded?
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>
>>>>>>> Hi Grzegorz,
>>>>>>>
>>>>>>> I’m using this command line for docker run
>>>>>>>
>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos -e
>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
>>>>>>> slave --net host --privileged --pid host mesos-slave
>>>>>>>
>>>>>>> where mesos-slave is the image built from the docker file in this
>>>>>>> repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>>>>
>>>>>>> I have tested successfully the deployment of dockerized applications
>>>>>>> through Marathon and dockerized jobs through Chronos and also the recovery
>>>>>>> seems to work fine with the flag docker_mesos_image. What is not working
>>>>>>> for me is the fetcher: it seems that when the executor is launched as
>>>>>>> separate container (thanks to the flag docker_mesos_image) the information
>>>>>>> about the URIs to be downloaded is lost…I hope someone can help to
>>>>>>> understand if this a bug or I’ m missing something.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
>>>>>>> gregory90@gmail.com> ha scritto:
>>>>>>>
>>>>>>> I've tried to use this flag, but cannot really run any container
>>>>>>> when this flag is set.
>>>>>>> I've raised this issue here:
>>>>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>>>>>>> here:
>>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
>>>>>>> sadly no one was able to help me...
>>>>>>>
>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>>>>>>> marica.antonacci@ba.infn.it> napisał:
>>>>>>>
>>>>>>>> OK, the problem I spotted is related to the usage of the
>>>>>>>> flag —docker_mesos_image that allows the executor to
>>>>>>>>
>>>>>>>>
>>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this
>>>>>>>> mesos slave instance. If an image is specified, the docker containerizer
>>>>>>>> assumes the slave is running in a docker container, and launches executors
>>>>>>>> with docker containers in order to recover them when the slave restarts and
>>>>>>>> recovers.
>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>>>>
>>>>>>>> Thank you
>>>>>>>> Marica
>>>>>>>>
>>>>>>>>
>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com>
>>>>>>>> ha scritto:
>>>>>>>>
>>>>>>>> no docker_mesos_image flag in my docker run, and the docker image
>>>>>>>> is build by myself.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <
>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>
>>>>>>>> Yes, I did check inside the container and the csv file was not
>>>>>>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>>>>>>
>>>>>>>>> Are you running your slave with the --docker_mesos_image flag? Can
>>>>>>>>> you please provide me the docker run command you are using to run your
>>>>>>>>> dockerized slave?
>>>>>>>>>
>>>>>>>>> Thank you very much
>>>>>>>>>
>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <
>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>
>>>>>>>>> Hi Marica,
>>>>>>>>>
>>>>>>>>> use your test-app json, i can run it correctly, the csv is truely
>>>>>>>>> download by mesos slave. please check mesos-master:5050 to check the task
>>>>>>>>> detail download files.
>>>>>>>>>
>>>>>>>>> you describe the app container why not found the csv, because the
>>>>>>>>> csv is download in slave container's folder, not in app container. so if
>>>>>>>>> you run
>>>>>>>>>
>>>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>>>
>>>>>>>>> the folder in app container is default value:
>>>>>>>>>
>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>>> but in real world, the sandbox is in slave container, not in app
>>>>>>>>> container.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>
>>>>>>>>>> Thank you very much,
>>>>>>>>>>
>>>>>>>>>> I’m using a sample application definition file, just for testing
>>>>>>>>>> purpose:
>>>>>>>>>>
>>>>>>>>>> {
>>>>>>>>>> "id": "test-app",
>>>>>>>>>> "container": {
>>>>>>>>>> "type": "DOCKER",
>>>>>>>>>> "docker": {
>>>>>>>>>> "image": "libmesos/ubuntu"
>>>>>>>>>> }
>>>>>>>>>> },
>>>>>>>>>> "cpus": 1,
>>>>>>>>>> "mem": 512,
>>>>>>>>>> *"uris": [
>>>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>>>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>>>>>>> ],*
>>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u
>>>>>>>>>> +%T; done"
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Here is the docker run command line:
>>>>>>>>>>
>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e
>>>>>>>>>> MESOS_IP=<SLAVE_IP> -e
>>>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>>>>> -e
>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>>>> MESOS_docker_mesos_image=mesos-slave
>>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>>>>>>> --privileged --pid host mesos-slave
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> As already mentioned, if I remove the environment variable
>>>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>>>>>>> snoqualmie.csv inside the sandbox.
>>>>>>>>>>
>>>>>>>>>> Thank you again! I’m looking forward to hearing about your
>>>>>>>>>> outcomes.
>>>>>>>>>> Best regards,
>>>>>>>>>> Marica
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <
>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>> Hi Marica,
>>>>>>>>>>
>>>>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>>>>
>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>
>>>>>>>>>>> No, using the socket:
>>>>>>>>>>>
>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <
>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>> docker in docker mode?
>>>>>>>>>>>
>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>
>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>>>>>>>> inside a docker container (using *docker_mesos_image* startup
>>>>>>>>>>>> flag) and you submit the deployment of a dockerized application or job the
>>>>>>>>>>>> fetcher step is not performed. On the other hand, if I request the
>>>>>>>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>>>>>>
>>>>>>>>>>>> Therefore, it seems that the information about the URIs gets
>>>>>>>>>>>> lost when the dockerized mesos slave spawns the executor docker container
>>>>>>>>>>>> that in turn launches the application docker container…Has anyone seen this
>>>>>>>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Marica
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Deshi Xiao
>>>>>>>>>> Twitter: xds2000
>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>> Via Orabona 4
>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Deshi Xiao
>>>>>>>>> Twitter: xds2000
>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>>>>>>> 2015-12-18 下午4.55.38.png>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Deshi Xiao
>>>>>>>> Twitter: xds2000
>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>>
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> ----------------------------------------------------------
>>>
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: Mesos fetcher in dockerized slave
Posted by Shuai Lin <li...@gmail.com>.
Hi Maria and Tim,
I'm setting up a test case for this scenario that would fail, after which
I'll begin on fixing it.
Is it feasible to include the fixing in the new release?
I'm not sure, does 0.27 have an estimated release date now?
Regards,
Shuai
On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <ti...@mesosphere.io> wrote:
> I can shepherd no problem.
>
> Tim
>
> On Dec 25, 2015, at 4:32 PM, Shuai Lin <li...@gmail.com> wrote:
>
> I'll work on it. @Tim could you shepherd it?
>
> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci <
> marica.antonacci@ba.infn.it> wrote:
>
>> Hi Tim and Shuai,
>>
>> thank you very much for your reply. I have opened a JIRA issue on this:
>> https://issues.apache.org/jira/browse/MESOS-4249
>> I hope it will be patched soon :)
>>
>> Best regards,
>> Marica
>>
>>
>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <ti...@mesosphere.io> ha
>> scritto:
>>
>> Hi Marica/Shuai,
>>
>> Sorry haven't been able to spend the time to repro, but looks like Shuai
>> confirmed it.
>>
>> Can one of you file a JIRA?
>>
>> Thanks!
>>
>> Tim
>>
>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com>
>> wrote:
>>
>>> Hi Marica,
>>>
>>> I can reproduce the problem exactly as you described in the first email
>>> of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable
>>> set, the fetcher works just fine; With it, the fetcher steps seems skipped.
>>> This looks like a bug to me.
>>>
>>> Regards,
>>> Shuai
>>>
>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <
>>> marica.antonacci@ba.infn.it> wrote:
>>>
>>>> Dear all,
>>>>
>>>> I have not solved this issue yet. Please, can anyone run the same test
>>>> and let me know if the fetcher is correctly invoked?
>>>> The test is really simple, just try to start a dockerized app (see json
>>>> definition file below) through marathon on a mesos slave running in a
>>>> docker container started with the option —docker_mesos_image=<mesos slave
>>>> image>.
>>>> I would appreciate very much any feedback.
>>>>
>>>> Sample Marathon app:
>>>> {
>>>> "id": "test-app",
>>>> "container": {
>>>> "type": "DOCKER",
>>>> "docker": {
>>>> "image": "libmesos/ubuntu"
>>>> }
>>>> },
>>>> "cpus": 1,
>>>> "mem": 512,
>>>> "uris": [ "
>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"
>>>> ],
>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>>>> done"
>>>> }
>>>>
>>>> Docker run command to start dockerized mesos slave:
>>>>
>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e
>>>> MESOS_CONTAINERIZERS=docker,mesos
>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos
>>>> --name slave --net host --privileged --pid host mesos-slave
>>>>
>>>> Thank you very much in advance!
>>>> Best regards,
>>>> Marica
>>>>
>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <
>>>> marica.antonacci@ba.infn.it> ha scritto:
>>>>
>>>> Dear Tim,
>>>>
>>>> I have collected some information from my test environment, starting
>>>> the slave container with and without the —docker_mesos_image startup flag.
>>>> Please let me know if you need further input. Thank you very much for your
>>>> support!
>>>>
>>>> Using the flag —docker_mesos_image:
>>>>
>>>> root@mesos-slave:~# docker ps
>>>> CONTAINER ID IMAGE COMMAND
>>>> CREATED STATUS PORTS NAMES
>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2
>>>> minutes ago Up 2 minutes
>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2
>>>> minutes ago Up 2 minutes
>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3
>>>> minutes ago Up 3 minutes slave
>>>>
>>>> *root@mesos-slave:~# docker logs slave*
>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task
>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task
>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown
>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>> to user 'root'
>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor
>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1;
>>>> mem(*):32 in work directory
>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task
>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor
>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container
>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task
>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor
>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework
>>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to
>>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for
>>>> executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>> 90.147.170.246:39381
>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating
>>>> container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to
>>>> update is identical to existing resources
>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task
>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor
>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@
>>>> 90.147.170.246:39381
>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update
>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>> 90.147.170.246:39381
>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received
>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826]
>>>> Checkpointing UPDATE for status update TASK_RUNNING (UUID:
>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update
>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for
>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@
>>>> 90.147.170.246:39381
>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received
>>>> status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826]
>>>> Checkpointing ACK for status update TASK_RUNNING (UUID:
>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>
>>>> root@mesos-slave:~# *docker logs
>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor*
>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>> --logbufsecs="0" --logging_level="INFO"
>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>> --stop_timeout="0ns"
>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>> --logbufsecs="0" --logging_level="INFO"
>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>> --stop_timeout="0ns"
>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave
>>>> db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>> Registered docker executor on 90.147.170.246
>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>
>>>> root@mesos-slave:~# *docker exec -it
>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>> bash*
>>>> root@mesos-slave:/# ls -R /tmp/
>>>> hsperfdata_root/ mesos/
>>>> root@mesos-slave:/# ls -R /tmp/mesos/
>>>> /tmp/mesos/:
>>>> *slaves*
>>>>
>>>> /tmp/mesos/slaves:
>>>> *db70e09f-f39d-491c-8480-73d9858c140b-S0*
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>>>> *frameworks*
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>>>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000*
>>>>
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>>>> *executors*
>>>>
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>>>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0*
>>>>
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>>>> *runs*
>>>>
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>>>> *d965f59b-cc1a-4081-95d2-f3370214c84d*
>>>>
>>>>
>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>>>> stderr stdout
>>>>
>>>>
>>>> root@mesos-slave:~# *docker exec -it
>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>> bash*
>>>> root@mesos-slave:/# env
>>>> HOSTNAME=mesos-slave
>>>> HOST=90.147.170.246
>>>> PORT0=31220
>>>> PORT_10000=31220
>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>> PWD=/
>>>> PORTS=31220
>>>>
>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>> SHLVL=1
>>>> HOME=/
>>>> MARATHON_APP_ID=/test-app
>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>>>> PORT=31220
>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>> _=/usr/bin/env
>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>>>> /mnt/mesos/sandbox:
>>>> stderr stdout
>>>>
>>>>
>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>> */var/log/mesos-slave.INFO*
>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>>>
>>>> /var/log/mesos:
>>>>
>>>>
>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is
>>>> created, the file is downloaded in the sandbox:
>>>>
>>>> root@mesos-slave:~# docker exec -it slave bash
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>> */var/log/mesos-fetcher.INFO*
>>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157
>>>> */var/log/mesos-slave.INFO*
>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>>>
>>>> /var/log/mesos:
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>>>> Log file created at: 2015/12/19 18:16:21
>>>> Running on machine: mesos-slave
>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started!
>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info:
>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/
>>>> www.stat.cmu.edu
>>>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI '
>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the
>>>> sandbox directory
>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI '
>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from '
>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>> to
>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched '
>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>> to
>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>> root@mesos-slave:/#
>>>>
>>>> root@mesos-slave:~# docker exec -it
>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4
>>>> bash
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/#
>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>>>> snoqualmie.csv stderr stdout
>>>> root@mesos-slave:/#
>>>>
>>>>
>>>> Best Regards,
>>>> Marica
>>>>
>>>>
>>>>
>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io> ha
>>>> scritto:
>>>>
>>>> HI Marica,
>>>>
>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't
>>>> seem possible we don't pass down the URI flags and if the fetcher failed
>>>> the container launch should have failed too.
>>>>
>>>> Also another possible situation is that web UI not really showing the
>>>> exact content of the sandbox, can you actually go into the directory and
>>>> see if the files are there?
>>>>
>>>> Thanks,
>>>>
>>>> Tim
>>>>
>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <
>>>> marica.antonacci@ba.infn.it> wrote:
>>>>
>>>>> Hi Tim,
>>>>>
>>>>> looking at the sandbox I can see only the stderr and stout file (see
>>>>> the attached screenshot). If I remove —docker_mesos_image (and therefore
>>>>> the executor is run inside the slave container) the file specified in the
>>>>> URI field is shown in the sandbox.
>>>>> Did you verify that the fetcher is called when using the
>>>>> —docker_mesos_image flag ?
>>>>>
>>>>> Thanks a lot for your feedback
>>>>> Marica
>>>>>
>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>
>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io>
>>>>> ha scritto:
>>>>>
>>>>> Hi Marica,
>>>>>
>>>>> It should work as we fetch all the files before we launch the executor
>>>>> and place them in the sandbox, and we mount the sandbox into that container
>>>>> as well.
>>>>>
>>>>> How did you verify that the file is not downloaded?
>>>>>
>>>>> Tim
>>>>>
>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>
>>>>>> Hi Grzegorz,
>>>>>>
>>>>>> I’m using this command line for docker run
>>>>>>
>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos -e
>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
>>>>>> slave --net host --privileged --pid host mesos-slave
>>>>>>
>>>>>> where mesos-slave is the image built from the docker file in this
>>>>>> repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>>>
>>>>>> I have tested successfully the deployment of dockerized applications
>>>>>> through Marathon and dockerized jobs through Chronos and also the recovery
>>>>>> seems to work fine with the flag docker_mesos_image. What is not working
>>>>>> for me is the fetcher: it seems that when the executor is launched as
>>>>>> separate container (thanks to the flag docker_mesos_image) the information
>>>>>> about the URIs to be downloaded is lost…I hope someone can help to
>>>>>> understand if this a bug or I’ m missing something.
>>>>>>
>>>>>> Cheers,
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>>
>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
>>>>>> gregory90@gmail.com> ha scritto:
>>>>>>
>>>>>> I've tried to use this flag, but cannot really run any container when
>>>>>> this flag is set.
>>>>>> I've raised this issue here:
>>>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>>>>>> here:
>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
>>>>>> sadly no one was able to help me...
>>>>>>
>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>>>>>> marica.antonacci@ba.infn.it> napisał:
>>>>>>
>>>>>>> OK, the problem I spotted is related to the usage of the
>>>>>>> flag —docker_mesos_image that allows the executor to
>>>>>>>
>>>>>>>
>>>>>>> --docker_mesos_image=VALUEThe docker image used to launch this
>>>>>>> mesos slave instance. If an image is specified, the docker containerizer
>>>>>>> assumes the slave is running in a docker container, and launches executors
>>>>>>> with docker containers in order to recover them when the slave restarts and
>>>>>>> recovers.
>>>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>>>
>>>>>>> Thank you
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com>
>>>>>>> ha scritto:
>>>>>>>
>>>>>>> no docker_mesos_image flag in my docker run, and the docker image
>>>>>>> is build by myself.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <
>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>
>>>>>>> Yes, I did check inside the container and the csv file was not
>>>>>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>>>>>
>>>>>>>> Are you running your slave with the --docker_mesos_image flag? Can
>>>>>>>> you please provide me the docker run command you are using to run your
>>>>>>>> dockerized slave?
>>>>>>>>
>>>>>>>> Thank you very much
>>>>>>>>
>>>>>>> Marica
>>>>>>>>
>>>>>>>>
>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com>
>>>>>>>> ha scritto:
>>>>>>>>
>>>>>>>> Hi Marica,
>>>>>>>>
>>>>>>>> use your test-app json, i can run it correctly, the csv is truely
>>>>>>>> download by mesos slave. please check mesos-master:5050 to check the task
>>>>>>>> detail download files.
>>>>>>>>
>>>>>>>> you describe the app container why not found the csv, because the
>>>>>>>> csv is download in slave container's folder, not in app container. so if
>>>>>>>> you run
>>>>>>>>
>>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>>
>>>>>>>> the folder in app container is default value:
>>>>>>>>
>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>> but in real world, the sandbox is in slave container, not in app
>>>>>>>> container.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>
>>>>>>>>> Thank you very much,
>>>>>>>>>
>>>>>>>>> I’m using a sample application definition file, just for testing
>>>>>>>>> purpose:
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> "id": "test-app",
>>>>>>>>> "container": {
>>>>>>>>> "type": "DOCKER",
>>>>>>>>> "docker": {
>>>>>>>>> "image": "libmesos/ubuntu"
>>>>>>>>> }
>>>>>>>>> },
>>>>>>>>> "cpus": 1,
>>>>>>>>> "mem": 512,
>>>>>>>>> *"uris": [
>>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>>>>>> ],*
>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u
>>>>>>>>> +%T; done"
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Here is the docker run command line:
>>>>>>>>>
>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e
>>>>>>>>> MESOS_IP=<SLAVE_IP> -e
>>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
>>>>>>>>> -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>>>>>> --privileged --pid host mesos-slave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> As already mentioned, if I remove the environment variable
>>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>>>>>> snoqualmie.csv inside the sandbox.
>>>>>>>>>
>>>>>>>>> Thank you again! I’m looking forward to hearing about your
>>>>>>>>> outcomes.
>>>>>>>>> Best regards,
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <
>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>
>>>>>>>>> Hi Marica,
>>>>>>>>>
>>>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>>>
>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>
>>>>>>>>>> No, using the socket:
>>>>>>>>>>
>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <
>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>> docker in docker mode?
>>>>>>>>>>
>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>
>>>>>>>>>>> Dear all,
>>>>>>>>>>>
>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>>>>>>> inside a docker container (using *docker_mesos_image* startup
>>>>>>>>>>> flag) and you submit the deployment of a dockerized application or job the
>>>>>>>>>>> fetcher step is not performed. On the other hand, if I request the
>>>>>>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>>>>>
>>>>>>>>>>> Therefore, it seems that the information about the URIs gets
>>>>>>>>>>> lost when the dockerized mesos slave spawns the executor docker container
>>>>>>>>>>> that in turn launches the application docker container…Has anyone seen this
>>>>>>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Marica
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Deshi Xiao
>>>>>>>>>> Twitter: xds2000
>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>> Via Orabona 4
>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Deshi Xiao
>>>>>>>>> Twitter: xds2000
>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Deshi Xiao
>>>>>>>> Twitter: xds2000
>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>>>>>> 2015-12-18 下午4.55.38.png>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>>
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Deshi Xiao
>>>>>>> Twitter: xds2000
>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: Mesos fetcher in dockerized slave
Posted by Timothy Chen <ti...@mesosphere.io>.
I can shepherd no problem.
Tim
> On Dec 25, 2015, at 4:32 PM, Shuai Lin <li...@gmail.com> wrote:
>
> I'll work on it. @Tim could you shepherd it?
>
>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
>> Hi Tim and Shuai,
>>
>> thank you very much for your reply. I have opened a JIRA issue on this: https://issues.apache.org/jira/browse/MESOS-4249
>> I hope it will be patched soon :)
>>
>> Best regards,
>> Marica
>>
>>
>>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <ti...@mesosphere.io> ha scritto:
>>>
>>> Hi Marica/Shuai,
>>>
>>> Sorry haven't been able to spend the time to repro, but looks like Shuai confirmed it.
>>>
>>> Can one of you file a JIRA?
>>>
>>> Thanks!
>>>
>>> Tim
>>>
>>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com> wrote:
>>>> Hi Marica,
>>>>
>>>> I can reproduce the problem exactly as you described in the first email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set, the fetcher works just fine; With it, the fetcher steps seems skipped. This looks like a bug to me.
>>>>
>>>> Regards,
>>>> Shuai
>>>>
>>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>>>> Dear all,
>>>>>
>>>>> I have not solved this issue yet. Please, can anyone run the same test and let me know if the fetcher is correctly invoked?
>>>>> The test is really simple, just try to start a dockerized app (see json definition file below) through marathon on a mesos slave running in a docker container started with the option —docker_mesos_image=<mesos slave image>.
>>>>> I would appreciate very much any feedback.
>>>>>
>>>>> Sample Marathon app:
>>>>> {
>>>>> "id": "test-app",
>>>>> "container": {
>>>>> "type": "DOCKER",
>>>>> "docker": {
>>>>> "image": "libmesos/ubuntu"
>>>>> }
>>>>> },
>>>>> "cpus": 1,
>>>>> "mem": 512,
>>>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>>>> }
>>>>>
>>>>> Docker run command to start dockerized mesos slave:
>>>>>
>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>>>>>
>>>>> Thank you very much in advance!
>>>>> Best regards,
>>>>> Marica
>>>>>
>>>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <ma...@ba.infn.it> ha scritto:
>>>>>>
>>>>>> Dear Tim,
>>>>>>
>>>>>> I have collected some information from my test environment, starting the slave container with and without the —docker_mesos_image startup flag. Please let me know if you need further input. Thank you very much for your support!
>>>>>>
>>>>>> Using the flag —docker_mesos_image:
>>>>>>
>>>>>> root@mesos-slave:~# docker ps
>>>>>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>>>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 minutes ago Up 3 minutes slave
>>>>>>
>>>>>> root@mesos-slave:~# docker logs slave
>>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user 'root'
>>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
>>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update is identical to existing resources
>>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@90.147.170.246:39381
>>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
>>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
>>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@90.147.170.246:39381
>>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>
>>>>>> root@mesos-slave:~# docker logs mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
>>>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>>>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>>> Registered docker executor on 90.147.170.246
>>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>>
>>>>>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor bash
>>>>>> root@mesos-slave:/# ls -R /tmp/
>>>>>> hsperfdata_root/ mesos/
>>>>>> root@mesos-slave:/# ls -R /tmp/mesos/
>>>>>> /tmp/mesos/:
>>>>>> slaves
>>>>>>
>>>>>> /tmp/mesos/slaves:
>>>>>> db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>>>>>> frameworks
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>>>>>> executors
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>>>>>> runs
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>>>>>> d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>
>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>>>>>> stderr stdout
>>>>>>
>>>>>>
>>>>>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d bash
>>>>>> root@mesos-slave:/# env
>>>>>> HOSTNAME=mesos-slave
>>>>>> HOST=90.147.170.246
>>>>>> PORT0=31220
>>>>>> PORT_10000=31220
>>>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>>>> PWD=/
>>>>>> PORTS=31220
>>>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>> SHLVL=1
>>>>>> HOME=/
>>>>>> MARATHON_APP_ID=/test-app
>>>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>>>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>>>>>> PORT=31220
>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>> _=/usr/bin/env
>>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>>>>>> /mnt/mesos/sandbox:
>>>>>> stderr stdout
>>>>>>
>>>>>>
>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>> /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>>>>>
>>>>>> /var/log/mesos:
>>>>>>
>>>>>>
>>>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is created, the file is downloaded in the sandbox:
>>>>>>
>>>>>> root@mesos-slave:~# docker exec -it slave bash
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>> /var/log/mesos-fetcher.INFO /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157 /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>>>>>
>>>>>> /var/log/mesos:
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>>>>>> Log file created at: 2015/12/19 18:16:21
>>>>>> Running on machine: mesos-slave
>>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started!
>>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the sandbox directory
>>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>> root@mesos-slave:/#
>>>>>>
>>>>>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 bash
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/#
>>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>>>>>> snoqualmie.csv stderr stdout
>>>>>> root@mesos-slave:/#
>>>>>>
>>>>>>
>>>>>> Best Regards,
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io> ha scritto:
>>>>>>>
>>>>>>> HI Marica,
>>>>>>>
>>>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't seem possible we don't pass down the URI flags and if the fetcher failed the container launch should have failed too.
>>>>>>>
>>>>>>> Also another possible situation is that web UI not really showing the exact content of the sandbox, can you actually go into the directory and see if the files are there?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>>>>>>> Hi Tim,
>>>>>>>>
>>>>>>>> looking at the sandbox I can see only the stderr and stout file (see the attached screenshot). If I remove —docker_mesos_image (and therefore the executor is run inside the slave container) the file specified in the URI field is shown in the sandbox.
>>>>>>>> Did you verify that the fetcher is called when using the —docker_mesos_image flag ?
>>>>>>>>
>>>>>>>> Thanks a lot for your feedback
>>>>>>>> Marica
>>>>>>>>
>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>
>>>>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io> ha scritto:
>>>>>>>>>
>>>>>>>>> Hi Marica,
>>>>>>>>>
>>>>>>>>> It should work as we fetch all the files before we launch the executor and place them in the sandbox, and we mount the sandbox into that container as well.
>>>>>>>>>
>>>>>>>>> How did you verify that the file is not downloaded?
>>>>>>>>>
>>>>>>>>> Tim
>>>>>>>>>
>>>>>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>>>>>>>>> Hi Grzegorz,
>>>>>>>>>>
>>>>>>>>>> I’m using this command line for docker run
>>>>>>>>>>
>>>>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>>>>>>>>>>
>>>>>>>>>> where mesos-slave is the image built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>>>>>>>
>>>>>>>>>> I have tested successfully the deployment of dockerized applications through Marathon and dockerized jobs through Chronos and also the recovery seems to work fine with the flag docker_mesos_image. What is not working for me is the fetcher: it seems that when the executor is launched as separate container (thanks to the flag docker_mesos_image) the information about the URIs to be downloaded is lost…I hope someone can help to understand if this a bug or I’ m missing something.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Marica
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gr...@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>> I've tried to use this flag, but cannot really run any container when this flag is set.
>>>>>>>>>>> I've raised this issue here: https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and here: https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but sadly no one was able to help me...
>>>>>>>>>>>
>>>>>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <ma...@ba.infn.it> napisał:
>>>>>>>>>>>> OK, the problem I spotted is related to the usage of the flag —docker_mesos_image that allows the executor to
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this mesos slave instance. If an image is specified, the docker containerizer assumes the slave is running in a docker container, and launches executors with docker containers in order to recover them when the slave restarts and recovers.
>>>>>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you
>>>>>>>>>>>> Marica
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>>>>>>
>>>>>>>>>>>>> no docker_mesos_image flag in my docker run, and the docker image is build by myself.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you very much
>>>>>>>>>>>>
>>>>>>>>>>>>>> Marica
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the folder in app container is default value:
>>>>>>>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> but in real world, the sandbox is in slave container, not in app container.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>>>>>>>>> Thank you very much,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I’m using a sample application definition file, just for testing purpose:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>> "id": "test-app",
>>>>>>>>>>>>>>>> "container": {
>>>>>>>>>>>>>>>> "type": "DOCKER",
>>>>>>>>>>>>>>>> "docker": {
>>>>>>>>>>>>>>>> "image": "libmesos/ubuntu"
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>> },
>>>>>>>>>>>>>>>> "cpus": 1,
>>>>>>>>>>>>>>>> "mem": 512,
>>>>>>>>>>>>>>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>>>>>>>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here is the docker run command line:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>>>>>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>>>>>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>> Marica
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>>>>>>>>>>> No, using the socket:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> docker in docker mode?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>> Marica
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>
>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>> Via Orabona 4
>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
Hi Tim and Shuai,
thank you very much for your reply. I have opened a JIRA issue on this: https://issues.apache.org/jira/browse/MESOS-4249
I hope it will be patched soon :)
Best regards,
Marica
Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <ti...@mesosphere.io> ha scritto:
> Hi Marica/Shuai,
>
> Sorry haven't been able to spend the time to repro, but looks like Shuai confirmed it.
>
> Can one of you file a JIRA?
>
> Thanks!
>
> Tim
>
> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com> wrote:
> Hi Marica,
>
> I can reproduce the problem exactly as you described in the first email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set, the fetcher works just fine; With it, the fetcher steps seems skipped. This looks like a bug to me.
>
> Regards,
> Shuai
>
> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
> Dear all,
>
> I have not solved this issue yet. Please, can anyone run the same test and let me know if the fetcher is correctly invoked?
> The test is really simple, just try to start a dockerized app (see json definition file below) through marathon on a mesos slave running in a docker container started with the option —docker_mesos_image=<mesos slave image>.
> I would appreciate very much any feedback.
>
> Sample Marathon app:
> {
> "id": "test-app",
> "container": {
> "type": "DOCKER",
> "docker": {
> "image": "libmesos/ubuntu"
> }
> },
> "cpus": 1,
> "mem": 512,
> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
> }
>
> Docker run command to start dockerized mesos slave:
>
> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>
> Thank you very much in advance!
> Best regards,
> Marica
>
> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <ma...@ba.infn.it> ha scritto:
>
>> Dear Tim,
>>
>> I have collected some information from my test environment, starting the slave container with and without the —docker_mesos_image startup flag. Please let me know if you need further input. Thank you very much for your support!
>>
>> Using the flag —docker_mesos_image:
>>
>> root@mesos-slave:~# docker ps
>> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 minutes ago Up 3 minutes slave
>>
>> root@mesos-slave:~# docker logs slave
>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user 'root'
>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update is identical to existing resources
>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@90.147.170.246:39381
>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@90.147.170.246:39381
>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>
>> root@mesos-slave:~# docker logs mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave db70e09f-f39d-491c-8480-73d9858c140b-S0
>> Registered docker executor on 90.147.170.246
>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>
>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor bash
>> root@mesos-slave:/# ls -R /tmp/
>> hsperfdata_root/ mesos/
>> root@mesos-slave:/# ls -R /tmp/mesos/
>> /tmp/mesos/:
>> slaves
>>
>> /tmp/mesos/slaves:
>> db70e09f-f39d-491c-8480-73d9858c140b-S0
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>> frameworks
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>> executors
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>> runs
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>> d965f59b-cc1a-4081-95d2-f3370214c84d
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>> stderr stdout
>>
>>
>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d bash
>> root@mesos-slave:/# env
>> HOSTNAME=mesos-slave
>> HOST=90.147.170.246
>> PORT0=31220
>> PORT_10000=31220
>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>> PWD=/
>> PORTS=31220
>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>> SHLVL=1
>> HOME=/
>> MARATHON_APP_ID=/test-app
>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>> PORT=31220
>> MESOS_SANDBOX=/mnt/mesos/sandbox
>> _=/usr/bin/env
>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>> /mnt/mesos/sandbox:
>> stderr stdout
>>
>>
>> root@mesos-slave:/# ls /var/log/mesos*
>> /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>
>> /var/log/mesos:
>>
>>
>> Disabling the flag —docker_mesos_image the fetcher is called, log is created, the file is downloaded in the sandbox:
>>
>> root@mesos-slave:~# docker exec -it slave bash
>> root@mesos-slave:/#
>> root@mesos-slave:/#
>> root@mesos-slave:/#
>> root@mesos-slave:/# ls /var/log/mesos*
>> /var/log/mesos-fetcher.INFO /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157 /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>
>> /var/log/mesos:
>> root@mesos-slave:/#
>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>> Log file created at: 2015/12/19 18:16:21
>> Running on machine: mesos-slave
>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started!
>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the sandbox directory
>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>> root@mesos-slave:/#
>>
>> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 bash
>> root@mesos-slave:/#
>> root@mesos-slave:/#
>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>> snoqualmie.csv stderr stdout
>> root@mesos-slave:/#
>>
>>
>> Best Regards,
>> Marica
>>
>>
>>
>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io> ha scritto:
>>
>>> HI Marica,
>>>
>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't seem possible we don't pass down the URI flags and if the fetcher failed the container launch should have failed too.
>>>
>>> Also another possible situation is that web UI not really showing the exact content of the sandbox, can you actually go into the directory and see if the files are there?
>>>
>>> Thanks,
>>>
>>> Tim
>>>
>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>> Hi Tim,
>>>
>>> looking at the sandbox I can see only the stderr and stout file (see the attached screenshot). If I remove —docker_mesos_image (and therefore the executor is run inside the slave container) the file specified in the URI field is shown in the sandbox.
>>> Did you verify that the fetcher is called when using the —docker_mesos_image flag ?
>>>
>>> Thanks a lot for your feedback
>>> Marica
>>>
>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>
>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io> ha scritto:
>>>
>>>> Hi Marica,
>>>>
>>>> It should work as we fetch all the files before we launch the executor and place them in the sandbox, and we mount the sandbox into that container as well.
>>>>
>>>> How did you verify that the file is not downloaded?
>>>>
>>>> Tim
>>>>
>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>>> Hi Grzegorz,
>>>>
>>>> I’m using this command line for docker run
>>>>
>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>>>>
>>>> where mesos-slave is the image built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>
>>>> I have tested successfully the deployment of dockerized applications through Marathon and dockerized jobs through Chronos and also the recovery seems to work fine with the flag docker_mesos_image. What is not working for me is the fetcher: it seems that when the executor is launched as separate container (thanks to the flag docker_mesos_image) the information about the URIs to be downloaded is lost…I hope someone can help to understand if this a bug or I’ m missing something.
>>>>
>>>> Cheers,
>>>> Marica
>>>>
>>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gr...@gmail.com> ha scritto:
>>>>
>>>>> I've tried to use this flag, but cannot really run any container when this flag is set.
>>>>> I've raised this issue here: https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and here: https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but sadly no one was able to help me...
>>>>>
>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <ma...@ba.infn.it> napisał:
>>>>> OK, the problem I spotted is related to the usage of the flag —docker_mesos_image that allows the executor to
>>>>>
>>>>>
>>>>> --docker_mesos_image=VALUE The docker image used to launch this mesos slave instance. If an image is specified, the docker containerizer assumes the slave is running in a docker container, and launches executors with docker containers in order to recover them when the slave restarts and recovers.
>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>
>>>>> Thank you
>>>>> Marica
>>>>>
>>>>>
>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>
>>>>>> no docker_mesos_image flag in my docker run, and the docker image is build by myself.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>
>>>>>> Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
>>>>>>
>>>>>> Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
>>>>>>
>>>>>> Thank you very much
>>>>>
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>
>>>>>>
>>>>>>
>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>
>>>>>>> Hi Marica,
>>>>>>>
>>>>>>> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>>>>>>>
>>>>>>> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>>>>>>>
>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>
>>>>>>> the folder in app container is default value:
>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>
>>>>>>> but in real world, the sandbox is in slave container, not in app container.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>> Thank you very much,
>>>>>>>
>>>>>>> I’m using a sample application definition file, just for testing purpose:
>>>>>>>
>>>>>>> {
>>>>>>> "id": "test-app",
>>>>>>> "container": {
>>>>>>> "type": "DOCKER",
>>>>>>> "docker": {
>>>>>>> "image": "libmesos/ubuntu"
>>>>>>> }
>>>>>>> },
>>>>>>> "cpus": 1,
>>>>>>> "mem": 512,
>>>>>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>>>>>> }
>>>>>>>
>>>>>>> Here is the docker run command line:
>>>>>>>
>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>>>>>>>
>>>>>>>
>>>>>>> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>>>>>>>
>>>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>>>> Best regards,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>
>>>>>>>> Hi Marica,
>>>>>>>>
>>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>>
>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>> No, using the socket:
>>>>>>>>
>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>
>>>>>>>>
>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>>
>>>>>>>>> docker in docker mode?
>>>>>>>>>
>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>>>>>>>
>>>>>>>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>
>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>> Best Regards,
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Deshi Xiao
>>>>>>>>> Twitter: xds2000
>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Deshi Xiao
>>>>>>>> Twitter: xds2000
>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Deshi Xiao
>>>>>>> Twitter: xds2000
>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Deshi Xiao
>>>>>> Twitter: xds2000
>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ----------------------------------------------------------
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ----------------------------------------------------------
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Marica/Shuai,
Sorry haven't been able to spend the time to repro, but looks like Shuai
confirmed it.
Can one of you file a JIRA?
Thanks!
Tim
On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <li...@gmail.com> wrote:
> Hi Marica,
>
> I can reproduce the problem exactly as you described in the first email of
> this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set,
> the fetcher works just fine; With it, the fetcher steps seems skipped. This
> looks like a bug to me.
>
> Regards,
> Shuai
>
> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <
> marica.antonacci@ba.infn.it> wrote:
>
>> Dear all,
>>
>> I have not solved this issue yet. Please, can anyone run the same test
>> and let me know if the fetcher is correctly invoked?
>> The test is really simple, just try to start a dockerized app (see json
>> definition file below) through marathon on a mesos slave running in a
>> docker container started with the option —docker_mesos_image=<mesos slave
>> image>.
>> I would appreciate very much any feedback.
>>
>> Sample Marathon app:
>> {
>> "id": "test-app",
>> "container": {
>> "type": "DOCKER",
>> "docker": {
>> "image": "libmesos/ubuntu"
>> }
>> },
>> "cpus": 1,
>> "mem": 512,
>> "uris": [ "
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"
>> ],
>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>> done"
>> }
>>
>> Docker run command to start dockerized mesos slave:
>>
>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e
>> MESOS_CONTAINERIZERS=docker,mesos
>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos
>> --name slave --net host --privileged --pid host mesos-slave
>>
>> Thank you very much in advance!
>> Best regards,
>> Marica
>>
>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <
>> marica.antonacci@ba.infn.it> ha scritto:
>>
>> Dear Tim,
>>
>> I have collected some information from my test environment, starting the
>> slave container with and without the —docker_mesos_image startup flag.
>> Please let me know if you need further input. Thank you very much for your
>> support!
>>
>> Using the flag —docker_mesos_image:
>>
>> root@mesos-slave:~# docker ps
>> CONTAINER ID IMAGE COMMAND CREATED
>> STATUS PORTS NAMES
>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2
>> minutes ago Up 2 minutes
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2
>> minutes ago Up 2 minutes
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3
>> minutes ago Up 3 minutes slave
>>
>> *root@mesos-slave:~# docker logs slave*
>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>> to user 'root'
>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1;
>> mem(*):32 in work directory
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container
>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework
>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to
>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>> 90.147.170.246:39381
>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container
>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update is
>> identical to existing resources
>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@
>> 90.147.170.246:39381
>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update
>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>> 90.147.170.246:39381
>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received
>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing
>> UPDATE for status update TASK_RUNNING (UUID:
>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update
>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for
>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@
>> 90.147.170.246:39381
>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received
>> status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing
>> ACK for status update TASK_RUNNING (UUID:
>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>
>> root@mesos-slave:~# *docker logs
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor*
>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>> --logbufsecs="0" --logging_level="INFO"
>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>> --stop_timeout="0ns"
>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>> --logbufsecs="0" --logging_level="INFO"
>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>> --stop_timeout="0ns"
>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave
>> db70e09f-f39d-491c-8480-73d9858c140b-S0
>> Registered docker executor on 90.147.170.246
>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>
>> root@mesos-slave:~# *docker exec -it
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>> bash*
>> root@mesos-slave:/# ls -R /tmp/
>> hsperfdata_root/ mesos/
>> root@mesos-slave:/# ls -R /tmp/mesos/
>> /tmp/mesos/:
>> *slaves*
>>
>> /tmp/mesos/slaves:
>> *db70e09f-f39d-491c-8480-73d9858c140b-S0*
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>> *frameworks*
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>> *executors*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>> *runs*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>> *d965f59b-cc1a-4081-95d2-f3370214c84d*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>> stderr stdout
>>
>>
>> root@mesos-slave:~# *docker exec -it
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>> bash*
>> root@mesos-slave:/# env
>> HOSTNAME=mesos-slave
>> HOST=90.147.170.246
>> PORT0=31220
>> PORT_10000=31220
>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>> PWD=/
>> PORTS=31220
>>
>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>> SHLVL=1
>> HOME=/
>> MARATHON_APP_ID=/test-app
>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>> PORT=31220
>> MESOS_SANDBOX=/mnt/mesos/sandbox
>> _=/usr/bin/env
>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>> /mnt/mesos/sandbox:
>> stderr stdout
>>
>>
>> root@mesos-slave:/# ls /var/log/mesos*
>> */var/log/mesos-slave.INFO*
>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>
>> /var/log/mesos:
>>
>>
>> Disabling the flag —docker_mesos_image the fetcher is called, log is
>> created, the file is downloaded in the sandbox:
>>
>> root@mesos-slave:~# docker exec -it slave bash
>> root@mesos-slave:/#
>> root@mesos-slave:/#
>> root@mesos-slave:/#
>> root@mesos-slave:/# ls /var/log/mesos*
>> */var/log/mesos-fetcher.INFO*
>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157
>> */var/log/mesos-slave.INFO*
>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>
>> /var/log/mesos:
>> root@mesos-slave:/#
>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>> Log file created at: 2015/12/19 18:16:21
>> Running on machine: mesos-slave
>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started!
>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info:
>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/
>> www.stat.cmu.edu
>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI '
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the
>> sandbox directory
>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI '
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from '
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> to
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched '
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> to
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>> root@mesos-slave:/#
>>
>> root@mesos-slave:~# docker exec -it
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4
>> bash
>> root@mesos-slave:/#
>> root@mesos-slave:/#
>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>> snoqualmie.csv stderr stdout
>> root@mesos-slave:/#
>>
>>
>> Best Regards,
>> Marica
>>
>>
>>
>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io> ha
>> scritto:
>>
>> HI Marica,
>>
>> Did you see the fetcher invoked at all from the Slave logs? Doesn't seem
>> possible we don't pass down the URI flags and if the fetcher failed the
>> container launch should have failed too.
>>
>> Also another possible situation is that web UI not really showing the
>> exact content of the sandbox, can you actually go into the directory and
>> see if the files are there?
>>
>> Thanks,
>>
>> Tim
>>
>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <
>> marica.antonacci@ba.infn.it> wrote:
>>
>>> Hi Tim,
>>>
>>> looking at the sandbox I can see only the stderr and stout file (see the
>>> attached screenshot). If I remove —docker_mesos_image (and therefore the
>>> executor is run inside the slave container) the file specified in the URI
>>> field is shown in the sandbox.
>>> Did you verify that the fetcher is called when using the
>>> —docker_mesos_image flag ?
>>>
>>> Thanks a lot for your feedback
>>> Marica
>>>
>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>
>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io> ha
>>> scritto:
>>>
>>> Hi Marica,
>>>
>>> It should work as we fetch all the files before we launch the executor
>>> and place them in the sandbox, and we mount the sandbox into that container
>>> as well.
>>>
>>> How did you verify that the file is not downloaded?
>>>
>>> Tim
>>>
>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
>>> marica.antonacci@ba.infn.it> wrote:
>>>
>>>> Hi Grzegorz,
>>>>
>>>> I’m using this command line for docker run
>>>>
>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>> -e MESOS_CONTAINERIZERS=docker,mesos -e
>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
>>>> slave --net host --privileged --pid host mesos-slave
>>>>
>>>> where mesos-slave is the image built from the docker file in this repo
>>>> https://github.com/maricaantonacci/mesos-slave-dev
>>>>
>>>> I have tested successfully the deployment of dockerized applications
>>>> through Marathon and dockerized jobs through Chronos and also the recovery
>>>> seems to work fine with the flag docker_mesos_image. What is not working
>>>> for me is the fetcher: it seems that when the executor is launched as
>>>> separate container (thanks to the flag docker_mesos_image) the information
>>>> about the URIs to be downloaded is lost…I hope someone can help to
>>>> understand if this a bug or I’ m missing something.
>>>>
>>>> Cheers,
>>>> Marica
>>>>
>>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
>>>> gregory90@gmail.com> ha scritto:
>>>>
>>>> I've tried to use this flag, but cannot really run any container when
>>>> this flag is set.
>>>> I've raised this issue here:
>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>>>> here:
>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
>>>> sadly no one was able to help me...
>>>>
>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>>>> marica.antonacci@ba.infn.it> napisał:
>>>>
>>>>> OK, the problem I spotted is related to the usage of the
>>>>> flag —docker_mesos_image that allows the executor to
>>>>>
>>>>>
>>>>> --docker_mesos_image=VALUEThe docker image used to launch this mesos
>>>>> slave instance. If an image is specified, the docker containerizer assumes
>>>>> the slave is running in a docker container, and launches executors with
>>>>> docker containers in order to recover them when the slave restarts and
>>>>> recovers.
>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>
>>>>> Thank you
>>>>> Marica
>>>>>
>>>>>
>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com>
>>>>> ha scritto:
>>>>>
>>>>> no docker_mesos_image flag in my docker run, and the docker image is
>>>>> build by myself.
>>>>>
>>>>>
>>>>>
>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it>:
>>>>>
>>>>> Yes, I did check inside the container and the csv file was not
>>>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>>>
>>>>>> Are you running your slave with the --docker_mesos_image flag? Can
>>>>>> you please provide me the docker run command you are using to run your
>>>>>> dockerized slave?
>>>>>>
>>>>>> Thank you very much
>>>>>>
>>>>> Marica
>>>>>>
>>>>>>
>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>
>>>>>
>>>>>>
>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com>
>>>>>> ha scritto:
>>>>>>
>>>>>> Hi Marica,
>>>>>>
>>>>>> use your test-app json, i can run it correctly, the csv is truely
>>>>>> download by mesos slave. please check mesos-master:5050 to check the task
>>>>>> detail download files.
>>>>>>
>>>>>> you describe the app container why not found the csv, because the csv
>>>>>> is download in slave container's folder, not in app container. so if you
>>>>>> run
>>>>>>
>>>>>> cd $MESOS_SANDBOX;
>>>>>>
>>>>>> the folder in app container is default value:
>>>>>>
>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>> but in real world, the sandbox is in slave container, not in app
>>>>>> container.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>
>>>>>>> Thank you very much,
>>>>>>>
>>>>>>> I’m using a sample application definition file, just for testing
>>>>>>> purpose:
>>>>>>>
>>>>>>> {
>>>>>>> "id": "test-app",
>>>>>>> "container": {
>>>>>>> "type": "DOCKER",
>>>>>>> "docker": {
>>>>>>> "image": "libmesos/ubuntu"
>>>>>>> }
>>>>>>> },
>>>>>>> "cpus": 1,
>>>>>>> "mem": 512,
>>>>>>> *"uris": [
>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>>>> ],*
>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u
>>>>>>> +%T; done"
>>>>>>> }
>>>>>>>
>>>>>>> Here is the docker run command line:
>>>>>>>
>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP>
>>>>>>> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
>>>>>>> -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>>>> --privileged --pid host mesos-slave
>>>>>>>
>>>>>>>
>>>>>>> As already mentioned, if I remove the environment variable
>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>>>> snoqualmie.csv inside the sandbox.
>>>>>>>
>>>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>>>> Best regards,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com>
>>>>>>> ha scritto:
>>>>>>>
>>>>>>> Hi Marica,
>>>>>>>
>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>
>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>
>>>>>>>> No, using the socket:
>>>>>>>>
>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>
>>>>>>>>
>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com>
>>>>>>>> ha scritto:
>>>>>>>>
>>>>>>>> docker in docker mode?
>>>>>>>>
>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>>>>> inside a docker container (using *docker_mesos_image* startup
>>>>>>>>> flag) and you submit the deployment of a dockerized application or job the
>>>>>>>>> fetcher step is not performed. On the other hand, if I request the
>>>>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>>>
>>>>>>>>> Therefore, it seems that the information about the URIs gets lost
>>>>>>>>> when the dockerized mesos slave spawns the executor docker container that
>>>>>>>>> in turn launches the application docker container…Has anyone seen this
>>>>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>
>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>> Best Regards,
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Deshi Xiao
>>>>>>>> Twitter: xds2000
>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>>
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Deshi Xiao
>>>>>>> Twitter: xds2000
>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Deshi Xiao
>>>>>> Twitter: xds2000
>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>>>> 2015-12-18 下午4.55.38.png>
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ----------------------------------------------------------
>>>
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
Dear all,
I have not solved this issue yet. Please, can anyone run the same test and let me know if the fetcher is correctly invoked?
The test is really simple, just try to start a dockerized app (see json definition file below) through marathon on a mesos slave running in a docker container started with the option —docker_mesos_image=<mesos slave image>.
I would appreciate very much any feedback.
Sample Marathon app:
{
"id": "test-app",
"container": {
"type": "DOCKER",
"docker": {
"image": "libmesos/ubuntu"
}
},
"cpus": 1,
"mem": 512,
"uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
"cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
}
Docker run command to start dockerized mesos slave:
# docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
Thank you very much in advance!
Best regards,
Marica
Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <ma...@ba.infn.it> ha scritto:
> Dear Tim,
>
> I have collected some information from my test environment, starting the slave container with and without the —docker_mesos_image startup flag. Please let me know if you need further input. Thank you very much for your support!
>
> Using the flag —docker_mesos_image:
>
> root@mesos-slave:~# docker ps
> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 minutes ago Up 3 minutes slave
>
> root@mesos-slave:~# docker logs slave
> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user 'root'
> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update is identical to existing resources
> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@90.147.170.246:39381
> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@90.147.170.246:39381
> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>
> root@mesos-slave:~# docker logs mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave db70e09f-f39d-491c-8480-73d9858c140b-S0
> Registered docker executor on 90.147.170.246
> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>
> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor bash
> root@mesos-slave:/# ls -R /tmp/
> hsperfdata_root/ mesos/
> root@mesos-slave:/# ls -R /tmp/mesos/
> /tmp/mesos/:
> slaves
>
> /tmp/mesos/slaves:
> db70e09f-f39d-491c-8480-73d9858c140b-S0
>
> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
> frameworks
>
> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>
> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
> executors
>
> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>
> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
> runs
>
> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
> d965f59b-cc1a-4081-95d2-f3370214c84d
>
> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
> stderr stdout
>
>
> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d bash
> root@mesos-slave:/# env
> HOSTNAME=mesos-slave
> HOST=90.147.170.246
> PORT0=31220
> PORT_10000=31220
> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> PWD=/
> PORTS=31220
> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
> SHLVL=1
> HOME=/
> MARATHON_APP_ID=/test-app
> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
> PORT=31220
> MESOS_SANDBOX=/mnt/mesos/sandbox
> _=/usr/bin/env
> root@mesos-slave:/# ls -R $MESOS_SANDBOX
> /mnt/mesos/sandbox:
> stderr stdout
>
>
> root@mesos-slave:/# ls /var/log/mesos*
> /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>
> /var/log/mesos:
>
>
> Disabling the flag —docker_mesos_image the fetcher is called, log is created, the file is downloaded in the sandbox:
>
> root@mesos-slave:~# docker exec -it slave bash
> root@mesos-slave:/#
> root@mesos-slave:/#
> root@mesos-slave:/#
> root@mesos-slave:/# ls /var/log/mesos*
> /var/log/mesos-fetcher.INFO /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157 /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>
> /var/log/mesos:
> root@mesos-slave:/#
> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
> Log file created at: 2015/12/19 18:16:21
> Running on machine: mesos-slave
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started!
> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the sandbox directory
> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
> root@mesos-slave:/#
>
> root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 bash
> root@mesos-slave:/#
> root@mesos-slave:/#
> root@mesos-slave:/# ls /mnt/mesos/sandbox/
> snoqualmie.csv stderr stdout
> root@mesos-slave:/#
>
>
> Best Regards,
> Marica
>
>
>
> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io> ha scritto:
>
>> HI Marica,
>>
>> Did you see the fetcher invoked at all from the Slave logs? Doesn't seem possible we don't pass down the URI flags and if the fetcher failed the container launch should have failed too.
>>
>> Also another possible situation is that web UI not really showing the exact content of the sandbox, can you actually go into the directory and see if the files are there?
>>
>> Thanks,
>>
>> Tim
>>
>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
>> Hi Tim,
>>
>> looking at the sandbox I can see only the stderr and stout file (see the attached screenshot). If I remove —docker_mesos_image (and therefore the executor is run inside the slave container) the file specified in the URI field is shown in the sandbox.
>> Did you verify that the fetcher is called when using the —docker_mesos_image flag ?
>>
>> Thanks a lot for your feedback
>> Marica
>>
>> <Schermata 2015-12-18 alle 10.15.58.png>
>>
>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io> ha scritto:
>>
>>> Hi Marica,
>>>
>>> It should work as we fetch all the files before we launch the executor and place them in the sandbox, and we mount the sandbox into that container as well.
>>>
>>> How did you verify that the file is not downloaded?
>>>
>>> Tim
>>>
>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
>>> Hi Grzegorz,
>>>
>>> I’m using this command line for docker run
>>>
>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>>>
>>> where mesos-slave is the image built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev
>>>
>>> I have tested successfully the deployment of dockerized applications through Marathon and dockerized jobs through Chronos and also the recovery seems to work fine with the flag docker_mesos_image. What is not working for me is the fetcher: it seems that when the executor is launched as separate container (thanks to the flag docker_mesos_image) the information about the URIs to be downloaded is lost…I hope someone can help to understand if this a bug or I’ m missing something.
>>>
>>> Cheers,
>>> Marica
>>>
>>>
>>>
>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gr...@gmail.com> ha scritto:
>>>
>>>> I've tried to use this flag, but cannot really run any container when this flag is set.
>>>> I've raised this issue here: https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and here: https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but sadly no one was able to help me...
>>>>
>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <ma...@ba.infn.it> napisał:
>>>> OK, the problem I spotted is related to the usage of the flag —docker_mesos_image that allows the executor to
>>>>
>>>>
>>>> --docker_mesos_image=VALUE The docker image used to launch this mesos slave instance. If an image is specified, the docker containerizer assumes the slave is running in a docker container, and launches executors with docker containers in order to recover them when the slave restarts and recovers.
>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>
>>>> Thank you
>>>> Marica
>>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha scritto:
>>>>
>>>>> no docker_mesos_image flag in my docker run, and the docker image is build by myself.
>>>>>
>>>>>
>>>>>
>>>>
>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>
>>>>> Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
>>>>>
>>>>> Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
>>>>>
>>>>> Thank you very much
>>>>
>>>>> Marica
>>>>>
>>>>>
>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>
>>>>>
>>>>>
>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>
>>>>>> Hi Marica,
>>>>>>
>>>>>> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>>>>>>
>>>>>> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>>>>>>
>>>>>> cd $MESOS_SANDBOX;
>>>>>>
>>>>>> the folder in app container is default value:
>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>
>>>>>> but in real world, the sandbox is in slave container, not in app container.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>> Thank you very much,
>>>>>>
>>>>>> I’m using a sample application definition file, just for testing purpose:
>>>>>>
>>>>>> {
>>>>>> "id": "test-app",
>>>>>> "container": {
>>>>>> "type": "DOCKER",
>>>>>> "docker": {
>>>>>> "image": "libmesos/ubuntu"
>>>>>> }
>>>>>> },
>>>>>> "cpus": 1,
>>>>>> "mem": 512,
>>>>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>>>>> }
>>>>>>
>>>>>> Here is the docker run command line:
>>>>>>
>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>>>>>>
>>>>>>
>>>>>> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>>>>>>
>>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>>> Best regards,
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>
>>>>>>> Hi Marica,
>>>>>>>
>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>
>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>> No, using the socket:
>>>>>>>
>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>>
>>>>>>>> docker in docker mode?
>>>>>>>>
>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>>>>>>
>>>>>>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>
>>>>>>>> Thanks a lot in advance for you help
>>>>>>>> Best Regards,
>>>>>>>> Marica
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Deshi Xiao
>>>>>>>> Twitter: xds2000
>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Deshi Xiao
>>>>>>> Twitter: xds2000
>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Deshi Xiao
>>>>>> Twitter: xds2000
>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>
>>>>
>>>> ----------------------------------------------------------
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ----------------------------------------------------------
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
Dear Tim,
I have collected some information from my test environment, starting the slave container with and without the —docker_mesos_image startup flag. Please let me know if you need further input. Thank you very much for your support!
Using the flag —docker_mesos_image:
root@mesos-slave:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 minutes ago Up 2 minutes mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 minutes ago Up 3 minutes slave
root@mesos-slave:~# docker logs slave
I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user 'root'
I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.335907 19480 docker.cpp:762] Starting container 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update is identical to existing resources
I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@90.147.170.246:39381
I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@90.147.170.246:39381
I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
root@mesos-slave:~# docker logs mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
--container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
--container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" --stop_timeout="0ns"
I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave db70e09f-f39d-491c-8480-73d9858c140b-S0
Registered docker executor on 90.147.170.246
Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor bash
root@mesos-slave:/# ls -R /tmp/
hsperfdata_root/ mesos/
root@mesos-slave:/# ls -R /tmp/mesos/
/tmp/mesos/:
slaves
/tmp/mesos/slaves:
db70e09f-f39d-491c-8480-73d9858c140b-S0
/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
frameworks
/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
246b272b-d649-47c0-88ca-6b1ff35f437a-0000
/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
executors
/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
runs
/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
d965f59b-cc1a-4081-95d2-f3370214c84d
/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
stderr stdout
root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d bash
root@mesos-slave:/# env
HOSTNAME=mesos-slave
HOST=90.147.170.246
PORT0=31220
PORT_10000=31220
MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
PORTS=31220
MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
SHLVL=1
HOME=/
MARATHON_APP_ID=/test-app
MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
PORT=31220
MESOS_SANDBOX=/mnt/mesos/sandbox
_=/usr/bin/env
root@mesos-slave:/# ls -R $MESOS_SANDBOX
/mnt/mesos/sandbox:
stderr stdout
root@mesos-slave:/# ls /var/log/mesos*
/var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
/var/log/mesos:
Disabling the flag —docker_mesos_image the fetcher is called, log is created, the file is downloaded in the sandbox:
root@mesos-slave:~# docker exec -it slave bash
root@mesos-slave:/#
root@mesos-slave:/#
root@mesos-slave:/#
root@mesos-slave:/# ls /var/log/mesos*
/var/log/mesos-fetcher.INFO /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157 /var/log/mesos-slave.INFO /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
/var/log/mesos:
root@mesos-slave:/#
root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
Log file created at: 2015/12/19 18:16:21
Running on machine: mesos-slave
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started!
I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the sandbox directory
I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
root@mesos-slave:/#
root@mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 bash
root@mesos-slave:/#
root@mesos-slave:/#
root@mesos-slave:/# ls /mnt/mesos/sandbox/
snoqualmie.csv stderr stdout
root@mesos-slave:/#
Best Regards,
Marica
Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <ti...@mesosphere.io> ha scritto:
> HI Marica,
>
> Did you see the fetcher invoked at all from the Slave logs? Doesn't seem possible we don't pass down the URI flags and if the fetcher failed the container launch should have failed too.
>
> Also another possible situation is that web UI not really showing the exact content of the sandbox, can you actually go into the directory and see if the files are there?
>
> Thanks,
>
> Tim
>
> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <ma...@ba.infn.it> wrote:
> Hi Tim,
>
> looking at the sandbox I can see only the stderr and stout file (see the attached screenshot). If I remove —docker_mesos_image (and therefore the executor is run inside the slave container) the file specified in the URI field is shown in the sandbox.
> Did you verify that the fetcher is called when using the —docker_mesos_image flag ?
>
> Thanks a lot for your feedback
> Marica
>
> <Schermata 2015-12-18 alle 10.15.58.png>
>
> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io> ha scritto:
>
>> Hi Marica,
>>
>> It should work as we fetch all the files before we launch the executor and place them in the sandbox, and we mount the sandbox into that container as well.
>>
>> How did you verify that the file is not downloaded?
>>
>> Tim
>>
>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
>> Hi Grzegorz,
>>
>> I’m using this command line for docker run
>>
>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>>
>> where mesos-slave is the image built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev
>>
>> I have tested successfully the deployment of dockerized applications through Marathon and dockerized jobs through Chronos and also the recovery seems to work fine with the flag docker_mesos_image. What is not working for me is the fetcher: it seems that when the executor is launched as separate container (thanks to the flag docker_mesos_image) the information about the URIs to be downloaded is lost…I hope someone can help to understand if this a bug or I’ m missing something.
>>
>> Cheers,
>> Marica
>>
>>
>>
>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gr...@gmail.com> ha scritto:
>>
>>> I've tried to use this flag, but cannot really run any container when this flag is set.
>>> I've raised this issue here: https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and here: https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but sadly no one was able to help me...
>>>
>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <ma...@ba.infn.it> napisał:
>>> OK, the problem I spotted is related to the usage of the flag —docker_mesos_image that allows the executor to
>>>
>>>
>>> --docker_mesos_image=VALUE The docker image used to launch this mesos slave instance. If an image is specified, the docker containerizer assumes the slave is running in a docker container, and launches executors with docker containers in order to recover them when the slave restarts and recovers.
>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>
>>> Thank you
>>> Marica
>>>
>>>
>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha scritto:
>>>
>>>> no docker_mesos_image flag in my docker run, and the docker image is build by myself.
>>>>
>>>>
>>>>
>>>
>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>
>>>> Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
>>>>
>>>> Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
>>>>
>>>> Thank you very much
>>>
>>>> Marica
>>>>
>>>>
>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>
>>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
>>>>
>>>>> Hi Marica,
>>>>>
>>>>> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>>>>>
>>>>> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>>>>>
>>>>> cd $MESOS_SANDBOX;
>>>>>
>>>>> the folder in app container is default value:
>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>
>>>>> but in real world, the sandbox is in slave container, not in app container.
>>>>>
>>>>>
>>>>>
>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>> Thank you very much,
>>>>>
>>>>> I’m using a sample application definition file, just for testing purpose:
>>>>>
>>>>> {
>>>>> "id": "test-app",
>>>>> "container": {
>>>>> "type": "DOCKER",
>>>>> "docker": {
>>>>> "image": "libmesos/ubuntu"
>>>>> }
>>>>> },
>>>>> "cpus": 1,
>>>>> "mem": 512,
>>>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>>>> }
>>>>>
>>>>> Here is the docker run command line:
>>>>>
>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>>>>>
>>>>>
>>>>> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>>>>>
>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>> Best regards,
>>>>> Marica
>>>>>
>>>>>
>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>
>>>>>> Hi Marica,
>>>>>>
>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>
>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>> No, using the socket:
>>>>>>
>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>
>>>>>>
>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>>
>>>>>>> docker in docker mode?
>>>>>>>
>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>>>>>
>>>>>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>
>>>>>>> Thanks a lot in advance for you help
>>>>>>> Best Regards,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Deshi Xiao
>>>>>>> Twitter: xds2000
>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Deshi Xiao
>>>>>> Twitter: xds2000
>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
>>>>
>>>> ----------------------------------------------------------
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Deshi Xiao
>>>> Twitter: xds2000
>>>> E-mail: xiaods(AT)gmail.com
>>>
>>>
>>> ----------------------------------------------------------
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by Tim Chen <ti...@mesosphere.io>.
HI Marica,
Did you see the fetcher invoked at all from the Slave logs? Doesn't seem
possible we don't pass down the URI flags and if the fetcher failed the
container launch should have failed too.
Also another possible situation is that web UI not really showing the exact
content of the sandbox, can you actually go into the directory and see if
the files are there?
Thanks,
Tim
On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <
marica.antonacci@ba.infn.it> wrote:
> Hi Tim,
>
> looking at the sandbox I can see only the stderr and stout file (see the
> attached screenshot). If I remove —docker_mesos_image (and therefore the
> executor is run inside the slave container) the file specified in the URI
> field is shown in the sandbox.
> Did you verify that the fetcher is called when using the
> —docker_mesos_image flag ?
>
> Thanks a lot for your feedback
> Marica
>
>
> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io> ha
> scritto:
>
> Hi Marica,
>
> It should work as we fetch all the files before we launch the executor and
> place them in the sandbox, and we mount the sandbox into that container as
> well.
>
> How did you verify that the file is not downloaded?
>
> Tim
>
> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
> marica.antonacci@ba.infn.it> wrote:
>
>> Hi Grzegorz,
>>
>> I’m using this command line for docker run
>>
>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>> -e MESOS_CONTAINERIZERS=docker,mesos -e
>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
>> slave --net host --privileged --pid host mesos-slave
>>
>> where mesos-slave is the image built from the docker file in this repo
>> https://github.com/maricaantonacci/mesos-slave-dev
>>
>> I have tested successfully the deployment of dockerized applications
>> through Marathon and dockerized jobs through Chronos and also the recovery
>> seems to work fine with the flag docker_mesos_image. What is not working
>> for me is the fetcher: it seems that when the executor is launched as
>> separate container (thanks to the flag docker_mesos_image) the information
>> about the URIs to be downloaded is lost…I hope someone can help to
>> understand if this a bug or I’ m missing something.
>>
>> Cheers,
>> Marica
>>
>>
>>
>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
>> gregory90@gmail.com> ha scritto:
>>
>> I've tried to use this flag, but cannot really run any container when
>> this flag is set.
>> I've raised this issue here:
>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>> here:
>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
>> sadly no one was able to help me...
>>
>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>> marica.antonacci@ba.infn.it> napisał:
>>
>>> OK, the problem I spotted is related to the usage of the
>>> flag —docker_mesos_image that allows the executor to
>>>
>>>
>>> --docker_mesos_image=VALUEThe docker image used to launch this mesos
>>> slave instance. If an image is specified, the docker containerizer assumes
>>> the slave is running in a docker container, and launches executors with
>>> docker containers in order to recover them when the slave restarts and
>>> recovers.
>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>
>>> Thank you
>>> Marica
>>>
>>>
>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha
>>> scritto:
>>>
>>> no docker_mesos_image flag in my docker run, and the docker image is
>>> build by myself.
>>>
>>>
>>>
>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it
>>> >:
>>>
>>> Yes, I did check inside the container and the csv file was not
>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>
>>>> Are you running your slave with the --docker_mesos_image flag? Can you
>>>> please provide me the docker run command you are using to run your
>>>> dockerized slave?
>>>>
>>>> Thank you very much
>>>>
>>> Marica
>>>>
>>>>
>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>
>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com>
>>>> ha scritto:
>>>>
>>>> Hi Marica,
>>>>
>>>> use your test-app json, i can run it correctly, the csv is truely
>>>> download by mesos slave. please check mesos-master:5050 to check the task
>>>> detail download files.
>>>>
>>>> you describe the app container why not found the csv, because the csv
>>>> is download in slave container's folder, not in app container. so if you
>>>> run
>>>>
>>>> cd $MESOS_SANDBOX;
>>>>
>>>> the folder in app container is default value:
>>>>
>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>> but in real world, the sandbox is in slave container, not in app
>>>> container.
>>>>
>>>>
>>>>
>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>> marica.antonacci@ba.infn.it>:
>>>>
>>>>> Thank you very much,
>>>>>
>>>>> I’m using a sample application definition file, just for testing
>>>>> purpose:
>>>>>
>>>>> {
>>>>> "id": "test-app",
>>>>> "container": {
>>>>> "type": "DOCKER",
>>>>> "docker": {
>>>>> "image": "libmesos/ubuntu"
>>>>> }
>>>>> },
>>>>> "cpus": 1,
>>>>> "mem": 512,
>>>>> *"uris": [
>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>> ],*
>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>>>>> done"
>>>>> }
>>>>>
>>>>> Here is the docker run command line:
>>>>>
>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e
>>>>> MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>> --privileged --pid host mesos-slave
>>>>>
>>>>>
>>>>> As already mentioned, if I remove the environment variable
>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>> snoqualmie.csv inside the sandbox.
>>>>>
>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>> Best regards,
>>>>> Marica
>>>>>
>>>>>
>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com>
>>>>> ha scritto:
>>>>>
>>>>> Hi Marica,
>>>>>
>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>
>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it>:
>>>>>
>>>>>> No, using the socket:
>>>>>>
>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>
>>>>>>
>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com>
>>>>>> ha scritto:
>>>>>>
>>>>>> docker in docker mode?
>>>>>>
>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>>> inside a docker container (using *docker_mesos_image* startup flag)
>>>>>>> and you submit the deployment of a dockerized application or job the
>>>>>>> fetcher step is not performed. On the other hand, if I request the
>>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>
>>>>>>> Therefore, it seems that the information about the URIs gets lost
>>>>>>> when the dockerized mesos slave spawns the executor docker container that
>>>>>>> in turn launches the application docker container…Has anyone seen this
>>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>
>>>>>>> Thanks a lot in advance for you help
>>>>>>> Best Regards,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Deshi Xiao
>>>>>> Twitter: xds2000
>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Deshi Xiao
>>>> Twitter: xds2000
>>>> E-mail: xiaods(AT)gmail.com
>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>> 2015-12-18 下午4.55.38.png>
>>>>
>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Deshi Xiao
>>> Twitter: xds2000
>>> E-mail: xiaods(AT)gmail.com
>>>
>>>
>>> ----------------------------------------------------------
>>>
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> ----------------------------------------------------------
>
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
Hi Tim,
looking at the sandbox I can see only the stderr and stout file (see the attached screenshot). If I remove —docker_mesos_image (and therefore the executor is run inside the slave container) the file specified in the URI field is shown in the sandbox.
Did you verify that the fetcher is called when using the —docker_mesos_image flag ?
Thanks a lot for your feedback
Marica
Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <ti...@mesosphere.io> ha scritto:
> Hi Marica,
>
> It should work as we fetch all the files before we launch the executor and place them in the sandbox, and we mount the sandbox into that container as well.
>
> How did you verify that the file is not downloaded?
>
> Tim
>
> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <ma...@ba.infn.it> wrote:
> Hi Grzegorz,
>
> I’m using this command line for docker run
>
> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
>
> where mesos-slave is the image built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev
>
> I have tested successfully the deployment of dockerized applications through Marathon and dockerized jobs through Chronos and also the recovery seems to work fine with the flag docker_mesos_image. What is not working for me is the fetcher: it seems that when the executor is launched as separate container (thanks to the flag docker_mesos_image) the information about the URIs to be downloaded is lost…I hope someone can help to understand if this a bug or I’ m missing something.
>
> Cheers,
> Marica
>
>
>
> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gr...@gmail.com> ha scritto:
>
>> I've tried to use this flag, but cannot really run any container when this flag is set.
>> I've raised this issue here: https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and here: https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but sadly no one was able to help me...
>>
>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <ma...@ba.infn.it> napisał:
>> OK, the problem I spotted is related to the usage of the flag —docker_mesos_image that allows the executor to
>>
>>
>> --docker_mesos_image=VALUE The docker image used to launch this mesos slave instance. If an image is specified, the docker containerizer assumes the slave is running in a docker container, and launches executors with docker containers in order to recover them when the slave restarts and recovers.
>> Has anyone used this flag and tested the behavior of the fetcher?
>>
>> Thank you
>> Marica
>>
>>
>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha scritto:
>>
>>> no docker_mesos_image flag in my docker run, and the docker image is build by myself.
>>>
>>>
>>>
>>
>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>
>>> Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
>>>
>>> Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
>>>
>>> Thank you very much
>>
>>> Marica
>>>
>>>
>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>
>>>
>>>
>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
>>>
>>>> Hi Marica,
>>>>
>>>> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>>>>
>>>> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>>>>
>>>> cd $MESOS_SANDBOX;
>>>>
>>>> the folder in app container is default value:
>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>
>>>> but in real world, the sandbox is in slave container, not in app container.
>>>>
>>>>
>>>>
>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>> Thank you very much,
>>>>
>>>> I’m using a sample application definition file, just for testing purpose:
>>>>
>>>> {
>>>> "id": "test-app",
>>>> "container": {
>>>> "type": "DOCKER",
>>>> "docker": {
>>>> "image": "libmesos/ubuntu"
>>>> }
>>>> },
>>>> "cpus": 1,
>>>> "mem": 512,
>>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>>> }
>>>>
>>>> Here is the docker run command line:
>>>>
>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>>>>
>>>>
>>>> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>>>>
>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>> Best regards,
>>>> Marica
>>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>>>>
>>>>> Hi Marica,
>>>>>
>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>
>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>> No, using the socket:
>>>>>
>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>
>>>>>
>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>>>>
>>>>>> docker in docker mode?
>>>>>>
>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>>> Dear all,
>>>>>>
>>>>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>>>>
>>>>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>>>>
>>>>>> Thanks a lot in advance for you help
>>>>>> Best Regards,
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Deshi Xiao
>>>>>> Twitter: xds2000
>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>
>>>> ----------------------------------------------------------
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Deshi Xiao
>>>> Twitter: xds2000
>>>> E-mail: xiaods(AT)gmail.com
>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
>>>
>>> ----------------------------------------------------------
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>>
>>>
>>>
>>> --
>>> Deshi Xiao
>>> Twitter: xds2000
>>> E-mail: xiaods(AT)gmail.com
>>
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Marica,
It should work as we fetch all the files before we launch the executor and
place them in the sandbox, and we mount the sandbox into that container as
well.
How did you verify that the file is not downloaded?
Tim
On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
marica.antonacci@ba.infn.it> wrote:
> Hi Grzegorz,
>
> I’m using this command line for docker run
>
> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
> -e MESOS_CONTAINERIZERS=docker,mesos -e
> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
> slave --net host --privileged --pid host mesos-slave
>
> where mesos-slave is the image built from the docker file in this repo
> https://github.com/maricaantonacci/mesos-slave-dev
>
> I have tested successfully the deployment of dockerized applications
> through Marathon and dockerized jobs through Chronos and also the recovery
> seems to work fine with the flag docker_mesos_image. What is not working
> for me is the fetcher: it seems that when the executor is launched as
> separate container (thanks to the flag docker_mesos_image) the information
> about the URIs to be downloaded is lost…I hope someone can help to
> understand if this a bug or I’ m missing something.
>
> Cheers,
> Marica
>
>
>
> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
> gregory90@gmail.com> ha scritto:
>
> I've tried to use this flag, but cannot really run any container when this
> flag is set.
> I've raised this issue here:
> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
> here:
> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
> sadly no one was able to help me...
>
> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
> marica.antonacci@ba.infn.it> napisał:
>
>> OK, the problem I spotted is related to the usage of the
>> flag —docker_mesos_image that allows the executor to
>>
>>
>> --docker_mesos_image=VALUEThe docker image used to launch this mesos
>> slave instance. If an image is specified, the docker containerizer assumes
>> the slave is running in a docker container, and launches executors with
>> docker containers in order to recover them when the slave restarts and
>> recovers.
>> Has anyone used this flag and tested the behavior of the fetcher?
>>
>> Thank you
>> Marica
>>
>>
>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha
>> scritto:
>>
>> no docker_mesos_image flag in my docker run, and the docker image is
>> build by myself.
>>
>>
>>
>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>
>> :
>>
>> Yes, I did check inside the container and the csv file was not downloaded
>>> as shown also by the app details (see the screenshot below).
>>>
>>> Are you running your slave with the --docker_mesos_image flag? Can you
>>> please provide me the docker run command you are using to run your
>>> dockerized slave?
>>>
>>> Thank you very much
>>>
>> Marica
>>>
>>>
>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>
>>
>>>
>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha
>>> scritto:
>>>
>>> Hi Marica,
>>>
>>> use your test-app json, i can run it correctly, the csv is truely
>>> download by mesos slave. please check mesos-master:5050 to check the task
>>> detail download files.
>>>
>>> you describe the app container why not found the csv, because the csv is
>>> download in slave container's folder, not in app container. so if you run
>>>
>>> cd $MESOS_SANDBOX;
>>>
>>> the folder in app container is default value:
>>>
>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>> but in real world, the sandbox is in slave container, not in app
>>> container.
>>>
>>>
>>>
>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it
>>> >:
>>>
>>>> Thank you very much,
>>>>
>>>> I’m using a sample application definition file, just for testing
>>>> purpose:
>>>>
>>>> {
>>>> "id": "test-app",
>>>> "container": {
>>>> "type": "DOCKER",
>>>> "docker": {
>>>> "image": "libmesos/ubuntu"
>>>> }
>>>> },
>>>> "cpus": 1,
>>>> "mem": 512,
>>>> *"uris": [
>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>> ],*
>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>>>> done"
>>>> }
>>>>
>>>> Here is the docker run command line:
>>>>
>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e
>>>> MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>> --privileged --pid host mesos-slave
>>>>
>>>>
>>>> As already mentioned, if I remove the environment variable
>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>> snoqualmie.csv inside the sandbox.
>>>>
>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>> Best regards,
>>>> Marica
>>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com>
>>>> ha scritto:
>>>>
>>>> Hi Marica,
>>>>
>>>> Could you please give a sample marathon json, i can test it asap.
>>>>
>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it
>>>> >:
>>>>
>>>>> No, using the socket:
>>>>>
>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>
>>>>>
>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com>
>>>>> ha scritto:
>>>>>
>>>>> docker in docker mode?
>>>>>
>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it>:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>> inside a docker container (using *docker_mesos_image* startup flag)
>>>>>> and you submit the deployment of a dockerized application or job the
>>>>>> fetcher step is not performed. On the other hand, if I request the
>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>
>>>>>> Therefore, it seems that the information about the URIs gets lost
>>>>>> when the dockerized mesos slave spawns the executor docker container that
>>>>>> in turn launches the application docker container…Has anyone seen this
>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>
>>>>>> Thanks a lot in advance for you help
>>>>>> Best Regards,
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Deshi Xiao
>>>> Twitter: xds2000
>>>> E-mail: xiaods(AT)gmail.com
>>>>
>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Deshi Xiao
>>> Twitter: xds2000
>>> E-mail: xiaods(AT)gmail.com
>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>> 2015-12-18 下午4.55.38.png>
>>>
>>>
>>> ----------------------------------------------------------
>>>
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
> ----------------------------------------------------------
>
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
Hi Grzegorz,
I’m using this command line for docker run
# docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-slave
where mesos-slave is the image built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev
I have tested successfully the deployment of dockerized applications through Marathon and dockerized jobs through Chronos and also the recovery seems to work fine with the flag docker_mesos_image. What is not working for me is the fetcher: it seems that when the executor is launched as separate container (thanks to the flag docker_mesos_image) the information about the URIs to be downloaded is lost…I hope someone can help to understand if this a bug or I’ m missing something.
Cheers,
Marica
Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gr...@gmail.com> ha scritto:
> I've tried to use this flag, but cannot really run any container when this flag is set.
> I've raised this issue here: https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and here: https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but sadly no one was able to help me...
>
> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <ma...@ba.infn.it> napisał:
> OK, the problem I spotted is related to the usage of the flag —docker_mesos_image that allows the executor to
>
>
> --docker_mesos_image=VALUE The docker image used to launch this mesos slave instance. If an image is specified, the docker containerizer assumes the slave is running in a docker container, and launches executors with docker containers in order to recover them when the slave restarts and recovers.
> Has anyone used this flag and tested the behavior of the fetcher?
>
> Thank you
> Marica
>
>
> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha scritto:
>
>> no docker_mesos_image flag in my docker run, and the docker image is build by myself.
>>
>>
>>
>
>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>
>> Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
>>
>> Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
>>
>> Thank you very much
>
>> Marica
>>
>>
>> <Schermata 2015-12-18 alle 10.15.58.png>
>
>>
>>
>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
>>
>>> Hi Marica,
>>>
>>> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>>>
>>> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>>>
>>> cd $MESOS_SANDBOX;
>>>
>>> the folder in app container is default value:
>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>
>>> but in real world, the sandbox is in slave container, not in app container.
>>>
>>>
>>>
>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>> Thank you very much,
>>>
>>> I’m using a sample application definition file, just for testing purpose:
>>>
>>> {
>>> "id": "test-app",
>>> "container": {
>>> "type": "DOCKER",
>>> "docker": {
>>> "image": "libmesos/ubuntu"
>>> }
>>> },
>>> "cpus": 1,
>>> "mem": 512,
>>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>>> }
>>>
>>> Here is the docker run command line:
>>>
>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>>>
>>>
>>> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>>>
>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>> Best regards,
>>> Marica
>>>
>>>
>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>>>
>>>> Hi Marica,
>>>>
>>>> Could you please give a sample marathon json, i can test it asap.
>>>>
>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>> No, using the socket:
>>>>
>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>
>>>>
>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>>>
>>>>> docker in docker mode?
>>>>>
>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>>> Dear all,
>>>>>
>>>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>>>
>>>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>>>
>>>>> Thanks a lot in advance for you help
>>>>> Best Regards,
>>>>> Marica
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>
>>>> ----------------------------------------------------------
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Deshi Xiao
>>>> Twitter: xds2000
>>>> E-mail: xiaods(AT)gmail.com
>>>
>>> ----------------------------------------------------------
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Deshi Xiao
>>> Twitter: xds2000
>>> E-mail: xiaods(AT)gmail.com
>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Shuai,
You need to specify the --pid=host flag.
Tim
On Fri, Dec 18, 2015 at 5:19 AM, Shuai Lin <li...@gmail.com> wrote:
> The problem happens to me if I don't specify the --docker_mesos_image
> flag. However, specifying the flag only makes things worse: the task is
> failed again and agin, but there does exist a container for this task.
>
> master and zookeeper is running on host, and slave is running inside a
> docker image:
>
> ```
> sudo docker run -it --rm \
> -e MESOS_HOSTNAME=localhost \
> -e MESOS_IP=127.0.0.1 \
> -e MESOS_MASTER=zk://127.0.0.1:2181/mesos \
> -v /sys/fs/cgroup:/sys/fs/cgroup \
> -v /var/run/docker.sock:/var/run/docker.sock \
> --name mesos-slave \
> --net host \
> --privileged \
> mesoscloud/mesos-slave:0.24.1-ubuntu-14.04
>
> ```
>
> However my setup may affect the outcome: master is 0.25.0 and slave is
> 0.24.1 (can't find a public docker image for mesos 2.5.1)
>
> Output of http http://127.0.0.1:8080/v2/apps (unrelevant part ommited)
>
> ```
> {
> "apps": [
> "container": {
> "docker": {
> "parameters": [],
> "privileged": false,
> "network": "BRIDGE",
> "image": "testapp:latest"
> },
> "volumes": [],
> "type": "DOCKER"
> },
> "uris": [
> "https://google.com/robots.txt"
> ],
> }
> ]
> }
> ```
>
> On Fri, Dec 18, 2015 at 7:11 PM, Grzegorz Graczyk <gr...@gmail.com>
> wrote:
>
>> I've tried to use this flag, but cannot really run any container when
>> this flag is set.
>> I've raised this issue here:
>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>> here:
>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
>> sadly no one was able to help me...
>>
>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>> marica.antonacci@ba.infn.it> napisał:
>>
>>> OK, the problem I spotted is related to the usage of the
>>> flag —docker_mesos_image that allows the executor to
>>>
>>>
>>> --docker_mesos_image=VALUEThe docker image used to launch this mesos
>>> slave instance. If an image is specified, the docker containerizer assumes
>>> the slave is running in a docker container, and launches executors with
>>> docker containers in order to recover them when the slave restarts and
>>> recovers.
>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>
>>> Thank you
>>> Marica
>>>
>>>
>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha
>>> scritto:
>>>
>>> no docker_mesos_image flag in my docker run, and the docker image is
>>> build by myself.
>>>
>>>
>>>
>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it
>>> >:
>>>
>>> Yes, I did check inside the container and the csv file was not
>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>
>>>> Are you running your slave with the --docker_mesos_image flag? Can you
>>>> please provide me the docker run command you are using to run your
>>>> dockerized slave?
>>>>
>>>> Thank you very much
>>>>
>>> Marica
>>>>
>>>>
>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>
>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com>
>>>> ha scritto:
>>>>
>>>> Hi Marica,
>>>>
>>>> use your test-app json, i can run it correctly, the csv is truely
>>>> download by mesos slave. please check mesos-master:5050 to check the task
>>>> detail download files.
>>>>
>>>> you describe the app container why not found the csv, because the csv
>>>> is download in slave container's folder, not in app container. so if you
>>>> run
>>>>
>>>> cd $MESOS_SANDBOX;
>>>>
>>>> the folder in app container is default value:
>>>>
>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>> but in real world, the sandbox is in slave container, not in app
>>>> container.
>>>>
>>>>
>>>>
>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>> marica.antonacci@ba.infn.it>:
>>>>
>>>>> Thank you very much,
>>>>>
>>>>> I’m using a sample application definition file, just for testing
>>>>> purpose:
>>>>>
>>>>> {
>>>>> "id": "test-app",
>>>>> "container": {
>>>>> "type": "DOCKER",
>>>>> "docker": {
>>>>> "image": "libmesos/ubuntu"
>>>>> }
>>>>> },
>>>>> "cpus": 1,
>>>>> "mem": 512,
>>>>> *"uris": [
>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>> ],*
>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>>>>> done"
>>>>> }
>>>>>
>>>>> Here is the docker run command line:
>>>>>
>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e
>>>>> MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>> --privileged --pid host mesos-slave
>>>>>
>>>>>
>>>>> As already mentioned, if I remove the environment variable
>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>> snoqualmie.csv inside the sandbox.
>>>>>
>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>> Best regards,
>>>>> Marica
>>>>>
>>>>>
>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com>
>>>>> ha scritto:
>>>>>
>>>>> Hi Marica,
>>>>>
>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>
>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it>:
>>>>>
>>>>>> No, using the socket:
>>>>>>
>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>
>>>>>>
>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com>
>>>>>> ha scritto:
>>>>>>
>>>>>> docker in docker mode?
>>>>>>
>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>>> inside a docker container (using *docker_mesos_image* startup flag)
>>>>>>> and you submit the deployment of a dockerized application or job the
>>>>>>> fetcher step is not performed. On the other hand, if I request the
>>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>
>>>>>>> Therefore, it seems that the information about the URIs gets lost
>>>>>>> when the dockerized mesos slave spawns the executor docker container that
>>>>>>> in turn launches the application docker container…Has anyone seen this
>>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>
>>>>>>> Thanks a lot in advance for you help
>>>>>>> Best Regards,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Deshi Xiao
>>>>>> Twitter: xds2000
>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Deshi Xiao
>>>> Twitter: xds2000
>>>> E-mail: xiaods(AT)gmail.com
>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>> 2015-12-18 下午4.55.38.png>
>>>>
>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Deshi Xiao
>>> Twitter: xds2000
>>> E-mail: xiaods(AT)gmail.com
>>>
>>>
>>> ----------------------------------------------------------
>>>
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
Re: Mesos fetcher in dockerized slave
Posted by Shuai Lin <li...@gmail.com>.
The problem happens to me if I don't specify the --docker_mesos_image flag.
However, specifying the flag only makes things worse: the task is failed
again and agin, but there does exist a container for this task.
master and zookeeper is running on host, and slave is running inside a
docker image:
```
sudo docker run -it --rm \
-e MESOS_HOSTNAME=localhost \
-e MESOS_IP=127.0.0.1 \
-e MESOS_MASTER=zk://127.0.0.1:2181/mesos \
-v /sys/fs/cgroup:/sys/fs/cgroup \
-v /var/run/docker.sock:/var/run/docker.sock \
--name mesos-slave \
--net host \
--privileged \
mesoscloud/mesos-slave:0.24.1-ubuntu-14.04
```
However my setup may affect the outcome: master is 0.25.0 and slave is
0.24.1 (can't find a public docker image for mesos 2.5.1)
Output of http http://127.0.0.1:8080/v2/apps (unrelevant part ommited)
```
{
"apps": [
"container": {
"docker": {
"parameters": [],
"privileged": false,
"network": "BRIDGE",
"image": "testapp:latest"
},
"volumes": [],
"type": "DOCKER"
},
"uris": [
"https://google.com/robots.txt"
],
}
]
}
```
On Fri, Dec 18, 2015 at 7:11 PM, Grzegorz Graczyk <gr...@gmail.com>
wrote:
> I've tried to use this flag, but cannot really run any container when this
> flag is set.
> I've raised this issue here:
> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
> here:
> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
> sadly no one was able to help me...
>
> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
> marica.antonacci@ba.infn.it> napisał:
>
>> OK, the problem I spotted is related to the usage of the
>> flag —docker_mesos_image that allows the executor to
>>
>>
>> --docker_mesos_image=VALUEThe docker image used to launch this mesos
>> slave instance. If an image is specified, the docker containerizer assumes
>> the slave is running in a docker container, and launches executors with
>> docker containers in order to recover them when the slave restarts and
>> recovers.
>> Has anyone used this flag and tested the behavior of the fetcher?
>>
>> Thank you
>> Marica
>>
>>
>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha
>> scritto:
>>
>> no docker_mesos_image flag in my docker run, and the docker image is
>> build by myself.
>>
>>
>>
>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>
>> :
>>
>> Yes, I did check inside the container and the csv file was not downloaded
>>> as shown also by the app details (see the screenshot below).
>>>
>>> Are you running your slave with the --docker_mesos_image flag? Can you
>>> please provide me the docker run command you are using to run your
>>> dockerized slave?
>>>
>>> Thank you very much
>>>
>> Marica
>>>
>>>
>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>
>>
>>>
>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha
>>> scritto:
>>>
>>> Hi Marica,
>>>
>>> use your test-app json, i can run it correctly, the csv is truely
>>> download by mesos slave. please check mesos-master:5050 to check the task
>>> detail download files.
>>>
>>> you describe the app container why not found the csv, because the csv is
>>> download in slave container's folder, not in app container. so if you run
>>>
>>> cd $MESOS_SANDBOX;
>>>
>>> the folder in app container is default value:
>>>
>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>> but in real world, the sandbox is in slave container, not in app
>>> container.
>>>
>>>
>>>
>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it
>>> >:
>>>
>>>> Thank you very much,
>>>>
>>>> I’m using a sample application definition file, just for testing
>>>> purpose:
>>>>
>>>> {
>>>> "id": "test-app",
>>>> "container": {
>>>> "type": "DOCKER",
>>>> "docker": {
>>>> "image": "libmesos/ubuntu"
>>>> }
>>>> },
>>>> "cpus": 1,
>>>> "mem": 512,
>>>> *"uris": [
>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>> ],*
>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>>>> done"
>>>> }
>>>>
>>>> Here is the docker run command line:
>>>>
>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e
>>>> MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>> --privileged --pid host mesos-slave
>>>>
>>>>
>>>> As already mentioned, if I remove the environment variable
>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>> snoqualmie.csv inside the sandbox.
>>>>
>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>> Best regards,
>>>> Marica
>>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com>
>>>> ha scritto:
>>>>
>>>> Hi Marica,
>>>>
>>>> Could you please give a sample marathon json, i can test it asap.
>>>>
>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it
>>>> >:
>>>>
>>>>> No, using the socket:
>>>>>
>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>
>>>>>
>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com>
>>>>> ha scritto:
>>>>>
>>>>> docker in docker mode?
>>>>>
>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>> marica.antonacci@ba.infn.it>:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>> inside a docker container (using *docker_mesos_image* startup flag)
>>>>>> and you submit the deployment of a dockerized application or job the
>>>>>> fetcher step is not performed. On the other hand, if I request the
>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>
>>>>>> Therefore, it seems that the information about the URIs gets lost
>>>>>> when the dockerized mesos slave spawns the executor docker container that
>>>>>> in turn launches the application docker container…Has anyone seen this
>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>
>>>>>> Thanks a lot in advance for you help
>>>>>> Best Regards,
>>>>>> Marica
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Deshi Xiao
>>>> Twitter: xds2000
>>>> E-mail: xiaods(AT)gmail.com
>>>>
>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Deshi Xiao
>>> Twitter: xds2000
>>> E-mail: xiaods(AT)gmail.com
>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>> 2015-12-18 下午4.55.38.png>
>>>
>>>
>>> ----------------------------------------------------------
>>>
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
Re: Mesos fetcher in dockerized slave
Posted by Grzegorz Graczyk <gr...@gmail.com>.
I've tried to use this flag, but cannot really run any container when this
flag is set.
I've raised this issue here:
https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and here:
https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351
but
sadly no one was able to help me...
pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
marica.antonacci@ba.infn.it> napisał:
> OK, the problem I spotted is related to the usage of the
> flag —docker_mesos_image that allows the executor to
>
>
> --docker_mesos_image=VALUEThe docker image used to launch this mesos
> slave instance. If an image is specified, the docker containerizer assumes
> the slave is running in a docker container, and launches executors with
> docker containers in order to recover them when the slave restarts and
> recovers.
> Has anyone used this flag and tested the behavior of the fetcher?
>
> Thank you
> Marica
>
>
> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha
> scritto:
>
> no docker_mesos_image flag in my docker run, and the docker image is
> build by myself.
>
>
>
> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>
> Yes, I did check inside the container and the csv file was not downloaded
>> as shown also by the app details (see the screenshot below).
>>
>> Are you running your slave with the --docker_mesos_image flag? Can you
>> please provide me the docker run command you are using to run your
>> dockerized slave?
>>
>> Thank you very much
>>
> Marica
>>
>>
>> <Schermata 2015-12-18 alle 10.15.58.png>
>>
>
>>
>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha
>> scritto:
>>
>> Hi Marica,
>>
>> use your test-app json, i can run it correctly, the csv is truely
>> download by mesos slave. please check mesos-master:5050 to check the task
>> detail download files.
>>
>> you describe the app container why not found the csv, because the csv is
>> download in slave container's folder, not in app container. so if you run
>>
>> cd $MESOS_SANDBOX;
>>
>> the folder in app container is default value:
>>
>> MESOS_SANDBOX=/mnt/mesos/sandbox
>> but in real world, the sandbox is in slave container, not in app
>> container.
>>
>>
>>
>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>
>> :
>>
>>> Thank you very much,
>>>
>>> I’m using a sample application definition file, just for testing purpose:
>>>
>>> {
>>> "id": "test-app",
>>> "container": {
>>> "type": "DOCKER",
>>> "docker": {
>>> "image": "libmesos/ubuntu"
>>> }
>>> },
>>> "cpus": 1,
>>> "mem": 512,
>>> *"uris": [
>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>> ],*
>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>>> done"
>>> }
>>>
>>> Here is the docker run command line:
>>>
>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e
>>> MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>> --privileged --pid host mesos-slave
>>>
>>>
>>> As already mentioned, if I remove the environment variable
>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>> snoqualmie.csv inside the sandbox.
>>>
>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>> Best regards,
>>> Marica
>>>
>>>
>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha
>>> scritto:
>>>
>>> Hi Marica,
>>>
>>> Could you please give a sample marathon json, i can test it asap.
>>>
>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>
>>> :
>>>
>>>> No, using the socket:
>>>>
>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>
>>>>
>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com>
>>>> ha scritto:
>>>>
>>>> docker in docker mode?
>>>>
>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>> marica.antonacci@ba.infn.it>:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I'm testing the URIs fetching mechanism for both Marathon applications
>>>>> and Chronos jobs and I have found that if the slave is running inside a
>>>>> docker container (using *docker_mesos_image* startup flag) and you
>>>>> submit the deployment of a dockerized application or job the fetcher step
>>>>> is not performed. On the other hand, if I request the deployment of a
>>>>> non-dockerized application, the URIs are correctly fetched. Moreover, if I
>>>>> don’t provide the docker_mesos_image flag, the fetcher works fine again for
>>>>> both dockerized and non-dockerized applications.
>>>>>
>>>>> Therefore, it seems that the information about the URIs gets lost when
>>>>> the dockerized mesos slave spawns the executor docker container that in
>>>>> turn launches the application docker container…Has anyone seen this problem
>>>>> before? I would like to know if there is a workaround or a fixing.
>>>>>
>>>>> Thanks a lot in advance for you help
>>>>> Best Regards,
>>>>> Marica
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Deshi Xiao
>>>> Twitter: xds2000
>>>> E-mail: xiaods(AT)gmail.com
>>>>
>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Deshi Xiao
>>> Twitter: xds2000
>>> E-mail: xiaods(AT)gmail.com
>>>
>>>
>>> ----------------------------------------------------------
>>>
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>> 2015-12-18 下午4.55.38.png>
>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
>
>
> ----------------------------------------------------------
>
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
OK, the problem I spotted is related to the usage of the flag —docker_mesos_image that allows the executor to
--docker_mesos_image=VALUE The docker image used to launch this mesos slave instance. If an image is specified, the docker containerizer assumes the slave is running in a docker container, and launches executors with docker containers in order to recover them when the slave restarts and recovers.
Has anyone used this flag and tested the behavior of the fetcher?
Thank you
Marica
Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xi...@gmail.com> ha scritto:
> no docker_mesos_image flag in my docker run, and the docker image is build by myself.
>
>
>
> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
> Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
>
> Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
>
> Thank you very much
> Marica
>
>
> <Schermata 2015-12-18 alle 10.15.58.png>
>
>
> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
>
>> Hi Marica,
>>
>> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>>
>> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>>
>> cd $MESOS_SANDBOX;
>>
>> the folder in app container is default value:
>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>
>> but in real world, the sandbox is in slave container, not in app container.
>>
>>
>>
>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>> Thank you very much,
>>
>> I’m using a sample application definition file, just for testing purpose:
>>
>> {
>> "id": "test-app",
>> "container": {
>> "type": "DOCKER",
>> "docker": {
>> "image": "libmesos/ubuntu"
>> }
>> },
>> "cpus": 1,
>> "mem": 512,
>> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
>> }
>>
>> Here is the docker run command line:
>>
>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>>
>>
>> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>>
>> Thank you again! I’m looking forward to hearing about your outcomes.
>> Best regards,
>> Marica
>>
>>
>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>>
>>> Hi Marica,
>>>
>>> Could you please give a sample marathon json, i can test it asap.
>>>
>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>> No, using the socket:
>>>
>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>
>>>
>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>>
>>>> docker in docker mode?
>>>>
>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>>> Dear all,
>>>>
>>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>>
>>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>>
>>>> Thanks a lot in advance for you help
>>>> Best Regards,
>>>> Marica
>>>>
>>>>
>>>> ----------------------------------------------------------
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antonacci@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Deshi Xiao
>>>> Twitter: xds2000
>>>> E-mail: xiaods(AT)gmail.com
>>>
>>> ----------------------------------------------------------
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Deshi Xiao
>>> Twitter: xds2000
>>> E-mail: xiaods(AT)gmail.com
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
Yes, I did check inside the container and the csv file was not downloaded as shown also by the app details (see the screenshot below).
Are you running your slave with the --docker_mesos_image flag? Can you please provide me the docker run command you are using to run your dockerized slave?
Thank you very much
Marica
Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xi...@gmail.com> ha scritto:
> Hi Marica,
>
> use your test-app json, i can run it correctly, the csv is truely download by mesos slave. please check mesos-master:5050 to check the task detail download files.
>
> you describe the app container why not found the csv, because the csv is download in slave container's folder, not in app container. so if you run
>
> cd $MESOS_SANDBOX;
>
> the folder in app container is default value:
> MESOS_SANDBOX=/mnt/mesos/sandbox
>
> but in real world, the sandbox is in slave container, not in app container.
>
>
>
> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
> Thank you very much,
>
> I’m using a sample application definition file, just for testing purpose:
>
> {
> "id": "test-app",
> "container": {
> "type": "DOCKER",
> "docker": {
> "image": "libmesos/ubuntu"
> }
> },
> "cpus": 1,
> "mem": 512,
> "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
> }
>
> Here is the docker run command line:
>
> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
> -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
>
>
> As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
>
> Thank you again! I’m looking forward to hearing about your outcomes.
> Best regards,
> Marica
>
>
> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
>
>> Hi Marica,
>>
>> Could you please give a sample marathon json, i can test it asap.
>>
>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>> No, using the socket:
>>
>> -v /var/run/docker.sock:/var/run/docker.sock
>>
>>
>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>>
>>> docker in docker mode?
>>>
>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>>> Dear all,
>>>
>>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>>
>>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>>
>>> Thanks a lot in advance for you help
>>> Best Regards,
>>> Marica
>>>
>>>
>>> ----------------------------------------------------------
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Deshi Xiao
>>> Twitter: xds2000
>>> E-mail: xiaods(AT)gmail.com
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png>
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by tommy xiao <xi...@gmail.com>.
Hi Marica,
use your test-app json, i can run it correctly, the csv is truely download
by mesos slave. please check mesos-master:5050 to check the task detail
download files.
you describe the app container why not found the csv, because the csv is
download in slave container's folder, not in app container. so if you run
cd $MESOS_SANDBOX;
the folder in app container is default value:
MESOS_SANDBOX=/mnt/mesos/sandbox
but in real world, the sandbox is in slave container, not in app container.
2015-12-18 16:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
> Thank you very much,
>
> I’m using a sample application definition file, just for testing purpose:
>
> {
> "id": "test-app",
> "container": {
> "type": "DOCKER",
> "docker": {
> "image": "libmesos/ubuntu"
> }
> },
> "cpus": 1,
> "mem": 512,
> *"uris": [
> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
> ],*
> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
> done"
> }
>
> Here is the docker run command line:
>
> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
> -e MESOS_CONTAINERIZERS=docker,mesos \
> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e
> MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
> -v /sys/fs/cgroup:/sys/fs/cgroup -v
> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
> --privileged --pid host mesos-slave
>
>
> As already mentioned, if I remove the environment variable
> MESOS_docker_mesos_image the fetcher works fine and I can see the file
> snoqualmie.csv inside the sandbox.
>
> Thank you again! I’m looking forward to hearing about your outcomes.
> Best regards,
> Marica
>
>
> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha
> scritto:
>
> Hi Marica,
>
> Could you please give a sample marathon json, i can test it asap.
>
> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>
>> No, using the socket:
>>
>> -v /var/run/docker.sock:/var/run/docker.sock
>>
>>
>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha
>> scritto:
>>
>> docker in docker mode?
>>
>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>
>> :
>>
>>> Dear all,
>>>
>>> I'm testing the URIs fetching mechanism for both Marathon applications
>>> and Chronos jobs and I have found that if the slave is running inside a
>>> docker container (using *docker_mesos_image* startup flag) and you
>>> submit the deployment of a dockerized application or job the fetcher step
>>> is not performed. On the other hand, if I request the deployment of a
>>> non-dockerized application, the URIs are correctly fetched. Moreover, if I
>>> don’t provide the docker_mesos_image flag, the fetcher works fine again for
>>> both dockerized and non-dockerized applications.
>>>
>>> Therefore, it seems that the information about the URIs gets lost when
>>> the dockerized mesos slave spawns the executor docker container that in
>>> turn launches the application docker container…Has anyone seen this problem
>>> before? I would like to know if there is a workaround or a fixing.
>>>
>>> Thanks a lot in advance for you help
>>> Best Regards,
>>> Marica
>>>
>>>
>>> ----------------------------------------------------------
>>>
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antonacci@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
>
>
> ----------------------------------------------------------
>
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
Thank you very much,
I’m using a sample application definition file, just for testing purpose:
{
"id": "test-app",
"container": {
"type": "DOCKER",
"docker": {
"image": "libmesos/ubuntu"
}
},
"cpus": 1,
"mem": 512,
"uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv" ],
"cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done"
}
Here is the docker run command line:
# docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZERS=docker,mesos \
-e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
-v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock --name slave --net host --privileged --pid host mesos-slave
As already mentioned, if I remove the environment variable MESOS_docker_mesos_image the fetcher works fine and I can see the file snoqualmie.csv inside the sandbox.
Thank you again! I’m looking forward to hearing about your outcomes.
Best regards,
Marica
Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xi...@gmail.com> ha scritto:
> Hi Marica,
>
> Could you please give a sample marathon json, i can test it asap.
>
> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
> No, using the socket:
>
> -v /var/run/docker.sock:/var/run/docker.sock
>
>
> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
>
>> docker in docker mode?
>>
>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>> Dear all,
>>
>> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>>
>> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>>
>> Thanks a lot in advance for you help
>> Best Regards,
>> Marica
>>
>>
>> ----------------------------------------------------------
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Deshi Xiao
>> Twitter: xds2000
>> E-mail: xiaods(AT)gmail.com
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
Re: Mesos fetcher in dockerized slave
Posted by tommy xiao <xi...@gmail.com>.
Hi Marica,
Could you please give a sample marathon json, i can test it asap.
2015-12-18 5:11 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
> No, using the socket:
>
> -v /var/run/docker.sock:/var/run/docker.sock
>
>
> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha
> scritto:
>
> docker in docker mode?
>
> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>
>> Dear all,
>>
>> I'm testing the URIs fetching mechanism for both Marathon applications
>> and Chronos jobs and I have found that if the slave is running inside a
>> docker container (using *docker_mesos_image* startup flag) and you
>> submit the deployment of a dockerized application or job the fetcher step
>> is not performed. On the other hand, if I request the deployment of a
>> non-dockerized application, the URIs are correctly fetched. Moreover, if I
>> don’t provide the docker_mesos_image flag, the fetcher works fine again for
>> both dockerized and non-dockerized applications.
>>
>> Therefore, it seems that the information about the URIs gets lost when
>> the dockerized mesos slave spawns the executor docker container that in
>> turn launches the application docker container…Has anyone seen this problem
>> before? I would like to know if there is a workaround or a fixing.
>>
>> Thanks a lot in advance for you help
>> Best Regards,
>> Marica
>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
>
>
> ----------------------------------------------------------
>
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
Re: Mesos fetcher in dockerized slave
Posted by Tim Chen <ti...@mesosphere.io>.
Can you share exactly how you run the slave in a docker container?
Tim
On Thu, Dec 17, 2015 at 1:11 PM, Marica Antonacci <
marica.antonacci@ba.infn.it> wrote:
> No, using the socket:
>
> -v /var/run/docker.sock:/var/run/docker.sock
>
>
> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha
> scritto:
>
> docker in docker mode?
>
> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
>
>> Dear all,
>>
>> I'm testing the URIs fetching mechanism for both Marathon applications
>> and Chronos jobs and I have found that if the slave is running inside a
>> docker container (using *docker_mesos_image* startup flag) and you
>> submit the deployment of a dockerized application or job the fetcher step
>> is not performed. On the other hand, if I request the deployment of a
>> non-dockerized application, the URIs are correctly fetched. Moreover, if I
>> don’t provide the docker_mesos_image flag, the fetcher works fine again for
>> both dockerized and non-dockerized applications.
>>
>> Therefore, it seems that the information about the URIs gets lost when
>> the dockerized mesos slave spawns the executor docker container that in
>> turn launches the application docker container…Has anyone seen this problem
>> before? I would like to know if there is a workaround or a fixing.
>>
>> Thanks a lot in advance for you help
>> Best Regards,
>> Marica
>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antonacci@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
>
>
> ----------------------------------------------------------
>
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
Re: Mesos fetcher in dockerized slave
Posted by Marica Antonacci <ma...@ba.infn.it>.
No, using the socket:
-v /var/run/docker.sock:/var/run/docker.sock
Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xi...@gmail.com> ha scritto:
> docker in docker mode?
>
> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <ma...@ba.infn.it>:
> Dear all,
>
> I'm testing the URIs fetching mechanism for both Marathon applications and Chronos jobs and I have found that if the slave is running inside a docker container (using docker_mesos_image startup flag) and you submit the deployment of a dockerized application or job the fetcher step is not performed. On the other hand, if I request the deployment of a non-dockerized application, the URIs are correctly fetched. Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works fine again for both dockerized and non-dockerized applications.
>
> Therefore, it seems that the information about the URIs gets lost when the dockerized mesos slave spawns the executor docker container that in turn launches the application docker container…Has anyone seen this problem before? I would like to know if there is a workaround or a fixing.
>
> Thanks a lot in advance for you help
> Best Regards,
> Marica
>
>
> ----------------------------------------------------------
> Marica ANTONACCI
> INFN - National Institute of Nuclear Physics
> Via Orabona 4
> 70126 Bari - ITALY
> Phone +39 080 5443244
> Skype: marica.antonacci
> e-mail marica.antonacci@ba.infn.it
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
----------------------------------------------------------
Marica ANTONACCI
INFN - National Institute of Nuclear Physics
Via Orabona 4
70126 Bari - ITALY
Phone +39 080 5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it