You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Steven Schlansker <ss...@opentable.com> on 2014/09/05 01:23:12 UTC
Trying out Docker containerizer: fails with no interesting output
I am trying to integrate the Docker containerizer into the Singularity framework (https://github.com/HubSpot/Singularity)
I have filled in my containerInfo:
"containerInfo": {
"type": "DOCKER",
"docker": {
"image": "registry.mesos-vpcqa.otenv.com/demo-server:latest-master"
}
}
I am leaving ‘command’ blank as I wish to have the default command executed.
Interestingly enough, the Docker containerizer seems to pass on the TaskInfo:
I0904 23:12:13.941849 32497 exec.cpp:132] Version: 0.20.0
I0904 23:12:13.947185 32500 exec.cpp:206] Executor registered on slave 20140826-180432-1594050058-5050-23467-6
F0904 23:12:13.949040 32502 executor.cpp:134] Check failed: task.command().has_value() Shell command of task DemoService-DemoService.18-1409872328807-2-10-us_west_2a is not specified!
*** Check failure stack trace: ***
@ 0x7f54dc76b4bd google::LogMessage::Fail()
@ 0x7f54dc76d2fd google::LogMessage::SendToLog()
@ 0x7f54dc76b0ac google::LogMessage::Flush()
@ 0x7f54dc76dbf9 google::LogMessageFatal::~LogMessageFatal()
@ 0x42abc2 mesos::internal::CommandExecutorProcess::launchTask()
@ 0x7f54dc7160e2 process::ProcessManager::resume()
@ 0x7f54dc7163ec process::schedule()
@ 0x7f54db190182 start_thread
@ 0x7f54daebd38d (unknown)
Aborted
but if I add some (otherwise unnecessary) URIs to fetch it then is happy to accept it
"uris": [ "https://my.s3.amazonaws.com/something.tar.gz”]
then I start to see Docker containerizer output in the slave log:
I0904 22:58:24.092098 1288 docker.cpp:815] Starting container '479dedef-4312-4441-be72-6570caaf2b8a' for task 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' (and executor 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a') of framework 'Singularity'
I0904 22:58:28.327430 1288 slave.cpp:2535] Monitoring executor 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' of framework 'Singularity' in container '479dedef-4312-4441-be72-6570caaf2b8a'
I0904 22:58:28.368155 1291 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: a428c917-973d-4c7e-a8a3-43fdf9b49e99) for task DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework Singularity
I0904 22:58:29.331042 1291 slave.cpp:2087] Handling status update TASK_FINISHED (UUID: de9b8003-9b61-4b87-98fc-32b01f875a95) for task DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework Singularity from executor(1)@10.70.6.186:56465
I0904 22:58:30.689954 1291 docker.cpp:1720] Executor for container '479dedef-4312-4441-be72-6570caaf2b8a' has exited
I0904 22:58:30.690076 1291 docker.cpp:1545] Destroying container '479dedef-4312-4441-be72-6570caaf2b8a'
I0904 22:58:30.690500 1291 docker.cpp:1642] Running docker kill on container '479dedef-4312-4441-be72-6570caaf2b8a'
stderr:
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0904 22:58:24.317865 26387 fetcher.cpp:126] Downloading ‘xxx' to '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
I0904 22:58:24.387418 26387 fetcher.cpp:64] Extracted resource '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx' into '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a'
I0904 22:58:27.870132 26811 exec.cpp:132] Version: 0.20.0
I0904 22:58:28.155580 26819 exec.cpp:206] Executor registered on slave 20140826-180432-1594050058-5050-23467-8
stdout:
Registered executor on 10.70.6.186
Starting task DemoService-DemoService.15-1409871503777-2-10-us_west_2a
Forked command at 26949
/bin/sh -c exit `docker wait mesos-479dedef-4312-4441-be72-6570caaf2b8a`
Command exited with status 0 (pid: 26949)
When I try to lookup the Docker container with ‘docker ps -a’ it looks like no container was ever created, and then the ‘docker wait’ exits immediately.
Any clues as to how I could debug such a situation?
Thanks!
Steven
Re: Trying out Docker containerizer: fails with no interesting output
Posted by David Greenberg <ds...@gmail.com>.
For those interested, the ticket is
https://issues.apache.org/jira/browse/MESOS-1730
On Fri, Sep 5, 2014 at 4:23 PM, Steven Schlansker <sschlansker@opentable.com
> wrote:
> Thanks, that was definitely a gotcha!
>
> On Sep 4, 2014, at 4:27 PM, David Greenberg <ds...@gmail.com>
> wrote:
>
> > Even though command is blank, you must set shell to false. There's a
> ticket for this that I don't have off-hand.
> >
> > On Thursday, September 4, 2014, Steven Schlansker <
> sschlansker@opentable.com> wrote:
> > I am trying to integrate the Docker containerizer into the Singularity
> framework (https://github.com/HubSpot/Singularity)
> >
> > I have filled in my containerInfo:
> > "containerInfo": {
> > "type": "DOCKER",
> > "docker": {
> > "image": "registry.mesos-vpcqa.otenv.com/demo-server:latest-master
> "
> > }
> > }
> >
> > I am leaving ‘command’ blank as I wish to have the default command
> executed.
> >
> > Interestingly enough, the Docker containerizer seems to pass on the
> TaskInfo:
> > I0904 23:12:13.941849 32497 exec.cpp:132] Version: 0.20.0
> > I0904 23:12:13.947185 32500 exec.cpp:206] Executor registered on slave
> 20140826-180432-1594050058-5050-23467-6
> > F0904 23:12:13.949040 32502 executor.cpp:134] Check failed:
> task.command().has_value() Shell command of task
> DemoService-DemoService.18-1409872328807-2-10-us_west_2a is not specified!
> > *** Check failure stack trace: ***
> > @ 0x7f54dc76b4bd google::LogMessage::Fail()
> > @ 0x7f54dc76d2fd google::LogMessage::SendToLog()
> > @ 0x7f54dc76b0ac google::LogMessage::Flush()
> > @ 0x7f54dc76dbf9 google::LogMessageFatal::~LogMessageFatal()
> > @ 0x42abc2
> mesos::internal::CommandExecutorProcess::launchTask()
> > @ 0x7f54dc7160e2 process::ProcessManager::resume()
> > @ 0x7f54dc7163ec process::schedule()
> > @ 0x7f54db190182 start_thread
> > @ 0x7f54daebd38d (unknown)
> > Aborted
> >
> > but if I add some (otherwise unnecessary) URIs to fetch it then is happy
> to accept it
> >
> > "uris": [ "https://my.s3.amazonaws.com/something.tar.gz”]
> >
> > then I start to see Docker containerizer output in the slave log:
> >
> > I0904 22:58:24.092098 1288 docker.cpp:815] Starting container
> '479dedef-4312-4441-be72-6570caaf2b8a' for task
> 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' (and executor
> 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a') of framework
> 'Singularity'
> > I0904 22:58:28.327430 1288 slave.cpp:2535] Monitoring executor
> 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' of framework
> 'Singularity' in container '479dedef-4312-4441-be72-6570caaf2b8a'
> > I0904 22:58:28.368155 1291 status_update_manager.cpp:320] Received
> status update TASK_RUNNING (UUID: a428c917-973d-4c7e-a8a3-43fdf9b49e99) for
> task DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework
> Singularity
> > I0904 22:58:29.331042 1291 slave.cpp:2087] Handling status update
> TASK_FINISHED (UUID: de9b8003-9b61-4b87-98fc-32b01f875a95) for task
> DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework
> Singularity from executor(1)@10.70.6.186:56465
> > I0904 22:58:30.689954 1291 docker.cpp:1720] Executor for container
> '479dedef-4312-4441-be72-6570caaf2b8a' has exited
> > I0904 22:58:30.690076 1291 docker.cpp:1545] Destroying container
> '479dedef-4312-4441-be72-6570caaf2b8a'
> > I0904 22:58:30.690500 1291 docker.cpp:1642] Running docker kill on
> container '479dedef-4312-4441-be72-6570caaf2b8a'
> >
> > stderr:
> > WARNING: Logging before InitGoogleLogging() is written to STDERR
> > I0904 22:58:24.317865 26387 fetcher.cpp:126] Downloading ‘xxx' to
> '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
> > I0904 22:58:24.387418 26387 fetcher.cpp:64] Extracted resource
> '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
> into
> '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a'
> > I0904 22:58:27.870132 26811 exec.cpp:132] Version: 0.20.0
> > I0904 22:58:28.155580 26819 exec.cpp:206] Executor registered on slave
> 20140826-180432-1594050058-5050-23467-8
> >
> > stdout:
> >
> > Registered executor on 10.70.6.186
> > Starting task DemoService-DemoService.15-1409871503777-2-10-us_west_2a
> > Forked command at 26949
> > /bin/sh -c exit `docker wait mesos-479dedef-4312-4441-be72-6570caaf2b8a`
> > Command exited with status 0 (pid: 26949)
> >
> > When I try to lookup the Docker container with ‘docker ps -a’ it looks
> like no container was ever created, and then the ‘docker wait’ exits
> immediately.
> >
> > Any clues as to how I could debug such a situation?
> > Thanks!
> > Steven
>
>
Re: Trying out Docker containerizer: fails with no interesting output
Posted by Steven Schlansker <ss...@opentable.com>.
Thanks, that was definitely a gotcha!
On Sep 4, 2014, at 4:27 PM, David Greenberg <ds...@gmail.com> wrote:
> Even though command is blank, you must set shell to false. There's a ticket for this that I don't have off-hand.
>
> On Thursday, September 4, 2014, Steven Schlansker <ss...@opentable.com> wrote:
> I am trying to integrate the Docker containerizer into the Singularity framework (https://github.com/HubSpot/Singularity)
>
> I have filled in my containerInfo:
> "containerInfo": {
> "type": "DOCKER",
> "docker": {
> "image": "registry.mesos-vpcqa.otenv.com/demo-server:latest-master"
> }
> }
>
> I am leaving ‘command’ blank as I wish to have the default command executed.
>
> Interestingly enough, the Docker containerizer seems to pass on the TaskInfo:
> I0904 23:12:13.941849 32497 exec.cpp:132] Version: 0.20.0
> I0904 23:12:13.947185 32500 exec.cpp:206] Executor registered on slave 20140826-180432-1594050058-5050-23467-6
> F0904 23:12:13.949040 32502 executor.cpp:134] Check failed: task.command().has_value() Shell command of task DemoService-DemoService.18-1409872328807-2-10-us_west_2a is not specified!
> *** Check failure stack trace: ***
> @ 0x7f54dc76b4bd google::LogMessage::Fail()
> @ 0x7f54dc76d2fd google::LogMessage::SendToLog()
> @ 0x7f54dc76b0ac google::LogMessage::Flush()
> @ 0x7f54dc76dbf9 google::LogMessageFatal::~LogMessageFatal()
> @ 0x42abc2 mesos::internal::CommandExecutorProcess::launchTask()
> @ 0x7f54dc7160e2 process::ProcessManager::resume()
> @ 0x7f54dc7163ec process::schedule()
> @ 0x7f54db190182 start_thread
> @ 0x7f54daebd38d (unknown)
> Aborted
>
> but if I add some (otherwise unnecessary) URIs to fetch it then is happy to accept it
>
> "uris": [ "https://my.s3.amazonaws.com/something.tar.gz”]
>
> then I start to see Docker containerizer output in the slave log:
>
> I0904 22:58:24.092098 1288 docker.cpp:815] Starting container '479dedef-4312-4441-be72-6570caaf2b8a' for task 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' (and executor 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a') of framework 'Singularity'
> I0904 22:58:28.327430 1288 slave.cpp:2535] Monitoring executor 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' of framework 'Singularity' in container '479dedef-4312-4441-be72-6570caaf2b8a'
> I0904 22:58:28.368155 1291 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: a428c917-973d-4c7e-a8a3-43fdf9b49e99) for task DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework Singularity
> I0904 22:58:29.331042 1291 slave.cpp:2087] Handling status update TASK_FINISHED (UUID: de9b8003-9b61-4b87-98fc-32b01f875a95) for task DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework Singularity from executor(1)@10.70.6.186:56465
> I0904 22:58:30.689954 1291 docker.cpp:1720] Executor for container '479dedef-4312-4441-be72-6570caaf2b8a' has exited
> I0904 22:58:30.690076 1291 docker.cpp:1545] Destroying container '479dedef-4312-4441-be72-6570caaf2b8a'
> I0904 22:58:30.690500 1291 docker.cpp:1642] Running docker kill on container '479dedef-4312-4441-be72-6570caaf2b8a'
>
> stderr:
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I0904 22:58:24.317865 26387 fetcher.cpp:126] Downloading ‘xxx' to '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
> I0904 22:58:24.387418 26387 fetcher.cpp:64] Extracted resource '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx' into '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a'
> I0904 22:58:27.870132 26811 exec.cpp:132] Version: 0.20.0
> I0904 22:58:28.155580 26819 exec.cpp:206] Executor registered on slave 20140826-180432-1594050058-5050-23467-8
>
> stdout:
>
> Registered executor on 10.70.6.186
> Starting task DemoService-DemoService.15-1409871503777-2-10-us_west_2a
> Forked command at 26949
> /bin/sh -c exit `docker wait mesos-479dedef-4312-4441-be72-6570caaf2b8a`
> Command exited with status 0 (pid: 26949)
>
> When I try to lookup the Docker container with ‘docker ps -a’ it looks like no container was ever created, and then the ‘docker wait’ exits immediately.
>
> Any clues as to how I could debug such a situation?
> Thanks!
> Steven
Re: Trying out Docker containerizer: fails with no interesting output
Posted by David Greenberg <ds...@gmail.com>.
Even though command is blank, you must set shell to false. There's a ticket
for this that I don't have off-hand.
On Thursday, September 4, 2014, Steven Schlansker <ss...@opentable.com>
wrote:
> I am trying to integrate the Docker containerizer into the Singularity
> framework (https://github.com/HubSpot/Singularity)
>
> I have filled in my containerInfo:
> "containerInfo": {
> "type": "DOCKER",
> "docker": {
> "image": "registry.mesos-vpcqa.otenv.com/demo-server:latest-master"
> }
> }
>
> I am leaving ‘command’ blank as I wish to have the default command
> executed.
>
> Interestingly enough, the Docker containerizer seems to pass on the
> TaskInfo:
> I0904 23:12:13.941849 32497 exec.cpp:132] Version: 0.20.0
> I0904 23:12:13.947185 32500 exec.cpp:206] Executor registered on slave
> 20140826-180432-1594050058-5050-23467-6
> F0904 23:12:13.949040 32502 executor.cpp:134] Check failed:
> task.command().has_value() Shell command of task
> DemoService-DemoService.18-1409872328807-2-10-us_west_2a is not specified!
> *** Check failure stack trace: ***
> @ 0x7f54dc76b4bd google::LogMessage::Fail()
> @ 0x7f54dc76d2fd google::LogMessage::SendToLog()
> @ 0x7f54dc76b0ac google::LogMessage::Flush()
> @ 0x7f54dc76dbf9 google::LogMessageFatal::~LogMessageFatal()
> @ 0x42abc2
> mesos::internal::CommandExecutorProcess::launchTask()
> @ 0x7f54dc7160e2 process::ProcessManager::resume()
> @ 0x7f54dc7163ec process::schedule()
> @ 0x7f54db190182 start_thread
> @ 0x7f54daebd38d (unknown)
> Aborted
>
> but if I add some (otherwise unnecessary) URIs to fetch it then is happy
> to accept it
>
> "uris": [ "https://my.s3.amazonaws.com/something.tar.gz”]
>
> then I start to see Docker containerizer output in the slave log:
>
> I0904 22:58:24.092098 1288 docker.cpp:815] Starting container
> '479dedef-4312-4441-be72-6570caaf2b8a' for task
> 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' (and executor
> 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a') of framework
> 'Singularity'
> I0904 22:58:28.327430 1288 slave.cpp:2535] Monitoring executor
> 'DemoService-DemoService.15-1409871503777-2-10-us_west_2a' of framework
> 'Singularity' in container '479dedef-4312-4441-be72-6570caaf2b8a'
> I0904 22:58:28.368155 1291 status_update_manager.cpp:320] Received status
> update TASK_RUNNING (UUID: a428c917-973d-4c7e-a8a3-43fdf9b49e99) for task
> DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework
> Singularity
> I0904 22:58:29.331042 1291 slave.cpp:2087] Handling status update
> TASK_FINISHED (UUID: de9b8003-9b61-4b87-98fc-32b01f875a95) for task
> DemoService-DemoService.15-1409871503777-2-10-us_west_2a of framework
> Singularity from executor(1)@10.70.6.186:56465
> I0904 22:58:30.689954 1291 docker.cpp:1720] Executor for container
> '479dedef-4312-4441-be72-6570caaf2b8a' has exited
> I0904 22:58:30.690076 1291 docker.cpp:1545] Destroying container
> '479dedef-4312-4441-be72-6570caaf2b8a'
> I0904 22:58:30.690500 1291 docker.cpp:1642] Running docker kill on
> container '479dedef-4312-4441-be72-6570caaf2b8a'
>
> stderr:
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I0904 22:58:24.317865 26387 fetcher.cpp:126] Downloading ‘xxx' to
> '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
> I0904 22:58:24.387418 26387 fetcher.cpp:64] Extracted resource
> '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a/xxx'
> into
> '/mnt/mesos/slaves/20140826-180432-1594050058-5050-23467-8/frameworks/Singularity/executors/DemoService-DemoService.15-1409871503777-2-10-us_west_2a/runs/479dedef-4312-4441-be72-6570caaf2b8a'
> I0904 22:58:27.870132 26811 exec.cpp:132] Version: 0.20.0
> I0904 22:58:28.155580 26819 exec.cpp:206] Executor registered on slave
> 20140826-180432-1594050058-5050-23467-8
>
> stdout:
>
> Registered executor on 10.70.6.186
> Starting task DemoService-DemoService.15-1409871503777-2-10-us_west_2a
> Forked command at 26949
> /bin/sh -c exit `docker wait mesos-479dedef-4312-4441-be72-6570caaf2b8a`
> Command exited with status 0 (pid: 26949)
>
> When I try to lookup the Docker container with ‘docker ps -a’ it looks
> like no container was ever created, and then the ‘docker wait’ exits
> immediately.
>
> Any clues as to how I could debug such a situation?
> Thanks!
> Steven