You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by Oskar Stenman <os...@magine.com> on 2015/03/13 15:08:59 UTC
Problem getting docker container running
Hi!
I'm investigating aurora + mesos + docker and i'm stuck.
I can create the hello world docker-task in aurora, it gets assigned a
slave, the docker-container is launched but the executor immediately
terminates and it ends up in "task lost" state.
Can anyone make any sense of this or tell me how to troubleshoot further?
--
[image: MagineTV]
*Oskar Stenman*
Network Architect
*Magine TV*
oskar.stenman@magine.com | Mob: +46 70 565 21 52
Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
<http://www.magine.com/>
Privileged and/or Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such a person), you may not
copy or deliver this message to anyone. In such case,
you should destroy this message and kindly notify the sender by reply email.
Re: Problem getting docker container running
Posted by Steve Niemitz <st...@tellapart.com>.
>From the mesos slave log it looks like the executor is failing. Most
likely the issue is your image doesn't have the native libraries needed to
run it. The next step would be to look in the sandbox for a failed run
(you can find the path in the slave logs) and look at the stderr log for
errors.
On Fri, Mar 13, 2015 at 10:08 AM, Oskar Stenman <os...@magine.com>
wrote:
> Hi!
>
> I'm investigating aurora + mesos + docker and i'm stuck.
>
> I can create the hello world docker-task in aurora, it gets assigned a
> slave, the docker-container is launched but the executor immediately
> terminates and it ends up in "task lost" state.
>
> Can anyone make any sense of this or tell me how to troubleshoot further?
>
>
> --
> [image: MagineTV]
>
> *Oskar Stenman*
> Network Architect
>
> *Magine TV*
> oskar.stenman@magine.com | Mob: +46 70 565 21 52
> Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> <http://www.magine.com/>
>
> Privileged and/or Confidential Information may be contained in this
> message. If you are not the addressee indicated in this message
> (or responsible for delivery of the message to such a person), you may not
> copy or deliver this message to anyone. In such case,
> you should destroy this message and kindly notify the sender by reply
> email.
>
Re: Problem getting docker container running
Posted by Oskar Stenman <os...@magine.com>.
The majority of the resolution to this happened on irc, #aurora / freenode,
thanks to Steve :)
i'm including it here "for logging" / searchability.
We're running all of mesos, aurora, thermos-observer, zookeeper and such in
docker containers, mesos-slave in a container managing other containers on
the same layer of "containerization" puts special requirements on the
mesos-slave container.
simple schematic showing how things are layered:
(host -> docker
docker -> mesos-slave
docker -> mesos-task
docker -> ...
)
This requires that directories in the mesos-slave is on the same path in
both the slave and on the host since mesos-slave mounts those directories
from docker (host's) perspective into the mesos-task.
The mesos-slave also seems to have it's own executor which keeps track of
the aurora-executor's pid.
If mesos-slave can't find the slave pid it will assume the task was lost.
(this was the main issue i was banging my head against since this isn't
logged anywhere if it fails, slave just prints "task_lost" and executor
just exits..)
This happens if mesos-slave is running in a docker-container where it will
be in a isolated pid-space and therefore will not be able to find the pid
of the slave-container's executor.
By running the docker-container containing mesos-slave with "--pid=host"
the mesos-slave will run in the same pid-space as the host, and will
therefore be able to find aurora-executor's pid, and things will magically
work.. :>
For this (mesos-slave in docker-container, tasks in other containers and
such) mesos-slave, thermos_observer and all slave/mesos tasks needs to be
able to access the same directories.
/sys needs to be mounted into the mesos-slave /sys directory so it can
manage cgroups.
/var/run/thermos needs to be mounted into thermos_observer's
/var/run/thermos, this will also be mounted into all slave-tasks
/var/run/thermos directory, i'm assuming so all slave-tasks can report back
to thermos_observer that they're running or not..
/tmp/mesos (or the directory configured in mesos-slave where all tasks
sandboxes are stored) needs to be mounted from host's /tmp/mesos into the
mesos-slave's /tmp/mesos, mesos-slave will mount this from the host into
each mesos-task container.
Mesos-slave docker config:
/etc/init/mesos-slave.conf
#
# This properly starts and stops a docker container through upstart.
#
description "run mesos-slave"
respawn
script
docker pull docker-repo.service.consul:5000/trusty/mesos
. /etc/default/cluster
mkdir -p /tmp/mesos
docker create -i -t --name=consul_$UPSTART_JOB --net=host \
-v /run/thermos:/run/thermos \
-v /tmp/mesos:/tmp/mesos \
-v /sys:/sys \
-v /usr/bin/docker:/usr/bin/docker \
-v /var/run/docker.sock:/var/run/docker.sock \
-e ULIMIT="-n 8192" \
-e MESOS_resources="ports:[13000-15999]" \
-e MESOS_FRAMEWORKS_HOME=/data/services/mesos/frameworks \
-e MESOS_launcher_dir=/usr/libexec/mesos \
-e MESOS_attributes="role:slave;rack:$RACK;host:`hostname`" \
--pid=host \ndocker-repo.service.consul:5000/trusty/mesos \
/usr/sbin/mesos-slave
--master=zk://zookeeper.service.consul:2181/mesos/$CLUSTER
--containerizers=docker,mesos --log_dir=/var/lib/mesos/log
--executor_registration_timeout=5mins
exec docker start -a consul_$UPSTART_JOB
end script
post-stop script
docker stop -t 0 consul_$UPSTART_JOB
docker rm consul_$UPSTART_JOB
sleep 5
end script
This doesn't seem to work properly for now. something is weird with
stdout/stderr, i'll investigate a bit further.
/etc/init/thermos-executor.conf:
#
# This properly starts and stops a docker container through upstart.
#
description "run thermos-observer"
respawn
script
docker pull docker-repo.service.consul:5000/trusty/mesos
. /etc/default/cluster
docker create -i -t --name=$UPSTART_JOB --net=host \
-v /var/run/thermos:/var/run/thermos \
docker-repo.service.consul:5000/trusty/mesos
/usr/local/bin/thermos_observer.pex --log_dir=/data/log/thermos-observer
--log_simple
exec docker start -a $UPSTART_JOB
end script
post-stop script
docker stop -t 0 $UPSTART_JOB
docker rm $UPSTART_JOB
end script
On Fri, Mar 13, 2015 at 6:18 PM, Steve Niemitz <st...@tellapart.com> wrote:
> I just noticed this line in your previous email:
> "Everything's included in the image, that's the image i'm running
> mesos-master,
> aurora, and mesos-slave from."
>
> Are you saying you're running the slave inside a docker container? If so,
> mesos does not support running docker containers from inside another docker
> container.
>
> On Fri, Mar 13, 2015 at 1:04 PM, Oskar Stenman <os...@magine.com>
> wrote:
>
> > Thanks for the reply!
> >
> > The only files i find in the workdir is stderr, stdout and
> > thermos_executor.pex
> > Nothing is cleaning the tmp directory, everything (except /data) is
> running
> > on ramdisk on these machines, it's the absolute minimal ubuntu "minbase"
> > (debootstrapped) installation i've found possible and still generally
> > usable. (around 100MB with docker and some other tools i include)
> >
> >
> > root@s1:~# ls
> >
> >
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781
> > -l
> > total 29000
> > -rw-r--r-x 1 root root 519 Mar 13 13:39 stderr
> > -rw-r--r-x 1 root root 0 Mar 13 13:39 stdout
> > -rwxr-xr-x 1 root root 29690639 Mar 13 13:39 thermos_executor.pex
> >
> > STDERR:
> > root@s1:~# cat
> >
> >
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781/stderr
> > WARNING: Logging before InitGoogleLogging() is written to STDERR
> > I0313 13:39:45.608578 258 fetcher.cpp:76] Fetching URI
> > '/usr/local/bin/thermos_executor.pex'
> > I0313 13:39:45.609048 258 fetcher.cpp:179] Copying resource from
> > '/usr/local/bin/thermos_executor.pex' to
> >
> >
> '/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
> >
> > /Oskar
> >
> > On Fri, Mar 13, 2015 at 5:54 PM, Bill Farner <wf...@apache.org> wrote:
> >
> > > When the task launches, a sandbox directory is created, in the above
> > e-mail
> > > it
> > > was
> > >
> >
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781.
> > > Is there anything else in that directory? Please post anything logs
> you
> > > find in there as they may provide useful clues.
> > >
> > > Possibly unrelated, your slave work directory is /tmp. I have observed
> > > that some environments a temp cleaner process runs that automatically
> > > deletes things under here. This may not fix the issue at hand here,
> but
> > i
> > > suggest you move this outside /tmp as it is critical state for the
> slave.
> > >
> > >
> > > -=Bill
> > >
> > > On Fri, Mar 13, 2015 at 9:24 AM, Oskar Stenman <
> oskar.stenman@magine.com
> > >
> > > wrote:
> > >
> > > > I wasn't subscribed to the mailing-list (I'm subscribed now though)
> so
> > > > i'm sorry if this reply ends up in the wrong place..
> > > >
> > > > >From the mesos slave log it looks like the executor is failing.
> Most
> > > > likely the issue is your image doesn't have the native libraries
> needed
> > > to
> > > > run it. The next step would be to look in the sandbox for a failed
> run
> > > > (you can find the path in the slave logs) and look at the stderr log
> > for
> > > > errors.
> > > >
> > > > Everything's included in the image, that's the image i'm running
> > > > mesos-master, aurora, and mesos-slave from.
> > > >
> > > > Stdout was included in the last email:
> > > >
> > > > WARNING: Logging before InitGoogleLogging() is written to STDERR
> > > > I0313 13:39:45.608578 258 fetcher.cpp:76] Fetching URI
> > > > '/usr/local/bin/thermos_executor.pex'
> > > > I0313 13:39:45.609048 258 fetcher.cpp:179] Copying resource from
> > > > '/usr/local/bin/thermos_executor.pex' to
> > > >
> > > >
> > >
> >
> '/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
> > > >
> > > > Also, when relaunching the container manually:
> > > >
> > > >
> > > > 1. root@s1:/var/log/upstart# docker start -i
> > > > mesos-cdf5e59f-c2be-47ba-b30a-2a690657e248
> > > > 2. twitter.common.app debug: Initializing: twitter.common.log
> > > > (Logging subsystem.)
> > > > 3. Writing log files to disk in /mnt/mesos/sandbox
> > > > 4. I0313 14:50:36.273181 5 exec.cpp:132] Version: 0.21.1
> > > > 5. I0313 14:50:36.278043 29 exec.cpp:379] Executor asked to
> > > shutdown
> > > > 6. Killed
> > > >
> > > >
> > > > It seems to be really tricky to troubleshoot from my point of view as
> > > > i don't have any output at all.
> > > > Is the executor giving up immediately? is it even trying to connect
> to
> > > > the slave?
> > > > Parameters to the executor missing? (like job-config or something)
> > > > Directory doesn't contain what it wants?
> > > > Network config wrong?
> > > >
> > > > /Oskar
> > > >
> > > >
> > > > On Fri, Mar 13, 2015 at 3:08 PM, Oskar Stenman <
> > oskar.stenman@magine.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi!
> > > > >
> > > > > I'm investigating aurora + mesos + docker and i'm stuck.
> > > > >
> > > > > I can create the hello world docker-task in aurora, it gets
> assigned
> > a
> > > > > slave, the docker-container is launched but the executor
> immediately
> > > > > terminates and it ends up in "task lost" state.
> > > > >
> > > > > Can anyone make any sense of this or tell me how to troubleshoot
> > > further?
> > > > >
> > > > >
> > > > > --
> > > > > [image: MagineTV]
> > > > >
> > > > > *Oskar Stenman*
> > > > > Network Architect
> > > > >
> > > > > *Magine TV*
> > > > > oskar.stenman@magine.com | Mob: +46 70 565 21 52
> > > > > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> > > > > <http://www.magine.com/>
> > > > >
> > > > > Privileged and/or Confidential Information may be contained in this
> > > > > message. If you are not the addressee indicated in this message
> > > > > (or responsible for delivery of the message to such a person), you
> > may
> > > > not
> > > > > copy or deliver this message to anyone. In such case,
> > > > > you should destroy this message and kindly notify the sender by
> reply
> > > > > email.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > [image: MagineTV]
> > > >
> > > > *Oskar Stenman*
> > > > Network Architect
> > > >
> > > > *Magine TV*
> > > > oskar.stenman@magine.com | Mob: +46 70 565 21 52
> > > > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> > > > <http://www.magine.com/>
> > > >
> > > > Privileged and/or Confidential Information may be contained in this
> > > > message. If you are not the addressee indicated in this message
> > > > (or responsible for delivery of the message to such a person), you
> may
> > > not
> > > > copy or deliver this message to anyone. In such case,
> > > > you should destroy this message and kindly notify the sender by reply
> > > > email.
> > > >
> > >
> >
> >
> >
> > --
> > [image: MagineTV]
> >
> > *Oskar Stenman*
> > Network Architect
> >
> > *Magine TV*
> > oskar.stenman@magine.com | Mob: +46 70 565 21 52
> > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> > <http://www.magine.com/>
> >
> > Privileged and/or Confidential Information may be contained in this
> > message. If you are not the addressee indicated in this message
> > (or responsible for delivery of the message to such a person), you may
> not
> > copy or deliver this message to anyone. In such case,
> > you should destroy this message and kindly notify the sender by reply
> > email.
> >
>
--
[image: MagineTV]
*Oskar Stenman*
Network Architect
*Magine TV*
oskar.stenman@magine.com | Mob: +46 70 565 21 52
Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
<http://www.magine.com/>
Privileged and/or Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such a person), you may not
copy or deliver this message to anyone. In such case,
you should destroy this message and kindly notify the sender by reply email.
Re: Problem getting docker container running
Posted by Steve Niemitz <st...@tellapart.com>.
I just noticed this line in your previous email:
"Everything's included in the image, that's the image i'm running mesos-master,
aurora, and mesos-slave from."
Are you saying you're running the slave inside a docker container? If so,
mesos does not support running docker containers from inside another docker
container.
On Fri, Mar 13, 2015 at 1:04 PM, Oskar Stenman <os...@magine.com>
wrote:
> Thanks for the reply!
>
> The only files i find in the workdir is stderr, stdout and
> thermos_executor.pex
> Nothing is cleaning the tmp directory, everything (except /data) is running
> on ramdisk on these machines, it's the absolute minimal ubuntu "minbase"
> (debootstrapped) installation i've found possible and still generally
> usable. (around 100MB with docker and some other tools i include)
>
>
> root@s1:~# ls
>
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781
> -l
> total 29000
> -rw-r--r-x 1 root root 519 Mar 13 13:39 stderr
> -rw-r--r-x 1 root root 0 Mar 13 13:39 stdout
> -rwxr-xr-x 1 root root 29690639 Mar 13 13:39 thermos_executor.pex
>
> STDERR:
> root@s1:~# cat
>
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781/stderr
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I0313 13:39:45.608578 258 fetcher.cpp:76] Fetching URI
> '/usr/local/bin/thermos_executor.pex'
> I0313 13:39:45.609048 258 fetcher.cpp:179] Copying resource from
> '/usr/local/bin/thermos_executor.pex' to
>
> '/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
>
> /Oskar
>
> On Fri, Mar 13, 2015 at 5:54 PM, Bill Farner <wf...@apache.org> wrote:
>
> > When the task launches, a sandbox directory is created, in the above
> e-mail
> > it
> > was
> >
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781.
> > Is there anything else in that directory? Please post anything logs you
> > find in there as they may provide useful clues.
> >
> > Possibly unrelated, your slave work directory is /tmp. I have observed
> > that some environments a temp cleaner process runs that automatically
> > deletes things under here. This may not fix the issue at hand here, but
> i
> > suggest you move this outside /tmp as it is critical state for the slave.
> >
> >
> > -=Bill
> >
> > On Fri, Mar 13, 2015 at 9:24 AM, Oskar Stenman <oskar.stenman@magine.com
> >
> > wrote:
> >
> > > I wasn't subscribed to the mailing-list (I'm subscribed now though) so
> > > i'm sorry if this reply ends up in the wrong place..
> > >
> > > >From the mesos slave log it looks like the executor is failing. Most
> > > likely the issue is your image doesn't have the native libraries needed
> > to
> > > run it. The next step would be to look in the sandbox for a failed run
> > > (you can find the path in the slave logs) and look at the stderr log
> for
> > > errors.
> > >
> > > Everything's included in the image, that's the image i'm running
> > > mesos-master, aurora, and mesos-slave from.
> > >
> > > Stdout was included in the last email:
> > >
> > > WARNING: Logging before InitGoogleLogging() is written to STDERR
> > > I0313 13:39:45.608578 258 fetcher.cpp:76] Fetching URI
> > > '/usr/local/bin/thermos_executor.pex'
> > > I0313 13:39:45.609048 258 fetcher.cpp:179] Copying resource from
> > > '/usr/local/bin/thermos_executor.pex' to
> > >
> > >
> >
> '/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
> > >
> > > Also, when relaunching the container manually:
> > >
> > >
> > > 1. root@s1:/var/log/upstart# docker start -i
> > > mesos-cdf5e59f-c2be-47ba-b30a-2a690657e248
> > > 2. twitter.common.app debug: Initializing: twitter.common.log
> > > (Logging subsystem.)
> > > 3. Writing log files to disk in /mnt/mesos/sandbox
> > > 4. I0313 14:50:36.273181 5 exec.cpp:132] Version: 0.21.1
> > > 5. I0313 14:50:36.278043 29 exec.cpp:379] Executor asked to
> > shutdown
> > > 6. Killed
> > >
> > >
> > > It seems to be really tricky to troubleshoot from my point of view as
> > > i don't have any output at all.
> > > Is the executor giving up immediately? is it even trying to connect to
> > > the slave?
> > > Parameters to the executor missing? (like job-config or something)
> > > Directory doesn't contain what it wants?
> > > Network config wrong?
> > >
> > > /Oskar
> > >
> > >
> > > On Fri, Mar 13, 2015 at 3:08 PM, Oskar Stenman <
> oskar.stenman@magine.com
> > >
> > > wrote:
> > >
> > > > Hi!
> > > >
> > > > I'm investigating aurora + mesos + docker and i'm stuck.
> > > >
> > > > I can create the hello world docker-task in aurora, it gets assigned
> a
> > > > slave, the docker-container is launched but the executor immediately
> > > > terminates and it ends up in "task lost" state.
> > > >
> > > > Can anyone make any sense of this or tell me how to troubleshoot
> > further?
> > > >
> > > >
> > > > --
> > > > [image: MagineTV]
> > > >
> > > > *Oskar Stenman*
> > > > Network Architect
> > > >
> > > > *Magine TV*
> > > > oskar.stenman@magine.com | Mob: +46 70 565 21 52
> > > > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> > > > <http://www.magine.com/>
> > > >
> > > > Privileged and/or Confidential Information may be contained in this
> > > > message. If you are not the addressee indicated in this message
> > > > (or responsible for delivery of the message to such a person), you
> may
> > > not
> > > > copy or deliver this message to anyone. In such case,
> > > > you should destroy this message and kindly notify the sender by reply
> > > > email.
> > > >
> > >
> > >
> > >
> > > --
> > > [image: MagineTV]
> > >
> > > *Oskar Stenman*
> > > Network Architect
> > >
> > > *Magine TV*
> > > oskar.stenman@magine.com | Mob: +46 70 565 21 52
> > > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> > > <http://www.magine.com/>
> > >
> > > Privileged and/or Confidential Information may be contained in this
> > > message. If you are not the addressee indicated in this message
> > > (or responsible for delivery of the message to such a person), you may
> > not
> > > copy or deliver this message to anyone. In such case,
> > > you should destroy this message and kindly notify the sender by reply
> > > email.
> > >
> >
>
>
>
> --
> [image: MagineTV]
>
> *Oskar Stenman*
> Network Architect
>
> *Magine TV*
> oskar.stenman@magine.com | Mob: +46 70 565 21 52
> Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> <http://www.magine.com/>
>
> Privileged and/or Confidential Information may be contained in this
> message. If you are not the addressee indicated in this message
> (or responsible for delivery of the message to such a person), you may not
> copy or deliver this message to anyone. In such case,
> you should destroy this message and kindly notify the sender by reply
> email.
>
Re: Problem getting docker container running
Posted by Oskar Stenman <os...@magine.com>.
Thanks for the reply!
The only files i find in the workdir is stderr, stdout and
thermos_executor.pex
Nothing is cleaning the tmp directory, everything (except /data) is running
on ramdisk on these machines, it's the absolute minimal ubuntu "minbase"
(debootstrapped) installation i've found possible and still generally
usable. (around 100MB with docker and some other tools i include)
root@s1:~# ls
/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781
-l
total 29000
-rw-r--r-x 1 root root 519 Mar 13 13:39 stderr
-rw-r--r-x 1 root root 0 Mar 13 13:39 stdout
-rwxr-xr-x 1 root root 29690639 Mar 13 13:39 thermos_executor.pex
STDERR:
root@s1:~# cat
/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781/stderr
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0313 13:39:45.608578 258 fetcher.cpp:76] Fetching URI
'/usr/local/bin/thermos_executor.pex'
I0313 13:39:45.609048 258 fetcher.cpp:179] Copying resource from
'/usr/local/bin/thermos_executor.pex' to
'/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
/Oskar
On Fri, Mar 13, 2015 at 5:54 PM, Bill Farner <wf...@apache.org> wrote:
> When the task launches, a sandbox directory is created, in the above e-mail
> it
> was
> /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781.
> Is there anything else in that directory? Please post anything logs you
> find in there as they may provide useful clues.
>
> Possibly unrelated, your slave work directory is /tmp. I have observed
> that some environments a temp cleaner process runs that automatically
> deletes things under here. This may not fix the issue at hand here, but i
> suggest you move this outside /tmp as it is critical state for the slave.
>
>
> -=Bill
>
> On Fri, Mar 13, 2015 at 9:24 AM, Oskar Stenman <os...@magine.com>
> wrote:
>
> > I wasn't subscribed to the mailing-list (I'm subscribed now though) so
> > i'm sorry if this reply ends up in the wrong place..
> >
> > >From the mesos slave log it looks like the executor is failing. Most
> > likely the issue is your image doesn't have the native libraries needed
> to
> > run it. The next step would be to look in the sandbox for a failed run
> > (you can find the path in the slave logs) and look at the stderr log for
> > errors.
> >
> > Everything's included in the image, that's the image i'm running
> > mesos-master, aurora, and mesos-slave from.
> >
> > Stdout was included in the last email:
> >
> > WARNING: Logging before InitGoogleLogging() is written to STDERR
> > I0313 13:39:45.608578 258 fetcher.cpp:76] Fetching URI
> > '/usr/local/bin/thermos_executor.pex'
> > I0313 13:39:45.609048 258 fetcher.cpp:179] Copying resource from
> > '/usr/local/bin/thermos_executor.pex' to
> >
> >
> '/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
> >
> > Also, when relaunching the container manually:
> >
> >
> > 1. root@s1:/var/log/upstart# docker start -i
> > mesos-cdf5e59f-c2be-47ba-b30a-2a690657e248
> > 2. twitter.common.app debug: Initializing: twitter.common.log
> > (Logging subsystem.)
> > 3. Writing log files to disk in /mnt/mesos/sandbox
> > 4. I0313 14:50:36.273181 5 exec.cpp:132] Version: 0.21.1
> > 5. I0313 14:50:36.278043 29 exec.cpp:379] Executor asked to
> shutdown
> > 6. Killed
> >
> >
> > It seems to be really tricky to troubleshoot from my point of view as
> > i don't have any output at all.
> > Is the executor giving up immediately? is it even trying to connect to
> > the slave?
> > Parameters to the executor missing? (like job-config or something)
> > Directory doesn't contain what it wants?
> > Network config wrong?
> >
> > /Oskar
> >
> >
> > On Fri, Mar 13, 2015 at 3:08 PM, Oskar Stenman <oskar.stenman@magine.com
> >
> > wrote:
> >
> > > Hi!
> > >
> > > I'm investigating aurora + mesos + docker and i'm stuck.
> > >
> > > I can create the hello world docker-task in aurora, it gets assigned a
> > > slave, the docker-container is launched but the executor immediately
> > > terminates and it ends up in "task lost" state.
> > >
> > > Can anyone make any sense of this or tell me how to troubleshoot
> further?
> > >
> > >
> > > --
> > > [image: MagineTV]
> > >
> > > *Oskar Stenman*
> > > Network Architect
> > >
> > > *Magine TV*
> > > oskar.stenman@magine.com | Mob: +46 70 565 21 52
> > > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> > > <http://www.magine.com/>
> > >
> > > Privileged and/or Confidential Information may be contained in this
> > > message. If you are not the addressee indicated in this message
> > > (or responsible for delivery of the message to such a person), you may
> > not
> > > copy or deliver this message to anyone. In such case,
> > > you should destroy this message and kindly notify the sender by reply
> > > email.
> > >
> >
> >
> >
> > --
> > [image: MagineTV]
> >
> > *Oskar Stenman*
> > Network Architect
> >
> > *Magine TV*
> > oskar.stenman@magine.com | Mob: +46 70 565 21 52
> > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> > <http://www.magine.com/>
> >
> > Privileged and/or Confidential Information may be contained in this
> > message. If you are not the addressee indicated in this message
> > (or responsible for delivery of the message to such a person), you may
> not
> > copy or deliver this message to anyone. In such case,
> > you should destroy this message and kindly notify the sender by reply
> > email.
> >
>
--
[image: MagineTV]
*Oskar Stenman*
Network Architect
*Magine TV*
oskar.stenman@magine.com | Mob: +46 70 565 21 52
Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
<http://www.magine.com/>
Privileged and/or Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such a person), you may not
copy or deliver this message to anyone. In such case,
you should destroy this message and kindly notify the sender by reply email.
Re: Problem getting docker container running
Posted by Bill Farner <wf...@apache.org>.
When the task launches, a sandbox directory is created, in the above e-mail
it
was /tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781.
Is there anything else in that directory? Please post anything logs you
find in there as they may provide useful clues.
Possibly unrelated, your slave work directory is /tmp. I have observed
that some environments a temp cleaner process runs that automatically
deletes things under here. This may not fix the issue at hand here, but i
suggest you move this outside /tmp as it is critical state for the slave.
-=Bill
On Fri, Mar 13, 2015 at 9:24 AM, Oskar Stenman <os...@magine.com>
wrote:
> I wasn't subscribed to the mailing-list (I'm subscribed now though) so
> i'm sorry if this reply ends up in the wrong place..
>
> >From the mesos slave log it looks like the executor is failing. Most
> likely the issue is your image doesn't have the native libraries needed to
> run it. The next step would be to look in the sandbox for a failed run
> (you can find the path in the slave logs) and look at the stderr log for
> errors.
>
> Everything's included in the image, that's the image i'm running
> mesos-master, aurora, and mesos-slave from.
>
> Stdout was included in the last email:
>
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I0313 13:39:45.608578 258 fetcher.cpp:76] Fetching URI
> '/usr/local/bin/thermos_executor.pex'
> I0313 13:39:45.609048 258 fetcher.cpp:179] Copying resource from
> '/usr/local/bin/thermos_executor.pex' to
>
> '/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
>
> Also, when relaunching the container manually:
>
>
> 1. root@s1:/var/log/upstart# docker start -i
> mesos-cdf5e59f-c2be-47ba-b30a-2a690657e248
> 2. twitter.common.app debug: Initializing: twitter.common.log
> (Logging subsystem.)
> 3. Writing log files to disk in /mnt/mesos/sandbox
> 4. I0313 14:50:36.273181 5 exec.cpp:132] Version: 0.21.1
> 5. I0313 14:50:36.278043 29 exec.cpp:379] Executor asked to shutdown
> 6. Killed
>
>
> It seems to be really tricky to troubleshoot from my point of view as
> i don't have any output at all.
> Is the executor giving up immediately? is it even trying to connect to
> the slave?
> Parameters to the executor missing? (like job-config or something)
> Directory doesn't contain what it wants?
> Network config wrong?
>
> /Oskar
>
>
> On Fri, Mar 13, 2015 at 3:08 PM, Oskar Stenman <os...@magine.com>
> wrote:
>
> > Hi!
> >
> > I'm investigating aurora + mesos + docker and i'm stuck.
> >
> > I can create the hello world docker-task in aurora, it gets assigned a
> > slave, the docker-container is launched but the executor immediately
> > terminates and it ends up in "task lost" state.
> >
> > Can anyone make any sense of this or tell me how to troubleshoot further?
> >
> >
> > --
> > [image: MagineTV]
> >
> > *Oskar Stenman*
> > Network Architect
> >
> > *Magine TV*
> > oskar.stenman@magine.com | Mob: +46 70 565 21 52
> > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> > <http://www.magine.com/>
> >
> > Privileged and/or Confidential Information may be contained in this
> > message. If you are not the addressee indicated in this message
> > (or responsible for delivery of the message to such a person), you may
> not
> > copy or deliver this message to anyone. In such case,
> > you should destroy this message and kindly notify the sender by reply
> > email.
> >
>
>
>
> --
> [image: MagineTV]
>
> *Oskar Stenman*
> Network Architect
>
> *Magine TV*
> oskar.stenman@magine.com | Mob: +46 70 565 21 52
> Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> <http://www.magine.com/>
>
> Privileged and/or Confidential Information may be contained in this
> message. If you are not the addressee indicated in this message
> (or responsible for delivery of the message to such a person), you may not
> copy or deliver this message to anyone. In such case,
> you should destroy this message and kindly notify the sender by reply
> email.
>
Re: Problem getting docker container running
Posted by Oskar Stenman <os...@magine.com>.
I wasn't subscribed to the mailing-list (I'm subscribed now though) so
i'm sorry if this reply ends up in the wrong place..
>From the mesos slave log it looks like the executor is failing. Most
likely the issue is your image doesn't have the native libraries needed to
run it. The next step would be to look in the sandbox for a failed run
(you can find the path in the slave logs) and look at the stderr log for
errors.
Everything's included in the image, that's the image i'm running
mesos-master, aurora, and mesos-slave from.
Stdout was included in the last email:
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0313 13:39:45.608578 258 fetcher.cpp:76] Fetching URI
'/usr/local/bin/thermos_executor.pex'
I0313 13:39:45.609048 258 fetcher.cpp:179] Copying resource from
'/usr/local/bin/thermos_executor.pex' to
'/tmp/mesos/slaves/20150313-131712-1143806393-5050-6-S0/frameworks/20150306-112428-1177360825-5050-6-0000/executors/thermos-1426253925515-docker-test-devel-hello_docker-0-a9011a74-c2a2-4cb7-b402-d383fde58c41/runs/e53d1267-e341-46f5-9759-0361c7440781'
Also, when relaunching the container manually:
1. root@s1:/var/log/upstart# docker start -i
mesos-cdf5e59f-c2be-47ba-b30a-2a690657e248
2. twitter.common.app debug: Initializing: twitter.common.log
(Logging subsystem.)
3. Writing log files to disk in /mnt/mesos/sandbox
4. I0313 14:50:36.273181 5 exec.cpp:132] Version: 0.21.1
5. I0313 14:50:36.278043 29 exec.cpp:379] Executor asked to shutdown
6. Killed
It seems to be really tricky to troubleshoot from my point of view as
i don't have any output at all.
Is the executor giving up immediately? is it even trying to connect to
the slave?
Parameters to the executor missing? (like job-config or something)
Directory doesn't contain what it wants?
Network config wrong?
/Oskar
On Fri, Mar 13, 2015 at 3:08 PM, Oskar Stenman <os...@magine.com>
wrote:
> Hi!
>
> I'm investigating aurora + mesos + docker and i'm stuck.
>
> I can create the hello world docker-task in aurora, it gets assigned a
> slave, the docker-container is launched but the executor immediately
> terminates and it ends up in "task lost" state.
>
> Can anyone make any sense of this or tell me how to troubleshoot further?
>
>
> --
> [image: MagineTV]
>
> *Oskar Stenman*
> Network Architect
>
> *Magine TV*
> oskar.stenman@magine.com | Mob: +46 70 565 21 52
> Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
> <http://www.magine.com/>
>
> Privileged and/or Confidential Information may be contained in this
> message. If you are not the addressee indicated in this message
> (or responsible for delivery of the message to such a person), you may not
> copy or deliver this message to anyone. In such case,
> you should destroy this message and kindly notify the sender by reply
> email.
>
--
[image: MagineTV]
*Oskar Stenman*
Network Architect
*Magine TV*
oskar.stenman@magine.com | Mob: +46 70 565 21 52
Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com
<http://www.magine.com/>
Privileged and/or Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such a person), you may not
copy or deliver this message to anyone. In such case,
you should destroy this message and kindly notify the sender by reply email.