You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mesos.apache.org by Łukasz Adamczyk <lu...@adamczyk.czest.pl> on 2015/10/19 18:05:08 UTC

Agent with External Containerizer ignores mesos.internal.RunTaskMessage

Hello,

TL;DR;

I have odd problem with Mesos Agent + External Containerizer: I'm able to
schedule only 14 tasks per agent (sic!), after reaching that limit my
mesos-agent ignores mesos.internal.RunTaskMessage from mesos-master.

More details:

Web UI with 14/20 tasks with RUNNING status:
http://foobar.krakow.pl/mesosproblem/mesos-master-ui1.png
mesos-agent output with GLOG_V=20 (short version):
http://foobar.krakow.pl/mesosproblem/mesos-slave-tldr.txt
mesos-agent output with GLOG_V=20 (full version - 0.8MB):
http://foobar.krakow.pl/mesosproblem/mesos-slave.txt
mesos-master output with GLOG_V=20 (full version - 1.8MB):
http://foobar.krakow.pl/mesosproblem/mesos-master.txt
Web UI after mesos.internal.ShutdownMessage (due to health check timeout):
http://foobar.krakow.pl/mesosproblem/mesos-master-ui2.png

Tested on mesos 0.25 and 0.23, with Marathon 0.10 + External Containerizer
+ Custom Executor (libmesos 0.23). Can you suggest me what is wrong? Thanks
in advance!

Regards,
Łukasz

-- 
Łukasz Adamczyk

Re: Agent with External Containerizer ignores mesos.internal.RunTaskMessage

Posted by Łukasz Adamczyk <lu...@adamczyk.czest.pl>.

Hello dev@

We still didn't found any solution, that's why I decided to prepare
environment showing that problem: https://github.com/klocekPL/mesos-ec-debug
and simpler External Containerizer Program that runs mesos-executor without
any container to simplify code:
https://github.com/klocekPL/mesos-ec-debug/blob/master/ecp/ecp.py but show
general idea in our solution.

Provisioning

git clone https://github.com/klocekPL/mesos-ec-debug
cd mesos-ec-debug

vagrant up

Default configuration: 1 x mesos-master (1CPU, 512MB) + 1 x mesos-slave
(1CPU, 1GB)

Mesos Master WebUI

http://192.168.255.10:5050

Marathon WebUI

http://192.168.255.10:8080

Steps to reproduce problem

1. Add task

curl -X POST -H "Content-Type: application/json"
http://192.168.255.10:8080/v2/apps -d '
{
    "id": "sleep",
    "cmd": "sleep 1000000",
    "instances": 2,
    "cpus": 0.1,
    "mem": 16
}'

2. Check Mesos-master console: http://192.168.255.10:5050 Should work fine.

3. Scale up to 16+ instances (per node)

4. See how Mesos-slave will die :(

* env tested on Vagrant 1.7.x and VirtualBox 5.x

I'm open for any suggestions, thanks in advance!

PS: I found that someone had the same problem:
https://issues.apache.org/jira/browse/MESOS-1473

Regards,
Łukasz


On 28 October 2015 at 15:58, Łukasz Adamczyk <lu...@adamczyk.czest.pl>
wrote:

> First thing: External Containerizer will be deprecated. I know that and we
> are open for moving that functionality to module, but at this moment we
> need to use External Containerizer that should work, am I right?
>
> Second thing: We decided to make our External Containerizer and Executor
> open source: https://github.com/AVSystem/mesoslxc
>
> Does anyone have any idea what can be wrong?
>
> Regards,
> Łukasz
>
> On 21 October 2015 at 16:19, haosdent <ha...@gmail.com> wrote:
>
>> seems could not found the implemention of your External Containerizer.
>> -> seems
>> could not found the implemention of your External Containerizer in email.
>>
>> On Wed, Oct 21, 2015 at 10:19 PM, haosdent <ha...@gmail.com> wrote:
>>
>> > Sounds like your External Containerizer have problems and seems could
>> not
>> > found the implemention of your External Containerizer. By the way, do
>> you
>> > know External Containerizer is deprecated and we would let Containerizer
>> > become module. http://search-hadoop.com/m/0Vlr6yasmiJToni
>> >
>> > On Wed, Oct 21, 2015 at 10:06 PM, Łukasz Adamczyk <
>> > lukasz@adamczyk.czest.pl> wrote:
>> >
>> >> Same problem with /help http://pastebin.com/2KUPDWv9 Of course when I
>> use
>> >> Mesos or Docker Containerizer it works well, but we need our custom LXC
>> >> based containers that's why we need External Containerizer.
>> >>
>> >> Thanks for reply,
>> >> Łukasz
>> >>
>> >> On 21 October 2015 at 15:38, haosdent <ha...@gmail.com> wrote:
>> >>
>> >> > And is it possible for you to use MesosContainerizer or
>> >> DockerContainerizer
>> >> > instead of ExternalContainerizer?
>> >> >
>> >> > On Wed, Oct 21, 2015 at 9:34 PM, haosdent <ha...@gmail.com>
>> wrote:
>> >> >
>> >> > > Could "curl -vv http://10.0.0.184:5051/help" success?
>> >> > >
>> >> > > On Wed, Oct 21, 2015 at 9:30 PM, Łukasz Adamczyk <
>> >> > lukasz@adamczyk.czest.pl
>> >> > > > wrote:
>> >> > >
>> >> > >> Hello,
>> >> > >>
>> >> > >> Any ideas? Maybe someone can suggest me what I should check? IMHO
>> >> > >> mesos-agent hangs, it accepts new TCP connections (port 5051), so
>> >> > service
>> >> > >> is still listening, but suddenly stopped sending HTTP responses:
>> >> > >> http://pastebin.com/QsyMft7j
>> >> > >>
>> >> > >> Regards,
>> >> > >> Łukasz
>> >> > >>
>> >> > >> On 19 October 2015 at 18:05, Łukasz Adamczyk <
>> >> lukasz@adamczyk.czest.pl>
>> >> > >> wrote:
>> >> > >>
>> >> > >> > Hello,
>> >> > >> >
>> >> > >> > TL;DR;
>> >> > >> >
>> >> > >> > I have odd problem with Mesos Agent + External Containerizer:
>> I'm
>> >> able
>> >> > >> to
>> >> > >> > schedule only 14 tasks per agent (sic!), after reaching that
>> limit
>> >> my
>> >> > >> > mesos-agent ignores mesos.internal.RunTaskMessage from
>> >> mesos-master.
>> >> > >> >
>> >> > >> > More details:
>> >> > >> >
>> >> > >> > Web UI with 14/20 tasks with RUNNING status:
>> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-master-ui1.png
>> >> > >> > mesos-agent output with GLOG_V=20 (short version):
>> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-slave-tldr.txt
>> >> > >> > mesos-agent output with GLOG_V=20 (full version - 0.8MB):
>> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-slave.txt
>> >> > >> > mesos-master output with GLOG_V=20 (full version - 1.8MB):
>> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-master.txt
>> >> > >> > Web UI after mesos.internal.ShutdownMessage (due to health check
>> >> > >> timeout):
>> >> > >> > http://foobar.krakow.pl/mesosproblem/mesos-master-ui2.png
>> >> > >> >
>> >> > >> > Tested on mesos 0.25 and 0.23, with Marathon 0.10 + External
>> >> > >> Containerizer
>> >> > >> > + Custom Executor (libmesos 0.23). Can you suggest me what is
>> >> wrong?
>> >> > >> Thanks
>> >> > >> > in advance!
>> >> > >> >
>> >> > >> > Regards,
>> >> > >> > Łukasz
>> >> > >> >
>> >> > >> > --
>> >> > >> > Łukasz Adamczyk
>> >> > >> >
>> >> > >>
>> >> > >> --
>> >> > >> Łukasz Adamczyk
>> >> > >>
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Best Regards,
>> >> > > Haosdent Huang
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Best Regards,
>> >> > Haosdent Huang
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Łukasz Adamczyk
>> >>
>> >
>> >
>> >
>> > --
>> > Best Regards,
>> > Haosdent Huang
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
>
>
> --
> Łukasz Adamczyk
>



-- 
Łukasz Adamczyk