Re: Please Help me about hadoop on Mesos

[Vinod Kone <vi...@gmail.com>]

>
> I have some questions about running hadoop on top of Mesos, please help me.
> 1. when a tasktracker is launched, if n cpu core are allocated to it, it
> can only launch n-1 map tasks. Could someone tell me why? And, if I want to
> run map-only job, what should I do to run n map tasks on a n cpu resource
> offer?
>

This is because 1 cpu is allocated to the task tracker itself.


> 2. After a tasktracker is launched, in what condition it's status will
> update to FINISHED?
> In my cluster, sometimes it will never end until I restart the jobtracker.
> Sometimes it will end if there is no task or job in jobtracker to run.
>

The expected case is that the task tracker is finished/killed if there is
no task/job assigned to it. If there is an idle task tracker for a long
time it's probably a bug (@brenden can correct me if the semantics have
changed around this). Some logs would help diagnose the issue.


> 3. How to use DRF with weight? I run two frameworks on mesos, and I want
> to give them different proportion of resources.
>
> Give each framework a different role (FrameworkInfo.role) and give weights
to each role via master command line flags (see --roles and --weights via
./master --help).



> Please help me!
> Thank you very much!
>
>
>

Re: Re: Re: Please Help me about hadoop on Mesos

[HUO Jing <hu...@ihep.ac.cn>]

Can we use the TaskTracker just for once?  when the tasks that assigned to the tasktracker are done, kill it and release the resource. And when there are new resource offers comming, launch a new TaskTracker to do the rest of the tasks.


> -----原始邮件-----
> 发件人: "Vinod Kone" <vi...@gmail.com>
> 发送时间: 2014年1月28日 星期二
> 收件人: dev <de...@mesos.apache.org>
> 抄送: "user@mesos.apache.org" <us...@mesos.apache.org>
> 主题: Re: Re: Please Help me about hadoop on Mesos
> 
> On Mon, Jan 27, 2014 at 10:07 AM, HUO Jing <hu...@ihep.ac.cn> wrote:
> 
> > So, at the very beginning, if all the resource are assigned to hadoop, and
> > after that, there are always enough jobs in jobtracker, is that meanning
> > that the other framework will never get resource?
> > Is it fair to do so ?
> >
> 
> That is correct. Currently there is no concept of pre-emption of resources
> in mesos. While this is likely to change in the future, in the short term
> you could reserve resources to frameworks (see --resources on ./slave
> --help) to avoid starvation.

Re: Re: Please Help me about hadoop on Mesos

[Vinod Kone <vi...@gmail.com>]

On Mon, Jan 27, 2014 at 10:07 AM, HUO Jing <hu...@ihep.ac.cn> wrote:

> So, at the very beginning, if all the resource are assigned to hadoop, and
> after that, there are always enough jobs in jobtracker, is that meanning
> that the other framework will never get resource?
> Is it fair to do so ?
>

That is correct. Currently there is no concept of pre-emption of resources
in mesos. While this is likely to change in the future, in the short term
you could reserve resources to frameworks (see --resources on ./slave
--help) to avoid starvation.

Re: Re: Please Help me about hadoop on Mesos

[Vinod Kone <vi...@gmail.com>]

On Mon, Jan 27, 2014 at 10:07 AM, HUO Jing <hu...@ihep.ac.cn> wrote:

> So, at the very beginning, if all the resource are assigned to hadoop, and
> after that, there are always enough jobs in jobtracker, is that meanning
> that the other framework will never get resource?
> Is it fair to do so ?
>

That is correct. Currently there is no concept of pre-emption of resources
in mesos. While this is likely to change in the future, in the short term
you could reserve resources to frameworks (see --resources on ./slave
--help) to avoid starvation.

Re: Re: Please Help me about hadoop on Mesos

[HUO Jing <hu...@ihep.ac.cn>]

> 2. After a tasktracker is launched, in what condition it's status will
> > update to FINISHED?
> > In my cluster, sometimes it will never end until I restart the jobtracker.
> > Sometimes it will end if there is no task or job in jobtracker to run.
> 
> The expected case is that the task tracker is finished/killed if there is
> no task/job assigned to it. If there is an idle task tracker for a long
> time it's probably a bug (@brenden can correct me if the semantics have
> changed around this). Some logs would help diagnose the issue.

So, at the very beginning, if all the resource are assigned to hadoop, and after that, there are always enough jobs in jobtracker, is that meanning that the other framework will never get resource? 
Is it fair to do so ?


> -----原始邮件-----
> 发件人: "Vinod Kone" <vi...@gmail.com>
> 发送时间: 2014年1月28日 星期二
> 收件人: dev <de...@mesos.apache.org>
> 抄送: "user@mesos.apache.org" <us...@mesos.apache.org>
> 主题: Re: Please Help me about hadoop on Mesos
> 
> >
> > I have some questions about running hadoop on top of Mesos, please help me.
> > 1. when a tasktracker is launched, if n cpu core are allocated to it, it
> > can only launch n-1 map tasks. Could someone tell me why? And, if I want to
> > run map-only job, what should I do to run n map tasks on a n cpu resource
> > offer?
> >
> 
> This is because 1 cpu is allocated to the task tracker itself.
> 
> 
> > 2. After a tasktracker is launched, in what condition it's status will
> > update to FINISHED?
> > In my cluster, sometimes it will never end until I restart the jobtracker.
> > Sometimes it will end if there is no task or job in jobtracker to run.
> >
> 
> The expected case is that the task tracker is finished/killed if there is
> no task/job assigned to it. If there is an idle task tracker for a long
> time it's probably a bug (@brenden can correct me if the semantics have
> changed around this). Some logs would help diagnose the issue.
> 
> 
> > 3. How to use DRF with weight? I run two frameworks on mesos, and I want
> > to give them different proportion of resources.
> >
> > Give each framework a different role (FrameworkInfo.role) and give weights
> to each role via master command line flags (see --roles and --weights via
> ./master --help).
> 
> 
> 
> > Please help me!
> > Thank you very much!
> >
> >
> >