You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Yang <te...@gmail.com> on 2015/04/28 00:30:33 UTC
in YARN/MR2, can I still submit multiple jobs to one MR application master?
conceptually, the MR application master is similar to the old job tracker.
if so, can I submit multiple jobs to the same MR application master? it
looks like an odd use case, the context is that we have users generating
lots of MR jobs, and he currently has a little crude scheduler that
periodically launches jobs to the RM by just "hadoop jar ..."
instead I was thinking to "carve out" a MR2 allocation in RM first, then
periodically submit to the "job tracker"/application master, so that all
the jobs are localized to this allocation.
I was also thinking about using Tez instead of MR application master. Tez
replaces MR2 application master, not on top of it, right?
Thanks
Yang
Re: in YARN/MR2, can I still submit multiple jobs to one MR
application master?
Posted by Yang <te...@gmail.com>.
Vinod:
thanks.
the queue is the correct way to go. but a small technical issue is that in
our (as in most) environment, the most users share one "default" queue, and
more importantly, even if they have a special queue, the queue limit is
defined by ops team, not the user himself. i.e. he can not "self-police"
his queue usage: if he has a queue with max 100 tasks running, he can not
limit that down to 50
Yang
On Mon, Apr 27, 2015 at 5:07 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
> The MapReduce ApplicationMaster supports only one job. You can say that
> (YARN ResourceManager + a bunch of MR ApplicationMasters (one per job) =
> JobTracker).
>
> Tez does have a notion of multiple DAGs per YARN app.
>
> For your specific use-case, you can force that user to a queue and limit
> how much he/she can access.
>
> Thanks
> +Vinod
>
> On Apr 27, 2015, at 3:30 PM, Yang <te...@gmail.com> wrote:
>
> > conceptually, the MR application master is similar to the old job
> tracker.
> >
> > if so, can I submit multiple jobs to the same MR application master? it
> looks like an odd use case, the context is that we have users generating
> lots of MR jobs, and he currently has a little crude scheduler that
> periodically launches jobs to the RM by just "hadoop jar ..."
> >
> > instead I was thinking to "carve out" a MR2 allocation in RM first, then
> periodically submit to the "job tracker"/application master, so that all
> the jobs are localized to this allocation.
> >
> >
> > I was also thinking about using Tez instead of MR application master.
> Tez replaces MR2 application master, not on top of it, right?
> >
> > Thanks
> > Yang
>
>
Re: in YARN/MR2, can I still submit multiple jobs to one MR
application master?
Posted by Yang <te...@gmail.com>.
Vinod:
thanks.
the queue is the correct way to go. but a small technical issue is that in
our (as in most) environment, the most users share one "default" queue, and
more importantly, even if they have a special queue, the queue limit is
defined by ops team, not the user himself. i.e. he can not "self-police"
his queue usage: if he has a queue with max 100 tasks running, he can not
limit that down to 50
Yang
On Mon, Apr 27, 2015 at 5:07 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
> The MapReduce ApplicationMaster supports only one job. You can say that
> (YARN ResourceManager + a bunch of MR ApplicationMasters (one per job) =
> JobTracker).
>
> Tez does have a notion of multiple DAGs per YARN app.
>
> For your specific use-case, you can force that user to a queue and limit
> how much he/she can access.
>
> Thanks
> +Vinod
>
> On Apr 27, 2015, at 3:30 PM, Yang <te...@gmail.com> wrote:
>
> > conceptually, the MR application master is similar to the old job
> tracker.
> >
> > if so, can I submit multiple jobs to the same MR application master? it
> looks like an odd use case, the context is that we have users generating
> lots of MR jobs, and he currently has a little crude scheduler that
> periodically launches jobs to the RM by just "hadoop jar ..."
> >
> > instead I was thinking to "carve out" a MR2 allocation in RM first, then
> periodically submit to the "job tracker"/application master, so that all
> the jobs are localized to this allocation.
> >
> >
> > I was also thinking about using Tez instead of MR application master.
> Tez replaces MR2 application master, not on top of it, right?
> >
> > Thanks
> > Yang
>
>
Re: in YARN/MR2, can I still submit multiple jobs to one MR
application master?
Posted by Yang <te...@gmail.com>.
Vinod:
thanks.
the queue is the correct way to go. but a small technical issue is that in
our (as in most) environment, the most users share one "default" queue, and
more importantly, even if they have a special queue, the queue limit is
defined by ops team, not the user himself. i.e. he can not "self-police"
his queue usage: if he has a queue with max 100 tasks running, he can not
limit that down to 50
Yang
On Mon, Apr 27, 2015 at 5:07 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
> The MapReduce ApplicationMaster supports only one job. You can say that
> (YARN ResourceManager + a bunch of MR ApplicationMasters (one per job) =
> JobTracker).
>
> Tez does have a notion of multiple DAGs per YARN app.
>
> For your specific use-case, you can force that user to a queue and limit
> how much he/she can access.
>
> Thanks
> +Vinod
>
> On Apr 27, 2015, at 3:30 PM, Yang <te...@gmail.com> wrote:
>
> > conceptually, the MR application master is similar to the old job
> tracker.
> >
> > if so, can I submit multiple jobs to the same MR application master? it
> looks like an odd use case, the context is that we have users generating
> lots of MR jobs, and he currently has a little crude scheduler that
> periodically launches jobs to the RM by just "hadoop jar ..."
> >
> > instead I was thinking to "carve out" a MR2 allocation in RM first, then
> periodically submit to the "job tracker"/application master, so that all
> the jobs are localized to this allocation.
> >
> >
> > I was also thinking about using Tez instead of MR application master.
> Tez replaces MR2 application master, not on top of it, right?
> >
> > Thanks
> > Yang
>
>
Re: in YARN/MR2, can I still submit multiple jobs to one MR
application master?
Posted by Yang <te...@gmail.com>.
Vinod:
thanks.
the queue is the correct way to go. but a small technical issue is that in
our (as in most) environment, the most users share one "default" queue, and
more importantly, even if they have a special queue, the queue limit is
defined by ops team, not the user himself. i.e. he can not "self-police"
his queue usage: if he has a queue with max 100 tasks running, he can not
limit that down to 50
Yang
On Mon, Apr 27, 2015 at 5:07 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
> The MapReduce ApplicationMaster supports only one job. You can say that
> (YARN ResourceManager + a bunch of MR ApplicationMasters (one per job) =
> JobTracker).
>
> Tez does have a notion of multiple DAGs per YARN app.
>
> For your specific use-case, you can force that user to a queue and limit
> how much he/she can access.
>
> Thanks
> +Vinod
>
> On Apr 27, 2015, at 3:30 PM, Yang <te...@gmail.com> wrote:
>
> > conceptually, the MR application master is similar to the old job
> tracker.
> >
> > if so, can I submit multiple jobs to the same MR application master? it
> looks like an odd use case, the context is that we have users generating
> lots of MR jobs, and he currently has a little crude scheduler that
> periodically launches jobs to the RM by just "hadoop jar ..."
> >
> > instead I was thinking to "carve out" a MR2 allocation in RM first, then
> periodically submit to the "job tracker"/application master, so that all
> the jobs are localized to this allocation.
> >
> >
> > I was also thinking about using Tez instead of MR application master.
> Tez replaces MR2 application master, not on top of it, right?
> >
> > Thanks
> > Yang
>
>
Re: in YARN/MR2, can I still submit multiple jobs to one MR
application master?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The MapReduce ApplicationMaster supports only one job. You can say that (YARN ResourceManager + a bunch of MR ApplicationMasters (one per job) = JobTracker).
Tez does have a notion of multiple DAGs per YARN app.
For your specific use-case, you can force that user to a queue and limit how much he/she can access.
Thanks
+Vinod
On Apr 27, 2015, at 3:30 PM, Yang <te...@gmail.com> wrote:
> conceptually, the MR application master is similar to the old job tracker.
>
> if so, can I submit multiple jobs to the same MR application master? it looks like an odd use case, the context is that we have users generating lots of MR jobs, and he currently has a little crude scheduler that periodically launches jobs to the RM by just "hadoop jar ..."
>
> instead I was thinking to "carve out" a MR2 allocation in RM first, then periodically submit to the "job tracker"/application master, so that all the jobs are localized to this allocation.
>
>
> I was also thinking about using Tez instead of MR application master. Tez replaces MR2 application master, not on top of it, right?
>
> Thanks
> Yang
Re: in YARN/MR2, can I still submit multiple jobs to one MR
application master?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The MapReduce ApplicationMaster supports only one job. You can say that (YARN ResourceManager + a bunch of MR ApplicationMasters (one per job) = JobTracker).
Tez does have a notion of multiple DAGs per YARN app.
For your specific use-case, you can force that user to a queue and limit how much he/she can access.
Thanks
+Vinod
On Apr 27, 2015, at 3:30 PM, Yang <te...@gmail.com> wrote:
> conceptually, the MR application master is similar to the old job tracker.
>
> if so, can I submit multiple jobs to the same MR application master? it looks like an odd use case, the context is that we have users generating lots of MR jobs, and he currently has a little crude scheduler that periodically launches jobs to the RM by just "hadoop jar ..."
>
> instead I was thinking to "carve out" a MR2 allocation in RM first, then periodically submit to the "job tracker"/application master, so that all the jobs are localized to this allocation.
>
>
> I was also thinking about using Tez instead of MR application master. Tez replaces MR2 application master, not on top of it, right?
>
> Thanks
> Yang
Re: in YARN/MR2, can I still submit multiple jobs to one MR
application master?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The MapReduce ApplicationMaster supports only one job. You can say that (YARN ResourceManager + a bunch of MR ApplicationMasters (one per job) = JobTracker).
Tez does have a notion of multiple DAGs per YARN app.
For your specific use-case, you can force that user to a queue and limit how much he/she can access.
Thanks
+Vinod
On Apr 27, 2015, at 3:30 PM, Yang <te...@gmail.com> wrote:
> conceptually, the MR application master is similar to the old job tracker.
>
> if so, can I submit multiple jobs to the same MR application master? it looks like an odd use case, the context is that we have users generating lots of MR jobs, and he currently has a little crude scheduler that periodically launches jobs to the RM by just "hadoop jar ..."
>
> instead I was thinking to "carve out" a MR2 allocation in RM first, then periodically submit to the "job tracker"/application master, so that all the jobs are localized to this allocation.
>
>
> I was also thinking about using Tez instead of MR application master. Tez replaces MR2 application master, not on top of it, right?
>
> Thanks
> Yang
Re: in YARN/MR2, can I still submit multiple jobs to one MR
application master?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The MapReduce ApplicationMaster supports only one job. You can say that (YARN ResourceManager + a bunch of MR ApplicationMasters (one per job) = JobTracker).
Tez does have a notion of multiple DAGs per YARN app.
For your specific use-case, you can force that user to a queue and limit how much he/she can access.
Thanks
+Vinod
On Apr 27, 2015, at 3:30 PM, Yang <te...@gmail.com> wrote:
> conceptually, the MR application master is similar to the old job tracker.
>
> if so, can I submit multiple jobs to the same MR application master? it looks like an odd use case, the context is that we have users generating lots of MR jobs, and he currently has a little crude scheduler that periodically launches jobs to the RM by just "hadoop jar ..."
>
> instead I was thinking to "carve out" a MR2 allocation in RM first, then periodically submit to the "job tracker"/application master, so that all the jobs are localized to this allocation.
>
>
> I was also thinking about using Tez instead of MR application master. Tez replaces MR2 application master, not on top of it, right?
>
> Thanks
> Yang