You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Bharath Ravi <bh...@gmail.com> on 2011/11/07 22:54:36 UTC
Parallel mapred jobs in Yarn
Hi all,
I have a beginner's question:
How can I configure yarn to allow multiple parallel mapreduce jobs to run?
Currently, the execution is sequential: each submitted job waits for the
previous to run.
In MR1, this could be done by enabling the fairscheduler/capacityscheduler.
Is there a similar configuration in Yarn as well?
Thanks a lot!
--
Bharath Ravi
Re: Parallel mapred jobs in Yarn
Posted by Bharath Ravi <bh...@gmail.com>.
Thanks a lot all!
I'll try configuring the capacity scheduler on yarn, and get back to you if
I run into any issues.
On 9 November 2011 10:41, Robert Evans <ev...@yahoo-inc.com> wrote:
> The configuration options are somewhat different for yarn, then they are
> for MRV1. You probably want to generate the documentation for yarn
>
> mvn site
>
> And then read through it about how to set up your cluster
>
>
> ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/target/site/index.html
>
> There is documentation about setting up the capacity scheduler too. If
> you run into any issues then reply here and if the documentation needs to
> be cleaned up was can file a JIRA against the documentation. The
> documentation is kind of new so it would be good to get some real feedback
> on it.
>
> --Bobby Evans
>
> On 11/9/11 3:07 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com>
> wrote:
>
> FairScheduler isn't ported yet to YARN. The default scheduler is there (
> which is FifoScheduler) and CapacityScheduler can be configured too.
>
> HTH,
> +Vinod
>
>
> On Wed, Nov 9, 2011 at 10:14 AM, Bharath Ravi <bharathravi1@gmail.com
> >wrote:
>
> > Thanks, Prashant!
> > I'll try Yarn out with the Fairscheduler.
> >
> > On 8 November 2011 01:01, Prashant Sharma <pr...@gmail.com>
> > wrote:
> >
> > > Yes! , you can do the same in yarn as well.
> > > -P
> > >
> > > On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <bh...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have a beginner's question:
> > > > How can I configure yarn to allow multiple parallel mapreduce jobs to
> > > run?
> > > > Currently, the execution is sequential: each submitted job waits for
> > the
> > > > previous to run.
> > > >
> > > > In MR1, this could be done by enabling the
> > > fairscheduler/capacityscheduler.
> > > > Is there a similar configuration in Yarn as well?
> > > >
> > > > Thanks a lot!
> > > > --
> > > > Bharath Ravi
> > > >
> > >
> >
> >
> >
> > --
> > Bharath Ravi
> >
>
>
--
Bharath Ravi
Re: Parallel mapred jobs in Yarn
Posted by Robert Evans <ev...@yahoo-inc.com>.
The configuration options are somewhat different for yarn, then they are for MRV1. You probably want to generate the documentation for yarn
mvn site
And then read through it about how to set up your cluster
./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/target/site/index.html
There is documentation about setting up the capacity scheduler too. If you run into any issues then reply here and if the documentation needs to be cleaned up was can file a JIRA against the documentation. The documentation is kind of new so it would be good to get some real feedback on it.
--Bobby Evans
On 11/9/11 3:07 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
FairScheduler isn't ported yet to YARN. The default scheduler is there (
which is FifoScheduler) and CapacityScheduler can be configured too.
HTH,
+Vinod
On Wed, Nov 9, 2011 at 10:14 AM, Bharath Ravi <bh...@gmail.com>wrote:
> Thanks, Prashant!
> I'll try Yarn out with the Fairscheduler.
>
> On 8 November 2011 01:01, Prashant Sharma <pr...@gmail.com>
> wrote:
>
> > Yes! , you can do the same in yarn as well.
> > -P
> >
> > On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <bh...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I have a beginner's question:
> > > How can I configure yarn to allow multiple parallel mapreduce jobs to
> > run?
> > > Currently, the execution is sequential: each submitted job waits for
> the
> > > previous to run.
> > >
> > > In MR1, this could be done by enabling the
> > fairscheduler/capacityscheduler.
> > > Is there a similar configuration in Yarn as well?
> > >
> > > Thanks a lot!
> > > --
> > > Bharath Ravi
> > >
> >
>
>
>
> --
> Bharath Ravi
>
Re: Parallel mapred jobs in Yarn
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
FairScheduler isn't ported yet to YARN. The default scheduler is there (
which is FifoScheduler) and CapacityScheduler can be configured too.
HTH,
+Vinod
On Wed, Nov 9, 2011 at 10:14 AM, Bharath Ravi <bh...@gmail.com>wrote:
> Thanks, Prashant!
> I'll try Yarn out with the Fairscheduler.
>
> On 8 November 2011 01:01, Prashant Sharma <pr...@gmail.com>
> wrote:
>
> > Yes! , you can do the same in yarn as well.
> > -P
> >
> > On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <bh...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I have a beginner's question:
> > > How can I configure yarn to allow multiple parallel mapreduce jobs to
> > run?
> > > Currently, the execution is sequential: each submitted job waits for
> the
> > > previous to run.
> > >
> > > In MR1, this could be done by enabling the
> > fairscheduler/capacityscheduler.
> > > Is there a similar configuration in Yarn as well?
> > >
> > > Thanks a lot!
> > > --
> > > Bharath Ravi
> > >
> >
>
>
>
> --
> Bharath Ravi
>
Re: Parallel mapred jobs in Yarn
Posted by Bharath Ravi <bh...@gmail.com>.
Thanks, Prashant!
I'll try Yarn out with the Fairscheduler.
On 8 November 2011 01:01, Prashant Sharma <pr...@gmail.com> wrote:
> Yes! , you can do the same in yarn as well.
> -P
>
> On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <bh...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I have a beginner's question:
> > How can I configure yarn to allow multiple parallel mapreduce jobs to
> run?
> > Currently, the execution is sequential: each submitted job waits for the
> > previous to run.
> >
> > In MR1, this could be done by enabling the
> fairscheduler/capacityscheduler.
> > Is there a similar configuration in Yarn as well?
> >
> > Thanks a lot!
> > --
> > Bharath Ravi
> >
>
--
Bharath Ravi
Re: Parallel mapred jobs in Yarn
Posted by Prashant Sharma <pr...@gmail.com>.
Yes! , you can do the same in yarn as well.
-P
On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <bh...@gmail.com> wrote:
> Hi all,
>
> I have a beginner's question:
> How can I configure yarn to allow multiple parallel mapreduce jobs to run?
> Currently, the execution is sequential: each submitted job waits for the
> previous to run.
>
> In MR1, this could be done by enabling the fairscheduler/capacityscheduler.
> Is there a similar configuration in Yarn as well?
>
> Thanks a lot!
> --
> Bharath Ravi
>