You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Kai Ju Liu <ka...@tellapart.com> on 2011/10/26 23:55:21 UTC
Dropping 0.20.203 capacity scheduler into 0.20.2
Hi. I'm currently running a Hadoop cluster on Amazon's EMR service, which
appears to be the 0.20.2 codebase plus several patches from the
(deprecated?) 0.20.3 branch. I'm interested in switching from using the fair
scheduler to the capacity scheduler, but I'm also interested in the
user-limit-factor configuration parameter introduced in 0.20.203. This
parameter is not available in the EMR-supplied capacity scheduler jar, so I
was wondering if it's possible and safe to drop the 0.20.203 capacity
scheduler jar into my Hadoop library path.
Any information would be very helpful. Thanks!
Kai Ju
Re: Dropping 0.20.203 capacity scheduler into 0.20.2
Posted by Kai Ju Liu <ka...@tellapart.com>.
Hi Joey. I actually migrated from CDH3u0 to EMR a while back due to
stability issues that turned out to be completely AMI/AKI-related, so I may
consider the migration back at some point. If so, I'll definitely give Whirr
a shot. Thanks!
Kai Ju
On Wed, Oct 26, 2011 at 3:40 PM, Joey Echeverria <jo...@cloudera.com> wrote:
> You can also check out Apache Whirr (http://whirr.apache.org/) if you
> decide to roll your own Hadoop clusters on EC2. It's crazy easy to get
> a cluster up and running with it.
>
> -Joey
>
> On Wed, Oct 26, 2011 at 3:04 PM, Kai Ju Liu <ka...@tellapart.com> wrote:
> > Hi Arun. Thanks for the prompt reply! It's a bit of a bummer to hear
> that,
> > but I'll definitely look into the upgrade path. Thanks again!
> >
> > Kai Ju
> >
> > On Wed, Oct 26, 2011 at 3:01 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> >>
> >> Sorry. This mostly won't work... we have significant changes in the
> >> interface between the JobTracker and schedulers (FS/CS) b/w 20.2 and
> 20.203
> >> (performance, better limits etc.).
> >>
> >> Your best bet might be to provision Hadoop yourself on EC2 with
> 0.20.203+.
> >>
> >> Good luck!
> >>
> >> Arun
> >>
> >> On Oct 26, 2011, at 2:55 PM, Kai Ju Liu wrote:
> >>
> >> > Hi. I'm currently running a Hadoop cluster on Amazon's EMR service,
> >> > which appears to be the 0.20.2 codebase plus several patches from the
> >> > (deprecated?) 0.20.3 branch. I'm interested in switching from using
> the fair
> >> > scheduler to the capacity scheduler, but I'm also interested in the
> >> > user-limit-factor configuration parameter introduced in 0.20.203. This
> >> > parameter is not available in the EMR-supplied capacity scheduler jar,
> so I
> >> > was wondering if it's possible and safe to drop the 0.20.203 capacity
> >> > scheduler jar into my Hadoop library path.
> >> >
> >> > Any information would be very helpful. Thanks!
> >> >
> >> > Kai Ju
> >>
> >
> >
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>
Re: Dropping 0.20.203 capacity scheduler into 0.20.2
Posted by Joey Echeverria <jo...@cloudera.com>.
You can also check out Apache Whirr (http://whirr.apache.org/) if you
decide to roll your own Hadoop clusters on EC2. It's crazy easy to get
a cluster up and running with it.
-Joey
On Wed, Oct 26, 2011 at 3:04 PM, Kai Ju Liu <ka...@tellapart.com> wrote:
> Hi Arun. Thanks for the prompt reply! It's a bit of a bummer to hear that,
> but I'll definitely look into the upgrade path. Thanks again!
>
> Kai Ju
>
> On Wed, Oct 26, 2011 at 3:01 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>>
>> Sorry. This mostly won't work... we have significant changes in the
>> interface between the JobTracker and schedulers (FS/CS) b/w 20.2 and 20.203
>> (performance, better limits etc.).
>>
>> Your best bet might be to provision Hadoop yourself on EC2 with 0.20.203+.
>>
>> Good luck!
>>
>> Arun
>>
>> On Oct 26, 2011, at 2:55 PM, Kai Ju Liu wrote:
>>
>> > Hi. I'm currently running a Hadoop cluster on Amazon's EMR service,
>> > which appears to be the 0.20.2 codebase plus several patches from the
>> > (deprecated?) 0.20.3 branch. I'm interested in switching from using the fair
>> > scheduler to the capacity scheduler, but I'm also interested in the
>> > user-limit-factor configuration parameter introduced in 0.20.203. This
>> > parameter is not available in the EMR-supplied capacity scheduler jar, so I
>> > was wondering if it's possible and safe to drop the 0.20.203 capacity
>> > scheduler jar into my Hadoop library path.
>> >
>> > Any information would be very helpful. Thanks!
>> >
>> > Kai Ju
>>
>
>
--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
Re: Dropping 0.20.203 capacity scheduler into 0.20.2
Posted by Kai Ju Liu <ka...@tellapart.com>.
Hi Arun. Thanks for the prompt reply! It's a bit of a bummer to hear that,
but I'll definitely look into the upgrade path. Thanks again!
Kai Ju
On Wed, Oct 26, 2011 at 3:01 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Sorry. This mostly won't work... we have significant changes in the
> interface between the JobTracker and schedulers (FS/CS) b/w 20.2 and 20.203
> (performance, better limits etc.).
>
> Your best bet might be to provision Hadoop yourself on EC2 with 0.20.203+.
>
> Good luck!
>
> Arun
>
> On Oct 26, 2011, at 2:55 PM, Kai Ju Liu wrote:
>
> > Hi. I'm currently running a Hadoop cluster on Amazon's EMR service, which
> appears to be the 0.20.2 codebase plus several patches from the
> (deprecated?) 0.20.3 branch. I'm interested in switching from using the fair
> scheduler to the capacity scheduler, but I'm also interested in the
> user-limit-factor configuration parameter introduced in 0.20.203. This
> parameter is not available in the EMR-supplied capacity scheduler jar, so I
> was wondering if it's possible and safe to drop the 0.20.203 capacity
> scheduler jar into my Hadoop library path.
> >
> > Any information would be very helpful. Thanks!
> >
> > Kai Ju
>
>
Re: Dropping 0.20.203 capacity scheduler into 0.20.2
Posted by Arun C Murthy <ac...@hortonworks.com>.
Sorry. This mostly won't work... we have significant changes in the interface between the JobTracker and schedulers (FS/CS) b/w 20.2 and 20.203 (performance, better limits etc.).
Your best bet might be to provision Hadoop yourself on EC2 with 0.20.203+.
Good luck!
Arun
On Oct 26, 2011, at 2:55 PM, Kai Ju Liu wrote:
> Hi. I'm currently running a Hadoop cluster on Amazon's EMR service, which appears to be the 0.20.2 codebase plus several patches from the (deprecated?) 0.20.3 branch. I'm interested in switching from using the fair scheduler to the capacity scheduler, but I'm also interested in the user-limit-factor configuration parameter introduced in 0.20.203. This parameter is not available in the EMR-supplied capacity scheduler jar, so I was wondering if it's possible and safe to drop the 0.20.203 capacity scheduler jar into my Hadoop library path.
>
> Any information would be very helpful. Thanks!
>
> Kai Ju