You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Ben Clay <rb...@ncsu.edu> on 2010/12/20 23:25:29 UTC
Hadoop 0.21 support
Hi-
Is there a timeline for updating Mahout to work with Hadoop 0.21?
Hadoop 0.21 introduced some features we need (LATE scheduler) but when
running the Mahout Quickstart jobs on both Mahout 0.4 and the latest 0.5
trunk, I get a number of errors relating to the new Hadoop API. I had some
trouble finding details using Google, but I see the 0.5 trunk's pom.xml
indicates <hadoop.version>0.20.2</hadoop.version>
Thanks for any info!
-Ben
Re: Hadoop 0.21 support
Posted by Sean Owen <sr...@gmail.com>.
For the foreseeable future, we're on 0.20.2. A major reason is that
Amazon EMR is on 0.20.2 and not 0.21.0 yet.
I think I know what you mean. There is one tiny API change that makes
it impossible to run 0.20.2-compiled code on 0.21.0 and vice versa.
Strangely, you will find that the code compiles fine against 0.21.0,
and that you can then run it on 0.21.0. I believe an abstract class
was made an interface and, while that change didn't make the
compile-time situation any different for Mahout, it means that the
byte code can only be consistent with one version of Hadoop.
So: I'm saying you can change the Hadoop version to 0.21.0 and compile
again and it will work fine.
Well, not so fast. hadoop-core is now renamed and split into
hadoop-common and hadoop-mapred, if I remember rightly, so you need to
update and add these dependencies. And AFAIK 0.21.0 is still not in
Maven for some reason. So you'd have to install 0.21.0 in Maven
locally to get it to work.
At least, this is what I had to do to get it working when I tried this
a few weeks ago.
On Mon, Dec 20, 2010 at 10:25 PM, Ben Clay <rb...@ncsu.edu> wrote:
> Hi-
>
>
>
> Is there a timeline for updating Mahout to work with Hadoop 0.21?
>
>
>
> Hadoop 0.21 introduced some features we need (LATE scheduler) but when
> running the Mahout Quickstart jobs on both Mahout 0.4 and the latest 0.5
> trunk, I get a number of errors relating to the new Hadoop API. I had some
> trouble finding details using Google, but I see the 0.5 trunk's pom.xml
> indicates <hadoop.version>0.20.2</hadoop.version>
>
>
>
> Thanks for any info!
>
>
>
> -Ben
>
>
>
>
Re: Hadoop 0.21 support
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Correction. my github trunk patch seems to still show CDH3b2 here :
<hadoop.version>0.20.2+320</hadoop.version>
But i did build and run some of my code with 0.20.2+737 on another computer.
Obviously i can't verify all Mahout algorithms but everything we ever built
for 0.20 old api before still happily runs with cdh distribution in
production with no side effects.
-Dmitriy
On Mon, Dec 20, 2010 at 4:27 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Ben -- Also if it helps I have that CDH3 dependency integration e.g. on
> this branch made to most recent Mahout trunk :
> https://github.com/dlyubimov/ssvd-lsi at branch ssvd-givens and all
> others. You'd need to install CDH3 jars into your local maven repository
> before it compiles. Or you may just 'git diff trunk -- pom.xml' to see
> /generate just the dependency patch.
>
> -d
>
>
> On Mon, Dec 20, 2010 at 4:16 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> Ben --
>>
>> yes CDH distributions backport 0.21 patches. Basically the maven version
>> of CDH3b3 says 0.20.2+737 where 737 is the number of mostly 0.21 patches
>> that they backported to 0.20.2 platform. They still have pretty useful
>> patches missing but their approach is to do thorough testing of those
>> changes so perhaps some of the features are omited deliberately. Most ppl
>> that i know (including us) use that stuff in production because of the
>> combination of fixes that are hard to ignore (such as append api so
>> desperately needed for latest 0.89 HBase releases) with scrutiny for
>> stability that goes with it. AFAIK they also use these with their clients.
>>
>> -d
>>
>>
>> On Mon, Dec 20, 2010 at 3:58 PM, Ben Clay <rb...@ncsu.edu> wrote:
>>
>>> Dmitry-
>>>
>>> Thanks for looking into this.
>>>
>>> HADOOP-2141 is indeed what we need. I am a bit confused - the JIRA page
>>> (https://issues.apache.org/jira/browse/HADOOP-2141) shows it was
>>> included in
>>> Hadoop 0.21, but CDH3b3 appears to be based on 0.20 (shown as
>>> hadoop-0.20.noarch in their yum repo). Does Cloudera typically backport
>>> features like this?
>>>
>>> Regardless, I'll install CDH3b3 and use it. Thanks for the tip!
>>>
>>> -Ben
>>>
>>
Re: Hadoop 0.21 support
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Ben -- Also if it helps I have that CDH3 dependency integration e.g. on this
branch made to most recent Mahout trunk :
https://github.com/dlyubimov/ssvd-lsi at branch ssvd-givens and all others.
You'd need to install CDH3 jars into your local maven repository before it
compiles. Or you may just 'git diff trunk -- pom.xml' to see /generate just
the dependency patch.
-d
On Mon, Dec 20, 2010 at 4:16 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Ben --
>
> yes CDH distributions backport 0.21 patches. Basically the maven version of
> CDH3b3 says 0.20.2+737 where 737 is the number of mostly 0.21 patches that
> they backported to 0.20.2 platform. They still have pretty useful patches
> missing but their approach is to do thorough testing of those changes so
> perhaps some of the features are omited deliberately. Most ppl that i know
> (including us) use that stuff in production because of the combination of
> fixes that are hard to ignore (such as append api so desperately needed for
> latest 0.89 HBase releases) with scrutiny for stability that goes with it.
> AFAIK they also use these with their clients.
>
> -d
>
>
> On Mon, Dec 20, 2010 at 3:58 PM, Ben Clay <rb...@ncsu.edu> wrote:
>
>> Dmitry-
>>
>> Thanks for looking into this.
>>
>> HADOOP-2141 is indeed what we need. I am a bit confused - the JIRA page
>> (https://issues.apache.org/jira/browse/HADOOP-2141) shows it was included
>> in
>> Hadoop 0.21, but CDH3b3 appears to be based on 0.20 (shown as
>> hadoop-0.20.noarch in their yum repo). Does Cloudera typically backport
>> features like this?
>>
>> Regardless, I'll install CDH3b3 and use it. Thanks for the tip!
>>
>> -Ben
>>
>
Re: Hadoop 0.21 support
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Ben --
yes CDH distributions backport 0.21 patches. Basically the maven version of
CDH3b3 says 0.20.2+737 where 737 is the number of mostly 0.21 patches that
they backported to 0.20.2 platform. They still have pretty useful patches
missing but their approach is to do thorough testing of those changes so
perhaps some of the features are omited deliberately. Most ppl that i know
(including us) use that stuff in production because of the combination of
fixes that are hard to ignore (such as append api so desperately needed for
latest 0.89 HBase releases) with scrutiny for stability that goes with it.
AFAIK they also use these with their clients.
-d
On Mon, Dec 20, 2010 at 3:58 PM, Ben Clay <rb...@ncsu.edu> wrote:
> Dmitry-
>
> Thanks for looking into this.
>
> HADOOP-2141 is indeed what we need. I am a bit confused - the JIRA page
> (https://issues.apache.org/jira/browse/HADOOP-2141) shows it was included
> in
> Hadoop 0.21, but CDH3b3 appears to be based on 0.20 (shown as
> hadoop-0.20.noarch in their yum repo). Does Cloudera typically backport
> features like this?
>
> Regardless, I'll install CDH3b3 and use it. Thanks for the tip!
>
> -Ben
>
>
> -----Original Message-----
> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
> Sent: Monday, December 20, 2010 6:25 PM
> To: user@mahout.apache.org
> Subject: Re: Hadoop 0.21 support
>
> Also , just in case if you are talking about HADOOP-2141, it _is_ included
> in CDH3b3, and i did verify before that CDH3b3 compiles with Mahout.
>
> Thanks.
> -Dmitriy
>
> On Mon, Dec 20, 2010 at 3:03 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > Ben,
> >
> > Just curious, are you planning to use 0.21 or new api from Cloudera
> > distribution? Cloudera seems to compile fine although i am not sure if
> > they backported the feature you are talking about.
> >
> > -d
> >
> >
> > On Mon, Dec 20, 2010 at 2:25 PM, Ben Clay <rb...@ncsu.edu> wrote:
> >
> >> Hi-
> >>
> >>
> >>
> >> Is there a timeline for updating Mahout to work with Hadoop 0.21?
> >>
> >>
> >>
> >> Hadoop 0.21 introduced some features we need (LATE scheduler) but
> >> when running the Mahout Quickstart jobs on both Mahout 0.4 and the
> >> latest 0.5 trunk, I get a number of errors relating to the new Hadoop
> >> API. I had some trouble finding details using Google, but I see the
> >> 0.5 trunk's pom.xml indicates <hadoop.version>0.20.2</hadoop.version>
> >>
> >>
> >>
> >> Thanks for any info!
> >>
> >>
> >>
> >> -Ben
> >>
> >>
> >>
> >>
> >
>
>
RE: Hadoop 0.21 support
Posted by Ben Clay <rb...@ncsu.edu>.
Dmitry-
Thanks for looking into this.
HADOOP-2141 is indeed what we need. I am a bit confused - the JIRA page
(https://issues.apache.org/jira/browse/HADOOP-2141) shows it was included in
Hadoop 0.21, but CDH3b3 appears to be based on 0.20 (shown as
hadoop-0.20.noarch in their yum repo). Does Cloudera typically backport
features like this?
Regardless, I'll install CDH3b3 and use it. Thanks for the tip!
-Ben
-----Original Message-----
From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
Sent: Monday, December 20, 2010 6:25 PM
To: user@mahout.apache.org
Subject: Re: Hadoop 0.21 support
Also , just in case if you are talking about HADOOP-2141, it _is_ included
in CDH3b3, and i did verify before that CDH3b3 compiles with Mahout.
Thanks.
-Dmitriy
On Mon, Dec 20, 2010 at 3:03 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Ben,
>
> Just curious, are you planning to use 0.21 or new api from Cloudera
> distribution? Cloudera seems to compile fine although i am not sure if
> they backported the feature you are talking about.
>
> -d
>
>
> On Mon, Dec 20, 2010 at 2:25 PM, Ben Clay <rb...@ncsu.edu> wrote:
>
>> Hi-
>>
>>
>>
>> Is there a timeline for updating Mahout to work with Hadoop 0.21?
>>
>>
>>
>> Hadoop 0.21 introduced some features we need (LATE scheduler) but
>> when running the Mahout Quickstart jobs on both Mahout 0.4 and the
>> latest 0.5 trunk, I get a number of errors relating to the new Hadoop
>> API. I had some trouble finding details using Google, but I see the
>> 0.5 trunk's pom.xml indicates <hadoop.version>0.20.2</hadoop.version>
>>
>>
>>
>> Thanks for any info!
>>
>>
>>
>> -Ben
>>
>>
>>
>>
>
Re: Hadoop 0.21 support
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Also , just in case if you are talking about HADOOP-2141, it _is_ included
in CDH3b3, and i did verify before that CDH3b3 compiles with Mahout.
Thanks.
-Dmitriy
On Mon, Dec 20, 2010 at 3:03 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Ben,
>
> Just curious, are you planning to use 0.21 or new api from Cloudera
> distribution? Cloudera seems to compile fine although i am not sure if they
> backported the feature you are talking about.
>
> -d
>
>
> On Mon, Dec 20, 2010 at 2:25 PM, Ben Clay <rb...@ncsu.edu> wrote:
>
>> Hi-
>>
>>
>>
>> Is there a timeline for updating Mahout to work with Hadoop 0.21?
>>
>>
>>
>> Hadoop 0.21 introduced some features we need (LATE scheduler) but when
>> running the Mahout Quickstart jobs on both Mahout 0.4 and the latest 0.5
>> trunk, I get a number of errors relating to the new Hadoop API. I had
>> some
>> trouble finding details using Google, but I see the 0.5 trunk's pom.xml
>> indicates <hadoop.version>0.20.2</hadoop.version>
>>
>>
>>
>> Thanks for any info!
>>
>>
>>
>> -Ben
>>
>>
>>
>>
>
Re: Hadoop 0.21 support
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Ben,
Just curious, are you planning to use 0.21 or new api from Cloudera
distribution? Cloudera seems to compile fine although i am not sure if they
backported the feature you are talking about.
-d
On Mon, Dec 20, 2010 at 2:25 PM, Ben Clay <rb...@ncsu.edu> wrote:
> Hi-
>
>
>
> Is there a timeline for updating Mahout to work with Hadoop 0.21?
>
>
>
> Hadoop 0.21 introduced some features we need (LATE scheduler) but when
> running the Mahout Quickstart jobs on both Mahout 0.4 and the latest 0.5
> trunk, I get a number of errors relating to the new Hadoop API. I had some
> trouble finding details using Google, but I see the 0.5 trunk's pom.xml
> indicates <hadoop.version>0.20.2</hadoop.version>
>
>
>
> Thanks for any info!
>
>
>
> -Ben
>
>
>
>