Install mahout 0.8 with hadoop 2.0

[Sergey Svinarchuk <ss...@hortonworks.com>]

Hi all,

Somebody compile and install mahout with hadoop 2.0? If yes, that what
changes you make in mahout, that it have 100% passed unit tests and
successful work with hadoop 2.0?

Thanks

Re: Install mahout 0.8 with hadoop 2.0

[Carlos Mundi <cm...@gmail.com>]

Ted,

The mr v1 option is absolutely viable.  It's just question of how long a
project wants to rely on a legacy tail, and that's everyone's individual
lookout naturally.

Downpour style will help, as you say and surely know better than me; I
should think about that for my situation.

Thanks!
On Aug 14, 2013 12:37 PM, "Ted Dunning" <te...@gmail.com> wrote:

> Honest feedback is always welcome on this mailing list.  Don't ever worry
> about flames for that.
>
> Don't forget that mr v1 is an option with hadoop 2. Confusing as that may
> be.
>
> Iterative algos are, as you say, very important.  My current inclination
> is to lean toward a downpour style of implementation. That fits well with
> yarn but it also actually fits reasonably with mr v1.
>
> Sent from my iPhone
>
> On Aug 13, 2013, at 20:13, Carlos Mundi <cm...@gmail.com> wrote:
>
> > Anyway, I apologize if anyone takes offense.  None is meant, so please
> > flame me off-list if you must.  But since I self-identify as a member of
> > the "small demand" set Ted Dunning describes, I figure I can chime in.
>  As
> > always, YMMV.
>

Re: Install mahout 0.8 with hadoop 2.0

[Ted Dunning <te...@gmail.com>]

Honest feedback is always welcome on this mailing list.  Don't ever worry about flames for that.  

Don't forget that mr v1 is an option with hadoop 2. Confusing as that may be.  

Iterative algos are, as you say, very important.  My current inclination is to lean toward a downpour style of implementation. That fits well with yarn but it also actually fits reasonably with mr v1.  

Sent from my iPhone

On Aug 13, 2013, at 20:13, Carlos Mundi <cm...@gmail.com> wrote:

> Anyway, I apologize if anyone takes offense.  None is meant, so please
> flame me off-list if you must.  But since I self-identify as a member of
> the "small demand" set Ted Dunning describes, I figure I can chime in.  As
> always, YMMV.

Re: Install mahout 0.8 with hadoop 2.0

[Carlos Mundi <cm...@gmail.com>]

I recently asked the same core question on this list.  I certainly won't
argue with the statistics of small numbers.  But I will hazard a
prediction: the impetus for Mahout to support Hadoop 2 will appear about
the same time the elephant book gets updated for 2.0, provided Twister or
something like it does not get there first.

My own view, from the outside, is that the best use of limited Mahout dev
resources at the moment is stabilizing the core feature-set and positioning
Mahout in the increasingly crowded big analytics space.  I don't know how
easy that is, as I do not know how tightly Mahout is tied to the Hadoop MR
model.

Although it would be hard to overstate the importance of iterative algos,
it is also true that Hadoop MR excels on some very real killer apps.
Making those accessible and robust should take precedence, for now, over
the incremental ROI for chasing Hadoop 2.  I say this as someone who would
very much like to run Mahout against Hadoop 2.

Anyway, I apologize if anyone takes offense.  None is meant, so please
flame me off-list if you must.  But since I self-identify as a member of
the "small demand" set Ted Dunning describes, I figure I can chime in.  As
always, YMMV.

Cheers,
Carlos
On Aug 13, 2013 9:59 AM, "Ted Dunning" <te...@gmail.com> wrote:

> No.  There is very small demand for Mahout on Hadoop 2.0 so far and the
> forward/backward incompatibility of 2.0 has made it difficult to motivate
> moving to 2.0.
>
> The bigtop guys built a maven profile for 0.23 some time ago.  I don't know
> the status of that.
>
> I don't think that the differences are huge ... it is just the standard
> Hadoop forklift-the-world upgrade experience.
>
>
>
> On Tue, Aug 13, 2013 at 6:49 AM, Sergey Svinarchuk <
> ssvinarchuk@hortonworks.com> wrote:
>
> > Hi all,
> >
> > Somebody compile and install mahout with hadoop 2.0? If yes, that what
> > changes you make in mahout, that it have 100% passed unit tests and
> > successful work with hadoop 2.0?
> >
> > Thanks
> >
>

Re: Install mahout 0.8 with hadoop 2.0

[Sean Owen <sr...@gmail.com>]

I think it all minimally works on Hadoop 2.0.x, though I haven't tried
it recently -- it does require a recompile.

This is different from it working on MRv2 versus MRv1. I'm almost
certain it does not work on MRv2 and doubt it will.

The effort is not large, but it's subtle. A few hacks may fail in
mysterious ways, and certainly to properly use MRv2 you have to switch
to use the newer resource configuration scheme -- in terms of
megabytes not reducer slots and all that.

At least this was most of the work that i remember when I was
rebuilding some of this type of stuff on MRv2 + Hadoop 2.0.x

On Tue, Aug 13, 2013 at 5:58 PM, Ted Dunning <te...@gmail.com> wrote:
> No.  There is very small demand for Mahout on Hadoop 2.0 so far and the
> forward/backward incompatibility of 2.0 has made it difficult to motivate
> moving to 2.0.
>
> The bigtop guys built a maven profile for 0.23 some time ago.  I don't know
> the status of that.
>
> I don't think that the differences are huge ... it is just the standard
> Hadoop forklift-the-world upgrade experience.
>
>
>
> On Tue, Aug 13, 2013 at 6:49 AM, Sergey Svinarchuk <
> ssvinarchuk@hortonworks.com> wrote:
>
>> Hi all,
>>
>> Somebody compile and install mahout with hadoop 2.0? If yes, that what
>> changes you make in mahout, that it have 100% passed unit tests and
>> successful work with hadoop 2.0?
>>
>> Thanks
>>

Re: Install mahout 0.8 with hadoop 2.0

[Ted Dunning <te...@gmail.com>]

No.  There is very small demand for Mahout on Hadoop 2.0 so far and the
forward/backward incompatibility of 2.0 has made it difficult to motivate
moving to 2.0.

The bigtop guys built a maven profile for 0.23 some time ago.  I don't know
the status of that.

I don't think that the differences are huge ... it is just the standard
Hadoop forklift-the-world upgrade experience.



On Tue, Aug 13, 2013 at 6:49 AM, Sergey Svinarchuk <
ssvinarchuk@hortonworks.com> wrote:

> Hi all,
>
> Somebody compile and install mahout with hadoop 2.0? If yes, that what
> changes you make in mahout, that it have 100% passed unit tests and
> successful work with hadoop 2.0?
>
> Thanks
>