You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Hector Yee <he...@gmail.com> on 2011/06/16 19:34:07 UTC

Iterative jobs

What do people think of using Spark for iterative jobs:

http://www.spark-project.org/

Or is there a new version of hadoop that supports this kind of computation?

-- 
Yee Yang Li Hector
http://hectorgon.blogspot.com/ (tech + travel)
http://hectorgon.com (book reviews)

Re: Iterative jobs

Posted by Hector Yee <he...@gmail.com>.
When is that ready?
On Jun 18, 2011 12:38 PM, "Josh Patterson" <jo...@cloudera.com> wrote:
> Another approach is the next generation Map Reduce 2 framework that
> moves the processing framework up into "user space" so that you can
> plug in another framework, like a "Dryad" implementation, in theory.
>
> I think we'll see something like this happen, and then you could get
> your low latency small data chunks from hbase or from the dcache, etc.
>
> JP
>
> On Thu, Jun 16, 2011 at 1:34 PM, Hector Yee <he...@gmail.com> wrote:
>> What do people think of using Spark for iterative jobs:
>>
>> http://www.spark-project.org/
>>
>> Or is there a new version of hadoop that supports this kind of
computation?
>>
>> --
>> Yee Yang Li Hector
>> http://hectorgon.blogspot.com/ (tech + travel)
>> http://hectorgon.com (book reviews)
>>
>
>
>
> --
> Twitter: @jpatanooga
> Solution Architect @ Cloudera
> hadoop: http://www.cloudera.com
> blog: http://jpatterson.floe.tv

Re: Iterative jobs

Posted by Josh Patterson <jo...@cloudera.com>.
Another approach is the next generation Map Reduce 2 framework that
moves the processing framework up into "user space" so that you can
plug in another framework, like a "Dryad" implementation, in theory.

I think we'll see something like this happen, and then you could get
your low latency small data chunks from hbase or from the dcache, etc.

JP

On Thu, Jun 16, 2011 at 1:34 PM, Hector Yee <he...@gmail.com> wrote:
> What do people think of using Spark for iterative jobs:
>
> http://www.spark-project.org/
>
> Or is there a new version of hadoop that supports this kind of computation?
>
> --
> Yee Yang Li Hector
> http://hectorgon.blogspot.com/ (tech + travel)
> http://hectorgon.com (book reviews)
>



-- 
Twitter: @jpatanooga
Solution Architect @ Cloudera
hadoop: http://www.cloudera.com
blog: http://jpatterson.floe.tv

Re: Iterative jobs

Posted by Hector Yee <he...@gmail.com>.
Ok sounds good. The hadoop wrapper handles iterations good enough for now.

On Thu, Jun 16, 2011 at 10:54 AM, Sean Owen <sr...@gmail.com> wrote:

> I guess my question i, is it a better framework and for what kind of
> problem? It says it gains speed for iterative algorithms by keeping
> stuff in memory. That's a fine tradeoff to make but sounds about like
> a point on the same "efficient frontier" of tradeoffs that any good
> system lives on. It's also a tradeoff you can already kind of make on
> Hadoop, and that some of the implementations here already do: loading
> via distributed cache or side-loading from HDFS. At first glance I'd
> guess it's "a bit better" for some kinds of problem.
>
> So what's the "cost" of using this? You certainly wouldn't want to
> replace the Hadoop-based version as the audience for that is much
> greater. Having an additional implementation doesn't hurt anyone. But
> it's another dependency and item to support (unless it's not going to
> get supported) and there is some small harm to having one isolated
> orphan implementation in the project: we've been trying to kill rather
> than feed such orphans lately.
>
>
> I personally don't think that this project needs more algorithms, and
> am personally directing all my time to what I view as more essential
> infrastructure tasks. Or to put it another way: how about tidying up
> Hadoop before moving on? That's just me.
>
> My gut says it would be cool to implement the SVD on something like
> this to see how it goes. I don't yet see this is anything to move to.
>
>
> On Thu, Jun 16, 2011 at 6:34 PM, Hector Yee <he...@gmail.com> wrote:
> > What do people think of using Spark for iterative jobs:
> >
> > http://www.spark-project.org/
> >
> > Or is there a new version of hadoop that supports this kind of
> computation?
> >
> > --
> > Yee Yang Li Hector
> > http://hectorgon.blogspot.com/ (tech + travel)
> > http://hectorgon.com (book reviews)
> >
>



-- 
Yee Yang Li Hector
http://hectorgon.blogspot.com/ (tech + travel)
http://hectorgon.com (book reviews)

Re: Iterative jobs

Posted by Sean Owen <sr...@gmail.com>.
There are 55 open issues already -- not just new implementations but
bug reports and enhancement requests. Just look at
https://issues.apache.org/jira/browse/MAHOUT . I personally value work
on those more than work on new JIRA issues.

On Sun, Jun 19, 2011 at 10:01 PM, Hector Yee <he...@gmail.com> wrote:
> So rather than adding new learning algorithms we should make the existing
> ones map-reduceable?
> Do you have a JIRA ticket for those that need tidying up?

Re: Iterative jobs

Posted by Hector Yee <he...@gmail.com>.
So rather than adding new learning algorithms we should make the existing
ones map-reduceable?
Do you have a JIRA ticket for those that need tidying up?

On Thu, Jun 16, 2011 at 10:54 AM, Sean Owen <sr...@gmail.com> wrote:
>
> I personally don't think that this project needs more algorithms, and
> am personally directing all my time to what I view as more essential
> infrastructure tasks. Or to put it another way: how about tidying up
> Hadoop before moving on? That's just me.
>
> --
Yee Yang Li Hector
http://hectorgon.blogspot.com/ (tech + travel)
http://hectorgon.com (book reviews)

Re: Iterative jobs

Posted by Sean Owen <sr...@gmail.com>.
I guess my question i, is it a better framework and for what kind of
problem? It says it gains speed for iterative algorithms by keeping
stuff in memory. That's a fine tradeoff to make but sounds about like
a point on the same "efficient frontier" of tradeoffs that any good
system lives on. It's also a tradeoff you can already kind of make on
Hadoop, and that some of the implementations here already do: loading
via distributed cache or side-loading from HDFS. At first glance I'd
guess it's "a bit better" for some kinds of problem.

So what's the "cost" of using this? You certainly wouldn't want to
replace the Hadoop-based version as the audience for that is much
greater. Having an additional implementation doesn't hurt anyone. But
it's another dependency and item to support (unless it's not going to
get supported) and there is some small harm to having one isolated
orphan implementation in the project: we've been trying to kill rather
than feed such orphans lately.


I personally don't think that this project needs more algorithms, and
am personally directing all my time to what I view as more essential
infrastructure tasks. Or to put it another way: how about tidying up
Hadoop before moving on? That's just me.

My gut says it would be cool to implement the SVD on something like
this to see how it goes. I don't yet see this is anything to move to.


On Thu, Jun 16, 2011 at 6:34 PM, Hector Yee <he...@gmail.com> wrote:
> What do people think of using Spark for iterative jobs:
>
> http://www.spark-project.org/
>
> Or is there a new version of hadoop that supports this kind of computation?
>
> --
> Yee Yang Li Hector
> http://hectorgon.blogspot.com/ (tech + travel)
> http://hectorgon.com (book reviews)
>