You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Drew Farris <dr...@apache.org> on 2011/10/27 04:59:04 UTC
Composing Mahout workflow (Re: Improving Our JIRA State)
(Also a separate topic here)
On Wed, Oct 26, 2011 at 5:19 PM, Dan Brickley <da...@danbri.org> wrote:
>
> Also I've been thinking in very fuzzy terms about how to compose
> larger tasks from smaller pieces, and wondering what might be a more
> principled way of doing this than running each bin/mahout job by hand.
> Obviously coding it up is one way, but also little shell scripts or
> makefiles or (if forced at gunpoint) maybe Ant ...?
Well, there certainly seem to be a number of options out there, don't
forget to mention the FlumeJava items like Ted's work on Plume or
Cloudera Crunch. Is Oozie is an option for this as well? When I was
looking at the clustering code recently and saw the various, methods
starting with the run* prefix, I really wondered if there was a
standard way that we could package these chunks of code (steps), that
would allow them to be easily decomposed and re-combined in different
ways.
There's some talk about beanifying our workflow steps in
https://issues.apache.org/jira/browse/MAHOUT-612, but I can't say I
understand how this would allow us to reach the composable workflow
goal.
Re: Composing Mahout workflow (Re: Improving Our JIRA State)
Posted by Lance Norskog <go...@gmail.com>.
"You can even use the evolutionary optimizer to do the optimization more
cleanly."
"The hard part is to pass data structures around."
This is one use case- another is to classify data which "classifies well"
v.s. "classifies poorly" as part of fine-tuning your classification jobs.
On Thu, Oct 27, 2011 at 12:25 AM, Ted Dunning <te...@gmail.com> wrote:
> On Wed, Oct 26, 2011 at 10:16 PM, Lance Norskog <go...@gmail.com> wrote:
>
> > What about Groovy? Java does have scripting languages built in. Someone
> > (sorry can't remember) has some patches to make Mahout scala-friendly.
> >
>
> OK (maybe) for scala.
>
> Nix for Groovy. THe key problem is lack of a serializable closure.
>
> See http://tdunning.blogspot.com/2008/03/hello-world-for-map-reduce.html
>
> The ultimate results were that it was impossible to debug these systems due
> to some of (clever!) hacks I had to use to get things to work.
>
>
> > A use case for "programmable workflow engine" is to run the same
> > classification job 100 times with different tuning parameters, and save
> the
> > confusion matrices for further optimization. Which of these tools allows
> > this?
> >
>
> That is pretty easy.
>
> You can even use the evolutionary optimizer to do the optimization more
> cleanly.
>
> The hard part is to pass data structures around.
>
--
Lance Norskog
goksron@gmail.com
Re: Composing Mahout workflow (Re: Improving Our JIRA State)
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Oct 26, 2011 at 10:16 PM, Lance Norskog <go...@gmail.com> wrote:
> What about Groovy? Java does have scripting languages built in. Someone
> (sorry can't remember) has some patches to make Mahout scala-friendly.
>
OK (maybe) for scala.
Nix for Groovy. THe key problem is lack of a serializable closure.
See http://tdunning.blogspot.com/2008/03/hello-world-for-map-reduce.html
The ultimate results were that it was impossible to debug these systems due
to some of (clever!) hacks I had to use to get things to work.
> A use case for "programmable workflow engine" is to run the same
> classification job 100 times with different tuning parameters, and save the
> confusion matrices for further optimization. Which of these tools allows
> this?
>
That is pretty easy.
You can even use the evolutionary optimizer to do the optimization more
cleanly.
The hard part is to pass data structures around.
Re: Composing Mahout workflow (Re: Improving Our JIRA State)
Posted by Lance Norskog <go...@gmail.com>.
What about Groovy? Java does have scripting languages built in. Someone
(sorry can't remember) has some patches to make Mahout scala-friendly.
A use case for "programmable workflow engine" is to run the same
classification job 100 times with different tuning parameters, and save the
confusion matrices for further optimization. Which of these tools allows
this?
Lance
On Wed, Oct 26, 2011 at 7:59 PM, Drew Farris <dr...@apache.org> wrote:
> (Also a separate topic here)
>
> On Wed, Oct 26, 2011 at 5:19 PM, Dan Brickley <da...@danbri.org> wrote:
> >
> > Also I've been thinking in very fuzzy terms about how to compose
> > larger tasks from smaller pieces, and wondering what might be a more
> > principled way of doing this than running each bin/mahout job by hand.
> > Obviously coding it up is one way, but also little shell scripts or
> > makefiles or (if forced at gunpoint) maybe Ant ...?
>
> Well, there certainly seem to be a number of options out there, don't
> forget to mention the FlumeJava items like Ted's work on Plume or
> Cloudera Crunch. Is Oozie is an option for this as well? When I was
> looking at the clustering code recently and saw the various, methods
> starting with the run* prefix, I really wondered if there was a
> standard way that we could package these chunks of code (steps), that
> would allow them to be easily decomposed and re-combined in different
> ways.
>
> There's some talk about beanifying our workflow steps in
> https://issues.apache.org/jira/browse/MAHOUT-612, but I can't say I
> understand how this would allow us to reach the composable workflow
> goal.
>
--
Lance Norskog
goksron@gmail.com
Re: Composing Mahout workflow (Re: Improving Our JIRA State)
Posted by Ted Dunning <te...@gmail.com>.
On Thu, Oct 27, 2011 at 5:55 AM, Dan Brickley <da...@danbri.org> wrote:
> On 27 October 2011 12:54, Frank Scholten <fr...@frankscholten.nl> wrote:
> > On Thu, Oct 27, 2011 at 9:27 AM, Ted Dunning <te...@gmail.com>
> wrote:
> >
> >>> There's some talk about beanifying our workflow steps in
> >>> https://issues.apache.org/jira/browse/MAHOUT-612, but I can't say I
> >>> understand how this would allow us to reach the composable workflow
> >>> goal.
> >>
> >> I don't think it does. It just passes data around in files like we do
> now.
> >>
> >
> > Yes, MAHOUT-612 has beans for configuring kmeans, canopy and
> > seq2sparse and the configuration is serialized and deserialized at the
> > mappers and reducers. But it does not have a workflow engine or
> > something like that. You have to connect the inputs of one job to the
> > outputs of the other.
>
> Any sense for whether having all the inputs/outputs written to hdfs is
> a big problem, versus trying to plug things together in code so that
> it doesn't all get serialized? I mean, how much is it worth trying to
> do the latter instead of using the filesystem for integration.
>
THe major place I see that this cost is bad is where you have feature
extraction that could be folded into a map phase.
Re: Composing Mahout workflow (Re: Improving Our JIRA State)
Posted by Dan Brickley <da...@danbri.org>.
On 27 October 2011 12:54, Frank Scholten <fr...@frankscholten.nl> wrote:
> On Thu, Oct 27, 2011 at 9:27 AM, Ted Dunning <te...@gmail.com> wrote:
>
>>> There's some talk about beanifying our workflow steps in
>>> https://issues.apache.org/jira/browse/MAHOUT-612, but I can't say I
>>> understand how this would allow us to reach the composable workflow
>>> goal.
>>
>> I don't think it does. It just passes data around in files like we do now.
>>
>
> Yes, MAHOUT-612 has beans for configuring kmeans, canopy and
> seq2sparse and the configuration is serialized and deserialized at the
> mappers and reducers. But it does not have a workflow engine or
> something like that. You have to connect the inputs of one job to the
> outputs of the other.
Any sense for whether having all the inputs/outputs written to hdfs is
a big problem, versus trying to plug things together in code so that
it doesn't all get serialized? I mean, how much is it worth trying to
do the latter instead of using the filesystem for integration.
Dan
Re: Composing Mahout workflow (Re: Improving Our JIRA State)
Posted by Frank Scholten <fr...@frankscholten.nl>.
On Thu, Oct 27, 2011 at 9:27 AM, Ted Dunning <te...@gmail.com> wrote:
>> There's some talk about beanifying our workflow steps in
>> https://issues.apache.org/jira/browse/MAHOUT-612, but I can't say I
>> understand how this would allow us to reach the composable workflow
>> goal.
>
> I don't think it does. It just passes data around in files like we do now.
>
Yes, MAHOUT-612 has beans for configuring kmeans, canopy and
seq2sparse and the configuration is serialized and deserialized at the
mappers and reducers. But it does not have a workflow engine or
something like that. You have to connect the inputs of one job to the
outputs of the other.
Frank
Re: Composing Mahout workflow (Re: Improving Our JIRA State)
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Oct 26, 2011 at 7:59 PM, Drew Farris <dr...@apache.org> wrote:
> (Also a separate topic here)
>
> On Wed, Oct 26, 2011 at 5:19 PM, Dan Brickley <da...@danbri.org> wrote:
> >
> > Also I've been thinking in very fuzzy terms about how to compose
> > larger tasks from smaller pieces, and wondering what might be a more
> > principled way of doing this than running each bin/mahout job by hand.
> > Obviously coding it up is one way, but also little shell scripts or
> > makefiles or (if forced at gunpoint) maybe Ant ...?
>
> Well, there certainly seem to be a number of options out there, don't
> forget to mention the FlumeJava items like Ted's work on Plume or
> Cloudera Crunch. Is Oozie is an option for this as well? When I was
> looking at the clustering code recently and saw the various, methods
> starting with the run* prefix, I really wondered if there was a
> standard way that we could package these chunks of code (steps), that
> would allow them to be easily decomposed and re-combined in different
> ways.
>
I am still very convinced that lazy evaluation with execution plan rewriting
a la FlumeJava is a very important approach here. That allows your library
and my code to intermingle in the resulting map-reduce program.
> There's some talk about beanifying our workflow steps in
> https://issues.apache.org/jira/browse/MAHOUT-612, but I can't say I
> understand how this would allow us to reach the composable workflow
> goal.
>
I don't think it does. It just passes data around in files like we do now.