You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Timothy Potter <th...@gmail.com> on 2012/05/02 20:06:31 UTC
Mahout - Pig Hackday
Hi,
The Dachis Group data analytics team are big users of Pig and just a
little Mahout so far, but that's changing soon so we'd like to
contribute some of our works and know-how back to the Mahout / Pig
communities.
In the near term, we're planning a Pig-Mahout hackday (code-named
"Pigout") at our HQ in Austin, TX on Friday, May 11 to co-incide with
Twitter's Pig hackday
(http://www.meetup.com/PigUser/events/62108962/). We're actually
hoping to get some sort of remote connection to the Twitter group ;-)
We're really keen on Ted's pig-vector project
(https://github.com/tdunning/pig-vector) as we're building a number of
classifiers on Mahout's SGD framework, with the bulk of our data being
in Cassandra processed almost entirely with Pig. We'd love to hear
about any planned features for the pig-vector project we can help out
on. Any similar Pig-Mahout projects we should know about?
In general, we're reaching out today to see who else in the community
is interested in better Pig / Mahout integration and what types of
challenges they're facing? Any cool UDFs you'd like to share?
Lastly, let us know if you'll be in the Austin area on Friday, May 11,
2012 as we'd love to join forces with other local Mahout / Pig users
... might even bring in some BBQ and really pig-out!
Cheers,
Timothy Potter
Big Data Architect, Dachis Group
www.socialbusinessindex.com
Re: Mahout - Pig Hackday
Posted by Andy Schlaikjer <an...@gmail.com>.
Hi Tim, Ted,
I wanted to chime in here regarding Elephant Bird utilities for
Pig-Mahout integration. I'm the author of EB's SequenceFileLoader,
SequenceFileStorage, and all the supporting WritableConverters,
including the VectorWritableConverter which facilitates conversion of
Mahout Vector data to Pig tuple formats (and vice versa). Details are
in the javadocs here:
https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/mahout/VectorWritableConverter.java
https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/load/SequenceFileLoader.java
https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/store/SequenceFileStorage.java
These utils are already heavily used here at Twitter. I've also put
together a number of UDFs for working with Vector data in Pig which I
hope to open source in the next month.
There's been recent discussion on EB's mailing list regarding
improvements to the project's build organization and artifact
publishing. We're interested in splitting the codebase into smaller
modules to produce thinner jar artifacts and simplify maven dependency
specification. In the process, I'll likely migrate
VectorWritableConverter to this separate Twitter OSS project with
related Vector UDFs and depend on those EB modules which include
SequenceFileStorage and friends.
I'll keep this list posted!
Cheers,
Andy
@sagemintblue
On Wed, May 2, 2012 at 5:20 PM, Timothy Potter <th...@gmail.com> wrote:
> Thanks Ted! Removing the elephant-bird dependency / build problems
> sounds like a good task we should include in our plans for the hackday
> ... what are your thoughts on adding pig-vector to Mahout as a contrib
> module? Do you want to keep it separate or eventually make its way
> into the project?
>
> Praneet - thanks for throwing your hat in ;-) Sounds like you're doing
> some interesting things with Mahout and Pig already. Will definitely
> keep you in the loop as we work out the details ...
>
> Cheers,
> Tim
>
> On Wed, May 2, 2012 at 1:43 PM, praneet mhatre <pr...@gmail.com> wrote:
>> On Wed, May 2, 2012 at 11:13 AM, Ted Dunning <te...@gmail.com> wrote:
>>
>>> On Wed, May 2, 2012 at 11:06 AM, Timothy Potter <thelabdude@gmail.com
>>> >wrote:
>>>
>>> > We're really keen on Ted's pig-vector project
>>> > (https://github.com/tdunning/pig-vector) as we're building a number of
>>> > classifiers on Mahout's SGD framework, with the bulk of our data being
>>> > in Cassandra processed almost entirely with Pig. We'd love to hear
>>> > about any planned features for the pig-vector project we can help out
>>> > on. Any similar Pig-Mahout projects we should know about?
>>> >
>>>
>>> The huge problem with pig-vector is that dependency on elephant bird makes
>>> it really almost impossible to build. Elephant bird has obscure
>>> dependencies on things like yaml-beans. That is a problem because the
>>> yaml-beans maintainer has a forceful way of expressing his distaste for all
>>> things to do with Maven and thus refuses to publish any artifacts in
>>> standard ways. Actually, the maintainer has a rather forceful manner that
>>> he applies to all interactions as far as I can tell.
>>>
>>> On the other hand, the necessary capabilities that pig-vector needs from
>>> Elephant bird are quite minor and could probably be reasonably extract. I
>>> am under-water, however, and thus cannot finish that right away. I can and
>>> will assist anybody who has the necessary time and enthusiasm. This might
>>> make a very nice pig day effort.
>>>
>>>
>>> > In general, we're reaching out today to see who else in the community
>>> > is interested in better Pig / Mahout integration and what types of
>>> > challenges they're facing? Any cool UDFs you'd like to share?
>>> >
>>>
>>> Praneet at UCI (praneetmhatre@gmail.com) has been doing some interesting
>>> work here to do with feature sharding in pig. Perhaps he can speak up.
>>>
>>
>> Hello Timothy,
>>
>> I have tried writing sharded versions of classifiers and they seem to work
>> well. But my code requires a pre-processing step before the classification
>> and re-aggregation of results (which was easy when I worked with Weka).
>> However, to be able to do the same in Mahout, I need something like
>> pig-vector to take of the pre-processing part.
>>
>> So yes, I am very interested in Pig / Mahout integration! But admittedly I
>> only have introductory knowledge of Pig. And as far the integration part
>> goes, my contribution so far has been limited to testing the stuff Ted has
>> written.
>>
>> But the idea of Pig-Mahout hackday sounds great! And I would definitely
>> like to be involved in it.
>>
>>
>>
>> --
>> Praneet Mhatre
>> Graduate Student
>> Donald Bren School of ICS
>> University of California, Irvine
Re: Mahout - Pig Hackday
Posted by Timothy Potter <th...@gmail.com>.
Here's the event registration link for the aforementioned Pig-out:
http://pig-hackday-austin.eventbrite.com/
On Wed, May 2, 2012 at 11:28 PM, Jake Mannix <ja...@gmail.com> wrote:
> On Wed, May 2, 2012 at 9:34 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> On Wed, May 2, 2012 at 9:05 PM, Jake Mannix <ja...@gmail.com> wrote:
>>
>> > On Wed, May 2, 2012 at 8:07 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>> >
>> > > Making a pig module for mahout is a fine idea. The twitter guys may
>> have
>> > > something better, though, so we should explore that as well. Andy's
>> > > comments make that possibility very interesting.
>> > >
>> >
>> > What I'd want to suggest is that anyone who wants to move rapidly on
>> > pig/mahout
>> > integration should start a github repo which doesn't directly inject
>> itself
>> > into mahout,
>> > but stands separately for now, but then the maven dependency DAG rears
>> its
>> > ugly head:
>> >
>> > pig-vector depends on mahout-core
>> >
>> > so if we *do* want to start writing cool stuff *in mahout* which depends
>> on
>> > it,
>> >
>>
>> I think that we are fine if we just create a pig module in mahout. It can
>> depend on the external stuff and mahout-core. That would be the natural
>> time and place to put the fancy pig-vector-ish stuff anyway.
>>
>> So I am not worried about this. We would have separation of mahout-pig
>> stuff from mahout-core-ish stuff and all should be fine.
>
>
> Yeah, most likely the idea would be that mahout-pig would depend on more
> than just writables, in the long run: UDF wrappers for everything we stuff
> into one (a la Jimmy Lin et al's "Training a smarter pig" talk at
> Hadoop World)
>
>
>> > we're circularly dependently self-destruct. Now, if we had a proper
>> > mahout-writables
>> > maven module (*ahem*!), which had all the stuff pig-vector needed, and
>> > mahout-core
>> > depended on this, then mahout-core (or mahout-examples) could still
>> depend
>> > on
>> > pig-vector (or something like it, like the elephant-bird-loaders slim
>> dep)
>> > at some
>> > point.
>> >
>>
>> I would rather not have Mahout depend on unreleased github stuff. If it is
>> good enough to depend on, it is good enough to suck into the main
>> deliverable.
>>
>
> Oh I wasn't meaning core should depend on unreleased stuff, more
> like the elephant-bird slimmed down module, once released.
>
> --
>
> -jake
Re: Mahout - Pig Hackday
Posted by Jake Mannix <ja...@gmail.com>.
On Wed, May 2, 2012 at 9:34 PM, Ted Dunning <te...@gmail.com> wrote:
> On Wed, May 2, 2012 at 9:05 PM, Jake Mannix <ja...@gmail.com> wrote:
>
> > On Wed, May 2, 2012 at 8:07 PM, Ted Dunning <te...@gmail.com>
> wrote:
> >
> > > Making a pig module for mahout is a fine idea. The twitter guys may
> have
> > > something better, though, so we should explore that as well. Andy's
> > > comments make that possibility very interesting.
> > >
> >
> > What I'd want to suggest is that anyone who wants to move rapidly on
> > pig/mahout
> > integration should start a github repo which doesn't directly inject
> itself
> > into mahout,
> > but stands separately for now, but then the maven dependency DAG rears
> its
> > ugly head:
> >
> > pig-vector depends on mahout-core
> >
> > so if we *do* want to start writing cool stuff *in mahout* which depends
> on
> > it,
> >
>
> I think that we are fine if we just create a pig module in mahout. It can
> depend on the external stuff and mahout-core. That would be the natural
> time and place to put the fancy pig-vector-ish stuff anyway.
>
> So I am not worried about this. We would have separation of mahout-pig
> stuff from mahout-core-ish stuff and all should be fine.
Yeah, most likely the idea would be that mahout-pig would depend on more
than just writables, in the long run: UDF wrappers for everything we stuff
into one (a la Jimmy Lin et al's "Training a smarter pig" talk at
Hadoop World)
> > we're circularly dependently self-destruct. Now, if we had a proper
> > mahout-writables
> > maven module (*ahem*!), which had all the stuff pig-vector needed, and
> > mahout-core
> > depended on this, then mahout-core (or mahout-examples) could still
> depend
> > on
> > pig-vector (or something like it, like the elephant-bird-loaders slim
> dep)
> > at some
> > point.
> >
>
> I would rather not have Mahout depend on unreleased github stuff. If it is
> good enough to depend on, it is good enough to suck into the main
> deliverable.
>
Oh I wasn't meaning core should depend on unreleased stuff, more
like the elephant-bird slimmed down module, once released.
--
-jake
Re: Mahout - Pig Hackday
Posted by Ted Dunning <te...@gmail.com>.
On Wed, May 2, 2012 at 9:05 PM, Jake Mannix <ja...@gmail.com> wrote:
> On Wed, May 2, 2012 at 8:07 PM, Ted Dunning <te...@gmail.com> wrote:
>
> > Making a pig module for mahout is a fine idea. The twitter guys may have
> > something better, though, so we should explore that as well. Andy's
> > comments make that possibility very interesting.
> >
>
> What I'd want to suggest is that anyone who wants to move rapidly on
> pig/mahout
> integration should start a github repo which doesn't directly inject itself
> into mahout,
> but stands separately for now, but then the maven dependency DAG rears its
> ugly head:
>
> pig-vector depends on mahout-core
>
> so if we *do* want to start writing cool stuff *in mahout* which depends on
> it,
>
I think that we are fine if we just create a pig module in mahout. It can
depend on the external stuff and mahout-core. That would be the natural
time and place to put the fancy pig-vector-ish stuff anyway.
So I am not worried about this. We would have separation of mahout-pig
stuff from mahout-core-ish stuff and all should be fine.
> we're circularly dependently self-destruct. Now, if we had a proper
> mahout-writables
> maven module (*ahem*!), which had all the stuff pig-vector needed, and
> mahout-core
> depended on this, then mahout-core (or mahout-examples) could still depend
> on
> pig-vector (or something like it, like the elephant-bird-loaders slim dep)
> at some
> point.
>
I would rather not have Mahout depend on unreleased github stuff. If it is
good enough to depend on, it is good enough to suck into the main
deliverable.
Re: Mahout - Pig Hackday
Posted by Jake Mannix <ja...@gmail.com>.
On Wed, May 2, 2012 at 8:07 PM, Ted Dunning <te...@gmail.com> wrote:
> Making a pig module for mahout is a fine idea. The twitter guys may have
> something better, though, so we should explore that as well. Andy's
> comments make that possibility very interesting.
>
What I'd want to suggest is that anyone who wants to move rapidly on
pig/mahout
integration should start a github repo which doesn't directly inject itself
into mahout,
but stands separately for now, but then the maven dependency DAG rears its
ugly head:
pig-vector depends on mahout-core
so if we *do* want to start writing cool stuff *in mahout* which depends on
it,
we're circularly dependently self-destruct. Now, if we had a proper
mahout-writables
maven module (*ahem*!), which had all the stuff pig-vector needed, and
mahout-core
depended on this, then mahout-core (or mahout-examples) could still depend
on
pig-vector (or something like it, like the elephant-bird-loaders slim dep)
at some
point.
>
> On Wed, May 2, 2012 at 5:20 PM, Timothy Potter <th...@gmail.com>
> wrote:
>
> > Thanks Ted! Removing the elephant-bird dependency / build problems
> > sounds like a good task we should include in our plans for the hackday
> > ... what are your thoughts on adding pig-vector to Mahout as a contrib
> > module? Do you want to keep it separate or eventually make its way
> > into the project?
> >
>
--
-jake
Re: Mahout - Pig Hackday
Posted by Ted Dunning <te...@gmail.com>.
Making a pig module for mahout is a fine idea. The twitter guys may have
something better, though, so we should explore that as well. Andy's
comments make that possibility very interesting.
On Wed, May 2, 2012 at 5:20 PM, Timothy Potter <th...@gmail.com> wrote:
> Thanks Ted! Removing the elephant-bird dependency / build problems
> sounds like a good task we should include in our plans for the hackday
> ... what are your thoughts on adding pig-vector to Mahout as a contrib
> module? Do you want to keep it separate or eventually make its way
> into the project?
>
Re: Mahout - Pig Hackday
Posted by Timothy Potter <th...@gmail.com>.
Thanks Ted! Removing the elephant-bird dependency / build problems
sounds like a good task we should include in our plans for the hackday
... what are your thoughts on adding pig-vector to Mahout as a contrib
module? Do you want to keep it separate or eventually make its way
into the project?
Praneet - thanks for throwing your hat in ;-) Sounds like you're doing
some interesting things with Mahout and Pig already. Will definitely
keep you in the loop as we work out the details ...
Cheers,
Tim
On Wed, May 2, 2012 at 1:43 PM, praneet mhatre <pr...@gmail.com> wrote:
> On Wed, May 2, 2012 at 11:13 AM, Ted Dunning <te...@gmail.com> wrote:
>
>> On Wed, May 2, 2012 at 11:06 AM, Timothy Potter <thelabdude@gmail.com
>> >wrote:
>>
>> > We're really keen on Ted's pig-vector project
>> > (https://github.com/tdunning/pig-vector) as we're building a number of
>> > classifiers on Mahout's SGD framework, with the bulk of our data being
>> > in Cassandra processed almost entirely with Pig. We'd love to hear
>> > about any planned features for the pig-vector project we can help out
>> > on. Any similar Pig-Mahout projects we should know about?
>> >
>>
>> The huge problem with pig-vector is that dependency on elephant bird makes
>> it really almost impossible to build. Elephant bird has obscure
>> dependencies on things like yaml-beans. That is a problem because the
>> yaml-beans maintainer has a forceful way of expressing his distaste for all
>> things to do with Maven and thus refuses to publish any artifacts in
>> standard ways. Actually, the maintainer has a rather forceful manner that
>> he applies to all interactions as far as I can tell.
>>
>> On the other hand, the necessary capabilities that pig-vector needs from
>> Elephant bird are quite minor and could probably be reasonably extract. I
>> am under-water, however, and thus cannot finish that right away. I can and
>> will assist anybody who has the necessary time and enthusiasm. This might
>> make a very nice pig day effort.
>>
>>
>> > In general, we're reaching out today to see who else in the community
>> > is interested in better Pig / Mahout integration and what types of
>> > challenges they're facing? Any cool UDFs you'd like to share?
>> >
>>
>> Praneet at UCI (praneetmhatre@gmail.com) has been doing some interesting
>> work here to do with feature sharding in pig. Perhaps he can speak up.
>>
>
> Hello Timothy,
>
> I have tried writing sharded versions of classifiers and they seem to work
> well. But my code requires a pre-processing step before the classification
> and re-aggregation of results (which was easy when I worked with Weka).
> However, to be able to do the same in Mahout, I need something like
> pig-vector to take of the pre-processing part.
>
> So yes, I am very interested in Pig / Mahout integration! But admittedly I
> only have introductory knowledge of Pig. And as far the integration part
> goes, my contribution so far has been limited to testing the stuff Ted has
> written.
>
> But the idea of Pig-Mahout hackday sounds great! And I would definitely
> like to be involved in it.
>
>
>
> --
> Praneet Mhatre
> Graduate Student
> Donald Bren School of ICS
> University of California, Irvine
Re: Mahout - Pig Hackday
Posted by praneet mhatre <pr...@gmail.com>.
On Wed, May 2, 2012 at 11:13 AM, Ted Dunning <te...@gmail.com> wrote:
> On Wed, May 2, 2012 at 11:06 AM, Timothy Potter <thelabdude@gmail.com
> >wrote:
>
> > We're really keen on Ted's pig-vector project
> > (https://github.com/tdunning/pig-vector) as we're building a number of
> > classifiers on Mahout's SGD framework, with the bulk of our data being
> > in Cassandra processed almost entirely with Pig. We'd love to hear
> > about any planned features for the pig-vector project we can help out
> > on. Any similar Pig-Mahout projects we should know about?
> >
>
> The huge problem with pig-vector is that dependency on elephant bird makes
> it really almost impossible to build. Elephant bird has obscure
> dependencies on things like yaml-beans. That is a problem because the
> yaml-beans maintainer has a forceful way of expressing his distaste for all
> things to do with Maven and thus refuses to publish any artifacts in
> standard ways. Actually, the maintainer has a rather forceful manner that
> he applies to all interactions as far as I can tell.
>
> On the other hand, the necessary capabilities that pig-vector needs from
> Elephant bird are quite minor and could probably be reasonably extract. I
> am under-water, however, and thus cannot finish that right away. I can and
> will assist anybody who has the necessary time and enthusiasm. This might
> make a very nice pig day effort.
>
>
> > In general, we're reaching out today to see who else in the community
> > is interested in better Pig / Mahout integration and what types of
> > challenges they're facing? Any cool UDFs you'd like to share?
> >
>
> Praneet at UCI (praneetmhatre@gmail.com) has been doing some interesting
> work here to do with feature sharding in pig. Perhaps he can speak up.
>
Hello Timothy,
I have tried writing sharded versions of classifiers and they seem to work
well. But my code requires a pre-processing step before the classification
and re-aggregation of results (which was easy when I worked with Weka).
However, to be able to do the same in Mahout, I need something like
pig-vector to take of the pre-processing part.
So yes, I am very interested in Pig / Mahout integration! But admittedly I
only have introductory knowledge of Pig. And as far the integration part
goes, my contribution so far has been limited to testing the stuff Ted has
written.
But the idea of Pig-Mahout hackday sounds great! And I would definitely
like to be involved in it.
--
Praneet Mhatre
Graduate Student
Donald Bren School of ICS
University of California, Irvine
Re: Mahout - Pig Hackday
Posted by Ted Dunning <te...@gmail.com>.
On Wed, May 2, 2012 at 11:06 AM, Timothy Potter <th...@gmail.com>wrote:
> We're really keen on Ted's pig-vector project
> (https://github.com/tdunning/pig-vector) as we're building a number of
> classifiers on Mahout's SGD framework, with the bulk of our data being
> in Cassandra processed almost entirely with Pig. We'd love to hear
> about any planned features for the pig-vector project we can help out
> on. Any similar Pig-Mahout projects we should know about?
>
The huge problem with pig-vector is that dependency on elephant bird makes
it really almost impossible to build. Elephant bird has obscure
dependencies on things like yaml-beans. That is a problem because the
yaml-beans maintainer has a forceful way of expressing his distaste for all
things to do with Maven and thus refuses to publish any artifacts in
standard ways. Actually, the maintainer has a rather forceful manner that
he applies to all interactions as far as I can tell.
On the other hand, the necessary capabilities that pig-vector needs from
Elephant bird are quite minor and could probably be reasonably extract. I
am under-water, however, and thus cannot finish that right away. I can and
will assist anybody who has the necessary time and enthusiasm. This might
make a very nice pig day effort.
> In general, we're reaching out today to see who else in the community
> is interested in better Pig / Mahout integration and what types of
> challenges they're facing? Any cool UDFs you'd like to share?
>
Praneet at UCI (praneetmhatre@gmail.com) has been doing some interesting
work here to do with feature sharding in pig. Perhaps he can speak up.