spork?

[Tom Graves <tg...@yahoo.com>]

Hey everyone,

At one point I heard of pig on Spark (Spork).  The only thing I could find on it was https://github.com/dvryaboy/pig and it doesn't look like there has been any work on it in quite a while.  

Does anyone know the current status of that or has tried it?

Thanks,
Tom

Re: spork?

[Matei Zaharia <ma...@gmail.com>]

Yup, that was it, though I believe people at Twitter picked it up again recently. I’d suggest asking Dmitriy if you know him. I’ve seen interest in this from several other groups, and if there’s enough of it, maybe we can start another open source repo to track it. The work in that repo you pointed to was done over one week, and already had most of Pig’s operators working. (I helped out with this prototype over Twitter’s hack week.) That work also calls the Scala API directly, because it was done before we had a Java API; it should be easier with the Java one.

Matei

On Jan 22, 2014, at 9:07 AM, Tom Graves <tg...@yahoo.com> wrote:

> Hey everyone,
> 
> At one point I heard of pig on Spark (Spork).  The only thing I could find on it was https://github.com/dvryaboy/pig and it doesn't look like there has been any work on it in quite a while.  
> 
> Does anyone know the current status of that or has tried it?
> 
> Thanks,
> Tom

Re: spork?

[Matei Zaharia <ma...@gmail.com>]

Yeah, this is true for new jobs, but some organizations have large libraries of Pig scripts and UDFs. I think what would be ideal is a Pig-to-Scala or Pig-to-Python binding layer similar to Shark’s sql2rdd. That way you could run your existing Pig scripts, but also call them from Scala or Python to transition over to these programming languages.

Matei

On Jan 22, 2014, at 11:12 AM, Paul Brown <pr...@mult.ifario.us> wrote:

> Just an IMHO.  We've moved a number of Pig jobs to Spark jobs, and the
> benefits of a genuine conversion (type safety, terseness, debugging, lack
> of gratuitous shells and shims around UDFs, ...) are sufficient that a
> rewrite is probably preferable from an adapter unless you have an enormous
> Pig investment that you want to continue investing in.
> 
> -- Paul
> 
> 
> —
> prb@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/
> 
> 
> On Wed, Jan 22, 2014 at 9:07 AM, Tom Graves <tg...@yahoo.com> wrote:
> 
>> Hey everyone,
>> 
>> At one point I heard of pig on Spark (Spork).  The only thing I could find
>> on it was https://github.com/dvryaboy/pig and it doesn't look like there
>> has been any work on it in quite a while.
>> 
>> Does anyone know the current status of that or has tried it?
>> 
>> Thanks,
>> Tom

Re: spork?

[Paul Brown <pr...@mult.ifario.us>]

Just an IMHO.  We've moved a number of Pig jobs to Spark jobs, and the
benefits of a genuine conversion (type safety, terseness, debugging, lack
of gratuitous shells and shims around UDFs, ...) are sufficient that a
rewrite is probably preferable from an adapter unless you have an enormous
Pig investment that you want to continue investing in.

-- Paul


—
prb@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/


On Wed, Jan 22, 2014 at 9:07 AM, Tom Graves <tg...@yahoo.com> wrote:

> Hey everyone,
>
> At one point I heard of pig on Spark (Spork).  The only thing I could find
> on it was https://github.com/dvryaboy/pig and it doesn't look like there
> has been any work on it in quite a while.
>
> Does anyone know the current status of that or has tried it?
>
> Thanks,
> Tom