You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Andy Wagner <an...@gmail.com> on 2014/10/18 23:42:29 UTC
Scala DSL for Pig
Hi All,
I am creating an internal Scala DSL for Pig and am wondering if there is
any interest from others. I have quite a bit of experience in DSLs which I
wanted to use as a starting point to learn a bit about the big data tools
and Pig seemed like a simple enough language to use as a starting point.
The project is at an early state and not really even alpha ready although a
few simple test cases are working.
For those not familiar with DSLs, I have embedded a syntax very similar to
pig inside of scala so "pig-like" code can be written directly inside
scala.
Some simple working test cases can be seen here :
https://github.com/andywag/pigdsl/blob/master/src/test/scala/org/simplifide/pig/test/BasicParserTest.scala
A larger set of command can be seen here :
https://github.com/andywag/pigdsl/blob/master/src/test/scala/org/simplifide/pig/test/TestParser.scala
Any feedback on the usefulness of this would be greatly appreciated.
Thanks in Advance,
Andy
Re: Scala DSL for Pig
Posted by Andy Wagner <an...@gmail.com>.
Hi Matt,
Thanks for the links.
I think I am doing the opposite of what PigPen and Scalding are doing.
Those projects seem to be trying to make "MapReduce" operation more
Clojure/Scala like where I am trying to embed pig like commands in Scala.
The potential advantage of this is that it allows the use of either Scala
or Pig like syntax for operations. I have done this for other
languages(verilog mainly) with good results but given I am a novice in this
area I am doing this now as more of an academic exercise.
Hi Russel,
Thanks for the comments. I am planning on continuing although the language
is a bit more complicated than it appeared at first glance.
Andy
On Wed, Oct 22, 2014 at 1:46 PM, Matt Bossenbroek <
mbossenbroek@netflix.com.invalid> wrote:
> Check out PigPen (a Clojure version of what you're doing):
> https://github.com/Netflix/PigPen
>
> And if you really want to use Scala, also check out Scalding:
> https://github.com/twitter/scalding
>
> -Matt
>
>
> On Saturday, October 18, 2014 at 4:07 PM, Russell Jurney wrote:
>
> > I don't have any feedback except to say I think this is cool and you
> should
> > keep working on it :)
> > ᐧ
> >
> > On Sat, Oct 18, 2014 at 2:42 PM, Andy Wagner <andy.wagner2@gmail.com
> (mailto:andy.wagner2@gmail.com)> wrote:
> >
> > > Hi All,
> > >
> > > I am creating an internal Scala DSL for Pig and am wondering if there
> is
> > > any interest from others. I have quite a bit of experience in DSLs
> which I
> > > wanted to use as a starting point to learn a bit about the big data
> tools
> > > and Pig seemed like a simple enough language to use as a starting
> point.
> > > The project is at an early state and not really even alpha ready
> although a
> > > few simple test cases are working.
> > >
> > > For those not familiar with DSLs, I have embedded a syntax very
> similar to
> > > pig inside of scala so "pig-like" code can be written directly inside
> > > scala.
> > >
> > > Some simple working test cases can be seen here :
> > >
> > >
> https://github.com/andywag/pigdsl/blob/master/src/test/scala/org/simplifide/pig/test/BasicParserTest.scala
> > >
> > > A larger set of command can be seen here :
> > >
> > >
> https://github.com/andywag/pigdsl/blob/master/src/test/scala/org/simplifide/pig/test/TestParser.scala
> > >
> > > Any feedback on the usefulness of this would be greatly appreciated.
> > >
> > > Thanks in Advance,
> > > Andy
> > >
> >
> >
> >
> >
> > --
> > Russell Jurney twitter.com/rjurney russell.jurney@gmail.com (mailto:
> russell.jurney@gmail.com) datasyndrome.com (http://datasyndrome.com)
> >
> >
>
>
>
Re: Scala DSL for Pig
Posted by Matt Bossenbroek <mb...@netflix.com.INVALID>.
Check out PigPen (a Clojure version of what you're doing): https://github.com/Netflix/PigPen
And if you really want to use Scala, also check out Scalding: https://github.com/twitter/scalding
-Matt
On Saturday, October 18, 2014 at 4:07 PM, Russell Jurney wrote:
> I don't have any feedback except to say I think this is cool and you should
> keep working on it :)
> ᐧ
>
> On Sat, Oct 18, 2014 at 2:42 PM, Andy Wagner <andy.wagner2@gmail.com (mailto:andy.wagner2@gmail.com)> wrote:
>
> > Hi All,
> >
> > I am creating an internal Scala DSL for Pig and am wondering if there is
> > any interest from others. I have quite a bit of experience in DSLs which I
> > wanted to use as a starting point to learn a bit about the big data tools
> > and Pig seemed like a simple enough language to use as a starting point.
> > The project is at an early state and not really even alpha ready although a
> > few simple test cases are working.
> >
> > For those not familiar with DSLs, I have embedded a syntax very similar to
> > pig inside of scala so "pig-like" code can be written directly inside
> > scala.
> >
> > Some simple working test cases can be seen here :
> >
> > https://github.com/andywag/pigdsl/blob/master/src/test/scala/org/simplifide/pig/test/BasicParserTest.scala
> >
> > A larger set of command can be seen here :
> >
> > https://github.com/andywag/pigdsl/blob/master/src/test/scala/org/simplifide/pig/test/TestParser.scala
> >
> > Any feedback on the usefulness of this would be greatly appreciated.
> >
> > Thanks in Advance,
> > Andy
> >
>
>
>
>
> --
> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com (mailto:russell.jurney@gmail.com) datasyndrome.com (http://datasyndrome.com)
>
>
Re: Scala DSL for Pig
Posted by Russell Jurney <ru...@gmail.com>.
I don't have any feedback except to say I think this is cool and you should
keep working on it :)
ᐧ
On Sat, Oct 18, 2014 at 2:42 PM, Andy Wagner <an...@gmail.com> wrote:
> Hi All,
>
> I am creating an internal Scala DSL for Pig and am wondering if there is
> any interest from others. I have quite a bit of experience in DSLs which I
> wanted to use as a starting point to learn a bit about the big data tools
> and Pig seemed like a simple enough language to use as a starting point.
> The project is at an early state and not really even alpha ready although a
> few simple test cases are working.
>
> For those not familiar with DSLs, I have embedded a syntax very similar to
> pig inside of scala so "pig-like" code can be written directly inside
> scala.
>
> Some simple working test cases can be seen here :
>
> https://github.com/andywag/pigdsl/blob/master/src/test/scala/org/simplifide/pig/test/BasicParserTest.scala
>
> A larger set of command can be seen here :
>
> https://github.com/andywag/pigdsl/blob/master/src/test/scala/org/simplifide/pig/test/TestParser.scala
>
> Any feedback on the usefulness of this would be greatly appreciated.
>
> Thanks in Advance,
> Andy
>
--
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com