[jira] Created: (PIG-206) Right granularity for a pig script

["Mathieu Poumeyrol (JIRA)" <ji...@apache.org>]

Right granularity for a pig script
----------------------------------

                 Key: PIG-206
                 URL: https://issues.apache.org/jira/browse/PIG-206
             Project: Pig
          Issue Type: Wish
            Reporter: Mathieu Poumeyrol


I'd like to understand what people have in mind when they picture pig scripts...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-206) Right granularity for a pig script

["Mathieu Poumeyrol (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589586#action_12589586 ] 

Mathieu Poumeyrol commented on PIG-206:
---------------------------------------

All,

A bit of background, first. Over the last four or five years, my team and I have implemented several tools for our company [http://sk.idm.fr/opensource/software.html], one of them being a data processing framework (skprod) including a language which shares many goals and characteristics with Pig. 

The design was somehow "pre - MapReduce", and we are now facing some scalability issues which, combined with the cost of maintenance, make Pig look like a very good candidate for a replacement of skprod. The syntax is very different from what pig looks like, but the concepts maps quite easily. You may want to have a look to a getting started paper [http://sk.idm.fr/opensource/doc/skprod/index.html] to give an idea of it.

I am trying to "port" in Pig some of our existing data processing chains, and if many things looks very good, I now get an overall feeling that there is a difference of granularity : we were designing huge skprod script (the language itself has some builtin modularity) that perform a full featured task end to end, and usualy try to avoid chaining skprod scripts. But this approach does not map very well with pig as :
 - there is no way of defining "pig functions in pig". This lead obviously to pig code duplication.
 - every store statement is evaluated independently of the other, so there is no possibility for a script to detect the existence of a reusable intermediary result.

This lead me to think that I should maybe use Pig to run very small tasks, and find (or build ?) something on top of it to drive my general process calling pig as many times as needed, or generating a huge pig script...

At this point I'd realy like to know what you people think and where you plan to go... 

> Right granularity for a pig script
> ----------------------------------
>
>                 Key: PIG-206
>                 URL: https://issues.apache.org/jira/browse/PIG-206
>             Project: Pig
>          Issue Type: Wish
>            Reporter: Mathieu Poumeyrol
>
> I'd like to understand what people have in mind when they picture pig scripts...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-206) Right granularity for a pig script

["Alan Gates (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590125#action_12590125 ] 

Alan Gates commented on PIG-206:
--------------------------------

I do not think there is a consensus answer to your question at this point.  I have heard people advocate two possible tracks:

1) There's no reason to invent another scripting language, there is no lack of scripting languages.  It's better to embed pig in one or more of the existing ones.  This could be done in a way similar to DBI in perl.  Another suggestion of how to do it is a tighter integration with the actual parser of the language.  In python there are hooks into the parser that you can attach to, so that if the python parser can't understand a line of code it will hand it off to your module to interpret.  In this way pig could embed itself into python, make use of their constructs and syntax, etc.

The pros to this approach are that pig gets all the features of a scripting language for (nearly) free.

The cons are:
	A) Passing up certain optimization opportunities (eg loop unrolling) that you can do if you control the entire syntax tree.
	B) Risk getting tied to one language.  It may be difficult to maintain embeddings in multiple languages, leading to best support for only one.  This could lead to alienating users who don't know/like that one scripting language.
	C) Risk getting tied to an old version of a language.  If we embed in python, how much work is it for us when the next python version comes out and the interfaces change?  Are we going to be forever playing catchup?

2) We should make pig into a Turing complete language.  

Pros:
	A) Total control, we can design the language the way we want, optimize anything, etc.
	B) Self containment, users can write UDFs in pig directly rather than needing to go to a second language.
	
Cons:
	A) Pile of work!
	B) Usability.  Users don't want to learn a new syntax for while, if, etc.  It's easier for users to stick with already popular scripting languages. 




> Right granularity for a pig script
> ----------------------------------
>
>                 Key: PIG-206
>                 URL: https://issues.apache.org/jira/browse/PIG-206
>             Project: Pig
>          Issue Type: Wish
>            Reporter: Mathieu Poumeyrol
>
> I'd like to understand what people have in mind when they picture pig scripts...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.