You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by Apache Wiki <wi...@apache.org> on 2010/10/16 01:37:05 UTC
[Pig Wiki] Update of "TuringCompletePig" by AlanGates
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.
The "TuringCompletePig" page has been changed by AlanGates.
http://wiki.apache.org/pig/TuringCompletePig?action=diff&rev1=6&rev2=7
--------------------------------------------------
- = Making Pig Latin Turing Complete =
+ = Touring Complete Pig =
+
+ This document lays out the high level design for adding control flow and modularity constructs to Pig.
+
== Introduction ==
- As more users adopt Pig and begin writing their data processing in Pig Latin and as they use Pig to process more and more complex
- tasks, a consistent request from these users has been to add branches, loops, and functions to Pig Latin. This will enable Pig Latin to
- process a whole new class of problems. Consider, for example, an algorithm that needs to iterate until an error estimate is less
- than a given threshold. This might look like (this just suggests logic, not syntax):
+ Pig Latin is a data flow language. As such it does not offer users control flow and modularity features that are present in general purpose programming languages, including
+ functions, modules, loops, and branches. Given that it is a data flow language adding these constructs is neither straightforward nor reasonable.
+ However, users do want to be able to integrate standard programming techniques of separation and code sharing offered by functions and modules as well as
+ integration of control flow offered by functions, loops, and branches. This document proposes a way to accomplish these goals while preserving Pig Latin's data flow
+ orientation.
+ == The Proposal ==
- {{{
- error = 100.0;
- infile = 'original.data';
- while (error > 1.0) {
- A = load 'infile';
- B = group A all;
- C = foreach B generate flatten(doSomeCalculation(A)) as (result, error);
- error = foreach C generate error;
- store C into 'outfile';
- if (error > 1.0) mv 'outfile' 'infile';
- }
- }}}
-
- == Requirements ==
- The following should be provided by this Turing complete Pig Latin:
- 1. Branching. This will be satisfied by a standard `if / else if / else` functionality
- 1. Looping. This should include standard `while` and some form of `for`. for could be C style or Python style (foreach). Care needs to be taken to select syntax that does not cause confusion with the existing `foreach` operator in Pig Latin.
- 1. Functions.
- 1. Modules.
- 1. The ability to use local in memory variables in the Pig Latin script. For example, in the snippet given above the way `infile` is defined above the `while` and then used in the `load`.
- 1. The ability to "store" results into local in memory variables. For example, in the snippet given above the way the error calculation from the data processing is stored into `error` in the line `error = foreach C generate error;`.
-
- == Approach ==
+ === Approach ===
- There are two possible approaches to this. One is to add all of these features to Pig Latin itself. This has several advantages:
- * All Pig Latin operations will be first class objects in the language. There will not be a need to do quoted programming, like what happens when JDBC is used to write SQL inside a Java program.
- * There will be opportunities to do optimizations that are not available in embedded programming, such as loop unrolling, etc.
+ Since direct integration of data flow and control flow is neither reasonable nor desirable, a heuristic is needed to productively combine the two. One approach
+ that is being used in other projects is to view the control flow as building pipelines of data flow. This is used in [[http://www.cascading.org/|Cascading]] and
+ [[http://portal.acm.org/citation.cfm?id=1806596.1806638|FlumeJava]].
- However, the cost of this approach is incredible. It means turning Pig Latin into a full scripting language. And it means
- all kinds of tools like debuggers, etc. will never be available for Pig Latin users because the Pig team will not have the resources
- or expertise to develop and maintain such tools. And finally, does the world need another scripting language that starts with P?
+ Given this approach, it makes sense to embed Pig Latin in an existing scripting language. A scripting language has several advantages over a compiled language such
+ as Java:
+ * A compiled language requires a write, compile, package, deploy, run cycle. A scripting language cuts the compile, package, and deploy out of the cycle.
+ * Compiled languages are usually chosen for their greater speed and resource control (memory, etc.). Since time spent in the control flow portion of these scripts will generally represent less than 1% of the total run time, performance for the control flow is not the most important consideration. Similarly it is not expected that users will write control flows that require large amounts of memory, I/O, etc.
+ * Given that Java options for Hadoop already exist (Map Reduce, Cascading), my intuition is that many Pig users have chosen it precisely because they prefer its scripting approach. For them a scripting interface for control flow will be a more natural approach than Java.
+ * Scripting languages are traditionally used for glueing together modules of other functionality, which is what we are doing here.
+ (Note however, that if a given user prefers Java, there is nothing to prevent them from using the proposed interface with Java.)
- The second possible approach to this is to embed Pig Latin into an existing scripting language, such as Perl, Python, Ruby, etc. The
- advantages of this are:
- * Most of the requirements noted above (branching, looping, functions, and modules) are present in these languages.
- * For any of these languages whole hosts of tools such as debuggers, IDEs, etc. exist and could be used.
- * Users do not have to learn a new language.
+ We propose to solve these issues with two significant changes to Pig.
- There are a few significant drawbacks to this approach:
- * It leads to a quoted programming style which is unnatural and irritating for developers.
- * Which scripting language to choose? Perl, Python, and Ruby all have significant adoption and could make a claim to be the best choice.
- * Syntactic and semantic checking is usually delayed until an embedded bit of code is reached in the outer control flow. Given that Pig jobs can run for hours this can mean spending hours to discover a simple typo.
- Consider for example if Pig provided a Jython class that wrapped !PigServer and then we translated the above code snippet.
+ === Add Macro Expansion To Pig Latin ===
+ Part one of this proposal is to add macro expansion to Pig Latin. The existing `DEFINE` keyword will be expanded to allow definitions of Pig macros. A new `INLINE` keyword will be added
+ that inlines a macro at a particular point. A new `IMPORT` keyword will also be added to allow macros from other files to be inlined as well. See below for full details.
{{{
- error = 100.0
- infile = 'original.data'
- pig = PigServer()
- grunt = Grunt()
- while error > 1.0:
- pig.registerQuery("A = load 'infile'; \
+ define bot_cleanser(input, output, user) {
+ A = load '$input' using TempStorage();
+ B = filter A by not is_a_bot($user);
+ store B into '$output' using TempStorage();
+ }
+
+ A = load 'fact';
+ store A into 'bc_input' using TempStorage();
+ inline bot_cleanser('bc_input', 'bc_output', 'username');
+ B = load 'bc_output' using TempStorage();
+ C = group B by user;
+ ...
+ store Z into 'processed';
+ }}}
+
+ === Use Scripting Language For Control Flow ===
+
+ The second part of the proposal is to embed Pig Latin scripts in the host scripting language via a JDBC like compile, bind, run model. An example script would look like:
+
+ {{{
+ p = Pig.compile("""A = load '$in’;
+ C = group A by user;
+ D = foreach C generate group, myUDF(A);
+ store D into '$out’;
- B = group A all; \
+ F = group D all;
+ G = foreach F generate MAX(D);
+ store G into 'tmp’;
+ """)
- C = foreach B generate flatten(doSomeCalculation(A)) as (result, error); \
- PigIterator pi = pig.openIterator("C", 'outfile');
- output = grunt.exec("fs cat 'outfile'");
- bla = output.partition("\t");
- error = bla(2)
- if error >= 1.0:
- grunt.exec("fs mv 'outfile' 'infile'")
- }}}
-
- All of these references to `pig` and `grunt` as objects with command strings are undesirable.
- So while I believe that embedding is a much better approach due to the lower work load and the plethora of tools available for other
- languages, I do not believe the above is an acceptable way to do it. Thus I would like to place three additional requirements on
- embedded Pig Latin beyond those given above for Turing complete Pig Latin:
- 1.#7 Pig Latin should appear as a natural part of the language it is embedded in, not as quoted strings.
- 1. Syntactic and semantic checking should be done up front before the script is run.
- 1. It should be possible to write UDFs in the scripting language that Pig Latin is embedded in and reference them from Pig Latin.
-
- This overcomes two of the three drawbacks noted above. It does not provide for a way to do certain optimizations such as loop
- unrolling, but I think that is acceptable.
-
- Having rejected the quote style of programming we could choose the Domain Specific Language (DSL) option, where we define Pig operators in the
- target language. Again using Python as an example:
-
- {{{
- error = 100.0
+ error = 100.0
+ input = "original”
+ output = "output-0”
+ final = "final-output”
- infile = 'original.data'
- pig = PigServer()
- grunt = Grunt()
- while error > 1.0:
- A = pig.load(infile, { 'loader' => 'piggybank.MyLoader'});
- B = A.group(pig.ALL);
- C = B.foreach {
- innerBag = doSomeCalculation(:A);
- generate innerBag.flatten().as(:result, :error)
- }
-
- PigIterator pi = pig.openIterator(C, 'outfile');
- output = grunt.fs.cat('outfile'");
- bla = output.partition("\t");
- error = bla(2)
- if error >= 1.0:
- grunt.fs.mv('outfile', 'infile');
- }}}
+ for i in range(1, 100):
+ p.bind({'in':input, 'out':output}) # attaches $in, $out in Pig Latin to input, output Python variables
+ results = p.run()
- This meets requirements 7 and 9 above. It can partially but not fully meet 8. It can check that we use the right operators and pass
- them the right types. It cannot check the semantics of the operators, for example that `infile` exists and is readable. This might be ok,
- because it might turn out that things that cannot be checked at script compile time should not be checked up front anyway. As an example, it should not
- check for `infile` up front because the script may not have created it yet.
-
- This approach has the advantage that it will integrate very nicely with tools from the target language. Debuggers, IDE, etc. will all now
- view some form of Pig Latin as native to their language.
-
- It does however have drawback, which is that what we would be creating a new dialect of Pig Latin. There would be a Pig Latin dialect used when writing it
- directly, and a different dialect for embedding. This leads to confusion and duplication of effort. So I would like to suggest another
- requirement:
-
- 1.#10 Pig Latin should appear the same in the embedded form as in the non-embedded form.
- Given all these requirements, what might this look like? Again using the script snippet at the top and embedding it in Jython:
+ if results.getStatus("D") `= "FAILED" or results.getStatus("G") =` "FAILED":
+ raise "Pig job failed"
+ iter = results.getResults("G")
+ if iter.next() > 1:
+ input = output
+ output = "output-" + i.to_s
+ else:
+ H = Pig.fs("mv " + output + " " + final)
+ break
+ }}}
+
+ == Architecture ==
+
+ Pig will pass the control flow script to the host language for processing. This in turn will end up producing sets of data flows which will be passed to !PigServer.
+
+ === Scripting Language ===
+ The scripting language chosen must be able to execute in the JVM. This allows us to package a jar with Pig (or pull it via Ivy or Maven) and run it
+ without the user needing to download and install any packages. The candidates are Python, Ruby, Groovy, Scala, and Clojure, as far ask I know. (Scala and Clojure are not scripting
+ languages, though they can act like one.)
+
+ The code will be written in such a way that users can, with a minimal amount of coding, adapt it to the language of their choice. However, the Pig team will pick one language
+ to support as a default and reference implementation.
+
+ JRuby would perhaps be a good choice, except that it is under the GPL. If I understand the license correctly, packaging it as part of Pig would mean that Pig would need to be
+ under the GPL. That obviously is a non-starter. (Note, others have challenged this assertion on my part, saying that Pig itself would only become GPL if it '''modified''' and
+ packaged JRuby. If there is sufficient interest in Ruby we may need to pursue this further.)
+
+ Scala is cool and powerful, but relatively unknown at this point. Similarly Groovy seems to have a small user community and relatively few available libraries.
+
+ Clojure is a functional language, a paradigm which seems to engender one of love, terror, or confusion. As such, it probably is not a good choice for Pig Latin. Also, Cascalog
+ already exists for those who like Clojure.
+
+ This leaves Python, which has the advantages of a Java implementation, well established community, and extensive libraries. My impression is that Python also has a wider
+ adoption and user base than Ruby. One downside is that Pig Latin has more
+ Java like syntax (use of braces to open and close nested sections, semicolons to terminate lines) rather than Python like syntax. This mixing of multiple syntaxes may confuse
+ users.
+
+ === What Can Be Embedded ===
+ As noted in the examples, Pig data flow directives could be embedded in Python via a `Pig` construct. The following would be valid inside this construct:
+ * `COGROUP`
+ * `CROSS`
+ * `DISTINCT`
+ * `FILTER`
+ * `FOREACH`
+ * `GROUP`
+ * `JOIN`
+ * `LIMIT`
+ * `LOAD`
+ * `MAPREDUCE`
+ * `ORDER`
+ * `SAMPLE`
+ * `SPLIT`
+ * `STORE`
+ * `STREAM`
+ * `UNION`
+ * `REGISTER`
+ * `DEFINE`
+ * `set`
+
+ Static functions on this Pig object will be created for:
+ * `DEFINE`
+ * `REGISTER`
+ * `set`
+ * `fs`
+
+ Grunt will not be supported in this setup. So Grunt specific commands such as `mkdir, cd` etc. will not be supported. We are in the process of deprecating these
+ in standard Pig Latin anyway. The new `fs` commands will be supported.
+
+ In addition to accepting strings in the `compile` call, it will also be possible to read Pig Latin code from a file. The contents of this file must be valid Pig Latin.
+
+ Converge.pig:
{{{
+
+ A = load '$in’;
+ C = group A by user;
+ D = foreach C generate group, myUDF(A);
+ store D into '$out’;
+ F = group D all;
+ G = foreach F generate MAX(D);
+ store G into 'tmp’;
- error = 100.0
+ error = 100.0
- infile = 'original.data'
- while error > 1.0:
- PIG:
- A = load '$infile';
+ }}}
+
+ Converge.py:
+ {{{
+ p = Pig.compileFromFile("Converge.pig")
+ input = "original”
+ output = "output-0”
+ final = "final-output”
+
+ for i in range(1, 100):
+ p.bind({'in':input, 'out':output}) # attaches $in, $out in Pig Latin to input, output Python variables
+ results = p.run()
+
+ if results.getStatus("D") `= "FAILED" or results.getStatus("G") =` "FAILED":
+ raise "Pig job failed"
+ iter = results.getResults("G")
+ if iter.next() > 1:
+ input = output
+ output = "output-" + i.to_s
+ else:
+ H = Pig.fs("mv " + output + " " + final)
+ break
+ }}}
+
+
+ === Use of Parameters to Be Bound Later ===
+ When defining a Pig object, it will be legal to leave certain values undefined, replacing them with a parameter name. For example a user may wish to define a pipeline without yet defining
+ the location of the input to the pipeline. The parameter will be indicated by a dollar sign followed by a sequence of alpha-numeric or underscore characters. Values for these parameters must
+ be provided later at the time `bind()` is called on the Pig object. To call `run()` on a Pig object without all parameters being bound is an error. Parameters can be used in the following
+ locations in Pig Latin script:
+ * argument to the parallel directive
+ * locations for load and store functions
+ * constants in foreach or filter expressions
+ * field names
+ * number of record in a limit
+
+ === Parallel Execution of Similar Pipelines ===
+ A common use case is for users to run an identical pipeline across multiple input sets. For example, a user might want to run the same pipeline over data from different countries. A
+ for loop in the host language would support this, but would result in the pipelines running serially rather than in parallel. Many users wish to run them in parallel instead. To
+ meet this need, Pig will support binding lists of variables to the same Pig object in one bind call. Calling run on that object will then result in the script being run for each set of
+ bound variables in parallel.
+
+ For example, say that a pipeline exists that the user wishes to run for US, UK, and Brazil, all in parallel.
+
+ {{{
+ p = Pig.compile("""A = load '$in';
+ B = filter A by user is not null;
+ ...
+ store Z into '$out';
+ """)
+
+ # Pass a list of hashes, instead of one hash
+ p.bind([{'in':'us_raw','out':'us_processed'},
+ {'in':'uk_raw','out':'uk_processed'},
+ {'in':'brazil_raw','out':'brazil_processed'}])
+ p.run()
+ }}}
+
+ === Macro Expansion ===
+ Data entry points into macros will be handled by named locations loaded via a `TempStorage` loader. Data exit
+ points from macros will be handled by named locations stored via `TempStorage`.
+ Calling a macro without having stored data to each expected input will result in an error when the Pig script is compiled. Calling a macro without subsequently loading data from every
+ stored output is not an error. The Pig optimizer will handle deciding when the combination of `TempStorage` load/store pairs can be removed and when they need to be converted to actual
+ load/store operations using Pig's intermediate format. Stores that use `TempStorage` that are actually stored to disk will be marked as temporary and removed at the end of execution of
+ the host language script. The Pig optimizer will also handle removing portions of the pipeline that go to stores from macros that are not loaded in the
+ calling script.
+
+ These macros will also allow simple parameter substitution. This is not a generic substitution. These macro parameters can occur in the same places as the bind parameters defined above. In
+ cases where these arguments represent field names (such as the example given above) there is no way to enforce the expected type of the column.
+
+ The new `IMPORT` keyword can be used of files in Pig Latin:
+
+ bot_filter.pig:
+ {{{
+ define bot_cleanser(user) {
+ A = load 'bc_input' using TempStorage();
+ B = filter A by not is_a_bot($user);
+ store B into 'bc_output' using TempStorage();
+ }
+ }}}
+
+ main.pig:
+ {{{
+ import bot_filter.pig;
+
+ A = load 'fact';
+ store A into 'bc_input' using TempStorage();
+ inline bot_cleanser('username');
+ B = load 'bc_output' using TempStorage();
+ C = group B by user;
+ ...
+ store Z into 'processed';
+ }}}
+
+ In the embedded case, the `IMPORT` keyword can also refer to named Pig objects that are compiled elsewhere in the host language.
+
+ {{{
+
+ bot_cleanser = Pig.compile(bc, """define bot_cleanser(user) {
+ A = load 'bc_input' using TempStorage();
+ B = filter A by not is_a_bot($user);
+ store B into 'bc_output' using TempStorage();
+ }
+ """)
+
+ main = Pig.compile("""import bc;
+ A = load 'fact';
+ store A into 'bc_input' using TempStorage();
+ inline bot_cleanser('username');
+ B = load 'bc_output' using TempStorage();
+ C = group B by user;
+ ...
+ store Z into 'processed';
+ """)
+ }}}
+
+ Implementation note: this means that Pig will need to keep a symbol table of these named objects to search when resolving these index statements. Named modules in the host language will take
+ precedence over filenames when importing. So if there is an `import x` statement in an embedded pipeline and a separate Pig object named `x= and a file in the current working directory named =x`,
+ the Pig object will be used and not the file. If the file is desired, it can be referred to by the full path or by `./x`.
+
+ Implementation note: the parser will need to rewrite names in inlined macros. Consider a script like the following:
+
+ {{{
+ define my_macro (input, output) {
+ A = load '$input' using TempStorage();
+ B = filter A by my_filter(*);
+ store B into '$output' using TempStorage();
+ }
+
+ A = load 'foo';
+ B = load 'bar';
+ store B into 'bla' using TempStorage();
+ inline my_macro(bla, stuff);
+ C = load 'stuff' from TempStorage();
+ D = join A on $0, C on $0;
+ }}}
+
+ Since A has been re-assigned in the macro, the line `D = join A on $0, C on $0` will connect the A from the macro rather than the original A (which is clearly what the user intended). Therefore,
+ when the parser inlines the macro it will need to rewrite variable names to guarantee they are unique. The rewritten names will be of the form `__macro_<macro_name>_<original_id>_<sequence_number>`
+ where sequence number will be a unique identify in this script. So if the above was the third macro inlined in this script it would be rewritten to:
+ {{{
+
+ A = load 'foo';
+ B = load 'bar';
+ store B into 'bla' using TempStorage();
+ __macro_my_macro_A_3 = load '$input' using TempStorage();
+ __macro_my_macro_B_3 = filter __macro_my_macro_A_3 by my_filter(*);
+ store __macro_my_macro_B_3 into '$output' using TempStorage();
+ C = load 'stuff' from TempStorage();
+ D = join A on $0, C on $0;
+ }}}
+
+ The lines will need to be annotated with information about the original variable names so that the parser can give adequate error messages.
+
+ === API for Pig Object ===
+ The following describes the proposed API for the Pig class. This class will be built in Java so that it can be used across languages. The class is abstract since implementations of `bind()` will
+ need to differ depending on the target language.
+
+ {{{
+
+ public abstract class Pig {
+
+ /**
+ * Run a filesystem command. Any output from this command is written to
+ * stdout or stderr as appropriate.
+ * @param cmd Filesystem command to run along with its arguments as one
+ * string.
+ * @throws IOException
+ */
+ public static void fs(String cmd) throws IOException {...}
+
+ /**
+ * Register a jar for use in Pig. Once this is done this jar will be
+ * registered for <b>all subsequent</b> Pig pipelines in this script. If you wish to
+ * register it for only a single Pig pipeline, use register within that
+ * definition.
+ * @param jarfile Path of jar to include.
+ * @throws IOException if the indicated jarfile cannot be found.
+ */
+ public static void register(String jarfile) throws IOException {...}
+
+ /**
+ * Define an alias for a UDF or a streaming command. This definition
+ * will then be present for <b>all subsequent</b> Pig pipelines defined in this
+ * script. If you wish to define it for only a single Pig pipeline, use
+ * define within that definition.
+ * @param alias name of the defined alias
+ * @param definition string this alias is defined as
+ */
+ public static void define(String alias, String definition) {...}
+
+ /**
+ * Set a variable for use in Pig Latin. This set
+ * will then be present for <b>all subsequent</b> Pig pipelines defined in this
+ * script. If you wish to set it for only a single Pig pipeline, use
+ * set within that definition.
+ * @param var variable to set
+ * @param value to set it to
+ */
+ public static void set(String var, String value) {...}
+
+ /**
+ * Define a Pig pipeline.
+ * @param pl Pig Latin definition of the pipeline.
+ * @return Pig object representing this pipeline.
+ * @throws IOException if the Pig Latin does not compile.
+ */
+ public static Pig compile(String pl) throws IOException {...}
+
+ /**
+ * Define a named portion of a Pig pipeline. This allows it
+ * to be imported into another pipeline.
+ * @param name Name that will be used to define this pipeline.
+ * The namespace is global.
+ * @param pl Pig Latin definition of the pipeline.
+ * @return Pig object representing this pipeline.
+ * @throws IOException if the Pig Latin does not compile.
+ */
+ public static Pig compile(String name, String pl) throws IOException {...}
+
+ /**
+ * Define a Pig pipeline based on Pig Latin in a separate file.
+ * @param filename File to read Pig Latin from. This must be a
+ * purely Pig Latin file. It cannot contain host language constructs
+ * in it.
+ * @return Pig object representing this pipeline.
+ * @throws IOException if the Pig Latin does not compile or the file
+ * cannot be found.
+ */
+ public static Pig compileFromFile(String filename) throws IOException {...}
+
+ /**
+ * Define a named Pig pipeline based on Pig Latin in a separate file.
+ * This allows it to be imported into another pipeline.
+ * @param name Name that will be used to define this pipeline.
+ * The namespace is global.
+ * @param filename File to read Pig Latin from. This must be a
+ * purely Pig Latin file. It cannot contain host language constructs
+ * in it.
+ * @return Pig object representing this pipeline.
+ * @throws IOException if the Pig Latin does not compile or the file
+ * cannot be found.
+ */
+ public static Pig compileFromFile(String name,
+ String filename) throws IOException {...}
+
+ /**
+ * Bind a Pig object to variables in the host language (optional
+ * operation). This does an implicit mapping of variables in the host
+ * language to parameters in Pig Latin. For example, if the user
+ * provides a Pig Latin statement
+ * <tt> p = Pig.compile("A = load '$input';");</tt>
+ * and then calls this function it will look for a variable called
+ * <tt>input</tt> in the host language. Scoping rules of the host
+ * language will be followed in selecting which variable to bind. The
+ * variable bound must contain a string value. This method is optional
+ * because not all host languages may support searching for in scope
+ * variables.
+ * @throws IOException if host language variables are not found to resolve all
+ * Pig Latin parameters or if they contain unsupported types.
+ */
+ public abstract void bind() throws IOException;
+
+ /**
+ * Bind a Pig object to a set of variables. This must be called before the
+ * run if there are any variables in the Pig object. Values must be provided
+ * for all Pig Latin parameters.
+ * @param vars map of variables to bind. Keys should be parameters defined
+ * in the Pig Latin. Values should be strings that provide values for those
+ * parameters. They can be either constants or variables from the host
+ * language. Host language variables must contain strings.
+ * @throws IOException if there is not a key for each
+ * Pig Latin parameter or if they contain unsupported types.
+ */
+ public void bind(Map<String, String> vars) throws IOException {...}
+
+ /**
+ * Bind a Pig object to multiple sets of variables. This will
+ * cause the Pig Latin script to be executed in parallel over these sets of
+ * variables.
+ * @param vars list of maps of variables to bind. Keys should be parameters defined
+ * in the Pig Latin. Values should be strings that provide values for those
+ * variables. They can be either constants or variables from the host
+ * language. Host language variables must be strings.
+ * @throws IOException if there is not a key for each
+ * Pig Latin parameter or if they contain unsupported types.
+ */
+ public void bind(List<Map<String, String>> vars) throws IOException {...}
+
+ /**
+ * Run a pipeline on Hadoop. If there are no stores in this pipeline then nothing
+ * will be run. Bind must be called before this method.
+ * @return a list of PigResults, one for each map of variables passed
+ * to bind.
+ */
+ public List<PigResult> run() {...}
+
+ /**
+ * Run illustrate for a Pig pipeline. Results will be printed
+ * to stdout. If a list of variables have been bound to this Pig object,
+ * only one will be used for this illustrate. Bind must be called before
+ * this method.
+ * @throws IOException if illustrate fails.
+ */
+ public void illustrate() throws IOException {...}
+
+ /**
+ * Explain a Pig pipeline. Results will be printed to stdout.
+ * If a list of variables have been bound to this Pig object,
+ * only one will be used for this explain. Bind must be called before this
+ * method.
+ * @throws IOException if explain fails.
+ */
+ public void explain() throws IOException {...}
+
+ /**
+ * Describe the schema of an alias in a Pig pipeline.
+ * Results will be printed to stdout.
+ * If a list of variables have been bound to this Pig object,
+ * only one will be used for this describe. Bind must be called before
+ * this method.
+ * @param alias to be described
+ * @throws IOException if describe fails.
+ */
+ public void describe(String alias) throws IOException {...}
+ }
+
+ public class PigResult {
+
+ /**
+ * Get a summary status of all stores in the Pig pipeline.
+ * @return ExecJob.JOB_STATUS enum. COMPLETED indicates that all
+ * stores in the script finished successfully, FAILED indicates that
+ * at least one (not necessarily all) failed.
+ */
+ public ExecJob.JOB_STATUS getSummaryStatus() {...}
+
+ /**
+ * Get the status of a particular store in a Pig pipeline.
+ * @param alias Name of alias that was stored
+ * @return ExecJob.JOB_STATUS enum indicating status of the store or
+ * null if there is no such alias.
+ */
+ public ExecJob.JOB_STATUS getStatus(String alias) {...}
+
+ /**
+ * Get status of the first store in a Pig pipeline. This method should
+ * only be used in cases where there is a single store in the pipeline.
+ * @return ExecJob.JOB_STATUS enum indicating status of the store
+ */
+ public ExecJob.JOB_STATUS getStatus() {...}
+
+ /**
+ * Get statistics from a particular store in a Pig pipeline.
+ * @param alias Name of the alias that was stored.
+ * @return PigStats object or null if there is no such alias.
+ */
+ public PigStats getStatistics(String alias) {...}
+
+ /**
+ * Get statistics from the first store in a Pig pipeline. This method should
+ * only be used in cases where there is a single store in the pipeline.
+ * @return PigStats object
+ */
+ public PigStats getStatistics() {...}
+
+ /**
+ * Open an iterator to read results from a store.
+ * @param alias Name of the alias that was stored.
+ * @return iterator of tuples from this store or
+ * null if there is no such alias.
+ */
+ public Iterator<Tuple> getResults(String alias) {...}
+
+ /**
+ * Open an iterator to read results from the first store. This method should
+ * only be used in cases where there is a single store in the pipeline.
+ * @return iterator of tuples from the first store.
+ */
+ public Iterator<Tuple> getResults() {...}
+
+ }
+
+ }}}
+
+ === Requirements This Drives on the Parser ===
+ The Pig class will need a way to annotate the Pig Latin with notes that the new parser can then keep and place in the AST. For example line numbers and file names,
+ which will be completely different than the line numbers and file names the Pig Latin parser will see (since it will only see sections the constructed pipelines, not the script as a
+ whole). It is convenient to do this in comment syntax so that we do not introduce more tokens or keywords into Pig Latin. A comment of the form
+
+ {{{
+ /*@@@ key=value */
+ }}}
+
+ will be taken to be an annotation. Key here will be one of a set of predefined keys that the parser will understand how to use to annotate a node in the parse tree.
+ || '''Key''' || '''Meaning''' ||
+ || linenumber || Original line number in the source file that this line occurred on. ||
+ || filename || Original file this line occurred in. ||
+ || origid || Original id assigned to in this line (used because ids are rewritten during macro expansion). ||
+
+ The parser will need to change to have a parse only mode, where it does only parsing and some very basic semantic checks to support up front error checking on the
+ embedded Pig Latin. In this upfront mode the parser will also have to support certain uncertainties that would not be valid Pig Latin, such as
+ the presence of unresolved parameters in the code.
+
+ Finally, the parser will need to change to support the new macro expansion feature. This will require some integration with the parameter substitution functionality, as well as keeping a
+ symbol table of macros that have been seen.
+
+ === Changes to bin/pig ===
+ Currently `bin/pig`, the executable for Pig, invokes Java directly with `org.apache.pig.Main` as the main class. Now, with this embedded Pig Latin it will be necessary for
+ `bin/pig` to decide if it is working with standard Pig Latin (which it will not be possible to run through the host language) or embedded Pig Latin. Embedded Pig Latin will only
+ be supported in file mode, not interactive mode, so in any case except being invoked with a script file, `bin/pig` will know to use standard Pig Latin. In cases where `bin/pig` is
+ provided with a script it will look for a `!#` line at the beginning of the file. If the basename portion of the path matches `python` or `jython` it will use the reference
+ embedding implementation. If others add other implementations in the future these can be automatically invoked from here as well. The user can also explicitly request using
+ embedded Pig Latin by adding `--embedded` to the command line of `bin/pig`. If this is passed an argument that argument will be taken to refer to which language the Pig Latin is
+ embedded in. With no arguments it will be taken to refer to the reference implementation.
+
+ Once `bin/pig` has decided that this is an embedded Pig Latin script it will invoke
+ Java with the appropriate jars for that host language with the host language file. So, for example, if a user passes a Pig file embedded in Python to `bin/pig` as:
+
+ {{{
+ bin/pig myembedded.py
+ }}}
+
+ `bin/pig` will invoke this via Jython as
+
+ {{{
+ java -cp <jython jars>:<pig jars> /tmp/myembedded.py
+ }}}
+
+ == Use Cases ==
+ Talking to Pig users who desired this functionality, several use cases were discovered. Each of those will be discussed here along with an example of how Turing Complete Pig Latin would satisfy the
+ use case.
+
+ === Convergence ===
+ There are a whole class of problems that involve iterating over a pipeline an indeterminate number of times. Examples include running until values or error converge, graph traversal algorithms, etc.
+ `while` and `for` loops in the host language along with the ability to test the output of a pipeline and either run it again or move to the next step will enable users to satisfy this use case. For
+ example:
+
+ {{{
+ p = Pig.compile("""A = load '$in’;
+ C = group A by user;
+ D = foreach C generate group, myUDF(A);
+ store D into '$out’;
- B = group A all;
+ F = group D all;
- C = foreach B generate flatten(doSomeCalculation(A)) as (result, error);
- $error = foreach C generate error;
- store C into 'outfile';
- if error > 1.0:
- PIG:
- fs mv 'outfile' 'infile';
- infile = 'infile';
+ G = foreach F generate MAX(D);
+ store G into 'tmp’;
+ """)
+ error = 100.0
+ input = "original”
+ output = "output-0”
+ final = "final-output”
- def doSomeCalculation(A):
- for x in A:
+ for i in range(1, 100):
+ p.bind({'in':input, 'out':output}) # attaches $in, $out in Pig Latin to input, output Python variables
+ results = p.run()
+
+ if results.getStatus("D") `= "FAILED" or results.getStatus("G") =` "FAILED":
+ raise "Pig job failed"
+ iter = results.getResults("G")
+ if iter.next() > 1:
+ input = output
+ output = "output-" + i.to_s
+ else:
+ H = Pig.fs("mv " + output + " " + final)
+ break
+ }}}
+
+ === Function Calls ===
+ Users often decompose their Pig Latin into modules and then wish to invoke scripts from these modules. Today this is being done via the `exec` command. The new inline functionality will make this
+ easier and allow argument passing. Note that macro inlining is not a complete replacement for function calls, as it does not support recursive calls. However, we found no examples where
+ recursive function calls were necessary. An example of this use case is:
+
+ foo.pig:
+ {{{
+ define foo(input, output) {
+ A = load '$input' using TempStorage();
+ ...
+ store Z into '$output' using TempStorage();
+ }
+ }}}
+
+ bar.pig:
+ {{{
+ define bar(input, output) {
+ A = load '$input' using TempStorage();
+ ...
+ store Z into '$output' using TempStorage();
+ }
+ }}}
+
+ main.pig:
+ {{{
+ import foo.pig;
+ import bar.pig;
+
+ A = load 'raw';
+ store A into 'foo_in' using TempStorage();
+ inline foo('foo_in', 'foo_out');
+ B = load 'foo_out' using TempStorage();
+ store B into 'bar_in' using TempStorage();
+ inline bar('bar_in', 'bar_out');
+ C = load 'bar_out' using TempStorage();
+ ...
+ }}}
+
+ === Automated Pig Latin Generation ===
+ A number of user frameworks have sprung up that do automated generation of Pig Latin. Usually this is used to do things such as set the proper date for the data to be loaded, etc. An example of
+ this use case is:
+
+ {{{
+ today = sys.argv[1]
+ p = Pig.compile("""A = load 'fact' using HowlLoader();
+ B = filter A by datestamp = '$date';
- ...
+ ...
+ store Z into 'aggregated' using HowlStorage('datestamp = $date');
+ """)
+ p.bind({'date':today})
+ p.run()
}}}
+ ==== Conditional Compilation ====
+ A sub-use case of automated generation is conditional code generation. For example, different processing might be required based on whether this is weekday or a weekend:
+ str = "A = load 'input';"
+ if today.isWeekday():
+ str = str + "B = filter A by weekday_filter(*);"
- A preprocessor could then be applied to the above that would convert this to a form of Jython that the embedding functionality provided as part of
- the `pyg.tgz` patch already attached to [[https://issues.apache.org/jira/browse/PIG-928|PIG-928]] can run. In other words the above would be
- converted into something like the second example that uses the !PigServer interface. This preprocessor could also submit the
- Pig Latin portions of the script for syntactic and semantic checking.
-
- This preprocessor would find the Pig Latin segments of code via the `PIG:` tag. `PIG:` would have the same block scoping rules as
- other block operators in Python. Variables from the Python code would be imported to and exported from the Pig Latin via parameter
- substitution syntax (e.g. notice how variables `infile` and `error` appear as `$infile` and `$error` inside the Pig Latin).
-
- The last drawback that this proposal does not address is that we have to pick a particular
- scripting language to embed Pig Latin in. There are two solution I can see here:
- 1. Do our best to pick a good one and live with people's unhappiness.
- 1. Write the preprocessor in such a way that it is relatively easy to switch languages it would embed Pig Latin into.
-
- While option two sounds nice it greatly complicates the project. And realistically how many people are going to take on all the
- work to port it to another language? Based on this I suggest we go with option one and choose Jython as the scripting language. I
- vote for Jython for two reasons. One, in order to meet the goal of embedding functions from the scripting language directly as UDFs
- we have to pick a language that compiles to Java byte code. That leaves us with Jython, Jruby, Groovy, or !JavaScript. Out of that
- field we already have half of the implementation we need in Jython with [[https://issues.apache.org/jira/browse/PIG-928|PIG-928]].
-
- Thoughts? Preferences for one of the options I did not like? Comments welcome.
-
- == Approach 2 ==
- And now for something completely different.
-
- After thinking on the above for a week or so it occurs to me that in dismissing making Pig Latin itself Turing complete I am conflating two tasks
- that could be decoupled. The first is defining a grammar for the language and extending the parser. The second is building an execution engine to execute
- Pig Latin scripts. It is the second that I am concerned is too much work. Defining the grammar and building the parser is relatively easy (as
- we say in the Pig team at Yahoo, "parsers are easy").
-
- So what if we did extend Pig Latin itself to be Turing complete, but the first pass over the language was to compile it down to Java code that made
- use of the existing !PigServer class to execute the code? This meets all ten requirements given above (some extra work will need to be done to meet
- requirement 8 on up front semantic checking, but it is possible). It deals with my initial concern that supporting Turing completeness in Pig Latin
- is too much work. It also has the exceedingly nice feature that we do not have to pick any one scripting language. The more I talked to people the
- more I discovered some wanted Python, some Ruby, some Perl, some Groovy, etc. This avoids that problem. And the extensions to Pig Latin themselves
- will be simple enough that it should not be onerous for people to learn it. It also means that at some future time if we decide that we want more
- control over how the language is executed we can make changes without people needing to switch from whatever scripting language we embed it in.
-
- A significant downside to this proposal is now users have to have a Java compiler along to run their Pig Latin scripts.
-
- The other concerns I gave above about making Pig Latin Turing complete are somewhat addressed, but not totally. It would be possible, though
- painful, to use a Java debugger on the generated Java code. Syntax highlighting and completion files could be created for Vim, Emacs, Eclipse, and
- whatever other favorite editors people have.
-
- === Specifics ===
- The grammar of the language should be kept as simple as possible. The goal is not to create a general purpose programming language.
- Tasks requiring these features should still be written in UDFs in Java or a scripting language.
-
- Each Pig Latin file would be considered as a module. All functions would have global scope within that module and would be visible once the module is
- imported.
-
- The type system would be existing Pig Latin types (we may need to add a list type). Types would be bound at run time (this is necessary to support
- existing PL grammar where A = load ... is a declaration of A).
-
- The grammar would look something like:
-
- {{{
- program:
- import
- | register
- | define
- | func_definition
- | block
-
- import:
- IMPORT _modulename_ namespace_clause
-
- namespace_clause:
- (empty)
- | AS _namespacename_
-
- register:
- ... // as now
-
- define:
- ... // as now
-
- func_definition:
- DEF _functionname_ ( arg_list ) { block }
- // not sure about this, having DEF and DEFINE different keywords.
- // May want to reuse DEFINE here or DEFINE FUNCTION
-
- arg_list:
- expr
- | arg_list , expr
-
- block:
- statement
- | block statement
-
- statement:
- ;
- | assignment
- | if
- | while
- | for
- | return // only valid inside functions
- | CONTINUE ; // only valid inside loops
- | BREAK ; // only valid inside loops
- | split
- | store
- | dump
- | fs
-
- assignment:
- _var_ = expr ;
- | _var_ = LOAD _inputsrc_ ;
- ... // GROUP, FILTER, etc. as now
-
- statement_or_block:
- statement
- | { block }
-
- if:
- IF ( expr ) statement_or_block else
-
else:
+ str = str + "B = filter A by weekend_filter(*);"
+ str = str + "C = group B by user;"
- (empty)
- | ELSE statement_or_block
-
- while:
- WHILE ( expr ) statement_or_block
-
- for:
- FOR ( assignment ; expr ; expr ) statement_or_block
-
- return:
- RETURN ;
- | RETURN expr ;
-
- // split, dump, store, fs as now
}}}
- So the example given initially would look like:
+ ==== Parallel Execution ====
+ Another sub-use case of automated generation is parallel execution of identical pipelines. Often users have a single pipeline that they would like to run multiple data sets through in parallel.
+ Using the `bind` option with a map of variables satisfies this use case. For example, the following will run the pipeline for the US, the UK, and Brazil:
+
{{{
+ p = Pig.compile("""A = load '$in';
+ B = filter A by user is not null;
+ ...
+ store Z into '$out';
+ """)
+
+ p.bind([{'in':'us_raw','out':'us_processed'},
+ {'in':'uk_raw','out':'uk_processed'},
+ {'in':'brazil_raw','out':'brazil_processed'}])
+ p.run()
+ }}}
+
+ == Alternate Proposals ==
+ Other approaches were considered. These and the reasons for rejecting them are discussed here.
+
+ === Add a runtime to Pig Latin ===
+ We could go the whole nine yards and add a runtime to Pig Latin. The upside to this is that it will allow for a simpler design (no need to integrate with a
+ host language) and for greater control which will facilitate certain types of optimizations.
+
+ However, the drawbacks here are huge. The level of effort to fully implement a runtime is extensive. Also, given that there are no lack of scripting languages in
+ the world it seems a complete waste. And the resulting language will not have any debuggers, IDE support, etc.
+
+ === Turn Pig into a library ===
+ Pig could be changed become a library that could be embedded into a JVM compatible language of the users choice. This is simpler from the Pig team's perspective as there is no need
+ to support a pre-parser.
+
+ The drawbacks to this approach are that the results code is not Pig Latin anymore. It would instead look something like:
+
+ {{{
- error = 100.0;
+ error = 100.0
- infile = 'original.data';
- outfile = 'result.data';
- while (error > 1.0) {
- A = load infile;
- B = group A all;
- C = foreach B generate flatten(doSomeCalculation(A)) as (result, error);
- error = foreach C generate error;
- store C into outfile;
- if (error > 1.0) fs mv outfile infile;
+ input = "original”
+ output = "output-0”
+ final = "final-output”
+
+ for i in range(1, 100):
+ A = Pig.load(input)
+ C = A.group("user")
+ D = C.foreach("group", "myUDF(A)")
+ D.store(output)
+ F = D.group(Pig.all)
+ G = F.foreach("MAX(E)")
+ Result r = G.store("tmp")
- }
+
+ iter = r.iterator()
+ if iter.next() > 1:
+ input = output
+ output = "output-" + i.to_s
- }}}
-
- and would compile down to Java as:
-
- {{{
- import org.apache.pig.PigServer;
-
- public class PigLatinScript {
-
- public static void main(string[] args) {
- Object error = new Double(100.0);
- Object infile = new String("original.data");
- Object outfile = new String("result.data");
- while (error != null && (Double)error > 1.0) {
- PigServer ps = new PigServer();
- ps.registerQuery("A = load " + infile + ";");
- ps.registerQuery("B = group A all;");
- ps.registerQuery("C = foreach B generate flatten(doSomeCalculation(A)) as (result, error);");
- ps.registerQuery("error = foreach C generate error;");
- ps.store("C", outfile);
- Iterator<tuple> i = ps.openIterator("error");
- if (i.hasNext()) error = i.next();
- else error = null;
- if (error != null && (Double)error > 1.0) fs mv outfile infile;
- }
- }
- }
- }}}
-
- === Other Thoughts ===
- Whichever way we do it, we need to consider what built in variables we need in the system. For example, it would be really nice to have a
- status variable so that you could do something like:
-
- {{{
- ...
- store X into 'foo';
- if ($status == 0) { -- or "success" or whatever
- ...
- } else {
+ else:
- ...
- }
+ hdfs.mv(output, final)
+ break
}}}
+ This means we would have different dialects of Pig Latin, depending on whether it was embedded or not.
- == Approach 3 ==
- At the Pig contributor workshop in June 2010 Dmitriy Ryaboy proposed that we go the DSL route in Java. Thus the example given above becomes something like:
- {{{
+ The second issue is that there is no opportunity for checking up front. Because the Pig Latin is not invoked until the controlling script gets to it, it may be
+ hours before simple syntax errors are found.
- public class Main {
+ === Embed Pig Oracle Pro*C Style ===
+ A Pig pre-parser could be written that would allow directly embedding Pig Latin into the host language.
+ {{{
+ error = 100.0
+ input = "original”
+ output = "output-0”
+ final = "final-output”
- public static void main(String[] args) {
- float error = 100.0;
- String infile = "original.data";
- PigBuilder pig = new PigBuilder();
- while (error > 1.0) {
- PigRelation A = pig.load(infile, "piggybank.MyLoader");
- PigRelation B = A.group(pig.ALL);
- // It's not entirely clear to me how nested foreach works in this scenario
- PigRelation C = B.foreach(new MyFunc("A"));
-
- PigIterator pi = pig.openIterator(C, "outfile");
- Tuple t = pi.next();
- error = t.get(1);
- if (error >= 1.0) {
- pig.fs.mv('outfile', 'infile');
- }
- }
- }
- }
- }}}
- This would be accomplished by creating a public interface for Pig operators (here called !PigBuilder, but I'm not proposing that as the actual name) that would
- construct a logical plan and execute it when openIterator is called, much as !PigServer does today. Another way to look at this is !PigServer could be changed to
- expose Pig operators instead of just strings as it does today.
+ for i in range(1, 100):
+ P = Pig(
+ A = load '$input’;
+ C = group A by user;
+ D = foreach C generate group, myUDF(A);
+ store D into '$output’;
+ F = group D all;
+ G = foreach F generate MAX(E);
+ store G into 'tmp’;)
+ P.run()
+ iter = P.result("G").iterator()
+ if iter.next() > 1:
+ input = output
+ output = "output-" + i.to_s
+ else:
+ Pig.fs("mv " + output + " " + final)
+ break
+ }}}
- The beauty of doing this in Java is it facilitates it being used in scripting languages as well. Since Java packages can be directly imported into Jython, JRuby,
- Groovy, and other languages this immediately provides a scripting interface in the language of the users choice.
+ This would allow for up front error. It also has the nice feature that Pig code can be directly placed in the host language where it will be run, instead of needing to be compiled up
+ front and bound and ran later.
- This does violate requirement 10 above (that Pig Latin should appear the same in embedded and non-embedded form), but the cross language functionality may be worth
- it.
+ However, it has a few drawbacks. It requires a pre-parser, which is an added piece and will make it harder to embed Pig in new languages. It will also effectively create new languages
+ (ythonpay?, ubyray?) that will not match syntax highlighters, analysis tools, debuggers, etc. Finally, certain things that are doable in the compile/bind/run model are much harder to do
+ in this model (such as binding to multiple variables at the same time to get parallel pipelines, see below).
+ == Open Questions ==
+
+ 1. How will tools like Inspector Gadget and Karmasphere plug into this and make use of the Pig Latin? Ben suggested that we should build an interface in the pre-parser where tools could plug in and get a chance to look at the Pig Latin before the script is converted to Python. Assumably they would also want to be able to modify the output, doing things like annotating it, etc. Do they need it before it goes to the host language? Can we instead provide hooks to get the AST after the parser (not the pre-parser) sees the code? This would insulate these tools from half formed Pig Latin and incomplete pipelines.
+ 1. Should the Pig interface include the ability to pass streams where stdout and stderr will be written?
+