You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by ol...@apache.org on 2011/05/23 21:46:06 UTC
svn commit: r1126687 [3/3] - in /pig/branches/branch-0.9: ./
src/docs/src/documentation/content/xdocs/
Modified: pig/branches/branch-0.9/src/docs/src/documentation/content/xdocs/udf.xml
URL: http://svn.apache.org/viewvc/pig/branches/branch-0.9/src/docs/src/documentation/content/xdocs/udf.xml?rev=1126687&r1=1126686&r2=1126687&view=diff
==============================================================================
--- pig/branches/branch-0.9/src/docs/src/documentation/content/xdocs/udf.xml (original)
+++ pig/branches/branch-0.9/src/docs/src/documentation/content/xdocs/udf.xml Mon May 23 19:46:05 2011
@@ -25,23 +25,40 @@
</header>
<body>
-<!-- ================================================================== -->
-<!-- WRITING UDFS -->
+<section id="udfs">
+<title>Introduction</title>
+<p>
+Pig provides extensive support for user defined functions (UDFs) as a way to specify custom processing. Pig UDFs can currently be implemented in three languages: Java, Python, and JavaScript.
+</p>
-<section>
-<title>Writing UDFs</title>
+<p>
+The most extensive support is provided for Java functions. You can customize all parts of the processing including data load/store, column transformation, and aggregation. Java functions are also more efficient because they are implemented in the same language as Pig and because additional interfaces are supported such as the <a href="#algebraic-interface">Algebraic Interface</a> and the <a href="#accumulator-interface">Accumulator Interface</a>.
+</p>
<p>
-Pig provides extensive support for user defined functions (UDFs) as a way to specify custom processing.
-This section provides the information you need to write your own Java or Python UDFs.
-The next section describes Piggy Bank, a repository that allows you to access and contribute Java UDFs written by you and other Pig users.
+Limited support is provided for Python and JavaScript functions. These functions are new, still evolving, additions to the system. Currently only the basic interface is supported; load/store functions are not supported. Furthermore, JavaScript is provided as an experimental feature because it did not go through the same amount of testing as Java or Python. At runtime note that Pig will automatically detect the usage of a scripting UDF in the Pig script and will automatically ship the corresponding scripting jar, either Jython or Rhino, to the backend.</p>
+<p></p>
+
+<p>
+Pig also provides support for Piggy Bank, a repository for JAVA UDFs. Through Piggy Bank you can access Java UDFs written by other users and contribute Java UDFs that you have written.
</p>
+
+</section>
+
+
+<!-- ================================================================== -->
+<!-- WRITING UDFS -->
+
+<section id="udf-java">
+<title>Writing Java UDFs</title>
+
<!-- =============================================================== -->
-<section>
+<section id="eval-functions">
<title>Eval Functions</title>
-<section>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="eval-functions-use">
<title>How to Use a Simple Eval Function</title>
<p>Eval is the most common type of function. It can be used in <code>FOREACH</code> statements as shown in this script: </p>
@@ -81,7 +98,8 @@ transformation functions and include agg
</section>
-<section>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="eval-functions-write">
<title> How to Write a Simple Eval Function</title>
<p>Let's now look at the implementation of the <code>UPPER</code> UDF. </p>
@@ -131,7 +149,8 @@ jar -cf myudfs.jar myudfs
</section>
-<section>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="aggregate-functions">
<title>Aggregate Functions</title>
<p>Aggregate functions are another common type of eval function. Aggregate functions are usually applied to grouped data, as shown in this script: </p>
@@ -144,6 +163,11 @@ DUMP C;
</source>
<p>The script above uses the <code>COUNT</code> function to count the number of students with the same name. There are a couple of things to note about this script. First, even though we are using a function, there is no <code>register</code> command. Second, the function is not qualified with the package name. The reason for both is that <code>COUNT</code> is a <code>builtin</code> function meaning that it comes with the Pig distribution. These are the only two differences between builtins and UDFs. Builtins are discussed in more detail later in this document. </p>
+</section>
+
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="algebraic-interface">
+<title>Algebraic Interface</title>
<p>An aggregate function is an eval function that takes a bag and returns a scalar value. One interesting and useful property of many aggregate functions is that they can be computed incrementally in a distributed fashion. We call these functions <code>algebraic</code>. <code>COUNT</code> is an example of an algebraic function because we can count the number of elements in a subset of the data and then sum the counts to produce a final output. In the Hadoop world, this means that the partial computations can be done by the map and combiner, and the final result can be computed by the reducer. </p>
@@ -196,7 +220,90 @@ public interface Algebraic{
<p>Take a look at the <code>COUNT</code> implementation to see how this is done. Note that the <code>exec</code> function of the <code>Initial</code> and <code>Intermed</code> classes is parameterized with <code>Tuple</code> and the <code>exec</code> of the <code>Final</code> class is parameterized with the real type of the function, which in the case of the <code>COUNT</code> is <code>Long</code>. Also, note that the fully-qualified name of the class needs to be returned from <code>getInitial</code>, <code>getIntermed</code>, and <code>getFinal</code> methods. </p>
</section>
-<section>
+<!-- ++++++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="accumulator-interface">
+<title>Accumulator Interface</title>
+
+<p>In Pig, problems with memory usage can occur when data, which results from a group or cogroup operation, needs to be placed in a bag and passed in its entirety to a UDF.</p>
+
+<p>This problem is partially addressed by Algebraic UDFs that use the combiner and can deal with data being passed to them incrementally during different processing phases (map, combiner, and reduce). However, there are a number of UDFs that are not Algebraic, don't use the combiner, but still donât need to be given all data at once. </p>
+
+<p>The new Accumulator interface is designed to decrease memory usage by targeting such UDFs. For the functions that implement this interface, Pig guarantees that the data for the same key is passed continuously but in small increments. To work with incremental data, here is the interface a UDF needs to implement:</p>
+<source>
+public interface Accumulator <T> {
+ /**
+ * Process tuples. Each DataBag may contain 0 to many tuples for current key
+ */
+ public void accumulate(Tuple b) throws IOException;
+ /**
+ * Called when all tuples from current key have been passed to the accumulator.
+ * @return the value for the UDF for this key.
+ */
+ public T getValue();
+ /**
+ * Called after getValue() to prepare processing for next key.
+ */
+ public void cleanup();
+}
+</source>
+
+<p>There are several things to note here:</p>
+
+<ol>
+ <li>Each UDF must extend the EvalFunc class and implement all necessary functions there.</li>
+ <li>If a function is algebraic but can be used in a FOREACH statement with accumulator functions, it needs to implement the Accumulator interface in addition to the Algebraic interface.</li>
+ <li>The interface is parameterized with the return type of the function.</li>
+ <li>The accumulate function is guaranteed to be called one or more times, passing one or more tuples in a bag, to the UDF. (Note that the tuple that is passed to the accumulator has the same content as the one passed to exec â all the parameters passed to the UDF â one of which should be a bag).</li>
+ <li>The getValue function is called after all the tuples for a particular key have been processed to retrieve the final value.</li>
+ <li>The cleanup function is called after getValue but before the next value is processed.</li>
+</ol>
+
+
+<p>Here us a code snippet of the integer version of the MAX function that implements the interface:</p>
+<source>
+public class IntMax extends EvalFunc<Integer> implements Algebraic, Accumulator<Integer> {
+ â¦â¦.
+ /* Accumulator interface */
+
+ private Integer intermediateMax = null;
+
+ @Override
+ public void accumulate(Tuple b) throws IOException {
+ try {
+ Integer curMax = max(b);
+ if (curMax == null) {
+ return;
+ }
+ /* if bag is not null, initialize intermediateMax to negative infinity */
+ if (intermediateMax == null) {
+ intermediateMax = Integer.MIN_VALUE;
+ }
+ intermediateMax = java.lang.Math.max(intermediateMax, curMax);
+ } catch (ExecException ee) {
+ throw ee;
+ } catch (Exception e) {
+ int errCode = 2106;
+ String msg = "Error while computing max in " + this.getClass().getSimpleName();
+ throw new ExecException(msg, errCode, PigException.BUG, e);
+ }
+ }
+
+ @Override
+ public void cleanup() {
+ intermediateMax = null;
+ }
+
+ @Override
+ public Integer getValue() {
+ return intermediateMax;
+ }
+}
+</source>
+</section>
+
+
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="filter-functions">
<title> Filter Functions</title>
<p>Filter functions are eval functions that return a <code>boolean</code> value. Filter functions can be used anywhere a Boolean expression is appropriate, including the <code>FILTER</code> operator or <code>bincond</code> expression. </p>
@@ -264,8 +371,9 @@ public class IsEmpty extends FilterFunc
</source>
</section>
-<section>
-<title> Pig Types</title>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="pig-types">
+<title> Pig Types and Native Java Types</title>
<p>The main thing to know about Pig's type system is that Pig uses native Java types for almost all of its types, as shown in this table. </p>
@@ -389,11 +497,12 @@ public class TOKENIZE extends EvalFunc&l
</source>
</section>
-<section>
-<title> Schema</title>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="schemas">
+<title>Schemas and UDFs</title>
-<p>The latest version of Pig uses type information for validation and performance. It is important for UDFs to participate in type propagation. Until now, our UDFs made no effort to communicate their output schema to Pig. This is because, most of the time, Pig can figure out this information by using Java's <a href="http://java.sun.com/developer/technicalArticles/ALT/Reflection/"> Reflection</a>. If your UDF returns a scalar or a map, no work is required. However, if your UDF returns a <code>tuple</code> or a <code>bag</code> (of tuples), it needs to help Pig figure out the structure of the tuple. </p>
-<p>If a UDF returns a <code>tuple</code> or a <code>bag</code> and schema information is not provided, Pig assumes that the tuple contains a single field of type <code>bytearray</code>. If this is not the case, then not specifying the schema can cause failures. We look at this next. </p>
+<p>Pig uses type information for validation and performance. It is important for UDFs to participate in type propagation. Our UDFs generally make no effort to communicate their output schema to Pig. This is because Pig can usually figure out this information by using Java's <a href="http://java.sun.com/developer/technicalArticles/ALT/Reflection/"> Reflection</a>. If your UDF returns a scalar or a map, no work is required. However, if your UDF returns a tuple or a bag (of tuples), it needs to help Pig figure out the structure of the tuple. </p>
+<p>If a UDF returns a tuple or a bag and schema information is not provided, Pig assumes that the tuple contains a single field of type bytearray. If this is not the case, then not specifying the schema can cause failures. We look at this next. </p>
<p>Let's assume that we have UDF <code>Swap</code> that, given a tuple with two fields, swaps their order. Let's assume that the UDF does not specify a schema and look at the scripts below: </p>
<source>
@@ -480,7 +589,9 @@ public class DataType {
</source>
<p>You need to import the <code>org.apache.pig.data.DataType</code> class into your code to define schemas. You also need to import the schema class <code>org.apache.pig.impl.logicalLayer.schema.Schema</code>. </p>
-<p>The example above shows how to create an output schema for a tuple. Doing this for a bag is very similar. Let's extend the <code>TOKENIZE</code> function to do that: </p>
+<p>The example above shows how to create an output schema for a tuple. Doing this for a bag is very similar. Let's extend the <code>TOKENIZE</code> function to do that.</p>
+
+<p>As you can see in the example below, this is very similar to the output schema definition in the <code>Swap</code> function. One difference is that instead of reusing input schema, we create a brand new field schema to represent the tokens stored in the bag. The other difference is that the type of the schema created is BAG (not TUPLE). </p>
<source>
package org.apache.pig.builtin;
@@ -525,12 +636,14 @@ public class TOKENIZE extends EvalFunc&l
}
}
</source>
-
-<p>As you can see, this is very similar to the output schema definition in the <code>Swap</code> function. One difference is that instead of reusing input schema, we create a brand new field schema to represent the tokens stored in the bag. The other difference is that the type of the schema created is <code>BAG</code> (not TUPLE). </p>
+<p></p>
+<p>One more note about schemas and UDFs. Users have requested the ability to examine the input schema of the data before processing the data via a UDF. For example, they would like to know how to convert an input tuple to a map such that the keys in the map are the names of the input columns. Currently there is no way to do this. This is a feature we would like to support in the future.</p>
</section>
-<section>
+
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="error-handling">
<title> Error Handling</title>
<p>There are several types of errors that can occur in a UDF: </p>
@@ -545,7 +658,8 @@ public class TOKENIZE extends EvalFunc&l
</section>
-<section>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="function-overloading">
<title>Function Overloading</title>
<p>Before the type system was available in Pig, all values for the purpose of arithmetic calculations were assumed to be doubles as the safest choice. However, this is not very efficient if the data is actually of type integer or long. (We saw about a 2x slowdown of a query when using double where integer could be used.) Now that Pig supports types we can take advantage of the type information and choose the function that is most efficient for the provided operands. </p>
@@ -691,7 +805,8 @@ DUMP B;
</section>
-<section>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="reporting-progress">
<title>Reporting Progress</title>
<p>A challenge of running a large shared system is to make sure system resources are used efficiently. One aspect of this challenge is detecting runaway processes that are no longer making progress. Pig uses a heartbeat mechanism for this purpose. If any of the tasks stops sending a heartbeat, the system assumes that it is dead and kills it. </p>
@@ -716,7 +831,8 @@ public class UPPER extends EvalFunc<S
</source>
</section>
-<section>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="distributed-cache">
<title>Using Distributed Cache</title>
<p>Use getCacheFiles, an EvalFunc method, to return a list of HDFS files that need to be shipped to distributed cache. Inside EvalFunc, you can assume that these files already exist in distributed cache. For example:</p>
<source>
@@ -741,7 +857,8 @@ dump b;
</source>
</section>
-<section>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="import-lists">
<title>Import Lists</title>
<p>An import list allows you to specify the package to which a UDF or a group of UDFs belong,
eliminating the need to qualify the UDF on every call. An import list can be specified via the udf.import.list Java
@@ -768,7 +885,7 @@ pig -cp sds.jar -Dudf.import.list=com.ya
<!-- =============================================================== -->
<!-- BEGIN LOAD/STORE FUNCTIONS -->
-<section>
+<section id="load-store-functions">
<title> Load/Store Functions</title>
<p>The load/store user defined functions control how data goes into Pig and comes out of Pig. Often, the same function handles both input and output but that does not have to be the case. </p>
@@ -779,7 +896,8 @@ This enables you to create new LoadFunc
<strong>Note:</strong> Both the LoadFunc and StoreFunc implementations should use the Hadoop 20 API based classes (InputFormat/OutputFormat and related classes) under the <strong>new</strong> org.apache.hadoop.mapreduce package instead of the old org.apache.hadoop.mapred package.
</p>
-<section>
+<!-- +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="load-functions">
<title> Load Functions</title>
<p><a href="http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/LoadFunc.java?view=markup">LoadFunc</a>
abstract class has the main methods for loading data and for most use cases it would suffice to extend it. There are three other optional interfaces which can be implemented to achieve extended functionality: </p>
@@ -800,7 +918,7 @@ has methods to push operations from Pig
has methods to convert byte arrays to specific types. A loader implementation should implement this if casts (implicit or explicit) from DataByteArray fields to other types need to be supported. </li>
</ul>
- <p>The LoadFunc abstract class is the main class to extend for implementing a loader. The methods which need to be overriden are explained below:</p>
+ <p id="loadfunc-abstract">The LoadFunc abstract class is the main class to extend for implementing a loader. The methods which need to be overridden are explained below:</p>
<ul>
<li>getInputFormat(): This method is called by Pig to get the InputFormat used by the loader. The methods in the InputFormat (and underlying RecordReader) are called by Pig in the same manner (and in the same context) as by Hadoop in a MapReduce java program. If the InputFormat is a Hadoop packaged one, the implementation should use the new API based one under org.apache.hadoop.mapreduce. If it is a custom InputFormat, it should be implemented using the new API in org.apache.hadoop.mapreduce.<br></br> <br></br>
@@ -809,10 +927,10 @@ has methods to convert byte arrays to sp
<li>setLocation(): This method is called by Pig to communicate the load location to the loader. The loader should use this method to communicate the same information to the underlying InputFormat. This method is called multiple times by pig - implementations should bear this in mind and should ensure there are no inconsistent side effects due to the multiple calls. </li>
<li>prepareToRead(): Through this method the RecordReader associated with the InputFormat provided by the LoadFunc is passed to the LoadFunc. The RecordReader can then be used by the implementation in getNext() to return a tuple representing a record of data back to pig. </li>
- <li>getNext(): The meaning of getNext() has not changed and is called by Pig runtime to get the next tuple in the data - in this method the implementation should use the the underlying RecordReader and construct the tuple to return. </li>
+ <li>getNext(): The meaning of getNext() has not changed and is called by Pig runtime to get the next tuple in the data - in this method the implementation should use the underlying RecordReader and construct the tuple to return. </li>
</ul>
- <p>The following methods have default implementations in LoadFunc and should be overridden only if needed: </p>
+ <p id="loadfunc-default">The following methods have default implementations in LoadFunc and should be overridden only if needed: </p>
<ul>
<li>setUdfContextSignature(): This method will be called by Pig both in the front end and back end to pass a unique signature to the Loader. The signature can be used to store into the UDFContext any information which the Loader needs to store between various method invocations in the front end and back end. A use case is to store RequiredFieldList passed to it in LoadPushDown.pushProjection(RequiredFieldList) for use in the back end before returning tuples in getNext(). The default implementation in LoadFunc has an empty body. This method will be called before other methods. </li>
<li>relativeToAbsolutePath(): Pig runtime will call this method to allow the Loader to convert a relative load location to an absolute location. The default implementation provided in LoadFunc handles this for FileSystem locations. If the load source is something else, loader implementation may choose to override this.</li>
@@ -936,30 +1054,30 @@ public class SimpleTextLoader extends Lo
<!-- END LOAD FUNCTION -->
<!-- =============================================================== -->
-<section>
+<section id="store-functions">
<title> Store Functions</title>
-<p><a href="http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/StoreFunc.java?view=markup">StoreFunc</a>
+<p id="storefunc"><a href="http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/StoreFunc.java?view=markup">StoreFunc</a>
abstract class has the main methods for storing data and for most use cases it should suffice to extend it. There is an optional interface which can be implemented to achieve extended functionality: </p>
<ul>
-<li><a href="http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/StoreMetadata.java?view=markup">StoreMetadata:</a>
-This interface has methods to interact with metadata systems to store schema and store statistics. This interface is truely optional and should only be implemented if metadata needs to stored. </li>
+<li id="storemetadata"><a href="http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/StoreMetadata.java?view=markup">StoreMetadata:</a>
+This interface has methods to interact with metadata systems to store schema and store statistics. This interface is optional and should only be implemented if metadata needs to stored. </li>
</ul>
-<p>The methods which need to be overridden in StoreFunc are explained below: </p>
+<p id="storefunc-override">The methods which need to be overridden in StoreFunc are explained below: </p>
<ul>
<li>getOutputFormat(): This method will be called by Pig to get the OutputFormat used by the storer. The methods in the OutputFormat (and underlying RecordWriter and OutputCommitter) will be called by pig in the same manner (and in the same context) as by Hadoop in a map-reduce java program. If the OutputFormat is a hadoop packaged one, the implementation should use the new API based one under org.apache.hadoop.mapreduce. If it is a custom OutputFormat, it should be implemented using the new API under org.apache.hadoop.mapreduce. The checkOutputSpecs() method of the OutputFormat will be called by pig to check the output location up-front. This method will also be called as part of the Hadoop call sequence when the job is launched. So implementations should ensure that this method can be called multiple times without inconsistent side effects. </li>
<li>setStoreLocation(): This method is called by Pig to communicate the store location to the storer. The storer should use this method to communicate the same information to the underlying OutputFormat. This method is called multiple times by pig - implementations should bear in mind that this method is called multiple times and should ensure there are no inconsistent side effects due to the multiple calls. </li>
<li>prepareToWrite(): In the new API, writing of the data is through the OutputFormat provided by the StoreFunc. In prepareToWrite() the RecordWriter associated with the OutputFormat provided by the StoreFunc is passed to the StoreFunc. The RecordWriter can then be used by the implementation in putNext() to write a tuple representing a record of data in a manner expected by the RecordWriter. </li>
-<li>putNext(): The meaning of putNext() has not changed and is called by Pig runtime to write the next tuple of data - in the new API, this is the method wherein the implementation will use the the underlying RecordWriter to write the Tuple out.</li>
+<li>putNext(): The meaning of putNext() has not changed and is called by Pig runtime to write the next tuple of data - in the new API, this is the method wherein the implementation will use the underlying RecordWriter to write the Tuple out.</li>
</ul>
-<p>The following methods have default implementations in StoreFunc and should be overridden only if necessary: </p>
+<p id="storefunc-default">The following methods have default implementations in StoreFunc and should be overridden only if necessary: </p>
<ul>
<li>setStoreFuncUDFContextSignature(): This method will be called by Pig both in the front end and back end to pass a unique signature to the Storer. The signature can be used to store into the UDFContext any information which the Storer needs to store between various method invocations in the front end and back end. The default implementation in StoreFunc has an empty body. This method will be called before other methods.
</li>
<li>relToAbsPathForStoreLocation(): Pig runtime will call this method to allow the Storer to convert a relative store location to an absolute location. An implementation is provided in StoreFunc which handles this for FileSystem based locations. </li>
-<li>checkSchema(): A Store function should implement this function to check that a given schema describing the data to be written is acceptable to it. The default implementation in StoreFunc has an empty body. This method will be called before any calls to setStoreLocation(). </li>
+<li id="checkschema">checkSchema(): A Store function should implement this function to check that a given schema describing the data to be written is acceptable to it. The default implementation in StoreFunc has an empty body. This method will be called before any calls to setStoreLocation(). </li>
</ul>
<p><strong>Example Implementation</strong></p>
@@ -1170,152 +1288,92 @@ public class SimpleTextStorer extends St
<!-- END LOAD/STORE FUNCTIONS -->
+
+
<!-- =============================================================== -->
<section>
-<title>Accumulator Interface</title>
+<title>Advanced Topics</title>
-<p>In Pig, problems with memory usage can occur when data, which results from a group or cogroup operation, needs to be placed in a bag and passed in its entirety to a UDF.</p>
+<section id="udf-interfaces">
+<title>UDF Interfaces</title>
+<p>Java UDFs can be invoked multiple ways. The simplest UDF can just extend EvalFunc, which requires only the exec function to be implemented (see <a href="#eval-functions-write"> How to Write a Simple Eval Function</a>). Every eval UDF must implement this. Additionally, if a function is algebraic, it can implement <code>Algebraic</code> interface to significantly improve query performance in the cases when combiner can be used (see <a href="#Algebraic-Interface">Algebraic Interface</a>). Finally, a function that can process tuples in an incremental fashion can also implement the Accumulator interface to improve query memory consumption (see <a href="#Accumulator-Interface">Accumulator Interface</a>).
+</p>
-<p>This problem is partially addressed by Algebraic UDFs that use the combiner and can deal with data being passed to them incrementally during different processing phases (map, combiner, and reduce.) However, there are a number of UDFs that are not Algebraic, don't use the combiner, but still donât need to be given all data at once. </p>
+<p>The optimizer selects the exact method by which a UDF is invoked based on the UDF type and the query. Note that only a single interface is used at any given time. The optimizer tries to find the most efficient way to execute the function. If a combiner is used and the function implements the Algebraic interface then this interface will be used to invoke the function. If the combiner is not invoked but the accumulator can be used and the function implements Accumulator interface then that interface is used. If neither of the conditions is satisfied then the exec function is used to invoke the UDF.
+</p>
+ </section>
+
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="function-instantiation">
+<title>Function Instantiation</title>
+<p>One problem that users run into is when they make assumption about how many times a constructor for their UDF is called. For instance, they might be creating side files in the store function and doing it in the constructor seems like a good idea. The problem with this approach is that in most cases Pig instantiates functions on the client side to, for instance, examine the schema of the data. </p>
+<p>Users should not make assumptions about how many times a function is instantiated; instead, they should make their code resilient to multiple instantiations. For instance, they could check if the files exist before creating them. </p>
+</section>
-<p>The new Accumulator interface is designed to decrease memory usage by targeting such UDFs. For the functions that implement this interface, Pig guarantees that the data for the same key is passed continuously but in small increments. To work with incremental data, here is the interface a UDF needs to implement:</p>
+
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="udf-configurations">
+<title>Passing Configurations to UDFs</title>
+<p>The singleton UDFContext class provides two features to UDF writers. First, on the backend, it allows UDFs to get access to the JobConf object, by calling getJobConf. This is only available on the backend (at run time) as the JobConf has not yet been constructed on the front end (during planning time).</p>
+
+<p>Second, it allows UDFs to pass configuration information between instantiations of the UDF on the front and backends. UDFs can store information in a configuration object when they are constructed on the front end, or during other front end calls such as describeSchema. They can then read that information on the backend when exec (for EvalFunc) or getNext (for LoadFunc) is called. Note that information will not be passed between instantiations of the function on the backend. The communication channel only works from front end to back end.</p>
+
+<p>To store information, the UDF calls getUDFProperties. This returns a Properties object which the UDF can record the information in or read the information from. To avoid name space conflicts UDFs are required to provide a signature when obtaining a Properties object. This can be done in two ways. The UDF can provide its Class object (via this.getClass()). In this case, every instantiation of the UDF will be given the same Properties object. The UDF can also provide its Class plus an array of Strings. The UDF can pass its constructor arguments, or some other identifying strings. This allows each instantiation of the UDF to have a different properties object thus avoiding name space collisions between instantiations of the UDF.</p>
+</section>
+
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="udf-monitoring">
+<title>Monitoring Long-Running UDFs</title>
+<p>Sometimes one may discover that a UDF that executes very quickly in the vast majority of cases turns out to run exceedingly slowly on occasion. This can happen, for example, if a UDF uses complex regular expressions to parse free-form strings, or if a UDF uses some external service to communicate with. As of version 0.8, Pig provides a facility for monitoring the length of time a UDF is executing for every invocation, and terminating its execution if it runs too long. This facility can be turned on using a simple Java annotation:</p>
+
<source>
-public interface Accumulator <T> {
- /**
- * Process tuples. Each DataBag may contain 0 to many tuples for current key
- */
- public void accumulate(Tuple b) throws IOException;
- /**
- * Called when all tuples from current key have been passed to the accumulator.
- * @return the value for the UDF for this key.
- */
- public T getValue();
- /**
- * Called after getValue() to prepare processing for next key.
- */
- public void cleanup();
-}
+ import org.apache.pig.builtin.MonitoredUDF;
+
+ @MonitoredUDF
+ public class MyUDF extends EvalFunc<Integer> {
+ /* implementation goes here */
+ }
</source>
-<p>There are several things to note here:</p>
+<p>Simply annotating your UDF in this way will cause Pig to terminate the UDF's exec() method if it runs for more than 10 seconds, and return the default value of null. The duration of the timeout and the default value can be specified in the annotation, if desired:</p>
-<ol>
- <li>Each UDF must extend the EvalFunc class and implement all necessary functions there.</li>
- <li>If a function is algebraic but can be used in a FOREACH statement with accumulator functions, it needs to implement the Accumulator interface in addition to the Algebraic interface.</li>
- <li>The interface is parameterized with the return type of the function.</li>
- <li>The accumulate function is guaranteed to be called one or more times, passing one or more tuples in a bag, to the UDF. (Note that the tuple that is passed to the accumulator has the same content as the one passed to exec â all the parameters passed to the UDF â one of which should be a bag).</li>
- <li>The getValue function is called after all the tuples for a particular key have been processed to retrieve the final value.</li>
- <li>The cleanup function is called after getValue but before the next value is processed.</li>
-</ol>
+<source>
+ import org.apache.pig.builtin.MonitoredUDF;
+
+ @MonitoredUDF(timeUnit = TimeUnit.MILLISECONDS, duration = 100, intDefault = 10)
+ public class MyUDF extends EvalFunc<Integer> {
+ /* implementation goes here */
+ }
+</source>
+<p>intDefault, longDefault, doubleDefault, floatDefault, and stringDefault can be specified in the annotation; the correct default will be chosen based on the return type of the UDF. Custom defaults for tuples and bags are not supported at this time.</p>
-<p>Here us a code snippet of the integer version of the MAX function that implements the interface:</p>
-<source>
-public class IntMax extends EvalFunc<Integer> implements Algebraic, Accumulator<Integer> {
- â¦â¦.
- /* Accumulator interface */
-
- private Integer intermediateMax = null;
-
- @Override
- public void accumulate(Tuple b) throws IOException {
- try {
- Integer curMax = max(b);
- if (curMax == null) {
- return;
- }
- /* if bag is not null, initialize intermediateMax to negative infinity */
- if (intermediateMax == null) {
- intermediateMax = Integer.MIN_VALUE;
- }
- intermediateMax = java.lang.Math.max(intermediateMax, curMax);
- } catch (ExecException ee) {
- throw ee;
- } catch (Exception e) {
- int errCode = 2106;
- String msg = "Error while computing max in " + this.getClass().getSimpleName();
- throw new ExecException(msg, errCode, PigException.BUG, e);
- }
- }
+<p>If desired, custom logic can also be implemented for error handling by creating a subclass of MonitoredUDFExecutor.ErrorCallback, and overriding its handleError and/or handleTimeout methods. Both of those methods are static, and are passed in the instance of the EvalFunc that produced an exception, as well as an exception, so you may use any state you have in the UDF to process the errors as desired. The default behavior is to increment Hadoop counters every time an error is encountered. Once you have an implementation of the ErrorCallback that performs your custom logic, you can provide it in the annotation:</p>
- @Override
- public void cleanup() {
- intermediateMax = null;
- }
+<source>
+ import org.apache.pig.builtin.MonitoredUDF;
- @Override
- public Integer getValue() {
- return intermediateMax;
- }
-}
+ @MonitoredUDF(errorCallback=MySpecialErrorCallback.class)
+ public class MyUDF extends EvalFunc<Integer> {
+ /* implementation goes here */
+ }
</source>
-</section>
+<p>Currently the MonitoredUDF annotation works with regular and Algebraic UDFs, but has no effect on UDFs that run in the Accumulator mode.</p>
+
+</section>
+</section>
+</section>
- <!-- =============================================================== -->
- <section>
- <title>JavaScript UDFs</title>
- <section>
- <title>Syntax</title>
- <section>
- <title>Registering Scripts</title>
- <p>You can register JavaScript as shown here. This example uses org.apache.pig.scripting.js.JsScriptEngine to interpret JavaScript. You can develop and use custom script engines to support multiple programming languages and ways to interpret them. Currently, Pig identifies js as a keyword and ships the required scriptengine (js) to interpret it. </p>
- <source>
- register 'test.js' using javascript as myfuncs;
- </source>
-
- <p>The following syntax is also supported, where myfuncs is the namespace created for all the functions inside test.js.</p>
- <source>
- register 'test.js' using org.apache.pig.scripting.js.JsScriptEngine as myfuncs;
- </source>
-
- <p>A typical test.js looks like this:</p>
- <source>
-
- helloworld.outputSchema = "word:chararray";
- function helloworld() {
- return 'Hello, World';
- }
-
- complex.outputSchema = "word:chararray,num:long";
- function complex(word) {
- return {word:word, num:word.length};
- }
-
- </source>
-
- <p>The register statement above registers the js functions defined in test.js in Pigâs runtime within the defined namespace (myfuncs here). They can then be referred later on in the pig script as myfuncs.helloworld(), myfuncs.complex(), and myfuncs.square(). An example usage is:</p>
-
- <source>
- b = foreach a generate myfuncs.helloworld(), myfuncs.complex($0);
- </source>
-
- </section>
- <section>
- <title>outputSchema attribute and Schemas</title>
- <p>As JavaScript functions are first class objects, you can annotate them by adding attributes. Add an outputSchema attribute to your function so that Pig can identify return types for the script UDF. </p>
- <ul>
- <li>outputSchema - Defines schema for a script udf in a format that Pig understands and is able to parse. </li>
- </ul>
-
- <p>Sample Schema String - y:{t:(word:chararray,num:long)}, variable names inside a schema string are used for type conversion between Pig and JavaScript. Tuples are converted to Objects using the names and vice versa.</p>
- </section>
- </section>
-
- <section>
- <title>Inline Scripts</title>
- <p>Currently, Pig does not support UDFs using inline scripts. </p>
- </section>
- </section>
<!-- =============================================================== -->
-<section>
-<title>Python UDFs</title>
-<section>
-<title>Syntax</title>
-<section>
-<title>Registering Scripts</title>
-<p>You can register a Python script as shown here. This example uses org.apache.pig.scripting.jython.JythonScriptEngine to interpret the python script. You can develop and use custom script engines to support multiple programming languages and ways to interpret them. Currently, Pig identifies jython as a keyword and ships the required scriptengine (jython) to interpret it. </p>
+<section id="python-udfs">
+<title>Writing Python UDFs</title>
+
+<!-- ++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="register-python">
+<title>Registering the UDF</title>
+<p>You can register a Python script as shown here. This example uses org.apache.pig.scripting.jython.JythonScriptEngine to interpret the Python script. You can develop and use custom script engines to support multiple programming languages and ways to interpret them. Currently, Pig identifies jython as a keyword and ships the required scriptengine (jython) to interpret it. </p>
<source>
Register 'test.py' using jython as myfuncs;
</source>
@@ -1325,6 +1383,7 @@ Register 'test.py' using jython as myfun
register 'test.py' using org.apache.pig.scripting.jython.JythonScriptEngine as myfuncs;
</source>
+
<p>A typical test.py looks like this:</p>
<source>
@outputSchema("word:chararray")
@@ -1348,37 +1407,34 @@ def concat(str):
return str+str
</source>
-<p>The register statement above registers the python functions defined in test.py in Pigâs runtime within the defined namespace (myfuncs here). They can then be referred later on in the pig script as myfuncs.helloworld(), myfuncs.complex(), and myfuncs.square(). An example usage is:</p>
+<p>The register statement above registers the Python functions defined in test.py in Pigâs runtime within the defined namespace (myfuncs here). They can then be referred later on in the pig script as myfuncs.helloworld(), myfuncs.complex(), and myfuncs.square(). An example usage is:</p>
<source>
b = foreach a generate myfuncs.helloworld(), myfuncs.square(3);
</source>
+
+<!-- ++++++++++++++++++++++++++++++++++++++++++++++++ -->
</section>
-<section>
+<section id="decorators">
<title>Decorators and Schemas</title>
-<p>To annotate a python script so that Pig can identify return types, use python decorators to define output schema for the script UDF. </p>
+<p>To annotate a Python script so that Pig can identify return types, use Python decorators to define output schema for the script UDF. </p>
<ul>
-<li>outputSchema - Defines schema for a script udf in a format that Pig understands and is able to parse. </li>
-<li>outputFunctionSchema - Defines a script delegate function that defines schema for this function depending upon the input type. This is needed for functions that can accept generic types and perform generic operations on these types. A simple example is square which can accept multiple types. SchemaFunction for this type is a simple identity function (same schema as input). </li>
-<li>schemaFunction - Defines delegate function and is not registered to Pig. </li>
+<li id="outputschema">outputSchema - Defines schema for a script UDF in a format that Pig understands and is able to parse. </li>
+<li id="outputfunctionschema">outputFunctionSchema - Defines a script delegate function that defines schema for this function depending upon the input type. This is needed for functions that can accept generic types and perform generic operations on these types. A simple example is square which can accept multiple types. SchemaFunction for this type is a simple identity function (same schema as input). </li>
+<li id="schemafunction">schemaFunction - Defines delegate function and is not registered to Pig. </li>
</ul>
<p>When no decorator is specified, Pig assumes the output datatype as bytearray and converts the output generated by script function to bytearray. This is consistent with Pig's behavior in case of Java UDFs. </p>
<p>Sample Schema String - y:{t:(word:chararray,num:long)}, variable names inside a schema string are not used anywhere, they just make the syntax identifiable to the parser. </p>
</section>
-</section>
-
-<section>
-<title>Inline Scripts</title>
-<p>Currently, Pig does not support UDFs using inline scripts. </p>
-</section>
+<!-- ++++++++++++++++++++++++++++++++++++++++++++++++ -->
<section>
- <title>Sample Script UDFs</title>
- <p>Simple tasks like string manipulation, mathematical computations, and reorganizing data types can be easily done using python scripts without having to develop long and complex UDFs in Java. The overall overhead of using scripting language is much less and development cost is almost negligible. The following UDFs, developed in python, can be used with Pig. </p>
+ <title>Example Scripts</title>
+ <p>Simple tasks like string manipulation, mathematical computations, and reorganizing data types can be easily done using Python scripts without having to develop long and complex UDFs in Java. The overall overhead of using scripting language is much less and development cost is almost negligible. The following UDFs, developed in Python, can be used with Pig. </p>
<source>
mySampleLib.py
@@ -1428,93 +1484,133 @@ b = foreach a generate myfuncs.helloworl
return outBag
# Few comments-
- # pig mandates that a bag should be a bag of tuples, python UDFs should follow this pattern.
- # tuple in python are immutable, appending to a tuple is not possible.
+ # Pig mandates that a bag should be a bag of tuples, Python UDFs should follow this pattern.
+ # Tuples in Python are immutable, appending to a tuple is not possible.
</source>
</section>
+
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++++ -->
+ <section id="python-advanced">
+ <title>Advanced Topics</title>
+ <p>UDFs and Pig scripts are generally stored in separate files. For testing purposes you can combine the code in a single file - a "combined" script.
+ Note, however, if you then decide to embed this "combined" script in a host language, the language of the UDF must match the host language.</p>
+
+ <p>This example combines Python and Pig. This "combined" script can only be embedded in Python.</p>
+ <p>With Python you MUST use the <code>if __name__ == '__main__': </code> construct to separate UDFs and control flow. Otherwise the script will result in an error.</p>
+ <source>
+ #!/usr/bin/jython
+from org.apache.pig.scripting import *
-</section>
+@outputSchema("word:chararray")
+def helloworld():
+ return 'Hello, World'
+
+if __name__ == '__main__':
+ P = Pig.compile("""a = load '1.txt' as (a0, a1);
+ b = foreach a generate helloworld();
+ store b into 'myoutput'; """)
-<!-- =============================================================== -->
-<section>
-<title>Advanced Topics</title>
+result = P.bind().runSingle();
+ </source>
+ </section>
+</section>
-<section>
-<title>UDF Interfaces</title>
-<p>A UDF can be invoked multiple ways. The simplest UDF can just extend EvalFunc, which requires only the exec function to be implemented (see <a href="#How+to+Write+a+Simple+Eval+Function"> How to Write a Simple Eval Function</a>). Every eval UDF must implement this. Additionally, if a function is algebraic, it can implement <code>Algebraic</code> interface to significantly improve query performance in the cases when combiner can be used (see <a href="#Aggregate+Functions">Aggregate Functions</a>). Finally, a function that can process tuples in an incremental fashion can also implement the Accumulator interface to improve query memory consumption (see <a href="#Accumulator+Interface">Accumulator Interface</a>).
-</p>
-<p>The exact method by which UDF is invoked is selected by the optimizer based on the UDF type and the query. Note that only a single interface is used at any given time. The optimizer tries to find the most efficient way to execute the function. If a combiner is used and the function implements the Algebraic interface then this interface will be used to invoke the function. If the combiner is not invoked but the accumulator can be used and the function implements Accumulator interface then that interface is used. If neither of the conditions is satisfied then the exec function is used to invoke the UDF.
-</p>
- </section>
+ <!-- =============================================================== -->
+ <section id="js-udfs">
+ <title>Writing JavaScript UDFs</title>
+ <p><strong>Note:</strong> <em>JavaScript UDFs are an experimental feature.</em></p>
+
+<!-- ++++++++++++++++++++++++++++++++++++++++++++++++ -->
+ <section id="register-js">
+ <title>Registering the UDF</title>
+ <p>You can register JavaScript as shown here. This example uses org.apache.pig.scripting.js.JsScriptEngine to interpret JavaScript. You can develop and use custom script engines to support multiple programming languages and ways to interpret them. Currently, Pig identifies js as a keyword and ships the required scriptengine (js) to interpret it. </p>
+ <source>
+ register 'test.js' using javascript as myfuncs;
+ </source>
-<section>
-<title>Function Instantiation</title>
-<p>One problem that users run into is when they make assumption about how many times a constructor for their UDF is called. For instance, they might be creating side files in the store function and doing it in the constructor seems like a good idea. The problem with this approach is that in most cases Pig instantiates functions on the client side to, for instance, examine the schema of the data. </p>
-<p>Users should not make assumptions about how many times a function is instantiated; instead, they should make their code resilient to multiple instantiations. For instance, they could check if the files exist before creating them. </p>
-</section>
+ <p>The following syntax is also supported, where myfuncs is the namespace created for all the functions inside test.js.</p>
+ <source>
+ register 'test.js' using org.apache.pig.scripting.js.JsScriptEngine as myfuncs;
+ </source>
-<section>
-<title>Schemas</title>
-<p>One request from users is to have the ability to examine the input schema of the data before processing the data. For example, they would like to know how to convert an input tuple to a map such that the keys in the map are the names of the input columns. The current answer is that there is no way to do this. This is something we would like to support in the future. </p>
+ <p>The register statement above registers the js functions defined in test.js in Pigâs runtime within the defined namespace (myfuncs here). They can then be referred later on in the pig script as myfuncs.helloworld(), myfuncs.complex(), and myfuncs.square(). An example usage is:</p>
+
+ <source>
+ b = foreach a generate myfuncs.helloworld(), myfuncs.complex($0);
+ </source>
</section>
-<section>
-<title>Passing Configurations to UDFs</title>
-<p>The singleton UDFContext class provides two features to UDF writers. First, on the backend, it allows UDFs to get access to the JobConf object, by calling getJobConf. This is only available on the backend (at run time) as the JobConf has not yet been constructed on the front end (during planning time).</p>
-
-<p>Second, it allows UDFs to pass configuration information between instantiations of the UDF on the front and backends. UDFs can store information in a configuration object when they are constructed on the front end, or during other front end calls such as describeSchema. They can then read that information on the backend when exec (for EvalFunc) or getNext (for LoadFunc) is called. Note that information will not be passed between instantiations of the function on the backend. The communication channel only works from front end to back end.</p>
-
-<p>To store information, the UDF calls getUDFProperties. This returns a Properties object which the UDF can record the information in or read the information from. To avoid name space conflicts UDFs are required to provide a signature when obtaining a Properties object. This can be done in two ways. The UDF can provide its Class object (via this.getClass()). In this case, every instantiation of the UDF will be given the same Properties object. The UDF can also provide its Class plus an array of Strings. The UDF can pass its constructor arguments, or some other identifying strings. This allows each instantiation of the UDF to have a different properties object thus avoiding name space collisions between instantiations of the UDF.</p>
-</section>
-
-<section>
-<title>Monitoring long-running UDFs</title>
-<p>Sometimes one may discover that a UDF that executes very quickly in the vast majority of cases turns out to run exceedingly slowly on occasion. This can happen, for example, if a UDF uses complex regular expressions to parse free-form strings, or if a UDF uses some external service to communicate with. As of version 0.8, Pig provides a facility for monitoring the length of time a UDF is executing for every invocation, and terminating its execution if it runs too long. This facility can be turned on using a simple Java annotation:</p>
-
-<source>
- import org.apache.pig.builtin.MonitoredUDF;
-
- @MonitoredUDF
- public class MyUDF extends EvalFunc<Integer> {
- /* implementation goes here */
- }
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++++ -->
+ <section id="schema-return-types">
+ <title>Return Types and Schemas</title>
+ <p>Since JavaScript functions are first class objects, you can annotate them by adding attributes. Add an outputSchema attribute to your function so that Pig can identify return types for the script UDF. </p>
+ <ul>
+ <li>outputSchema - Defines schema for a script udf in a format that Pig understands and is able to parse. </li>
+ <li>Sample Schema String - y:{t:(word:chararray,num:long)} <br></br>Variable names inside a schema string are used for type conversion between Pig and JavaScript. Tuples are converted to Objects using the names and vice versa</li>
+ </ul>
+ </section>
+
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++++ -->
+ <section id="js-example">
+ <title>Example Scripts</title>
+ <p>A simple JavaScript UDF (udf.js) is shown here.</p>
+ <source>
+ helloworld.outputSchema = "word:chararray";
+function helloworld() {
+ return 'Hello, World';
+}
+
+complex.outputSchema = "word:chararray,num:long";
+function complex(word){
+ return {word:word, num:word.length};
+}
</source>
-
-<p>Simply annotating your UDF in this way will cause Pig to terminate the UDF's exec() method if it runs for more than 10 seconds, and return the default value of null. The duration of the timeout and the default value can be specified in the annotation, if desired:</p>
-
+
+<p>This Pig script registers the JavaScript UDF (udf.js).</p>
<source>
- import org.apache.pig.builtin.MonitoredUDF;
-
- @MonitoredUDF(timeUnit = TimeUnit.MILLISECONDS, duration = 100, intDefault = 10)
- public class MyUDF extends EvalFunc<Integer> {
- /* implementation goes here */
- }
-</source>
-
-<p>intDefault, longDefault, doubleDefault, floatDefault, and stringDefault can be specified in the annotation; the correct default will be chosen based on the return type of the UDF. Custom defaults for tuples and bags are not supported at this time.</p>
+ register âudf.jsâ using javascript as myfuncs;
+A = load âdataâ as (a0:chararray, a1:int);
+B = foreach A generate myfuncs.helloworld(), myfuncs.complex(a0);
+... ...
+</source>
+ </section>
+
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++++ -->
+ <section id="js-advanced">
+ <title>Advanced Topics</title>
-<p>If desired, custom logic can also be implemented for error handling by creating a subclass of MonitoredUDFExecutor.ErrorCallback, and overriding its handleError and/or handleTimeout methods. Both of those methods are static, and are passed in the instance of the EvalFunc that produced an exception, as well as an exception, so you may use any state you have in the UDF to process the errors as desired. The default behavior is to increment Hadoop counters every time an error is encountered. Once you have an implementation of the ErrorCallback that performs your custom logic, you can provide it in the annotation:</p>
-
-<source>
- import org.apache.pig.builtin.MonitoredUDF;
+ <p>UDFs and Pig scripts are generally stored in separate files. For testing purposes you can combine the code in a single file - a "combined" script.
+ Note, however, if you then decide to embed this "combined" script in a host language, the language of the UDF and the host language must match.</p>
+
+ <p>This example combines JavaScript and Pig. This "combined" script can only be embedded in JavaScript.</p>
+<p>With JavaScript, the control flow MUST be defined in the main function. Otherwise the script will result in an error.</p>
+ <source>
+importPackage(Packages.org.apache.pig.scripting.js)
+pig = org.apache.pig.scripting.js.JSPig;
- @MonitoredUDF(errorCallback=MySpecialErrorCallback.class)
- public class MyUDF extends EvalFunc<Integer> {
- /* implementation goes here */
- }
-</source>
+helloworld.outputSchema = "word:chararray"
+function helloworld() {
+ return 'Hello, World';
+}
-<p>Currently the MonitoredUDF annotation works with regular and Algebraic UDFs, but has no effect on UDFs that run in the Accumulator mode.</p>
+function main() {
+ var P = pig.compile(" a = load '1.txt' as (a0, a1);â+
+ âb = foreach a generate helloworld();â+
+ âstore b into 'myoutput';");
-</section>
-</section>
-</section>
+ var result = P.bind().runSingle();
+}
+ </source>
+
+ </section>
+
+</section>
<!-- ================================================================== -->
<!-- PIGGYBANK -->
-<section>
+<section id="piggybank">
<title>Piggy Bank</title>
<p>Piggy Bank is a place for Pig users to share the Java UDFs they have written for use with Pig.
The functions are contributed "as-is."
@@ -1522,9 +1618,10 @@ If you find a bug in a function, take th
If you don't find the UDF you need, take the time to write and contribute the function to Piggy Bank.
</p>
-<p><strong>Note:</strong> Piggy Bank currently supports Java UDFs. Support for Python UDFs will be added at a later date.</p>
+<p><strong>Note:</strong> Piggy Bank currently supports Java UDFs. Support for Python and JavaScript UDFs will be added at a later date.</p>
-<section>
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="piggbank-access">
<title>Accessing Functions</title>
<p>The Piggy Bank functions are currently distributed in source form. Users are required to checkout the code and build the package themselves. No binary distributions or nightly builds are available at this time. </p>
@@ -1562,14 +1659,15 @@ STORE TweetsInaug INTO 'meta/inaug/tweet
</source>
</section>
-<section>
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->
+<section id="piggybank-contribute">
<title>Contributing Functions</title>
<p>To contribute a Java function that you have written, do the following:</p>
<ol>
-<li>Check the existing javadoc to make sure that the function does not already exist as described in <a href="#Accessing+Functions">Accessing Functions</a>. </li>
-<li>Checkout the UDF code as described in <a href="#Accessing+Functions">Accessing Functions</a>. </li>
-<li>Place your java code in the directory that makes sense for your function. The directory structure currently has two levels: (1) function type, as described in <a href="#Accessing+Functions">Accessing Functions</a>, and (2) function subtype, for some of the types (like math or string for eval functions). If you think your function requires a new subtype, feel free to add one. </li>
+<li>Check the existing javadoc to make sure that the function does not already exist as described in <a href="#piggbank-access">Accessing Functions</a>. </li>
+<li>Checkout the UDF code as described in <a href="#piggbank-access">Accessing Functions</a>. </li>
+<li>Place your java code in the directory that makes sense for your function. The directory structure currently has two levels: (1) function type, as described in <a href="#piggbank-access">Accessing Functions</a>, and (2) function subtype, for some of the types (like math or string for eval functions). If you think your function requires a new subtype, feel free to add one. </li>
<li>Make sure that your function is well documented and uses the
<a href="http://download.oracle.com/javase/1.4.2/docs/tooldocs/solaris/javadoc.html">javadoc</a> style of documentation. </li>
<li>Make sure that your code follows Pig coding conventions described in <a href="http://wiki.apache.org/pig/HowToContribute">How to Contribute to Pig</a>.</li>
@@ -1577,9 +1675,9 @@ STORE TweetsInaug INTO 'meta/inaug/tweet
<li>Submit your patch following the process described in <a href="http://wiki.apache.org/pig/HowToContribute">How to Contribute to Pig</a>. </li>
</ol>
</section>
-
</section>
+
</body>
</document>