You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by da...@apache.org on 2011/03/26 01:57:34 UTC
svn commit: r1085620 - in /pig/trunk: CHANGES.txt
src/docs/src/documentation/content/xdocs/basic.xml
src/docs/src/documentation/content/xdocs/cont.xml
src/docs/src/documentation/content/xdocs/start.xml
src/docs/src/documentation/content/xdocs/udf.xml
Author: daijy
Date: Sat Mar 26 00:57:34 2011
New Revision: 1085620
URL: http://svn.apache.org/viewvc?rev=1085620&view=rev
Log:
PIG-1772: Pig 090 Documentation
Modified:
pig/trunk/CHANGES.txt
pig/trunk/src/docs/src/documentation/content/xdocs/basic.xml
pig/trunk/src/docs/src/documentation/content/xdocs/cont.xml
pig/trunk/src/docs/src/documentation/content/xdocs/start.xml
pig/trunk/src/docs/src/documentation/content/xdocs/udf.xml
Modified: pig/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/pig/trunk/CHANGES.txt?rev=1085620&r1=1085619&r2=1085620&view=diff
==============================================================================
--- pig/trunk/CHANGES.txt (original)
+++ pig/trunk/CHANGES.txt Sat Mar 26 00:57:34 2011
@@ -28,6 +28,8 @@ PIG-1680: HBaseStorage should work with
IMPROVEMENTS
+PIG-1772: Pig 090 Documentation (chandec via daijy)
+
PIG-1830: Type mismatch error in key from map, when doing GROUP on PigStorageSchema() variable (dvryaboy)
PIG-1566: Support globbing for registering jars in pig script (nrai via daijy)
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/basic.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/basic.xml?rev=1085620&r1=1085619&r2=1085620&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/basic.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/basic.xml Sat Mar 26 00:57:34 2011
@@ -302,9 +302,9 @@ A123
abc_123_BeX_
</source>
<p></p>
-<p>Invalid Identifies: </p>
+<p>Invalid identifiers: </p>
<source>
-_abc
+_A123
abc_$
A!B
</source>
@@ -836,7 +836,7 @@ DUMP X;
</section>
</section>
- <section>
+ <section id="map">
<title>Map</title>
<p>A map is a set of key value pairs.</p>
@@ -883,7 +883,7 @@ DUMP X;
<p>value</p>
</td>
<td>
- <p>Any data type.</p>
+ <p>Any data type (the defaults to bytearray).</p>
</td>
</tr>
</table>
@@ -1478,9 +1478,20 @@ C = FOREACH G GENERATE COUNT(*)
<!-- ++++++++++++++++++++++++++++++++++ -->
<section id="Schemas">
<title>Schemas</title>
+
+ <p>Schemas enable you to assign names to fields and declare types for fields. Schemas are optional but we encourage you to use them whenever possible; type declarations result in better parse-time error checking and more efficient code execution.</p>
+
+ <p>Schemas for <a href="#schema-simple">simple types</a> and <a href="#schema-complex">complex types</a> can be used anywhere a schema definition is appropriate.</p>
+
+ <p>Schemas are defined with the <a href="#LOAD">LOAD</a>, <a href="#STREAM">STREAM</a>, and <a href="#FOREACH">FOREACH</a> operators using the AS clause. If you define a schema using the LOAD operator, then it is the load function that enforces the schema
+ (see <a href="#LOAD">LOAD</a> and <a href="udf.html">User Defined Functions</a> for more information).</p>
+
+ <p></p>
+ <p><strong>Known Schema Handling</strong></p>
<p>Schemas enable you to assign names to fields and declare types for fields. Schemas are optional but we encourage you to use them whenever possible; type declarations result in better parse-time error checking and more efficient code execution. </p>
<p>Schemas are defined for the <a href="#LOAD">LOAD</a>, <a href="#STREAM">STREAM</a>, and <a href="#FOREACH">FOREACH</a> operators using the AS clause. If you define a schema using the LOAD operator, then it is the load function that enforces the schema
(see <a href="#LOAD">LOAD</a> and <a href="udf.html">User Defined Functions</a> for more information).</p>
+
<p></p>
<p><strong>Known Schema Handling</strong></p>
<p>Note the following:</p>
@@ -1527,7 +1538,7 @@ X = FOREACH A GENERATE f1 as user, f2 as
</source>
</section>
- <section>
+ <section id="schema-simple">
<title>Schemas for Simple Data Types</title>
<p>Simple data types include int, long, float, double, chararray, and bytearray.</p>
@@ -1616,7 +1627,7 @@ DUMP A;
</section></section>
- <section>
+ <section id="schema-complex">
<title>Schemas for Complex Data Types</title>
<p>Complex data types include tuples, bags, and maps.</p></section>
@@ -1802,11 +1813,11 @@ DUMP A;
<p>A map is a set of key value pairs.</p>
<section>
- <title>Syntax (where <> means optional)</title>
+ <title>Syntax (<> demotes optional)</title>
<table>
<tr>
<td>
- <p>alias<:map> [ ] </p>
+ <p>alias<:map> [ <type> ] </p>
</td>
</tr>
</table></section>
@@ -1814,36 +1825,31 @@ DUMP A;
<section>
<title>Terms</title>
<table>
- <tr>
- <td>
- <p>alias</p>
- </td>
- <td>
- <p>The name assigned to the map.</p>
- </td>
+ <tr>
+ <td><p>alias</p></td>
+ <td><p>The name assigned to the map.</p></td>
</tr>
<tr>
- <td>
- <p>:map</p>
- </td>
- <td>
- <p>(Optional) The data type, map (case insensitive).</p>
- </td>
+ <td><p>:map</p></td>
+ <td><p>(Optional) The data type, map (case insensitive).</p></td>
</tr>
<tr>
- <td>
- <p>[ ] Â </p>
- </td>
- <td>
- <p>The designation for a map, a set of straight brackets [ ].</p>
+ <td><p>[ ]</p></td>
+ <td><p>The designation for a map, a set of straight brackets [ ].</p></td>
+ </tr>
+ <tr>
+ <td><p>type</p></td>
+ <td><p>(Optional) The datatype (all types allowed, bytearray is the default).</p>
+ <p>The type applies to the map value only; the map key is always type chararray (see <a href="#Map">Map</a>).</p>
+ <p>If a type is declared then ALL values in the map must be of this type.</p>
</td>
</tr>
</table>
</section>
<section>
- <title>Example</title>
- <p>In this example the schema defines a map. The load statements are equivalent.</p>
+ <title>Examples</title>
+ <p>In this example the schema defines an untyped map (the map values default to bytearray). The load statements are equivalent.</p>
<source>
cat data;
[open#apache]
@@ -1859,6 +1865,22 @@ DUMP A;
([open#apache])
([apache#hadoop])
</source>
+
+<p>This example shows the use of a typed maps.</p>
+<source>
+/* Map types are declared*/
+a = load '1.txt' as(map[int]); --Map value is int
+b = foreach a generate (map[(i:int)])a0; -- Map value is tuple
+b = stream a through `cat` as (m:map[{(i:int,j:chararray)}]); -- Map value is bag
+
+/* The MapLookup of a typed map will result in a datatype of the map value */
+a = load '1.txt' as(map[int]);
+b = foreach a generate $0#'key';
+
+/* Schema for b */
+b: {int}
+
+</source>
</section></section>
<section>
@@ -7645,7 +7667,7 @@ B = FOREACH A GENERATE myFunc($0);
<table>
<tr>
<td>
- <p>REGISTER alias;</p>
+ <p>REGISTER path;</p>
</td>
</tr>
</table></section>
@@ -7655,13 +7677,11 @@ B = FOREACH A GENERATE myFunc($0);
<table>
<tr>
<td>
- <p>alias</p>
+ <p>path</p>
</td>
<td>
<p>The path to the JAR file (the full location URI is required). Do not place the name in quotes.</p>
- <p>Pig supports functions written in Java only.</p>
- <p>Pig supports JAR files stored in HDFS, Amazon S3, and other distributed files systems
- (see <a href="start.html#Pig+Scripts">Pig Scripts</a>).</p>
+
</td>
</tr>
</table>
@@ -7669,32 +7689,44 @@ B = FOREACH A GENERATE myFunc($0);
<section>
<title>Usage</title>
- <p>Use the REGISTER statement inside a Pig script to specify the path of the JAR file containing UDFs. </p>
- <p>You can register additional files (to use with your Pig script) via the command line using the -Dpig.additional.jars option.</p>
- <p>For more information see <a href="udf.html">User Defined Functions</a>.</p>
+ <p><strong>Pig Scripts</strong></p>
+
+ <p>Use the REGISTER statement inside a Pig script to specify a JAR file or a Python/JavaScript module. Pig supports JAR files and modules stored in local file systems as well as remote, distributed file systems such as HDFS and Amazon S3 (see <a href="start.html#Pig+Scripts">Pig Scripts</a>).</p>
+
+ <p>Additionally, JAR files stored in local file systems can be specified as a glob pattern using â*â. Pig will search for matching jars in the local file system, either the relative path (relative to your working directory) or the absolute path. Pig will pick up all JARs that match the glob.</p>
+
+ <p><strong>Command Line</strong></p>
+ <p>You can register additional files (to use with your Pig script) via the command line using the -Dpig.additional.jars option.
+For more information see <a href="udf.html">User Defined Functions</a>.</p>
</section>
<section>
<title>Examples</title>
-<p>In this example REGISTER states that myfunc.jar is located in the /src directory.</p>
+<p>In this example REGISTER states that the JavaScript module, myfunc.js, is located in the /src directory.</p>
<source>
/src $ java -jar pig.jar â
-REGISTER /src/myfunc.jar;
+REGISTER /src/myfunc.js;
A = LOAD 'students';
B = FOREACH A GENERATE myfunc.MyEvalFunc($0);
</source>
-<p>In this example additional jar files are registered via the command line.</p>
+<p>In this example additional JAR files are registered via the command line.</p>
<source>
pig -Dpig.additional.jars=my.jar:your.jar script.pig
</source>
-<p>In this example a jar file stored in HDFS is registered</p>
+<p>In this example a JAR file stored in HDFS is registered.</p>
<source>
java -cp pig.jar org.apache.pig.Main hdfs://nn.mydomain.com:9020/myscripts/script.pig
</source>
-<p></p>
+
+<p>This example shows how to specify a glob pattern using either a relative path or an absolute path.</p>
+<source>
+register /homes/user/pig/myfunc*.jar
+register count*.jar
+register jars/*.jar
+</source>
</section>
</section>
</section>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/cont.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/cont.xml?rev=1085620&r1=1085619&r2=1085620&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/cont.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/cont.xml Sat Mar 26 00:57:34 2011
@@ -32,27 +32,77 @@
<section>
<title>Embedded Pig and Control Flow </title>
- <p> To enable control flow, you can embed Pig Latin scripts in a host scripting language via a JDBC-like compile, bind, run model. The default scripting language is Jython which is shipped with Pig and available by default. </p>
+<p>To enable control flow, you can embed Pig Latin scripts in a host scripting language via a JDBC-like compile, bind, run model. Supported host languages include Python (via Jython) and JavaScript (via Rhino). You must make sure that the Jython jar and/or the Rhino jar are included in your class path if you want to used embedded Pig. At runtime Pig will automatically detect the usage of a scripting UDF in the Pig script and will ship the corresponding scripting jar to the backend.</p>
<section>
<title>Invocation Basics</title>
-<p>Embedded Pig Latin is supported in batch mode only, not interactive mode. When Pig is invoked with a script it will look for a #! line at the beginning of the file. If the basename portion of the path matches jython or python it will use the reference embedding implementation. </p>
+<p>Embedded Pig Latin is supported in batch mode only, not interactive mode. You can request that embedded Pig Latin be used by adding the <code>--embedded</code> option to the Pig command line. If this option is passed as an argument, that argument will refer to the language Pig Latin is embedded in, either Python or JavaScript. If no argument is specified, it is taken to refer to the reference implementation for Python.</p>
-<source>
-#! /usr/bin/jython
-</source>
-<p>You can also explicitly request that embedded Pig Latin be used by adding the <code>--embedded</code> option to the Pig command line. If this option is passed as an argument, that argument will refer to the language the Pig Latin is embedded in. If no argument is specified, it is taken to refer to the reference implementation.</p>
+ <p><strong>Python</strong></p>
+ <source>
+ $ pig myembedded.py
+OR
+$ java -cp <jython jars>:<pig jars>; [--embedded python] /tmp/myembedded.py
+ </source>
+ <p></p>
+ <p>Pig will look for the <code>#!/usr/bin/python</code> line in the script.</p>
+
+ <source>
+#!/usr/bin/python
+
+# explicitly import Pig class
+from org.apache.pig.scripting import Pig
+# COMPILE: compile method returns a Pig object that represents the pipeline
+P = Pig.compile("a = load '$in'; store a into '$out';")
+
+input = 'original'
+output = 'output'
+
+# BIND and RUN
+result = P.bind({'in':input, 'out':output}).runSingle()
+
+if result.isSuccessful() :
+ print 'Pig job succeeded'
+else :
+ raise 'Pig job failed'
+ </source>
+
+<p><strong>JavaScript</strong></p>
<source>
-$ pig myembedded.py
+$ pig myembedded.js
OR
-$ java -cp $lt;jython jars>:$lt;pig jars$gt; [--embedded jython] /tmp/myembedded.py
+$ java -cp <rhino jars>:<pig jars>; [--embedded javascript] /tmp/myembedded.js
</source>
+<p></p>
+<p>Pig will look for the *.js extension in the script.</p>
+<source>
+importPackage(Packages.org.apache.pig.scripting.js)
+
+Pig = org.apache.pig.scripting.js.JSPig
+
+function main() {
+ input = "original"
+ output = "output"
+
+ P = Pig.compile("A = load '$in'; store A into '$out';")
+
+ result = P.bind({'in':input, 'out':output}).runSingle()
+
+ if (result.isSuccessful()) {
+ print("Pig job succeeded")
+ } else {
+ print("Pig job failed")
+ }
+}
+</source>
+
+<p><strong>Invocation Process</strong></p>
<p>You invoke Pig Latin in the host scripting language through an embedded <a href="#pig-Object">Pig object</a>. </p>
-<p>The first step in this process is to compile your script. Compile is a static function on the Pig object and in its simplest form takes a fragment of Pig Latin that defines the pipeline as its input:</p>
+<p><strong>Compile:</strong> Compile is a static function on the Pig object and in its simplest form takes a fragment of Pig Latin that defines the pipeline as its input:</p>
<source>
# COMPILE: complie method returns a Pig object that represents the pipeline
@@ -62,7 +112,7 @@ P = Pig.compile("""A = load '$inâ;
<p>Compile returns an instance of Pig object. This object can have certain values undefined. For example, you may want to define a pipeline without yet specifying the location of the input to the pipeline. The parameter will be indicated by a dollar sign followed by a sequence of alpha-numeric or underscore characters. Values for these parameters must be provided later at the time bind() is called on the Pig object. To call run() on a Pig object without all parameters being bound is an error. </p>
<p></p>
-<p>The second step is to resolve the parameters during the <strong>bind </strong>call.</p>
+<p><strong>Bind:</strong> Resolve the parameters during the bind call.</p>
<source>
input = "originalâ
@@ -74,19 +124,18 @@ Q = P.bind({'in':input, 'out':output})
<p>Please note that all parameters must be resolved during bind. Having unbound parameters while running your script is an error. Also note that even if your script is fully defined during compile, bind without parameters still must be called.</p>
-<p>Bind call returns an instance of <a href="#BoundScript-Object">BoundScript object</a> that can be used to execute the pipeline. The simplest way to execute the pipeline is to call runSingle function. (However, as mentioned later, this works only if a single set of variables is bound to the parameters. Otherwise, if multiple set of variables are bound, an exception will be thrown if runSingle is called.)</p>
+<p><strong>Run:</strong> Bind call returns an instance of <a href="#BoundScript-Object">BoundScript object</a> that can be used to execute the pipeline. The simplest way to execute the pipeline is to call runSingle function. (However, as mentioned later, this works only if a single set of variables is bound to the parameters. Otherwise, if multiple set of variables are bound, an exception will be thrown if runSingle is called.)</p>
<source>
result = Q.runSingle()
</source>
-
<p>The function returns a <a href="#PigStats-Object">PigStats object</a> that tells you whether the run succeeded or failed. In case of success, additional run statistics are provided.</p>
-<p>Here is a complete example:</p>
-<p><strong>embedded.py</strong></p>
+<p><strong>Embedded Python Example</strong></p>
+<p>A complete embedded example is shown below.</p>
<source>
#! /usr/bin/python
@@ -164,14 +213,14 @@ P = Pig.compileFromFile(âP2â,
<section>
<title>Bind</title>
-<p>In its simplest form, bind takes no parameters. In this case, an implicit bind is performed.</p>
+<p>In its simplest form, bind takes no parameters. In this case, an implicit bind is performed; Pig internally constructs a map of parameters from the local variables specified by the user in the script.</p>
<source>
Q = P.bind ()
</source>
-<p>In the case where parameter substitution is needed, a map of parameters (parameter name -> parameter value) must be provided as was illustrated in the example above.</p>
+<p>In the case where parameter substitution is needed, a map of parameters (parameter name > parameter value) must be provided as was illustrated in the example above.</p>
<p>Finally, you might want to run the same pipeline in parallel with a different set of parameters, for instance for multiple dates. In this case, bind function, needs to be passed a list of maps with each element of the list containing parameters for a single invocation. In the example below, the pipeline is run for the US, the UK, and Brazil.</p>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/start.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/start.xml?rev=1085620&r1=1085619&r2=1085620&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/start.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/start.xml Sat Mar 26 00:57:34 2011
@@ -35,6 +35,9 @@
<ul>
<li> <strong>Hadoop 0.20.2</strong> - <a href="http://hadoop.apache.org/common/releases.html">http://hadoop.apache.org/common/releases.html</a></li>
<li> <strong>Java 1.6</strong> - <a href="http://java.sun.com/javase/downloads/index.jsp">http://java.sun.com/javase/downloads/index.jsp</a> (set JAVA_HOME to the root of your Java installation)</li>
+ <li> <strong>Python 2.5</strong> - <a href="http://jython.org/downloads.html">http://jython.org/downloads.html</a> (optional, for embedded Pig) </li>
+ <li> <strong>JavaScript 1.7</strong> - <a href="https://developer.mozilla.org/en/Rhino_downloads_archive">https://developer.mozilla.org/en/Rhino_downloads_archive</a> and <a href="http://mirrors.ibiblio.org/pub/mirrors/maven2/rhino/js/">http://mirrors.ibiblio.org/pub/mirrors/maven2/rhino/js/</a> (optional, for embedded Pig) </li>
+
<li> <strong>Ant 1.7</strong> - <a href="http://ant.apache.org/">http://ant.apache.org/</a> (optional, for builds) </li>
<li> <strong>JUnit 4.5</strong> - <a href="http://junit.sourceforge.net/">http://junit.sourceforge.net/</a> (optional, for unit tests) </li>
</ul>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/udf.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/udf.xml?rev=1085620&r1=1085619&r2=1085620&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/udf.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/udf.xml Sat Mar 26 00:57:34 2011
@@ -86,31 +86,30 @@ transformation functions and include agg
<p>Let's now look at the implementation of the <code>UPPER</code> UDF. </p>
<source>
-package myudfs;
-import java.io.IOException;
-import org.apache.pig.EvalFunc;
-import org.apache.pig.data.Tuple;
-import org.apache.pig.impl.util.WrappedIOException;
-
-public class UPPER extends EvalFunc<String>
-{
- public String exec(Tuple input) throws IOException {
- if (input == null || input.size() == 0)
- return null;
- try{
- String str = (String)input.get(0);
- return str.toUpperCase();
- }catch(Exception e){
- throw WrappedIOException.wrap("Caught exception processing input row ", e);
- }
- }
-}
-</source>
-
-<p>The first line indicates that the function is part of the <code>myudfs</code> package. The UDF class extends the <code>EvalFunc</code> class which is the base class for all eval functions. It is parameterized with the return type of the UDF which is a Java <code>String</code> in this case. We will look into the <code>EvalFunc</code> class in more detail later, but for now all we need to do is to implement the <code>exec</code> function. This function is invoked on every input tuple. The input into the function is a tuple with input parameters in the order they are passed to the function in the Pig script. In our example, it will contain a single string field corresponding to the student name. </p>
-<p>The first thing to decide is what to do with invalid data. This depends on the format of the data. If the data is of type <code>bytearray</code> it means that it has not yet been converted to its proper type. In this case, if the format of the data does not match the expected type, a NULL value should be returned. If, on the other hand, the input data is of another type, this means that the conversion has already happened and the data should be in the correct format. This is the case with our example and that's why it throws an error (line 16.) Note that <code>WrappedIOException</code> is a helper class to convert the actual exception to an IOException. </p>
-<p>Also, note that lines 10-11 check if the input data is null or empty and if so returns null. </p>
-<p>The actual function implementation is on lines 13-14 and is self-explanatory. </p>
+1 package myudfs;
+2 import java.io.IOException;
+3 import org.apache.pig.EvalFunc;
+4 import org.apache.pig.data.Tuple;
+5
+6 public class UPPER extends EvalFunc<String>
+7 {
+8 public String exec(Tuple input) throws IOException {
+9 if (input == null || input.size() == 0)
+10 return null;
+11 try{
+12 String str = (String)input.get(0);
+13 return str.toUpperCase();
+14 }catch(Exception e){
+15 throw new IOException("Caught exception processing input row ", e);
+16 }
+17 }
+18 }
+</source>
+
+<p>Line 1 indicates that the function is part of the <code>myudfs</code> package. The UDF class extends the <code>EvalFunc</code> class which is the base class for all eval functions. It is parameterized with the return type of the UDF which is a Java <code>String</code> in this case. We will look into the <code>EvalFunc</code> class in more detail later, but for now all we need to do is to implement the <code>exec</code> function. This function is invoked on every input tuple. The input into the function is a tuple with input parameters in the order they are passed to the function in the Pig script. In our example, it will contain a single string field corresponding to the student name. </p>
+<p>The first thing to decide is what to do with invalid data. This depends on the format of the data. If the data is of type <code>bytearray</code> it means that it has not yet been converted to its proper type. In this case, if the format of the data does not match the expected type, a NULL value should be returned. If, on the other hand, the input data is of another type, this means that the conversion has already happened and the data should be in the correct format. This is the case with our example and that's why it throws an error (line 15.) </p>
+<p>Also, note that lines 9-10 check if the input data is null or empty and if so returns null. </p>
+<p>The actual function implementation is on lines 12-13 and is self-explanatory. </p>
<p>Now that we have the function implemented, it needs to be compiled and included in a jar. You will need to build <code>pig.jar</code> to compile your UDF. You can use the following set of commands to checkout the code from SVN repository and create pig.jar: </p>
<source>
@@ -234,7 +233,6 @@ import org.apache.pig.backend.executione
import org.apache.pig.data.DataBag;
import org.apache.pig.data.Tuple;
import org.apache.pig.data.DataType;
-import org.apache.pig.impl.util.WrappedIOException;
public class IsEmpty extends FilterFunc {
public Boolean exec(Tuple input) throws IOException {
@@ -251,7 +249,7 @@ public class IsEmpty extends FilterFunc
DataType.findTypeName(values) + " for emptiness.");
}
} catch (ExecException ee) {
- throw WrappedIOException.wrap("Caught exception processing input row ", ee);
+ throw new IOException("Caught exception processing input row ", ee);
}
}
}
@@ -400,7 +398,7 @@ D = limit B 20;
dump D;
</source>
-<p>This script will result in the following error cause by line 4. </p>
+<p>This script will result in the following error cause by line 4 ( <code>C = foreach B generate $2;</code>). </p>
<source>
java.io.IOException: Out of bound access. Trying to access non-existent column: 2. Schema {bytearray,gpa: float} has 2 column(s).
@@ -445,7 +443,7 @@ public class Swap extends EvalFunc<Tu
}
</source>
-<p>The function creates a schema with a single field (of type <code>FieldSchema=) of type =tuple</code>. The name of the field is constructed using the <code>getSchemaName</code> function of the <code>EvalFunc</code> class. The name consists of the name of the UDF function, the first parameter passed to it, and a sequence number to guarantee uniqueness. In the previous script, if you replace <code>dump D;</code> with <code>describe B;</code> , you will see the following output:
+<p>The function creates a schema with a single field (of type <code>FieldSchema) of type tuple</code>. The name of the field is constructed using the <code>getSchemaName</code> function of the <code>EvalFunc</code> class. The name consists of the name of the UDF function, the first parameter passed to it, and a sequence number to guarantee uniqueness. In the previous script, if you replace <code>dump D;</code> with <code>describe B;</code> , you will see the following output:
</p>
<source>
@@ -521,7 +519,7 @@ public class TOKENIZE extends EvalFunc&l
}
</source>
-<p>As you can see, this is very similar to the output schema definition in the <code>Swap</code> function. One difference is that instead of reusing input schema, we create a brand new field schema to represent the tokens stored in the bag. The other difference is that the type of the schema created is <code>BAG</code> (not =TUPLE=). </p>
+<p>As you can see, this is very similar to the output schema definition in the <code>Swap</code> function. One difference is that instead of reusing input schema, we create a brand new field schema to represent the tokens stored in the bag. The other difference is that the type of the schema created is <code>BAG</code> (not TUPLE). </p>
</section>
@@ -537,7 +535,6 @@ public class TOKENIZE extends EvalFunc&l
<li><p>An error that affects the entire processing and is not likely to succeed on retry. An example of such a failure is the inability to open a lookup file because of file permission problems. Pig currently does not have a way to handle this case. Hadoop does not have a way to handle this case either. It will be handled the same way as 2 above. </p>
</li>
</ol>
-<p>Pig provides a helper class <code>WrappedIOException</code>. The intent here is to allow you to convert any exception into <code>IOException</code>. Its usage can be seen in the <code>UPPER</code> function in our first example. </p>
</section>
@@ -556,7 +553,6 @@ import org.apache.pig.EvalFunc;
import org.apache.pig.FuncSpec;
import org.apache.pig.data.Tuple;
import org.apache.pig.impl.logicalLayer.FrontendException;
-import org.apache.pig.impl.util.WrappedIOException;
import org.apache.pig.impl.logicalLayer.schema.Schema;
import org.apache.pig.data.DataType;
@@ -571,7 +567,7 @@ public class ABS extends EvalFunc<Dou
System.err.println("Failed to process input; error - " + nfe.getMessage());
return null;
} catch (Exception e){
- throw WrappedIOException.wrap("Caught exception processing input row ", e);
+ throw IOException("Caught exception processing input row ", e);
}
return Math.abs(d);
}
@@ -591,7 +587,6 @@ public class ABS extends EvalFunc<Dou
<source>
import java.io.IOException;
-import org.apache.pig.impl.util.WrappedIOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;
@@ -603,7 +598,7 @@ public class IntAbs extends EvalFunc<
try{
d = (Integer)input.get(0);
} catch (Exception e){
- throw WrappedIOException.wrap("Caught exception processing input row ", e);
+ throw newIOException("Caught exception processing input row ", e);
}
return Math.abs(d);
}
@@ -612,7 +607,7 @@ public class IntAbs extends EvalFunc<
<p>A note on error handling. The <code>ABS</code> class covers the case of the <code>bytearray</code> which means the data has not been converted yet to its actual type. This is why a null value is returned when <code>NumberFormatException</code> is encountered. However, the <code>IntAbs</code> function is only called if the data is already of type <code>Integer</code> which means it has already been converted to the real type and bad format has been dealt with. This is why an exception is thrown if the input can't be cast to <code>Integer</code>. </p>
-<p>The example above covers a reasonably simple case where the UDF only takes one parameter and there is a separate function for each parameter type. However, this will not always be the case. If Pig can't find an <code>exact match</code> it tries to do a <code>best match</code>. The rule for the best match is to find the most efficient function that can be used safely. This means that Pig must find the function that, for each input parameter, provides the smallest type that is equal to or greater than the input type. The type progression rules are: <code>int=->=long=->=float=->=double</code>. </p>
+<p>The example above covers a reasonably simple case where the UDF only takes one parameter and there is a separate function for each parameter type. However, this will not always be the case. If Pig can't find an <code>exact match</code> it tries to do a <code>best match</code>. The rule for the best match is to find the most efficient function that can be used safely. This means that Pig must find the function that, for each input parameter, provides the smallest type that is equal to or greater than the input type. The type progression rules are: <code>int>long>float>double</code>. </p>
<p>For instance, let's consider function <code>MAX</code> which is part of the <code>piggybank</code> described later in this document. Given two values, the function returns the larger value. The function table for <code>MAX</code> looks like this: </p>
<source>
@@ -707,7 +702,7 @@ public class UPPER extends EvalFunc<S
String str = (String)input.get(0);
return str.toUpperCase();
}catch(Exception e){
- throw WrappedIOException.wrap("Caught exception processing input row ", e);
+ throw new OException("Caught exception processing input row ", e);
}
}
}
@@ -1222,9 +1217,66 @@ public class IntMax extends EvalFunc<
}
}
</source>
-
</section>
+
+ <!-- =============================================================== -->
+ <section>
+ <title>JavaScript UDFs</title>
+ <section>
+ <title>Syntax</title>
+ <section>
+ <title>Registering Scripts</title>
+ <p>You can register JavaScript as shown here. This example uses org.apache.pig.scripting.js.JsScriptEngine to interpret JavaScript. You can develop and use custom script engines to support multiple programming languages and ways to interpret them. Currently, Pig identifies js as a keyword and ships the required scriptengine (js) to interpret it. </p>
+ <source>
+ Register 'test.js' using js as myfuncs;
+ </source>
+
+ <p>The following syntax is also supported, where myfuncs is the namespace created for all the functions inside test.js.</p>
+ <source>
+ register 'test.js' using org.apache.pig.scripting.js.JsScriptEngine as myfuncs;
+ </source>
+
+ <p>A typical test.js looks like this:</p>
+ <source>
+
+ helloworld.outputSchema = "word:chararray";
+ function helloworld() {
+ return 'Hello, World';
+ }
+
+ complex.outputSchema = "word:chararray,num:long";
+ function complex(word) {
+ return {word:word, num:word.length};
+ }
+
+ </source>
+
+ <p>The register statement above registers the js functions defined in test.js in Pigâs runtime within the defined namespace (myfuncs here). They can then be referred later on in the pig script as myfuncs.helloworld(), myfuncs.complex(), and myfuncs.square(). An example usage is:</p>
+
+ <source>
+ b = foreach a generate myfuncs.helloworld(), myfuncs.complex($0);
+ </source>
+
+ </section>
+ <section>
+ <title>outputSchema attribute and Schemas</title>
+ <p>As JavaScript functions are first class objects, you can annotate them by adding attributes. Add an outputSchema attribute to your function so that Pig can identify return types for the script UDF. </p>
+ <ul>
+ <li>outputSchema - Defines schema for a script udf in a format that Pig understands and is able to parse. </li>
+ </ul>
+
+ <p>Sample Schema String - y:{t:(word:chararray,num:long)}, variable names inside a schema string are used for type conversion between Pig and JavaScript. Tuples are converted to Objects using the names and vice versa.</p>
+ </section>
+ </section>
+
+ <section>
+ <title>Inline Scripts</title>
+ <p>Currently, Pig does not support UDFs using inline scripts. </p>
+ </section>
+ </section>
+
+
<!-- =============================================================== -->
<section>
<title>Python UDFs</title>
@@ -1292,62 +1344,64 @@ b = foreach a generate myfuncs.helloworl
<p>Currently, Pig does not support UDFs using inline scripts. </p>
</section>
-<section>
-<title>Sample Script UDFs</title>
-<p>Simple tasks like string manipulation, mathematical computations, and reorganizing data types can be easily done using python scripts without having to develop long and complex UDFs in Java. The overall overhead of using scripting language is much less and development cost is almost negligible. The following UDFs, developed in python, can be used with Pig. </p>
-<source>
-mySampleLib.py
----------------------
-#/usr/bin/python
-
-##################
-# Math functions #
-##################
-#Square - Square of a number of any data type
-@outputSchemaFunction("squareSchema")
-def square(num):
- return ((num)*(num))
-@schemaFunction("squareSchema")
-def squareSchema(input):
- return input
+ <section>
+ <title>Sample Script UDFs</title>
+ <p>Simple tasks like string manipulation, mathematical computations, and reorganizing data types can be easily done using python scripts without having to develop long and complex UDFs in Java. The overall overhead of using scripting language is much less and development cost is almost negligible. The following UDFs, developed in python, can be used with Pig. </p>
+
+ <source>
+ mySampleLib.py
+ ---------------------
+ #/usr/bin/python
+
+ ##################
+ # Math functions #
+ ##################
+ #Square - Square of a number of any data type
+ @outputSchemaFunction("squareSchema")
+ def square(num):
+ return ((num)*(num))
+ @schemaFunction("squareSchema")
+ def squareSchema(input):
+ return input
+
+ #Percent- Percentage
+ @outputSchema("percent:double")
+ def percent(num, total):
+ return num * 100 / total
+
+ ####################
+ # String Functions #
+ ####################
+ #commaFormat- format a number with commas, 12345-> 12,345
+ @outputSchema("numformat:chararray")
+ def commaFormat(num):
+ return '{:,}'.format(num)
+
+ #concatMultiple- concat multiple words
+ @outputSchema("onestring:chararray")
+ def concatMult4(word1, word2, word3, word4):
+ return word1 word2 word3 word4
+
+ #######################
+ # Data Type Functions #
+ #######################
+ #collectBag- collect elements of a bag into other bag
+ #This is useful UDF after group operation
+ @outputSchema("y:bag{t:tuple(len:int,word:chararray)}")
+ def collectBag(bag):
+ outBag = []
+ for word in bag:
+ tup=(len(bag), word[1])
+ outBag.append(tup)
+ return outBag
+
+ # Few comments-
+ # pig mandates that a bag should be a bag of tuples, python UDFs should follow this pattern.
+ # tuple in python are immutable, appending to a tuple is not possible.
+ </source>
+ </section>
-#Percent- Percentage
-@outputSchema("percent:double")
-def percent(num, total):
- return num * 100 / total
-
-####################
-# String Functions #
-####################
-#commaFormat- format a number with commas, 12345-> 12,345
-@outputSchema("numformat:chararray")
-def commaFormat(num):
- return '{:,}'.format(num)
-
-#concatMultiple- concat multiple words
-@outputSchema(âonestring:chararray")
-def concatMult4(word1, word2, word3, word4):
- return word1+word2+word3+word4
-
-#######################
-# Data Type Functions #
-#######################
-#collectBag- collect elements of a bag into other bag
-#This is useful UDF after group operation
-@outputSchema(ây:bag{t:tuple(len:int,word:chararray)}â)
-def collectBag(bag):
- outBag = []
- for word in bag:
- tup=(len(bag), word[1])
- outBag.append(tup)
- return outBag
-
-# Few comments-
-# pig mandates that a bag should be a bag of tuples, python UDFs should follow this pattern.
-# tuple in python are immutable, appending to a tuple is not possible.
-</source>
-</section>
</section>
<!-- =============================================================== -->