You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by ol...@apache.org on 2011/05/28 00:09:18 UTC
svn commit: r1128482 [2/3] - in /pig/trunk: ./
src/docs/src/documentation/content/xdocs/
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/cmds.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/cmds.xml?rev=1128482&r1=1128481&r2=1128482&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/cmds.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/cmds.xml Fri May 27 22:09:18 2011
@@ -31,7 +31,7 @@
<!-- +++++++++++++++++++++++++++++++++++++++ -->
<section id="fs">
<title>fs</title>
- <p>Invokes any FSShell command from within a Pig script or the Grunt shell.</p>
+ <p>Invokes any FsShell command from within a Pig script or the Grunt shell.</p>
<section>
<title>Syntax </title>
@@ -51,7 +51,7 @@
<p>subcommand</p>
</td>
<td>
- <p>The FSShell command.</p>
+ <p>The FsShell command.</p>
</td>
</tr>
<tr>
@@ -59,7 +59,7 @@
<p>subcommand_parameters</p>
</td>
<td>
- <p>The FSShell command parameters.</p>
+ <p>The FsShell command parameters.</p>
</td>
</tr>
</table>
@@ -68,10 +68,10 @@
<section>
<title>Usage</title>
- <p>Use the fs command to invoke any FSShell command from within a Pig script or Grunt shell.
+ <p>Use the fs command to invoke any FsShell command from within a Pig script or Grunt shell.
The fs command greatly extends the set of supported file system commands and the capabilities
supported for existing commands such as ls that will now support globing. For a complete list of
- FSShell commands, see
+ FsShell commands, see
<a href="http://hadoop.apache.org/common/docs/current/file_system_shell.html">File System Shell Guide</a></p>
</section>
@@ -208,7 +208,7 @@ grunt>
<title>Usage</title>
<p>Use the exec command to run a Pig script with no interaction between the script and the Grunt shell (batch mode). Aliases defined in the script are not available to the shell; however, the files produced as the output of the script and stored on the system are visible after the script is run. Aliases defined via the shell are not available to the script. </p>
<p>With the exec command, store statements will not trigger execution; rather, the entire script is parsed before execution starts. Unlike the run command, exec does not change the command history or remembers the handles used inside the script. Exec without any parameters can be used in scripts to force execution up to the point in the script where the exec occurs. </p>
- <p>For comparison, see the run command. Both the exec and run commands are useful for debugging because you can modify a Pig script in an editor and then rerun the script in the Grunt shell without leaving the shell. Also, both commands promote Pig script modularity as they allow you to reuse existing components.</p>
+ <p id="exec-debug">For comparison, see the run command. Both the exec and run commands are useful for debugging because you can modify a Pig script in an editor and then rerun the script in the Grunt shell without leaving the shell. Also, both commands promote Pig script modularity as they allow you to reuse existing components.</p>
</section>
<section>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/cont.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/cont.xml?rev=1128482&r1=1128481&r2=1128482&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/cont.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/cont.xml Fri May 27 22:09:18 2011
@@ -28,7 +28,8 @@
<p>To enable control flow, you can embed Pig Latin statements and Pig commands in the Python and JavaScript scripting languages using a JDBC-like compile, bind, run model. For Python, make sure the Jython jar is included in your class path. For JavaScript, make sure the Rhino jar is included in your classpath.</p>
-<p>(In addition to the discussion here, also see the Advanced Topics for Python and JavaScript in <a href="udf.html">User Defined Functions</a>.)</p>
+<p>Note that host languages and the languages of UDFs (included as part of the embedded Pig) are completely orthogonal. For example, a Pig Latin statement that registers a Python UDF may be embedded in Python, JavaScript, or Java. The exception to this rule is "combined" scripts â here the languages must match (see the <a href="udf.html#python-advanced">Advanced Topics for Python</a> and <a href="udf.html#js-advanced">Advanced Topics for JavaScript</a>). </p>
+
<!-- ============================================== -->
<section id="invocation-basics">
@@ -265,17 +266,27 @@ Pig.compile(...).bind(...).runSingle(pro
</section>
-<section id="pig-run-api">
+<section id="pigrunner-api">
<title>PigRunner API</title>
<p>Starting with Pig 0.8, some applications such as Oozie workflow invoke Pig using the PigRunner Java class rather than through the command line. For these applications, the PigRunner interface has been expanded to accommodate embedded Pig. PigRunner accepts Python and JavaScript scripts as input. These scripts can potentially contain multiple Pig pipelines; therefore, we need a way to return results for all of them.</p>
-<p id="pigstats">To do this and to preserve backward compatibility PigStats and related objects were expanded as shown below:</p>
+<p>To do this and to preserve backward compatibility PigStats and related objects were expanded as shown below:</p>
<ul>
-<li>getAllStats() and List< > getAllErrorMessages() were added to the PigStats class. The map returned from getAllStats is keyed on the name of the pipeline provided in the compile call. If the name was not compiled an internally generated id would be used. </li>
-<li>PigStats is now an abstract class with a new abstract method isEmbedded(). </li>
-<li>Two new classes that extend PigStats were created: SimplePigStats and EmbeddedPigStats. PigStats as it is today will become SimplePigStats. SimplePigStats.getAllStats() will return null. EmbeddedPigStats will return null for methods not listed in the proposal below. </li>
-<li>The PigProgressNotificationListener interface was modified to add script id to all its methods. </li>
+
+<li id="PigStats">PigStats is now an abstract class. (PigStats as it is today will become SimplePigStats.)</li>
+
+
+<li id="SimplePigStats">SimplePigStats is a new class that extends PigStats. SimplePigStats.getAllStats() will return null. </li>
+
+<li id="EmbeddedPigStats">EmbeddedPigStats is a new class that extends PigStats. EmbeddedPigStats will return null for methods not listed in the proposal below. </li>
+
+
+<li id="isembedded">isEmbedded() is a new abstract method that accommodates embedded Pig.</li>
+
+<li id="stats-messages">getAllStats() and List< > getAllErrorMessages() methods were added to the PigStats class. The map returned from getAllStats is keyed on the name of the pipeline provided in the compile call. If the name was not compiled an internally generated id would be used. </li>
+
+<li id="PigProgressNotificationListener2">The PigProgressNotificationListener interface was modified to add script id to all its methods. </li>
</ul>
<p></p>
<p>For more details, see <a href="#Java-Objects">Java Objects</a>.</p>
@@ -308,7 +319,7 @@ else:
</source>
</section>
-<section>
+<section id="convergence">
<title>Convergence</title>
<p>There is a class of problems that involve iterating over a data pipeline an indeterminate number of times until a certain value is reached. Examples arise in machine learning, graph traversal, and a host of numerical analysis problems which involve finding interpolations, extrapolations or regressions. The Python example below shows one way to achieve convergence using Pig scripts.</p>
@@ -349,7 +360,7 @@ for i in range(1, 4):
</source>
</section>
-<section>
+<section id="automated-pig-latin">
<title>Automated Pig Latin Generation</title>
<p>A number of user frameworks do automated generation of Pig Latin, such as setting the proper date for data loads.</p>
@@ -736,6 +747,8 @@ public abstract class PigStats {
<p>To enable control flow, you can embed Pig Latin statements and Pig commands in the Java programming language. </p>
+<p>Note that host languages and the languages of UDFs (included as part of the embedded Pig) are completely orthogonal. For example, a Pig Latin statement that registers a Java UDF may be embedded in Python, JavaScript, or Java. The exception to this rule is "combined" scripts â here the languages must match (see the <a href="udf.html#python-advanced">Advanced Topics for Python</a> and <a href="udf.html#js-advanced">Advanced Topics for JavaScript</a>). </p>
+
<section id="pigserver">
<title>PigServer Interface</title>
<p>Currently, <a href="http://pig.apache.org/docs/r0.9.0/api/org/apache/pig/PigServer.html">PigServer</a> is the main interface for embedding Pig in Java. PigServer can now be instantiated from multiple threads. (In the past, PigServer contained references to static data that prevented multiple instances of the object to be created from different threads within your application.) Please note that PigServer is NOT thread safe; the same object can't be shared across multiple threads. </p>
@@ -914,12 +927,12 @@ public static void runIdQuery(PigServer
<p><strong>Macro Definition</strong></p>
<p>A macro definition can appear anywhere in a Pig script as long as it appears prior to the first use. A macro definition can include references to other macros as long as the referenced macros are defined prior to the macro definition. Recursive references are not allowed. </p>
- <p>Note the following:</p>
+ <p>Note the following restrictions:</p>
<ul>
- <li>Macros are NOT allowed inside a <a href="basic.html#FOREACH">FOREACH</a> nested block.</li>
- <li>Macros can only contain Pig Latin statements. Grunt shell commands are not supported.</li>
- <li>Macros cannot include user-defined schema that has a name collision with an alias in the macro.</li>
- <li><a href="#Parameter-Sub">Parameter substitution</a> CANNOT be used inside of macros. Parameters should be explicitly passed to macros and parameter substitution used only at the top level.</li>
+ <li>Macros are not allowed inside a <a href="basic.html#nested-block">FOREACH</a> nested block.</li>
+ <li>Macros can only contain <a href="start.html#pl-statements">Pig Latin statements</a>. The <a href="basic.html#register">REGISTER</a> statement is not supported. The <a href="cmds.html#shell-cmds">shell commands</a> (used with Grunt) are not supported.</li>
+ <li>Macros cannot include a user-defined schema that has a name collision with an alias in the macro.</li>
+ <li><a href="#Parameter-Sub">Parameter substitution</a> cannot be used inside of macros. Parameters should be explicitly passed to macros and parameter substitution used only at the top level.</li>
</ul>
<p></p>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/func.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/func.xml?rev=1128482&r1=1128481&r2=1128482&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/func.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/func.xml Fri May 27 22:09:18 2011
@@ -55,7 +55,7 @@ decoded_strings = FOREACH encoded_string
<p>The <a href="basic.html#define">DEFINE</a> statement is used to bind a keyword to a Java method, as above. The first argument to the InvokeFor* constructor is the full path to the desired method. The second argument is a space-delimited ordered list of the classes of the method arguments. This can be omitted or an empty string if the method takes no arguments. Valid class names are string, long, float, double, and int. Invokers can also work with array arguments, represented in Pig as DataBags of single-tuple elements. Simply refer to string[], for example. Class names are not case sensitive. </p>
-<p>The ability to use invokers on methods that take array arguments makes methods like those in org.apache.commons.math.stat.StatUtils available (for processing the results of grouping your datasets, for example). This is helpful, but a word of caution: the resulting UDF will not be optimized for Hadoop, and the very significant benefits one gains from implementing the Algebraic and Accumulative interfaces are lost here. Be careful if you use invokers this way.</p>
+<p>The ability to use invokers on methods that take array arguments makes methods like those in org.apache.commons.math.stat.StatUtils available (for processing the results of grouping your datasets, for example). This is helpful, but a word of caution: the resulting UDF will not be optimized for Hadoop, and the very significant benefits one gains from implementing the Algebraic and Accumulator interfaces are lost here. Be careful if you use invokers this way.</p>
</section>
<!-- ======================================================== -->
@@ -1159,7 +1159,7 @@ dump b;
<ul>
<li>Sends an error message when the second script is executed: "ERROR 1118: Cannot cast bytes loaded from BinStorage. Please provide a custom converter."</li>
-<li>Allows you to use a custom converter to perform the casting. <br></br>
+<li id="custom-converter">Allows you to use a custom converter to perform the casting. <br></br>
<source>
a = load 'g/part*' using BinStorage('Utf8StorageConverter') as (id, d:bag{t:(v, s)});
b = foreach a generate (double)id, flatten(d);
@@ -1284,7 +1284,7 @@ STORE X INTO 'output' USING PigDump();
<table>
<tr>
<td>
- <p>field_delimiter</p>
+ <p id="field-delimiter">field_delimiter</p>
</td>
<td>
<p>Parameter. </p>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/index.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/index.xml?rev=1128482&r1=1128481&r2=1128482&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/index.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/index.xml Fri May 27 22:09:18 2011
@@ -29,8 +29,11 @@
become familiar with the Pig Latin operators and the supported data types.</p>
<p>Functions can be a part of almost every operator in Pig. The <a href="func.html">Built In Functions</a> guide describes Pig's built in functions.
- The <a href="udf.html">User Defined Functions</a> manual shows you how to how to write your own functions and how to access functions
- contributed by other Pig users. </p>
+ The <a href="udf.html">User Defined Functions</a> manual shows you how to how to write your own functions and how to access/contribute functions using the Piggy Bank repository.</p>
+
+ <p>The mechanisms featured in the <a href="cont.html">Control Structures</a> guide give you greater control over how your Pig scripts are structured and executed. The <a href="cont.html">Performance and Efficiency</a> guide provides valuable examples and suggestions for optimizing your code.</p>
+
+ <p>Finally, use Pig's <a href="cmds.html">Shell and Utility Commands</a> to run your programs and Pig's expanded <a href="test.html">Testing and Diagnostics</a> tools to examine and/or debug your programs.</p>
<p>If you have more questions, you can ask on the <a href="http://hadoop.apache.org/pig/mailing_lists.html">Pig Mailing Lists</a>.</p>
</body>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml?rev=1128482&r1=1128481&r2=1128482&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml Fri May 27 22:09:18 2011
@@ -149,7 +149,7 @@ D = foreach B generate group, MIN (A.gpa
<p>Pig allocates a fix amount of memory to store bags and spills to disk as soon as the memory limit is reached. This is very similar to how Hadoop decides when to spill data accumulated by the combiner. </p>
-<p>The amount of memory allocated to bags is determined by pig.cachedbag.memusage; the default is set to 20% (0.2) of available memory. Note that this memory is shared across all large bags used by the application.</p>
+<p id="memory-bags">The amount of memory allocated to bags is determined by pig.cachedbag.memusage; the default is set to 20% (0.2) of available memory. Note that this memory is shared across all large bags used by the application.</p>
</section>
@@ -188,7 +188,7 @@ can be combined to reduce the overall am
</li>
</ul>
-<section>
+<section id="splits">
<title>Explicit and Implicit Splits</title>
<p>There might be cases in which you want different processing on separate parts of the same data stream.</p>
<p>Example 1:</p>
@@ -226,7 +226,7 @@ Both are equivalent, but the performance
</ul>
</section>
-<section>
+<section id="data-store-performance">
<title>Storing Intermediate Results</title>
<p>Sometimes it is necessary to store intermediate results. </p>
@@ -340,7 +340,7 @@ A = LOAD 'sql://mytable' USING SQLLoader
</source>
</section>
-<section>
+<section id="Implicit-Dependencies">
<title>Implicit Dependencies</title>
<p>If a script has dependencies on the execution order outside of what Pig knows about, execution may fail. </p>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/pig-index.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/pig-index.xml?rev=1128482&r1=1128481&r2=1128482&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/pig-index.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/pig-index.xml Fri May 27 22:09:18 2011
@@ -58,42 +58,43 @@
<p></p>
<p></p>
-<!-- ==== A ================================================================== -->
+<!-- ==== Symbols ================================================================== -->
<p></p>
<p id="sym-index"><strong>Symbols</strong></p>
-<p>+ <a href="basic.html#arithmetic">addition operator</a></p>
-<p>?: <a href="basic.html#arithmetic">bincond operator</a></p>
-<p>/* */ <a href="start.html#comments">multi-line comments</a></p>
-<p>-- <a href="start.html#comments">single-line comments</a></p>
-<p># <a href="basic.html#deref">deference operator (map)</a></p>
-<p>. <a href="basic.html#deref">deference operator (tuple, bag)</a></p>
-<p>/ <a href="basic.html#arithmetic">division operator</a></p>
-<p>== <a href="basic.html#comparison">equal operator</a></p>
-<p>> <a href="basic.html#comparison">greater than operator</a></p>
-<p>>= <a href="basic.html#comparison">greater than or equal to operator</a></p>
-<p>< <a href="basic.html#comparison">less than operator</a></p>
-<p><= <a href="basic.html#comparison">less than or equal to operator</a></p>
-<p>.. <a href="basic.html#prexp">project-range expression</a></p>
-<p>% <a href="basic.html#arithmetic">modulo operator</a></p>
-<p>* <a href="basic.html#arithmetic">multiplication operator</a></p>
-<p>!= <a href="basic.html#comparison">not equal operator</a></p>
-<p>+ <a href="basic.html#sign">sign operator (positive)</a></p>
-<p>- <a href="basic.html#sign">sign operator (negative)</a></p>
-<p>* <a href="basic.html#sexp">star expression</a></p>
-<p>- <a href="basic.html#arithmetic">subtraction operator</a></p>
+<p> + <a href="basic.html#arithmetic">addition operator</a></p>
+<p> ?: <a href="basic.html#arithmetic">bincond operator</a></p>
+<p> /* */ <a href="start.html#comments">comments - multi-line</a></p>
+<p> -- <a href="start.html#comments">comments - single-line</a></p>
+<p> # <a href="basic.html#deref">deference operator (map)</a></p>
+<p> . <a href="basic.html#deref">deference operator (tuple, bag)</a></p>
+<p> :: <a href="basic.html#disambiguate">disambiguate operator</a></p>
+<p> / <a href="basic.html#arithmetic">division operator</a></p>
+<p> == <a href="basic.html#comparison">equal operator</a></p>
+<p> > <a href="basic.html#comparison">greater than operator</a></p>
+<p> >= <a href="basic.html#comparison">greater than or equal to operator</a></p>
+<p> < <a href="basic.html#comparison">less than operator</a></p>
+<p> <= <a href="basic.html#comparison">less than or equal to operator</a></p>
+<p> % <a href="basic.html#arithmetic">modulo operator</a></p>
+<p> * <a href="basic.html#arithmetic">multiplication operator</a></p>
+<p> != <a href="basic.html#comparison">not equal operator</a></p>
+<p> .. <a href="basic.html#prexp">project-range expression</a></p>
+<p> - <a href="basic.html#sign">sign operator (negative)</a></p>
+<p> + <a href="basic.html#sign">sign operator (positive)</a></p>
+<p> * <a href="basic.html#sexp">star expression</a></p>
+<p> - <a href="basic.html#arithmetic">subtraction operator</a></p>
<!-- ==== A ================================================================== -->
<p></p>
-<p id="a-index"><strong>A</strong> (<a href="#top">top</a>)</p>
+<p id="a-index"><strong>A</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
-<p><a href="func.html#abs">ABS</a></p>
+<p><a href="func.html#abs">ABS</a> function</p>
<p><a href="udf.html#accumulator-interface">accumulator interface</a></p>
-<p><a href="func.html#acos">ACOS</a></p>
+<p><a href="func.html#acos">ACOS</a> function</p>
<p><a href="udf.html#aggregate-functions">aggregate functions</a></p>
@@ -102,101 +103,134 @@
<p>aliases (for fields, relations). <em>See</em> referencing.</p>
<p><a href="start.html#dfs">Amazon S3</a></p>
+
<p><a href="basic.html#boolops">AND</a> (Boolean)</p>
+
<p><a href="basic.html#arithmetic">arithmetic operators</a></p>
-<p><a href="func.html#asin">ASIN</a></p>
-<p><a href="func.html#atan">ATAN</a></p>
-<p><a href="func.html#avg">AVG</a></p>
+<p><a href="func.html#asin">ASIN</a> function</p>
+
+<p><a href="func.html#atan">ATAN</a> function</p>
+
+<p><a href="basic.html#autoship">autoship</a> (streaming). <em>See also</em> ship</p>
+
+<p><a href="func.html#avg">AVG</a> function</p>
<!-- ==== B ================================================================== -->
<p></p>
-<p id="b-index"><strong>B</strong> (<a href="#top">top</a>)</p>
+<p id="b-index"><strong>B</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
<p><a href="perf.html#backward-compatibility">backward compatibility</a> (multi-query execution)</p>
+<p><a href="func.html#bag-tuple-functions">bag functions</a></p>
+
<p>bags (data type)
+<br></br> <a href="perf.html#memory-bags">and memory allocation</a>
<br></br> <a href="basic.html#relations">and relations</a>
-<br></br> <a href="basic.html#bag-schema">bag schema</a>
+<br></br> <a href="basic.html#bag-schema">and schemas</a>
<br></br> <a href="basic.html#schema-multi">schemas for multiple types</a>
<br></br> <a href="basic.html#bag">syntax</a>
</p>
-<p><a href="start.html#batch-mode">batch mode</a></p>
+<p><a href="start.html#batch-mode">batch mode</a>. <em>See also</em> memory management</p>
<p><a href="basic.html#arithmetic">bincond operator</a> ( ?: )</p>
-<p><a href="func.html#binstorage">BinStorage</a></p>
+<p><a href="func.html#binstorage">BinStorage</a> function</p>
-<p>Booleans
-<br></br> <a href="basic.html#bexp">expressions</a>
-<br></br> <a href="basic.html#boolops">AND operator</a>
-<br></br> <a href="basic.html#boolops">OR operator</a>
-<br></br> <a href="basic.html#boolops">NOT operator</a>
+<p><a href="basic.html#bexp">Boolean expressions</a></p>
+
+<p>Boolean operators
+<br></br> <a href="basic.html#boolops">AND</a> operator
+<br></br> <a href="basic.html#boolops">OR</a> operator
+<br></br> <a href="basic.html#boolops">NOT</a> operator
</p>
-<p><a href="cont.html#boundscript-object">BoundScript.java object</a></p>
+<p><a href="cont.html#boundscript-object">BoundScript.java</a> object</p>
<p><a href="start.html#build">building Pig</a></p>
+<p><a href="func.html#built-in-functions">built in functions</a></p>
<!-- ==== C ================================================================== -->
<p></p>
-<p id="c-index"><strong>C</strong> (<a href="#top">top</a>)</p>
+<p id="c-index"><strong>C</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
+
+
+<p><a href="basic.html#cache-about">cache</a> (streaming)</p>
<p><a href="basic.html#case-sensitivity">case sensitivity</a></p>
<p>casting types
-<br></br> <a href="basic.html#cast">operators</a>
+<br></br> <a href="basic.html#cast">cast operators</a>
+<br></br> <a href="func.html#custom-converter">custom converters</a> (BinStorage)
<br></br> <a href="basic.html#cast-relations">relations to scalars</a>
<br></br> <em>See also</em> types tables
</p>
-<p><a href="func.html#cbrt">CBRT</a></p>
+<p><a href="func.html#cbrt">CBRT</a> function</p>
-<p><a href="func.html#ceil">CEIL</a></p>
+<p><a href="func.html#ceil">CEIL</a> function</p>
-<p><a href="udf.html#checkschema">checkSchema</a></p>
+<p><a href="udf.html#checkschema">checkSchema</a> method</p>
-<p><a href="basic.html#cogroup">COGROUP</a></p>
+<p><a href="basic.html#cogroup">COGROUP</a> operator</p>
<p><a href="perf.html#combiner">combiner</a></p>
+<p><a href="start.html#comments">comments</a> (in Pig Scripts)</p>
+
<p><a href="basic.html#comparison">comparison operators</a></p>
-<p><a href="start.html#comments">comments</a> (in Pig Scripts)</p>
+<p>compression (of data)
+<br></br> <a href="func.html#handling-compression">handling compression</a>
+<br></br> <a href="perf.html#compression">compressing results of intermediate jobs</a>
+</p>
+
+<p><a href="func.html#concat">CONCAT</a> function</p>
+
+<p>constants
+<br></br> <a href="basic.html#constants">and data types</a>
+<br></br> <a href="basic.html#nulls-constants">and nulls</a>
+</p>
+
+<p><a href="cont.html#convergence">convergence</a> (Python example)</p>
-<p><a href="func.html#handling-compression">compression</a> (of data)</p>
+<p><a href="func.html#cosh">COS</a> function</p>
-<p><a href="func.html#concat">CONCAT</a></p>
-<p><a href="basic.html#constants">constants</a></p>
-<p><a href="func.html#cosh">COS</a></p>
-<p><a href="func.html#cosh">COSH</a></p>
-<p><a href="func.html#count">COUNT</a></p>
-<p><a href="func.html#count-star">COUNT_STAR</a></p>
+<p><a href="func.html#cosh">COSH</a> function </p>
-<p><a href="basic.html#cross">CROSS</a></p>
+<p><a href="func.html#count">COUNT</a> function</p>
+
+<p><a href="func.html#count-star">COUNT_STAR</a> function</p>
+
+<p><a href="basic.html#cross">CROSS</a> operator</p>
<!-- ==== D ================================================================== -->
<p></p>
-<p id="d-index"><strong>D</strong> (<a href="#top">top</a>)</p>
+<p id="d-index"><strong>D</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
+
+<p><a href="start.html#properties">-D</a> command line option</p>
<p>data
-<br></br> <a href="func.html#handling-compression">compression</a>
+<br></br> <a href="perf.html#combine-files">combining input files</a>
+<br></br> <a href="func.html#handling-compression">compression</a> (handling)
+<br></br> <a href="perf.html#compression">compression</a> (results of intermediate jobs)
<br></br> <a href="start.html#data-load">loading</a>
-<br></br> <a href="func.html#load-store-functions">load/store functions</a>
-<br></br> <a href="start.html#data-store">storing intermediate results</a>
+<br></br> <a href="func.html#load-store-functions">load/store functions</a> (built in functions)
+<br></br> <a href="udf.html#load-store-functions">load/store functions</a> (user defined functions)
<br></br> <a href="start.html#data-results">storing final results</a>
+<br></br> <a href="start.html#data-store">storing intermediate results</a> (and HDFS)
+<br></br> <a href="perf.html#data-store-performance">storing intermediate results</a> (and performance)
<br></br> <a href="start.html#data-work-with">working with</a>
</p>
-<p>data types
-<br></br> <a href="basic.html#data-types">in Pig Latin</a>
-</p>
+<p><a href="basic.html#data-types">data types</a> (Pig Latin)</p>
<p>debugging
<br></br> <a href="test.html#diagnostic-ops">diagnostic operators</a>
-<br></br> <a href="test.html#penny">Penny </a>
+<br></br> <a href="cmds.html#exec-debug">with exec and run commands</a>
+<br></br> <a href="test.html#penny">and Penny </a>
<br></br> <a href="start.html#debug">and Pig Latin</a>
</p>
@@ -207,19 +241,25 @@
<br></br> <a href="basic.html#deref">map</a> ( # )
</p>
-<p><a href="cont.html#define-macros">DEFINE (macros)</a> </p>
-<p><a href="basic.html#define-udfs">DEFINE (UDFs, streaming)</a> </p>
+<p><a href="cont.html#define-macros">DEFINE (macros)</a> operator</p>
+
+<p><a href="basic.html#define-udfs">DEFINE (UDFs, streaming)</a> operator</p>
-<p><a href="test.html#describe">DESCRIBE</a></p>
+<p><a href="test.html#describe">DESCRIBE</a> operator</p>
+
+<p><a href="func.html#diff">DIFF</a> function</p>
+
+<p><a href="basic.html#disambiguate">disambiguate operator</a> ( :: )</p>
<p><a href="start.html#dfs">distributed file systems</a> (and Pig Scripts)</p>
-<p><a href="basic.html#distinct">DISTINCT</a></p>
+<p><a href="basic.html#distinct">DISTINCT</a> operator</p>
+
+<p><a href="perf.html#distinct">DISTINCT and optimization</a></p>
<p><a href="udf.html#distributed-cache">distributed cache</a></p>
-<p><a href="start.html#download">downloading Pig</a></p>
-<p><a href="func.html#diff">DIFF</a></p>
+<p><a href="start.html#download">downloading Pig</a></p>
<p><a href="test.html#dump">DUMP</a> <em>See also </em> Store vs. Dump</p>
@@ -227,54 +267,50 @@
<!-- ==== E ================================================================== -->
<p></p>
-<p id="e-index"><strong>E</strong> (<a href="#top">top</a>)</p>
+<p id="e-index"><strong>E</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
<p>embedded Pig
<br></br> <a href="cont.html#invocation-basics">invocation basics</a>
<br></br> <a href="cont.html#invocation-details">invocation details</a> (compile, bind, run)
<br></br> <a href="cont.html#embed-java">and Java</a>
<br></br> <a href="cont.html#embed-python">and JavaScript</a>
-<br></br> <a href="cont.html#pig-run-api">and PigRunner API</a>
+<br></br> <a href="cont.html#pigrunner-api">and PigRunner API</a>
<br></br> <a href="cont.html#pigserver">and PigServer Interface</a>
<br></br> <a href="cont.html#embed-python">and Python</a>
</p>
-<p><a href="cont.html#pigstats">EmbeddedPigStats</a></p>
+<p><a href="cont.html#EmbeddedPigStats">EmbeddedPigStats</a> class</p>
<p><a href="perf.html#error-handling">error handling</a> (multi-query execution)</p>
-<p><a href="func.html#eval-functions">eval functions</a> (built in)</p>
+<p><a href="func.html#eval-functions">eval functions</a> (built in functions)</p>
-<p><a href="udf.html#eval-functions">eval functions</a> (UDFs)
-<br></br> <a href="udf.html#aggregate-functions">aggregate functions</a>
-<br></br> <a href="udf.html#distributed-cache">and distributed cache</a>
-<br></br> <a href="udf.html#error-handling">and error handling</a>
-<br></br> <a href="udf.html#function-overloading">and function overloading</a>
-<br></br> <a href="udf.html#filter-functions">filter functions</a>
-<br></br> <a href="udf.html#import-lists">and import lists</a>
-<br></br> <a href="udf.html#pig-types">and Pig types</a>
-<br></br> <a href="udf.html#reporting-progress">reporting progress of</a>
-<br></br> <a href="udf.html#schemas">and schemas</a>
-<br></br> <a href="udf.html#eval-functions-use">using, how to</a>
-<br></br> <a href="udf.html#eval-functions-write">writing, how to</a>
-</p>
+<p><a href="udf.html#eval-functions">eval functions</a> (user defined functions). <em>See also</em> Java UDFs</p>
+
+<p><a href="cmds.html#exec">exec</a> command</p>
-<p><a href="cmds.html#exec">exec</a></p>
+<p>executing Pig. <em>See</em> running Pig</p>
<p>exectution modes
<br></br> <a href="start.html#execution-modes">local mode</a>
<br></br> <a href="start.html#execution-modes">mapreduce mode</a>
</p>
-<p><a href="func.html#exp">EXP</a></p>
+<p>execution plans
+<br></br> <a href="test.html#logical-plan">logical plan</a>
+<br></br> <a href="test.html#mapreduce-plan">mapreduce plan</a>
+<br></br> <a href="test.html#physical-plan">physical plan</a>
+</p>
+
+<p><a href="func.html#exp">EXP</a> function</p>
-<p><a href="test.html#explain">EXPLAIN</a></p>
+<p><a href="test.html#explain">EXPLAIN</a> operator</p>
<p>expressions
-<br></br> <a href="basic.html#expressions">and Pig Latin</a>
<br></br> <a href="basic.html#bexp">Boolean expressions</a>
<br></br> <a href="basic.html#fexp">field expressions</a>
<br></br> <a href="basic.html#gexp">general expressions</a>
+<br></br> <a href="basic.html#expressions">and Pig Latin</a>
<br></br> <a href="basic.html#prexp">project-range expressions</a>
<br></br> <a href="basic.html#sexp">star expressions</a> ( * )
<br></br> <a href="basic.html#texp">tuple expressions</a>
@@ -282,81 +318,90 @@
<!-- ==== F ================================================================== -->
<p></p>
-<p id="f-index"><strong>F</strong> (<a href="#top">top</a>)</p>
+<p id="f-index"><strong>F</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
<p><a href="basic.html#fexp">field expressions</a></p>
<p>fields
<br></br> <a href="basic.html#relations">definition of</a>
+<br></br> <a href="func.html#field-delimiter">field delimiters</a>
<br></br> <a href="basic.html#ref-field">referencing</a>
-<br></br> <a href="basic.html#ref-field-complex">referencing and complex types</a>
+<br></br> <a href="basic.html#ref-field-complex">referencing complex types</a>
</p>
-<p><a href="basic.html#filter">FILTER</a></p>
+<p><a href="basic.html#filter">FILTER</a> operator</p>
-<p> <a href="udf.html#filter-functions">filter functions</a></p>
+<p><a href="perf.html#filter">FILTER and performance</a></p>
-<p><a href="basic.html#foreach">FOREACH</a></p>
+<p> <a href="udf.html#filter-functions">filter functions</a></p>
<p><a href="basic.html#flatten">flatten operator</a></p>
-<p><a href="func.html#floor">FLOOR</a></p>
+<p><a href="func.html#floor">FLOOR</a> function</p>
-<p><a href="cmds.html#fs">fs</a></p>
+<p><a href="basic.html#foreach">FOREACH</a> operator</p>
+
+<p><a href="cmds.html#fs">fs</a> command</p>
+
+<p><a href="cmds.html#fs">FsShell commands</a></p>
<!-- ==== G ================================================================== -->
<p></p>
-<p id="g-index"><strong>G</strong> (<a href="#top">top</a>)</p>
+<p id="g-index"><strong>G</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
<p><a href="basic.html#gexp">general expressions</a></p>
-<p><a href="udf.html#loadfunc-abstract">getInputFormat</a></p>
+<p><a href="cont.html#stats-messages">getAllErrorMessages</a> method</p>
+
+<p><a href="cont.html#stats-messages">getAllStats</a> method</p>
-<p><a href="func.html#pigstats">getAllErrorMessages</a></p>
-<p><a href="cont.html#pigstats">getAllStats</a></p>
+<p><a href="udf.html#getInputFormat">getInputFormat</a> method</p>
-<p><a href="udf.html#loadfunc-abstract">getNext</a></p>
-<p><a href="udf.html#storefunc-override">getOutputFormat</a></p>
+<p><a href="udf.html#getNext">getNext</a> method</p>
+
+<p><a href="udf.html#getOutputFormat">getOutputFormat</a> method</p>
+
+<p>globs
+<br></br> <a href="func.html#binstorage">and BinStorage function</a>
+<br></br> <a href="basic.html#load-glob">and LOAD operator</a>
+<br></br> <a href="basic.html#register-glob">and REGISTER statement</a>
+</p>
+
+<p><a href="basic.html#group">GROUP</a> operator</p>
-<p><a href="basic.html#group">GROUP</a></p>
<p><a href="start.html#interactive-mode">grunt shell</a></p>
<!-- ==== H ================================================================== -->
<p></p>
-<p id="h-index"><strong>H</strong> (<a href="#top">top</a>)</p>
+<p id="h-index"><strong>H</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
<p>Hadoop
+<br></br> <a href="cmds.html#fs">FsShell commands</a>
+<br></br> <a href="basic.html#load-glob">Hadoop globbing</a>
<br></br> <a href="test.html#hadoop-job-history-loader">HadoopJobHistoryLoader</a>
-<br></br> hadoop partitioner. <em>See </em> PARTITION BY
-<br></br> <a href="start.html#properties">properties</a>
-
+<br></br> hadoop partitioner. <em>See</em> PARTITION BY
+<br></br> <a href="start.html#hadoop-properties">Hadoop properties</a>
</p>
<p><a href="start.html#dfs">HDFS</a></p>
-<p><a href="cmds.html#help">help</a> (command)</p>
+
+<p><a href="cmds.html#help">help</a> command</p>
<p></p>
<!-- ==== I ================================================================== -->
<p></p>
-<p id="i-index"><strong>I</strong> (<a href="#top">top</a>)</p>
+<p id="i-index"><strong>I</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
-<p><a href="basic.html#identifiers">identifiers</a></p>
+<p><a href="basic.html#identifiers">identifiers</a> <em>See also</em> referencing</p>
-<p><a href="test.html#illustrate">ILLUSTRATE</a></p>
+<p><a href="test.html#illustrate">ILLUSTRATE</a> operator</p>
-<p><a href="cont.html#import-macros">IMPORT (macros)</a></p>
+<p><a href="perf.html#ImplicitSplitInserter">ImplicitSplitInserter</a> optimization rule</p>
+<p><a href="cont.html#import-macros">IMPORT (macros)</a> operator</p>
-<p><a href="perf.html#ImplicitSplitInserter">ImplicitSplitInserter </a></p>
-
-
-
-<p><a href="func.html#indexof">INDEXOF</a></p>
-
-
-<p><a href="cont.html#pigstats">isEmbedded</a></p>
-<p><a href="func.html#isempty">IsEmpty</a></p>
+<p><a href="func.html#indexof">INDEXOF</a> function</p>
<p>installing Pig
<br></br> <a href="start.html#build">builds</a>
@@ -366,73 +411,130 @@
<p><a href="start.html#interactive-mode">interactive mode</a></p>
+<p><a href="cont.html#isEmbedded">isEmbedded</a> method</p>
+
+<p><a href="func.html#isempty">IsEmpty</a> function</p>
+
+<p><a href="basic.html#is-not-null">is not null</a> operator</p>
+
+<p><a href="basic.html#is-null">is null</a> operator</p>
+
+
<!-- ==== J ================================================================== -->
<p></p>
-<p id="j-index"><strong>J</strong> (<a href="#top">top</a>)</p>
+<p id="j-index"><strong>J</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
+
+
+<p> <a href="cont.html#embed-java">Java and embedded Pig</a></p>
+<p>Java objects
+<br></br> <a href="cont.html#boundscript-object">BoundScript.java</a>
+<br></br> <a href="cont.html#pig-object">pig.java</a>
+<br></br> <a href="cont.html#PigProgressNotificationListener">PigProgressNotificationListener.java</a>
+<br></br> <a href="cont.html#pigstats-object">PigStats.java</a>
+</p>
+
+<p><a href="cont.html#embed-python">JavaScript and embedded Pig</a></p>
+
+<p><a href="udf.html#js-udfs">JavaScript UDFs</a>. <em>See also</em> UDFs</p>
+
+<p><a href="udf.html#udf-java">Java UDFs</a>
+<br></br> <a href="udf.html#eval-functions">eval functions</a>
+<br></br> <a href="udf.html#accumulator-interface">accumulator interface</a>
+<br></br> <a href="udf.html#aggregate-functions">aggregate functions</a>
+<br></br> <a href="udf.html#algebraic-interface">algebraic interface</a>
+<br></br> <a href="udf.html#distributed-cache">and distributed cache</a>
+<br></br> <a href="udf.html#error-handling">and error handling</a>
+<br></br> <a href="udf.html#filter-functions">filter functions</a>
+<br></br> <a href="udf.html#function-overloading">and function overloading</a>
+<br></br> <a href="udf.html#import-lists">and import lists</a>
+<br></br> <a href="udf.html#pig-types">and Pig types</a>
+<br></br> <a href="udf.html#reporting-progress">and reporting progress</a>
+<br></br> <a href="udf.html#schemas">and schemas</a>
+<br></br> <a href="udf.html#eval-functions-use">using the functions</a>
+<br></br> <a href="udf.html#eval-functions-write">writing the functions</a>
+<br></br> <a href="udf.html#load-store-functions">load/store functions</a>
+<br></br> <em>See also</em> UDFs
+</p>
-<p> <a href="cont.html#java-objects">Java objects</a></p>
+<p><a href="basic.html#join-inner">JOIN (inner)</a> operator</p>
-<p> <a href="udf.html#js-udfs">Javascript UDFs</a></p>
+<p><a href="basic.html#join-outer">JOIN (outer)</a> operator</p>
-<p><a href="basic.html#join-inner">JOIN (inner)</a> </p>
-<p><a href="basic.html#join-outer">JOIN (outer)</a></p>
+<p>joins
+<br></br> <a href="basic.html#join-inner">inner joins</a>
+<br></br> <a href="perf.html#join-optimizations">join optimizations</a>
+<br></br> <a href="perf.html#merge-joins">merge joins</a>
+<br></br> <a href="basic.html#join-outer">outer joins </a>
+<br></br> <a href="perf.html#replicated-joins">replicated joins</a>
+<br></br> <a href="basic.html#self-joins">self joins</a>
+<br></br> <a href="perf.html#skewed-joins">skewed joins</a>
+</p>
<!-- ==== K ================================================================== -->
<p></p>
-<p id="k-index"><strong>K</strong> (<a href="#top">top</a>)</p>
+<p id="k-index"><strong>K</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
<p>keywords. <em>See </em>reserved keywords</p>
-<p><a href="cmds.html#kill">kill</a> (command)</p>
+<p><a href="cmds.html#kill">kill</a> command</p>
<!-- ==== L ================================================================== -->
<p></p>
-<p id="l-index"><strong>L</strong> (<a href="#top">top</a>)</p>
+<p id="l-index"><strong>L</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
-<p><a href="func.html#last-index-of">LAST_INDEX_OF</a></p>
-<p><a href="func.html#lcfirst">LCFIRST</a></p>
-<p><a href="basic.html#limit">LIMIT</a></p>
-<p><a href="basic.html#load">LOAD</a></p>
-
-<p><a href="udf.html#load-store-functions">LoadCaster</a>. <em>See also</em> LoadFunc</p>
-
-<p><a href="udf.html#load-store-functions">LoadFunc</a>
-<br></br> <a href="udf.html#loadfunc-abstract">getInputFormat</a>
-<br></br> <a href="udf.html#loadfunc-abstract">getNext</a>
-<br></br> <a href="udf.html#load-store-functions">LoadCaster</a>
-<br></br> <a href="udf.html#load-store-functions">LoadMetadata</a>
-<br></br> <a href="udf.html#load-store-functions">LoadPushDown</a>
-<br></br> <a href="udf.html#loadfunc-abstract">prepareToRead</a>
-<br></br> <a href="udf.html#load-store-functions">pushProjection</a>
-<br></br> <a href="udf.html#loadfunc-default">relativeToAbsolutePath</a>
-<br></br> <a href="udf.html#loadfunc-abstract">setLocation</a>
-<br></br> <a href="udf.html#loadfunc-default">setUdfContextSignature</a>
+<p><a href="func.html#last-index-of">LAST_INDEX_OF</a> function</p>
+
+<p><a href="func.html#lcfirst">LCFIRST</a> function</p>
+
+<p><a href="basic.html#limit">LIMIT</a> operator</p>
+
+<p><a href="perf.html#limit">LIMIT and optimization</a></p>
+
+<p><a href="basic.html#load">LOAD</a> operator</p>
+
+<p><a href="udf.html#LoadCaster">LoadCaster</a> interface</p>
+
+<p><a href="udf.html#loadfunc">LoadFunc</a> class
+<br></br> <a href="udf.html#getInputFormat">getInputFormat</a> method
+<br></br> <a href="udf.html#getNext">getNext</a> method
+<br></br> <a href="udf.html#LoadCaster">LoadCaster</a> ubterface
+<br></br> <a href="udf.html#LoadMetadata">LoadMetadata</a> interface
+<br></br> <a href="udf.html#LoadPushDown">LoadPushDown</a> interface
+<br></br> <a href="udf.html#prepareToRead">prepareToRead</a> method
+<br></br> <a href="udf.html#pushProjection">pushProjection</a> method
+<br></br> <a href="udf.html#relativeToAbsolutePath">relativeToAbsolutePath</a> method
+<br></br> <a href="udf.html#setLocation">setLocation</a> method
+<br></br> <a href="udf.html#setUdfContextSignature">setUdfContextSignature</a> method
</p>
-<p><a href="udf.html#load-functions">Load Functions</a> (UDFs)</p>
+<p>Load Functions. <em>See</em> load/store functions</p>
-<p><a href="udf.html#load-store-functions">LoadMetadata</a></p>
-<p><a href="udf.html#load-store-functions">LoadPushDown</a></p>
+<p><a href="udf.html#LoadMetadata">LoadMetadata</a> interface</p>
+
+<p><a href="udf.html#LoadPushDown">LoadPushDown</a> interface</p>
<p>load/store functions
<br></br> <a href="func.html#load-store-functions">built in functions</a>
-<br></br> <a href="udf.html#load-store-functions">UDFs</a>
+<br></br> <a href="udf.html#load-store-functions">user defined functions (UDFs)</a>
</p>
<p><a href="start.html#execution-modes">local mode</a></p>
-<p><a href="func.html#log">LOG</a></p>
-<p><a href="func.html#log10">LOG10</a></p>
+<p><a href="func.html#log">LOG</a> function</p>
+
+<p><a href="func.html#log10">LOG10</a> function</p>
-<p> <a href="perf.html#LogicalExpressionSimplifier">LogicalExpressionSimplifier </a></p>
+<p><a href="test.html#logical-plan">logical execution plan</a></p>
-<p><a href="func.html#lower">LOWER</a></p>
+<p> <a href="perf.html#LogicalExpressionSimplifier">LogicalExpressionSimplifier</a> optimization rule</p>
+
+<p><a href="func.html#lower">LOWER</a> function</p>
<!-- ==== M ================================================================== -->
<p></p>
-<p id="m-index"><strong>M</strong> (<a href="#top">top</a>)</p>
+<p id="m-index"><strong>M</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
<p>macros
@@ -441,33 +543,36 @@
<br></br> <a href="cont.html#import-macros">importing macros</a>
</p>
-<p><a href="basic.html#mapreduce">MAPREDUCE</a> (operator)</p>
-
<p>MapReduce
-<br></br> <a href="test.html#mapreduce-job-ids">MapReduce job ids and Pig scripts</a>
+<br></br> <a href="test.html#mapreduce-job-ids">MapReduce job ids and Pig scripts</a>
<br></br> <a href="perf.html#Parallel">setting the number of reduce tasks</a>
</p>
+<p><a href="test.html#mapreduce-plan">mapreduce execution plan</a></p>
+
<p><a href="start.html#execution-modes">mapreduce mode</a></p>
+<p><a href="basic.html#mapreduce">MAPREDUCE</a> operator</p>
+
<p>maps (data type)
-<br></br> <a href="basic.html#map-schema">map schema</a>
+<br></br> <a href="basic.html#map-schema">and schemas</a>
<br></br> <a href="basic.html#schema-multi">schemas for multiple types</a>
<br></br> <a href="basic.html#map">syntax</a>
</p>
<p>matches. <em>See</em> pattern matching</p>
-<p><a href="func.html#max">MAX</a></p>
+<p><a href="func.html#math-functions">math functions</a></p>
-<p><a href="perf.html#memory-management">memory management</a></p>
+<p><a href="func.html#max">MAX</a> function</p>
-<p><a href="perf.html#merge-joins">merge joins</a></p>
+<p><a href="perf.html#memory-management">memory management</a>. <em>See also</em> batch mode</p>
+<p><a href="perf.html#MergeForEach">MergeForEach</a> optimization rule</p>
-<p><a href="perf.html#MergeForEach">MergeForEach</a></p>
+<p><a href="perf.html#merge-joins">merge joins</a></p>
-<p><a href="func.html#min">MIN</a></p>
+<p><a href="func.html#min">MIN</a> function</p>
<p><a href="basic.html#arithmetic">modulo operator</a> ( % )</p>
@@ -475,26 +580,31 @@
<!-- ==== N ================================================================== -->
<p></p>
-<p id="n-index"><strong>N</strong> (<a href="#top">top</a>)</p>
+<p id="n-index"><strong>N</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
<p>names (for fields, relations). <em>See</em> referencing.</p>
+<p><a href="basic.html#nested-block">nested blocks</a> (FOREACH operator) </p>
+
<p><a href="basic.html#boolops">NOT</a> (Boolean)</p>
+<p><a href="basic.html#null_operators">null operators</a></p>
+
<p>nulls
-<br></br> <a href="basic.html#nulls">and Pig Latin</a>
+<br></br> <a href="basic.html#nulls-constants">and constants</a>
+<br></br> <a href="perf.html#nulls">dropping before a join</a> (performance)
+<br></br> <a href="basic.html#nulls_join">and JOIN operator</a>
+<br></br> <a href="basic.html#nulls-load">and load functions</a>
<br></br> <a href="basic.html#nulls-ops-produce">operations that produce</a>
-<br></br> <a href="basic.html#null_operators">null operators</a>
+<br></br> <a href="basic.html#nulls">and Pig Latin</a>
</p>
-
<!-- ==== O ================================================================== -->
<p></p>
-<p id="o-index"><strong>O</strong> (<a href="#top">top</a>)</p>
+<p id="o-index"><strong>O</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
-<p> <a href="perf.html#OpLimitOptimizer">OpLimitOptimizer</a></p>
-
+<p> <a href="perf.html#OpLimitOptimizer">OpLimitOptimizer</a> optimization rule</p>
<p><a href="perf.html#optimization-rules">optimization rules</a>
<br></br> <a href="perf.html#ImplicitSplitInserter">ImplicitSplitInserter </a>
@@ -507,21 +617,23 @@
<br></br> <a href="perf.html#TypeCastInserter">TypeCastInserter</a>
</p>
-
<p><a href="basic.html#boolops">OR</a> (Boolean)</p>
-<p><a href="basic.html#order-by">ORDER BY</a></p>
-<p><a href="udf.html#outputfunctionschema">outputFunctionSchema</a></p>
-<p><a href="udf.html#outputschema">outputSchema</a></p>
+<p><a href="basic.html#order-by">ORDER BY</a> operator</p>
+
+<p><a href="udf.html#outputfunctionschema">outputFunctionSchema</a> Python decorator</p>
+
+<p><a href="udf.html#outputschema">outputSchema</a> Python decorator</p>
<!-- ==== P ================================================================== -->
<p></p>
-<p id="p-index"><strong>P</strong> (<a href="#top">top</a>)</p>
+<p id="p-index"><strong>P</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
+<p><a href="start.html#properties">-P</a> command line option</p>
<p>PARALLEL
-<br></br> <a href="perf.html#Use+the+Parallel+Features">and performance</a>
-<br></br> <a href="cmds.html#set">setting defalt_parallel</a>
+<br></br> <a href="perf.html#Parallel">and performance</a>
+<br></br> <a href="cmds.html#set">setting default_parallel</a>
</p>
<p><a href="cont.html#parameter-sub">parameter substitution</a></p>
@@ -529,63 +641,45 @@
<p>PARTITION BY
<br></br> <a href="basic.html#partition-by-cross">and CROSS</a>
<br></br> <a href="basic.html#partition-by-distinct">and DISTINCT</a>
-<br></br> <a href="basic.html#partitionby">and GROUP</a>
+<br></br> <a href="basic.html#partition-by-group">and GROUP</a>
<br></br> <a href="basic.html#partition-by-join-inner">and JOIN (inner)</a>
<br></br> <a href="basic.html#partition-by-join-inner">and JOIN (outer)</a>
</p>
<p><a href="basic.html#comparison">pattern matching</a></p>
-<p>Performance
-<br></br> <a href="perf.html#specialized-joins-performance">and merge joins</a>
-<br></br> <a href="perf.html#optimiztion-rules">optimization rules for</a>
-<br></br> <a href="perf.html#performance-enhancers">performance enhancers</a>
-</p>
-
<p><a href="test.html#penny">Penny</a> (monitoring and debugging)</p>
-<p><a href="test.html#pig-alias">pig.alias</a></p>
+<p>performance (writing efficient code)
+<br></br> <a href="perf.html#optimization-rules">optimization rules for</a>
+<br></br> <a href="perf.html#performance-enhancers">performance enhancers</a>
+<br></br> <em>See also</em> Pig Latin
+</p>
-<p><a href="test.html#pig-command-line">pig.command.line</a></p>
+<p><a href="test.html#physical-plan">physical execution plan</a></p>
-<p><a href="func.html#pigdump">PigDUMP</a></p>
+<p><a href="perf.html#memory-bags">pig.cachedbag.memusage</a> property</p>
-<p><a href="test.html#pig-hadoop-version">pig.hadoop.version</a></p>
+<p><a href="func.html#pigdump">PigDump</a> function</p>
<p><a href="udf.html#piggybank">Piggy Bank</a></p>
-<p><a href="test.html#pig-job-feature">pig.job.feature</a></p>
<p>Pig Latin
-<br></br> <a href="start.html#comments">comments</a>
-<br></br> <a href="start.html#pig-scripts">scripts</a>
-<br></br> <a href="start.html#pl-statements">statements</a>
+<br></br> <a href="cont.html#automated-pig-latin">automated generation of</a> (Python example)
+<br></br> <a href="start.html#pl-statements">Pig Latin statements</a>
+<br></br> <em>See also</em> performance (writing efficient code)
</p>
<p>Pig macros. <em>See</em> macros</p>
-<p><a href="cont.html#pig-object">pig.java object</a></p>
+<p><a href="cont.html#pig-object">pig.java</a> object</p>
-<p><a href="test.html#pig-input-dirs">pig.input.dirs</a></p>
+<p><a href="cont.html#pigstats">PigProgressNotificationListener</a> interface</p>
-<p><a href="test.html#pig-map-output-dirs">pig.map.output.dirs</a></p>
+<p><a href="cont.html#PigProgressNotificationListener">PigProgressNotificationListener.java</a> object</p>
-<p><a href="test.html#pig-parent-jobid">pig.parent.jobid</a></p>
-
-<p><a href="cont.html#pigstats">PigProgressNotificationListener interface</a></p>
-<p><a href="cont.html#PigProgressNotificationListener">PigProgressNotificationListener.java</a></p>
-
-<p><a href="test.html#pig-reduce-output-dirs">pig.reduce.output.dirs</a></p>
-
-<p><a href="cont.html#pig-run-api">PigRunner API</a></p>
-
-<p><a href="cont.html#pigserver">PigServer Interface</a></p>
-
-<p><a href="test.html#pig-script">pig.script</a></p>
-
-<p><a href="test.html#pig-script-features">pig.script.features</a></p>
-
-<p><a href="test.html#pig-script-id">pig.script.id</a></p>
+<p><a href="cont.html#pigrunner-api">PigRunner API</a>. <em>See also</em> PigStats class</p>
<p>Pig Scripts
<br></br> <a href="start.html#batch-mode">and batch mode</a>
@@ -596,14 +690,25 @@
<br></br> <a href="cmds.html#run">and run command</a>
</p>
-<p><a href="cont.html#pigstats">PigStats</a> (function)</p>
+<p><a href="cont.html#pigserver">PigServer</a> interface</p>
+
+<p><a href="cont.html#pigstats">PigStats</a> class
+<br></br> <a href="cont.html#EmbeddedPigStats">EmbeddedPigStats</a> class
+<br></br> <a href="cont.html#stats-messages">getAllErrorMessages</a> method
+<br></br> <a href="cont.html#stats-messages">getAllStats</a> method
+<br></br> <a href="cont.html#isembedded">isEmbedded</a> method
+<br></br> <a href="cont.html#SimplePigStats">SimplePigStats</a> class
+<br></br> <em>See also</em> PigRunner API
+</p>
+
+<p><a href="cont.html#pigstats-object">PigStats.java</a> object</p>
<p><a href="test.html#pig-statistics">Pig Statistics</a>
<br></br> <a href="test.html#pig-alias">pig.alias</a>
<br></br> <a href="test.html#pig-command-line">pig.command.line</a>
<br></br> <a href="test.html#pig-hadoop-version">pig.hadoop.version</a>
-<br></br> <a href="test.html#pig-job-feature">pig.job.feature</a>
<br></br> <a href="test.html#pig-input-dirs">pig.input.dirs</a>
+<br></br> <a href="test.html#pig-job-feature">pig.job.feature</a>
<br></br> <a href="test.html#pig-map-output-dirs">pig.map.output.dirs</a>
<br></br> <a href="test.html#pig-parent-jobid">pig.parent.jobid</a>
<br></br> <a href="test.html#pig-reduce-output-dirs">pig.reduce.output.dirs</a>
@@ -613,131 +718,156 @@
<br></br> <a href="test.html#pig-version">pig.version</a>
</p>
-<p><a href="func.html#binstorage">PigStats.java object</a></p>
-<p><a href="func.html#pigstorage">PigStorage</a> (function)</p>
+<p><a href="func.html#pigstorage">PigStorage</a> function</p>
<p><a href="start.html#tutorial">Pig tutorial</a></p>
<p>Pig types. <em>See</em> data types</p>
-<p><a href="test.html#pig-unit">PigUnit</a></p>
-
-<p><a href="test.html#pig-version">pig.version</a></p>
+<p><a href="test.html#pigunit">PigUnit</a></p>
<p><a href="basic.html#ref-field">positional notation</a></p>
-<p><a href="udf.html#loadfunc-abstract">prepareToRead</a></p>
-<p><a href="udf.html#storefunc-override">prepareToWrite</a></p>
+<p><a href="udf.html#prepareToRead">prepareToRead</a> method</p>
-<p><a href="basic.html#prexp">project-range expression</a></p>
+<p><a href="udf.html#prepareToWrite">prepareToWrite</a> method</p>
+
+<p>projection
+<br></br> <a href="basic.html#projection">example of</a>
+<br></br> <a href="perf.html#projection">and performance</a>
+</p>
+
+<p><a href="basic.html#prexp">project-range expressions</a></p>
<p>properties
-<br></br> <a href="start.html#properties">specifying Hadoop properties</a>
-<br></br> <a href="start.html#properties">specifying Pig properties</a>
+<br></br> <a href="start.html#hadoop-properties">specifying Hadoop properties</a>
+<br></br> <a href="start.html#pig-properties">specifying Pig properties</a>
</p>
-<p><a href="udf.html#load-store-functions">pushProjection</a></p>
-<p><a href="udf.html#storefunc-override">putNext</a></p>
-<p> <a href="udf.html#python-udfs">Python UDFs</a></p>
+<p> <a href="perf.html#PushDownExplodes">PushDownExplodes</a> optimization rule</p>
+
+<p><a href="udf.html#pushprojection">pushProjection</a> method</p>
+
+<p><a href="perf.html#PushUpFilters">PushUpFilters</a> optimization rule</p>
-<p> <a href="perf.html#PushDownExplodes">PushDownExplodes</a></p>
+<p><a href="udf.html#putNext">putNext</a> method</p>
+
+<p><a href="cont.html#embed-python">Python and embedded Pig</a></p>
+
+<p><a href="udf.html#python-udfs">Python UDFs</a>. <em>See also</em> UDFs</p>
-<p><a href="perf.html#PushUpFilters">PushUpFilters </a></p>
<!-- ==== Q ================================================================== -->
<p></p>
-<p id="q-index"><strong>Q</strong> (<a href="#top">top</a>)</p>
+<p id="q-index"><strong>Q</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
<p><a href="cmds.html#quit">quit</a> (command)</p>
<!-- ==== R ================================================================== -->
<p></p>
-<p id="r-index"><strong>R</strong> (<a href="#top">top</a>)</p>
+<p id="r-index"><strong>R</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
-<p><a href="func.html#random">RANDOM</a></p>
+<p><a href="func.html#random">RANDOM</a> function</p>
<p>referencing
<br></br> <a href="basic.html#ref-field">fields</a>
<br></br> <a href="basic.html#ref-field-complex">fields and complex types</a>
<br></br> <a href="basic.html#ref-relation">relations</a>
+<br></br> <em>See also</em> identifiers
</p>
-<p><a href="func.html#regex-extract">REGEX_EXTRACT</a></p>
-<p><a href="func.html#regex-extract-all">REGEX_EXTRACT_ALL</a></p>
-<p><a href="basic.html#register">REGISTER</a></p>
+<p><a href="func.html#regex-extract">REGEX_EXTRACT</a> function</p>
+
+<p><a href="func.html#regex-extract-all">REGEX_EXTRACT_ALL</a> function</p>
+
+<p><a href="basic.html#register">REGISTER</a> statement</p>
+
<p>regular expressions. <em>See</em> pattern matching</p>
<p>relations
-<br></br> <a href="basic.html#relations">and Pig Latin</a>
<br></br> <a href="basic.html#cast-relations">casting to scalars</a>
+<br></br> <a href="basic.html#relations">and Pig Latin</a>
<br></br> <a href="basic.html#ref-relation">referencing</a>
</p>
-<p><a href="udf.html#loadfunc-default">relativeToAbsolutePath</a></p>
-<p><a href="udf.html#storefunc-default">relToAbsPathForStoreLocation</a></p>
+<p><a href="udf.html#relativeToAbsolutePath">relativeToAbsolutePath</a> method</p>
-<p><a href="func.html#replace">REPLACE</a></p>
+<p><a href="udf.html#relToAbsPathForStoreLocation">relToAbsPathForStoreLocation</a> method</p>
-<p><a href="perf.html#replicated-joins">replicated joins</a></p>
+<p><a href="func.html#replace">REPLACE</a> function</p>
+<p><a href="perf.html#replicated-joins">replicated joins</a></p>
<p><a href="start.html#req">requirements</a> (for Pig)</p>
<p id="reserved-keywords"><a href="basic.html#reserved-keywords">reserved keywords</a></p>
-<p><a href="func.html#round">ROUND</a></p>
-
-<p><a href="cmds.html#run">run</a> (command)</p>
+<p><a href="func.html#round">ROUND</a> function</p>
+<p><a href="cmds.html#run">run</a> command</p>
+<p><a href="start.html#run">running Pig</a>
+<br></br> <a href="cmds.html#exec">exec</a> command
+<br></br> <a href="start.html#execution-modes">execution modes</a>
+<br></br> <a href="perf.html#Implicit-Dependencies">execution order</a>
+<br></br> <a href="test.html#execution-plans">execution plans</a>
+<br></br> <a href="perf.html#multi-query-execution">multi-query execution</a>
+<br></br> <a href="cmds.html#run">run</a> command
+</p>
<!-- ==== S ================================================================== -->
<p></p>
-<p id="s-index"><strong>S</strong> (<a href="#top">top</a>)</p>
+<p id="s-index"><strong>S</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
-<p><a href="basic.html#sample">SAMPLE</a></p>
+<p><a href="basic.html#sample">SAMPLE</a> operator</p>
-<p><a href="udf.html#schemafunction">schemaFunction</a></p>
+<p><a href="udf.html#schemafunction">schemaFunction</a> Python decorator</p>
<p>schemas
<br></br> <a href="basic.html#schema-complex">for complex data types</a> (tuples, bags, maps)
<br></br> <a href="udf.html#decorators">and decorators</a> (Python UDFs)
+<br></br> <a href="basic.html#schemaforeach">and FOREACH</a>
+<br></br> <a href="basic.html#schema-load">and LOAD, STREAM</a>
+<br></br> <a href="basic.html#onschema">ONSCHEMA clause</a> (UNION operator)
<br></br> <a href="basic.html#schemas">and Pig Latin</a>
<br></br> <a href="udf.html#schema-return-types">and return types</a> (JavaScript UDFs)
<br></br> <a href="basic.html#schema-simple">for simple data types</a> (int, long, float, double, chararray, bytearray)
<br></br> <a href="basic.html#unknown-schema">unknown (null) schemas</a>
</p>
+<p><a href="cmds.html#set">set</a> command</p>
+
+<p><a href="udf.html#setLocation">setLocation</a> method</p>
+<p><a href="udf.html#setStoreFuncUDFContextSignature">setStoreFuncUDFContextSignature</a> method</p>
-<p><a href="cmds.html#set">set</a></p>
+<p><a href="udf.html#setStoreLocation">setStoreLocation</a> method</p>
-<p><a href="udf.html#loadfunc-abstract">setLocation</a></p>
-<p><a href="udf.html#storefunc-default">setStoreFuncUDFContextSignature</a></p>
-<p><a href="udf.html#storefunc-override">setStoreLocation</a></p>
-<p><a href="udf.html#loadfunc-default">setUdfContextSignature</a></p>
+<p><a href="udf.html#setUdfContextSignature">setUdfContextSignature</a> method</p>
-<p><a href="cmds.html#sh">sh</a></p>
+<p><a href="cmds.html#sh">sh</a> command</p>
<p><a href="cmds.html#shell-cmds">shell commands</a></p>
+<p><a href="basic.html#ship-about">ship</a> (streaming). <em>See also</em> autoship</p>
+
<p>sign operators
-<br></br> <a href="basic.html#sign">positive</a> ( + )
<br></br> <a href="basic.html#sign">negative</a> ( - )
+<br></br> <a href="basic.html#sign">positive</a> ( + )
</p>
-<p><a href="perf.html#skewed-joins">skewed joins</a></p>
+<p><a href="cont.html#SimplePigStats">SimplePigStats</a> class</p>
-<p><a href="func.html#pigstats">SimplePigStats</a></p>
+<p><a href="func.html#sin">SIN</a> function</p>
-<p><a href="func.html#sin">SIN</a></p>
-<p><a href="func.html#sinh">SINH</a></p>
-<p><a href="func.html#size">SIZE</a></p>
+<p><a href="func.html#sinh">SINH</a> function</p>
-<p>software requirements. <em>See</em> requirements</p>
+<p><a href="func.html#size">SIZE</a> function</p>
-<p><a href="basic.html#split">SPLIT</a></p>
+<p><a href="perf.html#skewed-joins">skewed joins</a></p>
+
+<p>software requirements. <em>See</em> requirements</p>
<p>specialized joins
<br></br> <a href="perf.html#merge-joins">merge joins</a>
@@ -746,99 +876,121 @@
<br></br> <a href="perf.html#skewed-joins">skewed joins</a>
</p>
+<p><a href="basic.html#split">SPLIT</a> operator</p>
+<p><a href="perf.html#splits">splits</a> (implicit, explicit)</p>
+<p><a href="func.html#sqrt">SQRT</a> function</p>
-<p><a href="func.html#sqrt">SQRT</a></p>
<p><a href="basic.html#sexp">star expression</a> ( * )</p>
+
<p><a href="start.html#pl-statements">statements</a> (Pig Latin)</p>
-<p><a href="basic.html#store">STORE</a> <em>See also </em> Store vs. Dump</p>
+<p>statistics. <em>See</em> Pig statistics</p>
+<p><a href="basic.html#store">STORE</a> operator. <em>See also </em> Store vs. Dump</p>
-<p><a href="udf.html#storefunc">StoreFunc</a></p>
+<p>Store functions. <em>See</em> load/store functions</p>
-<p><a href="udf.html#store-functions">Store Functions</a> (UDFs)</p>
+<p><a href="udf.html#storefunc">StoreFunc</a> class
+<br></br> <a href="udf.html#checkSchema">checkSchema</a> method
+<br></br> <a href="udf.html#getOutputFormat">getOutputFormat</a> method
+<br></br> <a href="udf.html#prepareToWrite">prepareToWrite</a> method
+<br></br> <a href="udf.html#putNext">putNext</a> method
+<br></br> <a href="udf.html#relToAbsPathForStoreLocation">relToAbsPathForStoreLocation</a> method
+<br></br> <a href="udf.html#setStoreFuncUDFContextSignature">setStoreFuncUDFContextSignature</a> method
+<br></br> <a href="udf.html#setStoreLocation">setStoreLocation</a> method
+<br></br> <a href="udf.html#StoreMetadata">StoreMetadata</a> interface
+</p>
-<p><a href="udf.html#storemetadata">StoreMetadata</a></p>
+<p><a href="udf.html#storemetadata">StoreMetadata</a> interface</p>
<p><a href="perf.html#store-dump">Store vs. Dump</a> </p>
-<p><a href="basic.html#stream">STREAM</a></p>
+<p><a href="basic.html#stream">STREAM</a> operator</p>
-<p> <a href="perf.html#StreamOptimizer">StreamOptimizer</a></p>
+<p><a href="basic.html#define-udfs">streaming</a> (DEFINE operator)</p>
+<p> <a href="perf.html#StreamOptimizer">StreamOptimizer</a> optimization rule</p>
+<p><a href="func.html#string-functions">string functions</a></p>
-<p><a href="func.html#strsplit">STRSPLIT</a></p>
+<p><a href="func.html#strsplit">STRSPLIT</a> function</p>
-<p><a href="func.html#string-functions">string functions</a></p>
-<p><a href="func.html#substring">SUBSTRING</a></p>
-<p><a href="func.html#sum">SUM</a></p>
+<p><a href="func.html#substring">SUBSTRING</a> function</p>
+
+<p><a href="func.html#sum">SUM</a> function</p>
<!-- ==== T ================================================================== -->
<p></p>
-<p id="t-index"><strong>T</strong> (<a href="#top">top</a>)</p>
+<p id="t-index"><strong>T</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
+
+<p><a href="func.html#tan">TAN</a> function</p>
+
+<p><a href="func.html#tanh">TANH</a> function</p>
-<p><a href="func.html#tan">TAN</a></p>
-<p><a href="func.html#tanh">TANH</a></p>
-<p><a href="func.html#textloader">TextLoader</a></p>
+<p><a href="func.html#textloader">TextLoader</a> function</p>
-<p><a href="func.html#tobag">TOBAG</a></p>
+<p><a href="func.html#tobag">TOBAG</a> function</p>
-<p><a href="func.html#tokenize">TOKENIZE</a></p>
+<p><a href="func.html#tokenize">TOKENIZE</a> function</p>
-<p><a href="func.html#topx">TOP</a></p>
+<p><a href="func.html#topx">TOP</a> function</p>
-<p><a href="func.html#totuple">TOTUPLE</a></p>
+<p><a href="func.html#totuple">TOTUPLE</a> function</p>
-<p><a href="func.html#trim">TRIM</a></p>
+<p><a href="func.html#trim">TRIM</a> function</p>
-<p><a href="basic.html#texp">tuple expression</a></p>
+<p><a href="basic.html#texp">tuple expressions</a></p>
+
+<p><a href="func.html#bag-tuple-functions">tuple functions</a></p>
<p>tuples
<br></br> <a href="basic.html#relations">and relations</a>
+<br></br> <a href="basic.html#tuple-schema">and schemas</a>
<br></br> <a href="basic.html#tuple">syntax</a>
</p>
-<p> <a href="perf.html#TypeCastInserter">TypeCastInserter</a></p>
-
+<p> <a href="perf.html#TypeCastInserter">TypeCastInserter</a> optimiztion rule</p>
<p>type conversions. <em>See</em> casting types, types tables</p>
+<p><a href="perf.html#types">types and performance</a></p>
+
<p>types tables
<br></br> <a href="basic.html#types-table-add">for addition, subtraction</a>
-<br></br> <a href="basic.html#types-table-mult">for multiplication, division</a>
<br></br> <a href="basic.html#types-table-equal">for equal, not equal</a>
<br></br> <a href="basic.html#types-table-matches">for matches</a>
+<br></br> <a href="basic.html#types-table-mult">for multiplication, division</a>
<br></br> <a href="basic.html#types-table-negative">for negative (negation)</a>
<br></br> <a href="basic.html#types-table-nulls">for nulls</a>
<br></br> <em>See also</em> casting types
</p>
-<p><a href="start.html#tutorial">tutorial</a></p>
+
+<p><a href="start.html#tutorial">tutorial</a> (for Pig)</p>
<!-- ==== U ================================================================== -->
<p></p>
-<p id="u-index"><strong>U</strong> (<a href="#top">top</a>)</p>
+<p id="u-index"><strong>U</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
-<p><a href="func.html#ucfirst">UCFIRST</a></p>
+<p><a href="func.html#ucfirst">UCFIRST</a> function</p>
<p>UDFs
-<br></br> <a href="udf.html#accumulator-interface">accumulator interface</a>
-<br></br> <a href="udf.html#algebraic-interface">algebraic interface</a>
-<br></br> <a href="udf.html#udf-configurations">configurations, passing to </a>
-<br></br> <a href="udf.html#function-instantiation">function instantiation</a>
+<br></br> <a href="udf.html#function-instantiation">and function instantiation</a>
+<br></br> <a href="udf.html#udf-monitoring">and monitoring</a>
+<br></br> <a href="udf.html#udf-configurations">passing configurations to </a>
+<br></br> <a href="perf.html#accumulator-interface">and performance</a> (Accumulator Interface)
+<br></br> <a href="perf.html#algebraic-interface">and performance</a> (Algebraic Interface)
+<br></br> <a href="udf.html#piggybank">Piggy Bank</a> (repository)
<br></br> <a href="udf.html#udf-interfaces">UDF interfaces</a>
-<br></br> <a href="udf.html#load-store-functions">load/store functions</a>
-<br></br> <a href="udf.html#udf-monitoring">monitoring</a>
-<br></br> <a href="udf.html#piggybank">Piggy Bank</a>
-<br></br> <a href="udf.html#udf-configurations">UDFContext class</a>
+<br></br> <em>See also</em> Java UDFs, JavaScript UDFs, Python UDFs
</p>
-<p><a href="basic.html#union">UNION</a></p>
-<p><a href="func.html#upper">UPPER</a></p>
+<p><a href="basic.html#union">UNION</a> operator</p>
+
+<p><a href="func.html#upper">UPPER</a> function</p>
<p>user defined functions. <em>See</em> UDFs</p>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/start.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/start.xml?rev=1128482&r1=1128481&r2=1128482&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/start.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/start.xml Fri May 27 22:09:18 2011
@@ -350,7 +350,7 @@ DUMP B;
<!-- ++++++++++++++++++++++++++++++++++ -->
<section id="data-store">
- <title>Storing Intermediate Data</title>
+ <title>Storing Intermediate Results</title>
<p>Pig stores the intermediate data generated between MapReduce jobs in a temporary location on HDFS.
This location must already exist on HDFS prior to use.
@@ -359,7 +359,7 @@ DUMP B;
</section>
<section id="data-results">
- <title>Storing Results</title>
+ <title>Storing Final Results</title>
<p>Use the <a href="basic.html#STORE">STORE</a> operator and the <a href="udf.html#load-store-functions">load/store functions</a>
to write results to the file system (PigStorage is the default store function). </p>
<p><strong>Note:</strong> During the testing/debugging phase of your implementation, you can use DUMP to display results to your terminal screen.
@@ -394,7 +394,7 @@ However, in a production environment you
<title>Pig Properties</title>
<p>Pig supports a number of Java properties that you can use to customize Pig behavior. You can retrieve a list of the properties using the <a href="cmds.html#help">help properties</a> command. All of these properties are optional; none are required. </p>
<p></p>
-<p>To specify Pig properties use one of these mechanisms:</p>
+<p id="pig-properties">To specify Pig properties use one of these mechanisms:</p>
<ul>
<li>The pig.properties file (add the directory that contains the pig.properties file to the classpath)</li>
<li>The -D command line option and a Pig property (pig -Dpig.tmpfilecompression=true)</li>
@@ -403,7 +403,8 @@ However, in a production environment you
</ul>
<p><strong>Note:</strong> The properties file uses standard Java property file format.</p>
<p>The following precedence order is supported: pig.properties > -D Pig property > -P properties file > set command. This means that if the same property is provided using the âD command line option as well as the âP command line option and a properties file, the value of the property in the properties file will take precedence.</p>
-<p>To specify Hadoop properties you can use the same mechanisms:</p>
+
+<p id="hadoop-properties">To specify Hadoop properties you can use the same mechanisms:</p>
<ul>
<li>The hadoop-site.xml file (add the directory that contains the hadoop-site.xml file to the classpath)</li>
<li>The -D command line option and a Hadoop property (pig âDmapreduce.task.profile=true) </li>
@@ -413,7 +414,7 @@ However, in a production environment you
<p></p>
<p>The same precedence holds: hadoop-site.xml > -D Hadoop property > -P properties_file > set command.</p>
<p>Hadoop properties are not interpreted by Pig but are passed directly to Hadoop. Any Hadoop property can be passed this way. </p>
-<p>All properties that Pig collects, including Hadoop properties, are available to any UDF via the UDFContext. To get access to the properties, you can call the getJobConf method.</p>
+<p>All properties that Pig collects, including Hadoop properties, are available to any UDF via the UDFContext object. To get access to the properties, you can call the getJobConf method.</p>
</section>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/test.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/test.xml?rev=1128482&r1=1128481&r2=1128482&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/test.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/test.xml Fri May 27 22:09:18 2011
@@ -273,20 +273,20 @@ DUMP B;
</tr>
</table></section>
- <section>
+ <section id="execution-plans">
<title>Usage</title>
<p>Use the EXPLAIN operator to review the logical, physical, and map reduce execution plans that are used to compute the specified relationship. </p>
<p>If no script is given:</p>
<ul>
- <li>
- <p>The logical plan shows a pipeline of operators to be executed to build the relation. Type checking and backend-independent optimizations (such as applying filters early on) also apply.</p>
+ <li id="logical-plan">
+ <p >The logical plan shows a pipeline of operators to be executed to build the relation. Type checking and backend-independent optimizations (such as applying filters early on) also apply.</p>
</li>
- <li>
- <p>The physical plan shows how the logical operators are translated to backend-specific physical operators. Some backend optimizations also apply.</p>
+ <li id="physical-plan">
+ <p >The physical plan shows how the logical operators are translated to backend-specific physical operators. Some backend optimizations also apply.</p>
</li>
- <li>
- <p>The map reduce plan shows how the physical operators are grouped into map reduce jobs.</p>
+ <li id="mapreduce-plan">
+ <p >The mapreduce plan shows how the physical operators are grouped into map reduce jobs.</p>
</li>
</ul>
<p></p>
@@ -381,10 +381,10 @@ Local Rearrange[tuple]{chararray}(false)
<p>Use the ILLUSTRATE operator to review how data is transformed through a sequence of Pig Latin statements.
ILLUSTRATE allows you to test your programs on small datasets and get faster turnaround times. </p>
-<p>ILLUSTRATE is based on an example generator
+<p id="example-generator">ILLUSTRATE is based on an example generator
(see <a href="http://research.yahoo.com/files/paper_5.pdf">Generating Example Data for Dataflow Programs</a>).
-The algorithm works by retrieving a small sample of the input data and then propagating this data through the pipeline. However, some operators, such as JOIN or FILTER, can eliminate tuples from the data - and this could result in no data following through the pipeline. To address this issue, the algorithm will automatically generate example data, in near real-time. Thus, you might see data propagating through the pipeline that was not found in the original input data, but this data changes nothing and ensures that you will be able to examine the semantics of your Pig Latin statements.</p>
+The algorithm works by retrieving a small sample of the input data and then propagating this data through the pipeline. However, some operators, such as JOIN and FILTER, can eliminate tuples from the data - and this could result in no data following through the pipeline. To address this issue, the algorithm will automatically generate example data, in near real-time. Thus, you might see data propagating through the pipeline that was not found in the original input data, but this data changes nothing and ensures that you will be able to examine the semantics of your Pig Latin statements.</p>
<p>As shown in the examples below, you can use ILLUSTRATE to review a relation or an entire Pig script.</p>
</section>