You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by ol...@apache.org on 2010/10/12 03:03:12 UTC
svn commit: r1021560 - in
/pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs:
cookbook.xml piglatin_ref1.xml piglatin_ref2.xml udf.xml
Author: olga
Date: Tue Oct 12 01:03:12 2010
New Revision: 1021560
URL: http://svn.apache.org/viewvc?rev=1021560&view=rev
Log:
PIG-1600: Docs update (chandec via olgan)
Modified:
pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/cookbook.xml
pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref1.xml
pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref2.xml
pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/udf.xml
Modified: pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/cookbook.xml
URL: http://svn.apache.org/viewvc/pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/cookbook.xml?rev=1021560&r1=1021559&r2=1021560&view=diff
==============================================================================
--- pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/cookbook.xml (original)
+++ pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/cookbook.xml Tue Oct 12 01:03:12 2010
@@ -204,59 +204,64 @@ C = join A1 by t, B1 by x;
<section>
<title>Take Advantage of Join Optimizations</title>
-
-<section>
-<title>Regular Join Optimizations</title>
+<p><strong>Regular Join Optimizations</strong></p>
<p>Optimization for regular joins ensures that the last table in the join is not brought into memory but streamed through instead. Optimization reduces the amount of memory used which means you can avoid spilling the data and also should be able to scale your query to larger data volumes. </p>
<p>To take advantage of this optimization, make sure that the table with the largest number of tuples per key is the last table in your query.
In some of our tests we saw 10x performance improvement as the result of this optimization.</p>
-
<source>
small = load 'small_file' as (t, u, v);
large = load 'large_file' as (x, y, z);
C = join small by t, large by x;
</source>
-</section>
-
-<section>
-<title>Specialized Join Optimizations</title>
+<p><strong>Specialized Join Optimizations</strong></p>
<p>Optimization can also be achieved using fragment replicate joins, skewed joins, and merge joins.
For more information see <a href="piglatin_ref1.html#Specialized+Joins">Specialized Joins</a>.</p>
-</section>
</section>
<section>
-<title>Use the PARALLEL Clause</title>
+<title>Use the Parallel Features</title>
-<p>Use the PARALLEL clause to increase the parallelism of a job:</p>
-<ul>
-<li>PARALLEL sets the number of reduce tasks for the MapReduce jobs generated by Pig. The default value is 1 (one reduce task).</li>
-<li>PARALLEL only affects the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block. </li>
-<li>If you donât specify PARALLEL, you still get the same map parallelism but only one reduce task.</li>
-</ul>
-<p></p>
-<p>As noted, the default value for PARALLEL is 1 (one reduce task). However, the number of reducers you need for a particular construct in Pig that forms a MapReduce boundary depends entirely on (1) your data and the number of intermediate keys you are generating in your mappers and (2) the partitioner and distribution of map (combiner) output keys. In the best cases we have seen that a reducer processing about 500 MB of data behaves efficiently.</p>
+<p>You can set the number of reduce tasks for the MapReduce jobs generated by Pig using two parallel features.
+(The parallel features only affect the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block.)</p>
+
+<p><strong>You Set the Number of Reducers</strong></p>
+<p>Use the <a href="piglatin_ref2.html#set">set default parallel</a> command to set the number of reducers at the script level.</p>
-<p>You can include the PARALLEL clause with any operator that starts a reduce phase (see the example below). This includes
+<p>Alternatively, use the PARALLEL clause to set the number of reducers at the operator level.
+(In a script, the value set via the PARALLEL clause will override any value set via "set default parallel.")
+You can include the PARALLEL clause with any operator that starts a reduce phase:
<a href="piglatin_ref2.html#COGROUP">COGROUP</a>,
<a href="piglatin_ref2.html#CROSS">CROSS</a>,
<a href="piglatin_ref2.html#DISTINCT">DISTINCT</a>,
<a href="piglatin_ref2.html#GROUP">GROUP</a>,
<a href="piglatin_ref2.html#JOIN+%28inner%29">JOIN (inner)</a>,
<a href="piglatin_ref2.html#JOIN+%28outer%29">JOIN (outer)</a>, and
-<a href="piglatin_ref2.html#ORDER">ORDER</a>.
+<a href="piglatin_ref2.html#ORDER+BY">ORDER BY</a>.
</p>
-
-<p>You can also set the value of PARALLEL for all Pig scripts using the <a href="piglatin_ref2.html#set">set default parallel</a> command.</p>
+<p>The number of reducers you need for a particular construct in Pig that forms a MapReduce boundary depends entirely on (1) your data and the number of intermediate keys you are generating in your mappers and (2) the partitioner and distribution of map (combiner) output keys. In the best cases we have seen that a reducer processing about 1 GB of data behaves efficiently.</p>
+
+<p><strong>Let Pig Set the Number of Reducers</strong></p>
+<p>If neither "set default parallel" nor the PARALLEL clause are used, Pig sets the number of reducers using a heuristic based on the size of the input data. You can set the values for these properties:</p>
+<ul>
+ <li>pig.exec.reducers.bytes.per.reducer - Defines the number of input bytes per reduce; default value is 1000*1000*1000 (1GB).</li>
+ <li>pig.exec.reducers.max - Defines the upper bound on the number of reducers; default is 999. </li>
+</ul>
+<p></p>
+
+<p>The formula, shown below, is very simple and will improve over time. The computed value takes all inputs within the script into account and applies the computed value to all the jobs within Pig script.</p>
+
+<p><code>#reducers = MIN (pig.exec.reducers.max, total input size (in bytes) / bytes per reducer) </code></p>
+
+<p><strong>Examples</strong></p>
<p>In this example PARALLEL is used with the GROUP operator. </p>
<source>
A = LOAD 'myfile' AS (t, u, v);
B = GROUP A BY t PARALLEL 18;
-.....
+...
</source>
<p>In this example all the MapReduce jobs that get launched use 20 reducers.</p>
@@ -273,7 +278,6 @@ STORE D INTO âmysortedcountâ U
<section>
<title>Use the LIMIT Operator</title>
-
<p>Often you are not interested in the entire output but rather a sample or top results. In such cases, using LIMIT can yield a much better performance as we push the limit as high as possible to minimize the amount of data travelling through the pipeline. </p>
<p>Sample:
</p>
@@ -316,7 +320,56 @@ B = foreach A generate u;
C = distinct B;
dump C;
</source>
+</section>
+
+<section>
+<title>Compress the Results of Intermediate Jobs</title>
+<p>If your Pig script generates a sequence of MapReduce jobs, you can compress the output of the intermediate jobs using LZO compression. (Use the <a href="piglatin_ref2.html#EXPLAIN">EXPLAIN</a> operator to determine if your script produces multiple MapReduce Jobs.)</p>
+
+<p>By doing this, you will save HDFS space used to store the intermediate data used by PIG and potentially improve query execution speed. In general, the more intermediate data that is generated, the more benefits in storage and speed that result.</p>
+<p>You can set the value for these properties:</p>
+<ul>
+ <li>pig.tmpfilecompression - Determines if the temporary files should be compressed or not (set to false by default).</li>
+ <li>pig.tmpfilecompression.codec - Specifies which compression codec to use. Currently, Pig accepts "gz" and "lzo" as possible values. However, because LZO is under GPL license (and disabled by default) you will need to configure your cluster to use the LZO codec to take advantage of this feature. For details, see http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ.</li>
+</ul>
+<p></p>
+
+<p>On the non-trivial queries (one ran longer than a couple of minutes) we saw significant improvements both in terms of query latency and space usage. For some queries we saw up to 96% disk saving and up to 4x query speed up. Of course, the performance characteristics are very much query and data dependent and testing needs to be done to determine gains. We did not see any slowdown in the tests we peformed which means that you are at least saving on space while using compression.</p>
+
+<p>With gzip we saw a better compression (96-99%) but at a cost of 4% slowdown. Thus, we don't recommend using gzip. </p>
+
+<p><strong>Example</strong></p>
+<source>
+-- launch Pig script using lzo compression
+
+java -cp $PIG_HOME/pig.jar
+-Djava.library.path=<path to the lzo library>
+-Dpig.tmpfilecompression=true
+-Dpig.tmpfilecompression.codec=lzo org.apache.pig.Main myscript.pig
+</source>
+</section>
+
+<section>
+<title>Combine Small Input Files</title>
+<p>Processing input (either user input or intermediate input) from multiple small files can be inefficient because a separate map has to be created for each file. Pig can now combined small files so that they are processed as a single map.</p>
+
+<p>You can set the values for these properties:</p>
+
+<ul>
+<li>pig.maxCombinedSplitSize â Specifies the size, in bytes, of data to be processed by a single map. Smaller files are combined untill this size is reached. </li>
+<li>pig.splitCombination â Turns combine split files on or off (set to âtrueâ by default).</li>
+</ul>
+<p></p>
+
+<p>This feature works with <a href="piglatin_ref2.html#PigStorage">PigStorage</a>. However, if you are using a custom loader, please note the following:</p>
+
+<ul>
+<li>If your loader implementation makes use of the PigSplit object passed through the prepareToRead method, then you may need to rebuild the loader since the definition of PigSplit has been modified. </li>
+<li>The loader must be stateless across the invocations to the prepareToRead method. That is, the method should reset any internal states that are not affected by the RecordReader argument.</li>
+<li>If a loader implements IndexableLoadFunc, or implements OrderedLoadFunc and CollectableLoadFunc, its input splits won't be subject to possible combinations.</li>
+</ul>
+<p></p>
</section>
</section>
</body>
Modified: pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref1.xml
URL: http://svn.apache.org/viewvc/pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref1.xml?rev=1021560&r1=1021559&r2=1021560&view=diff
==============================================================================
--- pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref1.xml (original)
+++ pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref1.xml Tue Oct 12 01:03:12 2010
@@ -54,6 +54,7 @@
</li>
</ol>
+<!-- ++++++++++++++++++++++++++++++++++ -->
<section>
<title>Running Pig Latin </title>
@@ -128,7 +129,48 @@ DUMP B;
<p> </p>
<p>See <a href="#Multi-Query+Execution">Multi-Query Execution</a> for more information on how Pig Latin statements are processed.</p>
</section>
+
+ <!-- ++++++++++++++++++++++++++++++++++ -->
+ <section>
+ <title>Pig Latin Scripts</title>
+ <p>See the following:</p>
+ <ul>
+ <li><a href="tutorial.html#Running+the+Pig+Scripts+in+Local+Mode">Running the Pig Scripts in Local Mode</a></li>
+ <li><a href="tutorial.html#Running+the+Pig+Scripts+in+Mapreduce+Mode">Running the Pig Scripts in MapReduce Mode</a></li>
+ <li><a href="#Multi-Query+Execution">Multi-Query Execution</a></li>
+ </ul>
+ <p></p>
+ <p>Pig supports running scripts that are stored in HDFS, Amazon S3, or other distributed file systems (also see <a href="piglatin_ref2.html#REGISTER">REGISTER</a> for information about Jar files). The script's full location URI is required. For example, to run a Pig script on HDFS, do the following:</p>
+<source>
+java -cp pig.jar org.apache.pig.Main hdfs://nn.mydomain.com:9020/myscripts/script.pig
+</source>
+ </section>
+
+ <!-- ++++++++++++++++++++++++++++++++++ -->
+ <section>
+ <title>Using Comments in Pig Latin Scripts</title>
+ <p>If you place Pig Latin statements in a script, the script can include comments. </p>
+ <ol>
+ <li>
+ <p>For multi-line comments use /* â¦. */</p>
+ </li>
+ <li>
+ <p>For single line comments use --</p>
+ </li>
+ </ol>
+<source>
+/* myscript.pig
+My script includes three simple Pig Latin Statements.
+*/
+
+A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float); -- load statement
+B = FOREACH A GENERATE name; -- foreach statement
+DUMP B; --dump statement
+</source>
+</section>
+
+<!-- ++++++++++++++++++++++++++++++++++ -->
<section>
<title>Retrieving Pig Latin Results</title>
<p>Pig Latin includes operators you can use to retrieve the results of your Pig Latin statements: </p>
@@ -142,7 +184,16 @@ DUMP B;
</ol>
</section>
+ <!-- ++++++++++++++++++++++++++++++++++ -->
+ <section>
+ <title>Storing Intermediate Data</title>
+ <p>Pig stores the intermediate data generated between MapReduce jobs in a temporary location on HDFS.
+ This location must already exist on HDFS prior to use.
+ This location can be configured using the pig.temp.dir property. The property's default value is "/tmp" which is the same
+ as the hardcoded location in Pig 0.7.0 and earlier versions. </p>
+ </section>
+ <!-- ++++++++++++++++++++++++++++++++++ -->
<section>
<title>Debugging Pig Latin</title>
<p>Pig Latin includes operators that can help you debug your Pig Latin statements:</p>
@@ -157,8 +208,17 @@ DUMP B;
<p>Use the ILLUSTRATE operator to view the step-by-step execution of a series of statements.</p>
</li>
</ol>
+ <p></p>
+ <p>Complex Pig scripts often generate many MapReduce jobs. To help you debug a script, Pig prints a summary of the execution that shows which relations (aliases) are mapped to each MapReduce job. </p>
+<source>
+JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
+job_201004271216_12712 1 1 3 3 3 12 12 12 B,C GROUP_BY,COMBINER
+job_201004271216_12713 1 1 3 3 3 12 12 12 D SAMPLER
+job_201004271216_12714 1 1 3 3 3 12 12 12 D ORDER_BY,COMBINER hdfs://wilbur20.labs.corp.sp1.yahoo.com:9020/tmp/temp743703298/tmp-2019944040,
+</source>
</section>
-
+
+ <!-- ++++++++++++++++++++++++++++++++++ -->
<section>
<title>Working with Data</title>
<p>Pig Latin allows you to work with data in many ways. In general, and as a starting point:</p>
@@ -175,28 +235,8 @@ DUMP B;
</ol>
</section>
- <section>
- <title>Using Comments in Scripts</title>
- <p>If you place Pig Latin statements in a script, the script can include comments. </p>
- <ol>
- <li>
- <p>For multi-line comments use /* â¦. */</p>
- </li>
- <li>
- <p>For single line comments use --</p>
- </li>
- </ol>
-<source>
-/* myscript.pig
-My script includes three simple Pig Latin Statements.
-*/
-
-A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float); -- load statement
-B = FOREACH A GENERATE name; -- foreach statement
-DUMP B; --dump statement
-</source>
-</section>
+ <!-- ++++++++++++++++++++++++++++++++++ -->
<section>
<title>Case Sensitivity</title>
<p>The names (aliases) of relations and fields are case sensitive. The names of Pig Latin functions are case sensitive.
@@ -228,10 +268,39 @@ grunt> B = GROUP A BY f1;
grunt> C = FOREACH B GENERATE COUNT ($0);
grunt> DUMP C;
</source>
-</section>
+</section>
+
</section>
<!-- END PIG LATIN STATEMENTS -->
+
+
+<!-- ================================================================== -->
+<!-- DYNAMIC INVOKERS -->
+<section>
+<title>Dynamic Invokers</title>
+
+<p>Often you may need to use a simple function that is already provided by standard Java libraries, but for which a UDF has not been written. Dynamic Invokers allow you to refer to Java functions without having to wrap them in custom Pig UDFs, at the cost of doing some Java reflection on every function call.
+</p>
+
+<source>
+...
+DEFINE UrlDecode InvokeForString('java.net.URLDecoder.decode', 'String String');
+encoded_strings = LOAD 'encoded_strings.txt' as (encoded:chararray);
+decoded_strings = FOREACH encoded_strings GENERATE UrlDecode(encoded, 'UTF-8');
+...
+</source>
+
+<p>Currently, Dynamic Invokers can be used for any static function that accepts no arguments or some combination of strings, ints, longs, doubles, floats, or arrays of same, and returns a string, an int, a long, a double, or a float. Primitives only for the numbers, no capital-letter numeric classes as arguments. Depending on the return type, a specific kind of Invoker must be used: InvokeForString, InvokeForInt, InvokeForLong, InvokeForDouble, or InvokeForFloat.
+</p>
+
+<p>The <a href="piglatin_ref2.html#define">DEFINE</a> statement is used to bind a keyword to a Java method, as above. The first argument to the InvokeFor* constructor is the full path to the desired method. The second argument is a space-delimited ordered list of the classes of the method arguments. This can be omitted or an empty string if the method takes no arguments. Valid class names are string, long, float, double, and int. Invokers can also work with array arguments, represented in Pig as DataBags of single-tuple elements. Simply refer to string[], for example. Class names are not case-sensitive. </p>
+
+<p>The ability to use invokers on methods that take array arguments makes methods like those in org.apache.commons.math.stat.StatUtils available (for processing the results of grouping your datasets, for example). This is helpful, but a word of caution: the resulting UDF will not be optimized for Hadoop, and the very significant benefits one gains from implementing the Algebraic and Accumulative interfaces are lost here. Be careful if you use invokers this way.</p>
+
+</section>
+<!-- END DYNAMIC INVOKERS -->
+
<!-- ================================================================== -->
<!-- MEMORY MANAGEMENT -->
<section>
@@ -528,6 +597,7 @@ pig -optimizer_off [opt_rule | all ]
<p>Note that some rules are mandatory and cannot be turned off.</p>
+<!-- +++++++++++++++++++++++++++++++ -->
<section>
<title>ImplicitSplitInserter</title>
<p>Status: Mandatory</p>
@@ -545,35 +615,81 @@ C = COGROUP A BY $0, B BY $0;
</source>
</section>
+
+<!-- +++++++++++++++++++++++++++++++ -->
<section>
-<title>TypeCastInserter</title>
-<p>Status: Mandatory</p>
-<p>
-If you specify a <a href="piglatin_ref2.html#Schemas">schema</a> with the
-<a href="piglatin_ref2.html#LOAD">LOAD</a> statement, the optimizer will perform a pre-fix projection of the columns
-and <a href="piglatin_ref2.html#Cast+Operators">cast</a> the columns to the appropriate types. An example will illustrate the point.
-The LOAD statement (a) has a schema associated with it. The optimizer will insert a FOREACH operator that will project columns 0, 1 and 2
-and also cast them to chararray, int and float respectively.
-</p>
+<title>LogicalExpressionSimplifier</title>
+<p>This rule contains several types of simplifications.</p>
+
<source>
-A = LOAD 'input' AS (name: chararray, age: int, gpa: float);
-B = FILER A BY $1 == 1;
-C = GROUP A By $0;
+1) Constant pre-calculation
+
+B = FILTER A BY a0 > 5+7;
+is simplified to
+B = FILTER A BY a0 > 12;
+
+2) Elimination of negations
+
+B = FILTER A BY NOT (NOT(a0 > 5) OR a > 10);
+is simplified to
+B = FILTER A BY a0 > 5 AND a <= 10;
+
+3) Elimination of logical implied expression in AND
+
+B = FILTER A BY (a0 > 5 AND a0 > 7);
+is simplified to
+B = FILTER A BY a0 > 7;
+
+4) Elimination of logical implied expression in OR
+
+B = FILTER A BY ((a0 > 5) OR (a0 > 6 AND a1 > 15);
+is simplified to
+B = FILTER C BY a0 > 5;
+
+5) Equivalence elimination
+
+B = FILTER A BY (a0 v 5 AND a0 > 5);
+is simplified to
+B = FILTER A BY a0 > 5;
+
+6) Elimination of complementary expressions in OR
+
+B = FILTER A BY (a0 > 5 OR a0 <= 5);
+is simplified to non-filtering
+
+7) Elimination of naive TRUE expression
+
+B = FILTER A BY 1==1;
+is simplified to non-filtering
</source>
</section>
+
+<!-- +++++++++++++++++++++++++++++++ -->
<section>
-<title>StreamOptimizer</title>
-<p>
-Optimize when <a href="piglatin_ref2.html#LOAD">LOAD</a> precedes <a href="piglatin_ref2.html#STREAM">STREAM</a>
-and the loader class is the same as the serializer for the stream. Similarly, optimize when STREAM is followed by
-<a href="piglatin_ref2.html#STORE">STORE</a> and the deserializer class is same as the storage class.
-For both of these cases the optimization is to replace the loader/serializer with BinaryStorage which just moves bytes
-around and to replace the storer/deserializer with BinaryStorage.
-</p>
+<title>MergeForEach</title>
+<p>The objective of this rule is to merge together two feach statements, if these preconditions are met:</p>
+<ul>
+ <li>The foreach statements are consecutive. </li>
+ <li>The first foreach statement does not contain flatten. </li>
+ <li>The second foreach is not nested. </li>
+</ul>
+<source>
+-- Original code:
+
+A = LOAD 'file.txt' AS (a, b, c);
+B = FOREACH A GENERATE a+b AS u, c-b AS v;
+C = FOREACH B GENERATE $0+5, v;
+-- Optimized code:
+
+A = LOAD 'file.txt' AS (a, b, c);
+C = FOREACH A GENERATE a+b+5, c-b;
+</source>
</section>
+
+<!-- +++++++++++++++++++++++++++++++ -->
<section>
<title>OpLimitOptimizer</title>
<p>
@@ -600,6 +716,7 @@ C = FILTER B BY $0 < 10;
</source>
</section>
+<!-- +++++++++++++++++++++++++++++++ -->
<section>
<title>PushDownExplodes</title>
<p>
@@ -615,8 +732,313 @@ C = FOREACH A GENERATE FLATTEN($0), B, C
D = JOIN C BY $1, B BY $1;
</source>
</section>
-</section> <!-- END OPTIMIZATION RULES -->
+<!-- +++++++++++++++++++++++++++++++ -->
+<section>
+<title>StreamOptimizer</title>
+<p>
+Optimize when <a href="piglatin_ref2.html#LOAD">LOAD</a> precedes <a href="piglatin_ref2.html#STREAM">STREAM</a>
+and the loader class is the same as the serializer for the stream. Similarly, optimize when STREAM is followed by
+<a href="piglatin_ref2.html#STORE">STORE</a> and the deserializer class is same as the storage class.
+For both of these cases the optimization is to replace the loader/serializer with BinaryStorage which just moves bytes
+around and to replace the storer/deserializer with BinaryStorage.
+</p>
+</section>
+
+<!-- +++++++++++++++++++++++++++++++ -->
+<section>
+<title>TypeCastInserter</title>
+<p>Status: Mandatory</p>
+<p>
+If you specify a <a href="piglatin_ref2.html#Schemas">schema</a> with the
+<a href="piglatin_ref2.html#LOAD">LOAD</a> statement, the optimizer will perform a pre-fix projection of the columns
+and <a href="piglatin_ref2.html#Cast+Operators">cast</a> the columns to the appropriate types. An example will illustrate the point.
+The LOAD statement (a) has a schema associated with it. The optimizer will insert a FOREACH operator that will project columns 0, 1 and 2
+and also cast them to chararray, int and float respectively.
+</p>
+<source>
+A = LOAD 'input' AS (name: chararray, age: int, gpa: float);
+B = FILER A BY $1 == 1;
+C = GROUP A By $0;
+</source>
+</section>
+</section>
+<!-- END OPTIMIZATION RULES -->
+
+<!-- ================================================================== -->
+<!-- PIG PROPERTIES -->
+<section>
+<title>Pig Properties</title>
+ <p>
+The Pig "-propertyfile " option enables you to pass a set of Pig or Hadoop properties to a Pig job. If the value is present in both the property file passed from the command line as well as in default property file bundled into pig.jar, the properties passed from command line take precedence. This property, as well as all other properties defined in Pig, are available to your UDFs via UDFContext.getClientSystemProps()API call (see the <a href="udf.html">Pig UDF Manual</a>.) </p>
+
+<p>You can retrieve a list of all properties using the <a href="piglatin_ref2.html#help">help properties</a> command.</p>
+<p>You can set properties using the <a href="piglatin_ref2.html#set">set</a> command.</p>
+
+</section>
+<!-- END PIG PROPERTIES -->
+
+
+<!-- ==================================================================== -->
+
+<!-- PIG STATISTICS-->
+<section>
+<title>Pig Statistics</title>
+
+<p>Pig Statistics is a framework for collecting and storing script-level statistics for Pig Latin. Characteristics of Pig Latin scripts and the resulting MapReduce jobs are collected while the script is executed. These statistics are then available for Pig users and tools using Pig (such as Oozie) to retrieve after the job is done.</p>
+
+<p>The new Pig statistics and the existing Hadoop statistics can also be accessed via the Hadoop job history file (and job xml file). Piggybank has a HadoopJobHistoryLoader which acts as an example of using Pig itself to query these statistics (the loader can be used as a reference implementation but is NOT supported for production use).</p>
+
+<section>
+<title>Java API</title>
+
+<p>Several new public classes make it easier for external tools such as Oozie to integrate with Pig statistics. </p>
+
+<p>The Pig statistics are available here: <a href="http://pig.apache.org/docs/r0.8.0/api/">http://pig.apache.org/docs/r0.8.0/api/</a></p>
+
+<p>The stats classes are in the package: org.apache.pig.tools.pigstats</p>
+<ul>
+<li>PigStats</li>
+<li>JobStats</li>
+<li>OutputStats</li>
+<li>InputStats</li>
+</ul>
+<p></p>
+
+<p>The PigRunner class mimics the behavior of the Main class but gives users a statistics object back. Optionally, you can call the API with an implementation of progress listener which will be invoked by Pig runtime during the execution. </p>
+
+<source>
+package org.apache.pig;
+
+public abstract class PigRunner {
+ public static PigStats run(String[] args, PigProgressNotificationListener listener)
+}
+
+public interface PigProgressNotificationListener extends java.util.EventListener {
+ // just before the launch of MR jobs for the script
+ public void LaunchStartedNotification(int numJobsToLaunch);
+ // number of jobs submitted in a batch
+ public void jobsSubmittedNotification(int numJobsSubmitted);
+ // a job is started
+ public void jobStartedNotification(String assignedJobId);
+ // a job is completed successfully
+ public void jobFinishedNotification(JobStats jobStats);
+ // a job is failed
+ public void jobFailedNotification(JobStats jobStats);
+ // a user output is completed successfully
+ public void outputCompletedNotification(OutputStats outputStats);
+ // updates the progress as percentage
+ public void progressUpdatedNotification(int progress);
+ // the script execution is done
+ public void launchCompletedNotification(int numJobsSucceeded);
+}
+</source>
+
+
+</section>
+
+<section>
+<title>Job XML</title>
+<p>The following entries are included in job conf: </p>
+
+<table>
+<tr>
+<td>
+<p> <strong>Pig Statistic</strong> </p>
+</td>
+<td>
+<p> <strong>Description</strong></p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.script.id</p>
+</td>
+<td>
+<p>The UUID for the script. All jobs spawned by the script have the same script ID.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.script</p>
+</td>
+<td>
+<p>The base64 encoded script text.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.command.line</p>
+</td>
+<td>
+<p>The command line used to invoke the script.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.hadoop.version</p>
+</td>
+<td>
+<p>The Hadoop version installed.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.version</p>
+</td>
+<td>
+<p>The Pig version used.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.input.dirs</p>
+</td>
+<td>
+<p>A comma-separated list of input directories for the job.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.map.output.dirs</p>
+</td>
+<td>
+<p>A comma-separated list of output directories in the map phase of the job.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.reduce.output.dirs</p>
+</td>
+<td>
+<p>A comma-separated list of output directories in the reduce phase of the job.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.parent.jobid</p>
+</td>
+<td>
+<p>A comma-separated list of parent job ids.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.script.features</p>
+</td>
+<td>
+<p>A list of Pig features used in the script.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.job.feature</p>
+</td>
+<td>
+<p>A list of Pig features used in the job.</p>
+</td>
+</tr>
+<tr>
+<td>
+<p>pig.alias</p>
+</td>
+<td>
+<p>The alias associated with the job.</p>
+</td>
+</tr>
+</table>
+</section>
+
+<section>
+<title>Hadoop Job History Loader</title>
+<p>The HadoopJobHistoryLoader in Piggybank loads Hadoop job history files and job xml files from file system. For each MapReduce job, the loader produces a tuple with schema (j:map[], m:map[], r:map[]). The first map in the schema contains job-related entries. Here are some of important key names in the map: </p>
+
+<table>
+<tr>
+<td>
+<p>PIG_SCRIPT_ID</p>
+<p>CLUSTER </p>
+<p>QUEUE_NAME</p>
+<p>JOBID</p>
+<p>JOBNAME</p>
+<p>STATUS</p>
+</td>
+<td>
+<p>USER </p>
+<p>HADOOP_VERSION </p>
+<p>PIG_VERSION</p>
+<p>PIG_JOB_FEATURE</p>
+<p>PIG_JOB_ALIAS </p>
+<p>PIG_JOB_PARENTS</p>
+</td>
+<td>
+<p>SUBMIT_TIME</p>
+<p>LAUNCH_TIME</p>
+<p>FINISH_TIME</p>
+<p>TOTAL_MAPS</p>
+<p>TOTAL_REDUCES</p>
+</td>
+</tr>
+</table>
+<p></p>
+<p>Examples that use the loader to query Pig statistics are shown below.</p>
+</section>
+
+
+<section>
+<title>Examples</title>
+<p>Find scripts that generate more then three MapReduce jobs:</p>
+<source>
+a = load '/mapred/history/done' using HadoopJobHistoryLoader() as (j:map[], m:map[], r:map[]);
+b = group a by (j#'PIG_SCRIPT_ID', j#'USER', j#'JOBNAME');
+c = foreach b generate group.$1, group.$2, COUNT(a);
+d = filter c by $2 > 3;
+dump d;
+</source>
+
+<p>Find the running time of each script (in seconds): </p>
+<source>
+a = load '/mapred/history/done' using HadoopJobHistoryLoader() as (j:map[], m:map[], r:map[]);
+b = foreach a generate j#'PIG_SCRIPT_ID' as id, j#'USER' as user, j#'JOBNAME' as script_name,
+ (Long) j#'SUBMIT_TIME' as start, (Long) j#'FINISH_TIME' as end;
+c = group b by (id, user, script_name)
+d = foreach c generate group.user, group.script_name, (MAX(b.end) - MIN(b.start)/1000;
+dump d;
+</source>
+
+<p>Find the number of scripts run by user and queue on a cluster: </p>
+<source>
+a = load '/mapred/history/done' using HadoopJobHistoryLoader() as (j:map[], m:map[], r:map[]);
+b = foreach a generate j#'PIG_SCRIPT_ID' as id, j#'USER' as user, j#'QUEUE_NAME' as queue;
+c = group b by (id, user, queue) parallel 10;
+d = foreach c generate group.user, group.queue, COUNT(b);
+dump d;
+</source>
+
+<p>Find scripts that have failed jobs: </p>
+<source>
+a = load '/mapred/history/done' using HadoopJobHistoryLoader() as (j:map[], m:map[], r:map[]);
+b = foreach a generate (Chararray) j#'STATUS' as status, j#'PIG_SCRIPT_ID' as id, j#'USER' as user, j#'JOBNAME' as script_name, j#'JOBID' as job;
+c = filter b by status != 'SUCCESS';
+dump c;
+</source>
+
+<p>Find scripts that use only the default parallelism: </p>
+<source>
+a = load '/mapred/history/done' using HadoopJobHistoryLoader() as (j:map[], m:map[], r:map[]);
+b = foreach a generate j#'PIG_SCRIPT_ID' as id, j#'USER' as user, j#'JOBNAME' as script_name, (Long) r#'NUMBER_REDUCES' as reduces;
+c = group b by (id, user, script_name) parallel 10;
+d = foreach c generate group.user, group.script_name, MAX(b.reduces) as max_reduces;
+e = filter d by max_reduces == 1;
+dump e;
+</source>
+
+</section>
+
+
+</section>
+
+<!-- END PIG STATISTICS-->
<!-- ==================================================================== -->
@@ -624,7 +1046,8 @@ D = JOIN C BY $1, B BY $1;
<section>
<title>Specialized Joins</title>
<p>
-In certain cases, the performance of <a href="piglatin_ref2.html#JOIN+%28inner%29">inner joins</a> and <a href="piglatin_ref2.html#JOIN+%28outer%29">outer joins</a>
+In certain cases, the performance of <a href="piglatin_ref2.html#JOIN+%28inner%29">inner joins</a> and
+<a href="piglatin_ref2.html#JOIN+%28outer%29">outer joins</a>
can be optimized using replicated, skewed, or merge joins. </p>
@@ -794,8 +1217,8 @@ job performing the merge-join will proce
</section>
</section><!-- END MERGE JOIN -->
+</section>
-</section>
<!-- END SPECIALIZED JOINS-->
Modified: pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref2.xml
URL: http://svn.apache.org/viewvc/pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref2.xml?rev=1021560&r1=1021559&r2=1021560&view=diff
==============================================================================
--- pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref2.xml (original)
+++ pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/piglatin_ref2.xml Tue Oct 12 01:03:12 2010
@@ -1880,6 +1880,7 @@ A = LOAD 'mydata' AS (T1:(f1:int, f2:int
</listitem>
</itemizedlist>
<para>The value of a parameter, in either form, can be expressed in terms of other parameters as long as the values of the dependent parameters are already defined.</para>
+ <para>There are no hard limits on the size except that parameters need to fit into memory.</para>
</entry>
</row>
<row>
@@ -4919,16 +4920,8 @@ dump E;
<para>PARALLEL n</para>
</entry>
<entry>
- <para>Increase the parallelism of a job by specifying the number of reduce tasks, n. The default value for n is 1 (one reduce task). Note the following:</para>
- <itemizedlist>
- <listitem>
- <para>Parallel only affects the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block. </para>
- </listitem>
- <listitem>
- <para>If you donât specify parallel, you still get the same map parallelism but only one reduce task. </para>
- </listitem>
- </itemizedlist>
- <para>For more information, see the <ulink url="cookbook.html#Use+the+PARALLEL+Clause">Pig Cookbook</ulink>.</para>
+ <para>Increase the parallelism of a job by specifying the number of reduce tasks, n. </para>
+ <para>For more information, see <ulink url="cookbook.html#Use+the+Parallel+Features">Use the Parallel Features</ulink>.</para>
</entry>
</row></tbody></tgroup>
</informaltable></section>
@@ -5019,16 +5012,8 @@ DUMP X;
<para>PARALLEL n</para>
</entry>
<entry>
- <para>Increase the parallelism of a job by specifying the number of reduce tasks, n. The default value for n is 1 (one reduce task). Note the following:</para>
- <itemizedlist>
- <listitem>
- <para>Parallel only affects the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block. </para>
- </listitem>
- <listitem>
- <para>If you donât specify parallel, you still get the same map parallelism but only one reduce task. </para>
- </listitem>
- </itemizedlist>
- <para>For more information, see the <ulink url="cookbook.html#Use+the+PARALLEL+Clause">Pig Cookbook</ulink>.</para>
+ <para>Increase the parallelism of a job by specifying the number of reduce tasks, n.</para>
+ <para>For more information, see <ulink url="cookbook.html#Use+the+Parallel+Features">Use the Parallel Features</ulink>.</para>
</entry>
</row></tbody></tgroup>
</informaltable></section>
@@ -5226,7 +5211,7 @@ DUMP X;
<para>nested_op</para>
</entry>
<entry>
- <para>Allowed operations are DISTINCT, FILTER, LIMIT, and ORDER. </para>
+ <para>Allowed operations are DISTINCT, FILTER, LIMIT, and ORDER BY. </para>
<para>The FOREACHâ¦GENERATE operation itself is not allowed since this could lead to an arbitrary number of nesting levels.</para>
<para>You can also perform projections (see <xref linkend="nestedblock" />).</para>
</entry>
@@ -5656,16 +5641,8 @@ DUMP X;
<para>PARALLEL n</para>
</entry>
<entry>
- <para>Increase the parallelism of a job by specifying the number of reduce tasks, n. The default value for n is 1 (one reduce task). Note the following:</para>
- <itemizedlist>
- <listitem>
- <para>Parallel only affects the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block. </para>
- </listitem>
- <listitem>
- <para>If you donât specify parallel, you still get the same map parallelism but only one reduce task. </para>
- </listitem>
- </itemizedlist>
- <para>For more information, see the <ulink url="cookbook.html#Use+the+PARALLEL+Clause">Pig Cookbook</ulink>.</para>
+ <para>Increase the parallelism of a job by specifying the number of reduce tasks, n.</para>
+ <para>For more information, see <ulink url="cookbook.html#Use+the+Parallel+Features">Use the Parallel Features</ulink>.</para>
</entry>
</row></tbody></tgroup>
</informaltable></section>
@@ -6014,16 +5991,8 @@ public class SimpleCustomPartitioner ext
<para>PARALLEL n</para>
</entry>
<entry>
- <para>Increase the parallelism of a job by specifying the number of reduce tasks, n. The default value for n is 1 (one reduce task). Note the following:</para>
- <itemizedlist>
- <listitem>
- <para>Parallel only affects the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block. </para>
- </listitem>
- <listitem>
- <para>If you donât specify parallel, you still get the same map parallelism but only one reduce task. </para>
- </listitem>
- </itemizedlist>
- <para>For more information, see the <ulink url="cookbook.html#Use+the+PARALLEL+Clause">Pig Cookbook</ulink>.</para>
+ <para>Increase the parallelism of a job by specifying the number of reduce tasks, n. </para>
+ <para>For more information, see <ulink url="cookbook.html#Use+the+Parallel+Features">Use the Parallel Features</ulink>.</para>
</entry>
</row></tbody></tgroup>
</informaltable></section>
@@ -6231,16 +6200,8 @@ DUMP X;
<para>PARALLEL n</para>
</entry>
<entry>
- <para>Increase the parallelism of a job by specifying the number of reduce tasks, n. The default value for n is 1 (one reduce task). Note the following:</para>
- <itemizedlist>
- <listitem>
- <para>Parallel only affects the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block. </para>
- </listitem>
- <listitem>
- <para>If you donât specify parallel, you still get the same map parallelism but only one reduce task. </para>
- </listitem>
- </itemizedlist>
- <para>For more information, see the <ulink url="cookbook.html#Use+the+PARALLEL+Clause">Pig Cookbook</ulink>.</para>
+ <para>Increase the parallelism of a job by specifying the number of reduce tasks, n. </para>
+ <para>For more information, see <ulink url="cookbook.html#Use+the+Parallel+Features">Use the Parallel Features</ulink>.</para>
</entry>
</row>
</tbody></tgroup>
@@ -6670,16 +6631,8 @@ B = MAPREDUCE wordcount.jar STOE A INTO
<para>PARALLEL n</para>
</entry>
<entry>
- <para>Increase the parallelism of a job by specifying the number of reduce tasks, n. The default value for n is 1 (one reduce task). Note the following:</para>
- <itemizedlist>
- <listitem>
- <para>Parallel only affects the number of reduce tasks. Map parallelism is determined by the input file, one map for each HDFS block. </para>
- </listitem>
- <listitem>
- <para>If you donât specify parallel, you still get the same map parallelism but only one reduce task. </para>
- </listitem>
- </itemizedlist>
- <para>For more information, see the <ulink url="cookbook.html#Use+the+PARALLEL+Clause">Pig Cookbook</ulink>.</para>
+ <para>Increase the parallelism of a job by specifying the number of reduce tasks, n.</para>
+ <para>For more information, see <ulink url="cookbook.html#Use+the+Parallel+Features">Use the Parallel Features</ulink>.</para>
</entry>
</row></tbody></tgroup>
</informaltable></section>
@@ -7327,7 +7280,7 @@ DUMP U;
<section>
<title>DESCRIBE</title>
- <para>Returns the schema of an alias.</para>
+ <para>Returns the schema of a relation.</para>
<section>
<title>Syntax</title>
@@ -7354,7 +7307,10 @@ DUMP U;
<section>
<title>Usage</title>
- <para>Use the DESCRIBE operator to review the schema of a particular alias.</para></section>
+ <para>Use the DESCRIBE operator to view the schema of a relation.
+ You can view outer relations as well as relations defined in a nested FOREACH statement.</para>
+ <para>See also: <ulink url="piglatin_ref1.html#Debugging+Pig+Latin">Debugging Pig Latin</ulink></para>
+ </section>
<section>
<title>Example</title>
@@ -7407,6 +7363,17 @@ DESCRIBE d;
d: {long}
</programlisting>
+ <para>This example shows how to view the schema of a nested relation using the :: operator.</para>
+ <programlisting>
+A = LOAD 'studentab10k' AS (name, age, gpa);
+B = GROUP A BY name;
+C = FOREACH B {
+ D = DISTINCT A.age;
+ GENERATE COUNT(D), group;}
+
+DESCRIBE C::D;
+D: {age: bytearray}
+ </programlisting>
</section></section>
@@ -7446,6 +7413,7 @@ d: {long}
Note that production scripts <emphasis>should not</emphasis> use DUMP as it will disable multi-query optimizations and is likely to slow down execution
(see <ulink url="piglatin_ref1.html#Store+vs.+Dump">Store vs. Dump</ulink>).
</para>
+ <para>See also: <ulink url="piglatin_ref1.html#Debugging+Pig+Latin">Debugging Pig Latin</ulink></para>
</section>
<section>
@@ -7580,6 +7548,8 @@ DUMP B;
<para></para>
<para>If a script without an alias is specified, it will output the entire execution graph (logical, physical, or map reduce). </para>
<para>If a script with a alias is specified, it will output the plan for the given alias. </para>
+ <para></para>
+ <para>See also: <ulink url="piglatin_ref1.html#Debugging+Pig+Latin">Debugging Pig Latin</ulink></para>
</section>
<section>
@@ -8120,20 +8090,23 @@ B = FOREACH A GENERATE myFunc($0);
<para>alias</para>
</entry>
<entry>
- <para>The path of a Java JAR file. Do not place the name in quotes.</para>
+ <para>The path to the JAR file (the full location URI is required). Do not place the name in quotes.</para>
+ <para>Pig supports functions written in Java only.</para>
+ <para>Pig supports JAR files stored in HDFS, Amazon S3, and other distributed files systems (see also <ulink url="piglatin_ref1.html#Pig+Latin+Scripts">Pig Latin Scripts</ulink>).</para>
</entry>
</row></tbody></tgroup>
</informaltable></section>
<section>
<title>Usage</title>
- <para>Use the REGISTER statement inside a Pig script to specify the path of a Java JAR file containing UDFs. </para>
+ <para>Use the REGISTER statement inside a Pig script to specify the path of the JAR file containing UDFs. </para>
<para>You can register additional files (to use with your Pig script) via the command line using the -Dpig.additional.jars option.</para>
- <para>For more information about UDFs, see the User Defined Function Guide. Note that Pig currently only supports functions written in Java.</para></section>
+ <para>For more information see the <ulink url="udf.html">Pig UDF Manual</ulink>.</para>
+ </section>
<section>
- <title>Example</title>
- <para>In this example REGISTER states that myfunc.jar is located in the /src directory.</para>
+ <title>Examples</title>
+<para>In this example REGISTER states that myfunc.jar is located in the /src directory.</para>
<programlisting>
/src $ java -jar pig.jar â
@@ -8142,11 +8115,15 @@ A = LOAD 'students';
B = FOREACH A GENERATE myfunc.MyEvalFunc($0);
</programlisting>
- <para>In this example additional jar files are registered via the command line.</para>
+<para>In this example additional jar files are registered via the command line.</para>
<programlisting>
pig -Dpig.additional.jars=my.jar:your.jar script.pig
</programlisting>
+<para>In this example a jar file stored in HDFS is registered</para>
+<programlisting>
+java -cp pig.jar org.apache.pig.Main hdfs://nn.mydomain.com:9020/myscripts/script.pig
+</programlisting>
<para></para></section></section>
</section>
@@ -9213,7 +9190,11 @@ store A into âmyoutput.bzâ;
<para>BinStorage works with data that is represented on disk in machine-readable format.
BinStorage does NOT support <ulink url="#Handling+Compression">compression</ulink>.</para>
- <para>BinStorage is used internally by Pig to store the temporary data that is created between multiple map/reduce jobs.</para></section>
+ <para>BinStorage is used internally by Pig to store the temporary data that is created between multiple map/reduce jobs.</para>
+
+ <para>BinStorage supports multiple locations (files, directories, globs) as input.</para>
+
+ </section>
<section>
<title>Example</title>
@@ -9223,6 +9204,11 @@ A = LOAD 'data' USING BinStorage();
STORE X into 'output' USING BinStorage();
</programlisting>
+
+ <para>In this example BinStorage is used to load multiple locations.</para>
+<programlisting>
+A = LOAD 'input1.bin, input2.bin' USING BinStorage();
+</programlisting>
</section></section>
<section>
@@ -9259,6 +9245,8 @@ STORE X into 'output' USING BinStorage()
<para>PigStorage is the default function for the LOAD and STORE operators and works with both simple and complex data types. </para>
<para>PigStorage supports structured text files (in human-readable UTF-8 format). PigStorage also supports <ulink url="#Handling+Compression">compression</ulink>.</para>
+
+ <para>PigStorage supports multiple locations (files, directories, globs) as input.</para>
<para>Load statements â PigStorage expects data to be formatted using field delimiters, either the tab character ('\t') or other specified character.</para>
@@ -11999,14 +11987,14 @@ grunt> exec âparam p1=myparam1 �
<section>
<title>help</title>
- <para>Prints a list of Pig commands.</para>
+ <para>Prints a list of Pig commands or properties.</para>
<section>
<title>Syntax</title>
<informaltable frame="all">
<tgroup cols="1"><tbody><row>
<entry>
- <para>help     </para>
+ <para>-help [properties] Â </para>
</entry>
</row></tbody></tgroup>
</informaltable></section>
@@ -12016,29 +12004,53 @@ grunt> exec âparam p1=myparam1 � <informaltable frame="all">
<tgroup cols="2"><tbody><row>
<entry>
- <para>none</para>
+ <para>properties</para>
</entry>
<entry>
- <para>no parameters</para>
+ <para>List Pig properties.</para>
</entry>
</row></tbody></tgroup>
</informaltable></section>
<section>
<title>Usage</title>
- <para>The help command prints a list of Pig commands.</para></section>
+ <para>The help command prints a list of Pig commands or properties.</para></section>
<section>
<title>Example</title>
- <para>In this example the results of the help command (the first few lines) are shown.</para>
+ <para>Use "help" to get a list of commands.</para>
<programlisting>
-grunt> help
-Commands:
-<pig latin statement>;
-store <alias> into <filename> [using <functionSpec>]
-dump <alias>
-<emphasis>etc â¦</emphasis></programlisting>
- </section></section>
+$ pig -help
+
+Apache Pig version 0.8.0-dev (r987348)
+compiled Aug 19 2010, 16:38:44
+
+USAGE: Pig [options] [-] : Run interactively in grunt shell.
+ Pig [options] -e[xecute] cmd [cmd ...] : Run cmd(s).
+ Pig [options] [-f[ile]] file : Run cmds found in file.
+ options include:
+ -4, -log4jconf - Log4j configuration file, overrides log conf
+ -b, -brief - Brief logging (no timestamps)
+ -c, -check - Syntax check
+<emphasis>etc â¦</emphasis>
+</programlisting>
+
+<para>Use "-help properties" to get a list of properties.</para>
+<programlisting>
+$ pig -help properties
+
+The following properties are supported:
+ Logging:
+ verbose=true|false; default is false. This property is the same as -v switch
+ brief=true|false; default is false. This property is the same as -b switch
+ debug=OFF|ERROR|WARN|INFO|DEBUG; default is INFO. This property is the same as -d switch
+ aggregate.warning=true|false; default is true. If true, prints count of warnings
+ of each type rather than logging each warning.
+<emphasis>etc â¦</emphasis>
+</programlisting>
+
+ </section>
+ </section>
<section>
<title>kill</title>
@@ -12246,7 +12258,8 @@ grunt> run âparam out=myoutput m
<section>
<title>Usage</title>
- <para>The set command enables you to assign values to keys, as shown in the table. </para>
+ <para>Use the set command to assign values to keys, as shown in the table. All keys and their corresponding values (for Pig and Hadoop) are case sensitive. </para>
+
<informaltable frame="all">
<tgroup cols="3"><tbody><row>
<entry>
@@ -12268,7 +12281,7 @@ grunt> run âparam out=myoutput m
</entry>
<entry>
<para>Sets the number of reducers for all MapReduce jobs generated by Pig
- (see <ulink url="cookbook.html#Use+the+PARALLEL+Clause">Use the PARALLEL Clause</ulink>).</para>
+ (see <ulink url="cookbook.html#Use+the+Parallel+Features">Use the Parallel Features</ulink>).</para>
</entry>
</row>
<row>
@@ -12321,18 +12334,21 @@ grunt> run âparam out=myoutput m
</tbody></tgroup>
</informaltable>
<para/>
+
+ <para>
+All Pig and Hadoop properties can be set, either in the Pig script or via the Grunt command line.
+ </para>
</section>
<section>
- <title>Example</title>
- <para>In this example debug is set on, the job is assigned a name, and the number of reducers is set to 100.</para>
+ <title>Examples</title>
+ <para>In this example key value pairs are set at the command line.</para>
<programlisting>
-grunt> set debug 'on'
-grunt> set job.name 'my job'
-grunt> set default_parallel 100
+grunt> SET debug 'on'
+grunt> SET job.name 'my job'
+grunt> SET default_parallel 100
</programlisting>
-
<para>In this example default_parallel is set in the Pig script; all MapReduce jobs that get launched will use 20 reducers.</para>
<programlisting>
SET default_parallel 20;
@@ -12342,6 +12358,16 @@ C = FOREACH B GENERATE group, COUNT(A.t)
D = ORDER C BY mycount;
STORE D INTO 'mysortedcount' USING PigStorage();
</programlisting>
+
+
+<para>In this example multiple key value pairs are set in the Pig script. These key value pairs are put in job-conf by Pig (making the pairs available to Pig and Hadoop). This is a script-wide setting; if a key value is defined multiple times in the script the last value will take effect and will be set for all jobs generated by the script. </para>
+<programlisting>
+...
+SET mapred.map.tasks.speculative.execution false;
+SET pig.logfile mylogfile.log;
+SET my.arbitrary.key my.arbitary.value;
+...
+</programlisting>
</section>
</section>
Modified: pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/udf.xml
URL: http://svn.apache.org/viewvc/pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/udf.xml?rev=1021560&r1=1021559&r2=1021560&view=diff
==============================================================================
--- pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/udf.xml (original)
+++ pig/branches/branch-0.8/src/docs/src/documentation/content/xdocs/udf.xml Tue Oct 12 01:03:12 2010
@@ -30,7 +30,8 @@
<title>Overview</title>
<p>Pig provides extensive support for user-defined functions (UDFs) as a way to specify custom processing.
Functions can be a part of almost every operator in Pig.
-This document describes how to use existing functions as well as how to write your own functions.</p>
+This document describes how to use existing functions as well as how to write your own functions
+(see also <a href="piglatin_ref1.html#Pig+Properties">Pig Properties</a>.)</p>
</section>
<section>