svn commit: r1439759 - in /pig/trunk: CHANGES.txt src/docs/src/documentation/content/xdocs/perf.xml

[bi...@apache.org]

Author: billgraham
Date: Tue Jan 29 07:23:04 2013
New Revision: 1439759

URL: http://svn.apache.org/viewvc?rev=1439759&view=rev
Log:
PIG-3139: Document reducer estimation (billgraham)

Modified:
    pig/trunk/CHANGES.txt
    pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml

Modified: pig/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/pig/trunk/CHANGES.txt?rev=1439759&r1=1439758&r2=1439759&view=diff
==============================================================================
--- pig/trunk/CHANGES.txt (original)
+++ pig/trunk/CHANGES.txt Tue Jan 29 07:23:04 2013
@@ -24,6 +24,8 @@ INCOMPATIBLE CHANGES
 
 IMPROVEMENTS
 
+PIG-3139: Document reducer estimation (billgraham)
+
 PIG-2764: Add a biginteger and bigdecimal type to pig (jcoveney)
 
 PIG-3073: POUserFunc creating log spam for large scripts (jcoveney)

Modified: pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml?rev=1439759&r1=1439758&r2=1439759&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml Tue Jan 29 07:23:04 2013
@@ -172,6 +172,33 @@ D = foreach B generate group, MIN (A.gpa
 
 </section> 
 
+<!-- ================================================================== -->
+<!-- REDUCER ESTIMATION -->
+<section id="reducer-estimation">
+<title>Reducer Estimation</title>
+
+<p>
+By default Pig determines the number of reducers to use for a given job based on the size of the
+input to the map phase. The input data size is divided by the
+pig.exec.reducers.bytes.per.reducer parameter value (default 1GB) to determine the number of
+reducers. The maximum number of reducers for a job is limited by the pig.exec.reducers.max parameter
+(default 999).
+</p>
+
+<p>
+The default reducer estimation algorithm described above can be overridden by setting the
+pig.exec.reducer.estimator parameter to the fully qualified class name of an implementation of
+<a href="http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigReducerEstimator.java">org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigReducerEstimator</a>.
+The class must exist on the classpath of the process submitting the Pig job. If the
+pig.exec.reducer.estimator.arg parameter is set, the value will be passed to a constructor
+of the implementing class that takes a single String.
+</p>
+
+<p>
+</p>
+
+</section>
+
 
 <!-- ==================================================================== -->
 <!-- MULTI-QUERY EXECUTION-->