You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by da...@apache.org on 2012/01/11 01:36:03 UTC
svn commit: r1229832 - in /pig/trunk: CHANGES.txt
src/docs/src/documentation/content/xdocs/func.xml
src/docs/src/documentation/content/xdocs/perf.xml
src/docs/src/documentation/content/xdocs/pig-index.xml
Author: daijy
Date: Wed Jan 11 00:36:02 2012
New Revision: 1229832
URL: http://svn.apache.org/viewvc?rev=1229832&view=rev
Log:
PIG-2400: Document has based aggregation support
Modified:
pig/trunk/CHANGES.txt
pig/trunk/src/docs/src/documentation/content/xdocs/func.xml
pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml
pig/trunk/src/docs/src/documentation/content/xdocs/pig-index.xml
Modified: pig/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/pig/trunk/CHANGES.txt?rev=1229832&r1=1229831&r2=1229832&view=diff
==============================================================================
--- pig/trunk/CHANGES.txt (original)
+++ pig/trunk/CHANGES.txt Wed Jan 11 00:36:02 2012
@@ -214,6 +214,8 @@ PIG-2228: support partial aggregation in
BUG FIXES
+PIG-2400: Document has based aggregation support (chandec via daijy)
+
PIG-2444: Remove the Zebra *.xml documentation files from the TRUNK and Branch-10 (chandec via daijy)
PIG-2430: An EvalFunc which overrides getArgToFuncMapping with FuncSpec
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/func.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/func.xml?rev=1229832&r1=1229831&r2=1229832&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/func.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/func.xml Wed Jan 11 00:36:02 2012
@@ -1233,6 +1233,64 @@ dump X;
</section>
+ <!-- ++++++++++++++++++++++++++++++++++++++++++++++ -->
+ <section id="jsonloadstore">
+ <title>JsonLoader, JsonStorage</title>
+ <p>Load or store JSON data.</p>
+
+ <section>
+ <title>Syntax</title>
+ <table>
+ <tr>
+ <td>
+ <p>JsonLoader( [âschemaâ] ) </p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p>JsonStorage( ) </p>
+ </td>
+ </tr>
+ </table></section>
+
+ <section>
+ <title>Terms</title>
+ <table>
+ <tr>
+ <td>
+ <p>schema</p>
+ </td>
+ <td>
+ <p>An optional Pig schema, in single quotes.</p>
+ </td>
+ </tr>
+ </table></section>
+
+ <section>
+ <title>Usage</title>
+ <p>Use JsonLoader to load JSON data. </p>
+
+ <p>Use JsonStorage to store JSON data.</p>
+
+ <p>Note that there is no concept of delimit in JsonLoader or JsonStorer. The data is encoded in standard JSON format. JsonLoader optionally takes a schema as the construct argument.</p>
+ </section>
+
+ <section>
+ <title>Examples</title>
+ <p>In this example data is loaded with a schema. </p>
+<source>
+a = load 'a.json' using JsonLoader('a0:int,a1:{(a10:int,a11:chararray)},a2:(a20:double,a21:bytearray),a3:[chararray]');
+</source>
+
+ <p>In this example data is loaded without a schema; it assumes there is a .pig_schema (produced by JsonStorage) in the input directory. </p>
+<source>
+a = load 'a.json' using JsonLoader();
+</source>
+ </section></section>
+
+
+
+
<!-- ++++++++++++++++++++++++++++++++++++++++++++++ -->
@@ -1286,7 +1344,7 @@ STORE X INTO 'output' USING PigDump();
<table>
<tr>
<td>
- <p>PigStorage(field_delimiter)Â </p>
+ <p>PigStorage( [field_delimiter] , ['options'] )Â </p>
</td>
</tr>
</table></section>
@@ -1299,25 +1357,51 @@ STORE X INTO 'output' USING PigDump();
<p id="field-delimiter">field_delimiter</p>
</td>
<td>
- <p>Parameter. </p>
<p>The default field delimiter is tab ('\t'). </p>
<p>You can specify other characters as field delimiters; however, be sure to encase the characters in single quotes.</p>
</td>
</tr>
+ <tr>
+ <td>
+ <p id="pigstorage-options">'options'</p>
+ </td>
+ <td>
+ <p>A string that contains space-separated options (âoptionA optionB optionCâ)</p>
+ <p>Currently supported options are:</p>
+ <ul>
+ <li>(âschemaâ) - Stores the schema of the relation using a hidden JSON file.</li>
+ <li>(ânoschemaâ) - Ignores a stored schema during the load.</li>
+ </ul>
+ </td>
+ </tr>
</table></section>
<section>
<title>Usage</title>
<p>PigStorage is the default function used by Pig to load/store the data. PigStorage supports structured text files (in human-readable UTF-8 format) in compressed or uncompressed form (see <a href="#Handling-Compression">Handling Compression</a>). All Pig <a href="basic.html#data-types">data types</a> (both simple and complex) can be read/written using this function. The input data to the load can be a file, a directory or a glob.</p>
+ <p><strong>Load/Store Statements</strong></p>
<p>Load statements â PigStorage expects data to be formatted using field delimiters, either the tab character ('\t') or other specified character.</p>
<p>Store statements â PigStorage outputs data using field deliminters, either the tab character ('\t') or other specified character, and the line feed record delimiter ('\n'). </p>
+ <p><strong>Field/Record Delimiters</strong></p>
<p>Field Delimiters â For load and store statements the default field delimiter is the tab character ('\t'). You can use other characters as field delimiters, but separators such as ^A or Ctrl-A should be represented in Unicode (\u0001) using UTF-16 encoding (see Wikipedia <a href="http://en.wikipedia.org/wiki/ASCII">ASCII</a>, <a href="http://en.wikipedia.org/wiki/Unicode">Unicode</a>, and <a href="http://en.wikipedia.org/wiki/UTF-16">UTF-16</a>).</p>
<p>Record Deliminters â For load statements Pig interprets the line feed ( '\n' ), carriage return ( '\r' or CTRL-M) and combined CR + LF ( '\r\n' ) characters as record delimiters (do not use these characters as field delimiters). For store statements Pig uses the line feed ('\n') character as the record delimiter.</p>
+<p><strong>Schemas</strong></p>
+
+<p>If the schema option is specified, a hidden ".pig_schema" file is created in the output directory when storing data. It is used by PigStorage (with or without -schema) during loading to determine the field names and types of the data without the need for a user to explicitly provide the schema in an as clause, unless <code>noschema</code> is specified. No attempt to merge conflicting schemas is made during loading. The first schema encountered during a file system scan is used. </p>
+
+<p>Additionally, if the schema option is specified, a ".pig_headers" file is created in the output directory. This file simply lists the delimited aliases. This is intended to make export to tools that can read files with header lines easier (just cat the header to your data). </p>
+
+<p>If the schema option is NOT specified, a schema will not be written when storing data.</p>
+
+<p>If the noschema option is NOT specified, and a schema is found, it gets loaded when loading data.</p>
+
+<p>Note that regardless of whether or not you store the schema, you always need to specify the correct delimiter to read your data. If you store reading delimiter "#" and then load using the default delimiter, your data will not be parsed correctly.</p>
+
<p><strong>Complex Data Types</strong></p>
<p>The formats for complex data types are shown here:</p>
<ul>
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml?rev=1229832&r1=1229831&r2=1229832&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/perf.xml Wed Jan 11 00:36:02 2012
@@ -140,7 +140,21 @@ D = foreach B generate group, MIN (A.gpa
<p>Depending on your use case, it might be more efficient (improve performance) to split your script into multiple scripts.</p>
</section>
</section>
-
+
+<!-- ================================================================== -->
+<!-- HASH-BASED AGGREGATION IN MAP TASK-->
+ <section id="hash-based-aggregation">
+<title>Hash-based Aggregation in Map Task</title>
+
+<p> To improve performance, hash-based aggregation will aggregate records in the map task before sending them to the combiner. This optimization reduces the serializing/deserializing costs of the combiner by sending it fewer records.</p>
+
+<p><strong>Turning On Off</strong></p>
+<p>Hash-based aggregation has been shown to improve the speed of group-by operations by up to 50%. However, since this is a very new feature, it is currently turned OFF by default. To turn it ON, set the property pig.exec.mapPartAgg to true.</p>
+
+<p><strong>Configuring</strong></p>
+<p>If the group-by keys used for grouping don't result in a sufficient reduction in the number of records, the performance might be worse with this feature turned ON. To prevent this from happening, the feature turns itself off if the reduction in records sent to combiner is not more than a configurable threshold. This threshold can be set using the property pig.exec.mapPartAgg.minReduction. It is set to a default value of 10, which means that the number of records that get sent to the combiner should be reduced by a factor of 10 or more.</p>
+
+</section>
<!-- ================================================================== -->
<!-- MEMORY MANAGEMENT -->
Modified: pig/trunk/src/docs/src/documentation/content/xdocs/pig-index.xml
URL: http://svn.apache.org/viewvc/pig/trunk/src/docs/src/documentation/content/xdocs/pig-index.xml?rev=1229832&r1=1229831&r2=1229832&view=diff
==============================================================================
--- pig/trunk/src/docs/src/documentation/content/xdocs/pig-index.xml (original)
+++ pig/trunk/src/docs/src/documentation/content/xdocs/pig-index.xml Wed Jan 11 00:36:02 2012
@@ -102,6 +102,8 @@
<p><a href="udf.html#aggregate-functions">aggregate functions</a></p>
+<p><a href="perf.html#hash-based-aggregation">aggregations (hash-based) in map task</a></p>
+
<p><a href="udf.html#algebraic-interface">algebraic interface</a></p>
<p>aliases (for fields, relations). <em>See</em> referencing.</p>
@@ -437,6 +439,10 @@
<p id="j-index"><strong>J</strong> (<a href="#top">top</a>) ----------------------------------------------</p>
+<p> <a href="func.html#jsonloadstore">JsonLoader</a></p>
+
+<p> <a href="func.html#jsonloadstore">JsonStorage</a></p>
+
<p> <a href="cont.html#embed-java">Java and embedded Pig</a></p>
<p>Java objects
@@ -557,6 +563,9 @@
<br></br> <a href="cont.html#import-macros">importing macros</a>
</p>
+
+<p><a href="perf.html#hash-based-aggregation">map task and hash-based aggregation</a></p>
+
<p>MapReduce
<br></br> <a href="test.html#mapreduce-job-ids">MapReduce job ids and Pig scripts</a>
<br></br> <a href="perf.html#Parallel">setting the number of reduce tasks</a>
@@ -851,6 +860,7 @@
<br></br> <a href="basic.html#schema-load">and LOAD, STREAM</a>
<br></br> <a href="basic.html#onschema">ONSCHEMA clause</a> (UNION operator)
<br></br> <a href="basic.html#schemas">and Pig Latin</a>
+<br></br> <a href="func.html#pigstorage-options">and PigStorage</a> (function)
<br></br> <a href="udf.html#schema-return-types">and return types</a> (JavaScript UDFs)
<br></br> <a href="basic.html#schema-simple">for simple data types</a> (int, long, float, double, chararray, bytearray)
<br></br> <a href="basic.html#unknown-schema">unknown (null) schemas</a>