You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hcatalog-commits@incubator.apache.org by kh...@apache.org on 2012/09/06 08:09:05 UTC
svn commit: r1381506 - in /incubator/hcatalog/trunk: CHANGES.txt
src/docs/src/documentation/content/xdocs/inputoutput.xml
src/docs/src/documentation/content/xdocs/loadstore.xml
Author: khorgath
Date: Thu Sep 6 08:09:04 2012
New Revision: 1381506
URL: http://svn.apache.org/viewvc?rev=1381506&view=rev
Log:
HCATALOG-431 document hcat type to java class/pig type mapping (lefty via khorgath)
Modified:
incubator/hcatalog/trunk/CHANGES.txt
incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml
incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml
Modified: incubator/hcatalog/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/incubator/hcatalog/trunk/CHANGES.txt?rev=1381506&r1=1381505&r2=1381506&view=diff
==============================================================================
--- incubator/hcatalog/trunk/CHANGES.txt (original)
+++ incubator/hcatalog/trunk/CHANGES.txt Thu Sep 6 08:09:04 2012
@@ -38,6 +38,8 @@ Trunk (unreleased changes)
HCAT-427 Document storage-based authorization (lefty via gates)
IMPROVEMENTS
+ HCAT-431 document hcat type to java class/pig type mapping (lefty via khorgath)
+
HCAT-492 Document CTAS workaround for Hive with JSON serde (lefty via khorgath)
HCAT-487 HCatalog should tolerate a user-defined amount of bad records (traviscrawford)
Modified: incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml
URL: http://svn.apache.org/viewvc/incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml?rev=1381506&r1=1381505&r2=1381506&view=diff
==============================================================================
--- incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml (original)
+++ incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/inputoutput.xml Thu Sep 6 08:09:04 2012
@@ -149,6 +149,84 @@ will be returned.</p>
</section>
+<!-- ==================================================================== -->
+<section>
+ <title>HCatRecord</title>
+
+<p>HCatRecord is the type supported for storing values in HCatalog tables.</p>
+<p>The types in an HCatalog table schema determine the types of objects returned for different fields in HCatRecord. This table shows the mappings between Java classes for MapReduce programs and HCatalog data types:</p>
+
+<table>
+ <tr>
+ <th><p class="center">HCatalog Data Type</p></th>
+ <th><p class="center">Java Class in MapReduce</p></th>
+ <th><p class="cell">Values</p></th>
+ </tr>
+ <tr>
+ <td><p class="center">TINYINT</p></td>
+ <td><p class="center">java.lang.Byte</p></td>
+ <td><p class="cell">-128 to 127</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">SMALLINT</p></td>
+ <td><p class="center">java.lang.Short</p></td>
+ <td><p class="cell">-2<sup>15</sup> to 2<sup>15</sup>-1 (-32,768 to 32,767)</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">INT</p></td>
+ <td><p class="center">java.lang.Integer</p></td>
+ <td><p class="cell">-2<sup>31</sup> to 2<sup>31</sup>-1 (-2,147,483,648 to 2,147,483,647)</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">BIGINT</p></td>
+ <td><p class="center">java.lang.Long</p></td>
+ <td><p class="cell">-2<sup>63</sup> to 2<sup>63</sup>-1 (-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807)</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">BOOLEAN</p></td>
+ <td><p class="center">java.lang.Boolean</p></td>
+ <td><p class="cell">true or false</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">FLOAT</p></td>
+ <td><p class="center">java.lang.Float</p></td>
+ <td><p class="cell">single-precision floating-point value</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">DOUBLE</p></td>
+ <td><p class="center">java.lang.Double</p></td>
+ <td><p class="cell">double-precision floating-point value</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">BINARY</p></td>
+ <td><p class="center">byte[]</p></td>
+ <td><p class="cell">binary data</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">STRING</p></td>
+ <td><p class="center">java.lang.String</p></td>
+ <td><p class="cell">character string</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">STRUCT</p></td>
+ <td><p class="center">java.util.List</p></td>
+ <td><p class="cell">structured data</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">ARRAY</p></td>
+ <td><p class="center">java.util.List</p></td>
+ <td><p class="cell">values of one data type</p></td>
+ </tr>
+ <tr>
+ <td><p class="center">MAP</p></td>
+ <td><p class="center">java.util.Map</p></td>
+ <td><p class="cell">key-value pairs</p></td>
+ </tr>
+</table>
+
+</section>
+
+<!-- ==================================================================== -->
<section>
<title>Running MapReduce with HCatalog</title>
<p>
Modified: incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml
URL: http://svn.apache.org/viewvc/incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml?rev=1381506&r1=1381505&r2=1381506&view=diff
==============================================================================
--- incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml (original)
+++ incubator/hcatalog/trunk/src/docs/src/documentation/content/xdocs/loadstore.xml Thu Sep 6 08:09:04 2012
@@ -68,51 +68,131 @@ immediately following the load statement
<section>
<title>HCatalog Data Types</title>
-<p>Restrictions apply to the types of columns HCatLoader can read.</p>
-<p>HCatLoader can read <strong>only</strong> the data types listed in the table.
+<p>Restrictions apply to the types of columns HCatLoader can read from HCatalog-managed tables.</p>
+<p>HCatLoader can read <em><strong>only</strong></em> the data types listed in the table below.
The table shows how Pig will interpret the HCatalog data type.</p>
- <table>
- <tr>
- <td>
- <p><strong>HCatalog Data Type</strong></p>
- </td>
- <td>
- <p><strong>Pig Data Type</strong></p>
- </td>
- </tr>
- <tr>
- <td>
- <p>primitives (int, long, float, double, string) </p>
- </td>
- <td>
- <p>int, long, float, double, string to chararray </p>
- </td>
- </tr>
- <tr>
- <td>
- <p>map (key type should be string, valuetype must be string)</p>
- </td>
- <td>
- <p>map </p>
- </td>
- </tr>
- <tr>
- <td>
- <p>List<any type> </p>
- </td>
- <td>
- <p>bag </p>
- </td>
- </tr>
- <tr>
- <td>
- <p>struct<any type fields> </p>
- </td>
- <td>
- <p>tuple </p>
- </td>
+ <table>
+ <tr>
+ <th>
+ <p class="center">Primitives</p>
+ </th>
+ <th>
+ <p class="center"> </p>
+ </th>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell"><strong>HCatalog Data Type</strong></p>
+ </td>
+ <td>
+ <p class="cell"><strong>Pig Data Type</strong></p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">int</p>
+ </td>
+ <td>
+ <p class="cell">int</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">long</p>
+ </td>
+ <td>
+ <p class="cell">long</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">float</p>
+ </td>
+ <td>
+ <p class="cell">float</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">double</p>
+ </td>
+ <td>
+ <p class="cell">double</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">string</p>
+ </td>
+ <td>
+ <p class="cell">chararray</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">boolean</p>
+ </td>
+ <td>
+ <p class="cell">boolean</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">binary</p>
+ </td>
+ <td>
+ <p class="cell">bytearray</p>
+ </td>
+ </tr>
+</table>
+<table>
+ <tr>
+ <th>
+ <p class="center">Complex Types</p>
+ </th>
+ <th>
+ <p class="center"></p>
+ </th>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell"><strong>HCatalog Data Type</strong></p>
+ </td>
+ <td>
+ <p class="cell"><strong>Pig Data Type</strong></p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">map<br/>(key type should be string)</p>
+ </td>
+ <td>
+ <p class="cell">map </p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">List<em><any type></em> </p>
+ </td>
+ <td>
+ <p class="cell">bag </p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">struct<em><any type fields></em> </p>
+ </td>
+ <td>
+ <p class="cell">tuple </p>
+ </td>
</tr>
</table>
+
+<p><br/></p>
+<p>Currently HCatLoader cannot map the smallint and tinyint datatypes to Pig data types.
+This issue exists in HCatalog version 0.4.0;
+the fix for Jira issue <a href="https://issues.apache.org/jira/browse/HCATALOG-425">HCATALOG-425</a> should be available in HCatalog version 0.5.0.</p>
+
</section>
<!-- ==================================================================== -->
@@ -355,51 +435,132 @@ HCatStorer with no argument:</p>
<section>
<title>HCatalog Data Types</title>
-<p>Restrictions apply to the types of columns HCatStorer can write.</p>
-<p>HCatStorer can write <strong>only</strong> the data types listed in the table.
+<p>Restrictions apply to the types of columns HCatStorer can write to HCatalog-managed tables.</p>
+<p>HCatStorer can write <em><strong>only</strong></em> the data types listed in the table.
The table shows how Pig will interpret the HCatalog data type.</p>
- <table>
- <tr>
- <td>
- <p><strong>HCatalog Data Type</strong></p>
- </td>
- <td>
- <p><strong>Pig Data Type</strong></p>
- </td>
- </tr>
- <tr>
- <td>
- <p>primitives (int, long, float, double, string) </p>
- </td>
- <td>
- <p>int, long, float, double, string to chararray </p>
- </td>
- </tr>
- <tr>
- <td>
- <p>map (key type should be string, valuetype must be string)</p>
- </td>
- <td>
- <p>map </p>
- </td>
- </tr>
- <tr>
- <td>
- <p>List<any type> </p>
- </td>
- <td>
- <p>bag </p>
- </td>
- </tr>
- <tr>
- <td>
- <p>struct<any type fields> </p>
- </td>
- <td>
- <p>tuple </p>
- </td>
+
+ <table>
+ <tr>
+ <th>
+ <p class="center">Primitives</p>
+ </th>
+ <th>
+ <p class="center"> </p>
+ </th>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell"><strong>Pig Data Type</strong></p>
+ </td>
+ <td>
+ <p class="cell"><strong>HCatalog Data Type</strong></p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">int</p>
+ </td>
+ <td>
+ <p class="cell">int</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">long</p>
+ </td>
+ <td>
+ <p class="cell">long</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">float</p>
+ </td>
+ <td>
+ <p class="cell">float</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">double</p>
+ </td>
+ <td>
+ <p class="cell">double</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">chararray</p>
+ </td>
+ <td>
+ <p class="cell">string</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">boolean</p>
+ </td>
+ <td>
+ <p class="cell">boolean</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">bytearray</p>
+ </td>
+ <td>
+ <p class="cell">binary</p>
+ </td>
+ </tr>
+</table>
+<table>
+ <tr>
+ <th>
+ <p class="center">Complex Types</p>
+ </th>
+ <th>
+ <p class="center"> </p>
+ </th>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell"><strong>Pig Data Type</strong></p>
+ </td>
+ <td>
+ <p class="cell"><strong>HCatalog Data Type</strong></p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">map</p>
+ </td>
+ <td>
+ <p class="cell">map<br/>(key type should be string)</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">bag </p>
+ </td>
+ <td>
+ <p class="cell">List<em><any type></em> </p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <p class="cell">tuple </p>
+ </td>
+ <td>
+ <p class="cell">struct<em><any type fields></em> </p>
+ </td>
</tr>
</table>
+
+<p><br/></p>
+<p>Currently HCatLoader cannot map the smallint and tinyint datatypes to Pig data types.
+This issue exists in HCatalog version 0.4.0;
+the fix for Jira issue <a href="https://issues.apache.org/jira/browse/HCATALOG-425">HCATALOG-425</a> should be available in HCatalog version 0.5.0.</p>
+
</section>
</section>