You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Venkat Ranganathan <n....@live.com> on 2016/05/31 14:55:35 UTC
Review Request 48077: SQOOP-2935: Support complex types with HCatatog
integration
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48077/
-----------------------------------------------------------
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-2935
https://issues.apache.org/jira/browse/SQOOP-2935
Repository: sqoop-trunk
Description
-------
Support for importing and exporting to columns of Hive complex data type using the HCatalog integration path. Documentation added to the hcatalog page on the usage.
Diffs
-----
src/docs/user/hcatalog.txt 2ae1d54
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java 57a90b0
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java ebe7209
src/test/org/apache/sqoop/hcat/HCatalogExportTest.java 8aa0725
src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 67b7a78
src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java f4b1ea9
Diff: https://reviews.apache.org/r/48077/diff/
Testing
-------
Added unit tests for complex types for both import and export
Thanks,
Venkat Ranganathan
Re: Review Request 48077: SQOOP-2935: Support complex types with
HCatatog integration
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48077/#review135718
-----------------------------------------------------------
src/docs/user/hcatalog.txt (line 38)
<https://reviews.apache.org/r/48077/#comment200768>
Good point - will remove
src/docs/user/hcatalog.txt (line 334)
<https://reviews.apache.org/r/48077/#comment200769>
No, Maps are represented with '{' key : val '}'
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java (line 277)
<https://reviews.apache.org/r/48077/#comment200774>
it does not matter as the value == null will result in an exception later. But will add
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java (line 354)
<https://reviews.apache.org/r/48077/#comment200777>
HCatFieldType was used earlier and now we use Primitive cataegory from Hive for handling complex types and it differs. Both LONG and BIGINT are the same representation of the types.
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java (line 473)
<https://reviews.apache.org/r/48077/#comment200778>
Thanks for pointing it out. It should be > 0. For arrays, the elementschema is the only member and for maps also the valueschema will be one.
Thanks Sowmya for a quick review. will update the patch. There were also a few trailing blanks that I will remove.
- Venkat Ranganathan
On May 31, 2016, 7:55 a.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48077/
> -----------------------------------------------------------
>
> (Updated May 31, 2016, 7:55 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-2935
> https://issues.apache.org/jira/browse/SQOOP-2935
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> Support for importing and exporting to columns of Hive complex data type using the HCatalog integration path. Documentation added to the hcatalog page on the usage.
>
>
> Diffs
> -----
>
> src/docs/user/hcatalog.txt 2ae1d54
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java 57a90b0
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java ebe7209
> src/test/org/apache/sqoop/hcat/HCatalogExportTest.java 8aa0725
> src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 67b7a78
> src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java f4b1ea9
>
> Diff: https://reviews.apache.org/r/48077/diff/
>
>
> Testing
> -------
>
> Added unit tests for complex types for both import and export
>
>
> Thanks,
>
> Venkat Ranganathan
>
>
Re: Review Request 48077: SQOOP-2935: Support complex types with
HCatatog integration
Posted by Sowmya Ramesh <sr...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48077/#review135694
-----------------------------------------------------------
src/docs/user/hcatalog.txt (line 38)
<https://reviews.apache.org/r/48077/#comment200721>
Minor nit: typo?
src/docs/user/hcatalog.txt (line 334)
<https://reviews.apache.org/r/48077/#comment200722>
Minor nit: typo '{' -> '['?
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java (line 277)
<https://reviews.apache.org/r/48077/#comment200728>
missing break statement?
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java (line 453)
<https://reviews.apache.org/r/48077/#comment200734>
Can value be null here?
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java (line 354)
<https://reviews.apache.org/r/48077/#comment200747>
BIGINT is changed to LONG. Can category be BIGINT instead of long?
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java (line 468)
<https://reviews.apache.org/r/48077/#comment200741>
Can this cause NPE? If so shoudl it be handled as regular Exception, I see that you are catching exception and loggign error.
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java (line 473)
<https://reviews.apache.org/r/48077/#comment200754>
Can you help me understand why size > 1 and not zero?
- Sowmya Ramesh
On May 31, 2016, 2:55 p.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48077/
> -----------------------------------------------------------
>
> (Updated May 31, 2016, 2:55 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-2935
> https://issues.apache.org/jira/browse/SQOOP-2935
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> Support for importing and exporting to columns of Hive complex data type using the HCatalog integration path. Documentation added to the hcatalog page on the usage.
>
>
> Diffs
> -----
>
> src/docs/user/hcatalog.txt 2ae1d54
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java 57a90b0
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java ebe7209
> src/test/org/apache/sqoop/hcat/HCatalogExportTest.java 8aa0725
> src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 67b7a78
> src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java f4b1ea9
>
> Diff: https://reviews.apache.org/r/48077/diff/
>
>
> Testing
> -------
>
> Added unit tests for complex types for both import and export
>
>
> Thanks,
>
> Venkat Ranganathan
>
>
Re: Review Request 48077: SQOOP-2935: Support complex types with
HCatatog integration
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48077/
-----------------------------------------------------------
(Updated May 30, 2018, 2:49 p.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
Fixed formatting
Bugs: SQOOP-2935
https://issues.apache.org/jira/browse/SQOOP-2935
Repository: sqoop-trunk
Description
-------
Support for importing and exporting to columns of Hive complex data type using the HCatalog integration path. Documentation added to the hcatalog page on the usage.
Diffs (updated)
-----
src/docs/user/hcatalog.txt 2ae1d54d
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java 57a90b0a
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java 21ff60a7
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a
src/test/org/apache/sqoop/hcat/HCatalogExportTest.java a124dd0c
src/test/org/apache/sqoop/hcat/HCatalogImportTest.java c7e1ea6f
src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java 28b42dc3
Diff: https://reviews.apache.org/r/48077/diff/3/
Changes: https://reviews.apache.org/r/48077/diff/2-3/
Testing
-------
Added unit tests for complex types for both import and export
Thanks,
Venkat Ranganathan
Re: Review Request 48077: SQOOP-2935: Support complex types with
HCatatog integration
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48077/
-----------------------------------------------------------
(Updated May 30, 2018, 2:47 p.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
Fixed the issues. Apologies for the delay in submitting updated patch. Rebased and refactored to apply to latest trunk branch
Bugs: SQOOP-2935
https://issues.apache.org/jira/browse/SQOOP-2935
Repository: sqoop-trunk
Description
-------
Support for importing and exporting to columns of Hive complex data type using the HCatalog integration path. Documentation added to the hcatalog page on the usage.
Diffs (updated)
-----
src/docs/user/hcatalog.txt 2ae1d54d
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java 57a90b0a
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java 21ff60a7
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a
src/test/org/apache/sqoop/hcat/HCatalogExportTest.java a124dd0c
src/test/org/apache/sqoop/hcat/HCatalogImportTest.java c7e1ea6f
src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java 28b42dc3
Diff: https://reviews.apache.org/r/48077/diff/2/
Changes: https://reviews.apache.org/r/48077/diff/1-2/
Testing
-------
Added unit tests for complex types for both import and export
Thanks,
Venkat Ranganathan