You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Rakesh Sharma <ra...@expedia.com> on 2015/09/15 13:17:10 UTC
Review Request 38400: Review for issue SQOOP-2332: Support dynamic
partitions while sqoop import. Also support partition key formats for date
partitions
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38400/
-----------------------------------------------------------
Review request for Sqoop and Venkat Ranganathan.
Repository: sqoop-trunk
Description
-------
Review request for support of Dynamic partitons while importing data. Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column.
Diffs
-----
src/docs/user/hcatalog.txt 99ae4f5
src/java/org/apache/sqoop/SqoopOptions.java ace90fd
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java 878f765
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 4070c24
src/java/org/apache/sqoop/metastore/TableMetaData.java PRE-CREATION
src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec
src/java/org/apache/sqoop/tool/ImportTool.java 39af42c
src/java/org/apache/sqoop/util/DbUtil.java PRE-CREATION
src/test/org/apache/sqoop/hcat/HCatalogImportTest.java d97b870
Diff: https://reviews.apache.org/r/38400/diff/
Testing
-------
Yes
Thanks,
Rakesh Sharma
Re: Review Request 38400: Review for issue SQOOP-2332: Support dynamic
partitions while sqoop import. Also support partition key formats for date
partitions
Posted by Shashank Tandon <st...@expedia.com>.
> On Sept. 16, 2015, 3:24 a.m., Venkat Ranganathan wrote:
> > src/java/org/apache/sqoop/util/DbUtil.java, line 40
> > <https://reviews.apache.org/r/38400/diff/2/?file=1074073#file1074073line40>
> >
> > Not sure why these are needed. Don't we have columninfo queries that support most of the needs including support for column precision?
Add a new review request https://reviews.apache.org/r/39260/ to remove the dbutil and used existing column info queries.Since submitter cannot be changed in review board , so added a new review request to add myself as submitter with a latest patch.
> On Sept. 16, 2015, 3:24 a.m., Venkat Ranganathan wrote:
> > src/docs/user/hcatalog.txt, line 129
> > <https://reviews.apache.org/r/38400/diff/2/?file=1074066#file1074066line129>
> >
> > I think this is a separate feature by itself that I think should be in a separate JIRA?
This is added into the existing code because dynamic partition key format works only with dynamic partition feature.It is also dynamic partition key format feature is a optional switch and it is required only in case of partition column is date/time/timestamp.
> On Sept. 16, 2015, 3:24 a.m., Venkat Ranganathan wrote:
> > src/docs/user/hcatalog.txt, line 113
> > <https://reviews.apache.org/r/38400/diff/2/?file=1074066#file1074066line113>
> >
> > I think this can be eloborated a bit. Sqoop already supports dynamic partitions - as you can see from the tests that we have. What is missing today is the ability to identify during table creation that some partitions should be added to the partition by clause alone (without being in the static partition key=value spec list).
In the new review request https://reviews.apache.org/r/39260/ , this is properly described as "Dynamic partition already exists in Sqoop in case of Hive partitioned table already exists.The new changes will allow to support dynamic partition if Hive partitioned table does not exists.
Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column."
- Shashank
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38400/#review99160
-----------------------------------------------------------
On Sept. 15, 2015, 6:02 p.m., Rakesh Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38400/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 6:02 p.m.)
>
>
> Review request for Sqoop and Venkat Ranganathan.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> Review request for support of Dynamic partitons while importing data. Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column.
>
>
> Diffs
> -----
>
> src/docs/user/hcatalog.txt 99ae4f5
> src/java/org/apache/sqoop/SqoopOptions.java ace90fd
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java 878f765
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 4070c24
> src/java/org/apache/sqoop/metastore/TableMetaData.java PRE-CREATION
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec
> src/java/org/apache/sqoop/tool/ImportTool.java 39af42c
> src/java/org/apache/sqoop/util/DbUtil.java PRE-CREATION
> src/test/org/apache/sqoop/hcat/HCatalogImportTest.java d97b870
>
> Diff: https://reviews.apache.org/r/38400/diff/
>
>
> Testing
> -------
>
> Yes
>
>
> Thanks,
>
> Rakesh Sharma
>
>
Re: Review Request 38400: Review for issue SQOOP-2332: Support dynamic
partitions while sqoop import. Also support partition key formats for date
partitions
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38400/#review99160
-----------------------------------------------------------
src/docs/user/hcatalog.txt (line 113)
<https://reviews.apache.org/r/38400/#comment156064>
I think this can be eloborated a bit. Sqoop already supports dynamic partitions - as you can see from the tests that we have. What is missing today is the ability to identify during table creation that some partitions should be added to the partition by clause alone (without being in the static partition key=value spec list).
src/docs/user/hcatalog.txt (line 129)
<https://reviews.apache.org/r/38400/#comment156065>
I think this is a separate feature by itself that I think should be in a separate JIRA?
src/java/org/apache/sqoop/util/DbUtil.java (line 40)
<https://reviews.apache.org/r/38400/#comment156066>
Not sure why these are needed. Don't we have columninfo queries that support most of the needs including support for column precision?
- Venkat Ranganathan
On Sept. 15, 2015, 11:02 a.m., Rakesh Sharma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38400/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 11:02 a.m.)
>
>
> Review request for Sqoop and Venkat Ranganathan.
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> Review request for support of Dynamic partitons while importing data. Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column.
>
>
> Diffs
> -----
>
> src/docs/user/hcatalog.txt 99ae4f5
> src/java/org/apache/sqoop/SqoopOptions.java ace90fd
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java 878f765
> src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 4070c24
> src/java/org/apache/sqoop/metastore/TableMetaData.java PRE-CREATION
> src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec
> src/java/org/apache/sqoop/tool/ImportTool.java 39af42c
> src/java/org/apache/sqoop/util/DbUtil.java PRE-CREATION
> src/test/org/apache/sqoop/hcat/HCatalogImportTest.java d97b870
>
> Diff: https://reviews.apache.org/r/38400/diff/
>
>
> Testing
> -------
>
> Yes
>
>
> Thanks,
>
> Rakesh Sharma
>
>
Re: Review Request 38400: Review for issue SQOOP-2332: Support dynamic
partitions while sqoop import. Also support partition key formats for date
partitions
Posted by Rakesh Sharma <ra...@expedia.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38400/
-----------------------------------------------------------
(Updated Sept. 15, 2015, 6:02 p.m.)
Review request for Sqoop and Venkat Ranganathan.
Changes
-------
Removing extra whitespaces
Repository: sqoop-trunk
Description
-------
Review request for support of Dynamic partitons while importing data. Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column.
Diffs (updated)
-----
src/docs/user/hcatalog.txt 99ae4f5
src/java/org/apache/sqoop/SqoopOptions.java ace90fd
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java 878f765
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 4070c24
src/java/org/apache/sqoop/metastore/TableMetaData.java PRE-CREATION
src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec
src/java/org/apache/sqoop/tool/ImportTool.java 39af42c
src/java/org/apache/sqoop/util/DbUtil.java PRE-CREATION
src/test/org/apache/sqoop/hcat/HCatalogImportTest.java d97b870
Diff: https://reviews.apache.org/r/38400/diff/
Testing
-------
Yes
Thanks,
Rakesh Sharma