You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Shashank Tandon <st...@expedia.com> on 2015/10/13 10:56:00 UTC

Review Request 39260: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39260/
-----------------------------------------------------------

Review request for Sqoop and Venkat Ranganathan.


Repository: sqoop-trunk


Description
-------

New Review request for support of Dynamic partitons while importing data. This is in addition to the old review  https://reviews.apache.org/r/38400/ already exists, but in the review board submitter cannot be changed so adding another review request to add a new patch for the code review changes.
Functioanlity : Dynamic partition already exists in Sqoop in case of Hive partitioned table should already exists.The new changes will allow to support dynamic partition if Hive partitioned table is not exists.
Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column.


Diffs
-----

  src/docs/user/hcatalog.txt 99ae4f5 
  src/java/org/apache/sqoop/SqoopOptions.java db92b30 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 9d5a9ee 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
  src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 8db06d1 

Diff: https://reviews.apache.org/r/39260/diff/


Testing
-------

Yes


Thanks,

Shashank Tandon


Re: Review Request 39260: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39260/#review104140
-----------------------------------------------------------



src/docs/user/hcatalog.txt (line 115)
<https://reviews.apache.org/r/39260/#comment162408>

    I think we have to change the description a bit - currently it gives the impression that dynamic partitions are not supported.
    
    Can we say something alone the lines of:
    
    This has the list of column to be added as partitioning keys to the created table along with the static partitioning keys and values provided.  For dynamic partitions keys, the data type of the partition key will be forced to be string irrespective of the SQL datatype



src/java/org/apache/sqoop/tool/BaseSqoopTool.java (line 610)
<https://reviews.apache.org/r/39260/#comment162410>

    Please add a space after the '.'



src/test/org/apache/sqoop/hcat/HCatalogImportTest.java (line 277)
<https://reviews.apache.org/r/39260/#comment162409>

    Better to print a log message


- Venkat Ranganathan


On Oct. 13, 2015, 2:42 a.m., Shashank Tandon wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39260/
> -----------------------------------------------------------
> 
> (Updated Oct. 13, 2015, 2:42 a.m.)
> 
> 
> Review request for Sqoop and Venkat Ranganathan.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> New Review request for support of Dynamic partitons while importing data. This is in addition to the old review  https://reviews.apache.org/r/38400/ already exists, but in the review board submitter cannot be changed so adding another review request to add a new patch for the code review changes.
> Functioanlity : Dynamic partition already exists in Sqoop in case of Hive partitioned table already exists.The new changes will allow to support dynamic partition if Hive partitioned table does not exists.
> Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column.
> 
> 
> Diffs
> -----
> 
>   src/docs/user/hcatalog.txt 99ae4f5 
>   src/java/org/apache/sqoop/SqoopOptions.java db92b30 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 9d5a9ee 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
>   src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
>   src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 8db06d1 
> 
> Diff: https://reviews.apache.org/r/39260/diff/
> 
> 
> Testing
> -------
> 
> Yes
> 
> 
> Thanks,
> 
> Shashank Tandon
> 
>


Re: Review Request 39260: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

Posted by Shashank Tandon <st...@expedia.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39260/
-----------------------------------------------------------

(Updated Nov. 4, 2015, 7:04 a.m.)


Review request for Sqoop and Venkat Ranganathan.


Repository: sqoop-trunk


Description
-------

New Review request for support of Dynamic partitons while importing data. This is in addition to the old review  https://reviews.apache.org/r/38400/ already exists, but in the review board submitter cannot be changed so adding another review request to add a new patch for the code review changes.
Functioanlity : Dynamic partition already exists in Sqoop in case of Hive partitioned table already exists.The new changes will allow to support dynamic partition if Hive partitioned table does not exists.
This has the list of column to be added as partitioning keys to the created table along with the static partitioning keys and values provided.  For dynamic partitions keys, the data type of the partition key will be forced to be string irrespective of the SQL datatype
Also added support for date formats if user wants to do partitioning based on a date column.


Diffs (updated)
-----

  src/docs/user/hcatalog.txt 99ae4f5 
  src/java/org/apache/sqoop/SqoopOptions.java db92b30 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 9d5a9ee 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
  src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 8db06d1 

Diff: https://reviews.apache.org/r/39260/diff/


Testing
-------

Yes


Thanks,

Shashank Tandon


Re: Review Request 39260: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39260/#review104651
-----------------------------------------------------------



src/docs/user/hcatalog.txt (line 120)
<https://reviews.apache.org/r/39260/#comment162900>

    I think we have to fix the text here.
    
    Couple of things.   This option allows additional partitioning keys to be specified other than the static partitioning keys.   These additional partitioning keys will be added to the create table statement to hcatalog as partitioning keys.   These key columns should be present in the table otherwise imports will fail.
    The datatype of these columns in the HCatalog table will be string irrespective of the DB column type.
    
    Sqoop has support for dynamic partitions earlier, but was not allowing dynamic partition keys to be provided as part of create hcatalog table



src/test/org/apache/sqoop/hcat/HCatalogImportTest.java (line 272)
<https://reviews.apache.org/r/39260/#comment162899>

    Whitespace - Please remove tabs


- Venkat Ranganathan


On Oct. 26, 2015, 11:33 p.m., Shashank Tandon wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39260/
> -----------------------------------------------------------
> 
> (Updated Oct. 26, 2015, 11:33 p.m.)
> 
> 
> Review request for Sqoop and Venkat Ranganathan.
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> New Review request for support of Dynamic partitons while importing data. This is in addition to the old review  https://reviews.apache.org/r/38400/ already exists, but in the review board submitter cannot be changed so adding another review request to add a new patch for the code review changes.
> Functioanlity : Dynamic partition already exists in Sqoop in case of Hive partitioned table already exists.The new changes will allow to support dynamic partition if Hive partitioned table does not exists.
> This has the list of column to be added as partitioning keys to the created table along with the static partitioning keys and values provided.  For dynamic partitions keys, the data type of the partition key will be forced to be string irrespective of the SQL datatype
> Also added support for date formats if user wants to do partitioning based on a date column.
> 
> 
> Diffs
> -----
> 
>   src/docs/user/hcatalog.txt 99ae4f5 
>   src/java/org/apache/sqoop/SqoopOptions.java db92b30 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 9d5a9ee 
>   src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
>   src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
>   src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 8db06d1 
> 
> Diff: https://reviews.apache.org/r/39260/diff/
> 
> 
> Testing
> -------
> 
> Yes
> 
> 
> Thanks,
> 
> Shashank Tandon
> 
>


Re: Review Request 39260: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

Posted by Shashank Tandon <st...@expedia.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39260/
-----------------------------------------------------------

(Updated Oct. 27, 2015, 6:33 a.m.)


Review request for Sqoop and Venkat Ranganathan.


Repository: sqoop-trunk


Description (updated)
-------

New Review request for support of Dynamic partitons while importing data. This is in addition to the old review  https://reviews.apache.org/r/38400/ already exists, but in the review board submitter cannot be changed so adding another review request to add a new patch for the code review changes.
Functioanlity : Dynamic partition already exists in Sqoop in case of Hive partitioned table already exists.The new changes will allow to support dynamic partition if Hive partitioned table does not exists.
This has the list of column to be added as partitioning keys to the created table along with the static partitioning keys and values provided.  For dynamic partitions keys, the data type of the partition key will be forced to be string irrespective of the SQL datatype
Also added support for date formats if user wants to do partitioning based on a date column.


Diffs (updated)
-----

  src/docs/user/hcatalog.txt 99ae4f5 
  src/java/org/apache/sqoop/SqoopOptions.java db92b30 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 9d5a9ee 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
  src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 8db06d1 

Diff: https://reviews.apache.org/r/39260/diff/


Testing
-------

Yes


Thanks,

Shashank Tandon


Re: Review Request 39260: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

Posted by Shashank Tandon <st...@expedia.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39260/
-----------------------------------------------------------

(Updated Oct. 13, 2015, 9:42 a.m.)


Review request for Sqoop and Venkat Ranganathan.


Repository: sqoop-trunk


Description (updated)
-------

New Review request for support of Dynamic partitons while importing data. This is in addition to the old review  https://reviews.apache.org/r/38400/ already exists, but in the review board submitter cannot be changed so adding another review request to add a new patch for the code review changes.
Functioanlity : Dynamic partition already exists in Sqoop in case of Hive partitioned table already exists.The new changes will allow to support dynamic partition if Hive partitioned table does not exists.
Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column.


Diffs
-----

  src/docs/user/hcatalog.txt 99ae4f5 
  src/java/org/apache/sqoop/SqoopOptions.java db92b30 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 9d5a9ee 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
  src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 8db06d1 

Diff: https://reviews.apache.org/r/39260/diff/


Testing
-------

Yes


Thanks,

Shashank Tandon


Re: Review Request 39260: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

Posted by Shashank Tandon <st...@expedia.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39260/
-----------------------------------------------------------

(Updated Oct. 13, 2015, 9:30 a.m.)


Review request for Sqoop and Venkat Ranganathan.


Repository: sqoop-trunk


Description
-------

New Review request for support of Dynamic partitons while importing data. This is in addition to the old review  https://reviews.apache.org/r/38400/ already exists, but in the review board submitter cannot be changed so adding another review request to add a new patch for the code review changes.
Functioanlity : Dynamic partition already exists in Sqoop in case of Hive partitioned table should already exists.The new changes will allow to support dynamic partition if Hive partitioned table is not exists.
Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column.


Diffs (updated)
-----

  src/docs/user/hcatalog.txt 99ae4f5 
  src/java/org/apache/sqoop/SqoopOptions.java db92b30 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 9d5a9ee 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
  src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 8db06d1 

Diff: https://reviews.apache.org/r/39260/diff/


Testing
-------

Yes


Thanks,

Shashank Tandon


Re: Review Request 39260: Review for issue SQOOP-2332: Support dynamic partitions while sqoop import. Also support partition key formats for date partitions

Posted by Shashank Tandon <st...@expedia.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39260/
-----------------------------------------------------------

(Updated Oct. 13, 2015, 9:14 a.m.)


Review request for Sqoop and Venkat Ranganathan.


Repository: sqoop-trunk


Description
-------

New Review request for support of Dynamic partitons while importing data. This is in addition to the old review  https://reviews.apache.org/r/38400/ already exists, but in the review board submitter cannot be changed so adding another review request to add a new patch for the code review changes.
Functioanlity : Dynamic partition already exists in Sqoop in case of Hive partitioned table should already exists.The new changes will allow to support dynamic partition if Hive partitioned table is not exists.
Users can just tell the list of columns based on which partitioning should happen. Also support for date formats is added if user wants to do partitioning based on a date column.


Diffs (updated)
-----

  src/docs/user/hcatalog.txt 99ae4f5 
  src/java/org/apache/sqoop/SqoopOptions.java db92b30 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java aba2458 
  src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 9d5a9ee 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java a7f6aec 
  src/java/org/apache/sqoop/tool/ImportTool.java 39af42c 
  src/test/org/apache/sqoop/hcat/HCatalogImportTest.java 8db06d1 

Diff: https://reviews.apache.org/r/39260/diff/


Testing
-------

Yes


Thanks,

Shashank Tandon