You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Krisztian Kasa <kk...@hortonworks.com> on 2019/07/10 12:10:55 UTC
Review Request 71045: HIVE-21948
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71045/
-----------------------------------------------------------
Review request for hive, Jesús Camacho Rodríguez, Zoltan Haindrich, and Vineet Garg.
Bugs: HIVE-21948
https://issues.apache.org/jira/browse/HIVE-21948
Repository: hive-git
Description
-------
Implement parallel processing in Pre Upgrade Tool
=================================================
Process databases and tables paralelly using a thread pool for each. Thread pools size can be defined by command line options. If no options are given the default pool size is the number of cpu cores / 2
Diffs
-----
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CloseableThreadLocal.java PRE-CREATION
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactTablesState.java PRE-CREATION
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactionMetaInfo.java PRE-CREATION
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/NamedForkJoinWorkerThreadFactory.java PRE-CREATION
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java 0a7354d12b
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/RunOptions.java 66213d424c
upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestCloseableThreadLocal.java PRE-CREATION
upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestRunOptions.java PRE-CREATION
Diff: https://reviews.apache.org/r/71045/diff/1/
Testing
-------
TestPreUpgradeTool UT passed.
Manually:
1. deploy ambari and HDP 2.6.5 cluster.
2. create some hive tables and insert/update values
3. run PreUpgradeTool and check logs and output script
Thanks,
Krisztian Kasa
Re: Review Request 71045: HIVE-21948
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71045/#review216544
-----------------------------------------------------------
Ship it!
Ship It!
- Jesús Camacho Rodríguez
On July 11, 2019, 11:32 a.m., Krisztian Kasa wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71045/
> -----------------------------------------------------------
>
> (Updated July 11, 2019, 11:32 a.m.)
>
>
> Review request for hive, Jesús Camacho Rodríguez, Zoltan Haindrich, and Vineet Garg.
>
>
> Bugs: HIVE-21948
> https://issues.apache.org/jira/browse/HIVE-21948
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement parallel processing in Pre Upgrade Tool
> =================================================
> Process tables paralelly using a thread pool for each. Thread pools size can be defined by command line options. If no options are given the default pool size is the number of cpu cores.
>
> I had to remove database parallel processing because the ForkJoinPool implementation does not support nested parallel streams.
>
>
> Diffs
> -----
>
> upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CloseableThreadLocal.java PRE-CREATION
> upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactTablesState.java PRE-CREATION
> upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactionMetaInfo.java PRE-CREATION
> upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/NamedForkJoinWorkerThreadFactory.java PRE-CREATION
> upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java 0a7354d12b
> upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/RunOptions.java 66213d424c
> upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestCloseableThreadLocal.java PRE-CREATION
> upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestRunOptions.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/71045/diff/2/
>
>
> Testing
> -------
>
> TestPreUpgradeTool UT passed.
>
> Manually:
> 1. deploy ambari and HDP 2.6.5 cluster.
> 2. create some hive tables and insert/update values
> 3. run PreUpgradeTool and check logs and output script
>
>
> Thanks,
>
> Krisztian Kasa
>
>
Re: Review Request 71045: HIVE-21948
Posted by Krisztian Kasa <kk...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71045/
-----------------------------------------------------------
(Updated July 11, 2019, 11:32 a.m.)
Review request for hive, Jesús Camacho Rodríguez, Zoltan Haindrich, and Vineet Garg.
Bugs: HIVE-21948
https://issues.apache.org/jira/browse/HIVE-21948
Repository: hive-git
Description (updated)
-------
Implement parallel processing in Pre Upgrade Tool
=================================================
Process tables paralelly using a thread pool for each. Thread pools size can be defined by command line options. If no options are given the default pool size is the number of cpu cores.
I had to remove database parallel processing because the ForkJoinPool implementation does not support nested parallel streams.
Diffs (updated)
-----
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CloseableThreadLocal.java PRE-CREATION
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactTablesState.java PRE-CREATION
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactionMetaInfo.java PRE-CREATION
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/NamedForkJoinWorkerThreadFactory.java PRE-CREATION
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java 0a7354d12b
upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/RunOptions.java 66213d424c
upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestCloseableThreadLocal.java PRE-CREATION
upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestRunOptions.java PRE-CREATION
Diff: https://reviews.apache.org/r/71045/diff/2/
Changes: https://reviews.apache.org/r/71045/diff/1-2/
Testing
-------
TestPreUpgradeTool UT passed.
Manually:
1. deploy ambari and HDP 2.6.5 cluster.
2. create some hive tables and insert/update values
3. run PreUpgradeTool and check logs and output script
Thanks,
Krisztian Kasa