You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Rajesh Balamohan <ra...@gmail.com> on 2016/08/03 07:49:42 UTC
Re: Review Request 49881: HIVE-14204: Optimize loading loaddynamic
partitions
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49881/
-----------------------------------------------------------
(Updated Aug. 3, 2016, 7:49 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
- Removed fetching existing partitions in loadDynamicPartitions. This can be added as a follow on optimization later.
Bugs: HIVE-14204
https://issues.apache.org/jira/browse/HIVE-14204
Repository: hive-git
Description
-------
Lots of time is spent in sequential fashion to load dynamic partitioned dataset in driver side.
E.g simple dynamic partitioned load as follows takes 300+ seconds
INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from tpcds_bin_partitioned_orc_200.web_sales;
Time taken to load dynamic partitions: 309.22 seconds
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa7647b
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 5adfa02
metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java d624d1b
ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java b4ae1d1
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 02c17b5
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd
Diff: https://reviews.apache.org/r/49881/diff/
Testing
-------
Thanks,
Rajesh Balamohan
Re: Review Request 49881: HIVE-14204: Optimize loading loaddynamic
partitions
Posted by Rajesh Balamohan <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49881/
-----------------------------------------------------------
(Updated Aug. 4, 2016, 12:21 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
Rebasing after HIVE-14400
Bugs: HIVE-14204
https://issues.apache.org/jira/browse/HIVE-14204
Repository: hive-git
Description
-------
Lots of time is spent in sequential fashion to load dynamic partitioned dataset in driver side.
E.g simple dynamic partitioned load as follows takes 300+ seconds
INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from tpcds_bin_partitioned_orc_200.web_sales;
Time taken to load dynamic partitions: 309.22 seconds
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9f5f619
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 5adfa02
metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java d624d1b
ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java b4ae1d1
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 02c17b5
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 57433bb
Diff: https://reviews.apache.org/r/49881/diff/
Testing
-------
Thanks,
Rajesh Balamohan