You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by pengcheng xiong <px...@hortonworks.com> on 2014/09/11 23:13:48 UTC
Review Request 25557: improve the speed of col stats update speed
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25557/
-----------------------------------------------------------
Review request for hive.
Repository: hive-git
Description
-------
Major improvement
(1) All the partition status update/insert is now done in one transaction.
(2) Rather than to use a query to update per col per partition (total query = #col * # part),
now we use 1 query to delete everything and then use 1 query to insert everything. The transaction makes sure that this happens in ACID mode.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9df6656
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 33745e4
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 59d5244
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 637a39a
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 5c5ed7f
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 5905efe
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
ql/src/test/queries/clientpositive/analyze_tbl_part.q 9040bd4
ql/src/test/results/clientpositive/analyze_tbl_part.q.out 40b926c
Diff: https://reviews.apache.org/r/25557/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 25557: improve the speed of col stats update speed
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25557/
-----------------------------------------------------------
(Updated Oct. 1, 2014, 8:50 p.m.)
Review request for hive and Ashutosh Chauhan.
Repository: hive-git
Description
-------
Major improvement
(1) All the partition status update/insert is now done in one transaction.
(2) Rather than to use a query to update per col per partition (total query = #col * # part),
now we use 1 query to delete everything and then use 1 query to insert everything. The transaction makes sure that this happens in ACID mode.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cfb3f9a
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 33745e4
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5a8591a
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 637a39a
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 5c5ed7f
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 5905efe
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
ql/src/test/queries/clientpositive/analyze_tbl_part.q 9040bd4
ql/src/test/results/clientpositive/analyze_tbl_part.q.out 40b926c
Diff: https://reviews.apache.org/r/25557/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 25557: improve the speed of col stats update speed
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25557/
-----------------------------------------------------------
(Updated Sept. 15, 2014, 9:26 p.m.)
Review request for hive.
Repository: hive-git
Description
-------
Major improvement
(1) All the partition status update/insert is now done in one transaction.
(2) Rather than to use a query to update per col per partition (total query = #col * # part),
now we use 1 query to delete everything and then use 1 query to insert everything. The transaction makes sure that this happens in ACID mode.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cfb3f9a
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 33745e4
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5a8591a
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 637a39a
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 5c5ed7f
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 5905efe
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
ql/src/test/queries/clientpositive/analyze_tbl_part.q 9040bd4
ql/src/test/results/clientpositive/analyze_tbl_part.q.out 40b926c
Diff: https://reviews.apache.org/r/25557/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 25557: improve the speed of col stats update speed
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25557/
-----------------------------------------------------------
(Updated Sept. 12, 2014, 6:47 p.m.)
Review request for hive.
Changes
-------
address case insensitive problem
Repository: hive-git
Description
-------
Major improvement
(1) All the partition status update/insert is now done in one transaction.
(2) Rather than to use a query to update per col per partition (total query = #col * # part),
now we use 1 query to delete everything and then use 1 query to insert everything. The transaction makes sure that this happens in ACID mode.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9df6656
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 33745e4
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5a8591a
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 637a39a
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 5c5ed7f
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 5905efe
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
ql/src/test/queries/clientpositive/analyze_tbl_part.q 9040bd4
ql/src/test/results/clientpositive/analyze_tbl_part.q.out 40b926c
Diff: https://reviews.apache.org/r/25557/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 25557: improve the speed of col stats update speed
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25557/
-----------------------------------------------------------
(Updated Sept. 12, 2014, 3:53 a.m.)
Review request for hive.
Changes
-------
rebase to trunk
Repository: hive-git
Description
-------
Major improvement
(1) All the partition status update/insert is now done in one transaction.
(2) Rather than to use a query to update per col per partition (total query = #col * # part),
now we use 1 query to delete everything and then use 1 query to insert everything. The transaction makes sure that this happens in ACID mode.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9df6656
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 33745e4
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5a8591a
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 637a39a
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 5c5ed7f
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 5905efe
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
ql/src/test/queries/clientpositive/analyze_tbl_part.q 9040bd4
ql/src/test/results/clientpositive/analyze_tbl_part.q.out 40b926c
Diff: https://reviews.apache.org/r/25557/diff/
Testing
-------
Thanks,
pengcheng xiong