You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Vaibhav Gumashta <vg...@hortonworks.com> on 2017/05/11 21:07:59 UTC
Review Request 59205: HIVE-16579: CachedStore: improvements to
partition col
stats caching and cache column stats for unpartitioned table
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73e0290
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 88b9faf
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a83e12e
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c22a1db
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java b438479
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 39b1676
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java f6420f5
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 3e3fd20
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 91d8c2a
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/1/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174841
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Line 906 (original), 906 (patched)
<https://reviews.apache.org/r/59205/#comment248058>
Lets keep default time in seconds, just as before.
- Thejas Nair
On May 11, 2017, 9:07 p.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 11, 2017, 9:07 p.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73e0290
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 88b9faf
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a83e12e
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c22a1db
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java b438479
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 39b1676
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java f6420f5
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 3e3fd20
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 91d8c2a
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review175116
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
Line 266 (original)
<https://reviews.apache.org/r/59205/#comment248479>
don't we need this removePartitionColStatsFromCache call ?
- Thejas Nair
On May 16, 2017, 8:10 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 16, 2017, 8:10 a.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/9/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review175684
-----------------------------------------------------------
Ship it!
Ship It!
- Daniel Dai
On May 22, 2017, 7:06 p.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 22, 2017, 7:06 p.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/19/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 22, 2017, 7:06 p.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/19/
Changes: https://reviews.apache.org/r/59205/diff/18-19/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 22, 2017, 9:04 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7dedd23
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/18/
Changes: https://reviews.apache.org/r/59205/diff/17-18/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 21, 2017, 7:12 p.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Changes
-------
Patch v16 from jira
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/17/
Changes: https://reviews.apache.org/r/59205/diff/16-17/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 19, 2017, 8:41 p.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/16/
Changes: https://reviews.apache.org/r/59205/diff/15-16/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 19, 2017, 2:38 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/15/
Changes: https://reviews.apache.org/r/59205/diff/14-15/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 18, 2017, 7:14 p.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/14/
Changes: https://reviews.apache.org/r/59205/diff/13-14/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review175354
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 827 (patched)
<https://reviews.apache.org/r/59205/#comment248801>
There is no need to alter partition/partition stats/table stats here. HiveAlterHandler will invoke alterPartition, updateTableColumnStatistics and updatePartitionColumnStatistics respectively.
- Daniel Dai
On May 17, 2017, 6:47 p.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 17, 2017, 6:47 p.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/13/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 17, 2017, 6:47 p.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/13/
Changes: https://reviews.apache.org/r/59205/diff/12-13/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 17, 2017, 8:24 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/12/
Changes: https://reviews.apache.org/r/59205/diff/11-12/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 17, 2017, 7:49 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/11/
Changes: https://reviews.apache.org/r/59205/diff/10-11/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 16, 2017, 5:59 p.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/10/
Changes: https://reviews.apache.org/r/59205/diff/9-10/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 16, 2017, 8:10 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1cc838f
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java da6cd46
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java PRE-CREATION
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/9/
Changes: https://reviews.apache.org/r/59205/diff/8-9/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 16, 2017, 3:27 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Changes
-------
Please ignore v8 on review board - generated incorrectly.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6068f0d
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d408321
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/8/
Changes: https://reviews.apache.org/r/59205/diff/7-8/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 16, 2017, 3:25 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6068f0d
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java 6ff48ee
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/src/test/resources/testconfiguration.properties a378a5d
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
itests/util/src/main/java/org/apache/hive/beeline/QFile.java 3d9ca99
itests/util/src/main/java/org/apache/hive/beeline/QFileBeeLineClient.java 7c50e18
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java fcf6f27
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
ql/src/java/org/apache/hadoop/hive/ql/Driver.java f01efa5
ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 6dfaa9f
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java cf575de
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java 77bce97
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java b121eea
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 3b719af
ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java c7266bc
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java c734988
ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBCompact.java 201622e
ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBJoinTreeApplyPredicate.java e607f10
ql/src/test/org/apache/hadoop/hive/ql/parse/TestQBSubQuery.java 2674835
ql/src/test/org/apache/hadoop/hive/ql/parse/TestReplicationSemanticAnalyzer.java 80865bd
ql/src/test/org/apache/hadoop/hive/ql/parse/TestSemanticAnalyzerFactory.java 5849950
ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java a573808
ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java 58cb4b4
ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV1.java 5d01080
ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestPrivilegesV2.java c552ba7
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out PRE-CREATION
service/src/java/org/apache/hive/service/cli/operation/Operation.java 0b27608
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 0b51591
Diff: https://reviews.apache.org/r/59205/diff/7/
Changes: https://reviews.apache.org/r/59205/diff/6-7/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 15, 2017, 10:27 p.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6068f0d
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d408321
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/6/
Changes: https://reviews.apache.org/r/59205/diff/5-6/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Thejas Nair <th...@hortonworks.com>.
> On May 15, 2017, 8:19 a.m., Thejas Nair wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
> > Line 281 (original), 284 (patched)
> > <https://reviews.apache.org/r/59205/diff/2-5/?file=1717962#file1717962line285>
> >
> > This needs to be TimeUnit.MILLISECONDS, as you want it converted to MILLISECONDS
the signature is - getTimeVar(Configuration conf, ConfVars var, TimeUnit outUnit)
- Thejas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174936
-----------------------------------------------------------
On May 15, 2017, 7:59 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 15, 2017, 7:59 a.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/5/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174936
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 281 (original), 284 (patched)
<https://reviews.apache.org/r/59205/#comment248225>
This needs to be TimeUnit.MILLISECONDS, as you want it converted to MILLISECONDS
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 512 (original), 526 (patched)
<https://reviews.apache.org/r/59205/#comment248226>
isDatabaseCacheDirty.set(true); - this needs to be set after obtaining the lock
Otherwise, you can have case where it gets reset by background thread before actual update.
(Same with this call in other places)
- Thejas Nair
On May 15, 2017, 7:59 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 15, 2017, 7:59 a.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/5/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 15, 2017, 7:59 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/5/
Changes: https://reviews.apache.org/r/59205/diff/4-5/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174906
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 341 (patched)
<https://reviews.apache.org/r/59205/#comment248175>
this call has already been made in updateTables. Should we just get it from the cache to reduce the rawStore call ? This can be done as part of follow up jira if necessary.
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 300 (original), 374 (patched)
<https://reviews.apache.org/r/59205/#comment248179>
This still doesn't prevent the race condition I mentioned (in offline discussion).
If another thread calls a method that updates a database object, when this thread is here (let say that calls starts and finishes when this thread is at this line), it would still go ahead and update the cache. That would result in local changes getting lost.
This is not a regression in this patch, we can address that in a follow up patch.
To address that, we need a dirty-cache check after acquring the lock.
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 1131 (original), 1433 (patched)
<https://reviews.apache.org/r/59205/#comment248176>
this would be just like getPartition() . We get it from cache, if its there in cache. Otherwise, assume its not there (yet). This can be done as part of follow up jira if necessary.
- Thejas Nair
On May 14, 2017, 5:19 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 14, 2017, 5:19 a.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174905
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 1387 (patched)
<https://reviews.apache.org/r/59205/#comment248174>
There is no order-by in the SQL query above. Is it guaranteed that the list is ordered ?
Should we add a "order by" to above SQL query ?
- Thejas Nair
On May 14, 2017, 5:19 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 14, 2017, 5:19 a.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Thejas Nair <th...@hortonworks.com>.
> On May 14, 2017, 7:38 a.m., Daniel Dai wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
> > Line 381 (original), 526 (patched)
> > <https://reviews.apache.org/r/59205/diff/4/?file=1717995#file1717995line552>
> >
> > I think we only need a dirty flag, it will be reset by the next refresh, not end of individual call.
I agree, this activeDatabaseUpdateCalls params are not going to prevent the race-condition. Its better to use dirty flag instead.
- Thejas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174907
-----------------------------------------------------------
On May 14, 2017, 5:19 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 14, 2017, 5:19 a.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Thejas Nair <th...@hortonworks.com>.
> On May 14, 2017, 7:38 a.m., Daniel Dai wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
> > Lines 528 (patched)
> > <https://reviews.apache.org/r/59205/diff/4/?file=1717995#file1717995line554>
> >
> > If using dirty flag, databaseCacheLock can be removed, right?
We still need some locking/synchronization to prevent things like this -
1. If (isDirtyFlag == false) {
2. // if db gets updated in another thread while this thread is here, we lose the update.
3. updateDBCache()
2. }
A read/write lock similar to what is used here appropriate for that. The background thread is the one that holds the write lock.
However, not sure if we need to busy-wait with trylock though. (I figure, the intent is to give this one higher priority). The update DbCache part should not take that long. (The busy-wait can be dealt with later also).
- Thejas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174907
-----------------------------------------------------------
On May 14, 2017, 5:19 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 14, 2017, 5:19 a.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Daniel Dai <da...@gmail.com>.
> On May 14, 2017, 7:38 a.m., Daniel Dai wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
> > Lines 528 (patched)
> > <https://reviews.apache.org/r/59205/diff/4/?file=1717995#file1717995line554>
> >
> > If using dirty flag, databaseCacheLock can be removed, right?
>
> Thejas Nair wrote:
> We still need some locking/synchronization to prevent things like this -
>
>
> 1. If (isDirtyFlag == false) {
> 2. // if db gets updated in another thread while this thread is here, we lose the update.
> 3. updateDBCache()
> 2. }
>
> A read/write lock similar to what is used here appropriate for that. The background thread is the one that holds the write lock.
> However, not sure if we need to busy-wait with trylock though. (I figure, the intent is to give this one higher priority). The update DbCache part should not take that long. (The busy-wait can be dealt with later also).
A simpler solution is move dirtyFlag to SharedCache. SharedCache.refreshDatabases will check the flag and SharedCache.addDatabaseToCache/alterDatabaseInCache/removeDatabaseFromCache will set the flag. There is no synchronization issue in SharedCache since every method is synchronized.
- Daniel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174907
-----------------------------------------------------------
On May 14, 2017, 5:19 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 14, 2017, 5:19 a.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review174907
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 381 (original), 526 (patched)
<https://reviews.apache.org/r/59205/#comment248177>
I think we only need a dirty flag, it will be reset by the next refresh, not end of individual call.
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 528 (patched)
<https://reviews.apache.org/r/59205/#comment248178>
If using dirty flag, databaseCacheLock can be removed, right?
- Daniel Dai
On May 14, 2017, 5:19 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> -----------------------------------------------------------
>
> (Updated May 14, 2017, 5:19 a.m.)
>
>
> Review request for hive, Daniel Dai and Thejas Nair.
>
>
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-16579
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
> metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
>
>
> Diff: https://reviews.apache.org/r/59205/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 14, 2017, 5:19 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b897ffa
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/4/
Changes: https://reviews.apache.org/r/59205/diff/3-4/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 14, 2017, 2:21 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/3/
Changes: https://reviews.apache.org/r/59205/diff/2-3/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 59205: HIVE-16579: CachedStore: improvements to
partition
col stats caching and cache column stats for unpartitioned table
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
-----------------------------------------------------------
(Updated May 14, 2017, 1:28 a.m.)
Review request for hive, Daniel Dai and Thejas Nair.
Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-16579
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d6a80ae
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 91a3a38
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java b96c27e
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 870896c
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ed19f42
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 668499b
metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 5a187d8
metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 7beee42
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 0c7d8bb
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f613c30
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1720e37
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 0ab20d6
Diff: https://reviews.apache.org/r/59205/diff/2/
Changes: https://reviews.apache.org/r/59205/diff/1-2/
Testing
-------
Thanks,
Vaibhav Gumashta