You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Vaibhav Gumashta <vg...@hortonworks.com> on 2018/04/09 09:57:52 UTC
Review Request 66503: HIVE-19126: CachedStore: Use memory estimation
to limit cache size during prewarm
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
-----------------------------------------------------------
Review request for hive and Thejas Nair.
Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-19126
Diffs
-----
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java c47856de87
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java 995137f967
Diff: https://reviews.apache.org/r/66503/diff/1/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review200780
-----------------------------------------------------------
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Line 308 (original), 305 (patched)
<https://reviews.apache.org/r/66503/#comment281644>
(It would be good to point out that this is based on an estimation, not the exact memory footprint).
The maximum memory in bytes that the cached objects can use. Memory used is calculated based on estimated size of tables and partitions in the cache.
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 165 (patched)
<https://reviews.apache.org/r/66503/#comment281645>
I see LLAP uses HiveConf.getSizeVar(conf, ConfVars.LLAP_IO_MEMORY_MAX_SIZE);
to get a type of size
can you please check if MetastoreConf.getLongVar(conf, ConfVars.CACHED_RAW_STORE_MAX_CACHE_MEMORY) does the conversion to bytes ?
- Thejas Nair
On April 9, 2018, 10:42 p.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> -----------------------------------------------------------
>
> (Updated April 9, 2018, 10:42 p.m.)
>
>
> Review request for hive and Thejas Nair.
>
>
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Diffs
> -----
>
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java c47856de87
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java 940a1bf276
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/66503/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review201706
-----------------------------------------------------------
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
Lines 52 (patched)
<https://reviews.apache.org/r/66503/#comment283258>
standalone-metastore shall not depend on ql.
- Daniel Dai
On April 16, 2018, 9:42 p.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> -----------------------------------------------------------
>
> (Updated April 16, 2018, 9:42 p.m.)
>
>
> Review request for hive and Thejas Nair.
>
>
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Diffs
> -----
>
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1ce86bbdba
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java f007261daf
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java PRE-CREATION
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java d451f966b0
>
>
> Diff: https://reviews.apache.org/r/66503/diff/6/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
-----------------------------------------------------------
(Updated April 16, 2018, 9:42 p.m.)
Review request for hive and Thejas Nair.
Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-19126
Diffs (updated)
-----
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 1ce86bbdba
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java f007261daf
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java PRE-CREATION
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java d451f966b0
Diff: https://reviews.apache.org/r/66503/diff/6/
Changes: https://reviews.apache.org/r/66503/diff/5-6/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review201041
-----------------------------------------------------------
Ship it!
Ship It!
- Thejas Nair
On April 11, 2018, 10:53 p.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> -----------------------------------------------------------
>
> (Updated April 11, 2018, 10:53 p.m.)
>
>
> Review request for hive and Thejas Nair.
>
>
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Diffs
> -----
>
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java c47856de87
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java f007261daf
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/66503/diff/5/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
-----------------------------------------------------------
(Updated April 11, 2018, 10:53 p.m.)
Review request for hive and Thejas Nair.
Changes
-------
Changed some logging to trace level
Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-19126
Diffs (updated)
-----
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java c47856de87
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java f007261daf
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java PRE-CREATION
Diff: https://reviews.apache.org/r/66503/diff/5/
Changes: https://reviews.apache.org/r/66503/diff/4-5/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
-----------------------------------------------------------
(Updated April 11, 2018, 10:38 p.m.)
Review request for hive and Thejas Nair.
Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-19126
Diffs (updated)
-----
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java c47856de87
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java f007261daf
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java PRE-CREATION
Diff: https://reviews.apache.org/r/66503/diff/4/
Changes: https://reviews.apache.org/r/66503/diff/3-4/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review200783
-----------------------------------------------------------
Ship it!
Ship It!
- Thejas Nair
On April 10, 2018, 12:25 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> -----------------------------------------------------------
>
> (Updated April 10, 2018, 12:25 a.m.)
>
>
> Review request for hive and Thejas Nair.
>
>
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Diffs
> -----
>
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java c47856de87
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java 940a1bf276
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/66503/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
-----------------------------------------------------------
(Updated April 10, 2018, 12:25 a.m.)
Review request for hive and Thejas Nair.
Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-19126
Diffs (updated)
-----
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java c47856de87
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java 940a1bf276
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java PRE-CREATION
Diff: https://reviews.apache.org/r/66503/diff/3/
Changes: https://reviews.apache.org/r/66503/diff/2-3/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
-----------------------------------------------------------
(Updated April 9, 2018, 10:42 p.m.)
Review request for hive and Thejas Nair.
Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126
Repository: hive-git
Description
-------
https://issues.apache.org/jira/browse/HIVE-19126
Diffs (updated)
-----
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java c47856de87
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java 940a1bf276
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java PRE-CREATION
Diff: https://reviews.apache.org/r/66503/diff/2/
Changes: https://reviews.apache.org/r/66503/diff/1-2/
Testing
-------
Thanks,
Vaibhav Gumashta
Re: Review Request 66503: HIVE-19126: CachedStore: Use memory
estimation to limit cache size during prewarm
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/#review200741
-----------------------------------------------------------
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Line 217 (original), 222 (patched)
<https://reviews.apache.org/r/66503/#comment281590>
If the catalogs.add(rawStore.getCatalog(catName)) is moved to another line, safer to use {} for a block under for loop.
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
Lines 118 (patched)
<https://reviews.apache.org/r/66503/#comment281591>
I think we can just keep track of size of partitions in it (and perhaps tables) since that should be taking almost all the memory.
I am just worried that SharedCache is more complicated structure, and we might hit bugs around such structures in estimation. There are whole lot of other basic classes that we don't estimate size of.
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 1786 (patched)
<https://reviews.apache.org/r/66503/#comment281593>
In master the Validator impls have been moved to separate classes (recent commit I suppose). Can yo u also please add SizeValidator as a seperate class ?
- Thejas Nair
On April 9, 2018, 9:57 a.m., Vaibhav Gumashta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66503/
> -----------------------------------------------------------
>
> (Updated April 9, 2018, 9:57 a.m.)
>
>
> Review request for hive and Thejas Nair.
>
>
> Bugs: HIVE-19126
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> https://issues.apache.org/jira/browse/HIVE-19126
>
>
> Diffs
> -----
>
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java 6f4ec6f1ea
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java 2f7fa24558
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 0bbaf7e459
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java c47856de87
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 89b400697b
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java 995137f967
>
>
> Diff: https://reviews.apache.org/r/66503/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vaibhav Gumashta
>
>