You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2017/07/25 21:32:33 UTC
Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in
cache can impact heap usage
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/
-----------------------------------------------------------
(Updated July 25, 2017, 9:32 p.m.)
Review request for hive, Gopal V and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java dd9ad71b5a
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 0cbc8f6f4c
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 35b9d1f942
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java a2eb82947f
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java dc053ee7cf
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java b9d7a77d5b
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 601b622b49
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 4565d11988
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 13c7767a3b
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 03a955c6f7
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java de49fc84bb
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 7540e72b53
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 3f5f99ce35
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java cdd58df370
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java d47ba6b31a
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 77b7f5a2f7
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out b799527e30
Diff: https://reviews.apache.org/r/60289/diff/3/
Changes: https://reviews.apache.org/r/60289/diff/2-3/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in
cache can impact heap usage
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On Sept. 8, 2017, 6:24 p.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
> > Lines 52 (patched)
> > <https://reviews.apache.org/r/60289/diff/4/?file=1809787#file1809787line54>
> >
> > This file looks renamed from Parquet to generic MetadataCache but contains OrcSpecific objects. If it is generic remove orc related stuff or rename the class if it is orc specific?
It's a generic class for buffers, and contains some format-specific sub-parts
> On Sept. 8, 2017, 6:24 p.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
> > Lines 106 (patched)
> > <https://reviews.apache.org/r/60289/diff/4/?file=1809787#file1809787line108>
> >
> > why lock and unlock notification back to back?
To "use" the item as far as eviction policy is concerned.
> On Sept. 8, 2017, 6:24 p.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
> > Lines 134 (patched)
> > <https://reviews.apache.org/r/60289/diff/4/?file=1809787#file1809787line136>
> >
> > can you create follow up? will be useful for debugging. or this could be jmx info. something that can be looked easily instead of logs.
HIVE-17524
> On Sept. 8, 2017, 6:24 p.m., Prasanth_J wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
> > Lines 235 (patched)
> > <https://reviews.apache.org/r/60289/diff/4/?file=1809787#file1809787line237>
> >
> > is readFully fixed in hadoop 2.8? if so, now that hive moved to 2.8.0 can that be used here and other places?
We may be running against 2.7; also it anyway does the same thing and requires a cast, so there's no point
> On Sept. 8, 2017, 6:24 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java
> > Lines 148 (patched)
> > <https://reviews.apache.org/r/60289/diff/4/?file=1809792#file1809792line148>
> >
> > May be we should start using TypeDescription everywhere. OrcProto.Type can be huge object when compared to TypeDescription.
added HIVE-17525
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/#review184992
-----------------------------------------------------------
On Sept. 1, 2017, 12:41 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60289/
> -----------------------------------------------------------
>
> (Updated Sept. 1, 2017, 12:41 a.m.)
>
>
> Review request for hive, Gopal V and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java c5248ceb5f
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java f42622b892
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 6edd84b8b0
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b5db3029d1
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java dc053ee7cf
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java b9d7a77d5b
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 601b622b49
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 4565d11988
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java b61a8ca022
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 13c7767a3b
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 03a955c6f7
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 69a9f9f35e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 7540e72b53
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 690cce798e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java cdd58df370
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java d47ba6b31a
> ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19
> ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 77b7f5a2f7
> ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out b799527e30
> storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 403c3ada61
>
>
> Diff: https://reviews.apache.org/r/60289/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in
cache can impact heap usage
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/#review184992
-----------------------------------------------------------
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 52 (patched)
<https://reviews.apache.org/r/60289/#comment261216>
This file looks renamed from Parquet to generic MetadataCache but contains OrcSpecific objects. If it is generic remove orc related stuff or rename the class if it is orc specific?
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 106 (patched)
<https://reviews.apache.org/r/60289/#comment261217>
why lock and unlock notification back to back?
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 134 (patched)
<https://reviews.apache.org/r/60289/#comment261218>
can you create follow up? will be useful for debugging. or this could be jmx info. something that can be looked easily instead of logs.
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 225 (patched)
<https://reviews.apache.org/r/60289/#comment261221>
smallBuffer is not added LlapMetadataBuffers?
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 235 (patched)
<https://reviews.apache.org/r/60289/#comment261222>
is readFully fixed in hadoop 2.8? if so, now that hive moved to 2.8.0 can that be used here and other places?
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 244 (patched)
<https://reviews.apache.org/r/60289/#comment261223>
remove of change to debug/trace log
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java
Lines 148 (patched)
<https://reviews.apache.org/r/60289/#comment261224>
May be we should start using TypeDescription everywhere. OrcProto.Type can be huge object when compared to TypeDescription.
- Prasanth_J
On Sept. 1, 2017, 12:41 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60289/
> -----------------------------------------------------------
>
> (Updated Sept. 1, 2017, 12:41 a.m.)
>
>
> Review request for hive, Gopal V and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java c5248ceb5f
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java f42622b892
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 6edd84b8b0
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b5db3029d1
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java dc053ee7cf
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java b9d7a77d5b
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 601b622b49
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 4565d11988
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java b61a8ca022
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 13c7767a3b
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 03a955c6f7
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 69a9f9f35e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 7540e72b53
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 690cce798e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java cdd58df370
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java d47ba6b31a
> ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19
> ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9
> ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 77b7f5a2f7
> ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out b799527e30
> storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 403c3ada61
>
>
> Diff: https://reviews.apache.org/r/60289/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in
cache can impact heap usage
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/
-----------------------------------------------------------
(Updated Sept. 13, 2017, 1:58 a.m.)
Review request for hive, Gopal V and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 24c5db0e47
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java c5248ceb5f
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java f42622b892
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 6edd84b8b0
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b5db3029d1
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java dc053ee7cf
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java b9d7a77d5b
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 601b622b49
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 4565d11988
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java b61a8ca022
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 13c7767a3b
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 03a955c6f7
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 69a9f9f35e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 7540e72b53
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 690cce798e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java cdd58df370
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java d47ba6b31a
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 77b7f5a2f7
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out b799527e30
storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 403c3ada61
Diff: https://reviews.apache.org/r/60289/diff/5/
Changes: https://reviews.apache.org/r/60289/diff/4-5/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in
cache can impact heap usage
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/
-----------------------------------------------------------
(Updated Sept. 1, 2017, 12:41 a.m.)
Review request for hive, Gopal V and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java c5248ceb5f
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java f42622b892
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 6edd84b8b0
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b5db3029d1
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java dc053ee7cf
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java b9d7a77d5b
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 601b622b49
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 4565d11988
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java b61a8ca022
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java 13c7767a3b
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 03a955c6f7
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 69a9f9f35e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 7540e72b53
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 690cce798e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java cdd58df370
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java d47ba6b31a
ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19
ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9
ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 77b7f5a2f7
ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out b799527e30
storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 403c3ada61
Diff: https://reviews.apache.org/r/60289/diff/4/
Changes: https://reviews.apache.org/r/60289/diff/3-4/
Testing
-------
Thanks,
Sergey Shelukhin