You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2016/03/19 00:18:42 UTC
Review Request 45062: HIVE-13241 LLAP: Incremental Caching marks some
small chunks as "incomplete CB"
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45062/
-----------------------------------------------------------
Review request for hive, Gopal V and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java bae571e
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java c292b37
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java dbee823
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java eb251a8
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java e970137
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 901e58a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 29b51ec
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java ddba889
Diff: https://reviews.apache.org/r/45062/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 45062: HIVE-13241 LLAP: Incremental Caching marks
some small chunks as "incomplete CB"
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On March 22, 2016, 5:09 a.m., Gopal V wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java, line 844
> > <https://reviews.apache.org/r/45062/diff/1/?file=1307057#file1307057line844>
> >
> > is Data ever non-null here?
>
> Sergey Shelukhin wrote:
> Yes, in EncodedReaderImpl
> // 2.5. Remember the bad estimates for future reference.
> if (badEstimates != null && !badEstimates.isEmpty()) {
> // Relies on the fact that cache does not actually store these.
> DiskRange[] cacheKeys = badEstimates.toArray(new DiskRange[badEstimates.size()]);
> long[] result = cacheWrapper.putFileData(fileKey, cacheKeys, null, baseOffset);
> assert result == null; // We don't expect conflicts from bad estimates.
> }
and for non-null:
// 6. Finally, put uncompressed data to cache.
if (fileKey != null) {
long[] collisionMask = cacheWrapper.putFileData(fileKey, cacheKeys, targetBuffers, baseOffset);
processCacheCollisions(collisionMask, toDecompress, targetBuffers, csd.getCacheBuffers());
}
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45062/#review124732
-----------------------------------------------------------
On March 18, 2016, 11:18 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45062/
> -----------------------------------------------------------
>
> (Updated March 18, 2016, 11:18 p.m.)
>
>
> Review request for hive, Gopal V and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java bae571e
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java c292b37
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java dbee823
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java eb251a8
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java e970137
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 901e58a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 29b51ec
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java ddba889
>
> Diff: https://reviews.apache.org/r/45062/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 45062: HIVE-13241 LLAP: Incremental Caching marks
some small chunks as "incomplete CB"
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On March 22, 2016, 5:09 a.m., Gopal V wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java, line 844
> > <https://reviews.apache.org/r/45062/diff/1/?file=1307057#file1307057line844>
> >
> > is Data ever non-null here?
Yes, in EncodedReaderImpl
// 2.5. Remember the bad estimates for future reference.
if (badEstimates != null && !badEstimates.isEmpty()) {
// Relies on the fact that cache does not actually store these.
DiskRange[] cacheKeys = badEstimates.toArray(new DiskRange[badEstimates.size()]);
long[] result = cacheWrapper.putFileData(fileKey, cacheKeys, null, baseOffset);
assert result == null; // We don't expect conflicts from bad estimates.
}
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45062/#review124732
-----------------------------------------------------------
On March 18, 2016, 11:18 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45062/
> -----------------------------------------------------------
>
> (Updated March 18, 2016, 11:18 p.m.)
>
>
> Review request for hive, Gopal V and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java bae571e
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java c292b37
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java dbee823
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java eb251a8
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java e970137
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 901e58a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 29b51ec
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java ddba889
>
> Diff: https://reviews.apache.org/r/45062/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 45062: HIVE-13241 LLAP: Incremental Caching marks
some small chunks as "incomplete CB"
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45062/#review124732
-----------------------------------------------------------
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java (line 844)
<https://reviews.apache.org/r/45062/#comment187381>
is Data ever non-null here?
- Gopal V
On March 18, 2016, 11:18 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45062/
> -----------------------------------------------------------
>
> (Updated March 18, 2016, 11:18 p.m.)
>
>
> Review request for hive, Gopal V and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java bae571e
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java c292b37
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java dbee823
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java eb251a8
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java e970137
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 901e58a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 29b51ec
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java ddba889
>
> Diff: https://reviews.apache.org/r/45062/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 45062: HIVE-13241 LLAP: Incremental Caching marks
some small chunks as "incomplete CB"
Posted by Lefty Leverenz <le...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45062/#review127765
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 2551 - 2554)
<https://reviews.apache.org/r/45062/#comment191147>
It isn't necessary to include "(Hive 2.1?)" – eventually, the description would have to be updated and anyway the release number can be found on HIVE-9660.
Also, please spell out RG. Thanks.
- Lefty Leverenz
On March 18, 2016, 11:18 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45062/
> -----------------------------------------------------------
>
> (Updated March 18, 2016, 11:18 p.m.)
>
>
> Review request for hive, Gopal V and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java bae571e
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapOptionsProcessor.java c292b37
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java dbee823
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java eb251a8
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java e970137
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java 901e58a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 29b51ec
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java ddba889
>
> Diff: https://reviews.apache.org/r/45062/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>