You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2017/07/26 21:53:34 UTC
Review Request 61164: HIVE-17006 LLAP: Parquet caching
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/
-----------------------------------------------------------
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
Diffs
-----
common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e
itests/src/test/resources/testconfiguration.properties f66e19be3e
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 42129b7511
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 0cbc8f6f4c
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 35b9d1f942
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab
ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java f4fadbb61b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java 322178a2f7
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java 6a7a219dfe
ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f
ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out PRE-CREATION
storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java PRE-CREATION
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java PRE-CREATION
Diff: https://reviews.apache.org/r/61164/diff/1/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/#review184177
-----------------------------------------------------------
ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out
Lines 1 (patched)
<https://reviews.apache.org/r/61164/#comment260249>
verify that this runs thru the cache if possible
- Sergey Shelukhin
On July 27, 2017, 10:27 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61164/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 10:27 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e
> itests/src/test/resources/testconfiguration.properties f66e19be3e
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 42129b7511
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 0cbc8f6f4c
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 35b9d1f942
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab
> ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java f4fadbb61b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java 322178a2f7
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java 6a7a219dfe
> ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486241
> ql/src/test/queries/clientpositive/parquet_predicate_pushdown.q a38cdbe007
> ql/src/test/queries/clientpositive/parquet_types.q db37d2e1b2
> ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f
> ql/src/test/queries/clientpositive/vectorized_parquet.q e6ebdaac62
> ql/src/test/queries/clientpositive/vectorized_parquet_types.q 7467cb3cf6
> ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out PRE-CREATION
> storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java PRE-CREATION
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/61164/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/
-----------------------------------------------------------
(Updated Aug. 30, 2017, 10:21 p.m.)
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/FileUtils.java e784797f62
itests/src/test/resources/testconfiguration.properties fa6a2aaea0
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 42129b7511
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 0cbc8f6f4c
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapAllocatorBuffer.java 52144c2432
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 35b9d1f942
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java fa7f59d7dc
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 442c921a84
ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java f4fadbb61b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java 322178a2f7
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java 6a7a219dfe
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 8b99ae07ac
ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486241
ql/src/test/queries/clientpositive/parquet_predicate_pushdown.q a38cdbe007
ql/src/test/queries/clientpositive/parquet_types.q db37d2e1b2
ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f
ql/src/test/queries/clientpositive/vectorized_parquet.q db02ec04a5
ql/src/test/queries/clientpositive/vectorized_parquet_types.q 7467cb3cf6
ql/src/test/results/clientpositive/llap/parquet_predicate_pushdown.q.out aecbcfdfe4
ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/vectorized_parquet.q.out 8a84d3dc22
ql/src/test/results/clientpositive/llap/vectorized_parquet_types.q.out cdf6b3d225
storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java PRE-CREATION
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java PRE-CREATION
Diff: https://reviews.apache.org/r/61164/diff/3/
Changes: https://reviews.apache.org/r/61164/diff/2-3/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/#review184175
-----------------------------------------------------------
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
Lines 75 (patched)
<https://reviews.apache.org/r/61164/#comment260240>
comment about not needing to discard
ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java
Lines 60 (patched)
<https://reviews.apache.org/r/61164/#comment260241>
revise comment
ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java
Lines 223 (patched)
<https://reviews.apache.org/r/61164/#comment260242>
should this use "off"?
ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java
Lines 265 (patched)
<https://reviews.apache.org/r/61164/#comment260244>
.
ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java
Lines 307 (patched)
<https://reviews.apache.org/r/61164/#comment260245>
needs to be done?
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
Lines 198 (patched)
<https://reviews.apache.org/r/61164/#comment260246>
follow up
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
Lines 327 (patched)
<https://reviews.apache.org/r/61164/#comment260247>
consider reading into cache, then doing the same thing as above
- Sergey Shelukhin
On July 27, 2017, 10:27 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61164/
> -----------------------------------------------------------
>
> (Updated July 27, 2017, 10:27 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e
> itests/src/test/resources/testconfiguration.properties f66e19be3e
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 42129b7511
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 0cbc8f6f4c
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 35b9d1f942
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab
> ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java f4fadbb61b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java 322178a2f7
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java 6a7a219dfe
> ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486241
> ql/src/test/queries/clientpositive/parquet_predicate_pushdown.q a38cdbe007
> ql/src/test/queries/clientpositive/parquet_types.q db37d2e1b2
> ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f
> ql/src/test/queries/clientpositive/vectorized_parquet.q e6ebdaac62
> ql/src/test/queries/clientpositive/vectorized_parquet_types.q 7467cb3cf6
> ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out PRE-CREATION
> storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java PRE-CREATION
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/61164/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/
-----------------------------------------------------------
(Updated July 27, 2017, 10:27 p.m.)
Review request for hive and Prasanth_J.
Repository: hive-git
Description (updated)
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e
itests/src/test/resources/testconfiguration.properties f66e19be3e
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 42129b7511
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 0cbc8f6f4c
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 35b9d1f942
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab
ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java f4fadbb61b
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java 322178a2f7
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java 6a7a219dfe
ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486241
ql/src/test/queries/clientpositive/parquet_predicate_pushdown.q a38cdbe007
ql/src/test/queries/clientpositive/parquet_types.q db37d2e1b2
ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f
ql/src/test/queries/clientpositive/vectorized_parquet.q e6ebdaac62
ql/src/test/queries/clientpositive/vectorized_parquet_types.q 7467cb3cf6
ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out PRE-CREATION
storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java PRE-CREATION
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java PRE-CREATION
Diff: https://reviews.apache.org/r/61164/diff/2/
Changes: https://reviews.apache.org/r/61164/diff/1-2/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 61164: HIVE-17006 LLAP: Parquet caching
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61164/#review181494
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
Lines 17 (patched)
<https://reviews.apache.org/r/61164/#comment257111>
lots of spurious imports, will remove in the next iter
- Sergey Shelukhin
On July 26, 2017, 9:53 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61164/
> -----------------------------------------------------------
>
> (Updated July 26, 2017, 9:53 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java e8a3a7a49e
> itests/src/test/resources/testconfiguration.properties f66e19be3e
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 42129b7511
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 0cbc8f6f4c
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 35b9d1f942
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java 9b8b76102a
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 21394c6aab
> ql/src/java/org/apache/hadoop/hive/ql/io/LlapCacheOnlyInputFormatInterface.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java f4fadbb61b
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java 322178a2f7
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetFooterInputFromCache.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java 6a7a219dfe
> ql/src/test/queries/clientpositive/parquet_types_vectorization.q bb0e5b258f
> ql/src/test/results/clientpositive/llap/parquet_types_vectorization.q.out PRE-CREATION
> storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java PRE-CREATION
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/MemoryBufferOrBuffers.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/61164/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>