You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2016/03/02 03:07:03 UTC
Review Request 44248: HIVE-12995 LLAP: Synthetic file ids need
collision checks
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44248/
-----------------------------------------------------------
Review request for hive and Gopal V.
Repository: hive-git
Description
-------
see jira
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad
llap-server/src/java/org/apache/hadoop/hive/llap/cache/Cache.java cee23a9
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 17d9fdf
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 1132171
llap-server/src/java/org/apache/hadoop/hive/llap/cache/NoopCache.java d0461e8
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java d2c1907
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java b81e97d
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 18191da
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2597848
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b36cf64
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java 2e4e0c5
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 43c8fb3
orc/src/java/org/apache/orc/FileMetadata.java d63bdcc
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
ql/src/java/org/apache/hadoop/hive/ql/io/SyntheticFileId.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4a27ee7
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 6cec80e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java da673a5a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcCacheKey.java 9a0158e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 246ead6
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java b0ac503
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 9046589
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 3ef7abe
Diff: https://reviews.apache.org/r/44248/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 44248: HIVE-12995 LLAP: Synthetic file ids need
collision checks
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44248/
-----------------------------------------------------------
(Updated March 9, 2016, 2:32 a.m.)
Review request for hive and Gopal V.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5098851
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java d33f724
llap-server/src/java/org/apache/hadoop/hive/llap/cache/Cache.java cee23a9
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 17d9fdf
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 1132171
llap-server/src/java/org/apache/hadoop/hive/llap/cache/NoopCache.java d0461e8
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java d2c1907
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java b81e97d
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 18191da
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2597848
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b36cf64
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java 2e4e0c5
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 43c8fb3
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 8479d22
orc/src/java/org/apache/orc/FileMetadata.java d63bdcc
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
ql/src/java/org/apache/hadoop/hive/ql/io/SyntheticFileId.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4a27ee7
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 6cec80e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java da673a5a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcCacheKey.java 9a0158e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 246ead6
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java b0ac503
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/StreamUtils.java 9ac53af
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 9046589
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 3ef7abe
Diff: https://reviews.apache.org/r/44248/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 44248: HIVE-12995 LLAP: Synthetic file ids need
collision checks
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44248/#review122634
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java (line 48)
<https://reviews.apache.org/r/44248/#comment184743>
equals() need to change to handle the long to Long.
- Gopal V
On March 3, 2016, 11:20 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44248/
> -----------------------------------------------------------
>
> (Updated March 3, 2016, 11:20 p.m.)
>
>
> Review request for hive and Gopal V.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java d33f724
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/Cache.java cee23a9
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 17d9fdf
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 1132171
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/NoopCache.java d0461e8
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java d2c1907
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java b81e97d
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 18191da
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2597848
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b36cf64
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java 2e4e0c5
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 43c8fb3
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 8479d22
> orc/src/java/org/apache/orc/FileMetadata.java d63bdcc
> ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
> ql/src/java/org/apache/hadoop/hive/ql/io/SyntheticFileId.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4a27ee7
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 6cec80e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java da673a5a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcCacheKey.java 9a0158e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 246ead6
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java b0ac503
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
> storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 9046589
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 3ef7abe
>
> Diff: https://reviews.apache.org/r/44248/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 44248: HIVE-12995 LLAP: Synthetic file ids need
collision checks
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44248/#review122464
-----------------------------------------------------------
Fix it, then Ship it!
Comments mostly about mixed use of fileId fileKey - standardize to fileKey everywhere?
orc/src/java/org/apache/orc/FileMetadata.java (line 47)
<https://reviews.apache.org/r/44248/#comment184461>
Change to fileKey?
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java (line 98)
<https://reviews.apache.org/r/44248/#comment184462>
fileKey?
- Gopal V
On March 3, 2016, 11:20 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44248/
> -----------------------------------------------------------
>
> (Updated March 3, 2016, 11:20 p.m.)
>
>
> Review request for hive and Gopal V.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad
> llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java d33f724
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/Cache.java cee23a9
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 17d9fdf
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 1132171
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/NoopCache.java d0461e8
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java d2c1907
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java b81e97d
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 18191da
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2597848
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b36cf64
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java 2e4e0c5
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 43c8fb3
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 8479d22
> orc/src/java/org/apache/orc/FileMetadata.java d63bdcc
> ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
> ql/src/java/org/apache/hadoop/hive/ql/io/SyntheticFileId.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4a27ee7
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 6cec80e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java da673a5a
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcCacheKey.java 9a0158e
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 246ead6
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java b0ac503
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
> storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 9046589
> storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 3ef7abe
>
> Diff: https://reviews.apache.org/r/44248/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 44248: HIVE-12995 LLAP: Synthetic file ids need
collision checks
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44248/
-----------------------------------------------------------
(Updated March 3, 2016, 11:20 p.m.)
Review request for hive and Gopal V.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2723dad
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java d33f724
llap-server/src/java/org/apache/hadoop/hive/llap/cache/Cache.java cee23a9
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 17d9fdf
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 1132171
llap-server/src/java/org/apache/hadoop/hive/llap/cache/NoopCache.java d0461e8
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java d2c1907
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java b81e97d
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java 18191da
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java 2597848
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b36cf64
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java 2e4e0c5
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java 43c8fb3
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java 8479d22
orc/src/java/org/apache/orc/FileMetadata.java d63bdcc
ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java af64fc8
ql/src/java/org/apache/hadoop/hive/ql/io/SyntheticFileId.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 0ebcd2a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 4a27ee7
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 6cec80e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcBatchKey.java da673a5a
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/OrcCacheKey.java 9a0158e
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 246ead6
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java b0ac503
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 4fafe8c
storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 9046589
storage-api/src/java/org/apache/hadoop/hive/common/io/encoded/EncodedColumnBatch.java 3ef7abe
Diff: https://reviews.apache.org/r/44248/diff/
Testing
-------
Thanks,
Sergey Shelukhin