You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2017/05/01 21:13:39 UTC
Re: Review Request 58777: HIVE-16546: LLAP: Fail map join tasks if
hash table memory exceeds threshold
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58777/#review173510
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
Lines 225 (patched)
<https://reviews.apache.org/r/58777/#comment246502>
is it possible to handle 0/negative values differently, to have a complete off switch in case something goes wrong?
- Sergey Shelukhin
On April 28, 2017, 4:53 a.m., Prasanth_J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58777/
> -----------------------------------------------------------
>
> (Updated April 28, 2017, 4:53 a.m.)
>
>
> Review request for hive, Gunther Hagleitner, Sergey Shelukhin, and Siddharth Seth.
>
>
> Bugs: HIVE-16546
> https://issues.apache.org/jira/browse/HIVE-16546
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java PRE-CREATION
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java dbe00b6
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java 7fc3226
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 595d1bd
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java 04e24bd
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java a3bccc6
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java 04e89e8
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java c86e5f5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 6d71fef
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 7b13e90
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 24d3526
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 486d43a
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java 6242daf
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java 1a41961
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java 331867c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java b93e977
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java b6db3bc
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java 49ecdd1
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastKeyStore.java be51693
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java 6fe98f9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java 9140aee
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java d3efb11
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java 8bfa07c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMap.java add4788
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMultiSet.java faefdbb
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashSet.java 5328910
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java f13034f
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java 53ad7b4
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java 723c729
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java 05f1cf1
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastValueStore.java f9c5b34
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java c7e585c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashSet.java 93a89d7
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashTable.java 5fe7861
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java f921b9c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java ad77e87
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java b2893e7
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java d375d1b
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java 93b8a5d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenSparkSkewJoinProcessor.java 405c3ca
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d39b8bd
> ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 940630c
> ql/src/test/org/apache/hadoop/hive/ql/exec/mapjoin/TestMapJoinMemoryExhaustionHandler.java 16b5b17
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java a4ecd9f
>
>
> Diff: https://reviews.apache.org/r/58777/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Prasanth_J
>
>
Re: Review Request 58777: HIVE-16546: LLAP: Fail map join tasks if
hash table memory exceeds threshold
Posted by j....@gmail.com.
> On May 1, 2017, 9:13 p.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
> > Lines 225 (patched)
> > <https://reviews.apache.org/r/58777/diff/4/?file=1702355#file1702355line225>
> >
> > is it possible to handle 0/negative values differently, to have a complete off switch in case something goes wrong?
Setting value of 0 will not completely turn off the check.
- Prasanth_J
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58777/#review173510
-----------------------------------------------------------
On May 2, 2017, 12:22 a.m., Prasanth_J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58777/
> -----------------------------------------------------------
>
> (Updated May 2, 2017, 12:22 a.m.)
>
>
> Review request for hive, Gunther Hagleitner, Sergey Shelukhin, and Siddharth Seth.
>
>
> Bugs: HIVE-16546
> https://issues.apache.org/jira/browse/HIVE-16546
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java PRE-CREATION
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java dbe00b6
> ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java 7fc3226
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 595d1bd
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java 04e24bd
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java a3bccc6
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java 04e89e8
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java c86e5f5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 6d71fef
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 7b13e90
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 486d43a
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java 6242daf
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java 1a41961
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java 331867c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java b93e977
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java b6db3bc
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java 49ecdd1
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastKeyStore.java be51693
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java 6fe98f9
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java 9140aee
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java d3efb11
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java 8bfa07c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMap.java add4788
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMultiSet.java faefdbb
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashSet.java 5328910
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java f13034f
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java 53ad7b4
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java 723c729
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java 05f1cf1
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastValueStore.java f9c5b34
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java c7e585c
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashSet.java 93a89d7
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashTable.java 5fe7861
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java f921b9c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java ad77e87
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java b2893e7
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java d375d1b
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java 93b8a5d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenSparkSkewJoinProcessor.java 405c3ca
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d39b8bd
> ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb
> ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 940630c
> ql/src/test/org/apache/hadoop/hive/ql/exec/mapjoin/TestMapJoinMemoryExhaustionHandler.java 16b5b17
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java a4ecd9f
>
>
> Diff: https://reviews.apache.org/r/58777/diff/5/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Prasanth_J
>
>