You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by j....@gmail.com on 2017/08/02 09:57:11 UTC
Review Request 61374: HIVE-17220: Bloomfilter probing in semijoin
reduction is thrashing L1 dcache
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61374/
-----------------------------------------------------------
Review request for hive and Gopal V.
Bugs: HIVE-17220
https://issues.apache.org/jira/browse/HIVE-17220
Repository: hive-git
Description
-------
HIVE-17220: Bloomfilter probing in semijoin reduction is thrashing L1 dcache
Diffs
-----
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/AggrStatsInvalidatorFilter.java 4ca4229acdd2d5e45f1365bbe9e4c8f91d585168
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java 188a87e923ff6dba4484d7029f64f46ad87dc36f
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java 4b3eca093c00ebdfc558bd1d6126cd8fb55533ec
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java 67a7c508a3d12a2236e3b8d1789cb95f131fdb6b
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java 2413ae6d2648ff32c15d621466afb71c84f28ed1
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java 3e6e069548d0a7aefacfe8dba37ec7825889b1cf
storage-api/src/java/org/apache/hive/common/util/BloomFilter.java e9f419dcbe23f5044e2fb936944380c4ba226464
storage-api/src/java/org/apache/hive/common/util/BloomKFilter.java PRE-CREATION
storage-api/src/test/org/apache/hive/common/util/TestBloomFilter.java e4ee93a01433550bfd9c6401ffc61be4d54e645d
storage-api/src/test/org/apache/hive/common/util/TestBloomKFilter.java PRE-CREATION
Diff: https://reviews.apache.org/r/61374/diff/1/
Testing
-------
Thanks,
Prasanth_J
Re: Review Request 61374: HIVE-17220: Bloomfilter probing in semijoin
reduction is thrashing L1 dcache
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61374/#review182078
-----------------------------------------------------------
Ship it!
Ship It!
- Gopal V
On Aug. 3, 2017, 3:09 a.m., Prasanth_J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61374/
> -----------------------------------------------------------
>
> (Updated Aug. 3, 2017, 3:09 a.m.)
>
>
> Review request for hive and Gopal V.
>
>
> Bugs: HIVE-17220
> https://issues.apache.org/jira/browse/HIVE-17220
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-17220: Bloomfilter probing in semijoin reduction is thrashing L1 dcache
>
>
> Diffs
> -----
>
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/AggrStatsInvalidatorFilter.java 4ca4229
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java 188a87e
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java 4b3eca09
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java 67a7c50
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java 2413ae6
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java 3e6e069
> storage-api/src/java/org/apache/hive/common/util/BloomFilter.java e9f419d
> storage-api/src/java/org/apache/hive/common/util/BloomKFilter.java PRE-CREATION
> storage-api/src/test/org/apache/hive/common/util/TestBloomFilter.java e4ee93a
> storage-api/src/test/org/apache/hive/common/util/TestBloomKFilter.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/61374/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Prasanth_J
>
>
Re: Review Request 61374: HIVE-17220: Bloomfilter probing in semijoin
reduction is thrashing L1 dcache
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61374/
-----------------------------------------------------------
(Updated Aug. 3, 2017, 3:09 a.m.)
Review request for hive and Gopal V.
Changes
-------
Addressed Gopal's review comments.
Bugs: HIVE-17220
https://issues.apache.org/jira/browse/HIVE-17220
Repository: hive-git
Description
-------
HIVE-17220: Bloomfilter probing in semijoin reduction is thrashing L1 dcache
Diffs (updated)
-----
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/AggrStatsInvalidatorFilter.java 4ca4229
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java 188a87e
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java 4b3eca09
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java 67a7c50
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java 2413ae6
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java 3e6e069
storage-api/src/java/org/apache/hive/common/util/BloomFilter.java e9f419d
storage-api/src/java/org/apache/hive/common/util/BloomKFilter.java PRE-CREATION
storage-api/src/test/org/apache/hive/common/util/TestBloomFilter.java e4ee93a
storage-api/src/test/org/apache/hive/common/util/TestBloomKFilter.java PRE-CREATION
Diff: https://reviews.apache.org/r/61374/diff/2/
Changes: https://reviews.apache.org/r/61374/diff/1-2/
Testing
-------
Thanks,
Prasanth_J
Re: Review Request 61374: HIVE-17220: Bloomfilter probing in semijoin
reduction is thrashing L1 dcache
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61374/#review182030
-----------------------------------------------------------
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/AggrStatsInvalidatorFilter.java
Line 107 (original), 107 (patched)
<https://reviews.apache.org/r/61374/#comment257840>
.toArray(bitset)?
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java
Line 113 (original), 106 (patched)
<https://reviews.apache.org/r/61374/#comment257842>
use maxEntries?
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java
Line 102 (original), 99 (patched)
<https://reviews.apache.org/r/61374/#comment257843>
NonSyncArrayInputStream?
storage-api/src/java/org/apache/hive/common/util/BloomFilter.java
Line 79 (original), 76 (patched)
<https://reviews.apache.org/r/61374/#comment257844>
leave comment about the lack of a copy - the data structure no longer copies, so any modification to original array affects the BloomFilter.
storage-api/src/java/org/apache/hive/common/util/BloomKFilter.java
Lines 215 (patched)
<https://reviews.apache.org/r/61374/#comment257845>
More comments on the bitPos computation needed to explain this Bloom filter impl.
storage-api/src/java/org/apache/hive/common/util/BloomKFilter.java
Lines 221 (patched)
<https://reviews.apache.org/r/61374/#comment257846>
This won't unroll easily - better to use 1 loop + Arrays fill()
- Gopal V
On Aug. 2, 2017, 9:57 a.m., Prasanth_J wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61374/
> -----------------------------------------------------------
>
> (Updated Aug. 2, 2017, 9:57 a.m.)
>
>
> Review request for hive and Gopal V.
>
>
> Bugs: HIVE-17220
> https://issues.apache.org/jira/browse/HIVE-17220
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-17220: Bloomfilter probing in semijoin reduction is thrashing L1 dcache
>
>
> Diffs
> -----
>
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/AggrStatsInvalidatorFilter.java 4ca4229acdd2d5e45f1365bbe9e4c8f91d585168
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java 188a87e923ff6dba4484d7029f64f46ad87dc36f
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java 4b3eca093c00ebdfc558bd1d6126cd8fb55533ec
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java 67a7c508a3d12a2236e3b8d1789cb95f131fdb6b
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java 2413ae6d2648ff32c15d621466afb71c84f28ed1
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java 3e6e069548d0a7aefacfe8dba37ec7825889b1cf
> storage-api/src/java/org/apache/hive/common/util/BloomFilter.java e9f419dcbe23f5044e2fb936944380c4ba226464
> storage-api/src/java/org/apache/hive/common/util/BloomKFilter.java PRE-CREATION
> storage-api/src/test/org/apache/hive/common/util/TestBloomFilter.java e4ee93a01433550bfd9c6401ffc61be4d54e645d
> storage-api/src/test/org/apache/hive/common/util/TestBloomKFilter.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/61374/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Prasanth_J
>
>