You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "László Bodor (Jira)" <ji...@apache.org> on 2020/07/20 07:33:00 UTC
[jira] [Updated] (HIVE-23880) Bloom filters can be merged in a
parallel way in VectorUDAFBloomFilterMerge
[ https://issues.apache.org/jira/browse/HIVE-23880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
László Bodor updated HIVE-23880:
--------------------------------
Description:
Merging bloom filters in semijoin reduction can become the main bottleneck in case of large number of source mapper tasks (~1000) and a large amount of expected entries (50M) in bloom filters.
For example in TPCDS Q93:
{code}
select /*+ semi(store_returns, sr_item_sk, store_sales, 70000000)*/ ss_customer_sk
,sum(act_sales) sumsales
from (select ss_item_sk
,ss_ticket_number
,ss_customer_sk
,case when sr_return_quantity is not null then (ss_quantity-sr_return_quantity)*ss_sales_price
else (ss_quantity*ss_sales_price) end act_sales
from store_sales left outer join store_returns on (sr_item_sk = ss_item_sk
and sr_ticket_number = ss_ticket_number)
,reason
where sr_reason_sk = r_reason_sk
and r_reason_desc = 'reason 66') t
group by ss_customer_sk
order by sumsales, ss_customer_sk
limit 100;
{code}
On 10TB-30TB scale there is a chance that from 3-4 mins of query runtime 1-2 mins are spent with merging bloom filters, as in:
> Bloom filters can be merged in a parallel way in VectorUDAFBloomFilterMerge
> ---------------------------------------------------------------------------
>
> Key: HIVE-23880
> URL: https://issues.apache.org/jira/browse/HIVE-23880
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: lipwig-output3605036885489193068.svg
>
>
> Merging bloom filters in semijoin reduction can become the main bottleneck in case of large number of source mapper tasks (~1000) and a large amount of expected entries (50M) in bloom filters.
> For example in TPCDS Q93:
> {code}
> select /*+ semi(store_returns, sr_item_sk, store_sales, 70000000)*/ ss_customer_sk
> ,sum(act_sales) sumsales
> from (select ss_item_sk
> ,ss_ticket_number
> ,ss_customer_sk
> ,case when sr_return_quantity is not null then (ss_quantity-sr_return_quantity)*ss_sales_price
> else (ss_quantity*ss_sales_price) end act_sales
> from store_sales left outer join store_returns on (sr_item_sk = ss_item_sk
> and sr_ticket_number = ss_ticket_number)
> ,reason
> where sr_reason_sk = r_reason_sk
> and r_reason_desc = 'reason 66') t
> group by ss_customer_sk
> order by sumsales, ss_customer_sk
> limit 100;
> {code}
> On 10TB-30TB scale there is a chance that from 3-4 mins of query runtime 1-2 mins are spent with merging bloom filters, as in:
--
This message was sent by Atlassian Jira
(v8.3.4#803005)