You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Yuming Wang (Jira)" <ji...@apache.org> on 2020/08/16 14:03:00 UTC

[jira] [Created] (SPARK-32628) Use bloom filter to improve dynamicPartitionPruning

Yuming Wang created SPARK-32628:
-----------------------------------

             Summary: Use bloom filter to improve dynamicPartitionPruning
                 Key: SPARK-32628
                 URL: https://issues.apache.org/jira/browse/SPARK-32628
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.1.0
            Reporter: Yuming Wang


It will throw exception when {{spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly}} is disabled:
{code:sql}
select catalog_sales.* from  catalog_sales join catalog_returns  where cr_order_number = cs_sold_date_sk and cr_returned_time_sk < 40000;
{code}
{noformat}
20/08/16 06:44:42 ERROR TaskSetManager: Total size of serialized results of 494 tasks (1225.3 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB)
{noformat}

We can improve it with minimum, maximum and bloom filter to reduce serialized results.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org