You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Holden Karau (Jira)" <ji...@apache.org> on 2023/07/10 18:05:00 UTC

[jira] [Created] (SPARK-44358) Improve AQE required distribution (sort order) to auto-matically grow scope (or use previous sort)

Holden Karau created SPARK-44358:
------------------------------------

             Summary: Improve AQE required distribution (sort order) to auto-matically grow scope (or use previous sort)
                 Key: SPARK-44358
                 URL: https://issues.apache.org/jira/browse/SPARK-44358
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.5.0
            Reporter: Holden Karau


With new versions of Iceberg and other catalogs, Spark now can have sort order (or distribution) requirements pushed up from the storage level.

 

While sort(a,b,c) does satisfy sort(a,b) we don't automatically extend the scope to sort(a,b,c) even when sort(a,b) does not have sufficient cardinality. This can result in excessively large partitions during writes.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org