You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "kevinshin (Jira)" <ji...@apache.org> on 2023/03/09 09:34:00 UTC

[jira] [Created] (SPARK-42728) when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config

kevinshin created SPARK-42728:
---------------------------------

             Summary: when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config
                 Key: SPARK-42728
                 URL: https://issues.apache.org/jira/browse/SPARK-42728
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.2.3, 3.2.1
         Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0

 
            Reporter: kevinshin


spark config : 
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
 
Sql script: 
 
*create* *table* spark_catalog.test.iceberg5(id *bigint*, *data* string) *USING* iceberg;

*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 'c');

*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;

*drop* *table* spark_catalog.test.iceberg5 PURGE;

spark web :  

!image-2023-03-09-17-33-14-940.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org