You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "kevinshin (Jira)" <ji...@apache.org> on 2023/03/09 09:35:00 UTC
[jira] [Updated] (SPARK-42728) when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config
[ https://issues.apache.org/jira/browse/SPARK-42728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
kevinshin updated SPARK-42728:
------------------------------
Attachment: image-2023-03-09-17-34-26-428.png
> when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-42728
> URL: https://issues.apache.org/jira/browse/SPARK-42728
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.1, 3.2.3
> Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0
>
> Reporter: kevinshin
> Priority: Minor
> Attachments: image-2023-03-09-17-34-26-428.png
>
>
> spark config :
> spark.sql.adaptive.enabled true
> spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
> spark.sql.adaptive.coalescePartitions.enabled true
> spark.sql.adaptive.coalescePartitions.minPartitionNum 1
> spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
> spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
> spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
> spark.sql.catalog.spark_catalog.type hive
>
> Sql script:
>
> *create* *table* spark_catalog.test.iceberg5(id *bigint*, *data* string) *USING* iceberg;
> *INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 'c');
> *update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;
> *drop* *table* spark_catalog.test.iceberg5 PURGE;
> spark web :
> !image-2023-03-09-17-33-14-940.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org