You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "kevinshin (Jira)" <ji...@apache.org> on 2023/03/09 09:34:00 UTC
[jira] [Created] (SPARK-42728) when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config
kevinshin created SPARK-42728:
---------------------------------
Summary: when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config
Key: SPARK-42728
URL: https://issues.apache.org/jira/browse/SPARK-42728
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.3, 3.2.1
Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0
Reporter: kevinshin
spark config :
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
Sql script:
*create* *table* spark_catalog.test.iceberg5(id *bigint*, *data* string) *USING* iceberg;
*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 'c');
*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;
*drop* *table* spark_catalog.test.iceberg5 PURGE;
spark web :
!image-2023-03-09-17-33-14-940.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org