You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/22 17:35:27 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-5677-b2d06787824baeb08ebb2893678f7710615488db created (now e879dec10b)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-5677-b2d06787824baeb08ebb2893678f7710615488db
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at e879dec10b test: run the Iceberg Spark tests with the native Iceberg writer enabled (#5677)
This branch includes the following new commits:
new e879dec10b test: run the Iceberg Spark tests with the native Iceberg writer enabled (#5677)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: test: run the Iceberg Spark tests with the native Iceberg writer enabled (#5677)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-5677-b2d06787824baeb08ebb2893678f7710615488db
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit e879dec10bd825deb5bd1974daf98878a0240dc2
Author: Andy Grove <ag...@apache.org>
AuthorDate: Tue Sep 22 17:35:05 2026 +0000
test: run the Iceberg Spark tests with the native Iceberg writer enabled (#5677)
* test: enable the Iceberg split-operator and native write by default
Both toggles have defaulted to false since they landed, so no CI job has ever
exercised the two-operator plan or the iceberg-rust writer outside the handful of
suites that set them explicitly. Flip the defaults to find out what the Iceberg
Spark SQL suites and Comet's own Iceberg suites say when every write goes through
them.
Exploratory: `iceberg-writes.md` still documents both as off by default and is
deliberately not updated here.
* test: enable local table scan in the Iceberg test diffs
The Iceberg test sessions already configure the write split operator, but
they leave `spark.comet.exec.localTableScan.enabled` off. Spark emits a
row-based `LocalTableScanExec` for an inline `VALUES` list, and
`CometIcebergNativeWrite` sets `requiresNativeChildren`, so those writes are
declined and silently run on the JVM writer. Iceberg's suites seed data that
way constantly, so the native writer stayed hidden from most of the write
surface even with `spark.comet.iceberg.write.enabled` on.
Measured on Spark 4.1.3 with the Iceberg 1.11 runtime and the Iceberg CI
session config: unpartitioned `INSERT INTO ... VALUES` and
`df.writeTo(...).append()` both selected the JVM writer before this change
and the native writer after it.
Regenerated all four diffs against their own tags per the contributor guide.
Also document the flag and the existing `spark.comet.explainFallback.enabled`
setting, which reports every declined operator with its reason, and drop the
stale claim that the split-operator flag is off by default.
* test: pin the JVM Iceberg writer where tests assert its plan or file layout
CometIcebergWriteActionSuite's two-op tests relied on the native writer
defaulting to off. Pin it off in the suite's conf, and have
withNativeEnabled restore the previous values instead of unsetting them,
which would fall through to the new default.
The page-skipping scan test's byte bounds assume parquet-mr's layout,
which drops dictionary encoding for high-cardinality columns. The native
writer keeps a ~2 MB dictionary page per column, so write its fixture on
the JVM.
* test: enable the native Iceberg writer only in the Iceberg test diffs
Restore the experimental split-operator and native write configs to off by
default, and turn the native writer on in every Comet-configured session of
the Iceberg Spark test diffs instead. Revert the test pins that only existed
because the defaults had flipped.
---
dev/diffs/iceberg/1.10.0.diff | 398 +++++++++++++-------
dev/diffs/iceberg/1.11.0.diff | 204 +++++++----
dev/diffs/iceberg/1.8.1.diff | 402 ++++++++++++++-------
dev/diffs/iceberg/1.9.1.diff | 402 ++++++++++++++-------
.../contributor-guide/iceberg-spark-tests.md | 9 +
5 files changed, 932 insertions(+), 483 deletions(-)
diff --git a/dev/diffs/iceberg/1.10.0.diff b/dev/diffs/iceberg/1.10.0.diff
index 7bbf584c31..2a52ef5ce3 100644
--- a/dev/diffs/iceberg/1.10.0.diff
+++ b/dev/diffs/iceberg/1.10.0.diff
@@ -24,10 +24,10 @@ index 714be0831d..dede65f213 100644
// runtime dependencies for running Hive Catalog based integration test
integrationRuntimeOnly project(':iceberg-hive-metastore')
diff --git a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-index 4c1a509591..186a264a88 100644
+index 4c1a509591..d19f8f0e7d 100644
--- a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
+++ b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-@@ -59,7 +59,25 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
+@@ -59,7 +59,29 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
.config(
SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(), String.valueOf(RANDOM.nextBoolean()))
@@ -38,6 +38,8 @@ index 4c1a509591..186a264a88 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -48,16 +50,18 @@ index 4c1a509591..186a264a88 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
TestBase.sparkContext = JavaSparkContext.fromSparkContext(spark.sparkContext());
diff --git a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-index 58d054bd05..ef1c1cb6d1 100644
+index 58d054bd05..30efbc7f29 100644
--- a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
+++ b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-@@ -56,6 +56,15 @@ public class TestCallStatementParser {
+@@ -56,6 +56,17 @@ public class TestCallStatementParser {
.master("local[2]")
.config("spark.sql.extensions", IcebergSparkSessionExtensions.class.getName())
.config("spark.extra.prop", "value")
@@ -68,6 +72,8 @@ index 58d054bd05..ef1c1cb6d1 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -102,10 +108,10 @@ index 5a0a04edba..afe92a8b72 100644
}
diff --git a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-index b6ade2bff3..90a4839dad 100644
+index b6ade2bff3..60f220dabb 100644
--- a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
+++ b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-@@ -170,6 +170,15 @@ public class DeleteOrphanFilesBenchmark {
+@@ -170,6 +170,17 @@ public class DeleteOrphanFilesBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", catalogWarehouse())
@@ -116,16 +122,18 @@ index b6ade2bff3..90a4839dad 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local");
spark = builder.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-index 7cd41bd7e7..0b6d490529 100644
+index 7cd41bd7e7..7aac36bb7e 100644
--- a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
+++ b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-@@ -386,6 +386,15 @@ public class IcebergSortCompactionBenchmark {
+@@ -386,6 +386,17 @@ public class IcebergSortCompactionBenchmark {
"spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", getCatalogWarehouse())
@@ -136,16 +144,18 @@ index 7cd41bd7e7..0b6d490529 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]");
spark = builder.getOrCreate();
Configuration sparkHadoopConf = spark.sessionState().newHadoopConf();
diff --git a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-index 68c537e34a..2e421c7706 100644
+index 68c537e34a..30ab50d0c7 100644
--- a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
+++ b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-@@ -94,7 +94,18 @@ public abstract class IcebergSourceBenchmark {
+@@ -94,7 +94,20 @@ public abstract class IcebergSourceBenchmark {
}
protected void setupSpark(boolean enableDictionaryEncoding) {
@@ -160,6 +170,8 @@ index 68c537e34a..2e421c7706 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g");
if (!enableDictionaryEncoding) {
@@ -947,10 +959,10 @@ index a0f45e7610..78e7845911 100644
.filter(residual)
.caseSensitive(caseSensitive())
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-index 404ba72846..b375458f05 100644
+index 404ba72846..d53a3051a8 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-@@ -90,6 +90,15 @@ public abstract class SparkDistributedDataScanTestBase
+@@ -90,6 +90,17 @@ public abstract class SparkDistributedDataScanTestBase
.master("local[2]")
.config("spark.serializer", serializer)
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -961,16 +973,18 @@ index 404ba72846..b375458f05 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-index 9361c63176..16ce1c614f 100644
+index 9361c63176..9b48408755 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-@@ -69,6 +69,15 @@ public class TestSparkDistributedDataScanDeletes
+@@ -69,6 +69,17 @@ public class TestSparkDistributedDataScanDeletes
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -981,16 +995,18 @@ index 9361c63176..16ce1c614f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-index a218f965ea..e30a8d6c6f 100644
+index a218f965ea..82e7521a90 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-@@ -62,6 +62,15 @@ public class TestSparkDistributedDataScanFilterFiles
+@@ -62,6 +62,17 @@ public class TestSparkDistributedDataScanFilterFiles
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -1001,16 +1017,18 @@ index a218f965ea..e30a8d6c6f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-index acd4688440..33732e6654 100644
+index acd4688440..4dcb8c797b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-@@ -59,6 +59,15 @@ public class TestSparkDistributedDataScanReporting
+@@ -59,6 +59,17 @@ public class TestSparkDistributedDataScanReporting
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -1021,16 +1039,18 @@ index acd4688440..33732e6654 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/TestBase.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-index 3c32b46936..2a39c4605c 100644
+index 3c32b46936..b6e7d42fbb 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-@@ -79,6 +79,15 @@ public abstract class TestBase extends SparkTestHelperBase {
+@@ -79,6 +79,17 @@ public abstract class TestBase extends SparkTestHelperBase {
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
@@ -1041,16 +1061,18 @@ index 3c32b46936..2a39c4605c 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-index 6647a1b483..aeb16f85ac 100644
+index 6647a1b483..a55437bbb6 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-@@ -62,6 +62,15 @@ public abstract class ScanTestBase extends AvroDataTestBase {
+@@ -62,6 +62,17 @@ public abstract class ScanTestBase extends AvroDataTestBase {
SparkSession.builder()
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
.master("local[2]")
@@ -1061,16 +1083,18 @@ index 6647a1b483..aeb16f85ac 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
ScanTestBase.sc = JavaSparkContext.fromSparkContext(spark.sparkContext());
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-index 24a14bb64d..9329cd99b3 100644
+index 24a14bb64d..ede156ca43 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-@@ -144,7 +144,19 @@ public class TestCompressionSettings extends CatalogTestBase {
+@@ -144,7 +144,21 @@ public class TestCompressionSettings extends CatalogTestBase {
@BeforeAll
public static void startSpark() {
@@ -1085,6 +1109,8 @@ index 24a14bb64d..9329cd99b3 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1092,10 +1118,10 @@ index 24a14bb64d..9329cd99b3 100644
@BeforeEach
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-index 99d3f38ee7..f1a74280dd 100644
+index 99d3f38ee7..552cb0e1a8 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-@@ -74,7 +74,19 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
+@@ -74,7 +74,21 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
@BeforeAll
public static void startSpark() {
@@ -1110,6 +1136,8 @@ index 99d3f38ee7..f1a74280dd 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1117,10 +1145,10 @@ index 99d3f38ee7..f1a74280dd 100644
@AfterAll
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-index a70aeb1ba6..fae2359d01 100644
+index a70aeb1ba6..85924c34ce 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-@@ -123,6 +123,15 @@ public class TestFilteredScan {
+@@ -123,6 +123,17 @@ public class TestFilteredScan {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1131,16 +1159,18 @@ index a70aeb1ba6..fae2359d01 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
// define UDFs used by partition tests
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-index c03f7b94ec..30032fa107 100644
+index c03f7b94ec..0173be37aa 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-@@ -99,6 +99,15 @@ public class TestForwardCompatibility {
+@@ -99,6 +99,17 @@ public class TestForwardCompatibility {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1151,16 +1181,18 @@ index c03f7b94ec..30032fa107 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-index f4f57157e4..418364e3da 100644
+index f4f57157e4..51f25b8a29 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-@@ -51,6 +51,15 @@ public class TestIcebergSpark {
+@@ -51,6 +51,17 @@ public class TestIcebergSpark {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1171,16 +1203,18 @@ index f4f57157e4..418364e3da 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-index e1402396fa..3092902873 100644
+index e1402396fa..988fce91d8 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-@@ -118,6 +118,15 @@ public class TestPartitionPruning {
+@@ -118,6 +118,17 @@ public class TestPartitionPruning {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1191,16 +1225,18 @@ index e1402396fa..3092902873 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
TestPartitionPruning.sparkContext = JavaSparkContext.fromSparkContext(spark.sparkContext());
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-index b0ad930487..6764bf645a 100644
+index b0ad930487..0eacc04158 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-@@ -112,6 +112,15 @@ public class TestPartitionValues {
+@@ -112,6 +112,17 @@ public class TestPartitionValues {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1211,16 +1247,18 @@ index b0ad930487..6764bf645a 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-index 11865db7fc..c7c1e6a02f 100644
+index 11865db7fc..1cf9aba797 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-@@ -91,6 +91,15 @@ public class TestSnapshotSelection {
+@@ -91,6 +91,17 @@ public class TestSnapshotSelection {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1231,16 +1269,18 @@ index 11865db7fc..c7c1e6a02f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-index 833a0b6b5e..eb29e23aac 100644
+index 833a0b6b5e..d362c953ca 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-@@ -125,6 +125,15 @@ public class TestSparkDataFile {
+@@ -125,6 +125,17 @@ public class TestSparkDataFile {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1251,16 +1291,18 @@ index 833a0b6b5e..eb29e23aac 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
TestSparkDataFile.sparkContext = JavaSparkContext.fromSparkContext(spark.sparkContext());
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-index 8884654fe0..fde607663e 100644
+index 8884654fe0..ad97512f17 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-@@ -100,6 +100,15 @@ public class TestSparkDataWrite {
+@@ -100,6 +100,17 @@ public class TestSparkDataWrite {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1271,12 +1313,14 @@ index 8884654fe0..fde607663e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
-@@ -144,7 +153,7 @@ public class TestSparkDataWrite {
+@@ -144,7 +155,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1285,7 +1329,7 @@ index 8884654fe0..fde607663e 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
for (ManifestFile manifest :
SnapshotUtil.latestSnapshot(table, branch).allManifests(table.io())) {
-@@ -213,7 +222,7 @@ public class TestSparkDataWrite {
+@@ -213,7 +224,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1294,7 +1338,7 @@ index 8884654fe0..fde607663e 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -258,7 +267,7 @@ public class TestSparkDataWrite {
+@@ -258,7 +269,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1303,7 +1347,7 @@ index 8884654fe0..fde607663e 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -310,7 +319,7 @@ public class TestSparkDataWrite {
+@@ -310,7 +321,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1312,7 +1356,7 @@ index 8884654fe0..fde607663e 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -352,7 +361,7 @@ public class TestSparkDataWrite {
+@@ -352,7 +363,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1321,7 +1365,7 @@ index 8884654fe0..fde607663e 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -391,7 +400,7 @@ public class TestSparkDataWrite {
+@@ -391,7 +402,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1330,7 +1374,7 @@ index 8884654fe0..fde607663e 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
List<DataFile> files = Lists.newArrayList();
-@@ -459,7 +468,7 @@ public class TestSparkDataWrite {
+@@ -459,7 +470,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1339,7 +1383,7 @@ index 8884654fe0..fde607663e 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -707,7 +716,7 @@ public class TestSparkDataWrite {
+@@ -707,7 +718,7 @@ public class TestSparkDataWrite {
// Since write and commit succeeded, the rows should be readable
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1349,10 +1393,10 @@ index 8884654fe0..fde607663e 100644
.hasSize(records.size() + records2.size())
.containsExactlyInAnyOrder(
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-index 596d05d30b..acbacf65e0 100644
+index 596d05d30b..b4dea5fc6b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-@@ -88,6 +88,15 @@ public class TestSparkReadProjection extends TestReadProjection {
+@@ -88,6 +88,17 @@ public class TestSparkReadProjection extends TestReadProjection {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1363,16 +1407,18 @@ index 596d05d30b..acbacf65e0 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
ImmutableMap<String, String> config =
ImmutableMap.of(
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-index 16fa726032..ce27c5cd4d 100644
+index 16fa726032..70d38c02d3 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-@@ -132,7 +132,25 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -132,7 +132,29 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
.config("spark.ui.liveUpdate.period", 0)
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -1383,6 +1429,8 @@ index 16fa726032..ce27c5cd4d 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1393,12 +1441,14 @@ index 16fa726032..ce27c5cd4d 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
catalog =
-@@ -200,7 +218,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -200,7 +222,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
}
protected boolean countDeletes() {
@@ -1409,10 +1459,10 @@ index 16fa726032..ce27c5cd4d 100644
@Override
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-index baf7fa8f88..fb677d7114 100644
+index baf7fa8f88..5dcc93cdf8 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-@@ -182,7 +182,25 @@ public class TestSparkReaderWithBloomFilter {
+@@ -182,7 +182,29 @@ public class TestSparkReaderWithBloomFilter {
SparkSession.builder()
.master("local[2]")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -1423,6 +1473,8 @@ index baf7fa8f88..fb677d7114 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1433,16 +1485,18 @@ index baf7fa8f88..fb677d7114 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
catalog =
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-index dc4fc7e187..2ab6dd12c2 100644
+index dc4fc7e187..7991d061c0 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-@@ -69,6 +69,15 @@ public class TestStructuredStreaming {
+@@ -69,6 +69,17 @@ public class TestStructuredStreaming {
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
.config("spark.sql.shuffle.partitions", 4)
@@ -1453,16 +1507,18 @@ index dc4fc7e187..2ab6dd12c2 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-index 8b1e3fbfc7..c519a2be0f 100644
+index 8b1e3fbfc7..683bc7f908 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-@@ -75,7 +75,19 @@ public class TestTimestampWithoutZone extends TestBase {
+@@ -75,7 +75,21 @@ public class TestTimestampWithoutZone extends TestBase {
@BeforeAll
public static void startSpark() {
@@ -1477,6 +1533,8 @@ index 8b1e3fbfc7..c519a2be0f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1484,10 +1542,10 @@ index 8b1e3fbfc7..c519a2be0f 100644
@AfterAll
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-index c3fac70dd3..d3ffd034d2 100644
+index c3fac70dd3..a66d3c6c2c 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-@@ -84,6 +84,15 @@ public class TestWriteMetricsConfig {
+@@ -84,6 +84,17 @@ public class TestWriteMetricsConfig {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -1498,16 +1556,18 @@ index c3fac70dd3..d3ffd034d2 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
TestWriteMetricsConfig.sc = JavaSparkContext.fromSparkContext(spark.sparkContext());
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-index 5ce56b4fec..3b614e3fc9 100644
+index 5ce56b4fec..08cd27d83b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-@@ -63,7 +63,25 @@ public class TestAggregatePushDown extends CatalogTestBase {
+@@ -63,7 +63,29 @@ public class TestAggregatePushDown extends CatalogTestBase {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.iceberg.aggregate_pushdown", "true")
@@ -1518,6 +1578,8 @@ index 5ce56b4fec..3b614e3fc9 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1528,6 +1590,8 @@ index 5ce56b4fec..3b614e3fc9 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -1561,10 +1625,10 @@ index 69700d8436..49ea338a45 100644
// runtime dependencies for running Hive Catalog based integration test
integrationRuntimeOnly project(':iceberg-hive-metastore')
diff --git a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-index 4c1a509591..3b4e44cc13 100644
+index 4c1a509591..948766e45f 100644
--- a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
+++ b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-@@ -59,6 +59,15 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
+@@ -59,6 +59,17 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
.config(
SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(), String.valueOf(RANDOM.nextBoolean()))
@@ -1575,6 +1639,8 @@ index 4c1a509591..3b4e44cc13 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1595,10 +1661,10 @@ index 893f9931cf..17e5fef217 100644
protected void createTableWithDeleteGranularity(
diff --git a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-index ecf9e6f8a5..beca5f8521 100644
+index ecf9e6f8a5..14181c55c9 100644
--- a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
+++ b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-@@ -56,6 +56,15 @@ public class TestCallStatementParser {
+@@ -56,6 +56,17 @@ public class TestCallStatementParser {
.master("local[2]")
.config("spark.sql.extensions", IcebergSparkSessionExtensions.class.getName())
.config("spark.extra.prop", "value")
@@ -1609,6 +1675,8 @@ index ecf9e6f8a5..beca5f8521 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -1643,10 +1711,10 @@ index 8cb92224fc..777b9bb1dd 100644
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-index 64edb1002e..57767da086 100644
+index 64edb1002e..b9f137443a 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-@@ -179,6 +179,15 @@ public class DeleteOrphanFilesBenchmark {
+@@ -179,6 +179,17 @@ public class DeleteOrphanFilesBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", catalogWarehouse())
@@ -1657,16 +1725,18 @@ index 64edb1002e..57767da086 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local");
spark = builder.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-index 77b79384a6..8708ffa72a 100644
+index 77b79384a6..6773af5e86 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-@@ -392,6 +392,15 @@ public class IcebergSortCompactionBenchmark {
+@@ -392,6 +392,17 @@ public class IcebergSortCompactionBenchmark {
"spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", getCatalogWarehouse())
@@ -1677,16 +1747,18 @@ index 77b79384a6..8708ffa72a 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]");
spark = builder.getOrCreate();
Configuration sparkHadoopConf = spark.sessionState().newHadoopConf();
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
-index c6794e43c6..4c1a165a44 100644
+index c6794e43c6..8f683c8fc9 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
-@@ -239,6 +239,15 @@ public class DVReaderBenchmark {
+@@ -239,6 +239,17 @@ public class DVReaderBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", newWarehouseDir())
@@ -1697,16 +1769,18 @@ index c6794e43c6..4c1a165a44 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
-index ac74fb5a10..4aa5d5b361 100644
+index ac74fb5a10..ef736f66a3 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
-@@ -223,6 +223,15 @@ public class DVWriterBenchmark {
+@@ -223,6 +223,17 @@ public class DVWriterBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", newWarehouseDir())
@@ -1717,16 +1791,18 @@ index ac74fb5a10..4aa5d5b361 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-index 68c537e34a..2e421c7706 100644
+index 68c537e34a..30ab50d0c7 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-@@ -94,7 +94,18 @@ public abstract class IcebergSourceBenchmark {
+@@ -94,7 +94,20 @@ public abstract class IcebergSourceBenchmark {
}
protected void setupSpark(boolean enableDictionaryEncoding) {
@@ -1741,6 +1817,8 @@ index 68c537e34a..2e421c7706 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g");
if (!enableDictionaryEncoding) {
@@ -2528,10 +2606,10 @@ index a0f45e7610..78e7845911 100644
.filter(residual)
.caseSensitive(caseSensitive())
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-index 404ba72846..b375458f05 100644
+index 404ba72846..d53a3051a8 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-@@ -90,6 +90,15 @@ public abstract class SparkDistributedDataScanTestBase
+@@ -90,6 +90,17 @@ public abstract class SparkDistributedDataScanTestBase
.master("local[2]")
.config("spark.serializer", serializer)
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2542,16 +2620,18 @@ index 404ba72846..b375458f05 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-index 659507e4c5..2a3a87f079 100644
+index 659507e4c5..67a3765dca 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-@@ -73,6 +73,15 @@ public class TestSparkDistributedDataScanDeletes
+@@ -73,6 +73,17 @@ public class TestSparkDistributedDataScanDeletes
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2562,16 +2642,18 @@ index 659507e4c5..2a3a87f079 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-index a218f965ea..e30a8d6c6f 100644
+index a218f965ea..82e7521a90 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-@@ -62,6 +62,15 @@ public class TestSparkDistributedDataScanFilterFiles
+@@ -62,6 +62,17 @@ public class TestSparkDistributedDataScanFilterFiles
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2582,16 +2664,18 @@ index a218f965ea..e30a8d6c6f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-index 2665d7ba8d..c24c82341a 100644
+index 2665d7ba8d..0f43057229 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-@@ -63,6 +63,15 @@ public class TestSparkDistributedDataScanReporting
+@@ -63,6 +63,17 @@ public class TestSparkDistributedDataScanReporting
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2602,16 +2686,18 @@ index 2665d7ba8d..c24c82341a 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-index daf4e29ac0..235cf60557 100644
+index daf4e29ac0..e4a43af910 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-@@ -79,6 +79,15 @@ public abstract class TestBase extends SparkTestHelperBase {
+@@ -79,6 +79,17 @@ public abstract class TestBase extends SparkTestHelperBase {
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
@@ -2622,6 +2708,8 @@ index daf4e29ac0..235cf60557 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -2653,10 +2741,10 @@ index 0db6a65fd3..9ee8b9e084 100644
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetDictionaryEncodedVectorizedReads.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetDictionaryEncodedVectorizedReads.java
-index 973a17c9a3..4e4d3c02f1 100644
+index 973a17c9a3..2d1c2efb82 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetDictionaryEncodedVectorizedReads.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetDictionaryEncodedVectorizedReads.java
-@@ -65,6 +65,15 @@ public class TestParquetDictionaryEncodedVectorizedReads extends TestParquetVect
+@@ -65,6 +65,17 @@ public class TestParquetDictionaryEncodedVectorizedReads extends TestParquetVect
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2667,16 +2755,18 @@ index 973a17c9a3..4e4d3c02f1 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-index 1c5905744a..b9cf548290 100644
+index 1c5905744a..710c61c5f6 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-@@ -61,6 +61,15 @@ public abstract class ScanTestBase extends AvroDataTestBase {
+@@ -61,6 +61,17 @@ public abstract class ScanTestBase extends AvroDataTestBase {
ScanTestBase.spark =
SparkSession.builder()
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2687,16 +2777,18 @@ index 1c5905744a..b9cf548290 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[2]")
.getOrCreate();
ScanTestBase.sc = JavaSparkContext.fromSparkContext(spark.sparkContext());
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-index 19ec6d13dd..c5f73bf666 100644
+index 19ec6d13dd..f80066a775 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-@@ -144,7 +144,19 @@ public class TestCompressionSettings extends CatalogTestBase {
+@@ -144,7 +144,21 @@ public class TestCompressionSettings extends CatalogTestBase {
@BeforeAll
public static void startSpark() {
@@ -2711,6 +2803,8 @@ index 19ec6d13dd..c5f73bf666 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2718,10 +2812,10 @@ index 19ec6d13dd..c5f73bf666 100644
@BeforeEach
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-index a7702b169a..67fabf3cdc 100644
+index a7702b169a..c93a61e3e5 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-@@ -74,7 +74,19 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
+@@ -74,7 +74,21 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
@BeforeAll
public static void startSpark() {
@@ -2736,6 +2830,8 @@ index a7702b169a..67fabf3cdc 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2743,10 +2839,10 @@ index a7702b169a..67fabf3cdc 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-index fd7d52178f..36831cfb07 100644
+index fd7d52178f..6cd26df171 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-@@ -114,6 +114,15 @@ public class TestFilteredScan {
+@@ -114,6 +114,17 @@ public class TestFilteredScan {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2757,16 +2853,18 @@ index fd7d52178f..36831cfb07 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-index 153564f7d1..973eac1878 100644
+index 153564f7d1..c215a010f7 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-@@ -98,6 +98,15 @@ public class TestForwardCompatibility {
+@@ -98,6 +98,17 @@ public class TestForwardCompatibility {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2777,16 +2875,18 @@ index 153564f7d1..973eac1878 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-index f4f57157e4..418364e3da 100644
+index f4f57157e4..51f25b8a29 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-@@ -51,6 +51,15 @@ public class TestIcebergSpark {
+@@ -51,6 +51,17 @@ public class TestIcebergSpark {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2797,16 +2897,18 @@ index f4f57157e4..418364e3da 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-index e1402396fa..3092902873 100644
+index e1402396fa..988fce91d8 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-@@ -118,6 +118,15 @@ public class TestPartitionPruning {
+@@ -118,6 +118,17 @@ public class TestPartitionPruning {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2817,16 +2919,18 @@ index e1402396fa..3092902873 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
TestPartitionPruning.sparkContext = JavaSparkContext.fromSparkContext(spark.sparkContext());
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-index 0b6ab2052b..8682b4a7da 100644
+index 0b6ab2052b..f61e00992c 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-@@ -112,6 +112,15 @@ public class TestPartitionValues {
+@@ -112,6 +112,17 @@ public class TestPartitionValues {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2837,16 +2941,18 @@ index 0b6ab2052b..8682b4a7da 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-index 11865db7fc..c7c1e6a02f 100644
+index 11865db7fc..1cf9aba797 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-@@ -91,6 +91,15 @@ public class TestSnapshotSelection {
+@@ -91,6 +91,17 @@ public class TestSnapshotSelection {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2857,16 +2963,18 @@ index 11865db7fc..c7c1e6a02f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-index 3051e27d72..cb3011b222 100644
+index 3051e27d72..aac8a717be 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-@@ -125,6 +125,15 @@ public class TestSparkDataFile {
+@@ -125,6 +125,17 @@ public class TestSparkDataFile {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2877,16 +2985,18 @@ index 3051e27d72..cb3011b222 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
TestSparkDataFile.sparkContext = JavaSparkContext.fromSparkContext(spark.sparkContext());
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-index 4ccbf86f12..f915f22b21 100644
+index 4ccbf86f12..6841909944 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-@@ -100,6 +100,15 @@ public class TestSparkDataWrite {
+@@ -100,6 +100,17 @@ public class TestSparkDataWrite {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2897,12 +3007,14 @@ index 4ccbf86f12..f915f22b21 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
-@@ -144,7 +153,7 @@ public class TestSparkDataWrite {
+@@ -144,7 +155,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -2911,7 +3023,7 @@ index 4ccbf86f12..f915f22b21 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
for (ManifestFile manifest :
SnapshotUtil.latestSnapshot(table, branch).allManifests(table.io())) {
-@@ -213,7 +222,7 @@ public class TestSparkDataWrite {
+@@ -213,7 +224,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -2920,7 +3032,7 @@ index 4ccbf86f12..f915f22b21 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -258,7 +267,7 @@ public class TestSparkDataWrite {
+@@ -258,7 +269,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -2929,7 +3041,7 @@ index 4ccbf86f12..f915f22b21 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -310,7 +319,7 @@ public class TestSparkDataWrite {
+@@ -310,7 +321,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -2938,7 +3050,7 @@ index 4ccbf86f12..f915f22b21 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -352,7 +361,7 @@ public class TestSparkDataWrite {
+@@ -352,7 +363,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -2947,7 +3059,7 @@ index 4ccbf86f12..f915f22b21 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -391,7 +400,7 @@ public class TestSparkDataWrite {
+@@ -391,7 +402,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -2956,7 +3068,7 @@ index 4ccbf86f12..f915f22b21 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
List<DataFile> files = Lists.newArrayList();
-@@ -459,7 +468,7 @@ public class TestSparkDataWrite {
+@@ -459,7 +470,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -2965,7 +3077,7 @@ index 4ccbf86f12..f915f22b21 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -706,7 +715,7 @@ public class TestSparkDataWrite {
+@@ -706,7 +717,7 @@ public class TestSparkDataWrite {
// Since write and commit succeeded, the rows should be readable
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -2975,10 +3087,10 @@ index 4ccbf86f12..f915f22b21 100644
.hasSize(records.size() + records2.size())
.containsExactlyInAnyOrder(
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-index 596d05d30b..acbacf65e0 100644
+index 596d05d30b..b4dea5fc6b 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-@@ -88,6 +88,15 @@ public class TestSparkReadProjection extends TestReadProjection {
+@@ -88,6 +88,17 @@ public class TestSparkReadProjection extends TestReadProjection {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -2989,16 +3101,18 @@ index 596d05d30b..acbacf65e0 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
ImmutableMap<String, String> config =
ImmutableMap.of(
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-index 42699f4662..b8bf79886e 100644
+index 42699f4662..7544d94720 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-@@ -138,6 +138,15 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -138,6 +138,17 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
.config("spark.ui.liveUpdate.period", 0)
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -3009,12 +3123,14 @@ index 42699f4662..b8bf79886e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
-@@ -206,7 +215,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -206,7 +217,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
}
protected boolean countDeletes() {
@@ -3025,10 +3141,10 @@ index 42699f4662..b8bf79886e 100644
@Override
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-index baf7fa8f88..b58321b761 100644
+index baf7fa8f88..19e8ec83ba 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-@@ -182,6 +182,15 @@ public class TestSparkReaderWithBloomFilter {
+@@ -182,6 +182,17 @@ public class TestSparkReaderWithBloomFilter {
SparkSession.builder()
.master("local[2]")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -3039,16 +3155,18 @@ index baf7fa8f88..b58321b761 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-index 54048bbf21..2ebe9ab993 100644
+index 54048bbf21..b5ef214cb8 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-@@ -69,6 +69,15 @@ public class TestStructuredStreaming {
+@@ -69,6 +69,17 @@ public class TestStructuredStreaming {
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
.config("spark.sql.shuffle.partitions", 4)
@@ -3059,16 +3177,18 @@ index 54048bbf21..2ebe9ab993 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-index 8b1e3fbfc7..c519a2be0f 100644
+index 8b1e3fbfc7..683bc7f908 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-@@ -75,7 +75,19 @@ public class TestTimestampWithoutZone extends TestBase {
+@@ -75,7 +75,21 @@ public class TestTimestampWithoutZone extends TestBase {
@BeforeAll
public static void startSpark() {
@@ -3083,6 +3203,8 @@ index 8b1e3fbfc7..c519a2be0f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3090,10 +3212,10 @@ index 8b1e3fbfc7..c519a2be0f 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-index c3fac70dd3..d3ffd034d2 100644
+index c3fac70dd3..a66d3c6c2c 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-@@ -84,6 +84,15 @@ public class TestWriteMetricsConfig {
+@@ -84,6 +84,17 @@ public class TestWriteMetricsConfig {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -3104,16 +3226,18 @@ index c3fac70dd3..d3ffd034d2 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
TestWriteMetricsConfig.sc = JavaSparkContext.fromSparkContext(spark.sparkContext());
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-index 5ce56b4fec..d089cc1f69 100644
+index 5ce56b4fec..d160ddbfa2 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-@@ -63,6 +63,15 @@ public class TestAggregatePushDown extends CatalogTestBase {
+@@ -63,6 +63,17 @@ public class TestAggregatePushDown extends CatalogTestBase {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.iceberg.aggregate_pushdown", "true")
@@ -3124,6 +3248,8 @@ index 5ce56b4fec..d089cc1f69 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
diff --git a/dev/diffs/iceberg/1.11.0.diff b/dev/diffs/iceberg/1.11.0.diff
index 555002b51f..84490ccf0b 100644
--- a/dev/diffs/iceberg/1.11.0.diff
+++ b/dev/diffs/iceberg/1.11.0.diff
@@ -39,10 +39,10 @@ index e6455fa34f..08b685f61a 100644
// runtime dependencies for running Hive Catalog based integration test
integrationRuntimeOnly project(':iceberg-hive-metastore')
diff --git a/spark/v4.1/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java b/spark/v4.1/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-index f766fbb79a..59e4e6c285 100644
+index f766fbb79a..9424644328 100644
--- a/spark/v4.1/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
+++ b/spark/v4.1/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-@@ -71,6 +71,15 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
+@@ -71,6 +71,17 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
.config(
SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(), String.valueOf(RANDOM.nextBoolean()))
@@ -53,6 +53,8 @@ index f766fbb79a..59e4e6c285 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
@@ -101,10 +103,10 @@ index 934220e5d3..92132d4b8d 100644
}
diff --git a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-index 3fd84553f0..db71a30730 100644
+index 3fd84553f0..ef1f2a0f08 100644
--- a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
+++ b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-@@ -180,6 +180,15 @@ public class DeleteOrphanFilesBenchmark {
+@@ -180,6 +180,17 @@ public class DeleteOrphanFilesBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", catalogWarehouse())
@@ -115,16 +117,18 @@ index 3fd84553f0..db71a30730 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.master("local");
spark = builder.getOrCreate();
diff --git a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-index 683f6bb46d..bd5ef5d3c0 100644
+index 683f6bb46d..7c0a62700a 100644
--- a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
+++ b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-@@ -395,6 +395,15 @@ public class IcebergSortCompactionBenchmark {
+@@ -395,6 +395,17 @@ public class IcebergSortCompactionBenchmark {
"spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", getCatalogWarehouse())
@@ -135,16 +139,18 @@ index 683f6bb46d..bd5ef5d3c0 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.master("local[*]");
spark = builder.getOrCreate();
diff --git a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
-index 3f242ce228..ee19cc1248 100644
+index 3f242ce228..bdc5606af2 100644
--- a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
+++ b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
-@@ -240,6 +240,15 @@ public class DVReaderBenchmark {
+@@ -240,6 +240,17 @@ public class DVReaderBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", newWarehouseDir())
@@ -155,16 +161,18 @@ index 3f242ce228..ee19cc1248 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]")
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
-index db57897240..f3b47a62f6 100644
+index db57897240..397cfdf0ad 100644
--- a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
+++ b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
-@@ -224,6 +224,15 @@ public class DVWriterBenchmark {
+@@ -224,6 +224,17 @@ public class DVWriterBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", newWarehouseDir())
@@ -175,16 +183,18 @@ index db57897240..f3b47a62f6 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]")
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-index debe37866f..f48e355a4c 100644
+index debe37866f..f315197e7f 100644
--- a/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
+++ b/spark/v4.1/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-@@ -95,7 +95,18 @@ public abstract class IcebergSourceBenchmark {
+@@ -95,7 +95,20 @@ public abstract class IcebergSourceBenchmark {
}
protected void setupSpark(boolean enableDictionaryEncoding) {
@@ -199,16 +209,18 @@ index debe37866f..f48e355a4c 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g");
if (!enableDictionaryEncoding) {
builder
.config("parquet.dictionary.page.size", "1")
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-index d1c724425c..da7450dd95 100644
+index d1c724425c..9d6b8f99f1 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-@@ -90,6 +90,15 @@ public abstract class SparkDistributedDataScanTestBase
+@@ -90,6 +90,17 @@ public abstract class SparkDistributedDataScanTestBase
.master("local[2]")
.config("spark.serializer", serializer)
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -219,16 +231,18 @@ index d1c724425c..da7450dd95 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-index a21c6a08ec..104bfcd580 100644
+index a21c6a08ec..fe96bab0b4 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-@@ -73,6 +73,15 @@ public class TestSparkDistributedDataScanDeletes
+@@ -73,6 +73,17 @@ public class TestSparkDistributedDataScanDeletes
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -239,16 +253,18 @@ index a21c6a08ec..104bfcd580 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-index 5edf482822..24bd0ee141 100644
+index 5edf482822..b848d1fe53 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-@@ -62,6 +62,15 @@ public class TestSparkDistributedDataScanFilterFiles
+@@ -62,6 +62,17 @@ public class TestSparkDistributedDataScanFilterFiles
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -259,16 +275,18 @@ index 5edf482822..24bd0ee141 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-index e6f3c75475..6e2a7c6fb7 100644
+index e6f3c75475..8d82667b6f 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-@@ -63,6 +63,15 @@ public class TestSparkDistributedDataScanReporting
+@@ -63,6 +63,17 @@ public class TestSparkDistributedDataScanReporting
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -279,16 +297,18 @@ index e6f3c75475..6e2a7c6fb7 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/TestBase.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-index 507d7b313b..8fbdb40228 100644
+index 507d7b313b..268bbc0b0e 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-@@ -86,6 +86,15 @@ public abstract class TestBase extends SparkTestHelperBase {
+@@ -86,6 +86,17 @@ public abstract class TestBase extends SparkTestHelperBase {
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
@@ -299,6 +319,8 @@ index 507d7b313b..8fbdb40228 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config("spark.ui.enabled", "false")
@@ -330,10 +352,10 @@ index 8ce60f6275..75112b4354 100644
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetDictionaryEncodedVectorizedReads.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetDictionaryEncodedVectorizedReads.java
-index b61ecfa2f4..a3aafae4ee 100644
+index b61ecfa2f4..1b30884b1d 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetDictionaryEncodedVectorizedReads.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/data/vectorized/parquet/TestParquetDictionaryEncodedVectorizedReads.java
-@@ -66,6 +66,15 @@ public class TestParquetDictionaryEncodedVectorizedReads extends TestParquetVect
+@@ -66,6 +66,17 @@ public class TestParquetDictionaryEncodedVectorizedReads extends TestParquetVect
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -344,16 +366,18 @@ index b61ecfa2f4..a3aafae4ee 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-index da9cd63921..c713c27456 100644
+index da9cd63921..b5c5a2387f 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-@@ -62,6 +62,15 @@ public abstract class ScanTestBase extends AvroDataTestBase {
+@@ -62,6 +62,17 @@ public abstract class ScanTestBase extends AvroDataTestBase {
ScanTestBase.spark =
SparkSession.builder()
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -364,16 +388,18 @@ index da9cd63921..c713c27456 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[2]")
.config(TestBase.DISABLE_UI)
.getOrCreate();
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-index d381822483..f090b5bf14 100644
+index d381822483..4fba640639 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-@@ -144,7 +144,19 @@ public class TestCompressionSettings extends CatalogTestBase {
+@@ -144,7 +144,21 @@ public class TestCompressionSettings extends CatalogTestBase {
@BeforeAll
public static void startSpark() {
@@ -388,6 +414,8 @@ index d381822483..f090b5bf14 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -395,10 +423,10 @@ index d381822483..f090b5bf14 100644
@BeforeEach
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-index e67ec5fd62..ef19a1d349 100644
+index e67ec5fd62..c14eb05166 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-@@ -79,7 +79,19 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
+@@ -79,7 +79,21 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
@BeforeAll
public static void startSpark() {
@@ -413,6 +441,8 @@ index e67ec5fd62..ef19a1d349 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -420,10 +450,10 @@ index e67ec5fd62..ef19a1d349 100644
@AfterAll
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-index 24fecf4eb2..964a5bc003 100644
+index 24fecf4eb2..d0dd617525 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-@@ -117,6 +117,15 @@ public class TestFilteredScan {
+@@ -117,6 +117,17 @@ public class TestFilteredScan {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -434,16 +464,18 @@ index 24fecf4eb2..964a5bc003 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-index d0103ff46e..c634782764 100644
+index d0103ff46e..88beab8704 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-@@ -99,6 +99,15 @@ public class TestForwardCompatibility {
+@@ -99,6 +99,17 @@ public class TestForwardCompatibility {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -454,16 +486,18 @@ index d0103ff46e..c634782764 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-index a637b975fe..dc31aab58b 100644
+index a637b975fe..51efc3f9d3 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-@@ -52,6 +52,15 @@ public class TestIcebergSpark {
+@@ -52,6 +52,17 @@ public class TestIcebergSpark {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -474,16 +508,18 @@ index a637b975fe..dc31aab58b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-index 8098db81f9..a3b023a43e 100644
+index 8098db81f9..38f931b4e7 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-@@ -120,6 +120,15 @@ public class TestPartitionPruning {
+@@ -120,6 +120,17 @@ public class TestPartitionPruning {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -494,16 +530,18 @@ index 8098db81f9..a3b023a43e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
TestPartitionPruning.sparkContext = JavaSparkContext.fromSparkContext(spark.sparkContext());
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-index 9b5b22a73f..4aed7f06dc 100644
+index 9b5b22a73f..4b0e8f7ecd 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-@@ -113,6 +113,15 @@ public class TestPartitionValues {
+@@ -113,6 +113,17 @@ public class TestPartitionValues {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -514,16 +552,18 @@ index 9b5b22a73f..4aed7f06dc 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-index 3004e8fa5c..82ebbe9e81 100644
+index 3004e8fa5c..a113ecc1a7 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-@@ -98,6 +98,15 @@ public class TestSnapshotSelection {
+@@ -98,6 +98,17 @@ public class TestSnapshotSelection {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -534,16 +574,18 @@ index 3004e8fa5c..82ebbe9e81 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-index d719ca6751..00ec4f9cdf 100644
+index d719ca6751..2facababc7 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-@@ -127,6 +127,15 @@ public class TestSparkDataFile {
+@@ -127,6 +127,17 @@ public class TestSparkDataFile {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -554,16 +596,18 @@ index d719ca6751..00ec4f9cdf 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
TestSparkDataFile.sparkContext = JavaSparkContext.fromSparkContext(spark.sparkContext());
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-index c2f5afef0e..d970524cd8 100644
+index c2f5afef0e..debb42d975 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-@@ -104,6 +104,15 @@ public class TestSparkDataWrite {
+@@ -104,6 +104,17 @@ public class TestSparkDataWrite {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -574,12 +618,14 @@ index c2f5afef0e..d970524cd8 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
-@@ -149,7 +158,7 @@ public class TestSparkDataWrite {
+@@ -149,7 +160,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -588,7 +634,7 @@ index c2f5afef0e..d970524cd8 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
for (ManifestFile manifest :
SnapshotUtil.latestSnapshot(table, branch).allManifests(table.io())) {
-@@ -219,7 +228,7 @@ public class TestSparkDataWrite {
+@@ -219,7 +230,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -597,7 +643,7 @@ index c2f5afef0e..d970524cd8 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -264,7 +273,7 @@ public class TestSparkDataWrite {
+@@ -264,7 +275,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -606,7 +652,7 @@ index c2f5afef0e..d970524cd8 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -316,7 +325,7 @@ public class TestSparkDataWrite {
+@@ -316,7 +327,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -615,7 +661,7 @@ index c2f5afef0e..d970524cd8 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -358,7 +367,7 @@ public class TestSparkDataWrite {
+@@ -358,7 +369,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -624,7 +670,7 @@ index c2f5afef0e..d970524cd8 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -397,7 +406,7 @@ public class TestSparkDataWrite {
+@@ -397,7 +408,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -633,7 +679,7 @@ index c2f5afef0e..d970524cd8 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
List<DataFile> files = Lists.newArrayList();
-@@ -461,7 +470,7 @@ public class TestSparkDataWrite {
+@@ -461,7 +472,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -642,7 +688,7 @@ index c2f5afef0e..d970524cd8 100644
assertThat(actual).hasSameSizeAs(expected).isEqualTo(expected);
}
-@@ -814,7 +823,7 @@ public class TestSparkDataWrite {
+@@ -814,7 +825,7 @@ public class TestSparkDataWrite {
// Since write and commit succeeded, the rows should be readable
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -652,10 +698,10 @@ index c2f5afef0e..d970524cd8 100644
.hasSize(records.size() + records2.size())
.containsExactlyInAnyOrder(
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-index de6a5e5902..e98a15fe39 100644
+index de6a5e5902..c93329e595 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-@@ -89,6 +89,15 @@ public class TestSparkReadProjection extends TestReadProjection {
+@@ -89,6 +89,17 @@ public class TestSparkReadProjection extends TestReadProjection {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -666,16 +712,18 @@ index de6a5e5902..e98a15fe39 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
ImmutableMap<String, String> config =
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-index 0d61930571..0a6208cc21 100644
+index 0d61930571..e50e17ccce 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-@@ -138,6 +138,15 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -138,6 +138,17 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
.config("spark.ui.liveUpdate.period", 0)
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -686,12 +734,14 @@ index 0d61930571..0a6208cc21 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.enableHiveSupport()
.getOrCreate();
-@@ -210,7 +219,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -210,7 +221,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
}
protected boolean countDeletes() {
@@ -702,10 +752,10 @@ index 0d61930571..0a6208cc21 100644
@Override
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-index cb2f866fab..02c42b7299 100644
+index cb2f866fab..dce617701e 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-@@ -183,6 +183,15 @@ public class TestSparkReaderWithBloomFilter {
+@@ -183,6 +183,17 @@ public class TestSparkReaderWithBloomFilter {
SparkSession.builder()
.master("local[2]")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -716,16 +766,18 @@ index cb2f866fab..02c42b7299 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.enableHiveSupport()
.getOrCreate();
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-index 5e900ea0ba..2f7d6a08a1 100644
+index 5e900ea0ba..97ad872ce0 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-@@ -73,6 +73,15 @@ public class TestStructuredStreaming {
+@@ -73,6 +73,17 @@ public class TestStructuredStreaming {
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
.config("spark.sql.shuffle.partitions", 4)
@@ -736,16 +788,18 @@ index 5e900ea0ba..2f7d6a08a1 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-index 79781a8fc3..08bfa8098b 100644
+index 79781a8fc3..0c10694523 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-@@ -75,7 +75,19 @@ public class TestTimestampWithoutZone extends TestBase {
+@@ -75,7 +75,21 @@ public class TestTimestampWithoutZone extends TestBase {
@BeforeAll
public static void startSpark() {
@@ -760,6 +814,8 @@ index 79781a8fc3..08bfa8098b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -767,10 +823,10 @@ index 79781a8fc3..08bfa8098b 100644
@AfterAll
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-index ab2479d610..49383ca4a6 100644
+index ab2479d610..08a08034c9 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-@@ -86,6 +86,15 @@ public class TestWriteMetricsConfig {
+@@ -86,6 +86,17 @@ public class TestWriteMetricsConfig {
SparkSession.builder()
.master("local[2]")
.config("spark.driver.host", InetAddress.getLoopbackAddress().getHostAddress())
@@ -781,16 +837,18 @@ index ab2479d610..49383ca4a6 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
.getOrCreate();
TestWriteMetricsConfig.sc = JavaSparkContext.fromSparkContext(spark.sparkContext());
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-index 1669301d2d..422c7e4ace 100644
+index 1669301d2d..5400555465 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-@@ -63,6 +63,15 @@ public class TestAggregatePushDown extends CatalogTestBase {
+@@ -63,6 +63,17 @@ public class TestAggregatePushDown extends CatalogTestBase {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.iceberg.aggregate_pushdown", "true")
@@ -801,6 +859,8 @@ index 1669301d2d..422c7e4ace 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.config(TestBase.DISABLE_UI)
@@ -824,10 +884,10 @@ index e5a9d63b68..220d445851 100644
assertThat(planAsString).as("Should be no post scan filter").doesNotContain("Filter (");
}
diff --git a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/variant/TestVariantShredding.java b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/variant/TestVariantShredding.java
-index 8cdcf22e58..da3ea3560f 100644
+index 8cdcf22e58..d6d59275c1 100644
--- a/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/variant/TestVariantShredding.java
+++ b/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/variant/TestVariantShredding.java
-@@ -100,6 +100,15 @@ public class TestVariantShredding extends CatalogTestBase {
+@@ -100,6 +100,17 @@ public class TestVariantShredding extends CatalogTestBase {
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
.config(DISABLE_UI)
@@ -838,6 +898,8 @@ index 8cdcf22e58..da3ea3560f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
diff --git a/dev/diffs/iceberg/1.8.1.diff b/dev/diffs/iceberg/1.8.1.diff
index 718eeccd7e..ac66e9e2c4 100644
--- a/dev/diffs/iceberg/1.8.1.diff
+++ b/dev/diffs/iceberg/1.8.1.diff
@@ -51,10 +51,10 @@ index 6eb26e8b73..38d9cf3ca3 100644
// runtime dependencies for running Hive Catalog based integration test
integrationRuntimeOnly project(':iceberg-hive-metastore')
diff --git a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkExtensionsTestBase.java b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkExtensionsTestBase.java
-index 4f137f5b8d..f1122f0843 100644
+index 4f137f5b8d..1579d23ce0 100644
--- a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkExtensionsTestBase.java
+++ b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkExtensionsTestBase.java
-@@ -60,7 +60,25 @@ public abstract class SparkExtensionsTestBase extends SparkCatalogTestBase {
+@@ -60,7 +60,29 @@ public abstract class SparkExtensionsTestBase extends SparkCatalogTestBase {
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
.config(
SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(), String.valueOf(RANDOM.nextBoolean()))
@@ -65,6 +65,8 @@ index 4f137f5b8d..f1122f0843 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -75,6 +77,8 @@ index 4f137f5b8d..f1122f0843 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -95,10 +99,10 @@ index 5fa76f9da6..9184aaba83 100644
protected void createTableWithDeleteGranularity(
diff --git a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-index 55a413063e..61a9385ff3 100644
+index 55a413063e..025466e8b4 100644
--- a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
+++ b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-@@ -61,6 +61,15 @@ public class TestCallStatementParser {
+@@ -61,6 +61,17 @@ public class TestCallStatementParser {
.master("local[2]")
.config("spark.sql.extensions", IcebergSparkSessionExtensions.class.getName())
.config("spark.extra.prop", "value")
@@ -109,6 +113,8 @@ index 55a413063e..61a9385ff3 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -143,10 +149,10 @@ index 49119d319c..04fd3d4859 100644
}
diff --git a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-index b6ade2bff3..90a4839dad 100644
+index b6ade2bff3..60f220dabb 100644
--- a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
+++ b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-@@ -170,6 +170,15 @@ public class DeleteOrphanFilesBenchmark {
+@@ -170,6 +170,17 @@ public class DeleteOrphanFilesBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", catalogWarehouse())
@@ -157,16 +163,18 @@ index b6ade2bff3..90a4839dad 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local");
spark = builder.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-index b08c352819..63880b089d 100644
+index b08c352819..4c2636760d 100644
--- a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
+++ b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-@@ -386,6 +386,15 @@ public class IcebergSortCompactionBenchmark {
+@@ -386,6 +386,17 @@ public class IcebergSortCompactionBenchmark {
"spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", getCatalogWarehouse())
@@ -177,16 +185,18 @@ index b08c352819..63880b089d 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]");
spark = builder.getOrCreate();
Configuration sparkHadoopConf = spark.sessionState().newHadoopConf();
diff --git a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-index 68c537e34a..2e421c7706 100644
+index 68c537e34a..30ab50d0c7 100644
--- a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
+++ b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-@@ -94,7 +94,18 @@ public abstract class IcebergSourceBenchmark {
+@@ -94,7 +94,20 @@ public abstract class IcebergSourceBenchmark {
}
protected void setupSpark(boolean enableDictionaryEncoding) {
@@ -201,6 +211,8 @@ index 68c537e34a..2e421c7706 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g");
if (!enableDictionaryEncoding) {
@@ -1004,10 +1016,10 @@ index 780e1750a5..25f253eede 100644
.filter(residual)
.caseSensitive(caseSensitive())
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-index 404ba72846..b375458f05 100644
+index 404ba72846..d53a3051a8 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-@@ -90,6 +90,15 @@ public abstract class SparkDistributedDataScanTestBase
+@@ -90,6 +90,17 @@ public abstract class SparkDistributedDataScanTestBase
.master("local[2]")
.config("spark.serializer", serializer)
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -1018,16 +1030,18 @@ index 404ba72846..b375458f05 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-index 9361c63176..16ce1c614f 100644
+index 9361c63176..9b48408755 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-@@ -69,6 +69,15 @@ public class TestSparkDistributedDataScanDeletes
+@@ -69,6 +69,17 @@ public class TestSparkDistributedDataScanDeletes
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -1038,16 +1052,18 @@ index 9361c63176..16ce1c614f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-index a218f965ea..e30a8d6c6f 100644
+index a218f965ea..82e7521a90 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-@@ -62,6 +62,15 @@ public class TestSparkDistributedDataScanFilterFiles
+@@ -62,6 +62,17 @@ public class TestSparkDistributedDataScanFilterFiles
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -1058,16 +1074,18 @@ index a218f965ea..e30a8d6c6f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-index acd4688440..33732e6654 100644
+index acd4688440..4dcb8c797b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-@@ -59,6 +59,15 @@ public class TestSparkDistributedDataScanReporting
+@@ -59,6 +59,17 @@ public class TestSparkDistributedDataScanReporting
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -1078,16 +1096,18 @@ index acd4688440..33732e6654 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java
-index 3e8953fb95..51ce392476 100644
+index 3e8953fb95..2fd144ff74 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java
-@@ -77,6 +77,15 @@ public abstract class SparkTestBase extends SparkTestHelperBase {
+@@ -77,6 +77,17 @@ public abstract class SparkTestBase extends SparkTestHelperBase {
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
@@ -1098,6 +1118,8 @@ index 3e8953fb95..51ce392476 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1129,10 +1151,10 @@ index ad969384c5..f3c2b833e3 100644
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-index 3a269740b7..fc63449215 100644
+index 3a269740b7..9affe54215 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-@@ -54,7 +54,19 @@ public abstract class ScanTestBase extends AvroDataTest {
+@@ -54,7 +54,21 @@ public abstract class ScanTestBase extends AvroDataTest {
@BeforeAll
public static void startSpark() {
@@ -1147,6 +1169,8 @@ index 3a269740b7..fc63449215 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1154,10 +1178,10 @@ index 3a269740b7..fc63449215 100644
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-index 724c6edde2..ff01c10e1a 100644
+index 724c6edde2..84c51ce703 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-@@ -103,7 +103,19 @@ public class TestCompressionSettings extends SparkCatalogTestBase {
+@@ -103,7 +103,21 @@ public class TestCompressionSettings extends SparkCatalogTestBase {
@BeforeClass
public static void startSpark() {
@@ -1172,6 +1196,8 @@ index 724c6edde2..ff01c10e1a 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1179,10 +1205,10 @@ index 724c6edde2..ff01c10e1a 100644
@Parameterized.AfterParam
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-index 013b8d4386..007b1af65b 100644
+index 013b8d4386..c3ba8f8a90 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-@@ -76,7 +76,19 @@ public class TestDataSourceOptions extends SparkTestBaseWithCatalog {
+@@ -76,7 +76,21 @@ public class TestDataSourceOptions extends SparkTestBaseWithCatalog {
@BeforeClass
public static void startSpark() {
@@ -1197,6 +1223,8 @@ index 013b8d4386..007b1af65b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1204,10 +1232,10 @@ index 013b8d4386..007b1af65b 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-index ba13d005bd..b9778c1986 100644
+index ba13d005bd..41304b505d 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-@@ -118,7 +118,19 @@ public class TestFilteredScan {
+@@ -118,7 +118,21 @@ public class TestFilteredScan {
@BeforeClass
public static void startSpark() {
@@ -1222,6 +1250,8 @@ index ba13d005bd..b9778c1986 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1229,10 +1259,10 @@ index ba13d005bd..b9778c1986 100644
// define UDFs used by partition tests
Function<Object, Integer> bucket4 = Transforms.bucket(4).bind(Types.LongType.get());
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-index 9f97753094..fb5bfbad1b 100644
+index 9f97753094..0f4b4282cf 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-@@ -94,7 +94,19 @@ public class TestForwardCompatibility {
+@@ -94,7 +94,21 @@ public class TestForwardCompatibility {
@BeforeClass
public static void startSpark() {
@@ -1247,6 +1277,8 @@ index 9f97753094..fb5bfbad1b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1254,10 +1286,10 @@ index 9f97753094..fb5bfbad1b 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-index 0154506f86..12e916cbe4 100644
+index 0154506f86..3c3065d2b0 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-@@ -46,7 +46,19 @@ public class TestIcebergSpark {
+@@ -46,7 +46,21 @@ public class TestIcebergSpark {
@BeforeClass
public static void startSpark() {
@@ -1272,6 +1304,8 @@ index 0154506f86..12e916cbe4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1279,10 +1313,10 @@ index 0154506f86..12e916cbe4 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-index 639d37c793..5e397909d4 100644
+index 639d37c793..8defa81bd2 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-@@ -111,7 +111,19 @@ public class TestPartitionPruning {
+@@ -111,7 +111,21 @@ public class TestPartitionPruning {
@BeforeClass
public static void startSpark() {
@@ -1297,6 +1331,8 @@ index 639d37c793..5e397909d4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1304,10 +1340,10 @@ index 639d37c793..5e397909d4 100644
String optionKey = String.format("fs.%s.impl", CountOpenLocalFileSystem.scheme);
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-index ad0984ef42..99af33c86c 100644
+index ad0984ef42..1f066eeaa4 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-@@ -104,7 +104,19 @@ public class TestPartitionValues {
+@@ -104,7 +104,21 @@ public class TestPartitionValues {
@BeforeClass
public static void startSpark() {
@@ -1322,6 +1358,8 @@ index ad0984ef42..99af33c86c 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1329,10 +1367,10 @@ index ad0984ef42..99af33c86c 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-index 9fc576dde5..69a49f5091 100644
+index 9fc576dde5..209340458b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-@@ -83,7 +83,19 @@ public class TestSnapshotSelection {
+@@ -83,7 +83,21 @@ public class TestSnapshotSelection {
@BeforeClass
public static void startSpark() {
@@ -1347,6 +1385,8 @@ index 9fc576dde5..69a49f5091 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1354,10 +1394,10 @@ index 9fc576dde5..69a49f5091 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-index 16fde3c954..26e94f6dde 100644
+index 16fde3c954..4aa025b80b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-@@ -121,7 +121,19 @@ public class TestSparkDataFile {
+@@ -121,7 +121,21 @@ public class TestSparkDataFile {
@BeforeClass
public static void startSpark() {
@@ -1372,6 +1412,8 @@ index 16fde3c954..26e94f6dde 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1379,10 +1421,10 @@ index 16fde3c954..26e94f6dde 100644
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-index 63c18277aa..971f14b647 100644
+index 63c18277aa..b791b36345 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-@@ -89,7 +89,19 @@ public class TestSparkDataWrite {
+@@ -89,7 +89,21 @@ public class TestSparkDataWrite {
@BeforeClass
public static void startSpark() {
@@ -1397,13 +1439,15 @@ index 63c18277aa..971f14b647 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
}
@Parameterized.AfterParam
-@@ -138,7 +150,7 @@ public class TestSparkDataWrite {
+@@ -138,7 +152,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1412,7 +1456,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
for (ManifestFile manifest :
-@@ -208,7 +220,7 @@ public class TestSparkDataWrite {
+@@ -208,7 +222,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1421,7 +1465,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -254,7 +266,7 @@ public class TestSparkDataWrite {
+@@ -254,7 +268,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1430,7 +1474,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -307,7 +319,7 @@ public class TestSparkDataWrite {
+@@ -307,7 +321,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1439,7 +1483,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -350,7 +362,7 @@ public class TestSparkDataWrite {
+@@ -350,7 +364,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1448,7 +1492,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -390,7 +402,7 @@ public class TestSparkDataWrite {
+@@ -390,7 +404,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1457,7 +1501,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
-@@ -455,7 +467,7 @@ public class TestSparkDataWrite {
+@@ -455,7 +469,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1466,7 +1510,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -619,7 +631,7 @@ public class TestSparkDataWrite {
+@@ -619,7 +633,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1475,7 +1519,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
-@@ -705,7 +717,7 @@ public class TestSparkDataWrite {
+@@ -705,7 +719,7 @@ public class TestSparkDataWrite {
// Since write and commit succeeded, the rows should be readable
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1485,10 +1529,10 @@ index 63c18277aa..971f14b647 100644
"Number of rows should match", records.size() + records2.size(), actual.size());
assertThat(actual)
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-index 3a4b235c46..7633a1792d 100644
+index 3a4b235c46..3dcc182b95 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-@@ -86,7 +86,19 @@ public class TestSparkReadProjection extends TestReadProjection {
+@@ -86,7 +86,21 @@ public class TestSparkReadProjection extends TestReadProjection {
@BeforeClass
public static void startSpark() {
@@ -1503,6 +1547,8 @@ index 3a4b235c46..7633a1792d 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1510,10 +1556,10 @@ index 3a4b235c46..7633a1792d 100644
ImmutableMap.of(
"type", "hive",
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-index dda49b4946..69d3a0d24f 100644
+index dda49b4946..d7b7adbf19 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-@@ -131,7 +131,25 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -131,7 +131,29 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
.config("spark.ui.liveUpdate.period", 0)
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -1524,6 +1570,8 @@ index dda49b4946..69d3a0d24f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1534,12 +1582,14 @@ index dda49b4946..69d3a0d24f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
catalog =
-@@ -199,7 +217,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -199,7 +221,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
}
protected boolean countDeletes() {
@@ -1550,10 +1600,10 @@ index dda49b4946..69d3a0d24f 100644
@Override
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-index e5831b76e4..2433013d3e 100644
+index e5831b76e4..50b2e9475b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-@@ -176,7 +176,25 @@ public class TestSparkReaderWithBloomFilter {
+@@ -176,7 +176,29 @@ public class TestSparkReaderWithBloomFilter {
SparkSession.builder()
.master("local[2]")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -1564,6 +1614,8 @@ index e5831b76e4..2433013d3e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1574,16 +1626,18 @@ index e5831b76e4..2433013d3e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
catalog =
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-index f420f1b955..21273826ba 100644
+index f420f1b955..72ae4711a3 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-@@ -66,6 +66,15 @@ public class TestStructuredStreaming {
+@@ -66,6 +66,17 @@ public class TestStructuredStreaming {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.shuffle.partitions", 4)
@@ -1594,16 +1648,18 @@ index f420f1b955..21273826ba 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-index ac674e2e62..c18fed7cfb 100644
+index ac674e2e62..073205f796 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-@@ -73,7 +73,19 @@ public class TestTimestampWithoutZone extends SparkTestBase {
+@@ -73,7 +73,21 @@ public class TestTimestampWithoutZone extends SparkTestBase {
@BeforeClass
public static void startSpark() {
@@ -1618,6 +1674,8 @@ index ac674e2e62..c18fed7cfb 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1625,10 +1683,10 @@ index ac674e2e62..c18fed7cfb 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-index 73827b309b..ace66913d1 100644
+index 73827b309b..cb70ef1b02 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-@@ -80,7 +80,19 @@ public class TestWriteMetricsConfig {
+@@ -80,7 +80,21 @@ public class TestWriteMetricsConfig {
@BeforeClass
public static void startSpark() {
@@ -1643,6 +1701,8 @@ index 73827b309b..ace66913d1 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1650,10 +1710,10 @@ index 73827b309b..ace66913d1 100644
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-index 1a4e2f3e1c..047e0f1fb4 100644
+index 1a4e2f3e1c..cab52c90d8 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-@@ -65,7 +65,25 @@ public class TestAggregatePushDown extends SparkCatalogTestBase {
+@@ -65,7 +65,29 @@ public class TestAggregatePushDown extends SparkCatalogTestBase {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.iceberg.aggregate_pushdown", "true")
@@ -1664,6 +1724,8 @@ index 1a4e2f3e1c..047e0f1fb4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1674,6 +1736,8 @@ index 1a4e2f3e1c..047e0f1fb4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -1738,10 +1802,10 @@ index e2d2c7a7ac..95cd3c08b5 100644
// runtime dependencies for running Hive Catalog based integration test
integrationRuntimeOnly project(':iceberg-hive-metastore')
diff --git a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-index 578845e3da..02b6dc270f 100644
+index 578845e3da..2f40e84b56 100644
--- a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
+++ b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-@@ -57,6 +57,15 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
+@@ -57,6 +57,17 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
.config(
SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(), String.valueOf(RANDOM.nextBoolean()))
@@ -1752,16 +1816,18 @@ index 578845e3da..02b6dc270f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
diff --git a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-index ade19de36f..579b68e73f 100644
+index ade19de36f..f7caecda7a 100644
--- a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
+++ b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-@@ -56,6 +56,15 @@ public class TestCallStatementParser {
+@@ -56,6 +56,17 @@ public class TestCallStatementParser {
.master("local[2]")
.config("spark.sql.extensions", IcebergSparkSessionExtensions.class.getName())
.config("spark.extra.prop", "value")
@@ -1772,6 +1838,8 @@ index ade19de36f..579b68e73f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -1806,10 +1874,10 @@ index 8cb92224fc..777b9bb1dd 100644
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-index 64edb1002e..57767da086 100644
+index 64edb1002e..b9f137443a 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-@@ -179,6 +179,15 @@ public class DeleteOrphanFilesBenchmark {
+@@ -179,6 +179,17 @@ public class DeleteOrphanFilesBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", catalogWarehouse())
@@ -1820,16 +1888,18 @@ index 64edb1002e..57767da086 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local");
spark = builder.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-index a5d0456b0b..f52c387943 100644
+index a5d0456b0b..26b1bfbc82 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-@@ -392,6 +392,15 @@ public class IcebergSortCompactionBenchmark {
+@@ -392,6 +392,17 @@ public class IcebergSortCompactionBenchmark {
"spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", getCatalogWarehouse())
@@ -1840,16 +1910,18 @@ index a5d0456b0b..f52c387943 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]");
spark = builder.getOrCreate();
Configuration sparkHadoopConf = spark.sessionState().newHadoopConf();
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
-index c6794e43c6..4c1a165a44 100644
+index c6794e43c6..8f683c8fc9 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
-@@ -239,6 +239,15 @@ public class DVReaderBenchmark {
+@@ -239,6 +239,17 @@ public class DVReaderBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", newWarehouseDir())
@@ -1860,16 +1932,18 @@ index c6794e43c6..4c1a165a44 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
-index ac74fb5a10..4aa5d5b361 100644
+index ac74fb5a10..ef736f66a3 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
-@@ -223,6 +223,15 @@ public class DVWriterBenchmark {
+@@ -223,6 +223,17 @@ public class DVWriterBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", newWarehouseDir())
@@ -1880,16 +1954,18 @@ index ac74fb5a10..4aa5d5b361 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-index 68c537e34a..2e421c7706 100644
+index 68c537e34a..30ab50d0c7 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-@@ -94,7 +94,18 @@ public abstract class IcebergSourceBenchmark {
+@@ -94,7 +94,20 @@ public abstract class IcebergSourceBenchmark {
}
protected void setupSpark(boolean enableDictionaryEncoding) {
@@ -1904,6 +1980,8 @@ index 68c537e34a..2e421c7706 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g");
if (!enableDictionaryEncoding) {
@@ -2707,10 +2785,10 @@ index 780e1750a5..25f253eede 100644
.filter(residual)
.caseSensitive(caseSensitive())
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-index 404ba72846..b375458f05 100644
+index 404ba72846..d53a3051a8 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-@@ -90,6 +90,15 @@ public abstract class SparkDistributedDataScanTestBase
+@@ -90,6 +90,17 @@ public abstract class SparkDistributedDataScanTestBase
.master("local[2]")
.config("spark.serializer", serializer)
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2721,16 +2799,18 @@ index 404ba72846..b375458f05 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-index 659507e4c5..2a3a87f079 100644
+index 659507e4c5..67a3765dca 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-@@ -73,6 +73,15 @@ public class TestSparkDistributedDataScanDeletes
+@@ -73,6 +73,17 @@ public class TestSparkDistributedDataScanDeletes
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2741,16 +2821,18 @@ index 659507e4c5..2a3a87f079 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-index a218f965ea..e30a8d6c6f 100644
+index a218f965ea..82e7521a90 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-@@ -62,6 +62,15 @@ public class TestSparkDistributedDataScanFilterFiles
+@@ -62,6 +62,17 @@ public class TestSparkDistributedDataScanFilterFiles
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2761,16 +2843,18 @@ index a218f965ea..e30a8d6c6f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-index 2665d7ba8d..c24c82341a 100644
+index 2665d7ba8d..0f43057229 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-@@ -63,6 +63,15 @@ public class TestSparkDistributedDataScanReporting
+@@ -63,6 +63,17 @@ public class TestSparkDistributedDataScanReporting
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2781,16 +2865,18 @@ index 2665d7ba8d..c24c82341a 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-index de68351f6e..dea8226742 100644
+index de68351f6e..05dd0b360e 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-@@ -77,6 +77,15 @@ public abstract class TestBase extends SparkTestHelperBase {
+@@ -77,6 +77,17 @@ public abstract class TestBase extends SparkTestHelperBase {
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
@@ -2801,16 +2887,18 @@ index de68351f6e..dea8226742 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/parquet/vectorized/TestParquetDictionaryEncodedVectorizedReads.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/parquet/vectorized/TestParquetDictionaryEncodedVectorizedReads.java
-index bc4e722bc8..57770308b3 100644
+index bc4e722bc8..f29b72247b 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/parquet/vectorized/TestParquetDictionaryEncodedVectorizedReads.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/parquet/vectorized/TestParquetDictionaryEncodedVectorizedReads.java
-@@ -59,7 +59,19 @@ public class TestParquetDictionaryEncodedVectorizedReads extends TestParquetVect
+@@ -59,7 +59,21 @@ public class TestParquetDictionaryEncodedVectorizedReads extends TestParquetVect
@BeforeAll
public static void startSpark() {
@@ -2825,6 +2913,8 @@ index bc4e722bc8..57770308b3 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2832,10 +2922,10 @@ index bc4e722bc8..57770308b3 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-index 3a269740b7..fc63449215 100644
+index 3a269740b7..9affe54215 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-@@ -54,7 +54,19 @@ public abstract class ScanTestBase extends AvroDataTest {
+@@ -54,7 +54,21 @@ public abstract class ScanTestBase extends AvroDataTest {
@BeforeAll
public static void startSpark() {
@@ -2850,6 +2940,8 @@ index 3a269740b7..fc63449215 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2857,10 +2949,10 @@ index 3a269740b7..fc63449215 100644
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-index f411920a5d..ec8eebe117 100644
+index f411920a5d..d0af6f4916 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-@@ -144,7 +144,19 @@ public class TestCompressionSettings extends CatalogTestBase {
+@@ -144,7 +144,21 @@ public class TestCompressionSettings extends CatalogTestBase {
@BeforeAll
public static void startSpark() {
@@ -2875,6 +2967,8 @@ index f411920a5d..ec8eebe117 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2882,10 +2976,10 @@ index f411920a5d..ec8eebe117 100644
@BeforeEach
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-index c4ba96e634..dcf72c19b6 100644
+index c4ba96e634..657826084c 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-@@ -75,7 +75,19 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
+@@ -75,7 +75,21 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
@BeforeAll
public static void startSpark() {
@@ -2900,6 +2994,8 @@ index c4ba96e634..dcf72c19b6 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2907,10 +3003,10 @@ index c4ba96e634..dcf72c19b6 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-index 348173596e..03156f786b 100644
+index 348173596e..17f8d7ba56 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-@@ -110,7 +110,19 @@ public class TestFilteredScan {
+@@ -110,7 +110,21 @@ public class TestFilteredScan {
@BeforeAll
public static void startSpark() {
@@ -2925,6 +3021,8 @@ index 348173596e..03156f786b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2932,10 +3030,10 @@ index 348173596e..03156f786b 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-index 84c99a575c..e647ab924e 100644
+index 84c99a575c..9666a8dea1 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-@@ -93,7 +93,19 @@ public class TestForwardCompatibility {
+@@ -93,7 +93,21 @@ public class TestForwardCompatibility {
@BeforeAll
public static void startSpark() {
@@ -2950,6 +3048,8 @@ index 84c99a575c..e647ab924e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2957,10 +3057,10 @@ index 84c99a575c..e647ab924e 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-index 7eff93d204..2fdc87a0a8 100644
+index 7eff93d204..9a464a4a46 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-@@ -46,7 +46,19 @@ public class TestIcebergSpark {
+@@ -46,7 +46,21 @@ public class TestIcebergSpark {
@BeforeAll
public static void startSpark() {
@@ -2975,6 +3075,8 @@ index 7eff93d204..2fdc87a0a8 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2982,10 +3084,10 @@ index 7eff93d204..2fdc87a0a8 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-index 9464f687b0..8653560334 100644
+index 9464f687b0..3af55b03ca 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-@@ -112,7 +112,19 @@ public class TestPartitionPruning {
+@@ -112,7 +112,21 @@ public class TestPartitionPruning {
@BeforeAll
public static void startSpark() {
@@ -3000,6 +3102,8 @@ index 9464f687b0..8653560334 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3007,10 +3111,10 @@ index 9464f687b0..8653560334 100644
String optionKey = String.format("fs.%s.impl", CountOpenLocalFileSystem.scheme);
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-index 5c218f21c4..2f17343d59 100644
+index 5c218f21c4..555ce96d6e 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-@@ -107,7 +107,19 @@ public class TestPartitionValues {
+@@ -107,7 +107,21 @@ public class TestPartitionValues {
@BeforeAll
public static void startSpark() {
@@ -3025,6 +3129,8 @@ index 5c218f21c4..2f17343d59 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3032,10 +3138,10 @@ index 5c218f21c4..2f17343d59 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-index a7334a580c..68210dc8e8 100644
+index a7334a580c..5d882a41d6 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-@@ -87,7 +87,19 @@ public class TestSnapshotSelection {
+@@ -87,7 +87,21 @@ public class TestSnapshotSelection {
@BeforeAll
public static void startSpark() {
@@ -3050,6 +3156,8 @@ index a7334a580c..68210dc8e8 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3057,10 +3165,10 @@ index a7334a580c..68210dc8e8 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-index 182b1ef8f5..76e04b1472 100644
+index 182b1ef8f5..3178bc28c5 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-@@ -120,7 +120,19 @@ public class TestSparkDataFile {
+@@ -120,7 +120,21 @@ public class TestSparkDataFile {
@BeforeAll
public static void startSpark() {
@@ -3075,6 +3183,8 @@ index 182b1ef8f5..76e04b1472 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3082,10 +3192,10 @@ index 182b1ef8f5..76e04b1472 100644
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-index fb2b312bed..846f50e0c9 100644
+index fb2b312bed..7e66feb2f0 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-@@ -96,7 +96,19 @@ public class TestSparkDataWrite {
+@@ -96,7 +96,21 @@ public class TestSparkDataWrite {
@BeforeAll
public static void startSpark() {
@@ -3100,13 +3210,15 @@ index fb2b312bed..846f50e0c9 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
}
@AfterEach
-@@ -140,7 +152,7 @@ public class TestSparkDataWrite {
+@@ -140,7 +154,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3115,7 +3227,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
for (ManifestFile manifest :
-@@ -210,7 +222,7 @@ public class TestSparkDataWrite {
+@@ -210,7 +224,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3124,7 +3236,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -256,7 +268,7 @@ public class TestSparkDataWrite {
+@@ -256,7 +270,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3133,7 +3245,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -309,7 +321,7 @@ public class TestSparkDataWrite {
+@@ -309,7 +323,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3142,7 +3254,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -352,7 +364,7 @@ public class TestSparkDataWrite {
+@@ -352,7 +366,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3151,7 +3263,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -392,7 +404,7 @@ public class TestSparkDataWrite {
+@@ -392,7 +406,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3160,7 +3272,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
-@@ -458,7 +470,7 @@ public class TestSparkDataWrite {
+@@ -458,7 +472,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3169,7 +3281,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -622,7 +634,7 @@ public class TestSparkDataWrite {
+@@ -622,7 +636,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3178,7 +3290,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
-@@ -708,7 +720,7 @@ public class TestSparkDataWrite {
+@@ -708,7 +722,7 @@ public class TestSparkDataWrite {
// Since write and commit succeeded, the rows should be readable
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3188,10 +3300,10 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual)
.describedAs("Result rows should match")
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-index becf6a064d..91451c268e 100644
+index becf6a064d..9ce173b6ef 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-@@ -83,7 +83,19 @@ public class TestSparkReadProjection extends TestReadProjection {
+@@ -83,7 +83,21 @@ public class TestSparkReadProjection extends TestReadProjection {
@BeforeAll
public static void startSpark() {
@@ -3206,6 +3318,8 @@ index becf6a064d..91451c268e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3213,10 +3327,10 @@ index becf6a064d..91451c268e 100644
ImmutableMap.of(
"type", "hive",
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-index 4f1cef5d37..4a1a602ed4 100644
+index 4f1cef5d37..8966dac33b 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-@@ -136,6 +136,15 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -136,6 +136,17 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
.config("spark.ui.liveUpdate.period", 0)
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -3227,12 +3341,14 @@ index 4f1cef5d37..4a1a602ed4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
-@@ -204,7 +213,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -204,7 +215,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
}
protected boolean countDeletes() {
@@ -3243,10 +3359,10 @@ index 4f1cef5d37..4a1a602ed4 100644
@Override
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-index baf7fa8f88..b58321b761 100644
+index baf7fa8f88..19e8ec83ba 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-@@ -182,6 +182,15 @@ public class TestSparkReaderWithBloomFilter {
+@@ -182,6 +182,17 @@ public class TestSparkReaderWithBloomFilter {
SparkSession.builder()
.master("local[2]")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -3257,16 +3373,18 @@ index baf7fa8f88..b58321b761 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-index 17db46b85c..d424b97d2a 100644
+index 17db46b85c..6566440898 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-@@ -65,6 +65,15 @@ public class TestStructuredStreaming {
+@@ -65,6 +65,17 @@ public class TestStructuredStreaming {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.shuffle.partitions", 4)
@@ -3277,16 +3395,18 @@ index 17db46b85c..d424b97d2a 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-index 306444b9f2..e4082d9cd9 100644
+index 306444b9f2..fb75671ac6 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-@@ -75,7 +75,19 @@ public class TestTimestampWithoutZone extends TestBase {
+@@ -75,7 +75,21 @@ public class TestTimestampWithoutZone extends TestBase {
@BeforeAll
public static void startSpark() {
@@ -3301,6 +3421,8 @@ index 306444b9f2..e4082d9cd9 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3308,10 +3430,10 @@ index 306444b9f2..e4082d9cd9 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-index 841268a6be..97f7187314 100644
+index 841268a6be..f9c0a6b7af 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-@@ -80,7 +80,19 @@ public class TestWriteMetricsConfig {
+@@ -80,7 +80,21 @@ public class TestWriteMetricsConfig {
@BeforeAll
public static void startSpark() {
@@ -3326,6 +3448,8 @@ index 841268a6be..97f7187314 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3333,10 +3457,10 @@ index 841268a6be..97f7187314 100644
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-index 6e09252704..4b0f2e4d58 100644
+index 6e09252704..faacde70ff 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-@@ -60,6 +60,15 @@ public class TestAggregatePushDown extends CatalogTestBase {
+@@ -60,6 +60,17 @@ public class TestAggregatePushDown extends CatalogTestBase {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.iceberg.aggregate_pushdown", "true")
@@ -3347,6 +3471,8 @@ index 6e09252704..4b0f2e4d58 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
diff --git a/dev/diffs/iceberg/1.9.1.diff b/dev/diffs/iceberg/1.9.1.diff
index 139003cae3..4b757ceaa5 100644
--- a/dev/diffs/iceberg/1.9.1.diff
+++ b/dev/diffs/iceberg/1.9.1.diff
@@ -41,10 +41,10 @@ index 2cd8666892..4077116abb 100644
// runtime dependencies for running Hive Catalog based integration test
integrationRuntimeOnly project(':iceberg-hive-metastore')
diff --git a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-index 578845e3da..19a200892f 100644
+index 578845e3da..ce2b886a54 100644
--- a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
+++ b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-@@ -57,7 +57,25 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
+@@ -57,7 +57,29 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
.config(
SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(), String.valueOf(RANDOM.nextBoolean()))
@@ -55,6 +55,8 @@ index 578845e3da..19a200892f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -65,16 +67,18 @@ index 578845e3da..19a200892f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
TestBase.sparkContext = JavaSparkContext.fromSparkContext(spark.sparkContext());
diff --git a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-index 58d054bd05..ef1c1cb6d1 100644
+index 58d054bd05..30efbc7f29 100644
--- a/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
+++ b/spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-@@ -56,6 +56,15 @@ public class TestCallStatementParser {
+@@ -56,6 +56,17 @@ public class TestCallStatementParser {
.master("local[2]")
.config("spark.sql.extensions", IcebergSparkSessionExtensions.class.getName())
.config("spark.extra.prop", "value")
@@ -85,6 +89,8 @@ index 58d054bd05..ef1c1cb6d1 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -119,10 +125,10 @@ index 5a0a04edba..afe92a8b72 100644
}
diff --git a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-index b6ade2bff3..90a4839dad 100644
+index b6ade2bff3..60f220dabb 100644
--- a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
+++ b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-@@ -170,6 +170,15 @@ public class DeleteOrphanFilesBenchmark {
+@@ -170,6 +170,17 @@ public class DeleteOrphanFilesBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", catalogWarehouse())
@@ -133,16 +139,18 @@ index b6ade2bff3..90a4839dad 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local");
spark = builder.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-index b08c352819..63880b089d 100644
+index b08c352819..4c2636760d 100644
--- a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
+++ b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-@@ -386,6 +386,15 @@ public class IcebergSortCompactionBenchmark {
+@@ -386,6 +386,17 @@ public class IcebergSortCompactionBenchmark {
"spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", getCatalogWarehouse())
@@ -153,16 +161,18 @@ index b08c352819..63880b089d 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]");
spark = builder.getOrCreate();
Configuration sparkHadoopConf = spark.sessionState().newHadoopConf();
diff --git a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-index 68c537e34a..2e421c7706 100644
+index 68c537e34a..30ab50d0c7 100644
--- a/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
+++ b/spark/v3.4/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-@@ -94,7 +94,18 @@ public abstract class IcebergSourceBenchmark {
+@@ -94,7 +94,20 @@ public abstract class IcebergSourceBenchmark {
}
protected void setupSpark(boolean enableDictionaryEncoding) {
@@ -177,6 +187,8 @@ index 68c537e34a..2e421c7706 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g");
if (!enableDictionaryEncoding) {
@@ -964,10 +976,10 @@ index 780e1750a5..25f253eede 100644
.filter(residual)
.caseSensitive(caseSensitive())
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-index 404ba72846..b375458f05 100644
+index 404ba72846..d53a3051a8 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-@@ -90,6 +90,15 @@ public abstract class SparkDistributedDataScanTestBase
+@@ -90,6 +90,17 @@ public abstract class SparkDistributedDataScanTestBase
.master("local[2]")
.config("spark.serializer", serializer)
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -978,16 +990,18 @@ index 404ba72846..b375458f05 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-index 9361c63176..16ce1c614f 100644
+index 9361c63176..9b48408755 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-@@ -69,6 +69,15 @@ public class TestSparkDistributedDataScanDeletes
+@@ -69,6 +69,17 @@ public class TestSparkDistributedDataScanDeletes
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -998,16 +1012,18 @@ index 9361c63176..16ce1c614f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-index a218f965ea..e30a8d6c6f 100644
+index a218f965ea..82e7521a90 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-@@ -62,6 +62,15 @@ public class TestSparkDistributedDataScanFilterFiles
+@@ -62,6 +62,17 @@ public class TestSparkDistributedDataScanFilterFiles
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -1018,16 +1034,18 @@ index a218f965ea..e30a8d6c6f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-index acd4688440..33732e6654 100644
+index acd4688440..4dcb8c797b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-@@ -59,6 +59,15 @@ public class TestSparkDistributedDataScanReporting
+@@ -59,6 +59,17 @@ public class TestSparkDistributedDataScanReporting
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -1038,16 +1056,18 @@ index acd4688440..33732e6654 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java
-index 3e8953fb95..51ce392476 100644
+index 3e8953fb95..2fd144ff74 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java
-@@ -77,6 +77,15 @@ public abstract class SparkTestBase extends SparkTestHelperBase {
+@@ -77,6 +77,17 @@ public abstract class SparkTestBase extends SparkTestHelperBase {
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
@@ -1058,16 +1078,18 @@ index 3e8953fb95..51ce392476 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-index 06d5e0c44f..effc2bcc69 100644
+index 06d5e0c44f..d60cb9467a 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-@@ -57,7 +57,19 @@ public abstract class ScanTestBase extends AvroDataTest {
+@@ -57,7 +57,21 @@ public abstract class ScanTestBase extends AvroDataTest {
@BeforeAll
public static void startSpark() {
@@ -1082,6 +1104,8 @@ index 06d5e0c44f..effc2bcc69 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1089,10 +1113,10 @@ index 06d5e0c44f..effc2bcc69 100644
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-index 26cb4dcc95..d185f1175b 100644
+index 26cb4dcc95..4c0eabf6f7 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-@@ -104,7 +104,19 @@ public class TestCompressionSettings extends SparkCatalogTestBase {
+@@ -104,7 +104,21 @@ public class TestCompressionSettings extends SparkCatalogTestBase {
@BeforeClass
public static void startSpark() {
@@ -1107,6 +1131,8 @@ index 26cb4dcc95..d185f1175b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1114,10 +1140,10 @@ index 26cb4dcc95..d185f1175b 100644
@Before
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-index 013b8d4386..007b1af65b 100644
+index 013b8d4386..c3ba8f8a90 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-@@ -76,7 +76,19 @@ public class TestDataSourceOptions extends SparkTestBaseWithCatalog {
+@@ -76,7 +76,21 @@ public class TestDataSourceOptions extends SparkTestBaseWithCatalog {
@BeforeClass
public static void startSpark() {
@@ -1132,6 +1158,8 @@ index 013b8d4386..007b1af65b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1139,10 +1167,10 @@ index 013b8d4386..007b1af65b 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-index ba13d005bd..b9778c1986 100644
+index ba13d005bd..41304b505d 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-@@ -118,7 +118,19 @@ public class TestFilteredScan {
+@@ -118,7 +118,21 @@ public class TestFilteredScan {
@BeforeClass
public static void startSpark() {
@@ -1157,6 +1185,8 @@ index ba13d005bd..b9778c1986 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1164,10 +1194,10 @@ index ba13d005bd..b9778c1986 100644
// define UDFs used by partition tests
Function<Object, Integer> bucket4 = Transforms.bucket(4).bind(Types.LongType.get());
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-index 9f97753094..fb5bfbad1b 100644
+index 9f97753094..0f4b4282cf 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-@@ -94,7 +94,19 @@ public class TestForwardCompatibility {
+@@ -94,7 +94,21 @@ public class TestForwardCompatibility {
@BeforeClass
public static void startSpark() {
@@ -1182,6 +1212,8 @@ index 9f97753094..fb5bfbad1b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1189,10 +1221,10 @@ index 9f97753094..fb5bfbad1b 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-index 0154506f86..12e916cbe4 100644
+index 0154506f86..3c3065d2b0 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-@@ -46,7 +46,19 @@ public class TestIcebergSpark {
+@@ -46,7 +46,21 @@ public class TestIcebergSpark {
@BeforeClass
public static void startSpark() {
@@ -1207,6 +1239,8 @@ index 0154506f86..12e916cbe4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1214,10 +1248,10 @@ index 0154506f86..12e916cbe4 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-index 639d37c793..5e397909d4 100644
+index 639d37c793..8defa81bd2 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-@@ -111,7 +111,19 @@ public class TestPartitionPruning {
+@@ -111,7 +111,21 @@ public class TestPartitionPruning {
@BeforeClass
public static void startSpark() {
@@ -1232,6 +1266,8 @@ index 639d37c793..5e397909d4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1239,10 +1275,10 @@ index 639d37c793..5e397909d4 100644
String optionKey = String.format("fs.%s.impl", CountOpenLocalFileSystem.scheme);
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-index ad0984ef42..99af33c86c 100644
+index ad0984ef42..1f066eeaa4 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-@@ -104,7 +104,19 @@ public class TestPartitionValues {
+@@ -104,7 +104,21 @@ public class TestPartitionValues {
@BeforeClass
public static void startSpark() {
@@ -1257,6 +1293,8 @@ index ad0984ef42..99af33c86c 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1264,10 +1302,10 @@ index ad0984ef42..99af33c86c 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-index 9fc576dde5..69a49f5091 100644
+index 9fc576dde5..209340458b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-@@ -83,7 +83,19 @@ public class TestSnapshotSelection {
+@@ -83,7 +83,21 @@ public class TestSnapshotSelection {
@BeforeClass
public static void startSpark() {
@@ -1282,6 +1320,8 @@ index 9fc576dde5..69a49f5091 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1289,10 +1329,10 @@ index 9fc576dde5..69a49f5091 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-index 16fde3c954..26e94f6dde 100644
+index 16fde3c954..4aa025b80b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-@@ -121,7 +121,19 @@ public class TestSparkDataFile {
+@@ -121,7 +121,21 @@ public class TestSparkDataFile {
@BeforeClass
public static void startSpark() {
@@ -1307,6 +1347,8 @@ index 16fde3c954..26e94f6dde 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1314,10 +1356,10 @@ index 16fde3c954..26e94f6dde 100644
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-index 63c18277aa..971f14b647 100644
+index 63c18277aa..b791b36345 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-@@ -89,7 +89,19 @@ public class TestSparkDataWrite {
+@@ -89,7 +89,21 @@ public class TestSparkDataWrite {
@BeforeClass
public static void startSpark() {
@@ -1332,13 +1374,15 @@ index 63c18277aa..971f14b647 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
}
@Parameterized.AfterParam
-@@ -138,7 +150,7 @@ public class TestSparkDataWrite {
+@@ -138,7 +152,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1347,7 +1391,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
for (ManifestFile manifest :
-@@ -208,7 +220,7 @@ public class TestSparkDataWrite {
+@@ -208,7 +222,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1356,7 +1400,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -254,7 +266,7 @@ public class TestSparkDataWrite {
+@@ -254,7 +268,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1365,7 +1409,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -307,7 +319,7 @@ public class TestSparkDataWrite {
+@@ -307,7 +321,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1374,7 +1418,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -350,7 +362,7 @@ public class TestSparkDataWrite {
+@@ -350,7 +364,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1383,7 +1427,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -390,7 +402,7 @@ public class TestSparkDataWrite {
+@@ -390,7 +404,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1392,7 +1436,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
-@@ -455,7 +467,7 @@ public class TestSparkDataWrite {
+@@ -455,7 +469,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1401,7 +1445,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
}
-@@ -619,7 +631,7 @@ public class TestSparkDataWrite {
+@@ -619,7 +633,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1410,7 +1454,7 @@ index 63c18277aa..971f14b647 100644
Assert.assertEquals("Number of rows should match", expected.size(), actual.size());
Assert.assertEquals("Result rows should match", expected, actual);
-@@ -705,7 +717,7 @@ public class TestSparkDataWrite {
+@@ -705,7 +719,7 @@ public class TestSparkDataWrite {
// Since write and commit succeeded, the rows should be readable
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -1420,10 +1464,10 @@ index 63c18277aa..971f14b647 100644
"Number of rows should match", records.size() + records2.size(), actual.size());
assertThat(actual)
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-index 584a6b1c70..b93a23d6f1 100644
+index 584a6b1c70..5504841411 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-@@ -86,7 +86,19 @@ public class TestSparkReadProjection extends TestReadProjection {
+@@ -86,7 +86,21 @@ public class TestSparkReadProjection extends TestReadProjection {
@BeforeClass
public static void startSpark() {
@@ -1438,6 +1482,8 @@ index 584a6b1c70..b93a23d6f1 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1445,10 +1491,10 @@ index 584a6b1c70..b93a23d6f1 100644
ImmutableMap.of(
"type", "hive",
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-index dda49b4946..69d3a0d24f 100644
+index dda49b4946..d7b7adbf19 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-@@ -131,7 +131,25 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -131,7 +131,29 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
.config("spark.ui.liveUpdate.period", 0)
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -1459,6 +1505,8 @@ index dda49b4946..69d3a0d24f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1469,12 +1517,14 @@ index dda49b4946..69d3a0d24f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
catalog =
-@@ -199,7 +217,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -199,7 +221,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
}
protected boolean countDeletes() {
@@ -1485,10 +1535,10 @@ index dda49b4946..69d3a0d24f 100644
@Override
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-index e5831b76e4..2433013d3e 100644
+index e5831b76e4..50b2e9475b 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-@@ -176,7 +176,25 @@ public class TestSparkReaderWithBloomFilter {
+@@ -176,7 +176,29 @@ public class TestSparkReaderWithBloomFilter {
SparkSession.builder()
.master("local[2]")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -1499,6 +1549,8 @@ index e5831b76e4..2433013d3e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1509,16 +1561,18 @@ index e5831b76e4..2433013d3e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
catalog =
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-index 961d69b721..f12ac3e83b 100644
+index 961d69b721..f7d820b7ed 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-@@ -68,6 +68,15 @@ public class TestStructuredStreaming {
+@@ -68,6 +68,17 @@ public class TestStructuredStreaming {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.shuffle.partitions", 4)
@@ -1529,16 +1583,18 @@ index 961d69b721..f12ac3e83b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-index ac674e2e62..c18fed7cfb 100644
+index ac674e2e62..073205f796 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-@@ -73,7 +73,19 @@ public class TestTimestampWithoutZone extends SparkTestBase {
+@@ -73,7 +73,21 @@ public class TestTimestampWithoutZone extends SparkTestBase {
@BeforeClass
public static void startSpark() {
@@ -1553,6 +1609,8 @@ index ac674e2e62..c18fed7cfb 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1560,10 +1618,10 @@ index ac674e2e62..c18fed7cfb 100644
@AfterClass
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-index 73827b309b..ace66913d1 100644
+index 73827b309b..cb70ef1b02 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-@@ -80,7 +80,19 @@ public class TestWriteMetricsConfig {
+@@ -80,7 +80,21 @@ public class TestWriteMetricsConfig {
@BeforeClass
public static void startSpark() {
@@ -1578,6 +1636,8 @@ index 73827b309b..ace66913d1 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -1585,10 +1645,10 @@ index 73827b309b..ace66913d1 100644
}
diff --git a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-index 1a4e2f3e1c..047e0f1fb4 100644
+index 1a4e2f3e1c..cab52c90d8 100644
--- a/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
+++ b/spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-@@ -65,7 +65,25 @@ public class TestAggregatePushDown extends SparkCatalogTestBase {
+@@ -65,7 +65,29 @@ public class TestAggregatePushDown extends SparkCatalogTestBase {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.iceberg.aggregate_pushdown", "true")
@@ -1599,6 +1659,8 @@ index 1a4e2f3e1c..047e0f1fb4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1609,6 +1671,8 @@ index 1a4e2f3e1c..047e0f1fb4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -1673,10 +1737,10 @@ index 572c32f929..29013d26c0 100644
// runtime dependencies for running Hive Catalog based integration test
integrationRuntimeOnly project(':iceberg-hive-metastore')
diff --git a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-index 578845e3da..02b6dc270f 100644
+index 578845e3da..2f40e84b56 100644
--- a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
+++ b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/ExtensionsTestBase.java
-@@ -57,6 +57,15 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
+@@ -57,6 +57,17 @@ public abstract class ExtensionsTestBase extends CatalogTestBase {
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
.config(
SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(), String.valueOf(RANDOM.nextBoolean()))
@@ -1687,6 +1751,8 @@ index 578845e3da..02b6dc270f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -1707,10 +1773,10 @@ index b5d6415763..0763a723d3 100644
protected void createTableWithDeleteGranularity(
diff --git a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-index ecf9e6f8a5..beca5f8521 100644
+index ecf9e6f8a5..14181c55c9 100644
--- a/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
+++ b/spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
-@@ -56,6 +56,15 @@ public class TestCallStatementParser {
+@@ -56,6 +56,17 @@ public class TestCallStatementParser {
.master("local[2]")
.config("spark.sql.extensions", IcebergSparkSessionExtensions.class.getName())
.config("spark.extra.prop", "value")
@@ -1721,6 +1787,8 @@ index ecf9e6f8a5..beca5f8521 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
@@ -1755,10 +1823,10 @@ index 8cb92224fc..777b9bb1dd 100644
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-index 64edb1002e..57767da086 100644
+index 64edb1002e..b9f137443a 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/DeleteOrphanFilesBenchmark.java
-@@ -179,6 +179,15 @@ public class DeleteOrphanFilesBenchmark {
+@@ -179,6 +179,17 @@ public class DeleteOrphanFilesBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", catalogWarehouse())
@@ -1769,16 +1837,18 @@ index 64edb1002e..57767da086 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local");
spark = builder.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-index a5d0456b0b..f52c387943 100644
+index a5d0456b0b..26b1bfbc82 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/action/IcebergSortCompactionBenchmark.java
-@@ -392,6 +392,15 @@ public class IcebergSortCompactionBenchmark {
+@@ -392,6 +392,17 @@ public class IcebergSortCompactionBenchmark {
"spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", getCatalogWarehouse())
@@ -1789,16 +1859,18 @@ index a5d0456b0b..f52c387943 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]");
spark = builder.getOrCreate();
Configuration sparkHadoopConf = spark.sessionState().newHadoopConf();
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
-index c6794e43c6..4c1a165a44 100644
+index c6794e43c6..8f683c8fc9 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVReaderBenchmark.java
-@@ -239,6 +239,15 @@ public class DVReaderBenchmark {
+@@ -239,6 +239,17 @@ public class DVReaderBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", newWarehouseDir())
@@ -1809,16 +1881,18 @@ index c6794e43c6..4c1a165a44 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
-index ac74fb5a10..4aa5d5b361 100644
+index ac74fb5a10..ef736f66a3 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/DVWriterBenchmark.java
-@@ -223,6 +223,15 @@ public class DVWriterBenchmark {
+@@ -223,6 +223,17 @@ public class DVWriterBenchmark {
.config("spark.sql.catalog.spark_catalog", SparkSessionCatalog.class.getName())
.config("spark.sql.catalog.spark_catalog.type", "hadoop")
.config("spark.sql.catalog.spark_catalog.warehouse", newWarehouseDir())
@@ -1829,16 +1903,18 @@ index ac74fb5a10..4aa5d5b361 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.master("local[*]")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-index 68c537e34a..2e421c7706 100644
+index 68c537e34a..30ab50d0c7 100644
--- a/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
+++ b/spark/v3.5/spark/src/jmh/java/org/apache/iceberg/spark/source/IcebergSourceBenchmark.java
-@@ -94,7 +94,18 @@ public abstract class IcebergSourceBenchmark {
+@@ -94,7 +94,20 @@ public abstract class IcebergSourceBenchmark {
}
protected void setupSpark(boolean enableDictionaryEncoding) {
@@ -1853,6 +1929,8 @@ index 68c537e34a..2e421c7706 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g");
if (!enableDictionaryEncoding) {
@@ -2640,10 +2718,10 @@ index 780e1750a5..25f253eede 100644
.filter(residual)
.caseSensitive(caseSensitive())
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-index 404ba72846..b375458f05 100644
+index 404ba72846..d53a3051a8 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/SparkDistributedDataScanTestBase.java
-@@ -90,6 +90,15 @@ public abstract class SparkDistributedDataScanTestBase
+@@ -90,6 +90,17 @@ public abstract class SparkDistributedDataScanTestBase
.master("local[2]")
.config("spark.serializer", serializer)
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2654,16 +2732,18 @@ index 404ba72846..b375458f05 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-index 659507e4c5..2a3a87f079 100644
+index 659507e4c5..67a3765dca 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanDeletes.java
-@@ -73,6 +73,15 @@ public class TestSparkDistributedDataScanDeletes
+@@ -73,6 +73,17 @@ public class TestSparkDistributedDataScanDeletes
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2674,16 +2754,18 @@ index 659507e4c5..2a3a87f079 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-index a218f965ea..e30a8d6c6f 100644
+index a218f965ea..82e7521a90 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanFilterFiles.java
-@@ -62,6 +62,15 @@ public class TestSparkDistributedDataScanFilterFiles
+@@ -62,6 +62,17 @@ public class TestSparkDistributedDataScanFilterFiles
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2694,16 +2776,18 @@ index a218f965ea..e30a8d6c6f 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-index 2665d7ba8d..c24c82341a 100644
+index 2665d7ba8d..0f43057229 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/TestSparkDistributedDataScanReporting.java
-@@ -63,6 +63,15 @@ public class TestSparkDistributedDataScanReporting
+@@ -63,6 +63,17 @@ public class TestSparkDistributedDataScanReporting
.master("local[2]")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config(SQLConf.SHUFFLE_PARTITIONS().key(), "4")
@@ -2714,16 +2798,18 @@ index 2665d7ba8d..c24c82341a 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-index 3e9f3334ef..0f96c07739 100644
+index 3e9f3334ef..187d2c8435 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestBase.java
-@@ -77,6 +77,15 @@ public abstract class TestBase extends SparkTestHelperBase {
+@@ -77,6 +77,17 @@ public abstract class TestBase extends SparkTestHelperBase {
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
.config("spark.sql.legacy.respectNullabilityInTextDatasetConversion", "true")
@@ -2734,6 +2820,8 @@ index 3e9f3334ef..0f96c07739 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
@@ -2765,10 +2853,10 @@ index a31138ae01..501c8ac93e 100644
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/parquet/vectorized/TestParquetDictionaryEncodedVectorizedReads.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/parquet/vectorized/TestParquetDictionaryEncodedVectorizedReads.java
-index bc4e722bc8..57770308b3 100644
+index bc4e722bc8..f29b72247b 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/parquet/vectorized/TestParquetDictionaryEncodedVectorizedReads.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/data/parquet/vectorized/TestParquetDictionaryEncodedVectorizedReads.java
-@@ -59,7 +59,19 @@ public class TestParquetDictionaryEncodedVectorizedReads extends TestParquetVect
+@@ -59,7 +59,21 @@ public class TestParquetDictionaryEncodedVectorizedReads extends TestParquetVect
@BeforeAll
public static void startSpark() {
@@ -2783,6 +2871,8 @@ index bc4e722bc8..57770308b3 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2790,10 +2880,10 @@ index bc4e722bc8..57770308b3 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-index 0886df957d..eca0f74533 100644
+index 0886df957d..2134aa6271 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/ScanTestBase.java
-@@ -57,7 +57,19 @@ public abstract class ScanTestBase extends AvroDataTest {
+@@ -57,7 +57,21 @@ public abstract class ScanTestBase extends AvroDataTest {
@BeforeAll
public static void startSpark() {
@@ -2808,6 +2898,8 @@ index 0886df957d..eca0f74533 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2815,10 +2907,10 @@ index 0886df957d..eca0f74533 100644
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-index 6b7d861364..23419aa9b1 100644
+index 6b7d861364..95c054b2b0 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestCompressionSettings.java
-@@ -144,7 +144,19 @@ public class TestCompressionSettings extends CatalogTestBase {
+@@ -144,7 +144,21 @@ public class TestCompressionSettings extends CatalogTestBase {
@BeforeAll
public static void startSpark() {
@@ -2833,6 +2925,8 @@ index 6b7d861364..23419aa9b1 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2840,10 +2934,10 @@ index 6b7d861364..23419aa9b1 100644
@BeforeEach
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-index c4ba96e634..dcf72c19b6 100644
+index c4ba96e634..657826084c 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataSourceOptions.java
-@@ -75,7 +75,19 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
+@@ -75,7 +75,21 @@ public class TestDataSourceOptions extends TestBaseWithCatalog {
@BeforeAll
public static void startSpark() {
@@ -2858,6 +2952,8 @@ index c4ba96e634..dcf72c19b6 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2865,10 +2961,10 @@ index c4ba96e634..dcf72c19b6 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-index 348173596e..03156f786b 100644
+index 348173596e..17f8d7ba56 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
-@@ -110,7 +110,19 @@ public class TestFilteredScan {
+@@ -110,7 +110,21 @@ public class TestFilteredScan {
@BeforeAll
public static void startSpark() {
@@ -2883,6 +2979,8 @@ index 348173596e..03156f786b 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2890,10 +2988,10 @@ index 348173596e..03156f786b 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-index 84c99a575c..e647ab924e 100644
+index 84c99a575c..9666a8dea1 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
-@@ -93,7 +93,19 @@ public class TestForwardCompatibility {
+@@ -93,7 +93,21 @@ public class TestForwardCompatibility {
@BeforeAll
public static void startSpark() {
@@ -2908,6 +3006,8 @@ index 84c99a575c..e647ab924e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2915,10 +3015,10 @@ index 84c99a575c..e647ab924e 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-index 7eff93d204..2fdc87a0a8 100644
+index 7eff93d204..9a464a4a46 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSpark.java
-@@ -46,7 +46,19 @@ public class TestIcebergSpark {
+@@ -46,7 +46,21 @@ public class TestIcebergSpark {
@BeforeAll
public static void startSpark() {
@@ -2933,6 +3033,8 @@ index 7eff93d204..2fdc87a0a8 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2940,10 +3042,10 @@ index 7eff93d204..2fdc87a0a8 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-index 9464f687b0..8653560334 100644
+index 9464f687b0..3af55b03ca 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionPruning.java
-@@ -112,7 +112,19 @@ public class TestPartitionPruning {
+@@ -112,7 +112,21 @@ public class TestPartitionPruning {
@BeforeAll
public static void startSpark() {
@@ -2958,6 +3060,8 @@ index 9464f687b0..8653560334 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2965,10 +3069,10 @@ index 9464f687b0..8653560334 100644
String optionKey = String.format("fs.%s.impl", CountOpenLocalFileSystem.scheme);
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-index 5c218f21c4..2f17343d59 100644
+index 5c218f21c4..555ce96d6e 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestPartitionValues.java
-@@ -107,7 +107,19 @@ public class TestPartitionValues {
+@@ -107,7 +107,21 @@ public class TestPartitionValues {
@BeforeAll
public static void startSpark() {
@@ -2983,6 +3087,8 @@ index 5c218f21c4..2f17343d59 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -2990,10 +3096,10 @@ index 5c218f21c4..2f17343d59 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-index a7334a580c..68210dc8e8 100644
+index a7334a580c..5d882a41d6 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java
-@@ -87,7 +87,19 @@ public class TestSnapshotSelection {
+@@ -87,7 +87,21 @@ public class TestSnapshotSelection {
@BeforeAll
public static void startSpark() {
@@ -3008,6 +3114,8 @@ index a7334a580c..68210dc8e8 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3015,10 +3123,10 @@ index a7334a580c..68210dc8e8 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-index 182b1ef8f5..76e04b1472 100644
+index 182b1ef8f5..3178bc28c5 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataFile.java
-@@ -120,7 +120,19 @@ public class TestSparkDataFile {
+@@ -120,7 +120,21 @@ public class TestSparkDataFile {
@BeforeAll
public static void startSpark() {
@@ -3033,6 +3141,8 @@ index 182b1ef8f5..76e04b1472 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3040,10 +3150,10 @@ index 182b1ef8f5..76e04b1472 100644
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-index fb2b312bed..846f50e0c9 100644
+index fb2b312bed..7e66feb2f0 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkDataWrite.java
-@@ -96,7 +96,19 @@ public class TestSparkDataWrite {
+@@ -96,7 +96,21 @@ public class TestSparkDataWrite {
@BeforeAll
public static void startSpark() {
@@ -3058,13 +3168,15 @@ index fb2b312bed..846f50e0c9 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
}
@AfterEach
-@@ -140,7 +152,7 @@ public class TestSparkDataWrite {
+@@ -140,7 +154,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3073,7 +3185,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
for (ManifestFile manifest :
-@@ -210,7 +222,7 @@ public class TestSparkDataWrite {
+@@ -210,7 +224,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3082,7 +3194,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -256,7 +268,7 @@ public class TestSparkDataWrite {
+@@ -256,7 +270,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3091,7 +3203,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -309,7 +321,7 @@ public class TestSparkDataWrite {
+@@ -309,7 +323,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3100,7 +3212,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -352,7 +364,7 @@ public class TestSparkDataWrite {
+@@ -352,7 +366,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3109,7 +3221,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -392,7 +404,7 @@ public class TestSparkDataWrite {
+@@ -392,7 +406,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3118,7 +3230,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
-@@ -458,7 +470,7 @@ public class TestSparkDataWrite {
+@@ -458,7 +472,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3127,7 +3239,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
}
-@@ -622,7 +634,7 @@ public class TestSparkDataWrite {
+@@ -622,7 +636,7 @@ public class TestSparkDataWrite {
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3136,7 +3248,7 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual).as("Number of rows should match").hasSameSizeAs(expected);
assertThat(actual).as("Result rows should match").isEqualTo(expected);
-@@ -708,7 +720,7 @@ public class TestSparkDataWrite {
+@@ -708,7 +722,7 @@ public class TestSparkDataWrite {
// Since write and commit succeeded, the rows should be readable
Dataset<Row> result = spark.read().format("iceberg").load(targetLocation);
List<SimpleRecord> actual =
@@ -3146,10 +3258,10 @@ index fb2b312bed..846f50e0c9 100644
assertThat(actual)
.describedAs("Result rows should match")
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-index becf6a064d..91451c268e 100644
+index becf6a064d..9ce173b6ef 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadProjection.java
-@@ -83,7 +83,19 @@ public class TestSparkReadProjection extends TestReadProjection {
+@@ -83,7 +83,21 @@ public class TestSparkReadProjection extends TestReadProjection {
@BeforeAll
public static void startSpark() {
@@ -3164,6 +3276,8 @@ index becf6a064d..91451c268e 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3171,10 +3285,10 @@ index becf6a064d..91451c268e 100644
ImmutableMap.of(
"type", "hive",
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-index 4f1cef5d37..4a1a602ed4 100644
+index 4f1cef5d37..8966dac33b 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
-@@ -136,6 +136,15 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -136,6 +136,17 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
.config("spark.ui.liveUpdate.period", 0)
.config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -3185,12 +3299,14 @@ index 4f1cef5d37..4a1a602ed4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
-@@ -204,7 +213,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
+@@ -204,7 +215,8 @@ public class TestSparkReaderDeletes extends DeleteReadTests {
}
protected boolean countDeletes() {
@@ -3201,10 +3317,10 @@ index 4f1cef5d37..4a1a602ed4 100644
@Override
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-index baf7fa8f88..b58321b761 100644
+index baf7fa8f88..19e8ec83ba 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderWithBloomFilter.java
-@@ -182,6 +182,15 @@ public class TestSparkReaderWithBloomFilter {
+@@ -182,6 +182,17 @@ public class TestSparkReaderWithBloomFilter {
SparkSession.builder()
.master("local[2]")
.config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
@@ -3215,16 +3331,18 @@ index baf7fa8f88..b58321b761 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
.getOrCreate();
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-index c84a65cbe9..e17ba73ee4 100644
+index c84a65cbe9..df98d0b4e3 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestStructuredStreaming.java
-@@ -67,6 +67,15 @@ public class TestStructuredStreaming {
+@@ -67,6 +67,17 @@ public class TestStructuredStreaming {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.shuffle.partitions", 4)
@@ -3235,16 +3353,18 @@ index c84a65cbe9..e17ba73ee4 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.getOrCreate();
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-index 306444b9f2..e4082d9cd9 100644
+index 306444b9f2..fb75671ac6 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestTimestampWithoutZone.java
-@@ -75,7 +75,19 @@ public class TestTimestampWithoutZone extends TestBase {
+@@ -75,7 +75,21 @@ public class TestTimestampWithoutZone extends TestBase {
@BeforeAll
public static void startSpark() {
@@ -3259,6 +3379,8 @@ index 306444b9f2..e4082d9cd9 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3266,10 +3388,10 @@ index 306444b9f2..e4082d9cd9 100644
@AfterAll
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-index 841268a6be..97f7187314 100644
+index 841268a6be..f9c0a6b7af 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestWriteMetricsConfig.java
-@@ -80,7 +80,19 @@ public class TestWriteMetricsConfig {
+@@ -80,7 +80,21 @@ public class TestWriteMetricsConfig {
@BeforeAll
public static void startSpark() {
@@ -3284,6 +3406,8 @@ index 841268a6be..97f7187314 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
+ .getOrCreate();
@@ -3291,10 +3415,10 @@ index 841268a6be..97f7187314 100644
}
diff --git a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-index 6e09252704..4b0f2e4d58 100644
+index 6e09252704..faacde70ff 100644
--- a/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
+++ b/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestAggregatePushDown.java
-@@ -60,6 +60,15 @@ public class TestAggregatePushDown extends CatalogTestBase {
+@@ -60,6 +60,17 @@ public class TestAggregatePushDown extends CatalogTestBase {
SparkSession.builder()
.master("local[2]")
.config("spark.sql.iceberg.aggregate_pushdown", "true")
@@ -3305,6 +3429,8 @@ index 6e09252704..4b0f2e4d58 100644
+ .config("spark.comet.explainFallback.enabled", "true")
+ .config("spark.comet.scan.icebergNative.enabled", "true")
+ .config("spark.comet.write.iceberg.splitOperator.enabled", "true")
++ .config("spark.comet.iceberg.write.enabled", "true")
++ .config("spark.comet.exec.localTableScan.enabled", "true")
+ .config("spark.memory.offHeap.enabled", "true")
+ .config("spark.memory.offHeap.size", "10g")
.enableHiveSupport()
diff --git a/docs/source/contributor-guide/iceberg-spark-tests.md b/docs/source/contributor-guide/iceberg-spark-tests.md
index 75ad2f3a1a..fb0e262e34 100644
--- a/docs/source/contributor-guide/iceberg-spark-tests.md
+++ b/docs/source/contributor-guide/iceberg-spark-tests.md
@@ -35,6 +35,15 @@ Here is an overview of the changes that the diffs make to Iceberg:
native scan in every Comet-configured session. The flag is off by default for users, so Iceberg's own suites
are the only place the split plan (`IcebergCommit -> IcebergWrite`) is exercised against Iceberg's write,
commit, and row-level-operation tests. See [#5259]
+- Enable Comet's native (iceberg-rust) Parquet writer (`spark.comet.iceberg.write.enabled`) in the same sessions.
+ The native writer is experimental and off by default for users, so this is where it runs against Iceberg's
+ write, commit, and row-level-operation tests.
+- Enable `spark.comet.exec.localTableScan.enabled` in the same sessions. `CometIcebergNativeWrite` sets
+ `requiresNativeChildren`, so without this flag a write fed by an inline `VALUES` list keeps Spark's row-based
+ `LocalTableScanExec`, the conversion is declined, and the write silently runs on the JVM writer. Many Iceberg
+ suites seed their data that way, so leaving it off hides the native writer from most of the write surface.
+- Enable fallback logging (`spark.comet.explainFallback.enabled`) so that every operator Comet declines is
+ reported in the test output together with the reason it was declined.
[#3739]: https://github.com/apache/datafusion-comet/pull/3739
[#5259]: https://github.com/apache/datafusion-comet/issues/5259
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org