You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "malinjawi (via GitHub)" <gi...@apache.org> on 2026/10/16 16:43:40 UTC
[PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
malinjawi opened a new pull request, #13338:
URL: https://github.com/apache/gluten/pull/13338
## What changes are proposed in this pull request?
Rewrites `docs/get-started/VeloxDelta.md` to follow `VeloxIceberg.md`, so it lists per Spark/Delta Lake version what Gluten offloads:
- a version matrix with read, native write and test coverage
- write commands, table features and data types, each with a status
- deletion vectors, change data feed, and how natively written files differ
- known wrong-result and exception cases, with their issues
- Gluten options, Spark SQL options, read and write options, and Delta table properties, with a Gluten Support column
Statuses come from main's code, Gluten's tests and the nightly Delta Lake test job (Delta 4.2.0 suites on Spark 4.1). "Seen on 4.1" marks results from that job, and "From code" marks results with no test behind them yet.
MERGE is listed as PartialOffload, as it is on main. #13324 would change that.
This replaces #12050, which went stale.
## How was this patch tested?
Docs only. Checked the in-page anchors and links.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM Bob
Related issue: #12025
cc @zhouyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "ReemaAlzaid (via GitHub)" <gi...@apache.org>.
ReemaAlzaid commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4312964724
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
Review Comment:
I can send it to you as an svg here you go. And thanks!
<img width="2133" height="332" alt="delta_exmaple" src="https://github.com/user-attachments/assets/b5c9e4ec-ec00-4e4f-bb6a-e4e99fe59962" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan merged PR #13338:
URL: https://github.com/apache/gluten/pull/13338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4312917007
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back |
+| Fallback | Runs on Spark or Delta Lake |
+| Exception | Fails instead of falling back |
+| ResultMismatch | Can return wrong results or write wrong files |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
Review Comment:
For` INSERT INTO` the setting makes no difference. With native write off, Delta's writer writes the files. With native write on it's the same, because INSERT plans as `AppendDataExecV1` / `OverwriteByExpressionExecV1` (or `DeltaDynamicPartitionOverwriteCommand`), and Gluten doesn't wrap those for native write yet. In both cases the SELECT part can still run in Velox. To get native writes for an append today, `df.write.format("delta").mode("append").save(path)`.
I added these details onto this section now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4312983554
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,632 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+Gluten's Velox backend can accelerate eligible scans and query operators over Delta Lake tables. Read
+acceleration does not require native writes. Native Delta writes are experimental and disabled by
+default. This guide covers the supported Spark and Delta Lake versions, setup, feature support and
+known limitations.
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+## How a Delta query runs
+
+Take a Delta table partitioned by `sale_date`:
+
+```sql
+SELECT SUM(amount)
+FROM sales
+WHERE sale_date = DATE '2026-10-01'
+ AND amount > 100;
+```
+
+```mermaid
+flowchart LR
+ subgraph Planning["Spark driver"]
+ Snapshot["1. Delta snapshot<br/>Schema + active files"]
+ Select["2. Delta file pruning<br/>Partitions + file statistics"]
+ Plan["3. Spark plan<br/>Gluten offload checks"]
+ Snapshot --> Select --> Plan
+ end
+
+ subgraph Execution["Spark executors"]
+ Scan["4. Velox scan<br/>Reads the selected files"]
+ Compute["5. Velox operators<br/>Filter + SUM"]
+ Spark["Spark operators<br/>Operators Gluten does not offload"]
+ Scan --> Compute
+ Compute <-->|"Row and columnar conversions"| Spark
+ end
+
+ subgraph Storage["Delta table"]
+ Log[("_delta_log<br/>Metadata + file statistics")]
+ Data[("Parquet files")]
+ end
+
+ Result["6. Query result"]
+
+ Log --> Snapshot
+ Plan -->|"Offloaded operators"| Scan
+ Plan -->|"Other operators"| Spark
+ Data --> Scan
+ Data --> Spark
+ Compute --> Result
+ Spark --> Result
+```
+
+Delta Lake loads the snapshot and prunes partitions and files while Spark plans the query. Gluten then
+offloads the operators Velox supports, and the rest run on Spark, so one query can mix Velox and Spark
+operators. Delta Lake keeps the table metadata, the transaction log and commits.
+
+## Supported Spark and Delta Lake versions
+
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. On Spark 4.0 and 4.1, the Read and Native write columns assume ANSI mode off (see [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x)). The copied Delta Lake suites run with native write on and `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`, which is off by default.
+
+Spark 4.1 has the most tests, because the nightly job runs on it. That job builds Delta Lake 4.2.0 from source against Spark 4.1.0, which is newer than the Delta Lake 4.1.0 in the table, and runs its Spark test suites. It puts a Gluten bundle compiled against Spark 4.1.1 and Delta Lake 4.1.0 on the test classpath. It replaces Delta Lake's `DeltaSQLCommandTest` with Gluten's copy, which loads Gluten with native write on and the non-default `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`. It then runs `.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh`. That script cherry-picks two Delta Lake test changes that are not in 4.2.0, delta-io/delta#7104 and #7105, so Delta Lake's scan checks accept Gluten's scan node. It also sets a row-group row limit in two test fixtures. The same job also runs on pull requests that change Gluten's Delta code. "Seen on 4.1" on this page refers to that job.
+
+The job's known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector, change data feed, column mapping, time travel and partition filter reads, and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+```bash
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+```
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. `dev/release/build-release.sh` passes `-Pdelta` only for Spark 3.4 and 3.5, so its Spark 4.0 and 4.1 bundles have no Delta Lake support either. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add Delta Lake with `--packages` and the coordinate from the table above, which also adds its `delta-storage` dependency. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off. From code.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+```bash
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+```
+
+On Spark 4.0 or 4.1, use the Gluten bundle and the Delta Lake jar for that version, and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code. Turning ANSI mode off also changes Spark's own behavior for invalid input. For example, `CAST('a' AS INT)` returns NULL instead of failing.
+
+```properties
+spark.sql.ansi.enabled=false
+```
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. On Spark 3.x, turning ANSI mode on makes every query fall back in the same way. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Gluten passes ANSI mode to Velox. Velox does not implement ANSI behavior for every expression, so some invalid input returns a result where Spark fails ([#10134](https://github.com/apache/gluten/issues/10134)). For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). Cast overflow errors raised in Velox have a different exception type (see [Error types](#error-types)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back. |
+| Fallback | Runs on Spark or Delta Lake. |
+| Exception | Fails instead of falling back. |
+| ResultMismatch | Can return wrong results or write wrong files. |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job showed the result, with Delta Lake 4.2.0 on Spark 4.1.0.<br>
+Reported on 3.4, Reported on 4.1: the linked issue reports the result on that Spark version. No CI test checks it.<br>
+No label: a test in the Gluten repository that runs in CI checks the status on the stated versions.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver. From code.
+
+## Creating a table
+Fallback
+
+```sql
+CREATE TABLE t (id INT, name STRING) USING delta;
+```
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload. From code.
+
+Offload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+```
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+## Writing
+PartialOffload
+
+```sql
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+```
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+```sql
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+```
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+```
+
+On Spark 3.5 to 4.1, set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). Velox writes the files of this UPDATE. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+```scala
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+```
+
+The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the command's plan from the Spark UI SQL tab or a `QueryExecutionListener`. Each write also reports a nested `deltaTransactionalWrite` execution for the query that feeds it. That plan never has the prefix, whichever writer ran. From code.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks that a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5 run as Gluten native write commands. It also checks on 3.5 and 4.0 that a DELETE that rewrites a file runs in Gluten's transaction and that its scan runs in Velox. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload, PartialOffload when it writes deletion vectors (from code on 4.1), Fallback when it is metadata-only (from code) | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. |
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | `DeltaTable.optimize().executeCompaction()`, with or without `where(...)` | Offload | Bin-packing, by path or by name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans, the DML target scan and joins can run in Velox. On Spark 3.5 and 4.0, Gluten's tests check that a deletion-vector MERGE runs a scan of the target table in Velox. The other cases are from code. Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake in `DescribeDeltaHistorySuite`. Seen on 4.1. Three unpartitioned tests in `DeleteMetricsSuite` and `UpdateMetricsSuite` expect two added files and get one. The job sets `spark.default.parallelism=1`, which plans the five small input files as one scan partition (from code).
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload after native writes, PartialOffload after other writes | Compacts natively written tables (seen on 4.1). The auto compaction tests that fail in the nightly job fail because its `spark.default.parallelism=1` makes their first write produce one file (from code). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). |
+| Files with deletion vectors | Fallback | Offload |
+| Change data feed, no deletion vectors in range | Offload. PartialOffload when the table allows deletion vectors and the range has `_change_data` files, because those scans then fall back (from code). | Offload |
+| Change data feed, deletion vectors in range | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors and no `_change_data` files (from code) | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors (from code) |
+
+```sql
+SELECT count(1) AS count, name
+FROM t
+GROUP BY name;
+```
+
+Reads need no Gluten option. With `spark.sql.adaptive.enabled=false`, `EXPLAIN` shows the native scan as `FileDeltaScanTransformer` and `EXPLAIN FORMATTED` shows `DeltaScanTransformer`. With AQE on, the default, `EXPLAIN` and `EXPLAIN FORMATTED` of a query with a shuffle, such as this one, print the plan from before Gluten offloads it. The Spark UI plan graph of a finished query shows `DeltaScanTransformer`. Delta prunes partitions and skips files by statistics while it plans the query, before the scan runs.
+
+| Read | Status | Notes |
+|---|---|---|
+| By table name, ``delta.`/path` `` or `spark.read.format("delta").load(path)` | Offload | Same native scan. |
+| `VERSION AS OF` | Offload | Tested on Spark 3.4 to 4.1. |
+| `TIMESTAMP AS OF`, `versionAsOf`, `timestampAsOf` | Offload | From code. Delta Lake's `DeltaTimeTravelSuite` passes (seen on 4.1). |
+| Streaming source | Offload | Each micro-batch is a Delta file scan. From code. Delta Lake's `DeltaSourceSuite` passes (seen on 4.1). |
+| Dynamic partition pruning | Not tested | |
+
+### Deletion vectors
+Offload
+
+```sql
+OPTIMIZE t;
+ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
+DELETE FROM t WHERE id IN (3, 4);
+SELECT * FROM t;
+```
+
+On Spark 3.5 to 4.1, Velox drops the deleted rows in the scan. This needs Delta's `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default. The DML target scan, which finds the rows for new deletion vectors, also runs in Velox. Gluten tests check the scan on Spark 3.5 to 4.1 and the DML target scan on Spark 3.5 and 4.0. Delta writes the deletion-vector files.
+
+Fallback
+
+```sql
+SET spark.databricks.delta.deletionVectors.useMetadataRowIndex=false;
+SELECT * FROM t;
+RESET spark.databricks.delta.deletionVectors.useMetadataRowIndex;
+```
+
+| Case | Behavior |
+|---|---|
+| Spark 3.5 to 4.1 with `useMetadataRowIndex=false` | A scan that reads Delta's generated `__delta_internal_is_row_deleted` or `__delta_internal_row_index` column falls back with `fallback Delta scan requiring generated DV metadata`. Delta adds the first column when any file in the scanned table version, micro-batch, DML file set or `OPTIMIZE` bin has a deletion vector, so `OPTIMIZE` and `REORG TABLE ... APPLY (PURGE)` scans of those files also fall back (from code). It adds the second to DML target scans that write deletion vectors. Other scans offload. Tested on Spark 3.5 and 4.0. |
+| Spark 3.4, normal reads | A read falls back with `fallback Spark 3.4 Delta DV scan` when a file it selected has a deletion vector. Tested on Spark 3.4. A read whose selected files have none offloads (from code). |
+| Spark 3.4, other scans, such as DML target scans, `OPTIMIZE` scans, streaming micro-batches and change data feed scans of `_change_data` files | Fall back whenever the table protocol allows deletion vectors. Change data feed scans of added or removed files fall back only when a file they read has a deletion vector. From code. |
+| `spark.databricks.delta.stats.skipping=false` | Delta Lake itself rejects reads of a table with deletion vectors with `Cannot work with a non-pinned table snapshot of the TahoeFileIndex` (seen on 4.1). Before Delta Lake 4.1.0, the table feature alone triggers it (from code). |
+
+The fallback reasons show in [Gluten UI](GlutenUI.md).
+
+### Change data feed
+Offload
+
+```sql
+CREATE TABLE c (id INT, name STRING) USING delta
+ TBLPROPERTIES ('delta.enableChangeDataFeed' = true);
+INSERT INTO c VALUES (1, 'a'), (2, 'b');
+UPDATE c SET name = 'bb' WHERE id = 2;
+SELECT id, name, _change_type, _commit_version
+FROM table_changes('c', 1)
+WHERE id = 2;
+```
+
+Gluten expands a batch change data feed read into Delta file scans that run in Velox with the filters pushed down. Delta also adds an empty placeholder scan to the plan. That scan falls back when the table schema has a MAP at any nesting level (from code). If the expansion fails, Gluten logs a warning and the read uses Delta's change data feed scan. From code.
+
+PartialOffload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false;
+SELECT * FROM table_changes('c', 1);
+```
+
+A read whose version range adds or removes a file with a deletion vector uses Delta's change data feed scan. Insert-only ranges of a table with deletion vectors enabled still run in Velox. With `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false`, every batch change data feed read uses Delta's scan. Delta's scan reads the data files in a query of its own that Gluten also plans, and `EXPLAIN` does not show it (from code). That query's file scans can run in Velox, except scans of added or removed files that have deletion vectors and, on Spark 3.4, scans of `_change_data` files when the table allows deletion vectors (from code). Streaming change data feed reads ignore this option. They run as Delta file scans. A scan whose files have deletion vectors falls back, and on Spark 3.4 so does a scan of `_change_data` files when the table allows deletion vectors (from code).
+
+### Read metadata
+Offload
+
+```sql
+SELECT id, _metadata.file_path, _metadata.row_index
+FROM t
+WHERE _metadata.row_index = 2;
+```
+
+The native scan fills the `_metadata` file fields and `input_file_name()` (from code), and `_metadata.row_index`. On Spark 3.5 to 4.1, Delta Lake exposes `_metadata.row_index` only on tables that allow deletion vectors or have row tracking, so the example needs `t` from [Deletion vectors](#deletion-vectors) (from code). A query that selects only `_metadata.file_name` can fail when the table has a BIGINT data column named `file_name` (seen on 4.1, see [Read problems](#read-problems)). Gluten tests check `_metadata.row_index` in the native scan on Spark 3.5 and 4.0, and Delta Lake tests that read it pass (seen on 4.1). Spark 3.4 is not tested. On row-tracking tables, `_metadata.row_id` and `_metadata.row_commit_version` are read natively (from code). In the nightly job, they return correct values except in the cases in [Read problems](#read-problems) (seen on 4.1).
+
+Fallback
+
+```sql
+DESCRIBE HISTORY t;
+```
+
+`DESCRIBE HISTORY` reads each commit file on Spark, through Delta's LogStore in a `mapPartitions` job. When Delta replays the log for a snapshot, every scan under `_delta_log`, checkpoints included, stays on Spark with the reason `fallback Delta _delta_log scan`. From code.
+
+## Table features
+Offload/Fallback
+
+The Read column covers Spark 3.5 to 4.1. It covers Spark 3.4 only where the row names it. The Native write column applies to Spark 3.5 to 4.1 with native write on.
+
+| Feature | Read | Native write | Notes |
+|---|---|---|---|
+| Deletion vectors | Offload, Fallback on Spark 3.4 | PartialOffload | Gluten's tests check deletion vector DELETE and MERGE results with native write on, on Spark 3.5 and 4.0. Which writer writes each file is from code. See [Deletion vectors](#deletion-vectors). |
+| Change data feed | Offload, also on Spark 3.4. Ranges with deletion vectors fall back. On Spark 3.4, scans of `_change_data` files also fall back when the table allows deletion vectors (from code). | Offload | UPDATE and DELETE write the `_change_data` files with Velox (from code). Change data feed reads of their changes are correct (seen on 4.1). See [Change data feed](#change-data-feed). |
+| Column mapping, name mode | Offload, also on Spark 3.4 | ResultMismatch | Files have no Parquet field IDs. Seen on 4.1. See [Write problems](#write-problems). |
+| Column mapping, id mode | Offload, also on Spark 3.4. ResultMismatch when field IDs differ from physical names (seen on 4.1). | ResultMismatch, Exception when Spark's reader reads the files (seen on 4.1) | Gluten matches columns by physical name. See [Known problems](#known-problems). |
+| Identity columns | Offload | Offload | Seen on 4.1. High-water marks are correct. The projection that generates identity values runs on Spark (from code). |
+| Generated columns | Offload | Offload | Seen on 4.1. No Gluten test covers generated columns. Velox computes the generated values when it supports the expression (from code). Partition filters derived from them reach the native scan. `DATE_FORMAT` results for years above 9999 differ (see [Write problems](#write-problems)). |
+| CHECK constraints, NOT NULL | Offload | Offload | Delta checks the rows on the JVM before Velox writes them. Gluten's tests cover NOT NULL on Spark 3.5 and 4.0. Seen on 4.1. |
+| Row tracking | Offload, Exception for `_metadata.base_row_id` filters, ResultMismatch for `_metadata.row_id` when older files lack base row IDs | Offload | Seen on 4.1. See [Read problems](#read-problems). |
+| Liquid clustering | Offload | PartialOffload | Reads use the same native scan as other tables. Native commands write to clustered tables. `OPTIMIZE`, which clusters the data, uses Delta's writer. All three are from code. |
+| UniForm (IcebergCompatV1, V2) | | ResultMismatch, Exception when partitioned | The write status is from code. No test reads a UniForm table. |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Offload, Exception for FLOAT or DOUBLE partition columns, for TIMESTAMP_NTZ partition values with sub-millisecond digits and for some writes of TIMESTAMP_NTZ data columns | Seen on 4.1. Spark 3.5 and 4.0 are not covered. |
+| TIMESTAMP_NTZ | PartialOffload, also on Spark 3.4 | Offload, Exception for some writes (seen on 4.1) | See [DataType](#datatype). |
+| Variant, including shredding | Fallback | Fallback | Spark 4.x. From code. See [DataType](#datatype). |
+| Collations | Fallback | | Spark 4.x. From code. See [DataType](#datatype). |
+| materializePartitionColumns | | ResultMismatch | Spark 4.1. From code. |
+
+## DataType
+
+The Read column applies to Spark 3.4 to 4.1. On all of them, Gluten's tests read INT, BIGINT, DECIMAL(12,2), STRING, DATE, TIMESTAMP, TIMESTAMP_NTZ, STRUCT, ARRAY and MAP columns natively. The other read statuses are from code, unless a row says otherwise. The native write columns apply to Spark 3.5 to 4.1 with native write on. On Spark 3.5 and 4.0, Gluten's tests write BOOLEAN, INT, BIGINT, STRING, STRUCT and ARRAY columns natively. The other native write statuses are seen on 4.1, unless a row says they are from code.
+
+| Type | Read | Native write | Native write as partition column | Notes |
|---|---|---|---|---|
-| `spark-3.5` | Spark 3.5.x | 2.12 | 3.3.x | Supported |
-| `spark-4.0` | Spark 4.0.x | 2.13 | 4.0.x | Supported |
-
-Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The difference between
-the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta 3.3 vs Spark 4.0 +
-Delta 4.0), not a native-write capability gap.
-
-## Build and runtime notes
-
-Build Gluten with Delta support by enabling `-Pdelta` together with the Velox backend profile and a Spark profile.
-
-- Spark 3.5 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests`
-- Spark 4.0 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13 -Pjava-17 -DskipTests`
-
-Native Delta write is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite`
- - Default: `false`
- - Type: experimental
-
-Native change data feed scan offload is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan`
- - Default: `true`
- - Type: experimental
-
-The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan`
- - Default: `true`
- - Type: experimental
- - When disabled, the DML target scan that produces file paths and row indexes for
- deletion-vector writes stays on Spark; other scans are unaffected.
-
-| Feature | Delta minWriterVersion | Delta minReaderVersion | Iceberg format-version | Feature type | Supported by Gluten (Velox) |
-|---|---:|---:|---:|---|---|
-| Basic functionality | 2 | 1 | 1 | Writer | Yes |
-| CHECK constraints | 3 | 1 | N/A | Writer | No |
-| Change data feed | 4 | 1 | N/A | Writer | Yes |
-| Generated columns | 4 | 1 | N/A | Writer | Partial |
-| Column mapping | 5 | 2 | N/A | Reader and writer | Yes |
-| Identity columns | 6 | 1 | N/A | Writer | Yes |
-| Row tracking | 7 | 1 | 3 | Writer | Partial |
-| Deletion vectors | 7 | 3 | 3 | Reader and writer | Partial |
-| TimestampNTZ | 7 | 3 | 1 | Reader and writer | No |
-| Liquid clustering | 7 | 3 | 1 | Reader and writer | Yes |
-| Iceberg readers (UniForm) | 7 | 2 | N/A | Writer | Not tested |
-| Type widening | 7 | 3 | N/A | Reader and writer | Partial |
-| Variant | 7 | 3 | 3 | Reader and writer | Not tested |
-| Variant shredding | 7 | 3 | 3 | Reader and writer | Not tested |
-| Collations | 7 | 3 | N/A | Reader and writer | Not tested |
-| Protected checkpoints | 7 | 1 | N/A | Writer | Not tested |
+| BOOLEAN | Offload | Offload | Offload | |
+| TINYINT, SMALLINT, INT, BIGINT | Offload | Offload | Offload | |
+| FLOAT, DOUBLE | Offload | Offload | Exception | See [Write problems](#write-problems). |
+| DECIMAL | Offload | Offload | Offload | Precision above 18 included. |
+| STRING | Offload | Offload | Offload | |
+| CHAR, VARCHAR | Offload | Offload | | Stored as STRING. Delta checks the length on the JVM before the native write. Both statuses are from code. Reads return correct results (seen on 4.1). |
+| BINARY | Offload | Offload | Not tested | The read status is from code. One Delta Lake test writes a table with only a BINARY column, checks its file statistics and runs a DELETE that filters on that column (seen on 4.1). |
+| DATE | Offload | Offload | Offload | |
+| TIMESTAMP | Offload, ResultMismatch for sub-millisecond partition values (seen on 4.1) | Offload | Offload, Exception for sub-millisecond values | The partition column status is from code. See [Known problems](#known-problems). |
+| TIMESTAMP_NTZ | PartialOffload | Offload, Exception for some writes (seen on 4.1) | Offload, Exception for sub-millisecond values (seen on 4.1) | Filters, sorts, joins and other operators whose input or output includes a TIMESTAMP_NTZ column fall back, even when they do not use that column. Scans, aggregates, shuffles and some projections can run in Velox. A scan whose pushed filter has a TIMESTAMP_NTZ literal falls back (from code). Gluten's tests check on Spark 3.4 to 4.1 that a scan of the column runs in Velox, and that a filter on the column and a sort on another column cause a fallback. The other operator rules are from code. |
+| STRUCT, ARRAY, MAP | Offload | Offload | | |
+| VARIANT | Fallback | Fallback | | Spark 4.x. Shredded variant reads fall back too. Both statuses are from code. |
+| Collated STRING | Fallback | | | Spark 4.x. Scans that read the column fall back. Delta Lake 4.0.1 and 4.1.0 reject a collated column in a table schema with `DELTA_UNSUPPORTED_DATA_TYPES`. They cannot write to tables that have the `collations` feature. From code. |
+
+A plan node falls back when its output contains a type Velox does not support, such as VARIANT. With the default `spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames=true`, a scan that does not select such a column can still offload. From code. Delta Lake does not allow interval columns.
+
+## Format
+PartialOffload
+
+Delta data files are Parquet, and Gluten reads them with Velox's Parquet reader. Velox writes them only for native write commands on Spark 3.5 to 4.1. Otherwise Delta's Parquet writer writes them.
+
+### Natively written files
+
+| Property | Behavior |
+|---|---|
+| Compression | snappy (default), gzip, zstd (always level 3), lz4 (Hadoop framing), none. From code. Other codecs: see [Write problems](#write-problems). |
+| Row groups | Sized by `spark.gluten.sql.columnar.parquet.write.blockSize` and `spark.gluten.sql.native.parquet.write.blockRows`. `parquet.*` settings are ignored. From code. |
+| TIMESTAMP | INT64 microseconds. `spark.sql.parquet.outputTimestampType` is ignored. Delta Lake's writer uses INT96 by default. Spark reads both. From code. |
+| Parquet field IDs | Not written, for column-mapped and UniForm tables too. Seen on 4.1 for column mapping, from code for UniForm. See [Write problems](#write-problems). |
+| Spark schema footer entry | `org.apache.spark.sql.parquet.row.metadata` is not written ([#13288](https://github.com/apache/gluten/issues/13288)). Seen on 4.1. |
+| Nested field names | Lowercased below the top level when `spark.sql.caseSensitive=false`. Spark and Gluten match them case-insensitively. A case-sensitive reader can miss them. From code. |
+| Page index | Always written. From code. |
+| `spark.sql.parquet.writeLegacyFormat=true` | Applies to DECIMAL only. Arrays and maps keep the standard layout. From code. |
+| File statistics | numRecords, min, max and nullCount are computed in Velox (checked on Spark 3.5). Gluten's tests check the values against the rows in each file on Spark 3.5 to 4.1. Most tests in Delta's data-skipping suites pass (seen on 4.1). Of the 34 that fail, 22 check statistics and expect specific rows in each file after `repartition(2)`. The other 12 hit the `_metadata.file_name` failure in [Read problems](#read-problems). |
+| Data file paths | Come from Delta's committer, so names and random prefixes follow Delta. File names seen on 4.1, random prefixes from code. |
+
+## SQL
+
+| Statement | Status | Notes |
+|---|---|---|
+| `SELECT` | Offload | See [Reading](#reading). |
+| `table_changes()` | Offload | Ranges with deletion vectors fall back. See [Change data feed](#change-data-feed). |
+| `CREATE TABLE ... AS SELECT`, `REPLACE TABLE ... AS SELECT` | Offload | With native write on, Spark 3.5 to 4.1, and a DeltaCatalog target. Seen on 4.1, from code on 3.5 and 4.0. See [Creating a table](#creating-a-table). |
+| `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `OPTIMIZE`, `REORG` | | See [Write commands](#write-commands). |
+| `CREATE TABLE` without `AS SELECT`, `DROP TABLE`, `DESCRIBE` | Fallback | They read and write no data files. From code. |
+| `ALTER TABLE` | Fallback, PartialOffload when it reads data | Most forms read and write no data files. `ADD CONSTRAINT` scans the table, and that scan can run in Velox. On Spark 3.5 to 4.1, `ALTER COLUMN ... SYNC IDENTITY` also scans it, and `DROP FEATURE` and `SET TBLPROPERTIES ('delta.columnMapping.mode' = 'none')` can rewrite data files with Delta's writer. From code. |
+| `VACUUM`, `RESTORE`, `SHALLOW CLONE` | | They write no Parquet data files. From code. |
+| `CONVERT TO DELTA` | PartialOffload | Writes no Parquet data files. Unless the statement has `NO STATISTICS` or `spark.databricks.delta.stats.collect=false` is set, it scans the Parquet files for statistics. That scan can run in Velox. From code. |
+
+## Schema evolution
+PartialOffload
+
+The `mergeSchema` and `overwriteSchema` rows need native write on Spark 3.5 to 4.1. Otherwise Delta writes the files.
+
+| Change | Status | Notes |
+|---|---|---|
+| `mergeSchema` with `save(path)` | Offload | Velox writes the evolved schema (from code). On Spark 3.5 and 4.0, the copied `DeltaSuite` checks the results of a `mergeSchema` overwrite. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). |
+| `overwriteSchema` with `save(path)` | Offload | Delta applies the change before the write. From code. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). |
+| Schema evolution in SQL `INSERT` | PartialOffload | From code. |
+| Schema evolution in `MERGE` | PartialOffload, Exception for some STRUCT map key changes | The merged table contents match Delta Lake. Some STRUCT map key changes fail in Velox (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). The two TIMESTAMP_NTZ widening tests fail before the MERGE runs, in a setup write of TIMESTAMP_NTZ values. Seen on 4.1. |
+| `RENAME COLUMN` with column mapping | Offload | Reads after top-level and nested renames return correct results. After a top-level rename, partition pruning and file skipping still work. |
+| `DROP COLUMN` with column mapping | Offload | From code. Delta Lake's `DeltaDropColumnSuite` passes (seen on 4.1). |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ and for native writes to FLOAT or DOUBLE partition columns | Seen on 4.1. See [Table features](#table-features). |
+| `spark.sql.parquet.mergeSchema=true` | Fallback | Every Parquet scan falls back, Delta scans included. From code. |
+
+## Known problems
+
+### Read problems
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A column default stored as `EXISTS_DEFAULT` metadata, for example after `CONVERT TO DELTA`, reads as NULL for rows in files without that column. | ResultMismatch | Seen on 4.1. #11494 reports the same bug for plain Parquet on 3.4 to 4.1. | [#11494](https://github.com/apache/gluten/issues/11494) |
+| In column mapping id mode, when field IDs no longer match the Parquet column names, Gluten returns the column with the matching physical name. Delta Lake returns NULL when no file column has that ID (seen on 4.1). It returns the column the ID points to when another column has it (from code). | ResultMismatch | 3.4 to 4.1. Seen on 4.1 for the NULL case, from code otherwise. | [#12241](https://github.com/apache/gluten/issues/12241) |
+| On a TIMESTAMP-partitioned table with six partition values from .1 to .123456 seconds, `count(DISTINCT ts)` returns 3, as if the values were truncated to milliseconds. Equality filters on the column still return the right rows. | ResultMismatch | Seen on 4.1 | |
+| Reading a data column widened from DATE to TIMESTAMP_NTZ fails on files written before the change with `Converted type DATE is not allowed for requested type TIMESTAMP UTC`. A partition column widened the same way reads correctly. | Exception | Seen on 4.1 | |
+| A query that selects only `_metadata.file_name` from a table with a BIGINT data column named `file_name` reads the data column and fails with `Converted type BIGINT is not allowed for requested type VARCHAR`. | Exception | Seen on 4.1 | |
+| An equality filter on `_metadata.base_row_id` fails with `Synthesized column 'base_row_id' failed filter validation` when a scanned file has a different base row ID. | Exception | Seen on 4.1 | |
+| On a row-tracking table whose older files lack base row IDs, selecting `_metadata.row_id` returns rows. Delta raises an error. Delta Lake's test creates this state with a manual metadata commit. `ALTER TABLE ... SET TBLPROPERTIES` backfills base row IDs when it enables row tracking. | ResultMismatch | Seen on 4.1 | |
+| Reading or deleting from a 2-billion-row table with deletion vectors can grow native memory until the JVM is killed. Delta Lake's two 2-billion-row tests reproduce it. The nightly job force-fails both tests. | Exception | Reported on 4.1 | [#12387](https://github.com/apache/gluten/issues/12387) |
+| With `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default, selecting `__delta_internal_is_row_deleted` from a relation built by hand over `DeltaParquetFileFormat` drops the deleted rows and returns 1 for the rows it keeps. The column is not in the table schema, so SQL and DataFrame queries over the table cannot select it. | ResultMismatch | Seen on 4.1 | |
+| Scans of a table with many small files can be slower than vanilla Spark on EBS storage. In the report, two of five TPC-DS SF1000 queries take 2.2 and 3.4 times as long as on vanilla Spark, with 182,300 files in one table. A maintainer did not see the slowdown on a host with local NVMe storage. | | Reported on 3.4 | [#10778](https://github.com/apache/gluten/issues/10778) |
+
+### Write problems
+
+These need native write on, unless a row says otherwise.
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A write partitioned by a FLOAT or DOUBLE column fails with `Unsupported partition type`. | Exception | 3.5 to 4.1, seen on 4.1 | [#13138](https://github.com/apache/gluten/issues/13138) |
+| A write to a path scheme the native library cannot write fails at task time. This covers custom Hadoop file systems, and S3, GCS, HDFS or ABFS in a build without that connector. | Exception | 3.5 to 4.1, seen on 4.1 (custom file system), from code (missing connector) | [#4963](https://github.com/apache/gluten/issues/4963) |
+| Files written for a column mapping table, in name or id mode, have no Parquet field IDs, which the Delta Lake protocol requires. Delta Lake and Gluten read name-mode files by physical name. Readers that match by field ID cannot. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| Spark's own reader fails on natively written id-mode files with `Spark read schema expects field Ids, but Parquet file schema doesn't contain any field Ids`. Spark uses this reader when a scan falls back or Gluten is off. With `spark.sql.parquet.fieldId.read.ignoreMissing=true`, Spark skips this check and returns NULL for the data columns of these files (from code). That setting is not tested with these files. | Exception | 3.5 to 4.1, seen on 4.1 | |
+| Files written for a UniForm table have no Parquet field IDs, so the Iceberg metadata points at files without IDs. No error is raised. | ResultMismatch | 3.5 to 4.1, from code | [#12957](https://github.com/apache/gluten/pull/12957) |
+| A write to a partitioned UniForm table fails with `The file schema type should be equal with the input rowvector type`. | Exception | 3.5 to 4.1, from code | |
+| Writes to a table with the `materializePartitionColumns` feature leave the partition columns out of the data files. | ResultMismatch | 4.1, from code | |
+| Some writes of a TIMESTAMP_NTZ data column fail with `The file schema type should be equal with the input rowvector type`. Delta Lake's type widening tests fail this way when they write `CAST(value AS TIMESTAMP_NTZ)`. Delta Lake's data skipping tests also write that cast and pass. On Spark 4.x, `spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation=true` sends TIMESTAMP_NTZ writes to Delta's writer. It is not tested as a workaround. | Exception | Seen on 4.1 | |
+| A partitioned write fails in Velox's `PartitionIdGenerator` when a TIMESTAMP or TIMESTAMP_NTZ partition value has sub-millisecond digits. | Exception | 3.5 to 4.1, seen on 4.1 (TIMESTAMP_NTZ), from code (TIMESTAMP) | |
+| A generated partition column such as `DATE_FORMAT(ts, 'yyyy-MM')` stores `12345-07` for the year 12345. Spark stores `+12345-07`. Not checked with native write off. | ResultMismatch | Seen on 4.1 | |
+| The `compression` option set to brotli or lzo fails the write. So does `spark.sql.parquet.compression.codec` when the option is unset. The write fails even with the Hadoop codec class on the classpath. Without that class, Delta's writer fails too, and Spark does not ship it. | Exception | 3.5 to 4.1, from code | |
+| The `compression` option or `spark.sql.parquet.compression.codec` set to lz4_raw writes Snappy data into a `.lz4raw.parquet` file. | ResultMismatch | 3.5 to 4.1, from code | |
+| `maxRecordsPerFile` on an unpartitioned write counts rows but writes whole batches. When the limit is smaller than a batch, the whole batch goes into one file. The other files opened for that batch are 0-byte files that are not valid Parquet. | ResultMismatch | 3.5 to 4.1, from code | |
+| `spark.sql.maxConcurrentOutputFileWriters` above 0 makes a partitioned write with unsorted input fail with `Underlying columnar data is inaccessible from BatchCarrierRow`. | Exception | 3.5 to 4.1, from code | |
+| An unpartitioned native write whose first task gets no rows, for example a write of an empty DataFrame, leaves a 0-byte file that is not valid Parquet. Reading the table directory as plain Parquet can then fail with `is not a Parquet file (length is too low: 0)`. With statistics on, the file stays on disk until `VACUUM`. With `spark.databricks.delta.stats.collect=false`, Delta commits it. The last two sentences are from code. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| A statistics aggregation that Velox cannot offload fails the write with `ClassCastException`. With the default `spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation=false`, the TIMESTAMP_NTZ case from the issue no longer fails on main. Tested on Spark 3.5, seen on 4.1, from code on 4.0. On Spark 3.5 with that option set to true, the TIMESTAMP_NTZ case still fails (from code). | Exception | 3.5 to 4.1, from code | [#12538](https://github.com/apache/gluten/issues/12538) |
+
+### MERGE, UPDATE and DELETE problems
+
+MERGE does not use native write, so the MERGE rows also apply with native write off (from code).
Review Comment:
I double-checked this one and the statement holds as MERGE isn't wrapped for native write, so Delta's writer still writes its files (none of the MERGE traces on main go through Gluten's writer; #13324 adds the native path which is still WIP). These problems come from the parts of MERGE that run in Velox, so they show up either way.
I reworded it to tie it to the Write commands table, and fixed the deletion vector row that made it look like MERGE writes natively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4312796251
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
Review Comment:
Thanks, both added. The overview is at the top with your wording, and right after it there's a short "How a Delta query runs" section with your example query and diagram. I tweaked a couple of labels: the Spark box now says "Operators Gluten does not offload", and the left box is "Query planning", since Delta's log replay can also run on executors. I also added a closing line saying Delta Lake keeps the table metadata, the transaction log and commits.
One heads-up: our docs site uses an older just-the-docs theme without mermaid, so the diagram renders here on GitHub but would show as a code block on the site. Happy to swap it for a PNG if you'd prefer.
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back |
+| Fallback | Runs on Spark or Delta Lake |
+| Exception | Fails instead of falling back |
+| ResultMismatch | Can return wrong results or write wrong files |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+````
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+````
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+````
+
+Set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+````
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+````
+
+The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the plan from the Spark UI SQL tab or a `QueryExecutionListener`.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. |
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | | Offload | Bin-packing, by path or by name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans and the DML target scan can run in Velox. Joins can too (from code). Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake. Seen on 4.1.
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload | Compacts natively written tables (seen on 4.1). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name (from code) | Offload, matched by physical name (seen on 4.1) |
Review Comment:
Took the pointer to Read problems, it reads much cleaner. I kept the cells as plain "Offload" instead of "(from code)" / "(seen on 4.1)", though: the `column mapping mode = id` test in DeltaSuite runs on Spark 3.4 to 4.1 in CI and fails on any fallback, so the offload part is covered by a real test.
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back |
+| Fallback | Runs on Spark or Delta Lake |
+| Exception | Fails instead of falling back |
+| ResultMismatch | Can return wrong results or write wrong files |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+````
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+````
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+````
+
+Set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+````
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+````
+
+The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the plan from the Spark UI SQL tab or a `QueryExecutionListener`.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. |
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | | Offload | Bin-packing, by path or by name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans and the DML target scan can run in Velox. Joins can too (from code). Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake. Seen on 4.1.
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload | Compacts natively written tables (seen on 4.1). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name (from code) | Offload, matched by physical name (seen on 4.1) |
+| Files with deletion vectors | Fallback | Offload |
+| Change data feed, no deletion vectors in range | Offload. Scans of `_change_data` files fall back when the table allows deletion vectors (from code). | Offload |
+| Change data feed, deletion vectors in range | Fallback | Fallback |
+
+````
+SELECT count(1) AS count, name
+FROM t
+GROUP BY name;
+````
+
+Reads need no Gluten option. In `EXPLAIN` output, the native scan shows up as `FileDeltaScanTransformer`. `EXPLAIN FORMATTED` and the Spark UI plan graph show `DeltaScanTransformer`. Delta prunes partitions and skips files by statistics while it plans the query, before the scan runs.
+
+| Read | Status | Notes |
+|---|---|---|
+| By table name, ``delta.`/path` `` or `spark.read.format("delta").load(path)` | Offload | Same native scan. |
+| `VERSION AS OF` | Offload | Tested on Spark 3.4 to 4.1. |
+| `TIMESTAMP AS OF`, `versionAsOf`, `timestampAsOf` | Offload | From code. Delta Lake's `DeltaTimeTravelSuite` passes (seen on 4.1). |
+| Streaming source | Offload | Each micro-batch is a Delta file scan. From code. Delta Lake's `DeltaSourceSuite` passes (seen on 4.1). |
+| Dynamic partition pruning | Not tested | |
+
+### Deletion vectors
+Offload
+
+````
+ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
+DELETE FROM t WHERE id IN (3, 4);
+SELECT * FROM t;
+````
+
+On Spark 3.5 to 4.1, Velox drops the deleted rows in the scan. This needs Delta's `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default. The DML target scan, which finds the rows for new deletion vectors, also runs in Velox. Gluten tests check the scan on Spark 3.5 to 4.1 and the DML target scan on Spark 3.5 and 4.0. Delta writes the deletion-vector files.
+
+Fallback
+
+````
+SET spark.databricks.delta.deletionVectors.useMetadataRowIndex=false;
+SELECT * FROM t;
+````
+
+| Case | Behavior |
+|---|---|
+| Spark 3.5 to 4.1 with `useMetadataRowIndex=false` | A scan that reads Delta's generated `__delta_internal_is_row_deleted` or `__delta_internal_row_index` column falls back with `fallback Delta scan requiring generated DV metadata`. Delta adds the first column when any file in the scanned table version, micro-batch or DML file set has a deletion vector. It adds the second to DML target scans that write deletion vectors. Other scans offload. Tested on Spark 3.5 and 4.0. |
+| Spark 3.4, normal reads | A read falls back with `fallback Spark 3.4 Delta DV scan` when a file it selected has a deletion vector. Tested on Spark 3.4. A read whose selected files have none offloads (from code). |
+| Spark 3.4, DML target scans, streaming micro-batches and change data feed scans of `_change_data` files | Fall back whenever the table protocol allows deletion vectors. From code. |
+| `spark.databricks.delta.stats.skipping=false` | Delta Lake itself rejects reads of a table with deletion vectors with `Cannot work with a non-pinned table snapshot of the TahoeFileIndex` (seen on 4.1). Before Delta Lake 4.1.0, the table feature alone triggers it (from code). |
+
+The fallback reasons show in [Gluten UI](GlutenUI.md).
+
+### Change data feed
+Offload
+
+````
+CREATE TABLE c (id INT, name STRING) USING delta
+ TBLPROPERTIES ('delta.enableChangeDataFeed' = true);
+INSERT INTO c VALUES (1, 'a'), (2, 'b');
+UPDATE c SET name = 'bb' WHERE id = 2;
+SELECT id, name, _change_type, _commit_version
+FROM table_changes('c', 1)
+WHERE id = 2;
+````
+
+Gluten expands a batch change data feed read into Delta file scans that run in Velox with the filters pushed down. Delta also adds an empty placeholder scan to the plan. It falls back when the table has a MAP column (from code). If the expansion fails, Gluten logs a warning and the read uses Delta's change data feed scan on Spark. From code.
+
+Fallback
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false;
+SELECT * FROM table_changes('c', 1);
+````
+
+A read whose version range adds or removes a file with a deletion vector stays on Delta's change data feed scan on Spark. Insert-only ranges of a table with deletion vectors enabled still run in Velox. With `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false`, every batch change data feed read uses Delta's scan. Streaming change data feed reads ignore this option. They run as Delta file scans, and a scan whose files have deletion vectors falls back (from code).
+
+### Read metadata
+Offload
+
+````
+SELECT id, _metadata.file_path, _metadata.row_index
+FROM t
+WHERE _metadata.row_index = 2;
+````
+
+The native scan fills the `_metadata` file fields and `input_file_name()` (from code), and `_metadata.row_index`. Gluten tests check `_metadata.row_index` in the native scan on Spark 3.5 and 4.0, and Delta Lake tests that read it pass (seen on 4.1). Spark 3.4 is not tested. On row-tracking tables, `_metadata.row_id` and `_metadata.row_commit_version` are read natively (from code). They return correct values except in the cases in [Read problems](#read-problems) (seen on 4.1).
+
+Fallback
+
+````
+DESCRIBE HISTORY t;
+````
+
+Delta reads the transaction log on Spark. Every scan under `_delta_log`, checkpoints included, stays on Spark. From code. The reason is `fallback Delta _delta_log scan`.
+
+## Table features
+Offload/Fallback
+
+The Read column covers Spark 3.5 to 4.1. It covers Spark 3.4 only where the row names it. The Native write column applies to Spark 3.5 to 4.1 with native write on.
+
+| Feature | Read | Native write | Notes |
+|---|---|---|---|
+| Deletion vectors | Offload, Fallback on Spark 3.4 | PartialOffload | See [Deletion vectors](#deletion-vectors). |
+| Change data feed | Offload, also on Spark 3.4. Ranges with deletion vectors fall back. On Spark 3.4, scans of `_change_data` files also fall back when the table allows deletion vectors (from code). | Offload | UPDATE and DELETE write the `_change_data` files with Velox (from code). Change data feed reads of their changes are correct (seen on 4.1). See [Change data feed](#change-data-feed). |
+| Column mapping, name mode | Offload, also on Spark 3.4 | ResultMismatch | Files have no Parquet field IDs. Seen on 4.1. See [Write problems](#write-problems). |
+| Column mapping, id mode | Offload, also on Spark 3.4. ResultMismatch when field IDs differ from physical names (seen on 4.1). | ResultMismatch, Exception when Spark's reader reads the files (seen on 4.1) | Gluten matches columns by physical name. See [Known problems](#known-problems). |
+| Identity columns | Offload | Offload | Seen on 4.1. High-water marks are correct. The projection that generates identity values runs on Spark (from code). |
+| Generated columns | Offload | Offload | Seen on 4.1. No Gluten test covers generated columns. Velox computes the generated values when it supports the expression (from code). Partition filters derived from them reach the native scan. `DATE_FORMAT` results for years above 9999 differ (see [Write problems](#write-problems)). |
+| CHECK constraints, NOT NULL | Offload | Offload | Delta checks the rows on the JVM before Velox writes them. Gluten's tests cover NOT NULL on Spark 3.5 and 4.0. Seen on 4.1. |
+| Row tracking | Offload, Exception for `_metadata.base_row_id` filters | Offload | Seen on 4.1. |
+| Liquid clustering | Offload | PartialOffload | Native commands write to clustered tables. `OPTIMIZE`, which clusters the data, uses Delta's writer. Both are from code. |
+| UniForm (IcebergCompatV1, V2) | | ResultMismatch, Exception when partitioned | The write status is from code. No test reads a UniForm table. |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Offload, Exception for FLOAT or DOUBLE partition columns | Seen on 4.1. Spark 3.5 and 4.0 are not covered. |
+| TIMESTAMP_NTZ | PartialOffload, also on Spark 3.4 | Offload, Exception for some writes (seen on 4.1) | See [DataType](#datatype). |
+| Variant, including shredding | Fallback | Fallback | Spark 4.x. From code. See [DataType](#datatype). |
+| Collations | Fallback | | Spark 4.x. From code. See [DataType](#datatype). |
+| materializePartitionColumns | | ResultMismatch | Spark 4.1. From code. |
+
+## DataType
+
+The native write columns apply to Spark 3.5 to 4.1 with native write on. On Spark 3.5 and 4.0, Gluten's tests write BOOLEAN, INT, BIGINT, STRING, STRUCT and ARRAY columns natively. The other native write statuses are seen on 4.1, unless a row says they are from code.
+
+| Type | Read | Native write | Native write as partition column | Notes |
|---|---|---|---|---|
-| `spark-3.5` | Spark 3.5.x | 2.12 | 3.3.x | Supported |
-| `spark-4.0` | Spark 4.0.x | 2.13 | 4.0.x | Supported |
-
-Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The difference between
-the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta 3.3 vs Spark 4.0 +
-Delta 4.0), not a native-write capability gap.
-
-## Build and runtime notes
-
-Build Gluten with Delta support by enabling `-Pdelta` together with the Velox backend profile and a Spark profile.
-
-- Spark 3.5 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests`
-- Spark 4.0 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13 -Pjava-17 -DskipTests`
-
-Native Delta write is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite`
- - Default: `false`
- - Type: experimental
-
-Native change data feed scan offload is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan`
- - Default: `true`
- - Type: experimental
-
-The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan`
- - Default: `true`
- - Type: experimental
- - When disabled, the DML target scan that produces file paths and row indexes for
- deletion-vector writes stays on Spark; other scans are unaffected.
-
-| Feature | Delta minWriterVersion | Delta minReaderVersion | Iceberg format-version | Feature type | Supported by Gluten (Velox) |
-|---|---:|---:|---:|---|---|
-| Basic functionality | 2 | 1 | 1 | Writer | Yes |
-| CHECK constraints | 3 | 1 | N/A | Writer | No |
-| Change data feed | 4 | 1 | N/A | Writer | Yes |
-| Generated columns | 4 | 1 | N/A | Writer | Partial |
-| Column mapping | 5 | 2 | N/A | Reader and writer | Yes |
-| Identity columns | 6 | 1 | N/A | Writer | Yes |
-| Row tracking | 7 | 1 | 3 | Writer | Partial |
-| Deletion vectors | 7 | 3 | 3 | Reader and writer | Partial |
-| TimestampNTZ | 7 | 3 | 1 | Reader and writer | No |
-| Liquid clustering | 7 | 3 | 1 | Reader and writer | Yes |
-| Iceberg readers (UniForm) | 7 | 2 | N/A | Writer | Not tested |
-| Type widening | 7 | 3 | N/A | Reader and writer | Partial |
-| Variant | 7 | 3 | 3 | Reader and writer | Not tested |
-| Variant shredding | 7 | 3 | 3 | Reader and writer | Not tested |
-| Collations | 7 | 3 | N/A | Reader and writer | Not tested |
-| Protected checkpoints | 7 | 1 | N/A | Writer | Not tested |
+| BOOLEAN | Offload | Offload | Offload | |
+| TINYINT, SMALLINT, INT, BIGINT | Offload | Offload | Offload | |
+| FLOAT, DOUBLE | Offload | Offload | Exception | See [Write problems](#write-problems). |
+| DECIMAL | Offload | Offload | Offload | Precision above 18 included. |
+| STRING | Offload | Offload | Offload | |
+| CHAR, VARCHAR | Offload | Offload | | Stored as STRING. Delta checks the length on the JVM before the native write. Both statuses are from code. Reads return correct results (seen on 4.1). |
+| BINARY | Not tested | Not tested | Not tested | Only indirect coverage. |
+| DATE | Offload | Offload | Offload | |
+| TIMESTAMP | Offload, ResultMismatch for sub-millisecond partition values (seen on 4.1) | Offload | Offload, Exception for sub-millisecond values | The partition column status is from code. See [Known problems](#known-problems). |
+| TIMESTAMP_NTZ | PartialOffload | Offload, Exception for some writes (seen on 4.1) | Offload, Exception for sub-millisecond values (seen on 4.1) | Filters, sorts, joins and other operators whose input or output includes a TIMESTAMP_NTZ column fall back, even when they do not use that column. Scans, aggregates, shuffles and some projections can run in Velox. These operator rules are from code. |
+| STRUCT, ARRAY, MAP | Offload | Offload | | |
+| VARIANT | Fallback | Fallback | | Spark 4.x. Shredded variant reads fall back too. Both statuses are from code. |
+| Collated STRING | Fallback | | | Spark 4.x. Scans that read the column fall back. Delta Lake 4.0.1 and 4.1.0 reject a collated column in a table schema with `DELTA_UNSUPPORTED_DATA_TYPES`. They cannot write to tables that have the `collations` feature. From code. |
+
+A plan node falls back when its output contains a type Velox does not support, such as VARIANT. With the default `spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames=true`, a scan that does not select such a column can still offload. From code. Delta Lake does not allow interval columns.
+
+## Format
+PartialOffload
+
+Delta data files are Parquet, and Gluten reads them with Velox's Parquet reader. Velox writes them only for native write commands on Spark 3.5 to 4.1. Otherwise Delta's Parquet writer writes them.
+
+### Natively written files
+
+| Property | Behavior |
+|---|---|
+| Compression | snappy (default), gzip, zstd (always level 3), lz4 (Hadoop framing), none. From code. Other codecs: see [Write problems](#write-problems). |
+| Row groups | Sized by `spark.gluten.sql.columnar.parquet.write.blockSize` and `spark.gluten.sql.native.parquet.write.blockRows`. `parquet.*` settings are ignored. From code. |
+| TIMESTAMP | INT64 microseconds. `spark.sql.parquet.outputTimestampType` is ignored. Delta Lake's writer uses INT96 by default. Spark reads both. From code. |
+| Parquet field IDs | Not written, for column-mapped and UniForm tables too. Seen on 4.1 for column mapping, from code for UniForm. See [Write problems](#write-problems). |
+| Spark schema footer entry | `org.apache.spark.sql.parquet.row.metadata` is not written ([#13288](https://github.com/apache/gluten/issues/13288)). Seen on 4.1. |
+| Nested field names | Lowercased below the top level when `spark.sql.caseSensitive=false`. Spark and Gluten match them case-insensitively. A case-sensitive reader can miss them. From code. |
+| Page index | Always written. From code. |
+| `spark.sql.parquet.writeLegacyFormat=true` | Applies to DECIMAL only. Arrays and maps keep the standard layout. From code. |
+| File statistics | numRecords, min, max and nullCount are computed in Velox (checked on Spark 3.5). Gluten's tests check the values against the rows in each file on Spark 3.5 to 4.1. Most of Delta's data-skipping tests pass (seen on 4.1). 22 data-skipping tests that check statistics still fail. They expect specific rows in each file after `repartition(2)`. |
+| Data file paths | Come from Delta's committer, so names and random prefixes follow Delta. File names seen on 4.1, random prefixes from code. |
+
+## SQL
+
+| Statement | Status | Notes |
+|---|---|---|
+| `SELECT` | Offload | See [Reading](#reading). |
+| `table_changes()` | Offload | Ranges with deletion vectors fall back. See [Change data feed](#change-data-feed). |
+| `CREATE TABLE ... AS SELECT`, `REPLACE TABLE ... AS SELECT` | Offload | With native write on, Spark 3.5 to 4.1, and a DeltaCatalog target. Seen on 4.1. See [Creating a table](#creating-a-table). |
+| `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `OPTIMIZE`, `REORG` | | See [Write commands](#write-commands). |
+| `CREATE TABLE` without `AS SELECT`, `DROP TABLE`, `DESCRIBE` | Fallback | They read and write no data files. From code. |
+| `ALTER TABLE` | Fallback, PartialOffload when it reads data | Most forms read and write no data files. `ADD CONSTRAINT` and `ALTER COLUMN ... SYNC IDENTITY` scan the table, and that scan can run in Velox. `DROP FEATURE` and `SET TBLPROPERTIES ('delta.columnMapping.mode' = 'none')` can rewrite data files with Delta's writer. From code. |
+| `VACUUM`, `RESTORE`, `SHALLOW CLONE` | | They write no Parquet data files. From code. |
+| `CONVERT TO DELTA` | PartialOffload | Writes no Parquet data files. Unless the statement has `NO STATISTICS` or `spark.databricks.delta.stats.collect=false` is set, it scans the Parquet files for statistics. That scan can run in Velox. From code. |
+
+## Schema evolution
+PartialOffload
+
+The `mergeSchema` and `overwriteSchema` rows need native write on Spark 3.5 to 4.1. Otherwise Delta writes the files.
+
+| Change | Status | Notes |
+|---|---|---|
+| `mergeSchema` with `save(path)` | Offload | Velox writes the evolved schema. |
+| `overwriteSchema` with `save(path)` | Offload | Delta applies the change before the write. |
+| Schema evolution in SQL `INSERT` | PartialOffload | From code. |
+| Schema evolution in `MERGE` | PartialOffload | Results match Delta Lake except for some STRUCT map key changes. See [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems). The two TIMESTAMP_NTZ widening tests fail before the MERGE runs, in a setup write of TIMESTAMP_NTZ values. Seen on 4.1. |
+| `RENAME COLUMN` with column mapping | Offload | Reads after top-level and nested renames return correct results. After a top-level rename, partition pruning and file skipping still work. |
+| `DROP COLUMN` with column mapping | Offload | Seen on 4.1. Plan not checked. |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Seen on 4.1. See [Table features](#table-features). |
+| `spark.sql.parquet.mergeSchema=true` | Fallback | Every Parquet scan falls back, Delta scans included. From code. |
+
+## Known problems
+
+### Read problems
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A column default stored as `EXISTS_DEFAULT` metadata, for example after `CONVERT TO DELTA`, reads as NULL for rows in files without that column. | ResultMismatch | Seen on 4.1. #11494 reports the same bug for plain Parquet on 3.4 to 4.1. | [#11494](https://github.com/apache/gluten/issues/11494) |
+| In column mapping id mode, when field IDs no longer match the Parquet column names, Gluten returns the column with the matching physical name. Delta Lake returns NULL or the column the ID points to. | ResultMismatch | 3.4 to 4.1, seen on 4.1 | [#12241](https://github.com/apache/gluten/issues/12241) |
+| On a TIMESTAMP-partitioned table with six partition values from .1 to .123456 seconds, `count(DISTINCT ts)` returns 3, as if the values were truncated to milliseconds. Equality filters on the column still return the right rows. | ResultMismatch | Seen on 4.1 | |
+| Reading a column widened from DATE to TIMESTAMP_NTZ fails with `Converted type DATE is not allowed for requested type TIMESTAMP`. | Exception | Seen on 4.1 | |
+| A query that selects only `_metadata.file_name` from a table with a BIGINT data column named `file_name` reads the data column and fails with `Converted type BIGINT is not allowed for requested type VARCHAR`. | Exception | Seen on 4.1 | |
+| A filter on `_metadata.base_row_id` fails with `Synthesized column 'base_row_id' failed filter validation`. | Exception | Seen on 4.1 | |
+| On a row-tracking table whose older files lack base row IDs, selecting `_metadata.row_id` returns rows. Delta raises an error. Delta Lake's test creates this state with a manual metadata commit. `ALTER TABLE ... SET TBLPROPERTIES` backfills base row IDs when it enables row tracking. | ResultMismatch | Seen on 4.1 | |
+| Reading or deleting from a 2-billion-row table with deletion vectors can grow native memory until the JVM is killed. Delta Lake's two 2-billion-row tests reproduce it. The nightly job force-fails both tests. | Exception | Reported on 4.1 | [#12387](https://github.com/apache/gluten/issues/12387) |
+| With `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default, selecting `__delta_internal_is_row_deleted` from a relation built by hand over `DeltaParquetFileFormat` drops the deleted rows and returns 1 for the rows it keeps. The column is not in the table schema, so SQL and DataFrame queries over the table cannot select it. | ResultMismatch | Seen on 4.1 | |
+| Scans of a table with many small files can be slower than vanilla Spark on EBS storage. In the report, two of five TPC-DS SF1000 queries take 2.2 and 3.4 times as long as on vanilla Spark, with 182,300 files in one table. A maintainer did not see the slowdown on a host with local NVMe storage. | | Reported on 3.4 | [#10778](https://github.com/apache/gluten/issues/10778) |
+
+### Write problems
+
+These need native write on, unless a row says otherwise.
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A write partitioned by a FLOAT or DOUBLE column fails with `Unsupported partition type`. | Exception | 3.5 to 4.1, seen on 4.1 | [#13138](https://github.com/apache/gluten/issues/13138) |
+| A write to a path scheme the native library cannot write fails at task time. This covers custom Hadoop file systems, and S3, GCS, HDFS or ABFS in a build without that connector. | Exception | 3.5 to 4.1, seen on 4.1 (custom file system), from code (missing connector) | [#4963](https://github.com/apache/gluten/issues/4963) |
+| Files written for a column mapping table, in name or id mode, have no Parquet field IDs, which the Delta Lake protocol requires. Delta Lake and Gluten read name-mode files by physical name. Readers that match by field ID cannot. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| Spark's own reader fails on natively written id-mode files with `Spark read schema expects field Ids, but Parquet file schema doesn't contain any field Ids`. Spark uses this reader when a scan falls back or Gluten is off. With `spark.sql.parquet.fieldId.read.ignoreMissing=true`, Spark skips this check and returns NULL for the data columns of these files (from code). That setting is not tested with these files. | Exception | 3.5 to 4.1, seen on 4.1 | |
+| Files written for a UniForm table have no Parquet field IDs, so the Iceberg metadata points at files without IDs. No error is raised. | ResultMismatch | 3.5 to 4.1, from code | [#12957](https://github.com/apache/gluten/pull/12957) |
+| A write to a partitioned UniForm table fails with `The file schema type should be equal with the input rowvector type`. | Exception | 3.5 to 4.1, from code | |
+| Writes to a table with the `materializePartitionColumns` feature leave the partition columns out of the data files. | ResultMismatch | 4.1, from code | |
+| Some writes of a TIMESTAMP_NTZ data column fail with `The file schema type should be equal with the input rowvector type`. Delta Lake's type widening tests fail this way when they write `CAST(value AS TIMESTAMP_NTZ)`. Delta Lake's data skipping tests also write that cast and pass. On Spark 4.x, `spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation=true` sends TIMESTAMP_NTZ writes to Delta's writer. It is not tested as a workaround. | Exception | Seen on 4.1 | |
+| A partitioned write fails in Velox's `PartitionIdGenerator` when a TIMESTAMP or TIMESTAMP_NTZ partition value has sub-millisecond digits. | Exception | 3.5 to 4.1, seen on 4.1 (TIMESTAMP_NTZ), from code (TIMESTAMP) | |
+| A generated partition column such as `DATE_FORMAT(ts, 'yyyy-MM')` stores `12345-07` for the year 12345. Spark stores `+12345-07`. Not checked with native write off. | ResultMismatch | Seen on 4.1 | |
+| The `compression` option set to brotli or lzo fails the write. | Exception | 3.5 to 4.1, from code | |
+| The `compression` option set to lz4_raw writes Snappy data into a `.lz4raw.parquet` file. | ResultMismatch | 3.5 to 4.1, from code | |
+| `maxRecordsPerFile` on an unpartitioned write counts rows but writes whole batches. When the limit is smaller than a batch, the whole batch goes into one file and the files rotated for that batch are empty. | ResultMismatch | 3.5 to 4.1, from code | |
+| `spark.sql.maxConcurrentOutputFileWriters` above 0 makes a partitioned write with unsorted input fail with `Underlying columnar data is inaccessible from BatchCarrierRow`. | Exception | 3.5 to 4.1, from code | |
+| An unpartitioned native write whose first task gets no rows, for example a write of an empty DataFrame, leaves a 0-byte file that is not valid Parquet. Reading the table directory as plain Parquet can then fail with `is not a Parquet file (length is too low: 0)`. With statistics on, the file stays on disk until `VACUUM`. With `spark.databricks.delta.stats.collect=false`, Delta commits it. The last two sentences are from code. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| A statistics aggregation that Velox cannot offload fails the write with `ClassCastException`. The TIMESTAMP_NTZ case from the issue no longer fails on main. | Exception | 3.5 to 4.1, from code | [#12538](https://github.com/apache/gluten/issues/12538) |
+
+### MERGE, UPDATE and DELETE problems
+
+MERGE does not use native write, so the MERGE rows also apply with native write off (from code).
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| UPDATE or DELETE through a temp view that swaps column names, such as `SELECT value AS key, key AS value FROM t`, changes different rows than Delta Lake. In Delta Lake's test, a DELETE removes both rows where Delta Lake removes one. Not checked with native write off. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| MERGE operation metrics in `DESCRIBE HISTORY` are too high, and a MERGE that changes nothing records change data rows on a change data feed table. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | [#9003](https://github.com/apache/gluten/issues/9003) |
+| With ANSI mode on, `spark.gluten.sql.ansiFallback.enabled=false` and `spark.databricks.delta.updateAndMergeCastingFollowsAnsiEnabledFlag=true`, a MERGE that casts `ARRAY<BIGINT>` to `ARRAY<INT>` or `MAP<STRING,BIGINT>` to `MAP<STRING,INT>` with overflowing values completes without an error. This was seen in a `WHEN MATCHED THEN UPDATE` clause and in a `WHEN NOT MATCHED THEN INSERT` clause inside `foreachBatch`. The Delta Lake flag is internal and false by default. | ResultMismatch | Seen on 4.1 | |
+| MERGE schema evolution that adds or removes fields of a STRUCT map key can fail with `map key cannot be indeterminate`. | Exception | Seen on 4.1 | |
+
+### Error types
+
+These operations fail as in Delta Lake, with a different exception type or error class. A failed write commits nothing (from code). Seen on 4.1. Spark 3.5 and 4.0 share the code.
+
+| Failure | Delta Lake raises | Gluten raises |
+|---|---|---|
+| CHECK, NOT NULL or generated-column violation on a partitioned native write. The NOT NULL case is from code. | `DeltaInvariantViolationException` | `SparkException` (`TASK_WRITE_FAILED`) wrapping `GlutenException` |
+| `replaceWhere` violation on a partitioned native write | `DeltaAnalysisException` (`DELTA_REPLACE_WHERE_MISMATCH`) | `SparkException` (`TASK_WRITE_FAILED`) wrapping `GlutenException` |
+| Store-assignment cast overflow in a MERGE `WHEN MATCHED THEN UPDATE` clause | `DeltaArithmeticException` (`DELTA_CAST_OVERFLOW_IN_TABLE_WRITE`) | `GlutenException` |
+| ANSI cast overflow evaluated in Velox | `SparkArithmeticException` | `GlutenException` (`VeloxUserError`) |
+| Missing data file | `SparkException` (`FAILED_READ_FILE.FILE_NOT_EXIST`) on Spark 4.x | `SparkException` wrapping `GlutenException` (`FILE_NOT_FOUND`) |
+
+For errors raised on the JVM, the Delta Lake message stays in the `GlutenException` message. On unpartitioned `save(path)` writes, CHECK and NOT NULL violations keep Delta Lake's exception types. Tracked in [#11912](https://github.com/apache/gluten/issues/11912).
+
+## Configuration
+### Gluten Options
+
+| Gluten option | Default | Description |
+| --- | --- | --- |
+| spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite | false | Experimental. Velox writes the data files of the commands marked Offload in [Write commands](#write-commands). Spark 3.5 to 4.1. No effect on Spark 3.4. |
+| spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan | true | Experimental. When false, the DML target scan that reads row indexes for deletion vectors stays on Spark. Other scans still offload. |
+| spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan | true | Experimental. When false, batch change data feed reads use Delta Lake's change data feed scan. |
+| spark.gluten.sql.columnar.filescan | true | When false, every V1 file source scan (`FileSourceScanExec`) falls back, Delta scans included. No Gluten option turns off Delta scans alone. |
+| spark.gluten.sql.ansiFallback.enabled | true | With ANSI mode on, every query falls back. See [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x). |
+| spark.gluten.sql.columnar.parquet.write.blockSize | 128MB | Row-group size for native Delta writes. |
+| spark.gluten.sql.native.parquet.write.blockRows | 100000000 | Row-group row limit for native Delta writes. |
+| spark.gluten.sql.native.writer.enabled | (unset) | Controls native writes of non-Delta Parquet files. Unset means on. No effect on Delta writes. |
+| spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation | false | When true, every operator whose input or output includes TIMESTAMP_NTZ falls back. On Spark 4.x, native Delta writes of TIMESTAMP_NTZ schemas then use Delta's writer. From code. |
+| spark.gluten.sql.columnar.backend.velox.scanMapKeyPruningEnabled | false | When true, Velox skips map keys a query does not look up, if the lookups sit in filters and projections directly above the scan. Applies to Delta scans without column mapping, with or without deletion vectors. Off for column-mapped tables. |
+| spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames | true | Velox matches Parquet columns by name. Id-mode column mapping tables are read by physical name. When false, id-mode tables are read by position and name-mode tables still by physical name. From code. |
+| spark.gluten.sql.fallbackUnexpectedMetadataParquet | false | When true, Delta scans fall back if a sampled file is encrypted, carries legacy datetime rebase metadata, uses LZO or BROTLI, or has variant annotations. LZ4_RAW also counts on Spark 4.0 and 4.1. The variant check runs on Spark 4.1 only. The sample is the first `spark.gluten.sql.fallbackUnexpectedMetadataParquet.limit` files (default 10) under the table root. `_delta_log` files count, so the sample may contain no data files. The check is skipped when `spark.gluten.sql.scan.fileSchemeValidation.enabled=false`. From code. |
+| spark.gluten.sql.scan.fileSchemeValidation.enabled | true | Scans on a file scheme Velox cannot read fall back. |
+
+The three `spark.gluten.sql.columnar.backend.velox.delta.*` options are read per query, so `SET` changes them for the session.
+
+### Catalogs
+
+Gluten does not change DeltaCatalog. `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` are written natively only through a DeltaCatalog (from code). With native write on, `UPDATE`, `DELETE`, plain `OPTIMIZE` and `save(path)` are written natively with any catalog (from code).
+
+### SQL Extensions
+Fallback
+
+Supports `spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension`. Gluten does not offload the command nodes of Delta's own SQL commands, such as `VACUUM`, `RESTORE` and `DESCRIBE HISTORY`. Their `_delta_log` scans stay on Spark. From code. `OPTIMIZE` follows [Write commands](#write-commands). GlutenPlugin appends its own extension.
+
+### Runtime configuration
+The "Gluten Support" column uses these values:
+
+✅ Supported<br>
+❌ Not Supported<br>
+⚠️ Partial Support<br>
+🚫 Not applied or transparent to Gluten<br>
+
+A blank cell means unknown.
+
+### Spark SQL Options
+
+| Option | Default | Description | Gluten Support |
+| --- | --- | --- | --- |
+| spark.sql.extensions | (none) | Set to `io.delta.sql.DeltaSparkSessionExtension`. | ✅ |
+| spark.sql.catalog.spark_catalog | builtin on Spark 4.x, (unset) on 3.x | Set to `org.apache.spark.sql.delta.catalog.DeltaCatalog`. | ✅ |
+| spark.sql.ansi.enabled | true on Spark 4.x, false on 3.x | When true, every query falls back unless `spark.gluten.sql.ansiFallback.enabled=false`. | ⚠️ |
+| spark.sql.parquet.compression.codec | snappy | Codec for native writes when the `compression` option is unset. | ⚠️ |
+| spark.sql.files.maxRecordsPerFile | 0 | Exact on partitioned native writes. See [Write problems](#write-problems). | ⚠️ |
+| spark.sql.maxConcurrentOutputFileWriters | 0 | Above 0, partitioned native writes with unsorted input fail. From code. | ❌ |
+| spark.sql.files.ignoreMissingFiles | false | Passed to Velox. Native scans skip missing files when true. | ✅ |
+| spark.sql.parquet.mergeSchema | false | When true, Delta scans fall back. From code. | ❌ |
+| spark.sql.parquet.outputTimestampType | INT96 | Ignored by native writes. From code. See [Natively written files](#natively-written-files). | ❌ |
+| spark.sql.parquet.writeLegacyFormat | false | Native writes apply it to DECIMAL only. From code. See [Natively written files](#natively-written-files). | ⚠️ |
+| spark.sql.caseSensitive | false | When false, native writes lowercase nested struct field names. From code. | ⚠️ |
+| spark.sql.sources.partitionOverwriteMode | STATIC | With native write on, Velox writes the files of a dynamic overwrite through `save(path)`, and the results are correct (seen on 4.1). | ✅ |
+| spark.databricks.delta.deletionVectors.useMetadataRowIndex | true | When false, scans that read Delta's generated deletion vector columns fall back. See [Deletion vectors](#deletion-vectors). | ⚠️ |
+| spark.databricks.delta.stats.collect | true | Native writes compute file statistics in Velox. | ✅ |
+| spark.databricks.delta.stats.skipping | true | Delta skips files by statistics before the scan runs. The scan offloads either way, except in the deletion-vector case in [Deletion vectors](#deletion-vectors). | 🚫 |
+| spark.databricks.delta.optimizeWrite.enabled | (unset) | Native writes run the optimized write shuffle in Velox (from code). File counts match Delta Lake (seen on 4.1). | ✅ |
+| spark.databricks.delta.optimizeWrite.binSize | 512 (MiB) | Used by the native optimized write shuffle. From code. | ✅ |
+| spark.databricks.delta.autoCompact.enabled | (unset) | Compacts natively written tables. Seen on 4.1. | ✅ |
+| spark.databricks.delta.schema.autoMerge.enabled | false | Delta applies schema changes before the write. | 🚫 |
+| spark.databricks.delta.dynamicPartitionOverwrite.enabled | true | Delta decides which partitions to replace. | ✅ |
+| spark.databricks.delta.merge.repartitionBeforeWrite.enabled | true | MERGE uses Delta's writer, so this only shapes Delta's write. | 🚫 |
+| spark.databricks.delta.optimize.maxFileSize | 1073741824 (1 GiB) | Delta plans the bins. With native write on, Velox writes the files of plain `OPTIMIZE`. | ✅ |
+| spark.databricks.delta.optimize.minFileSize | 1073741824 (1 GiB) | Delta plans the bins. With native write on, Velox writes the files of plain `OPTIMIZE`. | ✅ |
+| spark.databricks.delta.write.txnAppId, spark.databricks.delta.write.txnVersion | (unset) | Idempotent writes. Commit only. | 🚫 |
+| spark.databricks.delta.commitInfo.userMetadata | (unset) | Commit only. | 🚫 |
+| spark.databricks.delta.properties.defaults.* | (unset) | Default for the matching `delta.*` table property on new tables. | 🚫 |
+
+`spark.databricks.delta.deletionVectors.useMetadataRowIndex`, `spark.databricks.delta.optimizeWrite.enabled`, `spark.databricks.delta.optimizeWrite.binSize` and `spark.databricks.delta.autoCompact.enabled` do not exist in Delta Lake 2.4.0.
+
+#### Read options
+
+| Option | Default | Description | Gluten Support |
+| --- | --- | --- | --- |
+| versionAsOf | (latest) | Version of the table snapshot to read. Seen on 4.1. | ✅ |
+| timestampAsOf | (latest) | Read the snapshot current at this timestamp. Seen on 4.1. | ✅ |
+| readChangeFeed | false | Read the change data feed. Ranges with deletion vectors fall back. | ⚠️ |
+| startingVersion, endingVersion | (none) | Change data feed version range. `startingVersion` also applies to streaming reads. | ✅ |
+| startingTimestamp, endingTimestamp | (none) | Change data feed timestamp range. `startingTimestamp` also applies to streaming reads. Seen on 4.1. | ✅ |
+| maxFilesPerTrigger | 1000 | Streaming: files per micro-batch | 🚫 |
+| maxBytesPerTrigger | (unset) | Streaming: bytes per micro-batch | 🚫 |
+| ignoreDeletes, ignoreChanges, skipChangeCommits | false | Streaming: how to treat commits that remove data | 🚫 |
+| excludeRegex | (none) | Streaming: skip files whose path matches | 🚫 |
+| failOnDataLoss | true | Streaming: fail when Delta log files the stream needs are missing | 🚫 |
+| allowSourceColumnDrop, allowSourceColumnRename, allowSourceColumnTypeChange | (unset) | Streaming schema tracking. Delta Lake 4.0.1 and 4.1.0. | 🚫 |
+
+#### Write options
+
+| Option | Default | Description | Gluten Support |
+| --- | --- | --- | --- |
+| compression | spark.sql.parquet.compression.codec | Native writes support the codecs in [Natively written files](#natively-written-files). | ⚠️ |
+| maxRecordsPerFile | spark.sql.files.maxRecordsPerFile | Exact on partitioned native writes. See [Write problems](#write-problems). | ⚠️ |
+| partitionOverwriteMode | spark.sql.sources.partitionOverwriteMode | `dynamic` works through `save(path)`. `INSERT OVERWRITE` and `insertInto` use Delta's writer (from code). | ✅ |
+| replaceWhere | (none) | A violating partitioned native write fails and commits nothing, with a different error type (see [Error types](#error-types)). | ⚠️ |
+| mergeSchema | spark.databricks.delta.schema.autoMerge.enabled | Delta applies the schema change before the write. | 🚫 |
+| overwriteSchema | false | Delta applies the schema change before the write. | 🚫 |
+| optimizeWrite | (unset) | Runs the rebalancing shuffle in Velox before the native write. From code. | ✅ |
+| txnAppId, txnVersion | (unset) | Idempotent writes. Commit only. | 🚫 |
+| userMetadata | (unset) | Stored in `commitInfo.userMetadata`. Commit only. | 🚫 |
+| dataChange | true | Sets `dataChange` on the file actions. Commit only. | 🚫 |
+| parquet.* (writer option or Hadoop setting) | | Ignored by native writes (from code). Use `spark.gluten.sql.columnar.parquet.write.blockSize` for row groups. | ❌ |
+| fs.* Hadoop file system options | | A custom file system scheme fails on native writes (seen on 4.1). Per-write options for supported stores, such as `fs.s3a.access.key`, are not tested with native writes. | ❌ |
Review Comment:
Split into two rows like you suggested. On the per-write options row I kept ❌ instead of leaving it blank, since from the code those options don't reach the native writer, so it's a known gap rather than just untested.
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
Review Comment:
Done, all the examples now use regular fences with a language (sql, bash, scala, properties).
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
Review Comment:
Renamed it to "Supported Spark and Delta Lake versions".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4312322099
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
Review Comment:
The doc uses 4-backtick fences without language identifiers for examples (SQL, shell). To improve readability and consistent rendering (including syntax highlighting) across the docs site, prefer standard fenced blocks with explicit languages (e.g., `sql`, `bash`, `scala`) unless there’s a concrete need for 4-backtick fencing.
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
Review Comment:
The section title says “Supported Spark version”, but the page immediately presents a multi-version matrix (and includes Delta Lake versions/jars). Consider renaming this heading to “Supported Spark versions” or “Supported Spark / Delta versions” to match the content and reduce reader confusion.
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back |
+| Fallback | Runs on Spark or Delta Lake |
+| Exception | Fails instead of falling back |
+| ResultMismatch | Can return wrong results or write wrong files |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+````
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+````
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+````
+
Review Comment:
Minor punctuation/grammar: add a trailing period to keep the table descriptions consistent with the other rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4312917007
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back |
+| Fallback | Runs on Spark or Delta Lake |
+| Exception | Fails instead of falling back |
+| ResultMismatch | Can return wrong results or write wrong files |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
Review Comment:
For `INSERT INTO` the setting makes no difference. With native write off, Delta's writer writes the files. With native write on it's the same, because INSERT plans as `AppendDataExecV1` / `OverwriteByExpressionExecV1` (or `DeltaDynamicPartitionOverwriteCommand`), and Gluten doesn't wrap those for native write yet. In both cases the SELECT part can still run in Velox. To get native writes for an append today, use `df.write.format("delta").mode("append").save(path)`.
I added these details to this section now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "ReemaAlzaid (via GitHub)" <gi...@apache.org>.
ReemaAlzaid commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4299929126
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
Review Comment:
Also we can add a diagram example to show them the process of delta, by showing Delta snapshot/file selection → Gluten planning → Velox and Spark execution
For example, consider a Delta table partitioned by `sale_date`:
```sql
SELECT SUM(amount)
FROM sales
WHERE sale_date = DATE '2026-10-01'
AND amount > 100;
```
```mermaid
flowchart LR
subgraph Planning["Spark driver"]
Snapshot["1. Spark + Delta snapshot<br/>Schema + active files"]
Select["2. Delta file pruning<br/>Partitions + available stats"]
Plan["3. Spark plan<br/>Gluten offload checks"]
Snapshot --> Select --> Plan
end
subgraph Execution["Spark executors"]
Scan["4. Velox scan<br/>Read selected files"]
Compute["5. Velox operators<br/>Filter + SUM"]
Spark["Spark operators<br/>Retained during planning"]
Scan --> Compute
Compute <-->|"Planned conversions"| Spark
end
subgraph Storage["Delta table"]
Log[("_delta_log<br/>Metadata + file statistics")]
Data[("Parquet files")]
end
Result["6. Query result"]
Log --> Snapshot
Plan -->|"Eligible operators"| Scan
Plan -->|"Retained operators"| Spark
Data --> Scan
Data --> Spark
Compute --> Result
Spark --> Result
```
A query can combine native and Spark operators. Delta responsibility for table metadata and transactions
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back |
+| Fallback | Runs on Spark or Delta Lake |
+| Exception | Fails instead of falling back |
+| ResultMismatch | Can return wrong results or write wrong files |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+````
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+````
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+````
+
+Set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+````
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+````
+
+The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the plan from the Spark UI SQL tab or a `QueryExecutionListener`.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. |
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | | Offload | Bin-packing, by path or by name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans and the DML target scan can run in Velox. Joins can too (from code). Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake. Seen on 4.1.
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload | Compacts natively written tables (seen on 4.1). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name (from code) | Offload, matched by physical name (seen on 4.1) |
Review Comment:
```suggestion
| Column mapping, id mode | Offload, matched by physical name (from code). Known ResultMismatch cases: see [Read problems](#read-problems). | Offload, matched by physical name (seen on 4.1). Known ResultMismatch cases: see [Read problems](#read-problems). |
```
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
Review Comment:
Can we add a brief overview before the version matrix? It would help readers understand that read acceleration and experimental native writing are separate capabilities
Suggested wording:
> Gluten’s Velox backend can accelerate eligible scans and query operators over Delta Lake tables. Read acceleration does not require enabling native writes. Native Delta writes are experimental and disabled by default. This guide covers Spark/Delta Lake version combinations, setup, feature support, and known limitations.
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back |
+| Fallback | Runs on Spark or Delta Lake |
+| Exception | Fails instead of falling back |
+| ResultMismatch | Can return wrong results or write wrong files |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+````
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+````
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+````
+
+Set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+````
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+````
+
+The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the plan from the Spark UI SQL tab or a `QueryExecutionListener`.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. |
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | | Offload | Bin-packing, by path or by name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans and the DML target scan can run in Velox. Joins can too (from code). Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake. Seen on 4.1.
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload | Compacts natively written tables (seen on 4.1). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name (from code) | Offload, matched by physical name (seen on 4.1) |
+| Files with deletion vectors | Fallback | Offload |
+| Change data feed, no deletion vectors in range | Offload. Scans of `_change_data` files fall back when the table allows deletion vectors (from code). | Offload |
+| Change data feed, deletion vectors in range | Fallback | Fallback |
+
+````
+SELECT count(1) AS count, name
+FROM t
+GROUP BY name;
+````
+
+Reads need no Gluten option. In `EXPLAIN` output, the native scan shows up as `FileDeltaScanTransformer`. `EXPLAIN FORMATTED` and the Spark UI plan graph show `DeltaScanTransformer`. Delta prunes partitions and skips files by statistics while it plans the query, before the scan runs.
+
+| Read | Status | Notes |
+|---|---|---|
+| By table name, ``delta.`/path` `` or `spark.read.format("delta").load(path)` | Offload | Same native scan. |
+| `VERSION AS OF` | Offload | Tested on Spark 3.4 to 4.1. |
+| `TIMESTAMP AS OF`, `versionAsOf`, `timestampAsOf` | Offload | From code. Delta Lake's `DeltaTimeTravelSuite` passes (seen on 4.1). |
+| Streaming source | Offload | Each micro-batch is a Delta file scan. From code. Delta Lake's `DeltaSourceSuite` passes (seen on 4.1). |
+| Dynamic partition pruning | Not tested | |
+
+### Deletion vectors
+Offload
+
+````
+ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
+DELETE FROM t WHERE id IN (3, 4);
+SELECT * FROM t;
+````
+
+On Spark 3.5 to 4.1, Velox drops the deleted rows in the scan. This needs Delta's `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default. The DML target scan, which finds the rows for new deletion vectors, also runs in Velox. Gluten tests check the scan on Spark 3.5 to 4.1 and the DML target scan on Spark 3.5 and 4.0. Delta writes the deletion-vector files.
+
+Fallback
+
+````
+SET spark.databricks.delta.deletionVectors.useMetadataRowIndex=false;
+SELECT * FROM t;
+````
+
+| Case | Behavior |
+|---|---|
+| Spark 3.5 to 4.1 with `useMetadataRowIndex=false` | A scan that reads Delta's generated `__delta_internal_is_row_deleted` or `__delta_internal_row_index` column falls back with `fallback Delta scan requiring generated DV metadata`. Delta adds the first column when any file in the scanned table version, micro-batch or DML file set has a deletion vector. It adds the second to DML target scans that write deletion vectors. Other scans offload. Tested on Spark 3.5 and 4.0. |
+| Spark 3.4, normal reads | A read falls back with `fallback Spark 3.4 Delta DV scan` when a file it selected has a deletion vector. Tested on Spark 3.4. A read whose selected files have none offloads (from code). |
+| Spark 3.4, DML target scans, streaming micro-batches and change data feed scans of `_change_data` files | Fall back whenever the table protocol allows deletion vectors. From code. |
+| `spark.databricks.delta.stats.skipping=false` | Delta Lake itself rejects reads of a table with deletion vectors with `Cannot work with a non-pinned table snapshot of the TahoeFileIndex` (seen on 4.1). Before Delta Lake 4.1.0, the table feature alone triggers it (from code). |
+
+The fallback reasons show in [Gluten UI](GlutenUI.md).
+
+### Change data feed
+Offload
+
+````
+CREATE TABLE c (id INT, name STRING) USING delta
+ TBLPROPERTIES ('delta.enableChangeDataFeed' = true);
+INSERT INTO c VALUES (1, 'a'), (2, 'b');
+UPDATE c SET name = 'bb' WHERE id = 2;
+SELECT id, name, _change_type, _commit_version
+FROM table_changes('c', 1)
+WHERE id = 2;
+````
+
+Gluten expands a batch change data feed read into Delta file scans that run in Velox with the filters pushed down. Delta also adds an empty placeholder scan to the plan. It falls back when the table has a MAP column (from code). If the expansion fails, Gluten logs a warning and the read uses Delta's change data feed scan on Spark. From code.
+
+Fallback
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false;
+SELECT * FROM table_changes('c', 1);
+````
+
+A read whose version range adds or removes a file with a deletion vector stays on Delta's change data feed scan on Spark. Insert-only ranges of a table with deletion vectors enabled still run in Velox. With `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false`, every batch change data feed read uses Delta's scan. Streaming change data feed reads ignore this option. They run as Delta file scans, and a scan whose files have deletion vectors falls back (from code).
+
+### Read metadata
+Offload
+
+````
+SELECT id, _metadata.file_path, _metadata.row_index
+FROM t
+WHERE _metadata.row_index = 2;
+````
+
+The native scan fills the `_metadata` file fields and `input_file_name()` (from code), and `_metadata.row_index`. Gluten tests check `_metadata.row_index` in the native scan on Spark 3.5 and 4.0, and Delta Lake tests that read it pass (seen on 4.1). Spark 3.4 is not tested. On row-tracking tables, `_metadata.row_id` and `_metadata.row_commit_version` are read natively (from code). They return correct values except in the cases in [Read problems](#read-problems) (seen on 4.1).
+
+Fallback
+
+````
+DESCRIBE HISTORY t;
+````
+
+Delta reads the transaction log on Spark. Every scan under `_delta_log`, checkpoints included, stays on Spark. From code. The reason is `fallback Delta _delta_log scan`.
+
+## Table features
+Offload/Fallback
+
+The Read column covers Spark 3.5 to 4.1. It covers Spark 3.4 only where the row names it. The Native write column applies to Spark 3.5 to 4.1 with native write on.
+
+| Feature | Read | Native write | Notes |
+|---|---|---|---|
+| Deletion vectors | Offload, Fallback on Spark 3.4 | PartialOffload | See [Deletion vectors](#deletion-vectors). |
+| Change data feed | Offload, also on Spark 3.4. Ranges with deletion vectors fall back. On Spark 3.4, scans of `_change_data` files also fall back when the table allows deletion vectors (from code). | Offload | UPDATE and DELETE write the `_change_data` files with Velox (from code). Change data feed reads of their changes are correct (seen on 4.1). See [Change data feed](#change-data-feed). |
+| Column mapping, name mode | Offload, also on Spark 3.4 | ResultMismatch | Files have no Parquet field IDs. Seen on 4.1. See [Write problems](#write-problems). |
+| Column mapping, id mode | Offload, also on Spark 3.4. ResultMismatch when field IDs differ from physical names (seen on 4.1). | ResultMismatch, Exception when Spark's reader reads the files (seen on 4.1) | Gluten matches columns by physical name. See [Known problems](#known-problems). |
+| Identity columns | Offload | Offload | Seen on 4.1. High-water marks are correct. The projection that generates identity values runs on Spark (from code). |
+| Generated columns | Offload | Offload | Seen on 4.1. No Gluten test covers generated columns. Velox computes the generated values when it supports the expression (from code). Partition filters derived from them reach the native scan. `DATE_FORMAT` results for years above 9999 differ (see [Write problems](#write-problems)). |
+| CHECK constraints, NOT NULL | Offload | Offload | Delta checks the rows on the JVM before Velox writes them. Gluten's tests cover NOT NULL on Spark 3.5 and 4.0. Seen on 4.1. |
+| Row tracking | Offload, Exception for `_metadata.base_row_id` filters | Offload | Seen on 4.1. |
+| Liquid clustering | Offload | PartialOffload | Native commands write to clustered tables. `OPTIMIZE`, which clusters the data, uses Delta's writer. Both are from code. |
+| UniForm (IcebergCompatV1, V2) | | ResultMismatch, Exception when partitioned | The write status is from code. No test reads a UniForm table. |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Offload, Exception for FLOAT or DOUBLE partition columns | Seen on 4.1. Spark 3.5 and 4.0 are not covered. |
+| TIMESTAMP_NTZ | PartialOffload, also on Spark 3.4 | Offload, Exception for some writes (seen on 4.1) | See [DataType](#datatype). |
+| Variant, including shredding | Fallback | Fallback | Spark 4.x. From code. See [DataType](#datatype). |
+| Collations | Fallback | | Spark 4.x. From code. See [DataType](#datatype). |
+| materializePartitionColumns | | ResultMismatch | Spark 4.1. From code. |
+
+## DataType
+
+The native write columns apply to Spark 3.5 to 4.1 with native write on. On Spark 3.5 and 4.0, Gluten's tests write BOOLEAN, INT, BIGINT, STRING, STRUCT and ARRAY columns natively. The other native write statuses are seen on 4.1, unless a row says they are from code.
+
+| Type | Read | Native write | Native write as partition column | Notes |
|---|---|---|---|---|
-| `spark-3.5` | Spark 3.5.x | 2.12 | 3.3.x | Supported |
-| `spark-4.0` | Spark 4.0.x | 2.13 | 4.0.x | Supported |
-
-Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The difference between
-the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta 3.3 vs Spark 4.0 +
-Delta 4.0), not a native-write capability gap.
-
-## Build and runtime notes
-
-Build Gluten with Delta support by enabling `-Pdelta` together with the Velox backend profile and a Spark profile.
-
-- Spark 3.5 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests`
-- Spark 4.0 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13 -Pjava-17 -DskipTests`
-
-Native Delta write is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite`
- - Default: `false`
- - Type: experimental
-
-Native change data feed scan offload is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan`
- - Default: `true`
- - Type: experimental
-
-The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan`
- - Default: `true`
- - Type: experimental
- - When disabled, the DML target scan that produces file paths and row indexes for
- deletion-vector writes stays on Spark; other scans are unaffected.
-
-| Feature | Delta minWriterVersion | Delta minReaderVersion | Iceberg format-version | Feature type | Supported by Gluten (Velox) |
-|---|---:|---:|---:|---|---|
-| Basic functionality | 2 | 1 | 1 | Writer | Yes |
-| CHECK constraints | 3 | 1 | N/A | Writer | No |
-| Change data feed | 4 | 1 | N/A | Writer | Yes |
-| Generated columns | 4 | 1 | N/A | Writer | Partial |
-| Column mapping | 5 | 2 | N/A | Reader and writer | Yes |
-| Identity columns | 6 | 1 | N/A | Writer | Yes |
-| Row tracking | 7 | 1 | 3 | Writer | Partial |
-| Deletion vectors | 7 | 3 | 3 | Reader and writer | Partial |
-| TimestampNTZ | 7 | 3 | 1 | Reader and writer | No |
-| Liquid clustering | 7 | 3 | 1 | Reader and writer | Yes |
-| Iceberg readers (UniForm) | 7 | 2 | N/A | Writer | Not tested |
-| Type widening | 7 | 3 | N/A | Reader and writer | Partial |
-| Variant | 7 | 3 | 3 | Reader and writer | Not tested |
-| Variant shredding | 7 | 3 | 3 | Reader and writer | Not tested |
-| Collations | 7 | 3 | N/A | Reader and writer | Not tested |
-| Protected checkpoints | 7 | 1 | N/A | Writer | Not tested |
+| BOOLEAN | Offload | Offload | Offload | |
+| TINYINT, SMALLINT, INT, BIGINT | Offload | Offload | Offload | |
+| FLOAT, DOUBLE | Offload | Offload | Exception | See [Write problems](#write-problems). |
+| DECIMAL | Offload | Offload | Offload | Precision above 18 included. |
+| STRING | Offload | Offload | Offload | |
+| CHAR, VARCHAR | Offload | Offload | | Stored as STRING. Delta checks the length on the JVM before the native write. Both statuses are from code. Reads return correct results (seen on 4.1). |
+| BINARY | Not tested | Not tested | Not tested | Only indirect coverage. |
+| DATE | Offload | Offload | Offload | |
+| TIMESTAMP | Offload, ResultMismatch for sub-millisecond partition values (seen on 4.1) | Offload | Offload, Exception for sub-millisecond values | The partition column status is from code. See [Known problems](#known-problems). |
+| TIMESTAMP_NTZ | PartialOffload | Offload, Exception for some writes (seen on 4.1) | Offload, Exception for sub-millisecond values (seen on 4.1) | Filters, sorts, joins and other operators whose input or output includes a TIMESTAMP_NTZ column fall back, even when they do not use that column. Scans, aggregates, shuffles and some projections can run in Velox. These operator rules are from code. |
+| STRUCT, ARRAY, MAP | Offload | Offload | | |
+| VARIANT | Fallback | Fallback | | Spark 4.x. Shredded variant reads fall back too. Both statuses are from code. |
+| Collated STRING | Fallback | | | Spark 4.x. Scans that read the column fall back. Delta Lake 4.0.1 and 4.1.0 reject a collated column in a table schema with `DELTA_UNSUPPORTED_DATA_TYPES`. They cannot write to tables that have the `collations` feature. From code. |
+
+A plan node falls back when its output contains a type Velox does not support, such as VARIANT. With the default `spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames=true`, a scan that does not select such a column can still offload. From code. Delta Lake does not allow interval columns.
+
+## Format
+PartialOffload
+
+Delta data files are Parquet, and Gluten reads them with Velox's Parquet reader. Velox writes them only for native write commands on Spark 3.5 to 4.1. Otherwise Delta's Parquet writer writes them.
+
+### Natively written files
+
+| Property | Behavior |
+|---|---|
+| Compression | snappy (default), gzip, zstd (always level 3), lz4 (Hadoop framing), none. From code. Other codecs: see [Write problems](#write-problems). |
+| Row groups | Sized by `spark.gluten.sql.columnar.parquet.write.blockSize` and `spark.gluten.sql.native.parquet.write.blockRows`. `parquet.*` settings are ignored. From code. |
+| TIMESTAMP | INT64 microseconds. `spark.sql.parquet.outputTimestampType` is ignored. Delta Lake's writer uses INT96 by default. Spark reads both. From code. |
+| Parquet field IDs | Not written, for column-mapped and UniForm tables too. Seen on 4.1 for column mapping, from code for UniForm. See [Write problems](#write-problems). |
+| Spark schema footer entry | `org.apache.spark.sql.parquet.row.metadata` is not written ([#13288](https://github.com/apache/gluten/issues/13288)). Seen on 4.1. |
+| Nested field names | Lowercased below the top level when `spark.sql.caseSensitive=false`. Spark and Gluten match them case-insensitively. A case-sensitive reader can miss them. From code. |
+| Page index | Always written. From code. |
+| `spark.sql.parquet.writeLegacyFormat=true` | Applies to DECIMAL only. Arrays and maps keep the standard layout. From code. |
+| File statistics | numRecords, min, max and nullCount are computed in Velox (checked on Spark 3.5). Gluten's tests check the values against the rows in each file on Spark 3.5 to 4.1. Most of Delta's data-skipping tests pass (seen on 4.1). 22 data-skipping tests that check statistics still fail. They expect specific rows in each file after `repartition(2)`. |
+| Data file paths | Come from Delta's committer, so names and random prefixes follow Delta. File names seen on 4.1, random prefixes from code. |
+
+## SQL
+
+| Statement | Status | Notes |
+|---|---|---|
+| `SELECT` | Offload | See [Reading](#reading). |
+| `table_changes()` | Offload | Ranges with deletion vectors fall back. See [Change data feed](#change-data-feed). |
+| `CREATE TABLE ... AS SELECT`, `REPLACE TABLE ... AS SELECT` | Offload | With native write on, Spark 3.5 to 4.1, and a DeltaCatalog target. Seen on 4.1. See [Creating a table](#creating-a-table). |
+| `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `OPTIMIZE`, `REORG` | | See [Write commands](#write-commands). |
+| `CREATE TABLE` without `AS SELECT`, `DROP TABLE`, `DESCRIBE` | Fallback | They read and write no data files. From code. |
+| `ALTER TABLE` | Fallback, PartialOffload when it reads data | Most forms read and write no data files. `ADD CONSTRAINT` and `ALTER COLUMN ... SYNC IDENTITY` scan the table, and that scan can run in Velox. `DROP FEATURE` and `SET TBLPROPERTIES ('delta.columnMapping.mode' = 'none')` can rewrite data files with Delta's writer. From code. |
+| `VACUUM`, `RESTORE`, `SHALLOW CLONE` | | They write no Parquet data files. From code. |
+| `CONVERT TO DELTA` | PartialOffload | Writes no Parquet data files. Unless the statement has `NO STATISTICS` or `spark.databricks.delta.stats.collect=false` is set, it scans the Parquet files for statistics. That scan can run in Velox. From code. |
+
+## Schema evolution
+PartialOffload
+
+The `mergeSchema` and `overwriteSchema` rows need native write on Spark 3.5 to 4.1. Otherwise Delta writes the files.
+
+| Change | Status | Notes |
+|---|---|---|
+| `mergeSchema` with `save(path)` | Offload | Velox writes the evolved schema. |
+| `overwriteSchema` with `save(path)` | Offload | Delta applies the change before the write. |
+| Schema evolution in SQL `INSERT` | PartialOffload | From code. |
+| Schema evolution in `MERGE` | PartialOffload | Results match Delta Lake except for some STRUCT map key changes. See [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems). The two TIMESTAMP_NTZ widening tests fail before the MERGE runs, in a setup write of TIMESTAMP_NTZ values. Seen on 4.1. |
+| `RENAME COLUMN` with column mapping | Offload | Reads after top-level and nested renames return correct results. After a top-level rename, partition pruning and file skipping still work. |
+| `DROP COLUMN` with column mapping | Offload | Seen on 4.1. Plan not checked. |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Seen on 4.1. See [Table features](#table-features). |
+| `spark.sql.parquet.mergeSchema=true` | Fallback | Every Parquet scan falls back, Delta scans included. From code. |
+
+## Known problems
+
+### Read problems
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A column default stored as `EXISTS_DEFAULT` metadata, for example after `CONVERT TO DELTA`, reads as NULL for rows in files without that column. | ResultMismatch | Seen on 4.1. #11494 reports the same bug for plain Parquet on 3.4 to 4.1. | [#11494](https://github.com/apache/gluten/issues/11494) |
+| In column mapping id mode, when field IDs no longer match the Parquet column names, Gluten returns the column with the matching physical name. Delta Lake returns NULL or the column the ID points to. | ResultMismatch | 3.4 to 4.1, seen on 4.1 | [#12241](https://github.com/apache/gluten/issues/12241) |
+| On a TIMESTAMP-partitioned table with six partition values from .1 to .123456 seconds, `count(DISTINCT ts)` returns 3, as if the values were truncated to milliseconds. Equality filters on the column still return the right rows. | ResultMismatch | Seen on 4.1 | |
+| Reading a column widened from DATE to TIMESTAMP_NTZ fails with `Converted type DATE is not allowed for requested type TIMESTAMP`. | Exception | Seen on 4.1 | |
+| A query that selects only `_metadata.file_name` from a table with a BIGINT data column named `file_name` reads the data column and fails with `Converted type BIGINT is not allowed for requested type VARCHAR`. | Exception | Seen on 4.1 | |
+| A filter on `_metadata.base_row_id` fails with `Synthesized column 'base_row_id' failed filter validation`. | Exception | Seen on 4.1 | |
+| On a row-tracking table whose older files lack base row IDs, selecting `_metadata.row_id` returns rows. Delta raises an error. Delta Lake's test creates this state with a manual metadata commit. `ALTER TABLE ... SET TBLPROPERTIES` backfills base row IDs when it enables row tracking. | ResultMismatch | Seen on 4.1 | |
+| Reading or deleting from a 2-billion-row table with deletion vectors can grow native memory until the JVM is killed. Delta Lake's two 2-billion-row tests reproduce it. The nightly job force-fails both tests. | Exception | Reported on 4.1 | [#12387](https://github.com/apache/gluten/issues/12387) |
+| With `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default, selecting `__delta_internal_is_row_deleted` from a relation built by hand over `DeltaParquetFileFormat` drops the deleted rows and returns 1 for the rows it keeps. The column is not in the table schema, so SQL and DataFrame queries over the table cannot select it. | ResultMismatch | Seen on 4.1 | |
+| Scans of a table with many small files can be slower than vanilla Spark on EBS storage. In the report, two of five TPC-DS SF1000 queries take 2.2 and 3.4 times as long as on vanilla Spark, with 182,300 files in one table. A maintainer did not see the slowdown on a host with local NVMe storage. | | Reported on 3.4 | [#10778](https://github.com/apache/gluten/issues/10778) |
+
+### Write problems
+
+These need native write on, unless a row says otherwise.
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A write partitioned by a FLOAT or DOUBLE column fails with `Unsupported partition type`. | Exception | 3.5 to 4.1, seen on 4.1 | [#13138](https://github.com/apache/gluten/issues/13138) |
+| A write to a path scheme the native library cannot write fails at task time. This covers custom Hadoop file systems, and S3, GCS, HDFS or ABFS in a build without that connector. | Exception | 3.5 to 4.1, seen on 4.1 (custom file system), from code (missing connector) | [#4963](https://github.com/apache/gluten/issues/4963) |
+| Files written for a column mapping table, in name or id mode, have no Parquet field IDs, which the Delta Lake protocol requires. Delta Lake and Gluten read name-mode files by physical name. Readers that match by field ID cannot. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| Spark's own reader fails on natively written id-mode files with `Spark read schema expects field Ids, but Parquet file schema doesn't contain any field Ids`. Spark uses this reader when a scan falls back or Gluten is off. With `spark.sql.parquet.fieldId.read.ignoreMissing=true`, Spark skips this check and returns NULL for the data columns of these files (from code). That setting is not tested with these files. | Exception | 3.5 to 4.1, seen on 4.1 | |
+| Files written for a UniForm table have no Parquet field IDs, so the Iceberg metadata points at files without IDs. No error is raised. | ResultMismatch | 3.5 to 4.1, from code | [#12957](https://github.com/apache/gluten/pull/12957) |
+| A write to a partitioned UniForm table fails with `The file schema type should be equal with the input rowvector type`. | Exception | 3.5 to 4.1, from code | |
+| Writes to a table with the `materializePartitionColumns` feature leave the partition columns out of the data files. | ResultMismatch | 4.1, from code | |
+| Some writes of a TIMESTAMP_NTZ data column fail with `The file schema type should be equal with the input rowvector type`. Delta Lake's type widening tests fail this way when they write `CAST(value AS TIMESTAMP_NTZ)`. Delta Lake's data skipping tests also write that cast and pass. On Spark 4.x, `spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation=true` sends TIMESTAMP_NTZ writes to Delta's writer. It is not tested as a workaround. | Exception | Seen on 4.1 | |
+| A partitioned write fails in Velox's `PartitionIdGenerator` when a TIMESTAMP or TIMESTAMP_NTZ partition value has sub-millisecond digits. | Exception | 3.5 to 4.1, seen on 4.1 (TIMESTAMP_NTZ), from code (TIMESTAMP) | |
+| A generated partition column such as `DATE_FORMAT(ts, 'yyyy-MM')` stores `12345-07` for the year 12345. Spark stores `+12345-07`. Not checked with native write off. | ResultMismatch | Seen on 4.1 | |
+| The `compression` option set to brotli or lzo fails the write. | Exception | 3.5 to 4.1, from code | |
+| The `compression` option set to lz4_raw writes Snappy data into a `.lz4raw.parquet` file. | ResultMismatch | 3.5 to 4.1, from code | |
+| `maxRecordsPerFile` on an unpartitioned write counts rows but writes whole batches. When the limit is smaller than a batch, the whole batch goes into one file and the files rotated for that batch are empty. | ResultMismatch | 3.5 to 4.1, from code | |
+| `spark.sql.maxConcurrentOutputFileWriters` above 0 makes a partitioned write with unsorted input fail with `Underlying columnar data is inaccessible from BatchCarrierRow`. | Exception | 3.5 to 4.1, from code | |
+| An unpartitioned native write whose first task gets no rows, for example a write of an empty DataFrame, leaves a 0-byte file that is not valid Parquet. Reading the table directory as plain Parquet can then fail with `is not a Parquet file (length is too low: 0)`. With statistics on, the file stays on disk until `VACUUM`. With `spark.databricks.delta.stats.collect=false`, Delta commits it. The last two sentences are from code. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| A statistics aggregation that Velox cannot offload fails the write with `ClassCastException`. The TIMESTAMP_NTZ case from the issue no longer fails on main. | Exception | 3.5 to 4.1, from code | [#12538](https://github.com/apache/gluten/issues/12538) |
+
+### MERGE, UPDATE and DELETE problems
+
+MERGE does not use native write, so the MERGE rows also apply with native write off (from code).
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| UPDATE or DELETE through a temp view that swaps column names, such as `SELECT value AS key, key AS value FROM t`, changes different rows than Delta Lake. In Delta Lake's test, a DELETE removes both rows where Delta Lake removes one. Not checked with native write off. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| MERGE operation metrics in `DESCRIBE HISTORY` are too high, and a MERGE that changes nothing records change data rows on a change data feed table. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | [#9003](https://github.com/apache/gluten/issues/9003) |
+| With ANSI mode on, `spark.gluten.sql.ansiFallback.enabled=false` and `spark.databricks.delta.updateAndMergeCastingFollowsAnsiEnabledFlag=true`, a MERGE that casts `ARRAY<BIGINT>` to `ARRAY<INT>` or `MAP<STRING,BIGINT>` to `MAP<STRING,INT>` with overflowing values completes without an error. This was seen in a `WHEN MATCHED THEN UPDATE` clause and in a `WHEN NOT MATCHED THEN INSERT` clause inside `foreachBatch`. The Delta Lake flag is internal and false by default. | ResultMismatch | Seen on 4.1 | |
+| MERGE schema evolution that adds or removes fields of a STRUCT map key can fail with `map key cannot be indeterminate`. | Exception | Seen on 4.1 | |
+
+### Error types
+
+These operations fail as in Delta Lake, with a different exception type or error class. A failed write commits nothing (from code). Seen on 4.1. Spark 3.5 and 4.0 share the code.
+
+| Failure | Delta Lake raises | Gluten raises |
+|---|---|---|
+| CHECK, NOT NULL or generated-column violation on a partitioned native write. The NOT NULL case is from code. | `DeltaInvariantViolationException` | `SparkException` (`TASK_WRITE_FAILED`) wrapping `GlutenException` |
+| `replaceWhere` violation on a partitioned native write | `DeltaAnalysisException` (`DELTA_REPLACE_WHERE_MISMATCH`) | `SparkException` (`TASK_WRITE_FAILED`) wrapping `GlutenException` |
+| Store-assignment cast overflow in a MERGE `WHEN MATCHED THEN UPDATE` clause | `DeltaArithmeticException` (`DELTA_CAST_OVERFLOW_IN_TABLE_WRITE`) | `GlutenException` |
+| ANSI cast overflow evaluated in Velox | `SparkArithmeticException` | `GlutenException` (`VeloxUserError`) |
+| Missing data file | `SparkException` (`FAILED_READ_FILE.FILE_NOT_EXIST`) on Spark 4.x | `SparkException` wrapping `GlutenException` (`FILE_NOT_FOUND`) |
+
+For errors raised on the JVM, the Delta Lake message stays in the `GlutenException` message. On unpartitioned `save(path)` writes, CHECK and NOT NULL violations keep Delta Lake's exception types. Tracked in [#11912](https://github.com/apache/gluten/issues/11912).
+
+## Configuration
+### Gluten Options
+
+| Gluten option | Default | Description |
+| --- | --- | --- |
+| spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite | false | Experimental. Velox writes the data files of the commands marked Offload in [Write commands](#write-commands). Spark 3.5 to 4.1. No effect on Spark 3.4. |
+| spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan | true | Experimental. When false, the DML target scan that reads row indexes for deletion vectors stays on Spark. Other scans still offload. |
+| spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan | true | Experimental. When false, batch change data feed reads use Delta Lake's change data feed scan. |
+| spark.gluten.sql.columnar.filescan | true | When false, every V1 file source scan (`FileSourceScanExec`) falls back, Delta scans included. No Gluten option turns off Delta scans alone. |
+| spark.gluten.sql.ansiFallback.enabled | true | With ANSI mode on, every query falls back. See [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x). |
+| spark.gluten.sql.columnar.parquet.write.blockSize | 128MB | Row-group size for native Delta writes. |
+| spark.gluten.sql.native.parquet.write.blockRows | 100000000 | Row-group row limit for native Delta writes. |
+| spark.gluten.sql.native.writer.enabled | (unset) | Controls native writes of non-Delta Parquet files. Unset means on. No effect on Delta writes. |
+| spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation | false | When true, every operator whose input or output includes TIMESTAMP_NTZ falls back. On Spark 4.x, native Delta writes of TIMESTAMP_NTZ schemas then use Delta's writer. From code. |
+| spark.gluten.sql.columnar.backend.velox.scanMapKeyPruningEnabled | false | When true, Velox skips map keys a query does not look up, if the lookups sit in filters and projections directly above the scan. Applies to Delta scans without column mapping, with or without deletion vectors. Off for column-mapped tables. |
+| spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames | true | Velox matches Parquet columns by name. Id-mode column mapping tables are read by physical name. When false, id-mode tables are read by position and name-mode tables still by physical name. From code. |
+| spark.gluten.sql.fallbackUnexpectedMetadataParquet | false | When true, Delta scans fall back if a sampled file is encrypted, carries legacy datetime rebase metadata, uses LZO or BROTLI, or has variant annotations. LZ4_RAW also counts on Spark 4.0 and 4.1. The variant check runs on Spark 4.1 only. The sample is the first `spark.gluten.sql.fallbackUnexpectedMetadataParquet.limit` files (default 10) under the table root. `_delta_log` files count, so the sample may contain no data files. The check is skipped when `spark.gluten.sql.scan.fileSchemeValidation.enabled=false`. From code. |
+| spark.gluten.sql.scan.fileSchemeValidation.enabled | true | Scans on a file scheme Velox cannot read fall back. |
+
+The three `spark.gluten.sql.columnar.backend.velox.delta.*` options are read per query, so `SET` changes them for the session.
+
+### Catalogs
+
+Gluten does not change DeltaCatalog. `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` are written natively only through a DeltaCatalog (from code). With native write on, `UPDATE`, `DELETE`, plain `OPTIMIZE` and `save(path)` are written natively with any catalog (from code).
+
+### SQL Extensions
+Fallback
+
+Supports `spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension`. Gluten does not offload the command nodes of Delta's own SQL commands, such as `VACUUM`, `RESTORE` and `DESCRIBE HISTORY`. Their `_delta_log` scans stay on Spark. From code. `OPTIMIZE` follows [Write commands](#write-commands). GlutenPlugin appends its own extension.
+
+### Runtime configuration
+The "Gluten Support" column uses these values:
+
+✅ Supported<br>
+❌ Not Supported<br>
+⚠️ Partial Support<br>
+🚫 Not applied or transparent to Gluten<br>
+
+A blank cell means unknown.
+
+### Spark SQL Options
+
+| Option | Default | Description | Gluten Support |
+| --- | --- | --- | --- |
+| spark.sql.extensions | (none) | Set to `io.delta.sql.DeltaSparkSessionExtension`. | ✅ |
+| spark.sql.catalog.spark_catalog | builtin on Spark 4.x, (unset) on 3.x | Set to `org.apache.spark.sql.delta.catalog.DeltaCatalog`. | ✅ |
+| spark.sql.ansi.enabled | true on Spark 4.x, false on 3.x | When true, every query falls back unless `spark.gluten.sql.ansiFallback.enabled=false`. | ⚠️ |
+| spark.sql.parquet.compression.codec | snappy | Codec for native writes when the `compression` option is unset. | ⚠️ |
+| spark.sql.files.maxRecordsPerFile | 0 | Exact on partitioned native writes. See [Write problems](#write-problems). | ⚠️ |
+| spark.sql.maxConcurrentOutputFileWriters | 0 | Above 0, partitioned native writes with unsorted input fail. From code. | ❌ |
+| spark.sql.files.ignoreMissingFiles | false | Passed to Velox. Native scans skip missing files when true. | ✅ |
+| spark.sql.parquet.mergeSchema | false | When true, Delta scans fall back. From code. | ❌ |
+| spark.sql.parquet.outputTimestampType | INT96 | Ignored by native writes. From code. See [Natively written files](#natively-written-files). | ❌ |
+| spark.sql.parquet.writeLegacyFormat | false | Native writes apply it to DECIMAL only. From code. See [Natively written files](#natively-written-files). | ⚠️ |
+| spark.sql.caseSensitive | false | When false, native writes lowercase nested struct field names. From code. | ⚠️ |
+| spark.sql.sources.partitionOverwriteMode | STATIC | With native write on, Velox writes the files of a dynamic overwrite through `save(path)`, and the results are correct (seen on 4.1). | ✅ |
+| spark.databricks.delta.deletionVectors.useMetadataRowIndex | true | When false, scans that read Delta's generated deletion vector columns fall back. See [Deletion vectors](#deletion-vectors). | ⚠️ |
+| spark.databricks.delta.stats.collect | true | Native writes compute file statistics in Velox. | ✅ |
+| spark.databricks.delta.stats.skipping | true | Delta skips files by statistics before the scan runs. The scan offloads either way, except in the deletion-vector case in [Deletion vectors](#deletion-vectors). | 🚫 |
+| spark.databricks.delta.optimizeWrite.enabled | (unset) | Native writes run the optimized write shuffle in Velox (from code). File counts match Delta Lake (seen on 4.1). | ✅ |
+| spark.databricks.delta.optimizeWrite.binSize | 512 (MiB) | Used by the native optimized write shuffle. From code. | ✅ |
+| spark.databricks.delta.autoCompact.enabled | (unset) | Compacts natively written tables. Seen on 4.1. | ✅ |
+| spark.databricks.delta.schema.autoMerge.enabled | false | Delta applies schema changes before the write. | 🚫 |
+| spark.databricks.delta.dynamicPartitionOverwrite.enabled | true | Delta decides which partitions to replace. | ✅ |
+| spark.databricks.delta.merge.repartitionBeforeWrite.enabled | true | MERGE uses Delta's writer, so this only shapes Delta's write. | 🚫 |
+| spark.databricks.delta.optimize.maxFileSize | 1073741824 (1 GiB) | Delta plans the bins. With native write on, Velox writes the files of plain `OPTIMIZE`. | ✅ |
+| spark.databricks.delta.optimize.minFileSize | 1073741824 (1 GiB) | Delta plans the bins. With native write on, Velox writes the files of plain `OPTIMIZE`. | ✅ |
+| spark.databricks.delta.write.txnAppId, spark.databricks.delta.write.txnVersion | (unset) | Idempotent writes. Commit only. | 🚫 |
+| spark.databricks.delta.commitInfo.userMetadata | (unset) | Commit only. | 🚫 |
+| spark.databricks.delta.properties.defaults.* | (unset) | Default for the matching `delta.*` table property on new tables. | 🚫 |
+
+`spark.databricks.delta.deletionVectors.useMetadataRowIndex`, `spark.databricks.delta.optimizeWrite.enabled`, `spark.databricks.delta.optimizeWrite.binSize` and `spark.databricks.delta.autoCompact.enabled` do not exist in Delta Lake 2.4.0.
+
+#### Read options
+
+| Option | Default | Description | Gluten Support |
+| --- | --- | --- | --- |
+| versionAsOf | (latest) | Version of the table snapshot to read. Seen on 4.1. | ✅ |
+| timestampAsOf | (latest) | Read the snapshot current at this timestamp. Seen on 4.1. | ✅ |
+| readChangeFeed | false | Read the change data feed. Ranges with deletion vectors fall back. | ⚠️ |
+| startingVersion, endingVersion | (none) | Change data feed version range. `startingVersion` also applies to streaming reads. | ✅ |
+| startingTimestamp, endingTimestamp | (none) | Change data feed timestamp range. `startingTimestamp` also applies to streaming reads. Seen on 4.1. | ✅ |
+| maxFilesPerTrigger | 1000 | Streaming: files per micro-batch | 🚫 |
+| maxBytesPerTrigger | (unset) | Streaming: bytes per micro-batch | 🚫 |
+| ignoreDeletes, ignoreChanges, skipChangeCommits | false | Streaming: how to treat commits that remove data | 🚫 |
+| excludeRegex | (none) | Streaming: skip files whose path matches | 🚫 |
+| failOnDataLoss | true | Streaming: fail when Delta log files the stream needs are missing | 🚫 |
+| allowSourceColumnDrop, allowSourceColumnRename, allowSourceColumnTypeChange | (unset) | Streaming schema tracking. Delta Lake 4.0.1 and 4.1.0. | 🚫 |
+
+#### Write options
+
+| Option | Default | Description | Gluten Support |
+| --- | --- | --- | --- |
+| compression | spark.sql.parquet.compression.codec | Native writes support the codecs in [Natively written files](#natively-written-files). | ⚠️ |
+| maxRecordsPerFile | spark.sql.files.maxRecordsPerFile | Exact on partitioned native writes. See [Write problems](#write-problems). | ⚠️ |
+| partitionOverwriteMode | spark.sql.sources.partitionOverwriteMode | `dynamic` works through `save(path)`. `INSERT OVERWRITE` and `insertInto` use Delta's writer (from code). | ✅ |
+| replaceWhere | (none) | A violating partitioned native write fails and commits nothing, with a different error type (see [Error types](#error-types)). | ⚠️ |
+| mergeSchema | spark.databricks.delta.schema.autoMerge.enabled | Delta applies the schema change before the write. | 🚫 |
+| overwriteSchema | false | Delta applies the schema change before the write. | 🚫 |
+| optimizeWrite | (unset) | Runs the rebalancing shuffle in Velox before the native write. From code. | ✅ |
+| txnAppId, txnVersion | (unset) | Idempotent writes. Commit only. | 🚫 |
+| userMetadata | (unset) | Stored in `commitInfo.userMetadata`. Commit only. | 🚫 |
+| dataChange | true | Sets `dataChange` on the file actions. Commit only. | 🚫 |
+| parquet.* (writer option or Hadoop setting) | | Ignored by native writes (from code). Use `spark.gluten.sql.columnar.parquet.write.blockSize` for row groups. | ❌ |
+| fs.* Hadoop file system options | | A custom file system scheme fails on native writes (seen on 4.1). Per-write options for supported stores, such as `fs.s3a.access.key`, are not tested with native writes. | ❌ |
Review Comment:
```suggestion
| File system schemes unsupported by the native library | | Native writes fail at task time. A custom Hadoop file system scheme failed on Spark 4.1. See [Write problems](#write-problems). | ❌ |
| fs.* per-write options for supported stores | | Per-write options such as `fs.s3a.access.key` have not been tested with native writes. | |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4312932056
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,632 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+Gluten's Velox backend can accelerate eligible scans and query operators over Delta Lake tables. Read
+acceleration does not require native writes. Native Delta writes are experimental and disabled by
+default. This guide covers the supported Spark and Delta Lake versions, setup, feature support and
+known limitations.
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+## How a Delta query runs
+
+Take a Delta table partitioned by `sale_date`:
+
+```sql
+SELECT SUM(amount)
+FROM sales
+WHERE sale_date = DATE '2026-10-01'
+ AND amount > 100;
+```
+
+```mermaid
+flowchart LR
+ subgraph Planning["Spark driver"]
+ Snapshot["1. Delta snapshot<br/>Schema + active files"]
+ Select["2. Delta file pruning<br/>Partitions + file statistics"]
+ Plan["3. Spark plan<br/>Gluten offload checks"]
+ Snapshot --> Select --> Plan
+ end
+
+ subgraph Execution["Spark executors"]
+ Scan["4. Velox scan<br/>Reads the selected files"]
+ Compute["5. Velox operators<br/>Filter + SUM"]
+ Spark["Spark operators<br/>Operators Gluten does not offload"]
+ Scan --> Compute
+ Compute <-->|"Row and columnar conversions"| Spark
+ end
+
+ subgraph Storage["Delta table"]
+ Log[("_delta_log<br/>Metadata + file statistics")]
+ Data[("Parquet files")]
+ end
+
+ Result["6. Query result"]
+
+ Log --> Snapshot
+ Plan -->|"Offloaded operators"| Scan
+ Plan -->|"Other operators"| Spark
+ Data --> Scan
+ Data --> Spark
+ Compute --> Result
+ Spark --> Result
+```
+
+Delta Lake loads the snapshot and prunes partitions and files while Spark plans the query. Gluten then
+offloads the operators Velox supports, and the rest run on Spark, so one query can mix Velox and Spark
+operators. Delta Lake keeps the table metadata, the transaction log and commits.
+
+## Supported Spark and Delta Lake versions
+
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. On Spark 4.0 and 4.1, the Read and Native write columns assume ANSI mode off (see [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x)). The copied Delta Lake suites run with native write on and `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`, which is off by default.
+
+Spark 4.1 has the most tests, because the nightly job runs on it. That job builds Delta Lake 4.2.0 from source against Spark 4.1.0, which is newer than the Delta Lake 4.1.0 in the table, and runs its Spark test suites. It puts a Gluten bundle compiled against Spark 4.1.1 and Delta Lake 4.1.0 on the test classpath. It replaces Delta Lake's `DeltaSQLCommandTest` with Gluten's copy, which loads Gluten with native write on and the non-default `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`. It then runs `.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh`. That script cherry-picks two Delta Lake test changes that are not in 4.2.0, delta-io/delta#7104 and #7105, so Delta Lake's scan checks accept Gluten's scan node. It also sets a row-group row limit in two test fixtures. The same job also runs on pull requests that change Gluten's Delta code. "Seen on 4.1" on this page refers to that job.
+
+The job's known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector, change data feed, column mapping, time travel and partition filter reads, and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+```bash
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+```
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. `dev/release/build-release.sh` passes `-Pdelta` only for Spark 3.4 and 3.5, so its Spark 4.0 and 4.1 bundles have no Delta Lake support either. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add Delta Lake with `--packages` and the coordinate from the table above, which also adds its `delta-storage` dependency. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off. From code.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+```bash
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+```
+
+On Spark 4.0 or 4.1, use the Gluten bundle and the Delta Lake jar for that version, and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code. Turning ANSI mode off also changes Spark's own behavior for invalid input. For example, `CAST('a' AS INT)` returns NULL instead of failing.
+
+```properties
+spark.sql.ansi.enabled=false
+```
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. On Spark 3.x, turning ANSI mode on makes every query fall back in the same way. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Gluten passes ANSI mode to Velox. Velox does not implement ANSI behavior for every expression, so some invalid input returns a result where Spark fails ([#10134](https://github.com/apache/gluten/issues/10134)). For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). Cast overflow errors raised in Velox have a different exception type (see [Error types](#error-types)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back. |
+| Fallback | Runs on Spark or Delta Lake. |
+| Exception | Fails instead of falling back. |
+| ResultMismatch | Can return wrong results or write wrong files. |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job showed the result, with Delta Lake 4.2.0 on Spark 4.1.0.<br>
+Reported on 3.4, Reported on 4.1: the linked issue reports the result on that Spark version. No CI test checks it.<br>
+No label: a test in the Gluten repository that runs in CI checks the status on the stated versions.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver. From code.
+
+## Creating a table
+Fallback
+
+```sql
+CREATE TABLE t (id INT, name STRING) USING delta;
+```
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload. From code.
+
+Offload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+```
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+## Writing
+PartialOffload
+
+```sql
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+```
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+```sql
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+```
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+```
+
+On Spark 3.5 to 4.1, set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). Velox writes the files of this UPDATE. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+```scala
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+```
+
+The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the command's plan from the Spark UI SQL tab or a `QueryExecutionListener`. Each write also reports a nested `deltaTransactionalWrite` execution for the query that feeds it. That plan never has the prefix, whichever writer ran. From code.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks that a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5 run as Gluten native write commands. It also checks on 3.5 and 4.0 that a DELETE that rewrites a file runs in Gluten's transaction and that its scan runs in Velox. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload, PartialOffload when it writes deletion vectors (from code on 4.1), Fallback when it is metadata-only (from code) | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. |
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | `DeltaTable.optimize().executeCompaction()`, with or without `where(...)` | Offload | Bin-packing, by path or by name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans, the DML target scan and joins can run in Velox. On Spark 3.5 and 4.0, Gluten's tests check that a deletion-vector MERGE runs a scan of the target table in Velox. The other cases are from code. Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake in `DescribeDeltaHistorySuite`. Seen on 4.1. Three unpartitioned tests in `DeleteMetricsSuite` and `UpdateMetricsSuite` expect two added files and get one. The job sets `spark.default.parallelism=1`, which plans the five small input files as one scan partition (from code).
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload after native writes, PartialOffload after other writes | Compacts natively written tables (seen on 4.1). The auto compaction tests that fail in the nightly job fail because its `spark.default.parallelism=1` makes their first write produce one file (from code). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). |
+| Files with deletion vectors | Fallback | Offload |
+| Change data feed, no deletion vectors in range | Offload. PartialOffload when the table allows deletion vectors and the range has `_change_data` files, because those scans then fall back (from code). | Offload |
+| Change data feed, deletion vectors in range | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors and no `_change_data` files (from code) | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors (from code) |
+
+```sql
+SELECT count(1) AS count, name
+FROM t
+GROUP BY name;
+```
+
+Reads need no Gluten option. With `spark.sql.adaptive.enabled=false`, `EXPLAIN` shows the native scan as `FileDeltaScanTransformer` and `EXPLAIN FORMATTED` shows `DeltaScanTransformer`. With AQE on, the default, `EXPLAIN` and `EXPLAIN FORMATTED` of a query with a shuffle, such as this one, print the plan from before Gluten offloads it. The Spark UI plan graph of a finished query shows `DeltaScanTransformer`. Delta prunes partitions and skips files by statistics while it plans the query, before the scan runs.
+
+| Read | Status | Notes |
+|---|---|---|
+| By table name, ``delta.`/path` `` or `spark.read.format("delta").load(path)` | Offload | Same native scan. |
+| `VERSION AS OF` | Offload | Tested on Spark 3.4 to 4.1. |
+| `TIMESTAMP AS OF`, `versionAsOf`, `timestampAsOf` | Offload | From code. Delta Lake's `DeltaTimeTravelSuite` passes (seen on 4.1). |
+| Streaming source | Offload | Each micro-batch is a Delta file scan. From code. Delta Lake's `DeltaSourceSuite` passes (seen on 4.1). |
+| Dynamic partition pruning | Not tested | |
+
+### Deletion vectors
+Offload
+
+```sql
+OPTIMIZE t;
+ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
+DELETE FROM t WHERE id IN (3, 4);
+SELECT * FROM t;
+```
+
+On Spark 3.5 to 4.1, Velox drops the deleted rows in the scan. This needs Delta's `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default. The DML target scan, which finds the rows for new deletion vectors, also runs in Velox. Gluten tests check the scan on Spark 3.5 to 4.1 and the DML target scan on Spark 3.5 and 4.0. Delta writes the deletion-vector files.
+
+Fallback
+
+```sql
+SET spark.databricks.delta.deletionVectors.useMetadataRowIndex=false;
+SELECT * FROM t;
+RESET spark.databricks.delta.deletionVectors.useMetadataRowIndex;
+```
+
+| Case | Behavior |
+|---|---|
+| Spark 3.5 to 4.1 with `useMetadataRowIndex=false` | A scan that reads Delta's generated `__delta_internal_is_row_deleted` or `__delta_internal_row_index` column falls back with `fallback Delta scan requiring generated DV metadata`. Delta adds the first column when any file in the scanned table version, micro-batch, DML file set or `OPTIMIZE` bin has a deletion vector, so `OPTIMIZE` and `REORG TABLE ... APPLY (PURGE)` scans of those files also fall back (from code). It adds the second to DML target scans that write deletion vectors. Other scans offload. Tested on Spark 3.5 and 4.0. |
+| Spark 3.4, normal reads | A read falls back with `fallback Spark 3.4 Delta DV scan` when a file it selected has a deletion vector. Tested on Spark 3.4. A read whose selected files have none offloads (from code). |
+| Spark 3.4, other scans, such as DML target scans, `OPTIMIZE` scans, streaming micro-batches and change data feed scans of `_change_data` files | Fall back whenever the table protocol allows deletion vectors. Change data feed scans of added or removed files fall back only when a file they read has a deletion vector. From code. |
+| `spark.databricks.delta.stats.skipping=false` | Delta Lake itself rejects reads of a table with deletion vectors with `Cannot work with a non-pinned table snapshot of the TahoeFileIndex` (seen on 4.1). Before Delta Lake 4.1.0, the table feature alone triggers it (from code). |
+
+The fallback reasons show in [Gluten UI](GlutenUI.md).
+
+### Change data feed
+Offload
+
+```sql
+CREATE TABLE c (id INT, name STRING) USING delta
+ TBLPROPERTIES ('delta.enableChangeDataFeed' = true);
+INSERT INTO c VALUES (1, 'a'), (2, 'b');
+UPDATE c SET name = 'bb' WHERE id = 2;
+SELECT id, name, _change_type, _commit_version
+FROM table_changes('c', 1)
+WHERE id = 2;
+```
+
+Gluten expands a batch change data feed read into Delta file scans that run in Velox with the filters pushed down. Delta also adds an empty placeholder scan to the plan. That scan falls back when the table schema has a MAP at any nesting level (from code). If the expansion fails, Gluten logs a warning and the read uses Delta's change data feed scan. From code.
+
+PartialOffload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false;
+SELECT * FROM table_changes('c', 1);
+```
+
+A read whose version range adds or removes a file with a deletion vector uses Delta's change data feed scan. Insert-only ranges of a table with deletion vectors enabled still run in Velox. With `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false`, every batch change data feed read uses Delta's scan. Delta's scan reads the data files in a query of its own that Gluten also plans, and `EXPLAIN` does not show it (from code). That query's file scans can run in Velox, except scans of added or removed files that have deletion vectors and, on Spark 3.4, scans of `_change_data` files when the table allows deletion vectors (from code). Streaming change data feed reads ignore this option. They run as Delta file scans. A scan whose files have deletion vectors falls back, and on Spark 3.4 so does a scan of `_change_data` files when the table allows deletion vectors (from code).
+
+### Read metadata
+Offload
+
+```sql
+SELECT id, _metadata.file_path, _metadata.row_index
+FROM t
+WHERE _metadata.row_index = 2;
+```
+
+The native scan fills the `_metadata` file fields and `input_file_name()` (from code), and `_metadata.row_index`. On Spark 3.5 to 4.1, Delta Lake exposes `_metadata.row_index` only on tables that allow deletion vectors or have row tracking, so the example needs `t` from [Deletion vectors](#deletion-vectors) (from code). A query that selects only `_metadata.file_name` can fail when the table has a BIGINT data column named `file_name` (seen on 4.1, see [Read problems](#read-problems)). Gluten tests check `_metadata.row_index` in the native scan on Spark 3.5 and 4.0, and Delta Lake tests that read it pass (seen on 4.1). Spark 3.4 is not tested. On row-tracking tables, `_metadata.row_id` and `_metadata.row_commit_version` are read natively (from code). In the nightly job, they return correct values except in the cases in [Read problems](#read-problems) (seen on 4.1).
+
+Fallback
+
+```sql
+DESCRIBE HISTORY t;
+```
+
+`DESCRIBE HISTORY` reads each commit file on Spark, through Delta's LogStore in a `mapPartitions` job. When Delta replays the log for a snapshot, every scan under `_delta_log`, checkpoints included, stays on Spark with the reason `fallback Delta _delta_log scan`. From code.
+
+## Table features
+Offload/Fallback
+
+The Read column covers Spark 3.5 to 4.1. It covers Spark 3.4 only where the row names it. The Native write column applies to Spark 3.5 to 4.1 with native write on.
+
+| Feature | Read | Native write | Notes |
+|---|---|---|---|
+| Deletion vectors | Offload, Fallback on Spark 3.4 | PartialOffload | Gluten's tests check deletion vector DELETE and MERGE results with native write on, on Spark 3.5 and 4.0. Which writer writes each file is from code. See [Deletion vectors](#deletion-vectors). |
+| Change data feed | Offload, also on Spark 3.4. Ranges with deletion vectors fall back. On Spark 3.4, scans of `_change_data` files also fall back when the table allows deletion vectors (from code). | Offload | UPDATE and DELETE write the `_change_data` files with Velox (from code). Change data feed reads of their changes are correct (seen on 4.1). See [Change data feed](#change-data-feed). |
+| Column mapping, name mode | Offload, also on Spark 3.4 | ResultMismatch | Files have no Parquet field IDs. Seen on 4.1. See [Write problems](#write-problems). |
+| Column mapping, id mode | Offload, also on Spark 3.4. ResultMismatch when field IDs differ from physical names (seen on 4.1). | ResultMismatch, Exception when Spark's reader reads the files (seen on 4.1) | Gluten matches columns by physical name. See [Known problems](#known-problems). |
+| Identity columns | Offload | Offload | Seen on 4.1. High-water marks are correct. The projection that generates identity values runs on Spark (from code). |
+| Generated columns | Offload | Offload | Seen on 4.1. No Gluten test covers generated columns. Velox computes the generated values when it supports the expression (from code). Partition filters derived from them reach the native scan. `DATE_FORMAT` results for years above 9999 differ (see [Write problems](#write-problems)). |
+| CHECK constraints, NOT NULL | Offload | Offload | Delta checks the rows on the JVM before Velox writes them. Gluten's tests cover NOT NULL on Spark 3.5 and 4.0. Seen on 4.1. |
+| Row tracking | Offload, Exception for `_metadata.base_row_id` filters, ResultMismatch for `_metadata.row_id` when older files lack base row IDs | Offload | Seen on 4.1. See [Read problems](#read-problems). |
+| Liquid clustering | Offload | PartialOffload | Reads use the same native scan as other tables. Native commands write to clustered tables. `OPTIMIZE`, which clusters the data, uses Delta's writer. All three are from code. |
+| UniForm (IcebergCompatV1, V2) | | ResultMismatch, Exception when partitioned | The write status is from code. No test reads a UniForm table. |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Offload, Exception for FLOAT or DOUBLE partition columns, for TIMESTAMP_NTZ partition values with sub-millisecond digits and for some writes of TIMESTAMP_NTZ data columns | Seen on 4.1. Spark 3.5 and 4.0 are not covered. |
+| TIMESTAMP_NTZ | PartialOffload, also on Spark 3.4 | Offload, Exception for some writes (seen on 4.1) | See [DataType](#datatype). |
+| Variant, including shredding | Fallback | Fallback | Spark 4.x. From code. See [DataType](#datatype). |
+| Collations | Fallback | | Spark 4.x. From code. See [DataType](#datatype). |
+| materializePartitionColumns | | ResultMismatch | Spark 4.1. From code. |
+
+## DataType
+
+The Read column applies to Spark 3.4 to 4.1. On all of them, Gluten's tests read INT, BIGINT, DECIMAL(12,2), STRING, DATE, TIMESTAMP, TIMESTAMP_NTZ, STRUCT, ARRAY and MAP columns natively. The other read statuses are from code, unless a row says otherwise. The native write columns apply to Spark 3.5 to 4.1 with native write on. On Spark 3.5 and 4.0, Gluten's tests write BOOLEAN, INT, BIGINT, STRING, STRUCT and ARRAY columns natively. The other native write statuses are seen on 4.1, unless a row says they are from code.
+
+| Type | Read | Native write | Native write as partition column | Notes |
|---|---|---|---|---|
-| `spark-3.5` | Spark 3.5.x | 2.12 | 3.3.x | Supported |
-| `spark-4.0` | Spark 4.0.x | 2.13 | 4.0.x | Supported |
-
-Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The difference between
-the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta 3.3 vs Spark 4.0 +
-Delta 4.0), not a native-write capability gap.
-
-## Build and runtime notes
-
-Build Gluten with Delta support by enabling `-Pdelta` together with the Velox backend profile and a Spark profile.
-
-- Spark 3.5 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests`
-- Spark 4.0 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13 -Pjava-17 -DskipTests`
-
-Native Delta write is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite`
- - Default: `false`
- - Type: experimental
-
-Native change data feed scan offload is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan`
- - Default: `true`
- - Type: experimental
-
-The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan`
- - Default: `true`
- - Type: experimental
- - When disabled, the DML target scan that produces file paths and row indexes for
- deletion-vector writes stays on Spark; other scans are unaffected.
-
-| Feature | Delta minWriterVersion | Delta minReaderVersion | Iceberg format-version | Feature type | Supported by Gluten (Velox) |
-|---|---:|---:|---:|---|---|
-| Basic functionality | 2 | 1 | 1 | Writer | Yes |
-| CHECK constraints | 3 | 1 | N/A | Writer | No |
-| Change data feed | 4 | 1 | N/A | Writer | Yes |
-| Generated columns | 4 | 1 | N/A | Writer | Partial |
-| Column mapping | 5 | 2 | N/A | Reader and writer | Yes |
-| Identity columns | 6 | 1 | N/A | Writer | Yes |
-| Row tracking | 7 | 1 | 3 | Writer | Partial |
-| Deletion vectors | 7 | 3 | 3 | Reader and writer | Partial |
-| TimestampNTZ | 7 | 3 | 1 | Reader and writer | No |
-| Liquid clustering | 7 | 3 | 1 | Reader and writer | Yes |
-| Iceberg readers (UniForm) | 7 | 2 | N/A | Writer | Not tested |
-| Type widening | 7 | 3 | N/A | Reader and writer | Partial |
-| Variant | 7 | 3 | 3 | Reader and writer | Not tested |
-| Variant shredding | 7 | 3 | 3 | Reader and writer | Not tested |
-| Collations | 7 | 3 | N/A | Reader and writer | Not tested |
-| Protected checkpoints | 7 | 1 | N/A | Writer | Not tested |
+| BOOLEAN | Offload | Offload | Offload | |
+| TINYINT, SMALLINT, INT, BIGINT | Offload | Offload | Offload | |
+| FLOAT, DOUBLE | Offload | Offload | Exception | See [Write problems](#write-problems). |
+| DECIMAL | Offload | Offload | Offload | Precision above 18 included. |
+| STRING | Offload | Offload | Offload | |
+| CHAR, VARCHAR | Offload | Offload | | Stored as STRING. Delta checks the length on the JVM before the native write. Both statuses are from code. Reads return correct results (seen on 4.1). |
+| BINARY | Offload | Offload | Not tested | The read status is from code. One Delta Lake test writes a table with only a BINARY column, checks its file statistics and runs a DELETE that filters on that column (seen on 4.1). |
+| DATE | Offload | Offload | Offload | |
+| TIMESTAMP | Offload, ResultMismatch for sub-millisecond partition values (seen on 4.1) | Offload | Offload, Exception for sub-millisecond values | The partition column status is from code. See [Known problems](#known-problems). |
+| TIMESTAMP_NTZ | PartialOffload | Offload, Exception for some writes (seen on 4.1) | Offload, Exception for sub-millisecond values (seen on 4.1) | Filters, sorts, joins and other operators whose input or output includes a TIMESTAMP_NTZ column fall back, even when they do not use that column. Scans, aggregates, shuffles and some projections can run in Velox. A scan whose pushed filter has a TIMESTAMP_NTZ literal falls back (from code). Gluten's tests check on Spark 3.4 to 4.1 that a scan of the column runs in Velox, and that a filter on the column and a sort on another column cause a fallback. The other operator rules are from code. |
+| STRUCT, ARRAY, MAP | Offload | Offload | | |
+| VARIANT | Fallback | Fallback | | Spark 4.x. Shredded variant reads fall back too. Both statuses are from code. |
+| Collated STRING | Fallback | | | Spark 4.x. Scans that read the column fall back. Delta Lake 4.0.1 and 4.1.0 reject a collated column in a table schema with `DELTA_UNSUPPORTED_DATA_TYPES`. They cannot write to tables that have the `collations` feature. From code. |
+
+A plan node falls back when its output contains a type Velox does not support, such as VARIANT. With the default `spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames=true`, a scan that does not select such a column can still offload. From code. Delta Lake does not allow interval columns.
+
+## Format
+PartialOffload
+
+Delta data files are Parquet, and Gluten reads them with Velox's Parquet reader. Velox writes them only for native write commands on Spark 3.5 to 4.1. Otherwise Delta's Parquet writer writes them.
+
+### Natively written files
+
+| Property | Behavior |
+|---|---|
+| Compression | snappy (default), gzip, zstd (always level 3), lz4 (Hadoop framing), none. From code. Other codecs: see [Write problems](#write-problems). |
+| Row groups | Sized by `spark.gluten.sql.columnar.parquet.write.blockSize` and `spark.gluten.sql.native.parquet.write.blockRows`. `parquet.*` settings are ignored. From code. |
+| TIMESTAMP | INT64 microseconds. `spark.sql.parquet.outputTimestampType` is ignored. Delta Lake's writer uses INT96 by default. Spark reads both. From code. |
+| Parquet field IDs | Not written, for column-mapped and UniForm tables too. Seen on 4.1 for column mapping, from code for UniForm. See [Write problems](#write-problems). |
+| Spark schema footer entry | `org.apache.spark.sql.parquet.row.metadata` is not written ([#13288](https://github.com/apache/gluten/issues/13288)). Seen on 4.1. |
+| Nested field names | Lowercased below the top level when `spark.sql.caseSensitive=false`. Spark and Gluten match them case-insensitively. A case-sensitive reader can miss them. From code. |
+| Page index | Always written. From code. |
+| `spark.sql.parquet.writeLegacyFormat=true` | Applies to DECIMAL only. Arrays and maps keep the standard layout. From code. |
+| File statistics | numRecords, min, max and nullCount are computed in Velox (checked on Spark 3.5). Gluten's tests check the values against the rows in each file on Spark 3.5 to 4.1. Most tests in Delta's data-skipping suites pass (seen on 4.1). Of the 34 that fail, 22 check statistics and expect specific rows in each file after `repartition(2)`. The other 12 hit the `_metadata.file_name` failure in [Read problems](#read-problems). |
+| Data file paths | Come from Delta's committer, so names and random prefixes follow Delta. File names seen on 4.1, random prefixes from code. |
+
+## SQL
+
+| Statement | Status | Notes |
+|---|---|---|
+| `SELECT` | Offload | See [Reading](#reading). |
+| `table_changes()` | Offload | Ranges with deletion vectors fall back. See [Change data feed](#change-data-feed). |
+| `CREATE TABLE ... AS SELECT`, `REPLACE TABLE ... AS SELECT` | Offload | With native write on, Spark 3.5 to 4.1, and a DeltaCatalog target. Seen on 4.1, from code on 3.5 and 4.0. See [Creating a table](#creating-a-table). |
+| `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `OPTIMIZE`, `REORG` | | See [Write commands](#write-commands). |
+| `CREATE TABLE` without `AS SELECT`, `DROP TABLE`, `DESCRIBE` | Fallback | They read and write no data files. From code. |
+| `ALTER TABLE` | Fallback, PartialOffload when it reads data | Most forms read and write no data files. `ADD CONSTRAINT` scans the table, and that scan can run in Velox. On Spark 3.5 to 4.1, `ALTER COLUMN ... SYNC IDENTITY` also scans it, and `DROP FEATURE` and `SET TBLPROPERTIES ('delta.columnMapping.mode' = 'none')` can rewrite data files with Delta's writer. From code. |
+| `VACUUM`, `RESTORE`, `SHALLOW CLONE` | | They write no Parquet data files. From code. |
+| `CONVERT TO DELTA` | PartialOffload | Writes no Parquet data files. Unless the statement has `NO STATISTICS` or `spark.databricks.delta.stats.collect=false` is set, it scans the Parquet files for statistics. That scan can run in Velox. From code. |
+
+## Schema evolution
+PartialOffload
+
+The `mergeSchema` and `overwriteSchema` rows need native write on Spark 3.5 to 4.1. Otherwise Delta writes the files.
+
+| Change | Status | Notes |
+|---|---|---|
+| `mergeSchema` with `save(path)` | Offload | Velox writes the evolved schema (from code). On Spark 3.5 and 4.0, the copied `DeltaSuite` checks the results of a `mergeSchema` overwrite. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). |
+| `overwriteSchema` with `save(path)` | Offload | Delta applies the change before the write. From code. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). |
+| Schema evolution in SQL `INSERT` | PartialOffload | From code. |
+| Schema evolution in `MERGE` | PartialOffload, Exception for some STRUCT map key changes | The merged table contents match Delta Lake. Some STRUCT map key changes fail in Velox (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). The two TIMESTAMP_NTZ widening tests fail before the MERGE runs, in a setup write of TIMESTAMP_NTZ values. Seen on 4.1. |
+| `RENAME COLUMN` with column mapping | Offload | Reads after top-level and nested renames return correct results. After a top-level rename, partition pruning and file skipping still work. |
+| `DROP COLUMN` with column mapping | Offload | From code. Delta Lake's `DeltaDropColumnSuite` passes (seen on 4.1). |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ and for native writes to FLOAT or DOUBLE partition columns | Seen on 4.1. See [Table features](#table-features). |
+| `spark.sql.parquet.mergeSchema=true` | Fallback | Every Parquet scan falls back, Delta scans included. From code. |
Review Comment:
Good catch, its a different setting from Delta's `mergeSchema` write option that row is Spark's Parquet read option `spark.sql.parquet.mergeSchema`.
Made that clearer and fixed the write options table to show `mergeSchema` as supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4312797887
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back |
+| Fallback | Runs on Spark or Delta Lake |
+| Exception | Fails instead of falling back |
+| ResultMismatch | Can return wrong results or write wrong files |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+````
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+````
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+````
+
Review Comment:
Fixed. The legend rows and the other table descriptions now end with a period consistently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-12025][DOC] Rewrite the Velox Delta Lake support page [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4312195394
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back |
+| Fallback | Runs on Spark or Delta Lake |
+| Exception | Fails instead of falling back |
+| ResultMismatch | Can return wrong results or write wrong files |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
Review Comment:
> with or without native write
Could you please also add more detail on this?
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,632 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+Gluten's Velox backend can accelerate eligible scans and query operators over Delta Lake tables. Read
+acceleration does not require native writes. Native Delta writes are experimental and disabled by
+default. This guide covers the supported Spark and Delta Lake versions, setup, feature support and
+known limitations.
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+## How a Delta query runs
+
+Take a Delta table partitioned by `sale_date`:
+
+```sql
+SELECT SUM(amount)
+FROM sales
+WHERE sale_date = DATE '2026-10-01'
+ AND amount > 100;
+```
+
+```mermaid
+flowchart LR
+ subgraph Planning["Spark driver"]
+ Snapshot["1. Delta snapshot<br/>Schema + active files"]
+ Select["2. Delta file pruning<br/>Partitions + file statistics"]
+ Plan["3. Spark plan<br/>Gluten offload checks"]
+ Snapshot --> Select --> Plan
+ end
+
+ subgraph Execution["Spark executors"]
+ Scan["4. Velox scan<br/>Reads the selected files"]
+ Compute["5. Velox operators<br/>Filter + SUM"]
+ Spark["Spark operators<br/>Operators Gluten does not offload"]
+ Scan --> Compute
+ Compute <-->|"Row and columnar conversions"| Spark
+ end
+
+ subgraph Storage["Delta table"]
+ Log[("_delta_log<br/>Metadata + file statistics")]
+ Data[("Parquet files")]
+ end
+
+ Result["6. Query result"]
+
+ Log --> Snapshot
+ Plan -->|"Offloaded operators"| Scan
+ Plan -->|"Other operators"| Spark
+ Data --> Scan
+ Data --> Spark
+ Compute --> Result
+ Spark --> Result
+```
+
+Delta Lake loads the snapshot and prunes partitions and files while Spark plans the query. Gluten then
+offloads the operators Velox supports, and the rest run on Spark, so one query can mix Velox and Spark
+operators. Delta Lake keeps the table metadata, the transaction log and commits.
+
+## Supported Spark and Delta Lake versions
+
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. On Spark 4.0 and 4.1, the Read and Native write columns assume ANSI mode off (see [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x)). The copied Delta Lake suites run with native write on and `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`, which is off by default.
+
+Spark 4.1 has the most tests, because the nightly job runs on it. That job builds Delta Lake 4.2.0 from source against Spark 4.1.0, which is newer than the Delta Lake 4.1.0 in the table, and runs its Spark test suites. It puts a Gluten bundle compiled against Spark 4.1.1 and Delta Lake 4.1.0 on the test classpath. It replaces Delta Lake's `DeltaSQLCommandTest` with Gluten's copy, which loads Gluten with native write on and the non-default `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`. It then runs `.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh`. That script cherry-picks two Delta Lake test changes that are not in 4.2.0, delta-io/delta#7104 and #7105, so Delta Lake's scan checks accept Gluten's scan node. It also sets a row-group row limit in two test fixtures. The same job also runs on pull requests that change Gluten's Delta code. "Seen on 4.1" on this page refers to that job.
+
+The job's known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector, change data feed, column mapping, time travel and partition filter reads, and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+```bash
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+```
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. `dev/release/build-release.sh` passes `-Pdelta` only for Spark 3.4 and 3.5, so its Spark 4.0 and 4.1 bundles have no Delta Lake support either. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add Delta Lake with `--packages` and the coordinate from the table above, which also adds its `delta-storage` dependency. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off. From code.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+```bash
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+```
+
+On Spark 4.0 or 4.1, use the Gluten bundle and the Delta Lake jar for that version, and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code. Turning ANSI mode off also changes Spark's own behavior for invalid input. For example, `CAST('a' AS INT)` returns NULL instead of failing.
+
+```properties
+spark.sql.ansi.enabled=false
+```
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. On Spark 3.x, turning ANSI mode on makes every query fall back in the same way. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Gluten passes ANSI mode to Velox. Velox does not implement ANSI behavior for every expression, so some invalid input returns a result where Spark fails ([#10134](https://github.com/apache/gluten/issues/10134)). For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). Cast overflow errors raised in Velox have a different exception type (see [Error types](#error-types)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back. |
+| Fallback | Runs on Spark or Delta Lake. |
+| Exception | Fails instead of falling back. |
+| ResultMismatch | Can return wrong results or write wrong files. |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job showed the result, with Delta Lake 4.2.0 on Spark 4.1.0.<br>
+Reported on 3.4, Reported on 4.1: the linked issue reports the result on that Spark version. No CI test checks it.<br>
+No label: a test in the Gluten repository that runs in CI checks the status on the stated versions.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver. From code.
+
+## Creating a table
+Fallback
+
+```sql
+CREATE TABLE t (id INT, name STRING) USING delta;
+```
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload. From code.
+
+Offload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+```
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+## Writing
+PartialOffload
+
+```sql
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+```
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+```sql
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+```
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+```
+
+On Spark 3.5 to 4.1, set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). Velox writes the files of this UPDATE. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+```scala
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+```
+
+The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the command's plan from the Spark UI SQL tab or a `QueryExecutionListener`. Each write also reports a nested `deltaTransactionalWrite` execution for the query that feeds it. That plan never has the prefix, whichever writer ran. From code.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks that a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5 run as Gluten native write commands. It also checks on 3.5 and 4.0 that a DELETE that rewrites a file runs in Gluten's transaction and that its scan runs in Velox. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload, PartialOffload when it writes deletion vectors (from code on 4.1), Fallback when it is metadata-only (from code) | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. |
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | `DeltaTable.optimize().executeCompaction()`, with or without `where(...)` | Offload | Bin-packing, by path or by name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans, the DML target scan and joins can run in Velox. On Spark 3.5 and 4.0, Gluten's tests check that a deletion-vector MERGE runs a scan of the target table in Velox. The other cases are from code. Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake in `DescribeDeltaHistorySuite`. Seen on 4.1. Three unpartitioned tests in `DeleteMetricsSuite` and `UpdateMetricsSuite` expect two added files and get one. The job sets `spark.default.parallelism=1`, which plans the five small input files as one scan partition (from code).
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload after native writes, PartialOffload after other writes | Compacts natively written tables (seen on 4.1). The auto compaction tests that fail in the nightly job fail because its `spark.default.parallelism=1` makes their first write produce one file (from code). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). |
+| Files with deletion vectors | Fallback | Offload |
+| Change data feed, no deletion vectors in range | Offload. PartialOffload when the table allows deletion vectors and the range has `_change_data` files, because those scans then fall back (from code). | Offload |
+| Change data feed, deletion vectors in range | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors and no `_change_data` files (from code) | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors (from code) |
+
+```sql
+SELECT count(1) AS count, name
+FROM t
+GROUP BY name;
+```
+
+Reads need no Gluten option. With `spark.sql.adaptive.enabled=false`, `EXPLAIN` shows the native scan as `FileDeltaScanTransformer` and `EXPLAIN FORMATTED` shows `DeltaScanTransformer`. With AQE on, the default, `EXPLAIN` and `EXPLAIN FORMATTED` of a query with a shuffle, such as this one, print the plan from before Gluten offloads it. The Spark UI plan graph of a finished query shows `DeltaScanTransformer`. Delta prunes partitions and skips files by statistics while it plans the query, before the scan runs.
+
+| Read | Status | Notes |
+|---|---|---|
+| By table name, ``delta.`/path` `` or `spark.read.format("delta").load(path)` | Offload | Same native scan. |
+| `VERSION AS OF` | Offload | Tested on Spark 3.4 to 4.1. |
+| `TIMESTAMP AS OF`, `versionAsOf`, `timestampAsOf` | Offload | From code. Delta Lake's `DeltaTimeTravelSuite` passes (seen on 4.1). |
+| Streaming source | Offload | Each micro-batch is a Delta file scan. From code. Delta Lake's `DeltaSourceSuite` passes (seen on 4.1). |
+| Dynamic partition pruning | Not tested | |
+
+### Deletion vectors
+Offload
+
+```sql
+OPTIMIZE t;
+ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
+DELETE FROM t WHERE id IN (3, 4);
+SELECT * FROM t;
+```
+
+On Spark 3.5 to 4.1, Velox drops the deleted rows in the scan. This needs Delta's `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default. The DML target scan, which finds the rows for new deletion vectors, also runs in Velox. Gluten tests check the scan on Spark 3.5 to 4.1 and the DML target scan on Spark 3.5 and 4.0. Delta writes the deletion-vector files.
+
+Fallback
+
+```sql
+SET spark.databricks.delta.deletionVectors.useMetadataRowIndex=false;
+SELECT * FROM t;
+RESET spark.databricks.delta.deletionVectors.useMetadataRowIndex;
+```
+
+| Case | Behavior |
+|---|---|
+| Spark 3.5 to 4.1 with `useMetadataRowIndex=false` | A scan that reads Delta's generated `__delta_internal_is_row_deleted` or `__delta_internal_row_index` column falls back with `fallback Delta scan requiring generated DV metadata`. Delta adds the first column when any file in the scanned table version, micro-batch, DML file set or `OPTIMIZE` bin has a deletion vector, so `OPTIMIZE` and `REORG TABLE ... APPLY (PURGE)` scans of those files also fall back (from code). It adds the second to DML target scans that write deletion vectors. Other scans offload. Tested on Spark 3.5 and 4.0. |
+| Spark 3.4, normal reads | A read falls back with `fallback Spark 3.4 Delta DV scan` when a file it selected has a deletion vector. Tested on Spark 3.4. A read whose selected files have none offloads (from code). |
+| Spark 3.4, other scans, such as DML target scans, `OPTIMIZE` scans, streaming micro-batches and change data feed scans of `_change_data` files | Fall back whenever the table protocol allows deletion vectors. Change data feed scans of added or removed files fall back only when a file they read has a deletion vector. From code. |
+| `spark.databricks.delta.stats.skipping=false` | Delta Lake itself rejects reads of a table with deletion vectors with `Cannot work with a non-pinned table snapshot of the TahoeFileIndex` (seen on 4.1). Before Delta Lake 4.1.0, the table feature alone triggers it (from code). |
+
+The fallback reasons show in [Gluten UI](GlutenUI.md).
+
+### Change data feed
+Offload
+
+```sql
+CREATE TABLE c (id INT, name STRING) USING delta
+ TBLPROPERTIES ('delta.enableChangeDataFeed' = true);
+INSERT INTO c VALUES (1, 'a'), (2, 'b');
+UPDATE c SET name = 'bb' WHERE id = 2;
+SELECT id, name, _change_type, _commit_version
+FROM table_changes('c', 1)
+WHERE id = 2;
+```
+
+Gluten expands a batch change data feed read into Delta file scans that run in Velox with the filters pushed down. Delta also adds an empty placeholder scan to the plan. That scan falls back when the table schema has a MAP at any nesting level (from code). If the expansion fails, Gluten logs a warning and the read uses Delta's change data feed scan. From code.
+
+PartialOffload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false;
+SELECT * FROM table_changes('c', 1);
+```
+
+A read whose version range adds or removes a file with a deletion vector uses Delta's change data feed scan. Insert-only ranges of a table with deletion vectors enabled still run in Velox. With `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false`, every batch change data feed read uses Delta's scan. Delta's scan reads the data files in a query of its own that Gluten also plans, and `EXPLAIN` does not show it (from code). That query's file scans can run in Velox, except scans of added or removed files that have deletion vectors and, on Spark 3.4, scans of `_change_data` files when the table allows deletion vectors (from code). Streaming change data feed reads ignore this option. They run as Delta file scans. A scan whose files have deletion vectors falls back, and on Spark 3.4 so does a scan of `_change_data` files when the table allows deletion vectors (from code).
+
+### Read metadata
+Offload
+
+```sql
+SELECT id, _metadata.file_path, _metadata.row_index
+FROM t
+WHERE _metadata.row_index = 2;
+```
+
+The native scan fills the `_metadata` file fields and `input_file_name()` (from code), and `_metadata.row_index`. On Spark 3.5 to 4.1, Delta Lake exposes `_metadata.row_index` only on tables that allow deletion vectors or have row tracking, so the example needs `t` from [Deletion vectors](#deletion-vectors) (from code). A query that selects only `_metadata.file_name` can fail when the table has a BIGINT data column named `file_name` (seen on 4.1, see [Read problems](#read-problems)). Gluten tests check `_metadata.row_index` in the native scan on Spark 3.5 and 4.0, and Delta Lake tests that read it pass (seen on 4.1). Spark 3.4 is not tested. On row-tracking tables, `_metadata.row_id` and `_metadata.row_commit_version` are read natively (from code). In the nightly job, they return correct values except in the cases in [Read problems](#read-problems) (seen on 4.1).
+
+Fallback
+
+```sql
+DESCRIBE HISTORY t;
+```
+
+`DESCRIBE HISTORY` reads each commit file on Spark, through Delta's LogStore in a `mapPartitions` job. When Delta replays the log for a snapshot, every scan under `_delta_log`, checkpoints included, stays on Spark with the reason `fallback Delta _delta_log scan`. From code.
+
+## Table features
+Offload/Fallback
+
+The Read column covers Spark 3.5 to 4.1. It covers Spark 3.4 only where the row names it. The Native write column applies to Spark 3.5 to 4.1 with native write on.
+
+| Feature | Read | Native write | Notes |
+|---|---|---|---|
+| Deletion vectors | Offload, Fallback on Spark 3.4 | PartialOffload | Gluten's tests check deletion vector DELETE and MERGE results with native write on, on Spark 3.5 and 4.0. Which writer writes each file is from code. See [Deletion vectors](#deletion-vectors). |
+| Change data feed | Offload, also on Spark 3.4. Ranges with deletion vectors fall back. On Spark 3.4, scans of `_change_data` files also fall back when the table allows deletion vectors (from code). | Offload | UPDATE and DELETE write the `_change_data` files with Velox (from code). Change data feed reads of their changes are correct (seen on 4.1). See [Change data feed](#change-data-feed). |
+| Column mapping, name mode | Offload, also on Spark 3.4 | ResultMismatch | Files have no Parquet field IDs. Seen on 4.1. See [Write problems](#write-problems). |
+| Column mapping, id mode | Offload, also on Spark 3.4. ResultMismatch when field IDs differ from physical names (seen on 4.1). | ResultMismatch, Exception when Spark's reader reads the files (seen on 4.1) | Gluten matches columns by physical name. See [Known problems](#known-problems). |
+| Identity columns | Offload | Offload | Seen on 4.1. High-water marks are correct. The projection that generates identity values runs on Spark (from code). |
+| Generated columns | Offload | Offload | Seen on 4.1. No Gluten test covers generated columns. Velox computes the generated values when it supports the expression (from code). Partition filters derived from them reach the native scan. `DATE_FORMAT` results for years above 9999 differ (see [Write problems](#write-problems)). |
+| CHECK constraints, NOT NULL | Offload | Offload | Delta checks the rows on the JVM before Velox writes them. Gluten's tests cover NOT NULL on Spark 3.5 and 4.0. Seen on 4.1. |
+| Row tracking | Offload, Exception for `_metadata.base_row_id` filters, ResultMismatch for `_metadata.row_id` when older files lack base row IDs | Offload | Seen on 4.1. See [Read problems](#read-problems). |
+| Liquid clustering | Offload | PartialOffload | Reads use the same native scan as other tables. Native commands write to clustered tables. `OPTIMIZE`, which clusters the data, uses Delta's writer. All three are from code. |
+| UniForm (IcebergCompatV1, V2) | | ResultMismatch, Exception when partitioned | The write status is from code. No test reads a UniForm table. |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Offload, Exception for FLOAT or DOUBLE partition columns, for TIMESTAMP_NTZ partition values with sub-millisecond digits and for some writes of TIMESTAMP_NTZ data columns | Seen on 4.1. Spark 3.5 and 4.0 are not covered. |
+| TIMESTAMP_NTZ | PartialOffload, also on Spark 3.4 | Offload, Exception for some writes (seen on 4.1) | See [DataType](#datatype). |
+| Variant, including shredding | Fallback | Fallback | Spark 4.x. From code. See [DataType](#datatype). |
+| Collations | Fallback | | Spark 4.x. From code. See [DataType](#datatype). |
+| materializePartitionColumns | | ResultMismatch | Spark 4.1. From code. |
+
+## DataType
+
+The Read column applies to Spark 3.4 to 4.1. On all of them, Gluten's tests read INT, BIGINT, DECIMAL(12,2), STRING, DATE, TIMESTAMP, TIMESTAMP_NTZ, STRUCT, ARRAY and MAP columns natively. The other read statuses are from code, unless a row says otherwise. The native write columns apply to Spark 3.5 to 4.1 with native write on. On Spark 3.5 and 4.0, Gluten's tests write BOOLEAN, INT, BIGINT, STRING, STRUCT and ARRAY columns natively. The other native write statuses are seen on 4.1, unless a row says they are from code.
+
+| Type | Read | Native write | Native write as partition column | Notes |
|---|---|---|---|---|
-| `spark-3.5` | Spark 3.5.x | 2.12 | 3.3.x | Supported |
-| `spark-4.0` | Spark 4.0.x | 2.13 | 4.0.x | Supported |
-
-Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The difference between
-the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta 3.3 vs Spark 4.0 +
-Delta 4.0), not a native-write capability gap.
-
-## Build and runtime notes
-
-Build Gluten with Delta support by enabling `-Pdelta` together with the Velox backend profile and a Spark profile.
-
-- Spark 3.5 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests`
-- Spark 4.0 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13 -Pjava-17 -DskipTests`
-
-Native Delta write is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite`
- - Default: `false`
- - Type: experimental
-
-Native change data feed scan offload is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan`
- - Default: `true`
- - Type: experimental
-
-The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan`
- - Default: `true`
- - Type: experimental
- - When disabled, the DML target scan that produces file paths and row indexes for
- deletion-vector writes stays on Spark; other scans are unaffected.
-
-| Feature | Delta minWriterVersion | Delta minReaderVersion | Iceberg format-version | Feature type | Supported by Gluten (Velox) |
-|---|---:|---:|---:|---|---|
-| Basic functionality | 2 | 1 | 1 | Writer | Yes |
-| CHECK constraints | 3 | 1 | N/A | Writer | No |
-| Change data feed | 4 | 1 | N/A | Writer | Yes |
-| Generated columns | 4 | 1 | N/A | Writer | Partial |
-| Column mapping | 5 | 2 | N/A | Reader and writer | Yes |
-| Identity columns | 6 | 1 | N/A | Writer | Yes |
-| Row tracking | 7 | 1 | 3 | Writer | Partial |
-| Deletion vectors | 7 | 3 | 3 | Reader and writer | Partial |
-| TimestampNTZ | 7 | 3 | 1 | Reader and writer | No |
-| Liquid clustering | 7 | 3 | 1 | Reader and writer | Yes |
-| Iceberg readers (UniForm) | 7 | 2 | N/A | Writer | Not tested |
-| Type widening | 7 | 3 | N/A | Reader and writer | Partial |
-| Variant | 7 | 3 | 3 | Reader and writer | Not tested |
-| Variant shredding | 7 | 3 | 3 | Reader and writer | Not tested |
-| Collations | 7 | 3 | N/A | Reader and writer | Not tested |
-| Protected checkpoints | 7 | 1 | N/A | Writer | Not tested |
+| BOOLEAN | Offload | Offload | Offload | |
+| TINYINT, SMALLINT, INT, BIGINT | Offload | Offload | Offload | |
+| FLOAT, DOUBLE | Offload | Offload | Exception | See [Write problems](#write-problems). |
+| DECIMAL | Offload | Offload | Offload | Precision above 18 included. |
+| STRING | Offload | Offload | Offload | |
+| CHAR, VARCHAR | Offload | Offload | | Stored as STRING. Delta checks the length on the JVM before the native write. Both statuses are from code. Reads return correct results (seen on 4.1). |
+| BINARY | Offload | Offload | Not tested | The read status is from code. One Delta Lake test writes a table with only a BINARY column, checks its file statistics and runs a DELETE that filters on that column (seen on 4.1). |
+| DATE | Offload | Offload | Offload | |
+| TIMESTAMP | Offload, ResultMismatch for sub-millisecond partition values (seen on 4.1) | Offload | Offload, Exception for sub-millisecond values | The partition column status is from code. See [Known problems](#known-problems). |
+| TIMESTAMP_NTZ | PartialOffload | Offload, Exception for some writes (seen on 4.1) | Offload, Exception for sub-millisecond values (seen on 4.1) | Filters, sorts, joins and other operators whose input or output includes a TIMESTAMP_NTZ column fall back, even when they do not use that column. Scans, aggregates, shuffles and some projections can run in Velox. A scan whose pushed filter has a TIMESTAMP_NTZ literal falls back (from code). Gluten's tests check on Spark 3.4 to 4.1 that a scan of the column runs in Velox, and that a filter on the column and a sort on another column cause a fallback. The other operator rules are from code. |
+| STRUCT, ARRAY, MAP | Offload | Offload | | |
+| VARIANT | Fallback | Fallback | | Spark 4.x. Shredded variant reads fall back too. Both statuses are from code. |
+| Collated STRING | Fallback | | | Spark 4.x. Scans that read the column fall back. Delta Lake 4.0.1 and 4.1.0 reject a collated column in a table schema with `DELTA_UNSUPPORTED_DATA_TYPES`. They cannot write to tables that have the `collations` feature. From code. |
+
+A plan node falls back when its output contains a type Velox does not support, such as VARIANT. With the default `spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames=true`, a scan that does not select such a column can still offload. From code. Delta Lake does not allow interval columns.
+
+## Format
+PartialOffload
+
+Delta data files are Parquet, and Gluten reads them with Velox's Parquet reader. Velox writes them only for native write commands on Spark 3.5 to 4.1. Otherwise Delta's Parquet writer writes them.
+
+### Natively written files
+
+| Property | Behavior |
+|---|---|
+| Compression | snappy (default), gzip, zstd (always level 3), lz4 (Hadoop framing), none. From code. Other codecs: see [Write problems](#write-problems). |
+| Row groups | Sized by `spark.gluten.sql.columnar.parquet.write.blockSize` and `spark.gluten.sql.native.parquet.write.blockRows`. `parquet.*` settings are ignored. From code. |
+| TIMESTAMP | INT64 microseconds. `spark.sql.parquet.outputTimestampType` is ignored. Delta Lake's writer uses INT96 by default. Spark reads both. From code. |
+| Parquet field IDs | Not written, for column-mapped and UniForm tables too. Seen on 4.1 for column mapping, from code for UniForm. See [Write problems](#write-problems). |
+| Spark schema footer entry | `org.apache.spark.sql.parquet.row.metadata` is not written ([#13288](https://github.com/apache/gluten/issues/13288)). Seen on 4.1. |
+| Nested field names | Lowercased below the top level when `spark.sql.caseSensitive=false`. Spark and Gluten match them case-insensitively. A case-sensitive reader can miss them. From code. |
+| Page index | Always written. From code. |
+| `spark.sql.parquet.writeLegacyFormat=true` | Applies to DECIMAL only. Arrays and maps keep the standard layout. From code. |
+| File statistics | numRecords, min, max and nullCount are computed in Velox (checked on Spark 3.5). Gluten's tests check the values against the rows in each file on Spark 3.5 to 4.1. Most tests in Delta's data-skipping suites pass (seen on 4.1). Of the 34 that fail, 22 check statistics and expect specific rows in each file after `repartition(2)`. The other 12 hit the `_metadata.file_name` failure in [Read problems](#read-problems). |
+| Data file paths | Come from Delta's committer, so names and random prefixes follow Delta. File names seen on 4.1, random prefixes from code. |
+
+## SQL
+
+| Statement | Status | Notes |
+|---|---|---|
+| `SELECT` | Offload | See [Reading](#reading). |
+| `table_changes()` | Offload | Ranges with deletion vectors fall back. See [Change data feed](#change-data-feed). |
+| `CREATE TABLE ... AS SELECT`, `REPLACE TABLE ... AS SELECT` | Offload | With native write on, Spark 3.5 to 4.1, and a DeltaCatalog target. Seen on 4.1, from code on 3.5 and 4.0. See [Creating a table](#creating-a-table). |
+| `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `OPTIMIZE`, `REORG` | | See [Write commands](#write-commands). |
+| `CREATE TABLE` without `AS SELECT`, `DROP TABLE`, `DESCRIBE` | Fallback | They read and write no data files. From code. |
+| `ALTER TABLE` | Fallback, PartialOffload when it reads data | Most forms read and write no data files. `ADD CONSTRAINT` scans the table, and that scan can run in Velox. On Spark 3.5 to 4.1, `ALTER COLUMN ... SYNC IDENTITY` also scans it, and `DROP FEATURE` and `SET TBLPROPERTIES ('delta.columnMapping.mode' = 'none')` can rewrite data files with Delta's writer. From code. |
+| `VACUUM`, `RESTORE`, `SHALLOW CLONE` | | They write no Parquet data files. From code. |
+| `CONVERT TO DELTA` | PartialOffload | Writes no Parquet data files. Unless the statement has `NO STATISTICS` or `spark.databricks.delta.stats.collect=false` is set, it scans the Parquet files for statistics. That scan can run in Velox. From code. |
+
+## Schema evolution
+PartialOffload
+
+The `mergeSchema` and `overwriteSchema` rows need native write on Spark 3.5 to 4.1. Otherwise Delta writes the files.
+
+| Change | Status | Notes |
+|---|---|---|
+| `mergeSchema` with `save(path)` | Offload | Velox writes the evolved schema (from code). On Spark 3.5 and 4.0, the copied `DeltaSuite` checks the results of a `mergeSchema` overwrite. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). |
+| `overwriteSchema` with `save(path)` | Offload | Delta applies the change before the write. From code. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). |
+| Schema evolution in SQL `INSERT` | PartialOffload | From code. |
+| Schema evolution in `MERGE` | PartialOffload, Exception for some STRUCT map key changes | The merged table contents match Delta Lake. Some STRUCT map key changes fail in Velox (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). The two TIMESTAMP_NTZ widening tests fail before the MERGE runs, in a setup write of TIMESTAMP_NTZ values. Seen on 4.1. |
+| `RENAME COLUMN` with column mapping | Offload | Reads after top-level and nested renames return correct results. After a top-level rename, partition pruning and file skipping still work. |
+| `DROP COLUMN` with column mapping | Offload | From code. Delta Lake's `DeltaDropColumnSuite` passes (seen on 4.1). |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ and for native writes to FLOAT or DOUBLE partition columns | Seen on 4.1. See [Table features](#table-features). |
+| `spark.sql.parquet.mergeSchema=true` | Fallback | Every Parquet scan falls back, Delta scans included. From code. |
Review Comment:
it seems mergeScheme is support already in write section
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,632 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+Gluten's Velox backend can accelerate eligible scans and query operators over Delta Lake tables. Read
+acceleration does not require native writes. Native Delta writes are experimental and disabled by
+default. This guide covers the supported Spark and Delta Lake versions, setup, feature support and
+known limitations.
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+## How a Delta query runs
+
+Take a Delta table partitioned by `sale_date`:
+
+```sql
+SELECT SUM(amount)
+FROM sales
+WHERE sale_date = DATE '2026-10-01'
+ AND amount > 100;
+```
+
+```mermaid
+flowchart LR
+ subgraph Planning["Spark driver"]
+ Snapshot["1. Delta snapshot<br/>Schema + active files"]
+ Select["2. Delta file pruning<br/>Partitions + file statistics"]
+ Plan["3. Spark plan<br/>Gluten offload checks"]
+ Snapshot --> Select --> Plan
+ end
+
+ subgraph Execution["Spark executors"]
+ Scan["4. Velox scan<br/>Reads the selected files"]
+ Compute["5. Velox operators<br/>Filter + SUM"]
+ Spark["Spark operators<br/>Operators Gluten does not offload"]
+ Scan --> Compute
+ Compute <-->|"Row and columnar conversions"| Spark
+ end
+
+ subgraph Storage["Delta table"]
+ Log[("_delta_log<br/>Metadata + file statistics")]
+ Data[("Parquet files")]
+ end
+
+ Result["6. Query result"]
+
+ Log --> Snapshot
+ Plan -->|"Offloaded operators"| Scan
+ Plan -->|"Other operators"| Spark
+ Data --> Scan
+ Data --> Spark
+ Compute --> Result
+ Spark --> Result
+```
+
+Delta Lake loads the snapshot and prunes partitions and files while Spark plans the query. Gluten then
+offloads the operators Velox supports, and the rest run on Spark, so one query can mix Velox and Spark
+operators. Delta Lake keeps the table metadata, the transaction log and commits.
+
+## Supported Spark and Delta Lake versions
+
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. On Spark 4.0 and 4.1, the Read and Native write columns assume ANSI mode off (see [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x)). The copied Delta Lake suites run with native write on and `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`, which is off by default.
+
+Spark 4.1 has the most tests, because the nightly job runs on it. That job builds Delta Lake 4.2.0 from source against Spark 4.1.0, which is newer than the Delta Lake 4.1.0 in the table, and runs its Spark test suites. It puts a Gluten bundle compiled against Spark 4.1.1 and Delta Lake 4.1.0 on the test classpath. It replaces Delta Lake's `DeltaSQLCommandTest` with Gluten's copy, which loads Gluten with native write on and the non-default `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`. It then runs `.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh`. That script cherry-picks two Delta Lake test changes that are not in 4.2.0, delta-io/delta#7104 and #7105, so Delta Lake's scan checks accept Gluten's scan node. It also sets a row-group row limit in two test fixtures. The same job also runs on pull requests that change Gluten's Delta code. "Seen on 4.1" on this page refers to that job.
+
+The job's known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector, change data feed, column mapping, time travel and partition filter reads, and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+```bash
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+```
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. `dev/release/build-release.sh` passes `-Pdelta` only for Spark 3.4 and 3.5, so its Spark 4.0 and 4.1 bundles have no Delta Lake support either. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add Delta Lake with `--packages` and the coordinate from the table above, which also adds its `delta-storage` dependency. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off. From code.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+```bash
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+```
+
+On Spark 4.0 or 4.1, use the Gluten bundle and the Delta Lake jar for that version, and add `--conf spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code. Turning ANSI mode off also changes Spark's own behavior for invalid input. For example, `CAST('a' AS INT)` returns NULL instead of failing.
+
+```properties
+spark.sql.ansi.enabled=false
+```
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. On Spark 3.x, turning ANSI mode on makes every query fall back in the same way. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Gluten passes ANSI mode to Velox. Velox does not implement ANSI behavior for every expression, so some invalid input returns a result where Spark fails ([#10134](https://github.com/apache/gluten/issues/10134)). For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). Cast overflow errors raised in Velox have a different exception type (see [Error types](#error-types)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description |
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back. |
+| Fallback | Runs on Spark or Delta Lake. |
+| Exception | Fails instead of falling back. |
+| ResultMismatch | Can return wrong results or write wrong files. |
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI checks it.<br>
+Seen on 4.1: the nightly Delta Lake job showed the result, with Delta Lake 4.2.0 on Spark 4.1.0.<br>
+Reported on 3.4, Reported on 4.1: the linked issue reports the result on that Spark version. No CI test checks it.<br>
+No label: a test in the Gluten repository that runs in CI checks the status on the stated versions.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver. From code.
+
+## Creating a table
+Fallback
+
+```sql
+CREATE TABLE t (id INT, name STRING) USING delta;
+```
+
+A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload. From code.
+
+Offload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+```
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+## Writing
+PartialOffload
+
+```sql
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+```
+
+Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code.
+
+Fallback
+
+```sql
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+```
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+```
+
+On Spark 3.5 to 4.1, set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). Velox writes the files of this UPDATE. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+```scala
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+```
+
+The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the command's plan from the Spark UI SQL tab or a `QueryExecutionListener`. Each write also reports a nested `deltaTransactionalWrite` execution for the query that feeds it. That plan never has the prefix, whichever writer ran. From code.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks that a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5 run as Gluten native write commands. It also checks on 3.5 and 4.0 that a DELETE that rewrites a file runs in Gluten's transaction and that its scan runs in Velox. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload, PartialOffload when it writes deletion vectors (from code on 4.1), Fallback when it is metadata-only (from code) | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. |
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | `DeltaTable.optimize().executeCompaction()`, with or without `where(...)` | Offload | Bin-packing, by path or by name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans, the DML target scan and joins can run in Velox. On Spark 3.5 and 4.0, Gluten's tests check that a deletion-vector MERGE runs a scan of the target table in Velox. The other cases are from code. Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake in `DescribeDeltaHistorySuite`. Seen on 4.1. Three unpartitioned tests in `DeleteMetricsSuite` and `UpdateMetricsSuite` expect two added files and get one. The job sets `spark.default.parallelism=1`, which plans the five small input files as one scan partition (from code).
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload after native writes, PartialOffload after other writes | Compacts natively written tables (seen on 4.1). The auto compaction tests that fail in the nightly job fail because its `spark.default.parallelism=1` makes their first write produce one file (from code). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). |
+| Files with deletion vectors | Fallback | Offload |
+| Change data feed, no deletion vectors in range | Offload. PartialOffload when the table allows deletion vectors and the range has `_change_data` files, because those scans then fall back (from code). | Offload |
+| Change data feed, deletion vectors in range | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors and no `_change_data` files (from code) | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors (from code) |
+
+```sql
+SELECT count(1) AS count, name
+FROM t
+GROUP BY name;
+```
+
+Reads need no Gluten option. With `spark.sql.adaptive.enabled=false`, `EXPLAIN` shows the native scan as `FileDeltaScanTransformer` and `EXPLAIN FORMATTED` shows `DeltaScanTransformer`. With AQE on, the default, `EXPLAIN` and `EXPLAIN FORMATTED` of a query with a shuffle, such as this one, print the plan from before Gluten offloads it. The Spark UI plan graph of a finished query shows `DeltaScanTransformer`. Delta prunes partitions and skips files by statistics while it plans the query, before the scan runs.
+
+| Read | Status | Notes |
+|---|---|---|
+| By table name, ``delta.`/path` `` or `spark.read.format("delta").load(path)` | Offload | Same native scan. |
+| `VERSION AS OF` | Offload | Tested on Spark 3.4 to 4.1. |
+| `TIMESTAMP AS OF`, `versionAsOf`, `timestampAsOf` | Offload | From code. Delta Lake's `DeltaTimeTravelSuite` passes (seen on 4.1). |
+| Streaming source | Offload | Each micro-batch is a Delta file scan. From code. Delta Lake's `DeltaSourceSuite` passes (seen on 4.1). |
+| Dynamic partition pruning | Not tested | |
+
+### Deletion vectors
+Offload
+
+```sql
+OPTIMIZE t;
+ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
+DELETE FROM t WHERE id IN (3, 4);
+SELECT * FROM t;
+```
+
+On Spark 3.5 to 4.1, Velox drops the deleted rows in the scan. This needs Delta's `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default. The DML target scan, which finds the rows for new deletion vectors, also runs in Velox. Gluten tests check the scan on Spark 3.5 to 4.1 and the DML target scan on Spark 3.5 and 4.0. Delta writes the deletion-vector files.
+
+Fallback
+
+```sql
+SET spark.databricks.delta.deletionVectors.useMetadataRowIndex=false;
+SELECT * FROM t;
+RESET spark.databricks.delta.deletionVectors.useMetadataRowIndex;
+```
+
+| Case | Behavior |
+|---|---|
+| Spark 3.5 to 4.1 with `useMetadataRowIndex=false` | A scan that reads Delta's generated `__delta_internal_is_row_deleted` or `__delta_internal_row_index` column falls back with `fallback Delta scan requiring generated DV metadata`. Delta adds the first column when any file in the scanned table version, micro-batch, DML file set or `OPTIMIZE` bin has a deletion vector, so `OPTIMIZE` and `REORG TABLE ... APPLY (PURGE)` scans of those files also fall back (from code). It adds the second to DML target scans that write deletion vectors. Other scans offload. Tested on Spark 3.5 and 4.0. |
+| Spark 3.4, normal reads | A read falls back with `fallback Spark 3.4 Delta DV scan` when a file it selected has a deletion vector. Tested on Spark 3.4. A read whose selected files have none offloads (from code). |
+| Spark 3.4, other scans, such as DML target scans, `OPTIMIZE` scans, streaming micro-batches and change data feed scans of `_change_data` files | Fall back whenever the table protocol allows deletion vectors. Change data feed scans of added or removed files fall back only when a file they read has a deletion vector. From code. |
+| `spark.databricks.delta.stats.skipping=false` | Delta Lake itself rejects reads of a table with deletion vectors with `Cannot work with a non-pinned table snapshot of the TahoeFileIndex` (seen on 4.1). Before Delta Lake 4.1.0, the table feature alone triggers it (from code). |
+
+The fallback reasons show in [Gluten UI](GlutenUI.md).
+
+### Change data feed
+Offload
+
+```sql
+CREATE TABLE c (id INT, name STRING) USING delta
+ TBLPROPERTIES ('delta.enableChangeDataFeed' = true);
+INSERT INTO c VALUES (1, 'a'), (2, 'b');
+UPDATE c SET name = 'bb' WHERE id = 2;
+SELECT id, name, _change_type, _commit_version
+FROM table_changes('c', 1)
+WHERE id = 2;
+```
+
+Gluten expands a batch change data feed read into Delta file scans that run in Velox with the filters pushed down. Delta also adds an empty placeholder scan to the plan. That scan falls back when the table schema has a MAP at any nesting level (from code). If the expansion fails, Gluten logs a warning and the read uses Delta's change data feed scan. From code.
+
+PartialOffload
+
+```sql
+SET spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false;
+SELECT * FROM table_changes('c', 1);
+```
+
+A read whose version range adds or removes a file with a deletion vector uses Delta's change data feed scan. Insert-only ranges of a table with deletion vectors enabled still run in Velox. With `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false`, every batch change data feed read uses Delta's scan. Delta's scan reads the data files in a query of its own that Gluten also plans, and `EXPLAIN` does not show it (from code). That query's file scans can run in Velox, except scans of added or removed files that have deletion vectors and, on Spark 3.4, scans of `_change_data` files when the table allows deletion vectors (from code). Streaming change data feed reads ignore this option. They run as Delta file scans. A scan whose files have deletion vectors falls back, and on Spark 3.4 so does a scan of `_change_data` files when the table allows deletion vectors (from code).
+
+### Read metadata
+Offload
+
+```sql
+SELECT id, _metadata.file_path, _metadata.row_index
+FROM t
+WHERE _metadata.row_index = 2;
+```
+
+The native scan fills the `_metadata` file fields and `input_file_name()` (from code), and `_metadata.row_index`. On Spark 3.5 to 4.1, Delta Lake exposes `_metadata.row_index` only on tables that allow deletion vectors or have row tracking, so the example needs `t` from [Deletion vectors](#deletion-vectors) (from code). A query that selects only `_metadata.file_name` can fail when the table has a BIGINT data column named `file_name` (seen on 4.1, see [Read problems](#read-problems)). Gluten tests check `_metadata.row_index` in the native scan on Spark 3.5 and 4.0, and Delta Lake tests that read it pass (seen on 4.1). Spark 3.4 is not tested. On row-tracking tables, `_metadata.row_id` and `_metadata.row_commit_version` are read natively (from code). In the nightly job, they return correct values except in the cases in [Read problems](#read-problems) (seen on 4.1).
+
+Fallback
+
+```sql
+DESCRIBE HISTORY t;
+```
+
+`DESCRIBE HISTORY` reads each commit file on Spark, through Delta's LogStore in a `mapPartitions` job. When Delta replays the log for a snapshot, every scan under `_delta_log`, checkpoints included, stays on Spark with the reason `fallback Delta _delta_log scan`. From code.
+
+## Table features
+Offload/Fallback
+
+The Read column covers Spark 3.5 to 4.1. It covers Spark 3.4 only where the row names it. The Native write column applies to Spark 3.5 to 4.1 with native write on.
+
+| Feature | Read | Native write | Notes |
+|---|---|---|---|
+| Deletion vectors | Offload, Fallback on Spark 3.4 | PartialOffload | Gluten's tests check deletion vector DELETE and MERGE results with native write on, on Spark 3.5 and 4.0. Which writer writes each file is from code. See [Deletion vectors](#deletion-vectors). |
+| Change data feed | Offload, also on Spark 3.4. Ranges with deletion vectors fall back. On Spark 3.4, scans of `_change_data` files also fall back when the table allows deletion vectors (from code). | Offload | UPDATE and DELETE write the `_change_data` files with Velox (from code). Change data feed reads of their changes are correct (seen on 4.1). See [Change data feed](#change-data-feed). |
+| Column mapping, name mode | Offload, also on Spark 3.4 | ResultMismatch | Files have no Parquet field IDs. Seen on 4.1. See [Write problems](#write-problems). |
+| Column mapping, id mode | Offload, also on Spark 3.4. ResultMismatch when field IDs differ from physical names (seen on 4.1). | ResultMismatch, Exception when Spark's reader reads the files (seen on 4.1) | Gluten matches columns by physical name. See [Known problems](#known-problems). |
+| Identity columns | Offload | Offload | Seen on 4.1. High-water marks are correct. The projection that generates identity values runs on Spark (from code). |
+| Generated columns | Offload | Offload | Seen on 4.1. No Gluten test covers generated columns. Velox computes the generated values when it supports the expression (from code). Partition filters derived from them reach the native scan. `DATE_FORMAT` results for years above 9999 differ (see [Write problems](#write-problems)). |
+| CHECK constraints, NOT NULL | Offload | Offload | Delta checks the rows on the JVM before Velox writes them. Gluten's tests cover NOT NULL on Spark 3.5 and 4.0. Seen on 4.1. |
+| Row tracking | Offload, Exception for `_metadata.base_row_id` filters, ResultMismatch for `_metadata.row_id` when older files lack base row IDs | Offload | Seen on 4.1. See [Read problems](#read-problems). |
+| Liquid clustering | Offload | PartialOffload | Reads use the same native scan as other tables. Native commands write to clustered tables. `OPTIMIZE`, which clusters the data, uses Delta's writer. All three are from code. |
+| UniForm (IcebergCompatV1, V2) | | ResultMismatch, Exception when partitioned | The write status is from code. No test reads a UniForm table. |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Offload, Exception for FLOAT or DOUBLE partition columns, for TIMESTAMP_NTZ partition values with sub-millisecond digits and for some writes of TIMESTAMP_NTZ data columns | Seen on 4.1. Spark 3.5 and 4.0 are not covered. |
+| TIMESTAMP_NTZ | PartialOffload, also on Spark 3.4 | Offload, Exception for some writes (seen on 4.1) | See [DataType](#datatype). |
+| Variant, including shredding | Fallback | Fallback | Spark 4.x. From code. See [DataType](#datatype). |
+| Collations | Fallback | | Spark 4.x. From code. See [DataType](#datatype). |
+| materializePartitionColumns | | ResultMismatch | Spark 4.1. From code. |
+
+## DataType
+
+The Read column applies to Spark 3.4 to 4.1. On all of them, Gluten's tests read INT, BIGINT, DECIMAL(12,2), STRING, DATE, TIMESTAMP, TIMESTAMP_NTZ, STRUCT, ARRAY and MAP columns natively. The other read statuses are from code, unless a row says otherwise. The native write columns apply to Spark 3.5 to 4.1 with native write on. On Spark 3.5 and 4.0, Gluten's tests write BOOLEAN, INT, BIGINT, STRING, STRUCT and ARRAY columns natively. The other native write statuses are seen on 4.1, unless a row says they are from code.
+
+| Type | Read | Native write | Native write as partition column | Notes |
|---|---|---|---|---|
-| `spark-3.5` | Spark 3.5.x | 2.12 | 3.3.x | Supported |
-| `spark-4.0` | Spark 4.0.x | 2.13 | 4.0.x | Supported |
-
-Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The difference between
-the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta 3.3 vs Spark 4.0 +
-Delta 4.0), not a native-write capability gap.
-
-## Build and runtime notes
-
-Build Gluten with Delta support by enabling `-Pdelta` together with the Velox backend profile and a Spark profile.
-
-- Spark 3.5 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests`
-- Spark 4.0 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13 -Pjava-17 -DskipTests`
-
-Native Delta write is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite`
- - Default: `false`
- - Type: experimental
-
-Native change data feed scan offload is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan`
- - Default: `true`
- - Type: experimental
-
-The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan`
- - Default: `true`
- - Type: experimental
- - When disabled, the DML target scan that produces file paths and row indexes for
- deletion-vector writes stays on Spark; other scans are unaffected.
-
-| Feature | Delta minWriterVersion | Delta minReaderVersion | Iceberg format-version | Feature type | Supported by Gluten (Velox) |
-|---|---:|---:|---:|---|---|
-| Basic functionality | 2 | 1 | 1 | Writer | Yes |
-| CHECK constraints | 3 | 1 | N/A | Writer | No |
-| Change data feed | 4 | 1 | N/A | Writer | Yes |
-| Generated columns | 4 | 1 | N/A | Writer | Partial |
-| Column mapping | 5 | 2 | N/A | Reader and writer | Yes |
-| Identity columns | 6 | 1 | N/A | Writer | Yes |
-| Row tracking | 7 | 1 | 3 | Writer | Partial |
-| Deletion vectors | 7 | 3 | 3 | Reader and writer | Partial |
-| TimestampNTZ | 7 | 3 | 1 | Reader and writer | No |
-| Liquid clustering | 7 | 3 | 1 | Reader and writer | Yes |
-| Iceberg readers (UniForm) | 7 | 2 | N/A | Writer | Not tested |
-| Type widening | 7 | 3 | N/A | Reader and writer | Partial |
-| Variant | 7 | 3 | 3 | Reader and writer | Not tested |
-| Variant shredding | 7 | 3 | 3 | Reader and writer | Not tested |
-| Collations | 7 | 3 | N/A | Reader and writer | Not tested |
-| Protected checkpoints | 7 | 1 | N/A | Writer | Not tested |
+| BOOLEAN | Offload | Offload | Offload | |
+| TINYINT, SMALLINT, INT, BIGINT | Offload | Offload | Offload | |
+| FLOAT, DOUBLE | Offload | Offload | Exception | See [Write problems](#write-problems). |
+| DECIMAL | Offload | Offload | Offload | Precision above 18 included. |
+| STRING | Offload | Offload | Offload | |
+| CHAR, VARCHAR | Offload | Offload | | Stored as STRING. Delta checks the length on the JVM before the native write. Both statuses are from code. Reads return correct results (seen on 4.1). |
+| BINARY | Offload | Offload | Not tested | The read status is from code. One Delta Lake test writes a table with only a BINARY column, checks its file statistics and runs a DELETE that filters on that column (seen on 4.1). |
+| DATE | Offload | Offload | Offload | |
+| TIMESTAMP | Offload, ResultMismatch for sub-millisecond partition values (seen on 4.1) | Offload | Offload, Exception for sub-millisecond values | The partition column status is from code. See [Known problems](#known-problems). |
+| TIMESTAMP_NTZ | PartialOffload | Offload, Exception for some writes (seen on 4.1) | Offload, Exception for sub-millisecond values (seen on 4.1) | Filters, sorts, joins and other operators whose input or output includes a TIMESTAMP_NTZ column fall back, even when they do not use that column. Scans, aggregates, shuffles and some projections can run in Velox. A scan whose pushed filter has a TIMESTAMP_NTZ literal falls back (from code). Gluten's tests check on Spark 3.4 to 4.1 that a scan of the column runs in Velox, and that a filter on the column and a sort on another column cause a fallback. The other operator rules are from code. |
+| STRUCT, ARRAY, MAP | Offload | Offload | | |
+| VARIANT | Fallback | Fallback | | Spark 4.x. Shredded variant reads fall back too. Both statuses are from code. |
+| Collated STRING | Fallback | | | Spark 4.x. Scans that read the column fall back. Delta Lake 4.0.1 and 4.1.0 reject a collated column in a table schema with `DELTA_UNSUPPORTED_DATA_TYPES`. They cannot write to tables that have the `collations` feature. From code. |
+
+A plan node falls back when its output contains a type Velox does not support, such as VARIANT. With the default `spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames=true`, a scan that does not select such a column can still offload. From code. Delta Lake does not allow interval columns.
+
+## Format
+PartialOffload
+
+Delta data files are Parquet, and Gluten reads them with Velox's Parquet reader. Velox writes them only for native write commands on Spark 3.5 to 4.1. Otherwise Delta's Parquet writer writes them.
+
+### Natively written files
+
+| Property | Behavior |
+|---|---|
+| Compression | snappy (default), gzip, zstd (always level 3), lz4 (Hadoop framing), none. From code. Other codecs: see [Write problems](#write-problems). |
+| Row groups | Sized by `spark.gluten.sql.columnar.parquet.write.blockSize` and `spark.gluten.sql.native.parquet.write.blockRows`. `parquet.*` settings are ignored. From code. |
+| TIMESTAMP | INT64 microseconds. `spark.sql.parquet.outputTimestampType` is ignored. Delta Lake's writer uses INT96 by default. Spark reads both. From code. |
+| Parquet field IDs | Not written, for column-mapped and UniForm tables too. Seen on 4.1 for column mapping, from code for UniForm. See [Write problems](#write-problems). |
+| Spark schema footer entry | `org.apache.spark.sql.parquet.row.metadata` is not written ([#13288](https://github.com/apache/gluten/issues/13288)). Seen on 4.1. |
+| Nested field names | Lowercased below the top level when `spark.sql.caseSensitive=false`. Spark and Gluten match them case-insensitively. A case-sensitive reader can miss them. From code. |
+| Page index | Always written. From code. |
+| `spark.sql.parquet.writeLegacyFormat=true` | Applies to DECIMAL only. Arrays and maps keep the standard layout. From code. |
+| File statistics | numRecords, min, max and nullCount are computed in Velox (checked on Spark 3.5). Gluten's tests check the values against the rows in each file on Spark 3.5 to 4.1. Most tests in Delta's data-skipping suites pass (seen on 4.1). Of the 34 that fail, 22 check statistics and expect specific rows in each file after `repartition(2)`. The other 12 hit the `_metadata.file_name` failure in [Read problems](#read-problems). |
+| Data file paths | Come from Delta's committer, so names and random prefixes follow Delta. File names seen on 4.1, random prefixes from code. |
+
+## SQL
+
+| Statement | Status | Notes |
+|---|---|---|
+| `SELECT` | Offload | See [Reading](#reading). |
+| `table_changes()` | Offload | Ranges with deletion vectors fall back. See [Change data feed](#change-data-feed). |
+| `CREATE TABLE ... AS SELECT`, `REPLACE TABLE ... AS SELECT` | Offload | With native write on, Spark 3.5 to 4.1, and a DeltaCatalog target. Seen on 4.1, from code on 3.5 and 4.0. See [Creating a table](#creating-a-table). |
+| `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `OPTIMIZE`, `REORG` | | See [Write commands](#write-commands). |
+| `CREATE TABLE` without `AS SELECT`, `DROP TABLE`, `DESCRIBE` | Fallback | They read and write no data files. From code. |
+| `ALTER TABLE` | Fallback, PartialOffload when it reads data | Most forms read and write no data files. `ADD CONSTRAINT` scans the table, and that scan can run in Velox. On Spark 3.5 to 4.1, `ALTER COLUMN ... SYNC IDENTITY` also scans it, and `DROP FEATURE` and `SET TBLPROPERTIES ('delta.columnMapping.mode' = 'none')` can rewrite data files with Delta's writer. From code. |
+| `VACUUM`, `RESTORE`, `SHALLOW CLONE` | | They write no Parquet data files. From code. |
+| `CONVERT TO DELTA` | PartialOffload | Writes no Parquet data files. Unless the statement has `NO STATISTICS` or `spark.databricks.delta.stats.collect=false` is set, it scans the Parquet files for statistics. That scan can run in Velox. From code. |
+
+## Schema evolution
+PartialOffload
+
+The `mergeSchema` and `overwriteSchema` rows need native write on Spark 3.5 to 4.1. Otherwise Delta writes the files.
+
+| Change | Status | Notes |
+|---|---|---|
+| `mergeSchema` with `save(path)` | Offload | Velox writes the evolved schema (from code). On Spark 3.5 and 4.0, the copied `DeltaSuite` checks the results of a `mergeSchema` overwrite. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). |
+| `overwriteSchema` with `save(path)` | Offload | Delta applies the change before the write. From code. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). |
+| Schema evolution in SQL `INSERT` | PartialOffload | From code. |
+| Schema evolution in `MERGE` | PartialOffload, Exception for some STRUCT map key changes | The merged table contents match Delta Lake. Some STRUCT map key changes fail in Velox (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). The two TIMESTAMP_NTZ widening tests fail before the MERGE runs, in a setup write of TIMESTAMP_NTZ values. Seen on 4.1. |
+| `RENAME COLUMN` with column mapping | Offload | Reads after top-level and nested renames return correct results. After a top-level rename, partition pruning and file skipping still work. |
+| `DROP COLUMN` with column mapping | Offload | From code. Delta Lake's `DeltaDropColumnSuite` passes (seen on 4.1). |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ and for native writes to FLOAT or DOUBLE partition columns | Seen on 4.1. See [Table features](#table-features). |
+| `spark.sql.parquet.mergeSchema=true` | Fallback | Every Parquet scan falls back, Delta scans included. From code. |
+
+## Known problems
+
+### Read problems
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A column default stored as `EXISTS_DEFAULT` metadata, for example after `CONVERT TO DELTA`, reads as NULL for rows in files without that column. | ResultMismatch | Seen on 4.1. #11494 reports the same bug for plain Parquet on 3.4 to 4.1. | [#11494](https://github.com/apache/gluten/issues/11494) |
+| In column mapping id mode, when field IDs no longer match the Parquet column names, Gluten returns the column with the matching physical name. Delta Lake returns NULL when no file column has that ID (seen on 4.1). It returns the column the ID points to when another column has it (from code). | ResultMismatch | 3.4 to 4.1. Seen on 4.1 for the NULL case, from code otherwise. | [#12241](https://github.com/apache/gluten/issues/12241) |
+| On a TIMESTAMP-partitioned table with six partition values from .1 to .123456 seconds, `count(DISTINCT ts)` returns 3, as if the values were truncated to milliseconds. Equality filters on the column still return the right rows. | ResultMismatch | Seen on 4.1 | |
+| Reading a data column widened from DATE to TIMESTAMP_NTZ fails on files written before the change with `Converted type DATE is not allowed for requested type TIMESTAMP UTC`. A partition column widened the same way reads correctly. | Exception | Seen on 4.1 | |
+| A query that selects only `_metadata.file_name` from a table with a BIGINT data column named `file_name` reads the data column and fails with `Converted type BIGINT is not allowed for requested type VARCHAR`. | Exception | Seen on 4.1 | |
+| An equality filter on `_metadata.base_row_id` fails with `Synthesized column 'base_row_id' failed filter validation` when a scanned file has a different base row ID. | Exception | Seen on 4.1 | |
+| On a row-tracking table whose older files lack base row IDs, selecting `_metadata.row_id` returns rows. Delta raises an error. Delta Lake's test creates this state with a manual metadata commit. `ALTER TABLE ... SET TBLPROPERTIES` backfills base row IDs when it enables row tracking. | ResultMismatch | Seen on 4.1 | |
+| Reading or deleting from a 2-billion-row table with deletion vectors can grow native memory until the JVM is killed. Delta Lake's two 2-billion-row tests reproduce it. The nightly job force-fails both tests. | Exception | Reported on 4.1 | [#12387](https://github.com/apache/gluten/issues/12387) |
+| With `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default, selecting `__delta_internal_is_row_deleted` from a relation built by hand over `DeltaParquetFileFormat` drops the deleted rows and returns 1 for the rows it keeps. The column is not in the table schema, so SQL and DataFrame queries over the table cannot select it. | ResultMismatch | Seen on 4.1 | |
+| Scans of a table with many small files can be slower than vanilla Spark on EBS storage. In the report, two of five TPC-DS SF1000 queries take 2.2 and 3.4 times as long as on vanilla Spark, with 182,300 files in one table. A maintainer did not see the slowdown on a host with local NVMe storage. | | Reported on 3.4 | [#10778](https://github.com/apache/gluten/issues/10778) |
+
+### Write problems
+
+These need native write on, unless a row says otherwise.
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A write partitioned by a FLOAT or DOUBLE column fails with `Unsupported partition type`. | Exception | 3.5 to 4.1, seen on 4.1 | [#13138](https://github.com/apache/gluten/issues/13138) |
+| A write to a path scheme the native library cannot write fails at task time. This covers custom Hadoop file systems, and S3, GCS, HDFS or ABFS in a build without that connector. | Exception | 3.5 to 4.1, seen on 4.1 (custom file system), from code (missing connector) | [#4963](https://github.com/apache/gluten/issues/4963) |
+| Files written for a column mapping table, in name or id mode, have no Parquet field IDs, which the Delta Lake protocol requires. Delta Lake and Gluten read name-mode files by physical name. Readers that match by field ID cannot. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| Spark's own reader fails on natively written id-mode files with `Spark read schema expects field Ids, but Parquet file schema doesn't contain any field Ids`. Spark uses this reader when a scan falls back or Gluten is off. With `spark.sql.parquet.fieldId.read.ignoreMissing=true`, Spark skips this check and returns NULL for the data columns of these files (from code). That setting is not tested with these files. | Exception | 3.5 to 4.1, seen on 4.1 | |
+| Files written for a UniForm table have no Parquet field IDs, so the Iceberg metadata points at files without IDs. No error is raised. | ResultMismatch | 3.5 to 4.1, from code | [#12957](https://github.com/apache/gluten/pull/12957) |
+| A write to a partitioned UniForm table fails with `The file schema type should be equal with the input rowvector type`. | Exception | 3.5 to 4.1, from code | |
+| Writes to a table with the `materializePartitionColumns` feature leave the partition columns out of the data files. | ResultMismatch | 4.1, from code | |
+| Some writes of a TIMESTAMP_NTZ data column fail with `The file schema type should be equal with the input rowvector type`. Delta Lake's type widening tests fail this way when they write `CAST(value AS TIMESTAMP_NTZ)`. Delta Lake's data skipping tests also write that cast and pass. On Spark 4.x, `spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation=true` sends TIMESTAMP_NTZ writes to Delta's writer. It is not tested as a workaround. | Exception | Seen on 4.1 | |
+| A partitioned write fails in Velox's `PartitionIdGenerator` when a TIMESTAMP or TIMESTAMP_NTZ partition value has sub-millisecond digits. | Exception | 3.5 to 4.1, seen on 4.1 (TIMESTAMP_NTZ), from code (TIMESTAMP) | |
+| A generated partition column such as `DATE_FORMAT(ts, 'yyyy-MM')` stores `12345-07` for the year 12345. Spark stores `+12345-07`. Not checked with native write off. | ResultMismatch | Seen on 4.1 | |
+| The `compression` option set to brotli or lzo fails the write. So does `spark.sql.parquet.compression.codec` when the option is unset. The write fails even with the Hadoop codec class on the classpath. Without that class, Delta's writer fails too, and Spark does not ship it. | Exception | 3.5 to 4.1, from code | |
+| The `compression` option or `spark.sql.parquet.compression.codec` set to lz4_raw writes Snappy data into a `.lz4raw.parquet` file. | ResultMismatch | 3.5 to 4.1, from code | |
+| `maxRecordsPerFile` on an unpartitioned write counts rows but writes whole batches. When the limit is smaller than a batch, the whole batch goes into one file. The other files opened for that batch are 0-byte files that are not valid Parquet. | ResultMismatch | 3.5 to 4.1, from code | |
+| `spark.sql.maxConcurrentOutputFileWriters` above 0 makes a partitioned write with unsorted input fail with `Underlying columnar data is inaccessible from BatchCarrierRow`. | Exception | 3.5 to 4.1, from code | |
+| An unpartitioned native write whose first task gets no rows, for example a write of an empty DataFrame, leaves a 0-byte file that is not valid Parquet. Reading the table directory as plain Parquet can then fail with `is not a Parquet file (length is too low: 0)`. With statistics on, the file stays on disk until `VACUUM`. With `spark.databricks.delta.stats.collect=false`, Delta commits it. The last two sentences are from code. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| A statistics aggregation that Velox cannot offload fails the write with `ClassCastException`. With the default `spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation=false`, the TIMESTAMP_NTZ case from the issue no longer fails on main. Tested on Spark 3.5, seen on 4.1, from code on 4.0. On Spark 3.5 with that option set to true, the TIMESTAMP_NTZ case still fails (from code). | Exception | 3.5 to 4.1, from code | [#12538](https://github.com/apache/gluten/issues/12538) |
+
+### MERGE, UPDATE and DELETE problems
+
+MERGE does not use native write, so the MERGE rows also apply with native write off (from code).
Review Comment:
this statement seems conflicting with the previous write support part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org