You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "felipepessoto (via GitHub)" <gi...@apache.org> on 2026/10/15 00:00:54 UTC
[I] [VL][Delta] Missing Spark schema metadata in native Parquet footer [gluten]
felipepessoto opened a new issue, #13288:
URL: https://github.com/apache/gluten/issues/13288
### Backend
VL (Velox)
### Bug description
Delta's column-mapping concurrency test reaches a Parquet footer assertion after the observer fix in apache/gluten#13134:
```text
org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite
Enable column mapping feature - txnInterleaved: true
footerMetadata.containsKey(
org.apache.spark.sql.execution.datasources.parquet.ParquetReadSupport.SPARK_METADATA_KEY
) was false
```
The test previously timed out because its transaction observer was replaced. Preserving that observer lets it execute the UPDATE/column-mapping interleaving and reach footer validation. The remaining failure is separate from observer lifecycle handling.
**Observed:** at least one resulting Parquet file lacks Spark's schema metadata entry. The failure occurs at the presence check, before the test inspects the embedded column-mapping metadata. This is a native-writer metadata compatibility gap to investigate, not evidence here of incorrect row values.
**Expected:** native Delta writes should preserve the Spark schema metadata required by this path, or the test's expectation should be reviewed if that metadata is not required by the native writer's compatibility contract. Do not suppress the assertion or remove the known-failure entry without establishing the intended behavior.
### Reproduction and evidence
Run the existing Delta-on-Gluten harness against Delta v4.2.0 / Spark 4.1 with native Delta writes enabled, selecting:
```text
org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite
test: Enable column mapping feature - txnInterleaved: true
```
The test prepares an UPDATE while column mapping is disabled, enables column mapping concurrently, completes the transaction, checks the resulting rows, and then inspects the active files' Parquet footers.
- [Observed failing job](https://github.com/apache/gluten/actions/runs/36155269247/job/108177714662), using PR head `b9bb710055aabe37f37a554fb016029d283ec98e`.
- Artifact: `delta-spark-ut-reports-shard-0`, report `TEST-org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite.xml`.
- [Delta's footer assertion](https://github.com/delta-io/delta/blob/v4.2.0/spark/src/test/scala/org/apache/spark/sql/delta/FeatureEnablementConcurrencySuite.scala#L101-L111).
- [The interleaved test](https://github.com/delta-io/delta/blob/v4.2.0/spark/src/test/scala/org/apache/spark/sql/delta/FeatureEnablementConcurrencySuite.scala#L378-L414).
### Gluten version
Observed on the observer-fix branch in apache/gluten#13134; the test remains in the known-failure baseline. This issue has not yet been reproduced independently against current main.
### Spark version
Spark 4.1; Delta v4.2.0.
### Spark configurations
```text
spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true
```
Other settings are those supplied by the Delta Spark UT workflow and its `DeltaSQLCommandTest` harness.
### Related
- Follow-up to apache/gluten#12743.
- Exposed by apache/gluten#13134; intentionally outside that observer fix.
- Separate from apache/gluten#12241, which concerns native reader field-ID matching.
This issue was written with the assistance of GitHub Copilot CLI 1.0.88.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org