You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "felipepessoto (via GitHub)" <gi...@apache.org> on 2026/10/15 00:00:54 UTC

[I] [VL][Delta] Missing Spark schema metadata in native Parquet footer [gluten]

felipepessoto opened a new issue, #13288:
URL: https://github.com/apache/gluten/issues/13288

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   Delta's column-mapping concurrency test reaches a Parquet footer assertion after the observer fix in apache/gluten#13134:
   
   ```text
   org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite
   Enable column mapping feature - txnInterleaved: true
   
   footerMetadata.containsKey(
     org.apache.spark.sql.execution.datasources.parquet.ParquetReadSupport.SPARK_METADATA_KEY
   ) was false
   ```
   
   The test previously timed out because its transaction observer was replaced. Preserving that observer lets it execute the UPDATE/column-mapping interleaving and reach footer validation. The remaining failure is separate from observer lifecycle handling.
   
   **Observed:** at least one resulting Parquet file lacks Spark's schema metadata entry. The failure occurs at the presence check, before the test inspects the embedded column-mapping metadata. This is a native-writer metadata compatibility gap to investigate, not evidence here of incorrect row values.
   
   **Expected:** native Delta writes should preserve the Spark schema metadata required by this path, or the test's expectation should be reviewed if that metadata is not required by the native writer's compatibility contract. Do not suppress the assertion or remove the known-failure entry without establishing the intended behavior.
   
   ### Reproduction and evidence
   
   Run the existing Delta-on-Gluten harness against Delta v4.2.0 / Spark 4.1 with native Delta writes enabled, selecting:
   
   ```text
   org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite
   test: Enable column mapping feature - txnInterleaved: true
   ```
   
   The test prepares an UPDATE while column mapping is disabled, enables column mapping concurrently, completes the transaction, checks the resulting rows, and then inspects the active files' Parquet footers.
   
   - [Observed failing job](https://github.com/apache/gluten/actions/runs/36155269247/job/108177714662), using PR head `b9bb710055aabe37f37a554fb016029d283ec98e`.
   - Artifact: `delta-spark-ut-reports-shard-0`, report `TEST-org.apache.spark.sql.delta.FeatureEnablementConcurrencySuite.xml`.
   - [Delta's footer assertion](https://github.com/delta-io/delta/blob/v4.2.0/spark/src/test/scala/org/apache/spark/sql/delta/FeatureEnablementConcurrencySuite.scala#L101-L111).
   - [The interleaved test](https://github.com/delta-io/delta/blob/v4.2.0/spark/src/test/scala/org/apache/spark/sql/delta/FeatureEnablementConcurrencySuite.scala#L378-L414).
   
   ### Gluten version
   
   Observed on the observer-fix branch in apache/gluten#13134; the test remains in the known-failure baseline. This issue has not yet been reproduced independently against current main.
   
   ### Spark version
   
   Spark 4.1; Delta v4.2.0.
   
   ### Spark configurations
   
   ```text
   spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true
   ```
   
   Other settings are those supplied by the Delta Spark UT workflow and its `DeltaSQLCommandTest` harness.
   
   ### Related
   
   - Follow-up to apache/gluten#12743.
   - Exposed by apache/gluten#13134; intentionally outside that observer fix.
   - Separate from apache/gluten#12241, which concerns native reader field-ID matching.
   
   This issue was written with the assistance of GitHub Copilot CLI 1.0.88.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org