You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "malinjawi (via GitHub)" <gi...@apache.org> on 2026/10/15 10:02:52 UTC

[PR] [VL] Write Delta MERGE data files natively [gluten]

malinjawi opened a new pull request, #13324:
URL: https://github.com/apache/gluten/pull/13324

   ## What changes are proposed in this pull request?
   
   With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite` on, UPDATE, DELETE, OPTIMIZE and CTAS already write their data files natively, but MERGE still goes through Delta's writer. This PR adds MERGE.
   
   - `OffloadDeltaCommand` wraps `MergeIntoCommand` in a new `GlutenDeltaMergeIntoCommand`, which runs it in a Gluten transaction like the other commands.
   - A new flag, `spark.gluten.sql.columnar.backend.velox.delta.enableNativeMergeWrite` (default true), turns off just MERGE.
   - MERGE keeps Delta's writer for tables the native writer doesn't get right yet:
     - column mapping (native files have no Parquet field ids)
     - IcebergCompat
     - `materializePartitionColumns`
     - FLOAT/DOUBLE partition columns (#13138)
     - TIMESTAMP_NTZ
     - nested upper-case field names under case-insensitive analysis
     - stats collection off
     - concurrent output writers
     - `maxRecordsPerFile` on unpartitioned tables
   - The check runs at planning and again in `writeFiles` against the transaction's own metadata, in case the table changed in between.
   - If another transaction observer is installed on the thread, MERGE keeps Delta's writer so that observer isn't replaced. This can go once #13134 lands.
   - The Spark 3.5 `GlutenOptimisticTransaction` gets the Velox schema fallback that the Spark 4.x one already has.
   
   It's a draft because the Delta test harness turns native write on, so its MERGE suites now write natively. I'll go through what that run shows before marking this ready.
   
   ## How was this patch tested?
   
   - `GlutenDeltaMergeWriteGateSuite` has one case per fallback reason. It passes locally on Spark 3.5 and 4.1.
   - `GlutenDeltaNativeMergeWriteSuite` (gluten-ut):
     - Update/delete/insert, insert-only, partitioned, CDF, deletion-vector and DeltaTable API MERGEs write native files and read back the same as a twin table written without Gluten.
     - Each fallback case, and the observer case, keeps Delta's writer.
   - The Delta UT gate runs on this PR.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: IBM Bob
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Write Delta MERGE data files natively [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13324:
URL: https://github.com/apache/gluten/pull/13324#issuecomment-6267513403

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Write Delta MERGE data files natively [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13324:
URL: https://github.com/apache/gluten/pull/13324#issuecomment-6190460771

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL] Write Delta MERGE data files natively [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13324:
URL: https://github.com/apache/gluten/pull/13324#issuecomment-6294878894

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org