You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2026/06/09 07:56:21 UTC
[PR] [spark] Support Merge Into DELETE for data evolution tables [paimon]
leaves12138 opened a new pull request, #8182:
URL: https://github.com/apache/paimon/pull/8182
## What changed
- Add a data-evolution delete rewriter that rewrites normal and blob files by retained row-id ranges.
- Wire matched `DELETE` actions into Spark DataEvolution `MERGE INTO`, including mixed update/delete sequencing and Spark 4.0 parity.
- Add RowTracking and Blob coverage for merge delete and update+delete cases.
## Why
DataEvolution tables currently handle merge updates/inserts, but matched deletes need to physically rewrite the affected row-id ranges so normal files and corresponding blob files stay aligned.
## Validation
- `mvn -pl paimon-spark/paimon-spark-common -am -Pspark3,fast-build -DskipTests compile`
- `JAVA_HOME=/Users/yejunhao/Library/Java/JavaVirtualMachines/ms-17.0.16/Contents/Home PATH=/Users/yejunhao/Library/Java/JavaVirtualMachines/ms-17.0.16/Contents/Home/bin:$PATH mvn -pl paimon-spark/paimon-spark-4.0 -am -Pspark4,fast-build -DskipTests compile`
- `mvn -pl paimon-spark/paimon-spark-3.5 -am -Pspark3,fast-build -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.RowTrackingTest -Dtest=none test` ran 37 tests; the new data-evolution delete tests did not fail, while 5 existing local failures hit codegen loader / generated class instantiation issues.
- `mvn -pl paimon-spark/paimon-spark-ut -am -Pspark3,fast-build -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.BlobTestWithV2Write -Dtest=none test` ran 15 tests; the new blob delete test did not fail, while 2 existing local failures hit the same codegen loader issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support Merge Into DELETE for data evolution tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #8182: [spark] Support Merge Into DELETE for data evolution tables
URL: https://github.com/apache/paimon/pull/8182
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support Merge Into DELETE for data evolution tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8182:
URL: https://github.com/apache/paimon/pull/8182#issuecomment-4727750625
It's too troublesome to maintain the index after deletion. Maybe the correct solution is deletion-vector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org