You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2025/08/18 07:39:48 UTC
[PR] [spark] Support DELETE for row tracking table [paimon]
Zouxxyy opened a new pull request, #6086:
URL: https://github.com/apache/paimon/pull/6086
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Subtask of https://github.com/apache/paimon/issues/6022, support spark DELETE for row tracking table
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support DELETE for row tracking table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6086:
URL: https://github.com/apache/paimon/pull/6086#issuecomment-3195906411
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support DELETE for row tracking table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6086:
URL: https://github.com/apache/paimon/pull/6086#discussion_r2281619222
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/PaimonCommand.scala:
##########
@@ -54,7 +54,10 @@ trait PaimonCommand
with SQLConfHelper {
lazy val dvSafeWriter: PaimonSparkWriter = {
- if (table.primaryKeys().isEmpty && table.bucketMode() == BucketMode.HASH_FIXED) {
+ if (
+ table.coreOptions().deletionVectorsEnabled() && table.primaryKeys().isEmpty && table
Review Comment:
Why add dv? Do compact for bucketed Append table is dangerous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support DELETE for row tracking table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6086:
URL: https://github.com/apache/paimon/pull/6086#discussion_r2281557533
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/DeleteFromPaimonTableCommand.scala:
##########
@@ -147,10 +147,13 @@ case class DeleteFromPaimonTableCommand(
// Step4: build a dataframe that contains the unchanged data, and write out them.
val toRewriteScanRelation = Filter(Not(condition), newRelation)
- val data = createDataset(sparkSession, toRewriteScanRelation)
+ var data = createDataset(sparkSession, toRewriteScanRelation)
+ if (coreOptions.rowTrackingEnabled()) {
+ data = selectWithRowLineageMetaCols(data)
+ }
// only write new files, should have no compaction
- val addCommitMessage = dvSafeWriter.writeOnly().write(data)
+ val addCommitMessage = dvSafeWriter.withRowLineage().write(data)
Review Comment:
Should keep writeOnly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support DELETE for row tracking table [paimon]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #6086:
URL: https://github.com/apache/paimon/pull/6086#discussion_r2281592436
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/PaimonCommand.scala:
##########
@@ -54,7 +54,10 @@ trait PaimonCommand
with SQLConfHelper {
lazy val dvSafeWriter: PaimonSparkWriter = {
- if (table.primaryKeys().isEmpty && table.bucketMode() == BucketMode.HASH_FIXED) {
+ if (
+ table.coreOptions().deletionVectorsEnabled() && table.primaryKeys().isEmpty && table
Review Comment:
Currently, we always use dvSafeWriter regardless of whether DV is enabled or not.
Here I add a new judgment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support DELETE for row tracking table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #6086:
URL: https://github.com/apache/paimon/pull/6086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org