You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "kerwin-zk (via GitHub)" <gi...@apache.org> on 2026/06/03 02:32:39 UTC
[PR] [spark] Support V2 DML for row-tracking append-only tables [paimon]
kerwin-zk opened a new pull request, #8094:
URL: https://github.com/apache/paimon/pull/8094
### Purpose
Support V2 DML for row-tracking append-only tables
### Tests
CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support V2 DML for row-tracking append-only tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #8094: [spark] Support V2 DML for row-tracking append-only tables
URL: https://github.com/apache/paimon/pull/8094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support V2 DML for row-tracking append-only tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8094:
URL: https://github.com/apache/paimon/pull/8094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support V2 DML for row-tracking append-only tables [paimon]
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #8094:
URL: https://github.com/apache/paimon/pull/8094#issuecomment-4610829254
> I left one inline comment about the Spark 3.5 row-tracking V2 path. Spark 3.5 lacks metadata-aware row-level writes, so this likely needs a version gate or Spark 3.5-specific handling before merge.
@JingsongLi done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support V2 DML for row-tracking append-only tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8094:
URL: https://github.com/apache/paimon/pull/8094#discussion_r3346218212
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/SparkTable.scala:
##########
@@ -115,8 +113,8 @@ object SparkTable {
case fs: FileStoreTable =>
fs.primaryKeys().isEmpty &&
!sparkTable.coreOptions.deletionVectorsEnabled() &&
- !sparkTable.coreOptions.rowTrackingEnabled() &&
- !sparkTable.coreOptions.dataEvolutionEnabled()
+ !sparkTable.coreOptions.dataEvolutionEnabled() &&
+ !SparkTypeUtils.containsCharType(fs.rowType())
Review Comment:
This now enables V2 row-level ops for row-tracking tables on Spark 3.5 as well. Spark 3.5 does not have the metadata-aware row-level write path: DataWriter only has write(T) (no write(metadata, data)), MetadataColumn does not expose preserve flags, and V2Writes builds ReplaceData from rd.dataInput after stripping metadata columns. Since PaimonBatchWriteBase still creates PaimonV2MetadataAwareDataWriter whenever row tracking + COW, Spark 3.5 will call the inherited single-arg write(record) path and rewrite rows with the plain writer, losing the original _ROW_ID/_SEQUENCE_NUMBER for copied/updated rows instead of preserving them. Could we keep row-tracking tables on the V1 path for Spark 3.5, e.g. only allow row-tracking V2 ops on Spark >= 4.0, or add an explicit Spark 3.5 implementation/test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org