You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "kerwin-zk (via GitHub)" <gi...@apache.org> on 2026/07/07 03:42:27 UTC
[PR] [spark] Support deleted record metrics for V2 delete [paimon]
kerwin-zk opened a new pull request, #8486:
URL: https://github.com/apache/paimon/pull/8486
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Implement the todo left by #7114 in `PaimonV2Write.supportedCustomMetrics`: `support record metrics for row level ops`.
This PR adds a `number of deleted records` commit metric for V2 copy-on-write DELETE. The V2 row-level operation path only applies to append tables without deletion vectors or data evolution, so the file-level row counts are exact and the number of deleted records is exactly:
```
deletedRecords = rowCount(removed files) - rowCount(rewritten files)
```
The value is computed at commit time from the commit messages (no extra IO) and posted through the existing driver metrics mechanism.
For UPDATE and MERGE, the rewritten files mix copied rows with modified rows, so record-level counts cannot be derived from file statistics and remain unreported.
### Tests
Extended `PaimonMetricTest` ("Paimon Metric: delete") to assert `deletedRecords` for both a partial-file delete and a whole-file delete.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support deleted record metrics for V2 delete [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on PR #8486:
URL: https://github.com/apache/paimon/pull/8486#issuecomment-4931453285
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support deleted record metrics for V2 delete [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8486:
URL: https://github.com/apache/paimon/pull/8486#discussion_r3534180636
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/write/PaimonV2Write.scala:
##########
@@ -84,11 +84,16 @@ class PaimonV2Write(
PaimonAddedTableFilesMetric()
)
if (copyOnWriteScan.isEmpty) {
- // todo: support record metrics for row level ops
buffer += PaimonInsertedRecordsMetric()
}
if (copyOnWriteScan.nonEmpty) {
buffer += PaimonDeletedTableFilesMetric()
+ // For DELETE, the number of deleted records is exactly the row count difference between
+ // the removed files and the rewritten files. For UPDATE and MERGE, the rewritten files
+ // mix copied rows with modified rows, so record metrics cannot be derived from file stats.
+ if (operationType.contains(Snapshot.Operation.DELETE)) {
Review Comment:
This condition misses the DELETE plan used by the added test: `metrics("deletedRecords")` is absent before the write runs. I ran `mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.PaimonMetricTest -Dtest=none test`, and `Paimon Metric: delete` fails with `NoSuchElementException: key not found: deletedRecords` at `PaimonMetricTest.scala:188`. Since Spark builds the command metrics from `supportedCustomMetrics`, we need to register `PaimonDeletedRecordsMetric` for the row-level DELETE write path that actually reaches this test, or ensure `operationType` is populated before `supportedCustomMetrics` is queried.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support deleted record metrics for V2 delete [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron merged PR #8486:
URL: https://github.com/apache/paimon/pull/8486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org