You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "sundapeng (via GitHub)" <gi...@apache.org> on 2026/08/18 15:54:47 UTC
[PR] [core] Report what a format table commit wrote to the catalog [paimon]
sundapeng opened a new pull request, #9296:
URL: https://github.com/apache/paimon/pull/9296
> **Stacked on #9295** — its commit shows up here too until it merges. Review only
> `[core] Report what a format table commit wrote to the catalog`.
### Purpose
The writer already counted the rows and the bytes, the commit already knows which partitions it
wrote, and #9295 gave the catalog somewhere to put both. This connects them, behind
`format-table.commit.report-partition-statistics`.
Which mode a commit uses follows from what it did to the partition. An appending commit saw only its
own files, so it reports an increment: a Flink sink commits once per writer subtask, and N
increments over one partition add up to what the job wrote. An overwriting commit replaced
everything the partitions held, so what it wrote is the total and it reports that.
Static prefix overwrite is why a pure increment cannot express this. Clearing a prefix empties every
partition beneath it, including ones this commit writes nothing to; their old data is gone and no
increment says so. Those partitions report zero, an exact zero because they really are empty, and
stay registered. The partition set is the catalog's authority and statistics never touch it; only
ADD PARTITION and DROP PARTITION do. The directories emptied that way come out of the deletion
listing, which already had to walk them, so the numbers cost no extra IO: the row count and byte
size come from the writer, the file count from the commit messages, and the last file creation time
from the commit's own clock.
### Why the option is off by default
A non-empty `ADD` report makes the create-partitions request unsafe to replay, so a 429 or 503 the
client used to retry by itself now surfaces to the caller instead of risking a double-counted
increment. That is a change to how an existing write path fails, and it should be opted into rather
than inherited.
An increment can also drift for reasons no client can see: a writer that is not Paimon, a file
deleted out of band. Convergence is a later full report over the same partition, which the next two
PRs teach MSCK REPAIR TABLE and ANALYZE TABLE to produce.
### Tests
`FormatTableCommitStatisticsTest`: append, dynamic overwrite, static prefix overwrite of a partition
this commit does not write, the summation of the independent increments of concurrent writers of one
partition, a listing that answers under another scheme, a nested directory in the value-only layout,
and reporting turned off.
### API and Format
One new option, off by default. No API or format change.
### Documentation
The option is described in the generated core configuration table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Report partition statistics from format table commits [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng closed pull request #9296: [core] Report partition statistics from format table commits
URL: https://github.com/apache/paimon/pull/9296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Report partition statistics from format table commits [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9296:
URL: https://github.com/apache/paimon/pull/9296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support reporting partition statistics from format table commits [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9296:
URL: https://github.com/apache/paimon/pull/9296#discussion_r3821550947
##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2654,6 +2654,17 @@ public String toString() {
.noDefaultValue()
.withDescription("Format table commit hive sync uri.");
+ public static final ConfigOption<Boolean> FORMAT_TABLE_COMMIT_REPORT_PARTITION_STATISTICS =
+ ConfigOptions.key("format-table.commit.report-partition-statistics")
Review Comment:
Why we need add this option? What are the downsides of the report?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support reporting partition statistics from format table commits [paimon]
Posted by "sundapeng (via GitHub)" <gi...@apache.org>.
sundapeng commented on code in PR #9296:
URL: https://github.com/apache/paimon/pull/9296#discussion_r3822552945
##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2654,6 +2654,17 @@ public String toString() {
.noDefaultValue()
.withDescription("Format table commit hive sync uri.");
+ public static final ConfigOption<Boolean> FORMAT_TABLE_COMMIT_REPORT_PARTITION_STATISTICS =
+ ConfigOptions.key("format-table.commit.report-partition-statistics")
Review Comment:
The report costs nothing, and Paimon tables report on every commitSnapshot without an option, so I dropped it and made this unconditional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org