You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "liuml07 (via GitHub)" <gi...@apache.org> on 2025/07/03 04:22:04 UTC
[PR] [FLINK-36679][runtime] Add a metric to track checkpoint _metadata size [flink]
liuml07 opened a new pull request, #26744:
URL: https://github.com/apache/flink/pull/26744
## What is the purpose of the change
Add a metric to track checkpoint _metadata size. Check use cases of this metric in the JIRA: https://issues.apache.org/jira/browse/FLINK-36679
## Brief change log
Add a metric `lastCheckpointMetadataSize` to track checkpoint _metadata size
## Verifying this change
Updated existing unit tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36679][runtime] Add a metric to track checkpoint _metadata size [flink]
Posted by "liuml07 (via GitHub)" <gi...@apache.org>.
liuml07 commented on code in PR #26744:
URL: https://github.com/apache/flink/pull/26744#discussion_r2184799731
##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCheckpointStatsTracker.java:
##########
@@ -543,6 +551,18 @@ public Long getValue() {
}
}
+ private class LatestCompletedCheckpointMetadataSizeGauge implements Gauge<Long> {
+ @Override
+ public Long getValue() {
+ CompletedCheckpointStats completed = latestCompletedCheckpoint;
+ if (completed != null) {
+ return completed.getMetadataSize();
+ } else {
+ return -1L;
+ }
Review Comment:
Actually this should be returning `completed.getMetadataSize()` instead of `completed.getStateSize()`. Fortunately we have the test that fails so it's caught.
I used the if-else following other methods convention in this class. But using ternary operator seems more concise, so I still switched per this comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36679][runtime] Add a metric to track checkpoint _metadata size [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #26744:
URL: https://github.com/apache/flink/pull/26744#discussion_r2184788515
##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/AbstractCheckpointStats.java:
##########
@@ -93,6 +93,11 @@ public abstract class AbstractCheckpointStats implements Serializable {
*/
public abstract long getCheckpointedSize();
+ /**
+ * @return The metadata file size, 0 if unknown.
Review Comment:
```suggestion
* @return The metadata file size, -1 if unknown.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36679][runtime] Add a metric to track checkpoint _metadata size [flink]
Posted by "ferenc-csaky (via GitHub)" <gi...@apache.org>.
ferenc-csaky merged PR #26744:
URL: https://github.com/apache/flink/pull/26744
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36679][runtime] Add a metric to track checkpoint _metadata size [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #26744:
URL: https://github.com/apache/flink/pull/26744#discussion_r2184189726
##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/DefaultCheckpointStatsTracker.java:
##########
@@ -543,6 +551,18 @@ public Long getValue() {
}
}
+ private class LatestCompletedCheckpointMetadataSizeGauge implements Gauge<Long> {
+ @Override
+ public Long getValue() {
+ CompletedCheckpointStats completed = latestCompletedCheckpoint;
+ if (completed != null) {
+ return completed.getMetadataSize();
+ } else {
+ return -1L;
+ }
Review Comment:
```suggestion
return completed != null ? completed.getStateSize() : -1L;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36679][runtime] Add a metric to track checkpoint _metadata size [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26744:
URL: https://github.com/apache/flink/pull/26744#issuecomment-3030629690
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "599794bb0c46e235237c90668546b4639ea54b6d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "599794bb0c46e235237c90668546b4639ea54b6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 599794bb0c46e235237c90668546b4639ea54b6d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36679][runtime] Add a metric to track checkpoint _metadata size [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #26744:
URL: https://github.com/apache/flink/pull/26744#discussion_r2182249865
##########
docs/content.zh/docs/ops/metrics.md:
##########
@@ -1181,6 +1181,11 @@ Note that for failed checkpoints, metrics are updated on a best efforts basis an
<td>The full size of the last checkpoint (in bytes).</td>
<td>Gauge</td>
</tr>
+ <tr>
+ <td>lastCheckpointMetadataSize</td>
+ <td>上次检查点 _metadata 文件的大小（字节）.</td>
Review Comment:
Well, we can keep English here since this page has not been translated to Chinese yet.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/AbstractCheckpointStats.java:
##########
@@ -93,6 +93,13 @@ public abstract class AbstractCheckpointStats implements Serializable {
*/
public abstract long getCheckpointedSize();
+ /**
+ * @return The metadata file size, 0 if unknown.
+ */
+ public long getMetadataSize() {
Review Comment:
better keep this method abstract
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org