You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Mingliang Liu (Jira)" <ji...@apache.org> on 2025/07/01 21:50:00 UTC
[jira] [Updated] (FLINK-36679) Add a metric to track checkpoint _metadata size
[ https://issues.apache.org/jira/browse/FLINK-36679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingliang Liu updated FLINK-36679:
----------------------------------
Description:
Currently we expose multiple metrics for the checkpoint size. One specific interesting data point is the {{_metadata}} file size, which can also be added as a metric. The {{_metadata}} file has multiple types of data to store including operator states, coordinator states and properties. Its size should be scoped to a reasonable range, otherwise job may take too long to restore from checkpoints and/or fail to start when its size exceeding RPC frame limit.
However, we saw multiple times the {{_metadata}} file bloats up to 100MB~3GB causing job slow and/or fail to start. In user mail list (for e.g. [[1]|https://lists.apache.org/thread/dttjs2v412xd7slrrx94837ch8wjfo11], [[2]|https://lists.apache.org/thread/yj66dnbs7xrmbspdltq3yfptccm25llt]) and FLINK-32658 community reported similar problems.
Tracking the metadata size can be helpful for operations. Currently we have following use cases:
* The first case was primarily informational. When a job fails to deploy because the payload exceeds the Akka frame size, it’s often unclear what caused the issue. By monitoring the metrics of the checkpoint _metadata file, the problem becomes more apparent. As a workaround to restore the job, users typically increase the Akka frame size config, or redeploy it from an empty state and rewind the Kafka broker’s committed offsets for the consumer group.
* The second case involved a suspected bug in Kafka source v1, which occurs when the operator UID changes. In this scenario, the union state of the old operator grows exponentially with each savepoint, eventually resulting in a multiple GB _metadata file that exceeds the Akka frame size. We recommended that users use the State Processor API to rewrite the checkpoint file, omitting the problematic operator state for the obsolete operator. Although we haven’t identified the root cause, we advised users to maintain deterministic operator UIDs. In the long term, we plan to migrate to Kafka source v2 (FLIP-27).
* The final case involved a _metadata file that was growing in size, but still contained valid data. However, the file became too large to allow for quick job restarts or rescaling. We advised the user to improve the serialization of the operator state if possible, or to minimize heavy usage of operator state to avoid such issues.
was:
Currently we expose multiple metrics for the checkpoint size. One specific interesting data point is the {{_metadata}} file size, which can also be added as a metric. The {{_metadata}} file has multiple types of data to store including operator states, coordinator states and properties. Its size should be scoped to a reasonable range, otherwise job may take too long to restore from checkpoints and/or fail to start when its size exceeding RPC frame limit.
However, we saw multiple times the {{_metadata}} file bloats up to 100MB~3GB causing job slow and/or fail to start. In user mail list (for e.g. [[1]|https://lists.apache.org/thread/dttjs2v412xd7slrrx94837ch8wjfo11], [[2]|https://lists.apache.org/thread/yj66dnbs7xrmbspdltq3yfptccm25llt]) and FLINK-32658 community reported similar problems.
Tracking the metadata size can be helpful for operations.
> Add a metric to track checkpoint _metadata size
> -----------------------------------------------
>
> Key: FLINK-36679
> URL: https://issues.apache.org/jira/browse/FLINK-36679
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Affects Versions: 1.18.1, 1.20.0, 1.19.1
> Reporter: Mingliang Liu
> Priority: Major
> Labels: pull-request-available
>
> Currently we expose multiple metrics for the checkpoint size. One specific interesting data point is the {{_metadata}} file size, which can also be added as a metric. The {{_metadata}} file has multiple types of data to store including operator states, coordinator states and properties. Its size should be scoped to a reasonable range, otherwise job may take too long to restore from checkpoints and/or fail to start when its size exceeding RPC frame limit.
> However, we saw multiple times the {{_metadata}} file bloats up to 100MB~3GB causing job slow and/or fail to start. In user mail list (for e.g. [[1]|https://lists.apache.org/thread/dttjs2v412xd7slrrx94837ch8wjfo11], [[2]|https://lists.apache.org/thread/yj66dnbs7xrmbspdltq3yfptccm25llt]) and FLINK-32658 community reported similar problems.
> Tracking the metadata size can be helpful for operations. Currently we have following use cases:
> * The first case was primarily informational. When a job fails to deploy because the payload exceeds the Akka frame size, it’s often unclear what caused the issue. By monitoring the metrics of the checkpoint _metadata file, the problem becomes more apparent. As a workaround to restore the job, users typically increase the Akka frame size config, or redeploy it from an empty state and rewind the Kafka broker’s committed offsets for the consumer group.
> * The second case involved a suspected bug in Kafka source v1, which occurs when the operator UID changes. In this scenario, the union state of the old operator grows exponentially with each savepoint, eventually resulting in a multiple GB _metadata file that exceeds the Akka frame size. We recommended that users use the State Processor API to rewrite the checkpoint file, omitting the problematic operator state for the obsolete operator. Although we haven’t identified the root cause, we advised users to maintain deterministic operator UIDs. In the long term, we plan to migrate to Kafka source v2 (FLIP-27).
> * The final case involved a _metadata file that was growing in size, but still contained valid data. However, the file became too large to allow for quick job restarts or rescaling. We advised the user to improve the serialization of the operator state if possible, or to minimize heavy usage of operator state to avoid such issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)