You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by pa...@apache.org on 2026/04/30 22:17:38 UTC
(flink) branch release-2.3 updated: [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 (#28075)
This is an automated email from the ASF dual-hosted git repository.
RocMarshal pushed a commit to branch release-2.3
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-2.3 by this push:
new 84cd7548d1d [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 (#28075)
84cd7548d1d is described below
commit 84cd7548d1dddcd9dfecd4e49f287fc8554ae57c
Author: Yuepeng Pan <hi...@gmail.com>
AuthorDate: Fri May 1 06:17:30 2026 +0800
[FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 (#28075)
(cherry picked from commit 21d40d42b8b09c217971439486c688d85516440d)
Co-authored-by: spuru9 <si...@gmail.com>
Co-authored-by: XComp <mp...@confluent.io>
Co-authored-by: davidradl <da...@uk.ibm.com>
Co-authored-by: Sergey Nuyanzin <sn...@gmail.com>
Co-authored-by: och5351 <dh...@gmail.com>
---
docs/content.zh/docs/deployment/elastic_scaling.md | 78 ++++++++++++++++++++++
docs/content/docs/deployment/elastic_scaling.md | 77 +++++++++++++++++++++
.../generated/all_jobmanager_section.html | 2 +-
.../shortcodes/generated/web_configuration.html | 2 +-
.../org/apache/flink/configuration/WebOptions.java | 3 +
5 files changed, 160 insertions(+), 2 deletions(-)
diff --git a/docs/content.zh/docs/deployment/elastic_scaling.md b/docs/content.zh/docs/deployment/elastic_scaling.md
index d155e70d90f..96d4bb5ffac 100644
--- a/docs/content.zh/docs/deployment/elastic_scaling.md
+++ b/docs/content.zh/docs/deployment/elastic_scaling.md
@@ -189,4 +189,82 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of `AdaptiveScheduler` rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the `AdaptiveScheduler` enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent rescale records (up to the configured [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size) limit).
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - <u>Rescale UUID</u>: The unique ID in a rescale consists of 32 hexadecimal characters (The UUID definition below is identical to the one here).
+ - <u>Attempt ID</u>: The number of rescale attempts triggered on the same job resource requirements.
+ - <u>Requirements ID</u>: The unique UUID of resource requirements.
+ - <u>Trigger Cause</u>: The reason that triggered a rescale.
+ - <u>Terminal State</u>: The end state of a rescale.
+ - <u>Terminated Reason</u>: The reason of the rescale lifecycle termination.
+ - <u>Start Time</u>: The start time of a rescale.
+ - <u>Duration</u>: Duration from the start of the rescale to its completion or until now if the rescale operation hasn't completed, yet.
+ - <u>End Time</u>: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - <u>ID</u>: The unique UUID of target `Job Vertex`.
+ - <u>Name</u>: The short name of target vertex.
+ - <u>Slot Sharing Group ID</u>: The unique UUID of target `Slot Sharing Group`.
+ - <u>Previous Parallelism</u>: The parallelism of target vertex before the current rescale.
+ - <u>Acquired Parallelism</u>: The parallelism of target vertex after the current rescale.
+ - <u>Sufficient Parallelism</u>: The minimal parallelism of the vertex that would have allowed the rescale operation to go through even if the desired parallelism wasn't possible to be reached.
+ - <u>Desired Parallelism</u>: The desired parallelism of a `Job Vertex` that was specified in the initial change request that triggered the rescale operation.
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - <u>Slot Sharing Group ID</u>: The UUID of the `Slot Sharing Group` to which target slot belongs.
+ - <u>Slot Sharing Group Name</u>: The name of the `Slot Sharing Group` to which the slot belongs.
+ - <u>Previous Slot</u>: The number of slots before the rescale.
+ - <u>Acquired Slot</u>: The number of slots after the rescale.
+ - <u>Desired Slot</u>: The desired number of slots of the rescale.
+ - <u>Sufficient Slot</u>: The minimal number of slots to deploy tasks in the rescale.
+ - <u>Request Profile</u>: The request resource profile of the `Slot Sharing Group` in the rescale.
+ - <u>Acquired Profile</u>: The acquired resource profile of the `Slot Sharing Group` in the rescale.
+ - The internal `Scheduler State History`] of `AdaptiveScheduler` within a rescale (see [`AdaptiveScheduler states in FLIP-160`](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173083547#FLIP160:AdaptiveScheduler-Statemachineofthescheduler) for further details)
+ - <u>State</u>: The scheduler state name.
+ - <u>Enter Time</u>: The time to enter the state.
+ - <u>Leave Time</u>: The time to leave the state.
+ - <u>Duration</u>: Time spent in the state (Leave Time − Enter Time).
+ - <u>Exception</u>: The exception information about current rescale within the state.
+- `Summary`
+ This sub-page displays the total number of rescale events that have occurred since the job was launched,
+ along with the respective counts of failures and successes.
+ Additionally, it provides statistical summaries of the rescale history,
+ such as rescale duration statistics categorized by rescale status, including `Min`, `Max`, `Avg`, and `P50` metrics, etc.
+- `Configuration`
+ This sub-page displays the relevant parameter values used by the `AdaptiveScheduler` during rescaling operations for the current streaming job.
+
+### More details
+
+See the [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) for more details.
+
{{< top >}}
diff --git a/docs/content/docs/deployment/elastic_scaling.md b/docs/content/docs/deployment/elastic_scaling.md
index d0cdfdd06ca..cfe125067f9 100644
--- a/docs/content/docs/deployment/elastic_scaling.md
+++ b/docs/content/docs/deployment/elastic_scaling.md
@@ -198,4 +198,81 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of `AdaptiveScheduler` rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the `AdaptiveScheduler` enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent rescale records (up to the configured [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size) limit).
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - <u>Rescale UUID</u>: The unique ID in a rescale consists of 32 hexadecimal characters (The UUID definition below is identical to the one here).
+ - <u>Attempt ID</u>: The number of rescale attempts triggered on the same job resource requirements.
+ - <u>Requirements ID</u>: The unique UUID of resource requirements.
+ - <u>Trigger Cause</u>: The reason that triggered a rescale.
+ - <u>Terminal State</u>: The end state of a rescale.
+ - <u>Terminated Reason</u>: The reason of the rescale lifecycle termination.
+ - <u>Start Time</u>: The start time of a rescale.
+ - <u>Duration</u>: Duration from the start of the rescale to its completion or until now if the rescale operation hasn't completed, yet.
+ - <u>End Time</u>: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - <u>ID</u>: The unique UUID of target `Job Vertex`.
+ - <u>Name</u>: The short name of target vertex.
+ - <u>Slot Sharing Group ID</u>: The unique UUID of target `Slot Sharing Group`.
+ - <u>Previous Parallelism</u>: The parallelism of target vertex before the current rescale.
+ - <u>Acquired Parallelism</u>: The parallelism of target vertex after the current rescale.
+ - <u>Sufficient Parallelism</u>: The minimal parallelism of the vertex that would have allowed the rescale operation to go through even if the desired parallelism wasn't possible to be reached.
+ - <u>Desired Parallelism</u>: The desired parallelism of a `Job Vertex` that was specified in the initial change request that triggered the rescale operation.
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - <u>Slot Sharing Group ID</u>: The UUID of the `Slot Sharing Group` to which target slot belongs.
+ - <u>Slot Sharing Group Name</u>: The name of the `Slot Sharing Group` to which the slot belongs.
+ - <u>Previous Slot</u>: The number of slots before the rescale.
+ - <u>Acquired Slot</u>: The number of slots after the rescale.
+ - <u>Desired Slot</u>: The desired number of slots of the rescale.
+ - <u>Sufficient Slot</u>: The minimal number of slots to deploy tasks in the rescale.
+ - <u>Request Profile</u>: The request resource profile of the `Slot Sharing Group` in the rescale.
+ - <u>Acquired Profile</u>: The acquired resource profile of the `Slot Sharing Group` in the rescale.
+ - The internal `Scheduler State History`] of `AdaptiveScheduler` within a rescale (see [`AdaptiveScheduler states in FLIP-160`](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173083547#FLIP160:AdaptiveScheduler-Statemachineofthescheduler) for further details)
+ - <u>State</u>: The scheduler state name.
+ - <u>Enter Time</u>: The time to enter the state.
+ - <u>Leave Time</u>: The time to leave the state.
+ - <u>Duration</u>: Time spent in the state (Leave Time − Enter Time).
+ - <u>Exception</u>: The exception information about current rescale within the state.
+- `Summary`
+ This sub-page displays the total number of rescale events that have occurred since the job was launched,
+ along with the respective counts of failures and successes.
+ Additionally, it provides statistical summaries of the rescale history,
+ such as rescale duration statistics categorized by rescale status, including `Min`, `Max`, `Avg`, and `P50` metrics, etc.
+- `Configuration`
+ This sub-page displays the relevant parameter values used by the `AdaptiveScheduler` during rescaling operations for the current streaming job.
+
+### More details
+
+See the [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) for more details.
+
{{< top >}}
diff --git a/docs/layouts/shortcodes/generated/all_jobmanager_section.html b/docs/layouts/shortcodes/generated/all_jobmanager_section.html
index 87d8875a6df..49ccf35c8da 100644
--- a/docs/layouts/shortcodes/generated/all_jobmanager_section.html
+++ b/docs/layouts/shortcodes/generated/all_jobmanager_section.html
@@ -162,7 +162,7 @@
<td><h5>web.adaptive-scheduler.rescale-history.size</h5></td>
<td style="word-wrap: break-word;">0</td>
<td>Integer</td>
- <td>The maximum number of the rescale records per job whose scheduler is <code class="highlighter-rouge">Adaptive</code>. The feature will be disabled when the configuration value is smaller or equals to 0.</td>
+ <td>The maximum number of the rescale records per job whose scheduler is <code class="highlighter-rouge">Adaptive</code>. The feature will be disabled when the configuration value is smaller or equals to 0.<br />Note that high numbers may cause memory issues on the JobManager side.</td>
</tr>
<tr>
<td><h5>web.exception-history-size</h5></td>
diff --git a/docs/layouts/shortcodes/generated/web_configuration.html b/docs/layouts/shortcodes/generated/web_configuration.html
index 38cf3c34a80..9b9c065278d 100644
--- a/docs/layouts/shortcodes/generated/web_configuration.html
+++ b/docs/layouts/shortcodes/generated/web_configuration.html
@@ -18,7 +18,7 @@
<td><h5>web.adaptive-scheduler.rescale-history.size</h5></td>
<td style="word-wrap: break-word;">0</td>
<td>Integer</td>
- <td>The maximum number of the rescale records per job whose scheduler is <code class="highlighter-rouge">Adaptive</code>. The feature will be disabled when the configuration value is smaller or equals to 0.</td>
+ <td>The maximum number of the rescale records per job whose scheduler is <code class="highlighter-rouge">Adaptive</code>. The feature will be disabled when the configuration value is smaller or equals to 0.<br />Note that high numbers may cause memory issues on the JobManager side.</td>
</tr>
<tr>
<td><h5>web.cancel.enable</h5></td>
diff --git a/flink-core/src/main/java/org/apache/flink/configuration/WebOptions.java b/flink-core/src/main/java/org/apache/flink/configuration/WebOptions.java
index ee5b7391ab3..73da95e19ec 100644
--- a/flink-core/src/main/java/org/apache/flink/configuration/WebOptions.java
+++ b/flink-core/src/main/java/org/apache/flink/configuration/WebOptions.java
@@ -153,6 +153,9 @@ public class WebOptions {
"The maximum number of the rescale records per job whose scheduler is %s. "
+ "The feature will be disabled when the configuration value is smaller or equals to 0.",
code(JobManagerOptions.SchedulerType.Adaptive.name()))
+ .linebreak()
+ .text(
+ "Note that high numbers may cause memory issues on the JobManager side.")
.build());
/** Timeout for asynchronous operations by the web monitor in milliseconds. */