You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "RocMarshal (via GitHub)" <gi...@apache.org> on 2026/04/23 04:09:32 UTC
[PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
RocMarshal opened a new pull request, #28003:
URL: https://github.com/apache/flink/pull/28003
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
[FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487
## Brief change log
[FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487
## Verifying this change
N.A
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented)
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` followed by the name of the tool, and uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name and Version]
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130949332
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
Review Comment:
Are the states documented?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3155215639
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,81 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - <u>Rescale UUID</u>: The unique ID in a rescale consists of 32 hexadecimal characters (The UUID definition below is identical to the one here).
+ - <u>Attempt ID</u>: The number of rescale attempts triggered on the same job resource requirements.
+ - <u>Requirements ID</u>: The unique UUID of resource requirements.
+ - <u>Trigger Cause</u>: The reason that triggered a rescale.
+ - <u>Terminal State</u>: The end state of a rescale.
+ - <u>Terminated Reason</u>: The reason of the rescale lifecycle termination.
+ - <u>Start Time</u>: The start time of a rescale.
+ - <u>Duration</u>: Duration from the start of the rescale to its completion or until now if the rescale operation hasn't completed, yet.
+ - <u>End Time</u>: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - <u>ID</u>: The unique UUID of target `Job Vertex`.
+ - <u>Name</u>: The short name of target vertex.
+ - <u>Slot Sharing Group ID</u>: The unique UUID of target `Slot Sharing Group`.
+ - <u>Previous Parallelism</u>: The parallelism of target vertex before the current rescale.
+ - <u>Acquired Parallelism</u>: The parallelism of target vertex after the current rescale.
+ - <u>Sufficient Parallelism</u>: The minimal parallelism of the vertex that would have allowed the rescale operation to go through even if the desired parallelism wasn't possible to be reached.
+ - <u>Desired Parallelism</u>: The desired parallelism of a `Job Vertex` that was specified in the initial change request that triggered the rescale operation.
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - <u>Slot Sharing Group ID</u>: The UUID of the `Slot Sharing Group` to which target slot belongs.
+ - <u>Slot Sharing Group Name</u>: The name of the `Slot Sharing Group` to which the slot belongs.
+ - <u>Previous Slot</u>: The number of slots before the rescale.
+ - <u>Acquired Slot</u>: The number of slots after the rescale.
+ - <u>Desired Slot</u>: The desired number of slots of the rescale.
+ - <u>Sufficient Slot</u>: The minimal number of slots to deploy tasks in the rescale.
+ - <u>Request Profile</u>: The request resource profile of the `Slot Sharing Group` in the rescale.
+ - <u>Acquired Profile</u>: The acquired resource profile of the `Slot Sharing Group` in the rescale.
+ - The internal [`Scheduler State History`](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173083547#FLIP160:AdaptiveScheduler-Statemachineofthescheduler) of `Adaptive Scheduler` within a rescale
Review Comment:
The link is not pointing to the Scheduler State History, so the link text might be misleading. Rather add something like
> (see [AdaptiveScheduler states in FLIP-160](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173083547#FLIP160:AdaptiveScheduler-Statemachineofthescheduler) for further details)
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,82 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
Review Comment:
Fair point. I guess that would be something that should be specified in the configuration docs, hm?
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,81 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - <u>Rescale UUID</u>: The unique ID in a rescale consists of 32 hexadecimal characters (The UUID definition below is identical to the one here).
+ - <u>Attempt ID</u>: The number of rescale attempts triggered on the same job resource requirements.
+ - <u>Requirements ID</u>: The unique UUID of resource requirements.
+ - <u>Trigger Cause</u>: The reason that triggered a rescale.
+ - <u>Terminal State</u>: The end state of a rescale.
+ - <u>Terminated Reason</u>: The reason of the rescale lifecycle termination.
+ - <u>Start Time</u>: The start time of a rescale.
+ - <u>Duration</u>: Duration from the start of the rescale to its completion or until now if the rescale operation hasn't completed, yet.
+ - <u>End Time</u>: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - <u>ID</u>: The unique UUID of target `Job Vertex`.
+ - <u>Name</u>: The short name of target vertex.
+ - <u>Slot Sharing Group ID</u>: The unique UUID of target `Slot Sharing Group`.
+ - <u>Previous Parallelism</u>: The parallelism of target vertex before the current rescale.
+ - <u>Acquired Parallelism</u>: The parallelism of target vertex after the current rescale.
+ - <u>Sufficient Parallelism</u>: The minimal parallelism of the vertex that would have allowed the rescale operation to go through even if the desired parallelism wasn't possible to be reached.
+ - <u>Desired Parallelism</u>: The desired parallelism of a `Job Vertex` that was specified in the initial change request that triggered the rescale operation.
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - <u>Slot Sharing Group ID</u>: The UUID of the `Slot Sharing Group` to which target slot belongs.
+ - <u>Slot Sharing Group Name</u>: The name of the `Slot Sharing Group` to which the slot belongs.
+ - <u>Previous Slot</u>: The number of slots before the rescale.
+ - <u>Acquired Slot</u>: The number of slots after the rescale.
+ - <u>Desired Slot</u>: The desired number of slots of the rescale.
+ - <u>Sufficient Slot</u>: The minimal number of slots to deploy tasks in the rescale.
+ - <u>Request Profile</u>: The request resource profile of the `Slot Sharing Group` in the rescale.
+ - <u>Acquired Profile</u>: The acquired resource profile of the `Slot Sharing Group` in the rescale.
+ - The internal [`Scheduler State History`](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173083547#FLIP160:AdaptiveScheduler-Statemachineofthescheduler) of `Adaptive Scheduler` within a rescale
Review Comment:
nit: We can also call it `AdaptiveScheduler` rather than `Adaptive Scheduler` because that's the actual component's name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132123125
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
Review Comment:
consize.
And I had a try on a new description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28003:
URL: https://github.com/apache/flink/pull/28003#issuecomment-4301695857
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3e5f3c5bbdd09c4d0ba94a78fcbc41bc47faee2b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e5f3c5bbdd09c4d0ba94a78fcbc41bc47faee2b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e5f3c5bbdd09c4d0ba94a78fcbc41bc47faee2b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal merged PR #28003:
URL: https://github.com/apache/flink/pull/28003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130969481
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
+ - Exception: The exception information about current rescale during the state
+- `Summary`
+ This sub-page displays the total number of rescale events that have occurred since the job was launched,
+ along with the respective counts of failures and successes.
+ Additionally, it provides statistical summaries of the rescale history,
+ such as rescale duration statistics categorized by rescale status, including `Min`, `Max`, `Avg`, and `Pnn` metrics.
Review Comment:
what is Pnn here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132189792
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
Review Comment:
this doesn't include the current state which will not have an end time.
This is determined by the collection mechanism of adaptive state. Simply put, when a rescale adds a state, that state already includes start and end times."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132189792
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
Review Comment:
this doesn't include the current state which will not have an end time.
This is determined by the collection mechanism of adaptive state. Simply put, when a rescale adds a state, that state already includes start and end times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3154325750
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
Review Comment:
I missed this comment. @davidradl has a point here. We could link to the [StateMachine section of FLIP-160](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173083547#FLIP160:AdaptiveScheduler-Statemachineofthescheduler). That's where it is documented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #28003:
URL: https://github.com/apache/flink/pull/28003#issuecomment-4344613100
Thank you all for comments.
Updated.
Looking forward to your next review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130952701
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
Review Comment:
does this include the current state which will not have an end time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130941233
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
Review Comment:
I would say UUID - and define that once
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132201097
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
+ - Exception: The exception information about current rescale during the state
Review Comment:
maybe `during`-> `within`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132165828
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132092275
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
Review Comment:
yes.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-RescaleID&ResourceRequirementsrequest
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #28003:
URL: https://github.com/apache/flink/pull/28003#issuecomment-4301678834
Hi, @XComp @spuru9 @och5351 Could you help take a look ?
Any input is appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130590975
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
Review Comment:
it talks of Terminated Reason which implies a termination, then in the wording in talks of completions and terminations as 2 different things. Implying that termination is an abnormal end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130930101
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
Review Comment:
can we have a reference to Job vertex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3129530935
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`:
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
Review Comment:
@RocMarshal Can you implement the suggestion for removing the substructure as done for Overview for others as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130955002
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
+ - Exception: The exception information about current rescale during the state
Review Comment:
what does during the state mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130577315
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
Review Comment:
do we need to say target rescale- can we just say rescale or are there other rescale types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130924160
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
Review Comment:
I assume it is always the Duration from the start of the rescale. If the rescale is ongoing I guess the end is now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132120075
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
Review Comment:
updated
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3129760193
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`:
Review Comment:
```suggestion
- `Overview`
```
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`:
Review Comment:
```suggestion
- `Overview`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130932805
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
Review Comment:
I assume JobVertx is the same as Job Vertex- I would suggest we are consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130927847
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
Review Comment:
I assume this is the same as Duration unless the rescale is ongoing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132139824
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
Review Comment:
Yes, +1 from me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3145244142
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters(The UUID definition below is identical to the one here)
Review Comment:
```suggestion
- Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters (The UUID definition below is identical to the one here)
```
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters(The UUID definition below is identical to the one here)
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique UUID of resource requirements
+ - Trigger Cause: The reason that triggered a rescale
+ - Terminal State: The end state of a rescale
+ - Terminated Reason: The reason of the rescale lifecycle termination
+ - Start Time: The start time of a rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique UUID of target `Job Vertex`
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique UUID of target `Slot Sharing Group`
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
Review Comment:
```suggestion
- Sufficient Parallelism: The minimal parallelism of the vertex that would have allow the rescale operation to go through even if the desired parallelism wasn't possible to be reached.
```
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters(The UUID definition below is identical to the one here)
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique UUID of resource requirements
+ - Trigger Cause: The reason that triggered a rescale
+ - Terminal State: The end state of a rescale
+ - Terminated Reason: The reason of the rescale lifecycle termination
+ - Start Time: The start time of a rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique UUID of target `Job Vertex`
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique UUID of target `Slot Sharing Group`
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of a `Job Vertex`
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The UUID of the `Slot Sharing Group` to which target slot belongs
+ - Slot Sharing Group Name: The name of the `Slot Sharing Group` to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the `Slot Sharing Group` in the rescale
+ - Acquired Profile: The acquired resource profile of the `Slot Sharing Group` in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
+ - Exception: The exception information about current rescale within the state
+- `Summary`
+ This sub-page displays the total number of rescale events that have occurred since the job was launched,
+ along with the respective counts of failures and successes.
+ Additionally, it provides statistical summaries of the rescale history,
+ such as rescale duration statistics categorized by rescale status, including `Min`, `Max`, `Avg`, and `P50` metrics, etc.
+- `Configuration`
+ This sub-page displays the relevant parameter values used by the Adaptive Scheduler during rescaling operations for the current streaming job.
+
+### How to use it
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
Review Comment:
Should we also mention that a negative value or 0 disables the feature?
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters(The UUID definition below is identical to the one here)
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique UUID of resource requirements
+ - Trigger Cause: The reason that triggered a rescale
+ - Terminal State: The end state of a rescale
+ - Terminated Reason: The reason of the rescale lifecycle termination
+ - Start Time: The start time of a rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique UUID of target `Job Vertex`
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique UUID of target `Slot Sharing Group`
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of a `Job Vertex`
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The UUID of the `Slot Sharing Group` to which target slot belongs
+ - Slot Sharing Group Name: The name of the `Slot Sharing Group` to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the `Slot Sharing Group` in the rescale
+ - Acquired Profile: The acquired resource profile of the `Slot Sharing Group` in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
+ - Exception: The exception information about current rescale within the state
+- `Summary`
+ This sub-page displays the total number of rescale events that have occurred since the job was launched,
+ along with the respective counts of failures and successes.
+ Additionally, it provides statistical summaries of the rescale history,
+ such as rescale duration statistics categorized by rescale status, including `Min`, `Max`, `Avg`, and `P50` metrics, etc.
+- `Configuration`
+ This sub-page displays the relevant parameter values used by the Adaptive Scheduler during rescaling operations for the current streaming job.
+
+### How to use it
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- `web.adaptive-scheduler.rescale-history.size`: `4`
Review Comment:
We could make this parameter clickable and link it to the corresponding entry in the configuration docs: `http://localhost:1313/flink/flink-docs-master/docs/deployment/config/#web-adaptive-scheduler-rescale-history-size`
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters(The UUID definition below is identical to the one here)
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique UUID of resource requirements
+ - Trigger Cause: The reason that triggered a rescale
+ - Terminal State: The end state of a rescale
+ - Terminated Reason: The reason of the rescale lifecycle termination
+ - Start Time: The start time of a rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique UUID of target `Job Vertex`
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique UUID of target `Slot Sharing Group`
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of a `Job Vertex`
Review Comment:
```suggestion
- Desired Parallelism: The desired parallelism of a `Job Vertex` that was specified in the initial change request that triggered the rescale operation.
```
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters(The UUID definition below is identical to the one here)
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique UUID of resource requirements
+ - Trigger Cause: The reason that triggered a rescale
+ - Terminal State: The end state of a rescale
+ - Terminated Reason: The reason of the rescale lifecycle termination
+ - Start Time: The start time of a rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique UUID of target `Job Vertex`
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique UUID of target `Slot Sharing Group`
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of a `Job Vertex`
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The UUID of the `Slot Sharing Group` to which target slot belongs
+ - Slot Sharing Group Name: The name of the `Slot Sharing Group` to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the `Slot Sharing Group` in the rescale
+ - Acquired Profile: The acquired resource profile of the `Slot Sharing Group` in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
+ - Exception: The exception information about current rescale within the state
+- `Summary`
+ This sub-page displays the total number of rescale events that have occurred since the job was launched,
+ along with the respective counts of failures and successes.
+ Additionally, it provides statistical summaries of the rescale history,
+ such as rescale duration statistics categorized by rescale status, including `Min`, `Max`, `Avg`, and `P50` metrics, etc.
+- `Configuration`
+ This sub-page displays the relevant parameter values used by the Adaptive Scheduler during rescaling operations for the current streaming job.
+
+### How to use it
Review Comment:
This section is so short that we could add the reference to the configuration in the intro of the `Rescale History` section. WDYT? ...probably as a 3rd paragraph
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale
Review Comment:
Would it makes sense to highlight the field names here (maybe by underlining them)? Right now, this looks like a big block of text. Adding some highlighting here might help seeing the structure of the text block.
Or make it a table per block. WDYT? 🤔
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
Review Comment:
```suggestion
- Attempt ID: The number of rescale attempts triggered on the same job resource requirements.
```
What about that? ☝️
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
Review Comment:
```suggestion
- Duration: Duration from the start of the rescale to its completion or until now if the rescale operation hasn't completed, yet.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3162756517
##########
flink-core/src/main/java/org/apache/flink/configuration/WebOptions.java:
##########
@@ -153,6 +153,11 @@ public class WebOptions {
"The maximum number of the rescale records per job whose scheduler is %s. "
+ "The feature will be disabled when the configuration value is smaller or equals to 0.",
code(JobManagerOptions.SchedulerType.Adaptive.name()))
+ .linebreak()
+ .text(
+ "Note that a larger value will theoretically increase memory usage within the JobManager. "
+ + "It's advised not to configure this value excessively to mitigate the risk of high memory overhead. "
+ + "Specifically, setting it to Integer.MAX_VALUE will result in JobManager memory exhaustion.")
Review Comment:
Either resolve the value or just write something like "high numbers may cause memory issues on the JobManager side". Because `Integer.MAX_VALUE` itself is not the only value that could cause memory issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3154981060
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,82 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
Review Comment:
Do we have any limitation for this
or can I put Integer.MAX_VALUE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3165939849
##########
flink-core/src/main/java/org/apache/flink/configuration/WebOptions.java:
##########
@@ -153,6 +153,11 @@ public class WebOptions {
"The maximum number of the rescale records per job whose scheduler is %s. "
+ "The feature will be disabled when the configuration value is smaller or equals to 0.",
code(JobManagerOptions.SchedulerType.Adaptive.name()))
+ .linebreak()
+ .text(
+ "Note that a larger value will theoretically increase memory usage within the JobManager. "
+ + "It's advised not to configure this value excessively to mitigate the risk of high memory overhead. "
+ + "Specifically, setting it to Integer.MAX_VALUE will result in JobManager memory exhaustion.")
Review Comment:
Thanks @XComp for your reminder.
Updated :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "och5351 (via GitHub)" <gi...@apache.org>.
och5351 commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3158880523
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,81 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
Review Comment:
In this part, while most people might already know this, how about changing it to be more explicit, like:
```md
This sub-page displays abbreviated information for the most recent rescale records (up to the configured `web.adaptive-scheduler.rescale-history.size` limit).
```
I think it makes the configuration key more visible and easier to reference for users who may not be familiar with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "och5351 (via GitHub)" <gi...@apache.org>.
och5351 commented on PR #28003:
URL: https://github.com/apache/flink/pull/28003#issuecomment-4348913442
LGTM
I think I just need to solve the remaining problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3129253082
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
+This primarily includes the following sub-pages:
+
+- `Rescales/Overview`:
+ This sub-page displays recent rescale records across various rescale terminal state,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale.
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggers the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale.
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID:The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The previous parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The acquired parallelism of target vertex after the current rescale
Review Comment:
```suggestion
- Previous Parallelism: The parallelism of target vertex before the current rescale
- Acquired Parallelism: The parallelism of target vertex after the current rescale
```
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
Review Comment:
```suggestion
positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
```
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
Review Comment:
```suggestion
Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
```
Nothing against the anchor tags, but this seems more cleaner and more used in the docs. Anchors are also used in a few places.
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
+This primarily includes the following sub-pages:
+
+- `Rescales/Overview`:
+ This sub-page displays recent rescale records across various rescale terminal state,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale.
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggers the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale.
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID:The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The previous parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The acquired parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex.
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name:The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The previous number of slots before the rescale
+ - Acquired Slot: The acquired number of slots after the rescale
Review Comment:
```suggestion
- Previous Slot: The number of slots before the rescale
- Acquired Slot: The number of slots after the rescale
```
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
+This primarily includes the following sub-pages:
+
+- `Rescales/Overview`:
+ This sub-page displays recent rescale records across various rescale terminal state,
Review Comment:
```suggestion
This sub-page displays recent rescale records across various rescale terminal states,
```
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
+This primarily includes the following sub-pages:
+
+- `Rescales/Overview`:
+ This sub-page displays recent rescale records across various rescale terminal state,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale.
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggers the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale.
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID:The unique ID of the slot sharing group consists of 32 hexadecimal characters
Review Comment:
```suggestion
- Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
```
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
+This primarily includes the following sub-pages:
+
+- `Rescales/Overview`:
+ This sub-page displays recent rescale records across various rescale terminal state,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale.
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggers the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale.
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID:The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The previous parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The acquired parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex.
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name:The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The previous number of slots before the rescale
+ - Acquired Slot: The acquired number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: The duration time from enter time to leave time of the state
+ - Exception: The exception information about current rescale during the state
+- `Rescales/Summary`
+ This sub-page displays the total number of rescale events that have occurred since the job was launched,
+ along with the respective counts of failures and successes.
+ Additionally, it provides statistical summaries of the rescale history,
+ such as rescale duration statistics categorized by rescale status, including `Min`, `Max`, `Avg`, and `Pnn` metrics.
+- `Rescales/Configuration`
+ This sub-page displays the relevant parameter values used by the Adaptive Scheduler during rescaling operations for the current streaming job.
+
+### How to use it
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- `web.adaptive-scheduler.rescale-history.size`: `4`
+
+### More details
+
+See the <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> for more details.
Review Comment:
```suggestion
See the [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) for more details.
```
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
+This primarily includes the following sub-pages:
+
+- `Rescales/Overview`:
+ This sub-page displays recent rescale records across various rescale terminal state,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale.
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggers the target rescale
Review Comment:
```suggestion
- Trigger Cause: The reason that triggered the target rescale
```
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
+This primarily includes the following sub-pages:
+
+- `Rescales/Overview`:
Review Comment:
```suggestion
- `Overview`:
```
We could let go of `Rescales/` because we have mentioned the are subpages. Also for the others in the list. Not very strong on this, you can choose which look apt.
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
+This primarily includes the following sub-pages:
+
+- `Rescales/Overview`:
+ This sub-page displays recent rescale records across various rescale terminal state,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale.
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggers the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale.
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID:The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The previous parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The acquired parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex.
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name:The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The previous number of slots before the rescale
+ - Acquired Slot: The acquired number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: The duration time from enter time to leave time of the state
Review Comment:
```suggestion
- Duration: Time spent in the state (Leave Time − Enter Time)
```
Little tautological
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced <a href="https://cwiki.apache.org/confluence/x/TQr0Ew">FLIP-495</a> to support recording and storing rescaling history,
+and <a href="https://cwiki.apache.org/confluence/x/vZCMEw">FLIP-487</a> to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a likely style.
+This primarily includes the following sub-pages:
+
+- `Rescales/Overview`:
+ This sub-page displays recent rescale records across various rescale terminal state,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale.
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggers the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale.
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale.
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID:The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The previous parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The acquired parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex.
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name:The name of the slot sharing group to which the slot belongs
Review Comment:
```suggestion
- Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3129726298
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`:
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
Review Comment:
Thanks @spuru9
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130574370
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
Review Comment:
I am not sure what this sentence means, this is one ID but it talks of `a rescale attempts` is it one or more attempts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3130580829
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
Review Comment:
is this a UUID like the other one ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #28003:
URL: https://github.com/apache/flink/pull/28003#issuecomment-4323800454
This PR hasn’t received any new comments for four days. If this continues, I plan to merge it on Tuesday.
Thanks all involved in the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3153766322
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,82 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale</u>
+ - <u>Rescale UUID</u>: The unique ID in a rescale consists of 32 hexadecimal characters (The UUID definition below is identical to the one here)
Review Comment:
I realize that we do not use the punctuation consistently in the enumeration. Can we add periods to the end of the sentences everywhere to make it consistent?
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,82 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale</u>
+ - <u>Rescale UUID</u>: The unique ID in a rescale consists of 32 hexadecimal characters (The UUID definition below is identical to the one here)
+ - <u>Attempt ID</u>: The number of rescale attempts triggered on the same job resource requirements
+ - <u>Requirements ID</u>: The unique UUID of resource requirements
+ - <u>Trigger Cause</u>: The reason that triggered a rescale
+ - <u>Terminal State</u>: The end state of a rescale
+ - <u>Terminated Reason</u>: The reason of the rescale lifecycle termination
+ - <u>Start Time</u>: The start time of a rescale
+ - <u>Duration</u>: Duration from the start of the rescale to its completion or until now if the rescale operation hasn't completed, yet.
+ - <u>End Time</u>: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - <u>ID</u>: The unique UUID of target `Job Vertex`
+ - <u>Name</u>: The short name of target vertex
+ - <u>Slot Sharing Group ID</u>: The unique UUID of target `Slot Sharing Group`
+ - <u>Previous Parallelism</u>: The parallelism of target vertex before the current rescale
+ - <u>Acquired Parallelism</u>: The parallelism of target vertex after the current rescale
+ - <u>Sufficient Parallelism</u>: The minimal parallelism of the vertex that would have allow the rescale operation to go through even if the desired parallelism wasn't possible to be reached
Review Comment:
```suggestion
- <u>Sufficient Parallelism</u>: The minimal parallelism of the vertex that would have allowed the rescale operation to go through even if the desired parallelism wasn't possible to be reached
```
sorry, that was my mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #28003:
URL: https://github.com/apache/flink/pull/28003#issuecomment-4328484860
Thank you @XComp very much for your review.
Updated.
PTAL if you had the free time.
any input is appreciated 🌹
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3154219550
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,82 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale</u>
+ - <u>Rescale UUID</u>: The unique ID in a rescale consists of 32 hexadecimal characters (The UUID definition below is identical to the one here)
+ - <u>Attempt ID</u>: The number of rescale attempts triggered on the same job resource requirements
+ - <u>Requirements ID</u>: The unique UUID of resource requirements
+ - <u>Trigger Cause</u>: The reason that triggered a rescale
+ - <u>Terminal State</u>: The end state of a rescale
+ - <u>Terminated Reason</u>: The reason of the rescale lifecycle termination
+ - <u>Start Time</u>: The start time of a rescale
+ - <u>Duration</u>: Duration from the start of the rescale to its completion or until now if the rescale operation hasn't completed, yet.
+ - <u>End Time</u>: The end time of a rescale if the rescale is terminated, current time else.
+ - The basic attributes and rescale change per `Job Vertex`
+ - <u>ID</u>: The unique UUID of target `Job Vertex`
+ - <u>Name</u>: The short name of target vertex
+ - <u>Slot Sharing Group ID</u>: The unique UUID of target `Slot Sharing Group`
+ - <u>Previous Parallelism</u>: The parallelism of target vertex before the current rescale
+ - <u>Acquired Parallelism</u>: The parallelism of target vertex after the current rescale
+ - <u>Sufficient Parallelism</u>: The minimal parallelism of the vertex that would have allow the rescale operation to go through even if the desired parallelism wasn't possible to be reached
Review Comment:
No worries.
Thank you @XComp
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3154216764
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,82 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of a rescale</u>
+ - <u>Rescale UUID</u>: The unique ID in a rescale consists of 32 hexadecimal characters (The UUID definition below is identical to the one here)
Review Comment:
Nice catch and proposal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "och5351 (via GitHub)" <gi...@apache.org>.
och5351 commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3158880523
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,81 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+You can enable rescale history for stream jobs with the Adaptive Scheduler enabled by setting the following configuration item to a positive integer.
+This value indicates the number of recent rescale records retained for the job.
+
+- [`web.adaptive-scheduler.rescale-history.size`]({{< ref "docs/deployment/config" >}}#web-adaptive-scheduler-rescale-history-size): `4`
+
+The default value of the configuration option is `0`. When the configuration value is less than or equal to `0`, this feature will be disabled.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
Review Comment:
In this part, while most people might already know this, how about changing it to be more explicit, like:
```suggestion
This sub-page displays abbreviated information for the most recent rescale records (up to the configured `web.adaptive-scheduler.rescale-history.size` limit).
```
I think it makes the configuration key more visible and easier to reference for users who may not be familiar with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132236591
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
+ - State: The scheduler state name
+ - Enter Time: The time to enter the state
+ - Leave Time: The time to leave the state
+ - Duration: Time spent in the state (Leave Time − Enter Time)
+ - Exception: The exception information about current rescale during the state
+- `Summary`
+ This sub-page displays the total number of rescale events that have occurred since the job was launched,
+ along with the respective counts of failures and successes.
+ Additionally, it provides statistical summaries of the rescale history,
+ such as rescale duration statistics categorized by rescale status, including `Min`, `Max`, `Avg`, and `Pnn` metrics.
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132179000
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
+ - Requirements ID: The unique ID of resource requirements consists of 32 hexadecimal characters
+ - Trigger Cause: The reason that triggered the target rescale
+ - Terminal State: The end state of the target rescale
+ - Terminated Reason: The reason for the completion or termination of the target rescale
+ - Start Time: The start time of the target rescale
+ - Duration: Duration from the start of the rescale to its completion or until now
+ - End Time: The end time of the target rescale
+ - The basic attributes and rescale change per `Job Vertex`
+ - ID: The unique ID of target JobVertex consists of 32 hexadecimal characters
+ - Name: The short name of target vertex
+ - Slot Sharing Group ID: The unique ID of the slot sharing group consists of 32 hexadecimal characters
+ - Previous Parallelism: The parallelism of target vertex before the current rescale
+ - Acquired Parallelism: The parallelism of target vertex after the current rescale
+ - Sufficient Parallelism: The minimal parallelism of target vertex to run
+ - Desired Parallelism: The desired parallelism of the target vertex
+ - The basic attributes and rescale change per `Slot Sharing Group`
+ - Slot Sharing Group ID: The ID of the slot sharing group to which the slot belongs consists of 32 hexadecimal characters
+ - Slot Sharing Group Name: The name of the slot sharing group to which the slot belongs
+ - Previous Slot: The number of slots before the rescale
+ - Acquired Slot: The number of slots after the rescale
+ - Desired Slot: The desired number of slots of the rescale
+ - Sufficient Slot: The minimal number of slots to deploy tasks in the rescale
+ - Request Profile: The request resource profile of the slot sharing group in the rescale
+ - Acquired Profile: The acquired resource profile of the slot sharing group in the rescale
+ - The internal `Scheduler State History` of `Adaptive Scheduler` within a rescale
Review Comment:
these states are an abstract representation of the scheduler's internal state; therefore, they were omitted to avoid introducing unnecessary ambiguity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3129530935
##########
docs/content/docs/deployment/elastic_scaling.md:
##########
@@ -198,4 +198,80 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`:
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `Rescales/History`
Review Comment:
@RocMarshal Can you implement the suggestion for removing the substructure if done for Overview for others as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38902][docs] Add user instructions and usage documentation for FLIP-487 [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #28003:
URL: https://github.com/apache/flink/pull/28003#discussion_r3132072038
##########
docs/content.zh/docs/deployment/elastic_scaling.md:
##########
@@ -189,4 +189,81 @@ cp ./examples/streaming/TopSpeedWindowing.jar lib/
仅支持如下的部署方式：[Application 模式下的 Standalone 部署]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode)（可以参考[上文](#getting-started)）、[Application 模式下的 Docker 部署]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) 以及 [Standalone 的 Kubernetes Application 集群模式]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster)。
[Adaptive 调度器的局限性](#limitations-1) 同样也适用于 Reactive 模式.
+
+## Rescale History
+
+Before Flink 2.3, users and developers were unable to inspect the internal details of Adaptive Scheduler rescaling history,
+causing operational inconvenience.
+For instance, users need visibility into specific resource changes, parallelism adjustments,
+and the time spent on each internal state transition during the rescaling process.
+This information is crucial for tuning parameters to achieve lower latency and higher stability in rescaling.
+
+Therefore, Flink community introduced [FLIP-495](https://cwiki.apache.org/confluence/x/TQr0Ew) to support recording and storing rescaling history,
+and [FLIP-487](https://cwiki.apache.org/confluence/x/vZCMEw) to enable querying via the REST API and displaying this history on the Web UI.
+
+### The Information and Style About Rescale History
+
+Since Flink version 2.3, a page for displaying `Rescales` has been introduced in the Web UI,
+positioned at the same hierarchical level as the `Checkpoints` page and featuring a similar style.
+This primarily includes the following sub-pages:
+
+- `Overview`
+ This sub-page displays recent rescale records across various rescale terminal states,
+ along with fundamental job rescale statistics—such as the total number of rescales since job startup and the counts of failures or successes.
+ Additionally, the page supports the display of detailed rescale information.
+
+- `History`
+ This sub-page displays abbreviated information for the most recent `n` rescale records based on configuration.
+ Additionally, the page supports the display of detailed rescale information as outlined below:
+ - The basic information of the target rescale
+ - Rescale UUID: The unique ID in a rescale consists of 32 hexadecimal characters
+ - Attempt ID: The number ID of a rescale attempts that occurred under the same resource requirements
Review Comment:
Thanks @davidradl
The detailed description of the rescaleIdInfo is here.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-RescaleID&ResourceRequirementsrequest
Do you think we should keep the picture mentioned in the doc link into the doc ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org