You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/03/24 07:11:17 UTC
[GitHub] [flink] rmetzger opened a new pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
rmetzger opened a new pull request #15355:
URL: https://github.com/apache/flink/pull/15355
## What is the purpose of the change
Describe how to use adaptive scheduler + some general improvements on the page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-805566957
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit c1fe86f51da51fe354c044ef5af546083c58c338 (Wed Mar 24 07:21:56 UTC 2021)
✅no warnings
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-805578029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348",
"triggerID" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"triggerType" : "PUSH"
}, {
"hash" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1fe86f51da51fe354c044ef5af546083c58c338 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348)
* 0abc610ce3d013b86e67d48263f1811874bb11c6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger closed pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
rmetzger closed pull request #15355:
URL: https://github.com/apache/flink/pull/15355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-805578029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348",
"triggerID" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1fe86f51da51fe354c044ef5af546083c58c338 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-805578029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348",
"triggerID" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"triggerType" : "PUSH"
}, {
"hash" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15406",
"triggerID" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"triggerType" : "PUSH"
}, {
"hash" : "765978b6268a9f3dfd4af55a7c80d595806458ab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "765978b6268a9f3dfd4af55a7c80d595806458ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1fe86f51da51fe354c044ef5af546083c58c338 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348)
* 0abc610ce3d013b86e67d48263f1811874bb11c6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15406)
* 765978b6268a9f3dfd4af55a7c80d595806458ab UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-806397586
Thanks a lot for your review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #15355:
URL: https://github.com/apache/flink/pull/15355#discussion_r600803890
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -100,13 +110,40 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
- **Deployment is only supported as a standalone application deployment**. Active resource providers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications.
The only supported deployment options are [Standalone in Application Mode]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode) ([described](#getting-started) on this page), [Docker in Application Mode]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) and [Standalone Kubernetes Application Cluster]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster).
-- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used.
-- **No support for [local recovery]({{< ref "docs/ops/state/large_state_tuning">}}#task-local-recovery)**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to download the entire state from the checkpoint storage.
-- **No support for local failover**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitation impacts only recovery time of embarrassingly parallel jobs: Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job.
-- **Limited integration with Flink's Web UI**: Reactive Mode allows that a job's parallelism can change over its lifetime. The web UI only shows the current parallelism the job.
-- **Limited Job metrics**: With the exception of `numRestarts` all [availability]({{< ref "docs/ops/metrics" >}}#availability) and [checkpointing]({{< ref "docs/ops/metrics" >}}#checkpointing) metrics with the `Job` scope are not working correctly.
+The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+
+
+## Adaptive Scheduler
+
+{{< hint danger >}}
+Using Adaptive Scheduler directly (not through Reactive Mode) is only advised for advanced users.
+{{< /hint >}}
+
+Adaptive Scheduler is a scheduler that can adjust the parallelism of a job based on the available slots. On start up, it requests the number of slots needed based on the parallelisms configured by the user in the streaming job. If the number of slots offered is lower than the requested slots, Adaptive Scheduler will reduce the parallelism so that it can start executing the job (or fail if insufficient slots are available). In Reactive Mode (see above) the parallelism requested is conceptually set to infinity, letting the job always use as many resources as possible. You can also use Adaptive Scheduler without Reactive Mode, but there are some practical limitations:
+- If you are using Adaptive Scheduler on a session cluster, there are no guarantees regarding the distribution of slots between multiple running jobs in the same session.
+- An active resource manager (native Kubernetes, YARN, Mesos) will request TaskManagers until the parallelism requested by the job is fulfilled, potentially allocating a lot of resources.
+One benefit of Adpative Scheduler over the default scheduler is that it can handle TaskManager losses gracefully, since it would just scale down in these cases.
+
+### Usage
+
+The following configuration parameters need to be set:
+
+- `jobmanager.scheduler: adaptive`: Change from the default scheduler to adaptive scheduler
+- `cluster.declarative-resource-management.enabled` Declarative resource management must be enabled (enabled by default).
+
+Depending on your usage scenario, we also recommend adjusting the parallelism of the job you are submitting to the adaptive scheduler. The parallelism configured determines the number of slots Adaptive Scheduler will request.
Review comment:
from the introduction of adaptive scheduler, it is probably clear what the parallelism does. I'll remove that sentence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-805578029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348",
"triggerID" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"triggerType" : "PUSH"
}, {
"hash" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15406",
"triggerID" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1fe86f51da51fe354c044ef5af546083c58c338 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348)
* 0abc610ce3d013b86e67d48263f1811874bb11c6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15406)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #15355:
URL: https://github.com/apache/flink/pull/15355#discussion_r600756628
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -88,10 +88,20 @@ If you manually set a parallelism in your job for individual operators or the en
Note that such a high maxParallelism might affect performance of the job, since more internal structures are needed to maintain [some internal structures](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html) of Flink.
+When enabling Reactive Mode, the `jobmanager.adaptive-scheduler.resource-wait-timeout` configuration key will default to `-1`. This means that the JobManager will run forever waiting for sufficient resources.
+If you want the JobManager to stop after a certain time without enough TaskManagers to run the job, configure `jobmanager.adaptive-scheduler.resource-wait-timeout`.
+
+With Reactive Mode enabled, the `jobmanager.adaptive-scheduler.resource-stabilization-timeout` configuration key will default to `0`: Flink will start runnning the job, as soon as there are sufficient resources available.
+In scenarios where TaskManagers are not connecting at the same time, but slowly one after another, this behavior leads to a job restart whenever a TaskManager connects. Increase this configuration value if you want to wait for the resources to stabilize before scheduling the job.
+
#### Recommendations
- **Configure periodic checkpointing for stateful jobs**: Reactive mode restores from the latest completed checkpoint on a rescale event. If no periodic checkpointing is enabled, your program will loose its state. Checkpointing also configures a **restart strategy**. Reactive mode will respect the configured restarting strategy: If no restarting strategy is configured, reactive mode will fail your job, instead of scaling it.
+- Downscaling in Reactive Mode might cause longer stalls in your processing because Flink waits for the heartbeat between JobManager and the stopped TaskManager(s) to time-out. You will see that your Flink job is stuck in the failing state for roughly 50 seconds before redeploying your job with a lower parallelism.
Review comment:
Thanks for the explanation with the states. I removed the reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-805578029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1fe86f51da51fe354c044ef5af546083c58c338 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #15355:
URL: https://github.com/apache/flink/pull/15355#discussion_r600745988
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -88,10 +88,20 @@ If you manually set a parallelism in your job for individual operators or the en
Note that such a high maxParallelism might affect performance of the job, since more internal structures are needed to maintain [some internal structures](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html) of Flink.
+When enabling Reactive Mode, the `jobmanager.adaptive-scheduler.resource-wait-timeout` configuration key will default to `-1`. This means that the JobManager will run forever waiting for sufficient resources.
+If you want the JobManager to stop after a certain time without enough TaskManagers to run the job, configure `jobmanager.adaptive-scheduler.resource-wait-timeout`.
+
+With Reactive Mode enabled, the `jobmanager.adaptive-scheduler.resource-stabilization-timeout` configuration key will default to `0`: Flink will start runnning the job, as soon as there are sufficient resources available.
+In scenarios where TaskManagers are not connecting at the same time, but slowly one after another, this behavior leads to a job restart whenever a TaskManager connects. Increase this configuration value if you want to wait for the resources to stabilize before scheduling the job.
+
#### Recommendations
- **Configure periodic checkpointing for stateful jobs**: Reactive mode restores from the latest completed checkpoint on a rescale event. If no periodic checkpointing is enabled, your program will loose its state. Checkpointing also configures a **restart strategy**. Reactive mode will respect the configured restarting strategy: If no restarting strategy is configured, reactive mode will fail your job, instead of scaling it.
+- Downscaling in Reactive Mode might cause longer stalls in your processing because Flink waits for the heartbeat between JobManager and the stopped TaskManager(s) to time-out. You will see that your Flink job is stuck in the failing state for roughly 50 seconds before redeploying your job with a lower parallelism.
+
+ The default timeout is configured to 50 seconds. Adjust the [`heartbeat.timeout`]({{< ref "docs/deployment/config">}}#heartbeat-timeout) configuration to a lower value, if your infrastructure permits this. Setting a low heartbeat timeout can lead to failures if a TaskManager fails to respond to a heartbeat, for example due to a network congestion or a long garbage collection pause. Note that the [`heartbeat.interval`]({{< ref "docs/deployment/config">}}#heartbeat-interval) always needs to be lower than the timeout.
Review comment:
Yes, I wanted to have a linebreak in the bullet item
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-805578029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348",
"triggerID" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"triggerType" : "PUSH"
}, {
"hash" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15406",
"triggerID" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"triggerType" : "PUSH"
}, {
"hash" : "765978b6268a9f3dfd4af55a7c80d595806458ab",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15408",
"triggerID" : "765978b6268a9f3dfd4af55a7c80d595806458ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0abc610ce3d013b86e67d48263f1811874bb11c6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15406)
* 765978b6268a9f3dfd4af55a7c80d595806458ab Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15408)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #15355:
URL: https://github.com/apache/flink/pull/15355#discussion_r600245551
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -100,13 +110,40 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
- **Deployment is only supported as a standalone application deployment**. Active resource providers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications.
The only supported deployment options are [Standalone in Application Mode]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode) ([described](#getting-started) on this page), [Docker in Application Mode]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) and [Standalone Kubernetes Application Cluster]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster).
-- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used.
-- **No support for [local recovery]({{< ref "docs/ops/state/large_state_tuning">}}#task-local-recovery)**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to download the entire state from the checkpoint storage.
-- **No support for local failover**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitation impacts only recovery time of embarrassingly parallel jobs: Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job.
-- **Limited integration with Flink's Web UI**: Reactive Mode allows that a job's parallelism can change over its lifetime. The web UI only shows the current parallelism the job.
-- **Limited Job metrics**: With the exception of `numRestarts` all [availability]({{< ref "docs/ops/metrics" >}}#availability) and [checkpointing]({{< ref "docs/ops/metrics" >}}#checkpointing) metrics with the `Job` scope are not working correctly.
+The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+
+
+## Adaptive Scheduler
+
+{{< hint danger >}}
+Using Adaptive Scheduler directly (not through Reactive Mode) is only advised for advanced users.
+{{< /hint >}}
+
+Adaptive Scheduler is a scheduler that can adjust the parallelism of a job based on the available slots. On start up, it requests the number of slots needed based on the parallelisms configured by the user in the streaming job. If the number of slots offered is lower than the requested slots, Adaptive Scheduler will reduce the parallelism so that it can start executing the job (or fail if insufficient slots are available). In Reactive Mode (see above) the parallelism requested is conceptually set to infinity, letting the job always use as many resources as possible. You can also use Adaptive Scheduler without Reactive Mode, but there are some practical limitations:
Review comment:
```suggestion
The Adaptive Scheduler can adjust the parallelism of a job based on available slots. It will automatically reduce the parallelism if not enough slots are available to run the job with the originally configured parallelism; be it due to not enough resources being available at the time of submission, or TaskManager outages during the job execution. If new slots become available the job will be scaled up again, up to the configured parallelism.
In Reactive Mode (see above) the configured parallelism is ignored and treated as if it was set to infinity, letting the job always use as many resources as possible.
You can also use Adaptive Scheduler without Reactive Mode, but there are some practical limitations:
```
I think this goes too far into implementation details of the declarative slot protocol.
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -100,13 +110,40 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
- **Deployment is only supported as a standalone application deployment**. Active resource providers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications.
The only supported deployment options are [Standalone in Application Mode]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode) ([described](#getting-started) on this page), [Docker in Application Mode]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) and [Standalone Kubernetes Application Cluster]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster).
-- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used.
-- **No support for [local recovery]({{< ref "docs/ops/state/large_state_tuning">}}#task-local-recovery)**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to download the entire state from the checkpoint storage.
-- **No support for local failover**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitation impacts only recovery time of embarrassingly parallel jobs: Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job.
-- **Limited integration with Flink's Web UI**: Reactive Mode allows that a job's parallelism can change over its lifetime. The web UI only shows the current parallelism the job.
-- **Limited Job metrics**: With the exception of `numRestarts` all [availability]({{< ref "docs/ops/metrics" >}}#availability) and [checkpointing]({{< ref "docs/ops/metrics" >}}#checkpointing) metrics with the `Job` scope are not working correctly.
+The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+
+
+## Adaptive Scheduler
+
+{{< hint danger >}}
+Using Adaptive Scheduler directly (not through Reactive Mode) is only advised for advanced users.
+{{< /hint >}}
+
+Adaptive Scheduler is a scheduler that can adjust the parallelism of a job based on the available slots. On start up, it requests the number of slots needed based on the parallelisms configured by the user in the streaming job. If the number of slots offered is lower than the requested slots, Adaptive Scheduler will reduce the parallelism so that it can start executing the job (or fail if insufficient slots are available). In Reactive Mode (see above) the parallelism requested is conceptually set to infinity, letting the job always use as many resources as possible. You can also use Adaptive Scheduler without Reactive Mode, but there are some practical limitations:
+- If you are using Adaptive Scheduler on a session cluster, there are no guarantees regarding the distribution of slots between multiple running jobs in the same session.
+- An active resource manager (native Kubernetes, YARN, Mesos) will request TaskManagers until the parallelism requested by the job is fulfilled, potentially allocating a lot of resources.
Review comment:
How is this a limitation of the adaptive scheduler? The default scheduler has the same behavior.
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -100,13 +110,40 @@ Since Reactive Mode is a new, experimental feature, not all features supported b
- **Deployment is only supported as a standalone application deployment**. Active resource providers (such as native Kubernetes, YARN or Mesos) are explicitly not supported. Standalone session clusters are not supported either. The application deployment is limited to single job applications.
The only supported deployment options are [Standalone in Application Mode]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}#application-mode) ([described](#getting-started) on this page), [Docker in Application Mode]({{< ref "docs/deployment/resource-providers/standalone/docker" >}}#application-mode-on-docker) and [Standalone Kubernetes Application Cluster]({{< ref "docs/deployment/resource-providers/standalone/kubernetes" >}}#deploy-application-cluster).
-- **Streaming jobs only**: The first version of Reactive Mode runs with streaming jobs only. When submitting a batch job, then the default scheduler will be used.
-- **No support for [local recovery]({{< ref "docs/ops/state/large_state_tuning">}}#task-local-recovery)**: Local recovery is a feature that schedules tasks to machines so that the state on that machine gets re-used if possible. The lack of this feature means that Reactive Mode will always need to download the entire state from the checkpoint storage.
-- **No support for local failover**: Local failover means that the scheduler is able to restart parts ("regions" in Flink's internals) of a failed job, instead of the entire job. This limitation impacts only recovery time of embarrassingly parallel jobs: Flink's default scheduler can restart failed parts, while Reactive Mode will restart the entire job.
-- **Limited integration with Flink's Web UI**: Reactive Mode allows that a job's parallelism can change over its lifetime. The web UI only shows the current parallelism the job.
-- **Limited Job metrics**: With the exception of `numRestarts` all [availability]({{< ref "docs/ops/metrics" >}}#availability) and [checkpointing]({{< ref "docs/ops/metrics" >}}#checkpointing) metrics with the `Job` scope are not working correctly.
+The [limitations of Adaptive Scheduler](#limitations-1) also apply to Reactive Mode.
+
+
+## Adaptive Scheduler
+
+{{< hint danger >}}
+Using Adaptive Scheduler directly (not through Reactive Mode) is only advised for advanced users.
+{{< /hint >}}
+
+Adaptive Scheduler is a scheduler that can adjust the parallelism of a job based on the available slots. On start up, it requests the number of slots needed based on the parallelisms configured by the user in the streaming job. If the number of slots offered is lower than the requested slots, Adaptive Scheduler will reduce the parallelism so that it can start executing the job (or fail if insufficient slots are available). In Reactive Mode (see above) the parallelism requested is conceptually set to infinity, letting the job always use as many resources as possible. You can also use Adaptive Scheduler without Reactive Mode, but there are some practical limitations:
+- If you are using Adaptive Scheduler on a session cluster, there are no guarantees regarding the distribution of slots between multiple running jobs in the same session.
+- An active resource manager (native Kubernetes, YARN, Mesos) will request TaskManagers until the parallelism requested by the job is fulfilled, potentially allocating a lot of resources.
+One benefit of Adpative Scheduler over the default scheduler is that it can handle TaskManager losses gracefully, since it would just scale down in these cases.
+
+### Usage
+
+The following configuration parameters need to be set:
+
+- `jobmanager.scheduler: adaptive`: Change from the default scheduler to adaptive scheduler
+- `cluster.declarative-resource-management.enabled` Declarative resource management must be enabled (enabled by default).
+
+Depending on your usage scenario, we also recommend adjusting the parallelism of the job you are submitting to the adaptive scheduler. The parallelism configured determines the number of slots Adaptive Scheduler will request.
Review comment:
In what circumstances would a 1.12 user that wants to use the adaptive scheduler adjust the parallelism, and why?
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -88,10 +88,20 @@ If you manually set a parallelism in your job for individual operators or the en
Note that such a high maxParallelism might affect performance of the job, since more internal structures are needed to maintain [some internal structures](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html) of Flink.
+When enabling Reactive Mode, the `jobmanager.adaptive-scheduler.resource-wait-timeout` configuration key will default to `-1`. This means that the JobManager will run forever waiting for sufficient resources.
+If you want the JobManager to stop after a certain time without enough TaskManagers to run the job, configure `jobmanager.adaptive-scheduler.resource-wait-timeout`.
+
+With Reactive Mode enabled, the `jobmanager.adaptive-scheduler.resource-stabilization-timeout` configuration key will default to `0`: Flink will start runnning the job, as soon as there are sufficient resources available.
+In scenarios where TaskManagers are not connecting at the same time, but slowly one after another, this behavior leads to a job restart whenever a TaskManager connects. Increase this configuration value if you want to wait for the resources to stabilize before scheduling the job.
+
#### Recommendations
- **Configure periodic checkpointing for stateful jobs**: Reactive mode restores from the latest completed checkpoint on a rescale event. If no periodic checkpointing is enabled, your program will loose its state. Checkpointing also configures a **restart strategy**. Reactive mode will respect the configured restarting strategy: If no restarting strategy is configured, reactive mode will fail your job, instead of scaling it.
+- Downscaling in Reactive Mode might cause longer stalls in your processing because Flink waits for the heartbeat between JobManager and the stopped TaskManager(s) to time-out. You will see that your Flink job is stuck in the failing state for roughly 50 seconds before redeploying your job with a lower parallelism.
+
+ The default timeout is configured to 50 seconds. Adjust the [`heartbeat.timeout`]({{< ref "docs/deployment/config">}}#heartbeat-timeout) configuration to a lower value, if your infrastructure permits this. Setting a low heartbeat timeout can lead to failures if a TaskManager fails to respond to a heartbeat, for example due to a network congestion or a long garbage collection pause. Note that the [`heartbeat.interval`]({{< ref "docs/deployment/config">}}#heartbeat-interval) always needs to be lower than the timeout.
Review comment:
```suggestion
The default timeout is configured to 50 seconds. Adjust the [`heartbeat.timeout`]({{< ref "docs/deployment/config">}}#heartbeat-timeout) configuration to a lower value, if your infrastructure permits this. Setting a low heartbeat timeout can lead to failures if a TaskManager fails to respond to a heartbeat, for example due to a network congestion or a long garbage collection pause. Note that the [`heartbeat.interval`]({{< ref "docs/deployment/config">}}#heartbeat-interval) always needs to be lower than the timeout.
```
Or is this intentional?
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -88,10 +88,20 @@ If you manually set a parallelism in your job for individual operators or the en
Note that such a high maxParallelism might affect performance of the job, since more internal structures are needed to maintain [some internal structures](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html) of Flink.
+When enabling Reactive Mode, the `jobmanager.adaptive-scheduler.resource-wait-timeout` configuration key will default to `-1`. This means that the JobManager will run forever waiting for sufficient resources.
+If you want the JobManager to stop after a certain time without enough TaskManagers to run the job, configure `jobmanager.adaptive-scheduler.resource-wait-timeout`.
+
+With Reactive Mode enabled, the `jobmanager.adaptive-scheduler.resource-stabilization-timeout` configuration key will default to `0`: Flink will start runnning the job, as soon as there are sufficient resources available.
+In scenarios where TaskManagers are not connecting at the same time, but slowly one after another, this behavior leads to a job restart whenever a TaskManager connects. Increase this configuration value if you want to wait for the resources to stabilize before scheduling the job.
+
#### Recommendations
- **Configure periodic checkpointing for stateful jobs**: Reactive mode restores from the latest completed checkpoint on a rescale event. If no periodic checkpointing is enabled, your program will loose its state. Checkpointing also configures a **restart strategy**. Reactive mode will respect the configured restarting strategy: If no restarting strategy is configured, reactive mode will fail your job, instead of scaling it.
+- Downscaling in Reactive Mode might cause longer stalls in your processing because Flink waits for the heartbeat between JobManager and the stopped TaskManager(s) to time-out. You will see that your Flink job is stuck in the failing state for roughly 50 seconds before redeploying your job with a lower parallelism.
Review comment:
```suggestion
- Downscaling in Reactive Mode might cause longer stalls in your processing because Flink waits for the heartbeat between JobManager and the stopped TaskManager(s) to time out. You will see that your Flink job is stuck in the failing state for roughly 50 seconds before redeploying your job with a lower parallelism.
```
Is `stuck in the failing state` referring to the actual job status in Flink? (asking because the job will never officially reach a FAILING state, because from there the only path is FINISHED. It will be exposed as RUNNING until the restart is triggered).
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -88,10 +88,20 @@ If you manually set a parallelism in your job for individual operators or the en
Note that such a high maxParallelism might affect performance of the job, since more internal structures are needed to maintain [some internal structures](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html) of Flink.
+When enabling Reactive Mode, the `jobmanager.adaptive-scheduler.resource-wait-timeout` configuration key will default to `-1`. This means that the JobManager will run forever waiting for sufficient resources.
+If you want the JobManager to stop after a certain time without enough TaskManagers to run the job, configure `jobmanager.adaptive-scheduler.resource-wait-timeout`.
+
+With Reactive Mode enabled, the `jobmanager.adaptive-scheduler.resource-stabilization-timeout` configuration key will default to `0`: Flink will start runnning the job, as soon as there are sufficient resources available.
+In scenarios where TaskManagers are not connecting at the same time, but slowly one after another, this behavior leads to a job restart whenever a TaskManager connects. Increase this configuration value if you want to wait for the resources to stabilize before scheduling the job.
Review comment:
Unrelated to this ticket, but these both sound like undesirable behaviors.
If we're fine with downscaling causing 30+seconds of downtime, then why can we not wait a few seconds for the stabilization?
Why is reactive mode now this special case where a cluster may sit idling indefinitely?
It seems a bit like we're optimizing the defaults for demos, and not actual production usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-805578029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348",
"triggerID" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c1fe86f51da51fe354c044ef5af546083c58c338 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
rmetzger commented on a change in pull request #15355:
URL: https://github.com/apache/flink/pull/15355#discussion_r600752855
##########
File path: docs/content.zh/docs/deployment/elastic_scaling.md
##########
@@ -88,10 +88,20 @@ If you manually set a parallelism in your job for individual operators or the en
Note that such a high maxParallelism might affect performance of the job, since more internal structures are needed to maintain [some internal structures](https://flink.apache.org/features/2017/07/04/flink-rescalable-state.html) of Flink.
+When enabling Reactive Mode, the `jobmanager.adaptive-scheduler.resource-wait-timeout` configuration key will default to `-1`. This means that the JobManager will run forever waiting for sufficient resources.
+If you want the JobManager to stop after a certain time without enough TaskManagers to run the job, configure `jobmanager.adaptive-scheduler.resource-wait-timeout`.
+
+With Reactive Mode enabled, the `jobmanager.adaptive-scheduler.resource-stabilization-timeout` configuration key will default to `0`: Flink will start runnning the job, as soon as there are sufficient resources available.
+In scenarios where TaskManagers are not connecting at the same time, but slowly one after another, this behavior leads to a job restart whenever a TaskManager connects. Increase this configuration value if you want to wait for the resources to stabilize before scheduling the job.
Review comment:
As you've said, these changes are unrelated and have been introduced in a separate PR https://github.com/apache/flink/pull/15159.
I wouldn't say we are optimizing for demos, rather for a good out of the box experience for people trying out reactive mode.
The settings are well documented, and people will/should explore reactive mode before using it in prod. I hope by then they've adjusted the settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15355: [FLINK-21076][docs] Add section about Adaptive Scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15355:
URL: https://github.com/apache/flink/pull/15355#issuecomment-805578029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15348",
"triggerID" : "c1fe86f51da51fe354c044ef5af546083c58c338",
"triggerType" : "PUSH"
}, {
"hash" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15406",
"triggerID" : "0abc610ce3d013b86e67d48263f1811874bb11c6",
"triggerType" : "PUSH"
}, {
"hash" : "765978b6268a9f3dfd4af55a7c80d595806458ab",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15408",
"triggerID" : "765978b6268a9f3dfd4af55a7c80d595806458ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 765978b6268a9f3dfd4af55a7c80d595806458ab Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15408)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org