You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/06/28 03:58:11 UTC
[GitHub] [flink] zhuzhurk opened a new pull request, #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
zhuzhurk opened a new pull request, #20087:
URL: https://github.com/apache/flink/pull/20087
## What is the purpose of the change
This PR adds config options for speculative scheduler.
It is needed to unblock the development of blocklist and speculative scheduler.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on PR #20087:
URL: https://github.com/apache/flink/pull/20087#issuecomment-1169577879
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a diff in pull request #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on code in PR #20087:
URL: https://github.com/apache/flink/pull/20087#discussion_r910779628
##########
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java:
##########
@@ -593,6 +593,40 @@ public enum SchedulerType {
code(SchedulerType.AdaptiveBatch.name()))
.build());
+ @Documentation.Section({
+ Documentation.Sections.EXPERT_SCHEDULING,
+ Documentation.Sections.ALL_JOB_MANAGER
+ })
+ public static final ConfigOption<Boolean> SPECULATIVE_ENABLED =
Review Comment:
I prefer to keep the option name not too long, as long as the we can identify it is part of speculative execution.
I will rename `BLOCK_SLOW_NODE_DURATION` to `SPECULATIVE_BLOCK_SLOW_NODE_DURATION`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk closed pull request #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
Posted by GitBox <gi...@apache.org>.
zhuzhurk closed pull request #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
URL: https://github.com/apache/flink/pull/20087
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a diff in pull request #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on code in PR #20087:
URL: https://github.com/apache/flink/pull/20087#discussion_r910774779
##########
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java:
##########
@@ -593,6 +593,40 @@ public enum SchedulerType {
code(SchedulerType.AdaptiveBatch.name()))
.build());
+ @Documentation.Section({
+ Documentation.Sections.EXPERT_SCHEDULING,
+ Documentation.Sections.ALL_JOB_MANAGER
+ })
+ public static final ConfigOption<Boolean> SPECULATIVE_ENABLED =
+ key("jobmanager.adaptive-batch-scheduler.speculative.enabled")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Controls whether to enable speculative execution.");
+
+ @Documentation.Section({
+ Documentation.Sections.EXPERT_SCHEDULING,
+ Documentation.Sections.ALL_JOB_MANAGER
+ })
+ public static final ConfigOption<Integer> SPECULATIVE_MAX_CONCURRENT_EXECUTIONS =
+ key("jobmanager.adaptive-batch-scheduler.speculative.max-concurrent-executions")
+ .intType()
+ .defaultValue(2)
+ .withDescription(
+ "Controls the maximum number of execution attempts of each operator, "
+ + "including the original one and speculative ones, that can "
+ + "execute concurrently.");
+
Review Comment:
Maybe "Controls the maximum number of execution attempts of each operator that can execute concurrently, including the original one and speculative ones"
##########
flink-core/src/main/java/org/apache/flink/configuration/JobManagerOptions.java:
##########
@@ -593,6 +593,40 @@ public enum SchedulerType {
code(SchedulerType.AdaptiveBatch.name()))
.build());
+ @Documentation.Section({
+ Documentation.Sections.EXPERT_SCHEDULING,
+ Documentation.Sections.ALL_JOB_MANAGER
+ })
+ public static final ConfigOption<Boolean> SPECULATIVE_ENABLED =
Review Comment:
I perfer to add a `ADAPTIVE_BATCH_SCHEDULER` prefix, the same goes for the other two options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #20087:
URL: https://github.com/apache/flink/pull/20087#issuecomment-1168192437
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4c4fb962bcea34f1f94b149ded8123d3f76131c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4c4fb962bcea34f1f94b149ded8123d3f76131c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4c4fb962bcea34f1f94b149ded8123d3f76131c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on PR #20087:
URL: https://github.com/apache/flink/pull/20087#issuecomment-1172010152
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on PR #20087:
URL: https://github.com/apache/flink/pull/20087#issuecomment-1171949278
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org