You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "huwh (via GitHub)" <gi...@apache.org> on 2023/03/20 08:26:26 UTC
[GitHub] [flink] huwh opened a new pull request, #22224: [FLINK-31448][runtime] use fine-grained slot manager as default.
huwh opened a new pull request, #22224:
URL: https://github.com/apache/flink/pull/22224
## What is the purpose of the change
Change the default slot manager to FineGrainedSlotManager
## Brief change log
- *change ci to use fine grained slot manager by default*
- *enable FineGrainedSlotManager by default*
## Verifying this change
*(example:)*
- *The default change should pass all ci pipelines*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong closed pull request #22224: [FLINK-31448][runtime] use fine-grained slot manager as default.
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong closed pull request #22224: [FLINK-31448][runtime] use fine-grained slot manager as default.
URL: https://github.com/apache/flink/pull/22224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a diff in pull request #22224: [FLINK-31448][runtime] use fine-grained slot manager as default.
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on code in PR #22224:
URL: https://github.com/apache/flink/pull/22224#discussion_r1181538748
##########
flink-core/src/main/java/org/apache/flink/configuration/ClusterOptions.java:
##########
@@ -140,7 +140,7 @@ public class ClusterOptions {
public static final ConfigOption<Boolean> ENABLE_FINE_GRAINED_RESOURCE_MANAGEMENT =
ConfigOptions.key("cluster.fine-grained-resource-management.enabled")
.booleanType()
- .defaultValue(false)
+ .defaultValue(true)
Review Comment:
In addition to changing the default value, I think we should annotate the configuration as `@Deprecated` and exclude it from documentation. Because we are expecting to remove the declarative slot manager in the next release, and this config will then become invalid.
##########
tools/ci/stage.sh:
##########
@@ -26,7 +26,7 @@ STAGE_CONNECTORS_2="connect_2"
STAGE_TESTS="tests"
STAGE_MISC="misc"
STAGE_CLEANUP="cleanup"
-STAGE_FINEGRAINED_RESOURCE_MANAGEMENT="finegrained_resource_management"
+STAGE_DECLARATIVE_RESOURCE_MANAGEMENT="declarative_resource_management"
Review Comment:
I wonder if we can simply remove this stage. I'd expect there to be barely any changes to the declarative slot manager before it gets removed. Maybe relying on the unit tests would be good enough.
##########
docs/content/docs/deployment/finegrained_resource.md:
##########
@@ -84,10 +84,12 @@ To use fine-grained resource management, you need to:
### Enable Fine-Grained Resource Management
-To enable fine-grained resource management, you need to configure the [cluster.fine-grained-resource-management.enabled]({{< ref "docs/deployment/config" >}}#cluster-fine-grained-resource-management-enabled) to true.
+The fine-grained resource management is enabled by default.
+
+To ensure this, you need to check that [cluster.fine-grained-resource-management.enabled]({{< ref "docs/deployment/config" >}}#cluster-fine-grained-resource-management-enabled) is not set to false.
{{< hint danger >}}
-Without this configuration, the Flink runtime cannot schedule the slots with your specified resource requirement and the job will fail with an exception.
+When this configuration is disabled, the Flink runtime cannot schedule the slots with your specified resource requirement and the job will fail with an exception.
{{< /hint >}}
Review Comment:
And the *Usage* section should also be updated, because "configure to enable fine-grained resource management" is no longer needed.
##########
tools/ci/stage.sh:
##########
@@ -26,7 +26,7 @@ STAGE_CONNECTORS_2="connect_2"
STAGE_TESTS="tests"
STAGE_MISC="misc"
STAGE_CLEANUP="cleanup"
-STAGE_FINEGRAINED_RESOURCE_MANAGEMENT="finegrained_resource_management"
+STAGE_DECLARATIVE_RESOURCE_MANAGEMENT="declarative_resource_management"
Review Comment:
Either way, we would need a JIRA ticket to remind us to remove declarative slot manager in the next release. A ticket with Blocker priority and 1.19/2.0 fixVersion should do the job.
##########
docs/content/docs/deployment/finegrained_resource.md:
##########
@@ -84,10 +84,12 @@ To use fine-grained resource management, you need to:
### Enable Fine-Grained Resource Management
-To enable fine-grained resource management, you need to configure the [cluster.fine-grained-resource-management.enabled]({{< ref "docs/deployment/config" >}}#cluster-fine-grained-resource-management-enabled) to true.
+The fine-grained resource management is enabled by default.
+
+To ensure this, you need to check that [cluster.fine-grained-resource-management.enabled]({{< ref "docs/deployment/config" >}}#cluster-fine-grained-resource-management-enabled) is not set to false.
{{< hint danger >}}
-Without this configuration, the Flink runtime cannot schedule the slots with your specified resource requirement and the job will fail with an exception.
+When this configuration is disabled, the Flink runtime cannot schedule the slots with your specified resource requirement and the job will fail with an exception.
{{< /hint >}}
Review Comment:
I think we can simply remove this section. Users now can use either coarse-grained or fine-grained resource management, by simply specifying fine-grained resource requirements in their (DataStream) codes or not, without having to worry about whether the feature is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] huwh commented on pull request #22224: [FLINK-31448][runtime] use fine-grained slot manager as default.
Posted by "huwh (via GitHub)" <gi...@apache.org>.
huwh commented on PR #22224:
URL: https://github.com/apache/flink/pull/22224#issuecomment-1477514720
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] huwh commented on pull request #22224: [FLINK-31448][runtime] use fine-grained slot manager as default.
Posted by "huwh (via GitHub)" <gi...@apache.org>.
huwh commented on PR #22224:
URL: https://github.com/apache/flink/pull/22224#issuecomment-1535812800
@xintongsong Thanks for the review. comments addressed.
I squash there commits to avoid causing too much pollution to the history of git commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22224: [FLINK-31448][runtime] use fine-grained slot manager as default.
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22224:
URL: https://github.com/apache/flink/pull/22224#issuecomment-1475812654
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2999833323b20329125100b26bdee5f221e9b7fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2999833323b20329125100b26bdee5f221e9b7fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2999833323b20329125100b26bdee5f221e9b7fd UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] huwh commented on a diff in pull request #22224: [FLINK-31448][runtime] use fine-grained slot manager as default.
Posted by "huwh (via GitHub)" <gi...@apache.org>.
huwh commented on code in PR #22224:
URL: https://github.com/apache/flink/pull/22224#discussion_r1185761661
##########
tools/ci/stage.sh:
##########
@@ -26,7 +26,7 @@ STAGE_CONNECTORS_2="connect_2"
STAGE_TESTS="tests"
STAGE_MISC="misc"
STAGE_CLEANUP="cleanup"
-STAGE_FINEGRAINED_RESOURCE_MANAGEMENT="finegrained_resource_management"
+STAGE_DECLARATIVE_RESOURCE_MANAGEMENT="declarative_resource_management"
Review Comment:
Ticket are created, https://issues.apache.org/jira/browse/FLINK-31449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org