You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by dawidwys <gi...@git.apache.org> on 2018/07/10 11:50:49 UTC
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
GitHub user dawidwys opened a pull request:
https://github.com/apache/flink/pull/6294
[FLINK-9013][docs] Document yarn.containers.vcores only being effective whe…
Added required configuration documentation on the cluster side for the parameter `yarn.containers.vcores` to take effect.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dawidwys/flink FLINK-9013
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6294.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6294
----
commit 2ccd82efa2ab109758ed96301c381c68883d1025
Author: Dawid Wysakowicz <dw...@...>
Date: 2018-07-10T09:55:39Z
[FLINK-9013] Document yarn.containers.vcores only being effective when adapting YARN config
----
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Posted by dawidwys <gi...@git.apache.org>.
Github user dawidwys commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r201399926
--- Diff: docs/ops/deployment/yarn_setup.md ---
@@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering
Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers,
some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the
--- End diff --
The way it is right now it implies there is more information on the CPU scheduling in the description of the parameter.
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Posted by dawidwys <gi...@git.apache.org>.
Github user dawidwys commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r201394304
--- Diff: docs/ops/deployment/yarn_setup.md ---
@@ -132,7 +132,10 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering
Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers,
some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the
-number of vcores with a custom value.
+number of vcores with a custom value. In order for this parameter to be used your cluster must have CPU scheduling enabled. You can do this e.g. by
+
+ * setting the `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler` or
+ * enabling `org.apache.hadoop.yarn.util.resource.DominantResourceCalculator` for `org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler`
--- End diff --
Changed it as you suggested.
---
[GitHub] flink issue #6294: [FLINK-9013][docs] Document yarn.containers.vcores only b...
Posted by dawidwys <gi...@git.apache.org>.
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/6294
@NicoK Could you have a look, as you were the one that created this issue?
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r202720265
--- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java ---
@@ -65,7 +65,11 @@
key("yarn.containers.vcores")
.defaultValue(-1)
.withDescription("The number of virtual cores (vcores) per YARN container. By default, the number of vcores" +
- " is set to the number of slots per TaskManager, if set, or to 1, otherwise.");
+ " is set to the number of slots per TaskManager, if set, or to 1, otherwise. In order for this parameter " +
+ "to be used your cluster must have CPU scheduling enabled. You can do this e.g. by setting the " +
+ "org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler or enabling " +
+ "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator for " +
+ "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler");
--- End diff --
Also, please put the configuration parameters into some code environment, e.g. `<tt>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</tt>`, for the config.html page.
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r202968456
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/description/TextElement.java ---
@@ -53,6 +55,16 @@ public static TextElement text(String text) {
return new TextElement(text, Collections.emptyList());
}
+ /**
+ * Tries to format the text as code.
+ *
+ * @return text element with applied formatting
+ */
+ public TextElement formatAsCode() {
--- End diff --
alternatively we could add an explicit `Code` `InlineElement`.
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r202719580
--- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java ---
@@ -65,7 +65,11 @@
key("yarn.containers.vcores")
.defaultValue(-1)
.withDescription("The number of virtual cores (vcores) per YARN container. By default, the number of vcores" +
- " is set to the number of slots per TaskManager, if set, or to 1, otherwise.");
+ " is set to the number of slots per TaskManager, if set, or to 1, otherwise. In order for this parameter " +
+ "to be used your cluster must have CPU scheduling enabled. You can do this e.g. by setting the " +
+ "org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler or enabling " +
+ "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator for " +
+ "org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler");
--- End diff --
Have you actually tried - back when I created this issue, simply changing to `DominantResourceCalculator` did not lead to the desired effect.
---
[GitHub] flink issue #6294: [FLINK-9013][docs] Document yarn.containers.vcores only b...
Posted by dawidwys <gi...@git.apache.org>.
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/6294
I've rebased it on top of rich formatting feature for documentation, therefore only the last commit applies to the issue.
I will also check the `DominantResourceCalculator` once again.
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r201398701
--- Diff: docs/ops/deployment/yarn_setup.md ---
@@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering
Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers,
some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the
--- End diff --
why not replace "`yarn.containers.vcores`" with the link?
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r202722162
--- Diff: docs/ops/deployment/yarn_setup.md ---
@@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering
Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers,
some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the
-number of vcores with a custom value.
+number of vcores with a custom value. In order for this parameter to work you should enable CPU scheduling in your cluster, see more [here]({{ site.baseurl }}/ops/config.html#yarn-containers-vcores)
--- End diff --
You should, however, never link on "here" - maybe put the link on "enable CPU scheduling" instead?
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r202720821
--- Diff: docs/ops/deployment/yarn_setup.md ---
@@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering
Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers,
some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the
--- End diff --
I agree with @zentol - a link here (in addition) would be better to simply click your way through the docs
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r201329612
--- Diff: docs/ops/deployment/yarn_setup.md ---
@@ -132,7 +132,10 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering
Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers,
some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the
-number of vcores with a custom value.
+number of vcores with a custom value. In order for this parameter to be used your cluster must have CPU scheduling enabled. You can do this e.g. by
+
+ * setting the `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler` or
+ * enabling `org.apache.hadoop.yarn.util.resource.DominantResourceCalculator` for `org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler`
--- End diff --
I would suggest adding this to the options documentation, and then simply link to the configuration section. (you can link to individual options!)
---