You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by NicoK <gi...@git.apache.org> on 2017/01/03 11:20:54 UTC
[GitHub] flink pull request #3056: [FLINK-3150] make YARN container invocation config...
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/3056
[FLINK-3150] make YARN container invocation configurable
By using the `yarn.container-start-command-template` configuration parameter, the Flink start command can be altered/extended. By default, it is set to `"%java% %jvmmem% %jvmopts% %logging% %class% %args% %redirects%"` with
- `java` = path to the Java executable
- `jvmmem` = JVM memory limits and tweaks
- `jvmopts` = misc options for the Java VM
- `logging` = logging-related configuration settings
- `class` = main class to execute
- `args` = arguments for the main class
- `redirects` = output redirects
@rmetzger this is based on your previous pull request (https://github.com/apache/flink/pull/1446/files), can you have a look?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink FLINK-3150
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3056.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3056
----
commit f36599778d5e28401d68bbdd39d0c55e2eee3d0b
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-02T15:17:09Z
[hotfix] fix YARN TM krb5 conf only available when logging enabled
commit f112d0c46d205623712fd8cda0c798152150dfb1
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-02T16:44:38Z
[FLINK-3150] add a test for BootstrapTools#getTaskManagerShellCommand
commit deb04f971764a675b5e39a5609465f0179fa8d02
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-03T10:53:23Z
[FLINK-3150] add a test for AbstractYarnClusterDescriptor#setupApplicationMasterContainer
commit a140f34ebe728c77f8afc051b3dda0066a252bc6
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-02T16:01:21Z
[FLINK-3150] make YARN container invocation configurable
By using the `yarn.container-start-command-template` configuration parameter,
the Flink start command can be altered/extended. By default, it is set to
`"%java% %jvmmem% %jvmopts% %logging% %class% %args% %redirects%"` with
- `java` = path to the Java executable
- `jvmmem` = JVM memory limits and tweaks
- `jvmopts` = misc options for the Java VM
- `logging` = logging-related configuration settings
- `class` = main class to execute
- `args` = arguments for the main class
- `redirects` = output redirects
This is based on a previous pull request and commits by @rmetzger, i.e.
https://github.com/apache/flink/pull/1446/files
commit b17d563a626da17c82476a4b76fd62da325e4396
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-02T17:26:31Z
[FLINK-3150] fe-factor BootstrapToolsTest#testGetTaskManagerShellCommand
-> reduce duplication of strings for easier test changes
commit 6ff6724c2935956a0deea51286eb0930652a165f
Author: Nico Kruber <ni...@data-artisans.com>
Date: 2017-01-02T17:27:33Z
[FLINK-3150] extend BootstrapToolsTest#testGetTaskManagerShellCommand
This now also tests some variants changing the new
`yarn.container-start-command-template` configuration parameter.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3056: [FLINK-3150] make YARN container invocation configurable
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3056
Thank you for looking into the issue. Before I'm merging it, I'd also like to test drive the change on a real cluster :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3056: [FLINK-3150] make YARN container invocation configurable
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/3056
+1 to merge. The cluster test was successful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3056: [FLINK-3150] make YARN container invocation config...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3056
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3056: [FLINK-3150] make YARN container invocation config...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/3056#discussion_r94954355
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java ---
@@ -347,43 +351,88 @@ public static String getTaskManagerShellCommand(
boolean hasKrb5,
Class<?> mainClass) {
--- End diff --
Okay, then lets leave it as is for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3056: [FLINK-3150] make YARN container invocation config...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/3056#discussion_r94798020
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java ---
@@ -347,43 +351,88 @@ public static String getTaskManagerShellCommand(
boolean hasKrb5,
Class<?> mainClass) {
--- End diff --
There were some subtle differences in the memory setup (is that really intended?)), log and config directories and files which is why I did not generalise the setup.
It may be done, however... question is whether it is easier to grasp afterwards. Or let's worry about this when there are three setups of this kind ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3056: [FLINK-3150] make YARN container invocation config...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/3056#discussion_r94634138
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java ---
@@ -347,43 +351,88 @@ public static String getTaskManagerShellCommand(
boolean hasKrb5,
Class<?> mainClass) {
--- End diff --
Did you consider using this method also for the JobManager / ApplicationMaster container invocation in `AbstractYarnClusterDescriptor.java` ? I'm not sure if my approach is the right one, but there is a lot of shared code between the two.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---