You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rmetzger <gi...@git.apache.org> on 2016/06/28 14:00:59 UTC
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/2177
[FLINK-4127] Check API compatbility for 1.1 in flink-core
I checked all the newly introduced methods in public APIs by going through the reports generated from japicmp.
I've also put the reports (before my PR) into the JIRA: https://issues.apache.org/jira/browse/FLINK-4127
I added the new configuration parameters to the documentation, and renamed some new configuration keys.
@uce @tillrohrmann @mxm: What do you think about the renaming of the configuration keys?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink flink4127
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2177.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2177
----
commit b4073b93c9be271068faa97a139b20b9e9d6e356
Author: Robert Metzger <rm...@apache.org>
Date: 2016-06-28T13:12:05Z
[FLINK-4127] Check API compatbility for 1.1 in flink-core
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68933286
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
--- End diff --
how about renaming the prefix to `containerized`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r69457483
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINERIZED_HEAP_CUTOFF_RATIO = "containerized.heap-cutoff-ratio";
/**
* Minimum amount of heap memory to remove in containers, as a safety margin.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_MIN = "containered.heap-cutoff-min";
+ public static final String CONTAINERIZED_HEAP_CUTOFF_MIN = "containerized.heap-cutoff-min";
/**
* Prefix for passing custom environment variables to Flink's master process.
* For example for passing LD_LIBRARY_PATH as an env variable to the AppMaster, set:
* yarn.application-master.env.LD_LIBRARY_PATH: "/usr/lib/native"
* in the flink-conf.yaml.
*/
- public static final String CONTAINERED_MASTER_ENV_PREFIX = "containered.application-master.env.";
+ public static final String CONTAINERIZED_MASTER_ENV_PREFIX = "containerized.master.env.";
--- End diff --
I renamed it.
I think we misunderstood each other. I thought our discussion regarding `container` / `containered` / `containerized` was about the prefix in general, not for the heap cutoff settings alone.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/2177
I'll rename `containerized` to `container` once I got a third confirmation from a committer on this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68907634
--- Diff: docs/setup/config.md ---
@@ -107,12 +109,20 @@ Please make sure to set the maximum ticket life span high long running jobs. The
If you are on YARN, then it is sufficient to authenticate the client with Kerberos. On a Flink standalone cluster you need to ensure that, initially, all nodes are authenticated with Kerberos using the `kinit` tool.
+### Resource Manager
+
+- `resourcemanager.rpc.port`: The config parameter defining the network port to connect to for communication with the resource manager.
--- End diff --
Is it possible to specify a port range here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/2177
So you are suggesting to use the heap cutoff also for the JVM started in standalone mode?
The cutoff was introduced for YARN because there is a difference in the amount of used memory for a container we specify to YARN vs the memory the JVM is actually using.
So if we specify a container of 1000 MB, it will be killed by YARN as soon as we use 1001 MBs. For operating systems, this doesn't apply and if users run into issues, the can just set a lower heap size.
For Mesos, I don't know how it behaves, but I hope that there is a similar requirement, otherwise, we would have prematurely optimized the configuration keys.
I'm against applying this mechanism to all JVMs we start.
For the environment variables of the standalone mode, its quite easy to put the environment variables into the config.sh file, or just set the env variables.
But true, we could do a follow up JIRA to extract those env variables and set them in the config.sh script.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68906518
--- Diff: docs/setup/config.md ---
@@ -85,6 +85,8 @@ The default fraction for managed memory can be adjusted using the `taskmanager.m
- `taskmanager.memory.preallocate`: Can be either of `true` or `false`. Specifies whether task managers should allocate all managed memory when starting up. (DEFAULT: false)
+- `taskmanager.runtime.large-record-handler`: Whether to use the LargeRecordHandler when spilling. This feature is experimental. (DEFAULT: false)
--- End diff --
@aljoscha, can you explain what the large record handler is doing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2177
Thanks for updating the documentation! I've made some suggestions regarding the names of the new configuration keys.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r69267193
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
/**
* Minimum amount of heap memory to remove in containers, as a safety margin.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_MIN = "containered.heap-cutoff-min";
+ public static final String CONTAINER_HEAP_CUTOFF_MIN = "container.heap-cutoff-min";
/**
* Prefix for passing custom environment variables to Flink's master process.
* For example for passing LD_LIBRARY_PATH as an env variable to the AppMaster, set:
* yarn.application-master.env.LD_LIBRARY_PATH: "/usr/lib/native"
* in the flink-conf.yaml.
*/
- public static final String CONTAINERED_MASTER_ENV_PREFIX = "containered.application-master.env.";
+ public static final String CONTAINER_MASTER_ENV_PREFIX = "container.application-master.env.";
/**
- * Similar to the {@see CONTAINERED_MASTER_ENV_PREFIX}, this configuration prefix allows
+ * Similar to the {@see CONTAINER_MASTER_ENV_PREFIX}, this configuration prefix allows
* setting custom environment variables for the workers (TaskManagers)
*/
- public static final String CONTAINERED_TASK_MANAGER_ENV_PREFIX = "containered.taskmanager.env.";
+ public static final String CONTAINER_TASK_MANAGER_ENV_PREFIX = "container.taskmanager.env.";
--- End diff --
Actually, I'm not so sure about this anymore, because it taskManager is a name used in the documentation everywhere. Maybe we keep this name here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2177
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68932129
--- Diff: docs/setup/config.md ---
@@ -85,6 +85,8 @@ The default fraction for managed memory can be adjusted using the `taskmanager.m
- `taskmanager.memory.preallocate`: Can be either of `true` or `false`. Specifies whether task managers should allocate all managed memory when starting up. (DEFAULT: false)
+- `taskmanager.runtime.large-record-handler`: Whether to use the LargeRecordHandler when spilling. This feature is experimental. (DEFAULT: false)
--- End diff --
I would say so, yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68918991
--- Diff: docs/setup/config.md ---
@@ -85,6 +85,8 @@ The default fraction for managed memory can be adjusted using the `taskmanager.m
- `taskmanager.memory.preallocate`: Can be either of `true` or `false`. Specifies whether task managers should allocate all managed memory when starting up. (DEFAULT: false)
+- `taskmanager.runtime.large-record-handler`: Whether to use the LargeRecordHandler when spilling. This feature is experimental. (DEFAULT: false)
--- End diff --
Maybe we should not mention this in the documentation then?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68933327
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
/**
* Minimum amount of heap memory to remove in containers, as a safety margin.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_MIN = "containered.heap-cutoff-min";
+ public static final String CONTAINER_HEAP_CUTOFF_MIN = "container.heap-cutoff-min";
/**
* Prefix for passing custom environment variables to Flink's master process.
* For example for passing LD_LIBRARY_PATH as an env variable to the AppMaster, set:
* yarn.application-master.env.LD_LIBRARY_PATH: "/usr/lib/native"
* in the flink-conf.yaml.
*/
- public static final String CONTAINERED_MASTER_ENV_PREFIX = "containered.application-master.env.";
+ public static final String CONTAINER_MASTER_ENV_PREFIX = "container.application-master.env.";
--- End diff --
That makes sense. Application master is a YARN specific term afaik.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68907947
--- Diff: docs/setup/config.md ---
@@ -230,8 +240,8 @@ definition. This scheme is used **ONLY** if no other scheme is specified (explic
## YARN
-- `yarn.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove a 15% of the memory from the requested heap as a safety margin.
-- `yarn.heap-cutoff-min`: (Default 384 MB) Minimum amount of memory to cut off the requested heap size.
+- `container.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove a 15% of the memory from the requested heap as a safety margin.
--- End diff --
Maybe insert *, for example,* after YARN because the parameter is not specific to YARN, is it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r69123978
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
--- End diff --
I'm ok with that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2177
I think it's a good idea to merge `container` and `containerized`.
I think the idea is to introduce some general container parameters, such as the `heap-cutoff-ratio`, which are valid not only for yarn but also for mesos. Therefore, the `container/containered/containerized` prefix was introduced. Given that we don't have mesos support yet it might be confusing. However, changing the parameter names too often, given that they are declared public evolving, will cause pain for users.
Which duplicate configuration parameters are you referring to @StephanEwen?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r69300607
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINERIZED_HEAP_CUTOFF_RATIO = "containerized.heap-cutoff-ratio";
/**
* Minimum amount of heap memory to remove in containers, as a safety margin.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_MIN = "containered.heap-cutoff-min";
+ public static final String CONTAINERIZED_HEAP_CUTOFF_MIN = "containerized.heap-cutoff-min";
/**
* Prefix for passing custom environment variables to Flink's master process.
* For example for passing LD_LIBRARY_PATH as an env variable to the AppMaster, set:
* yarn.application-master.env.LD_LIBRARY_PATH: "/usr/lib/native"
* in the flink-conf.yaml.
*/
- public static final String CONTAINERED_MASTER_ENV_PREFIX = "containered.application-master.env.";
+ public static final String CONTAINERIZED_MASTER_ENV_PREFIX = "containerized.master.env.";
--- End diff --
I thought this should be `CONTAINER_MASTER_ENV_PREFIX = container.master.env.`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2177
Yeah, then let's leave them as they were since we don't know yet what mesos will require there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r69553944
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
/**
* Minimum amount of heap memory to remove in containers, as a safety margin.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_MIN = "containered.heap-cutoff-min";
+ public static final String CONTAINER_HEAP_CUTOFF_MIN = "container.heap-cutoff-min";
/**
* Prefix for passing custom environment variables to Flink's master process.
* For example for passing LD_LIBRARY_PATH as an env variable to the AppMaster, set:
* yarn.application-master.env.LD_LIBRARY_PATH: "/usr/lib/native"
* in the flink-conf.yaml.
*/
- public static final String CONTAINERED_MASTER_ENV_PREFIX = "containered.application-master.env.";
+ public static final String CONTAINER_MASTER_ENV_PREFIX = "container.application-master.env.";
/**
- * Similar to the {@see CONTAINERED_MASTER_ENV_PREFIX}, this configuration prefix allows
+ * Similar to the {@see CONTAINER_MASTER_ENV_PREFIX}, this configuration prefix allows
* setting custom environment variables for the workers (TaskManagers)
*/
- public static final String CONTAINERED_TASK_MANAGER_ENV_PREFIX = "containered.taskmanager.env.";
+ public static final String CONTAINER_TASK_MANAGER_ENV_PREFIX = "container.taskmanager.env.";
--- End diff --
With this argument, we should also rename `CONTAINER_MASTER_ENV_PREFIX` to `container.jobmanager.env`, shouldn't we?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2177
There are a lot of config options that seem to exist in two variants: One for standalone setup, one for containered setup.
Why are we making this distinction?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r69124072
--- Diff: docs/setup/config.md ---
@@ -230,8 +240,8 @@ definition. This scheme is used **ONLY** if no other scheme is specified (explic
## YARN
-- `yarn.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove a 15% of the memory from the requested heap as a safety margin.
-- `yarn.heap-cutoff-min`: (Default 384 MB) Minimum amount of memory to cut off the requested heap size.
+- `container.heap-cutoff-ratio`: (Default 0.25) Percentage of heap space to remove from containers started by YARN. When a user requests a certain amount of memory for each TaskManager container (for example 4 GB), we can not pass this amount as the maximum heap space for the JVM (`-Xmx` argument) because the JVM is also allocating memory outside the heap. YARN is very strict with killing containers which are using more memory than requested. Therefore, we remove a 15% of the memory from the requested heap as a safety margin.
--- End diff --
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68819516
--- Diff: docs/setup/config.md ---
@@ -85,6 +85,8 @@ The default fraction for managed memory can be adjusted using the `taskmanager.m
- `taskmanager.memory.preallocate`: Can be either of `true` or `false`. Specifies whether task managers should allocate all managed memory when starting up. (DEFAULT: false)
+- `taskmanager.runtime.large-record-handler`: Whether to use the LargeRecordHandler when spilling. This feature is experimental. (DEFAULT: false)
--- End diff --
Have we documented when it would be useful to enable `LargeRecordHandler`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2177
Good changes @rmetzger.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2177
Sorry for chiming in late, but can't - `heap-cutoff-ratio` and `heap-cutoff-min` be general parameters that apply to all VMs we start?
Same for `master.env.*` and `taskmanager.env.*`. Shouldn't this also apply to all VMs that we start?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2177
That would be good to address. Can we get away with only `yarn.` and `taskmanager.`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68905882
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
/**
* Minimum amount of heap memory to remove in containers, as a safety margin.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_MIN = "containered.heap-cutoff-min";
+ public static final String CONTAINER_HEAP_CUTOFF_MIN = "container.heap-cutoff-min";
/**
* Prefix for passing custom environment variables to Flink's master process.
* For example for passing LD_LIBRARY_PATH as an env variable to the AppMaster, set:
* yarn.application-master.env.LD_LIBRARY_PATH: "/usr/lib/native"
* in the flink-conf.yaml.
*/
- public static final String CONTAINERED_MASTER_ENV_PREFIX = "containered.application-master.env.";
+ public static final String CONTAINER_MASTER_ENV_PREFIX = "container.application-master.env.";
/**
- * Similar to the {@see CONTAINERED_MASTER_ENV_PREFIX}, this configuration prefix allows
+ * Similar to the {@see CONTAINER_MASTER_ENV_PREFIX}, this configuration prefix allows
* setting custom environment variables for the workers (TaskManagers)
*/
- public static final String CONTAINERED_TASK_MANAGER_ENV_PREFIX = "containered.taskmanager.env.";
+ public static final String CONTAINER_TASK_MANAGER_ENV_PREFIX = "container.taskmanager.env.";
--- End diff --
How about we rename the config string to `container.worker.env`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/2177
But with this argument, it would be even better to leave the configuration parameters as they are in 1.0 until we are 100% certain that mesos needs a similar mechanism.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/2177
The YARN settings are now spread across three prefixes:
- `yarn.`
- `containerized.`
- `container.`
I don't think this is helpful for making the system easy to configure.
What's your opinion on this @tillrohrmann @uce @StephanEwen ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68905457
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
--- End diff --
IMHO the `containered` prefix reflects the meaning of this config key better than `container`. It cuts off a fraction from the containerized Flink process, not from the container itself which the latter would imply.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/2177
All the container* config keys are now renamed to `containerized.`, so we have `yarn.` and `containerized.` as prefixes. But these config keys will be undocumented for now.
In the documentation, we still only list the `yarn.` configuration keys (and they are also picked up by the system).
I guess with the 1.2 release (when we are adding Mesos support) we can reconsider this decision and see how we are naming / grouping the configuration keys.
I'm going to merge this pull request after 24 hours.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68905471
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
/**
* Minimum amount of heap memory to remove in containers, as a safety margin.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_MIN = "containered.heap-cutoff-min";
+ public static final String CONTAINER_HEAP_CUTOFF_MIN = "container.heap-cutoff-min";
--- End diff --
Same as above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2177
My thinking was that Mesos maybe didn't require this, yes. Having it for Yarn only makes for a better user experience right now. So we could leave it under the "container" namespace.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68908810
--- Diff: docs/setup/config.md ---
@@ -85,6 +85,8 @@ The default fraction for managed memory can be adjusted using the `taskmanager.m
- `taskmanager.memory.preallocate`: Can be either of `true` or `false`. Specifies whether task managers should allocate all managed memory when starting up. (DEFAULT: false)
+- `taskmanager.runtime.large-record-handler`: Whether to use the LargeRecordHandler when spilling. This feature is experimental. (DEFAULT: false)
--- End diff --
Not really, I'm afraid. I just recently disabled it by default because of the known issues.
What I can gather from the code is that it is a separate sort buffer that is intended for very large records. It is not obvious from the code what "very large records" are, however.
The known problem is that key serialization does not work correctly if the user specified a custom type or if Scala types are used because the `TypeAnalyzer` is used in the `LargeRecordHandler` to get a `TypeInformation` on the fly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68933356
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
/**
* Minimum amount of heap memory to remove in containers, as a safety margin.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_MIN = "containered.heap-cutoff-min";
+ public static final String CONTAINER_HEAP_CUTOFF_MIN = "container.heap-cutoff-min";
/**
* Prefix for passing custom environment variables to Flink's master process.
* For example for passing LD_LIBRARY_PATH as an env variable to the AppMaster, set:
* yarn.application-master.env.LD_LIBRARY_PATH: "/usr/lib/native"
* in the flink-conf.yaml.
*/
- public static final String CONTAINERED_MASTER_ENV_PREFIX = "containered.application-master.env.";
+ public static final String CONTAINER_MASTER_ENV_PREFIX = "container.application-master.env.";
/**
- * Similar to the {@see CONTAINERED_MASTER_ENV_PREFIX}, this configuration prefix allows
+ * Similar to the {@see CONTAINER_MASTER_ENV_PREFIX}, this configuration prefix allows
* setting custom environment variables for the workers (TaskManagers)
*/
- public static final String CONTAINERED_TASK_MANAGER_ENV_PREFIX = "containered.taskmanager.env.";
+ public static final String CONTAINER_TASK_MANAGER_ENV_PREFIX = "container.taskmanager.env.";
--- End diff --
That would be consistent with above renaming
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68907744
--- Diff: docs/setup/config.md ---
@@ -107,12 +109,20 @@ Please make sure to set the maximum ticket life span high long running jobs. The
If you are on YARN, then it is sufficient to authenticate the client with Kerberos. On a Flink standalone cluster you need to ensure that, initially, all nodes are authenticated with Kerberos using the `kinit` tool.
+### Resource Manager
+
+- `resourcemanager.rpc.port`: The config parameter defining the network port to connect to for communication with the resource manager.
+
### Other
- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the systems directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round-robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: The system's tmp dir).
+- `taskmanager.log.path`: The config parameter defining the taskmanager log file location
+
- `jobmanager.web.port`: Port of the JobManager's web interface (DEFAULT: 8081).
+- `jobmanager.web.tmpdir`: This configuration parameter allows defining the Flink web directory to be used by the web interface.
--- End diff --
what is a web directory?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68905853
--- Diff: flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -269,26 +269,26 @@
* Percentage of heap space to remove from containers (YARN / Mesos), to compensate
* for other JVM memory usage.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_RATIO = "containered.heap-cutoff-ratio";
+ public static final String CONTAINER_HEAP_CUTOFF_RATIO = "container.heap-cutoff-ratio";
/**
* Minimum amount of heap memory to remove in containers, as a safety margin.
*/
- public static final String CONTAINERED_HEAP_CUTOFF_MIN = "containered.heap-cutoff-min";
+ public static final String CONTAINER_HEAP_CUTOFF_MIN = "container.heap-cutoff-min";
/**
* Prefix for passing custom environment variables to Flink's master process.
* For example for passing LD_LIBRARY_PATH as an env variable to the AppMaster, set:
* yarn.application-master.env.LD_LIBRARY_PATH: "/usr/lib/native"
* in the flink-conf.yaml.
*/
- public static final String CONTAINERED_MASTER_ENV_PREFIX = "containered.application-master.env.";
+ public static final String CONTAINER_MASTER_ENV_PREFIX = "container.application-master.env.";
--- End diff --
How about we rename the config string to `container.master.env`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2177: [FLINK-4127] Check API compatbility for 1.1 in flink-core
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/2177
I addressed all comments. It seems to me that the configuration parameters for YARN are overly complicated now because they are separated in resource manager and YARN now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by mxm <gi...@git.apache.org>.
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68908671
--- Diff: docs/setup/config.md ---
@@ -107,12 +109,20 @@ Please make sure to set the maximum ticket life span high long running jobs. The
If you are on YARN, then it is sufficient to authenticate the client with Kerberos. On a Flink standalone cluster you need to ensure that, initially, all nodes are authenticated with Kerberos using the `kinit` tool.
+### Resource Manager
+
+- `resourcemanager.rpc.port`: The config parameter defining the network port to connect to for communication with the resource manager.
--- End diff --
No, it doesn't. It is currently not in use (except for testing code). In Yarn, the port of the application master is always the port of the job manager because the two run in the same actor system.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2177: [FLINK-4127] Check API compatbility for 1.1 in fli...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/2177#discussion_r68904419
--- Diff: docs/setup/config.md ---
@@ -85,6 +85,8 @@ The default fraction for managed memory can be adjusted using the `taskmanager.m
- `taskmanager.memory.preallocate`: Can be either of `true` or `false`. Specifies whether task managers should allocate all managed memory when starting up. (DEFAULT: false)
+- `taskmanager.runtime.large-record-handler`: Whether to use the LargeRecordHandler when spilling. This feature is experimental. (DEFAULT: false)
--- End diff --
That's a very good point, Greg. I think we didn't. As it stands, the large record handler seems to have known issues. Personally, I would therefore discourage people from using it, but it totally makes sense to add a paragraph about what it actually does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---