You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by zentol <gi...@git.apache.org> on 2018/02/13 08:32:45 UTC

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

GitHub user zentol opened a pull request:

    https://github.com/apache/flink/pull/5471

    [FLINK-8475][config][docs] Integrate TM options

    ## What is the purpose of the change
    
    This PR integrates the TaskManager `ConfigOptions` into the configuration docs generator.
    
    ## Brief change log
    
    * Port missing  taskmanager config constants to config options
    * Add missing descriptions to config options (derived from existing description/javadocs)
    * integrate TaskManager configuration table into `config.md`

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zentol/flink 8475_tm

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5471.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5471
    
----
commit edb996e1c21962e6d087c941e8d898879021b9fc
Author: zentol <ch...@...>
Date:   2018-01-22T17:15:34Z

    [FLINK-8475][config][docs] Integrate TM options

----


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5471#discussion_r167900592
  
    --- Diff: docs/_includes/generated/task_manager_configuration.html ---
    @@ -0,0 +1,146 @@
    +<table class="table table-bordered">
    +    <thead>
    +        <tr>
    +            <th class="text-left" style="width: 20%">Key</th>
    +            <th class="text-left" style="width: 15%">Default</th>
    +            <th class="text-left" style="width: 65%">Description</th>
    +        </tr>
    +    </thead>
    +    <tbody>
    +        <tr>
    +            <td><h5>task.cancellation.interval</h5></td>
    --- End diff --
    
    `cancellation.timeout`, I think I was referring to


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167836229

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

You didn't want to include that whole description of how Flink works? 😉

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167853874

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

-The following parameters configure Flink's TaskManagers.
-
-
-- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file.
-
-- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine.
-
-- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port).
-
-- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**)
-
-- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value.
-
-- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores).
-
-- `taskmanager.tmp.dirs`: The directory for temporary files, or a list of directories separated by the system's directory delimiter (for example ':' (colon) on Linux/Unix). If multiple directories are specified, then the temporary files will be distributed across the directories in a round robin fashion. The I/O manager component will spawn one reading and one writing thread per directory. A directory may be listed multiple times to have the I/O manager use multiple threads for it (for example if it is physically stored on a very fast disc or RAID) (DEFAULT: **The system's tmp dir**).
-
-- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for network buffers. This determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value or the min/max values below. Also note, that `taskmanager.network.memory.min` and `taskmanager.network.memory.max` may override this fraction. (DEFAULT: **0.1**)
-
-- `taskmanager.network.memory.min`: Minimum memory size for network buffers in bytes (DEFAULT: **64 MB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`.
-
-- `taskmanager.network.memory.max`: Maximum memory size for network buffers in bytes (DEFAULT: **1 GB**). Previously, this was determined from `taskmanager.network.numberOfBuffers` and `taskmanager.memory.segment-size`.
-
-- `taskmanager.network.numberOfBuffers` (deprecated, replaced by the three parameters above): The number of buffers available to the network stack. This number determines how many streaming data exchange channels a TaskManager can have at the same time and how well buffered the channels are. If a job is rejected or you get a warning that the system has not enough buffers available, increase this value (DEFAULT: **2048**). If set, it will be mapped to `taskmanager.network.memory.min` and `taskmanager.network.memory.max` based on `taskmanager.memory.segment-size`.
-
-- `taskmanager.memory.size`: The amount of memory (in megabytes) that the task manager reserves on the JVM's heap space for sorting, hash tables, and caching of intermediate results. If unspecified (-1), the memory manager will take a fixed ratio of the heap memory available to the JVM, as specified by `taskmanager.memory.fraction`. (DEFAULT: **-1**)
-
-- `taskmanager.memory.fraction`: The relative amount of memory (with respect to `taskmanager.heap.mb`, after subtracting the amount of memory used by network buffers) that the task manager reserves for sorting, hash tables, and caching of intermediate results. For example, a value of `0.8` means that a task manager reserves 80% of its memory (on-heap or off-heap depending on `taskmanager.memory.off-heap`) for internal data buffers, leaving 20% of free memory for the task manager's heap for objects created by user-defined functions. (DEFAULT: 0.7) This parameter is only evaluated, if `taskmanager.memory.size` is not set.
-
-- `taskmanager.debug.memory.startLogThread`: Causes the TaskManagers to periodically log memory and Garbage collection statistics. The statistics include current heap-, off-heap, and other memory pool utilization, as well as the time spent on garbage collection, by heap memory pool.
-
-- `taskmanager.debug.memory.logIntervalMs`: The interval (in milliseconds) in which the TaskManagers log the memory and garbage collection statistics. Only has an effect, if `taskmanager.debug.memory.startLogThread` is set to true.
-
-- `taskmanager.maxRegistrationDuration`: Defines the maximum time it can take for the TaskManager registration. If the duration is exceeded without a successful registration, then the TaskManager terminates. The max registration duration requires a time unit specifier (ms/s/min/h/d) (e.g. "10 min"). (DEFAULT: **Inf**)
-
-- `taskmanager.initial-registration-pause`: The initial registration pause between two consecutive registration attempts. The pause is doubled for each new registration attempt until it reaches the maximum registration pause. The initial registration pause requires a time unit specifier (ms/s/min/h/d) (e.g. "5 s"). (DEFAULT: **500 ms**)
-
-- `taskmanager.max-registration-pause`: The maximum registration pause between two consecutive registration attempts. The max registration pause requires a time unit specifier (ms/s/min/h/d) (e.g. "5 s"). (DEFAULT: **30 s**)
--- End diff --

This seems to not be in the new doc

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167855709

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

superceeded by `CoreOptions#TMP_DIRS`

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167861535

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

will revert to existing docs

---

[GitHub] flink issue #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/5471
  
    merging.


---

[GitHub] flink issue #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on the issue:

    https://github.com/apache/flink/pull/5471
  
    Feel free to merge once you addressed my other comment. I'm eager to merge this no matter what because all of this is better than what we had before...


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5471#discussion_r167835147
  
    --- Diff: docs/ops/config.md ---
    @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp
     
     ### TaskManager
     
    -The following parameters configure Flink's TaskManagers.
    -
    -
    -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file.
    --- End diff --
    
    Why don't we have this longer description for the new `taskmanager.host` setting?


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167852941

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

Why not include the longer description?

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5471#discussion_r167836313
  
    --- Diff: docs/ops/config.md ---
    @@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp
     
     ### TaskManager
     
    -The following parameters configure Flink's TaskManagers.
    -
    -
    -- `taskmanager.hostname`: The hostname of the network interface that the TaskManager binds to. By default, the TaskManager searches for network interfaces that can connect to the JobManager and other TaskManagers. This option can be used to define a hostname if that strategy fails for some reason. Because different TaskManagers need different values for this option, it usually is specified in an additional non-shared TaskManager-specific config file.
    -
    -- `taskmanager.rpc.port`: The task manager's IPC port (DEFAULT: **0**, which lets the OS choose a free port). Flink also accepts a list of ports ("50100,50101"), ranges ("50100-50200") or a combination of both. It is recommended to set a range of ports to avoid collisions when multiple TaskManagers are running on the same machine.
    -
    -- `taskmanager.data.port`: The task manager's port used for data exchange operations (DEFAULT: **0**, which lets the OS choose a free port).
    -
    -- `taskmanager.data.ssl.enabled`: Enable SSL support for the taskmanager data transport. This is applicable only when the global ssl flag security.ssl.enabled is set to true (DEFAULT: **true**)
    -
    -- `taskmanager.heap.mb`: JVM heap size (in megabytes) for the TaskManagers, which are the parallel workers of the system. In contrast to Hadoop, Flink runs operators (e.g., join, aggregate) and user-defined functions (e.g., Map, Reduce, CoGroup) inside the TaskManager (including sorting/hashing/caching), so this value should be as large as possible (DEFAULT: **512**). On YARN setups, this value is automatically configured to the size of the TaskManager's YARN container, minus a certain tolerance value.
    -
    -- `taskmanager.numberOfTaskSlots`: The number of parallel operator or user function instances that a single TaskManager can run (DEFAULT: **1**). If this value is larger than 1, a single TaskManager takes multiple instances of a function or operator. That way, the TaskManager can utilize multiple CPU cores, but at the same time, the available memory is divided between the different operator or function instances. This value is typically proportional to the number of physical CPU cores that the TaskManager's machine has (e.g., equal to the number of cores, or half the number of cores).
    --- End diff --
    
    Why not include this one in the new description?


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167863017

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

Actually yes, I believe this doesn't belong in there. The YARN note is useful though...

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167856138

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

the generator excludes deprecated options on the assumption that they were subsumed by other options.

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167836095

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

Why don't we keep `This is applicable only when the global ssl flag security.ssl.enabled is set to true`?

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167856613

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

BlobServerOptions, see #5391

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167856770

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

old key was deprecated
```
public static final ConfigOption<Long> TASK_CANCELLATION_INTERVAL =
key("task.cancellation.interval")
.defaultValue(30000L)
.withDeprecatedKeys("task.cancellation-interval");
```

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5471#discussion_r167860765
  
    --- Diff: docs/_includes/generated/task_manager_configuration.html ---
    @@ -0,0 +1,146 @@
    +<table class="table table-bordered">
    +    <thead>
    +        <tr>
    +            <th class="text-left" style="width: 20%">Key</th>
    +            <th class="text-left" style="width: 15%">Default</th>
    +            <th class="text-left" style="width: 65%">Description</th>
    +        </tr>
    +    </thead>
    +    <tbody>
    +        <tr>
    +            <td><h5>task.cancellation.interval</h5></td>
    --- End diff --
    
    there's a description for cancellation.interval, which other one are you referring to?


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167853223

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

Isn't this deprecated but you can still use it? I don't see it in the new doc.

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167853972

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

Where were those migrated to?

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/5471


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5471#discussion_r167854183
  
    --- Diff: docs/_includes/generated/task_manager_configuration.html ---
    @@ -0,0 +1,146 @@
    +<table class="table table-bordered">
    +    <thead>
    +        <tr>
    +            <th class="text-left" style="width: 20%">Key</th>
    +            <th class="text-left" style="width: 15%">Default</th>
    +            <th class="text-left" style="width: 65%">Description</th>
    +        </tr>
    +    </thead>
    +    <tbody>
    +        <tr>
    +            <td><h5>task.cancellation.interval</h5></td>
    --- End diff --
    
    The description for those two should probably be carried over from the old doc


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167854639

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

You know why the new option is `task.cancellation.interval`?

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167853029

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

This question is probably true for a lot of these changes.

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167853447

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

It looks like this and the next config didn't make it into the new doc

---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by aljoscha <gi...@git.apache.org>.

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167836369

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

Does this not work anymore?

---

[GitHub] flink issue #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/5471
  
    @aljoscha I've ported the missing options and modified plenty of descriptions.


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167856324

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

weren't ported to ConfigOptions yet.

---

[GitHub] flink issue #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on the issue:

    https://github.com/apache/flink/pull/5471
  
    man these long descriptions are gonna wreck the table...


---

[GitHub] flink pull request #5471: [FLINK-8475][config][docs] Integrate TM options

Posted by zentol <gi...@git.apache.org>.

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5471#discussion_r167856481

--- Diff: docs/ops/config.md ---
@@ -268,59 +268,7 @@ These parameters configure the default HDFS used by Flink. Setups that do not sp

### TaskManager

yes missed this one

---