You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Stephan Ewen (Jira)" <ji...@apache.org> on 2020/01/22 10:49:00 UTC
[jira] [Updated] (FLINK-15698) Restructure Configuration Docs to
separate "Getting Started", "Common Operations" and "Tuning/Debugging"
[ https://issues.apache.org/jira/browse/FLINK-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephan Ewen updated FLINK-15698:
---------------------------------
Description:
In order to make the configuration documentation easier to understand for users, I would propose to he these major sections.
* Basic Setup / Getting Started
* Common Setup Options
* Deployment Targets (K8s, Yarn, Mesos)
* State Backends
* Expert/Internal
* Deprecated Options
The details of what to put into these sections is as follows:
*Basic Setups*
This section can be manual, does not need to be auto-generated. It should also be a bit more verbose and not just list options and descriptions.
- "Master and TaskManager Processes"
-- taskmanager.memory.process.size, taskmanager.memory.flink.size, jobmanager.heap.size
-- rest.address, rest.port
-- jobmanager.rpc.address, jobmanager.rpc.port
- "Parallelism"
-- taskmanager.numberOfTaskSlots
-- parallelism.default
- "Checkpointing"
-- state.backend
-- state.checkpoints.dir
-- state.savepoints.dir
- Directories
-- io.tmp.dirs
-- blob.storage.directory
-- web.upload.dir
- Miscellaneous
** web.submit.enable
** cluster.evenly-spread-out-slots
*Setup and Common Operations*
(These are options everyone may need at some point. They are not to alter behavior/debug/tune, but to configure the behavior and interaction with the environment)
Hosts and Ports
- rest.address
- rest.port
- rest.bind-port
- jobmanager.rpc.address
- jobmanager.rpc.port
- taskmanager.host
- taskmanager.rpc.port
- taskmanager.data.port
Memory Configuration
- all the FLIP-49 memory options
- taskmanager.debug.memory.log
- taskmanager.debug.memory.log-interval
Fault Tolerance
- restart-strategy
- restart strategy specific options
- jobmanager.execution.failover-strategy
- heartbeat.interval
- heartbeat.timeout
State Backends
High Availability
- High Availability (HA)
- ZooKeeper-based HA Mode
Security
- SSL Settings
- ZooKeeper Security
- Kerberos-based Security
Metrics
File Systems
History Server
Python Runtime
Queryable State
ClassLoading
- Preamble links to [https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html]
- classloader.resolve-order
- classloader.parent-first-patterns.default
- classloader.parent-first-patterns.additional
*Deployment Resource Management*
- Kubernetes
- Yarn
- Mesos
*State Backends*
(these options are mainly for specific setups and performance tuning of the respective backends)
- RocksDB State Backend Options
- RocksDB Configurable Options (rename to RocksDB Native Options)
- RocksDB Native Metrics
*Debugging & Expert Setups*
(Normally, users should not need to touch any of these options)
Debugging Scheduling
- slot.idle.timeout
- slot.request.timeout
- cluster.registration.*
Full Rest Options
Full JobManager Options
Full TaskManager Options
Resource Manager Options
Akka Options
Network Options
- Netty Shuffle Environment
- Network Communication (via Netty)
*Deprecated*
- DataSet Optimizer
- DataSet Algorithms
was:
In order to make the configuration documentation easier to understand for users, I would propose to he these major sections.
* Basic Setup / Getting Started
* Common Setup Options
* Deployment Targets (K8s, Yarn, Mesos)
* State Backends
* Expert/Internal
* Deprecated Options
The details of what to put into these sections is as follows:
*Basic Setups*
This section can be manual, does not need to be auto-generated. It should also be a bit more verbose and not just list options and descriptions.
- "Master and TaskManager Processes"
-- taskmanager.memory.process.size, taskmanager.memory.flink.size, jobmanager.heap.size)
-- rest.address, rest.port
-- jobmanager.rpc.address, jobmanager.rpc.port
-- web.submit.enable
- "Parallelism"
-- taskmanager.numberOfTaskSlots
-- parallelism.default
- "Checkpointing"
-- state.backend
-- state.checkpoints.dir
-- state.savepoints.dir
- Directories
-- io.tmp.dirs
-- blob.storage.directory
-- web.upload.dir
- Miscellaneous
-- cluster.evenly-spread-out-slots
*Setup and Common Operations*
(These are options everyone may need at some point. They are not to alter behavior/debug/tune, but to configure the behavior and interaction with the environment)
Hosts and Ports
- rest.address
- rest.port
- rest.bind-port
- jobmanager.rpc.address
- jobmanager.rpc.port
- taskmanager.host
- taskmanager.rpc.port
- taskmanager.data.port
Memory Configuration
- all the FLIP-49 memory options
- taskmanager.debug.memory.log
- taskmanager.debug.memory.log-interval
Fault Tolerance
- restart-strategy
- restart strategy specific options
- jobmanager.execution.failover-strategy
- heartbeat.interval
- heartbeat.timeout
State Backends
High Availability
- High Availability (HA)
- ZooKeeper-based HA Mode
Security
- SSL Settings
- ZooKeeper Security
- Kerberos-based Security
Metrics
File Systems
History Server
Python Runtime
Queryable State
ClassLoading
- Preamble links to [https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html]
- classloader.resolve-order
- classloader.parent-first-patterns.default
- classloader.parent-first-patterns.additional
*Deployment Resource Management*
- Kubernetes
- Yarn
- Mesos
*State Backends*
(these options are mainly for specific setups and performance tuning of the respective backends)
- RocksDB State Backend Options
- RocksDB Configurable Options (rename to RocksDB Native Options)
- RocksDB Native Metrics
*Debugging & Expert Setups*
(Normally, users should not need to touch any of these options)
Debugging Scheduling
- slot.idle.timeout
- slot.request.timeout
- cluster.registration.*
Full Rest Options
Full JobManager Options
Full TaskManager Options
Resource Manager Options
Akka Options
Network Options
- Netty Shuffle Environment
- Network Communication (via Netty)
*Deprecated*
- DataSet Optimizer
- DataSet Algorithms
> Restructure Configuration Docs to separate "Getting Started", "Common Operations" and "Tuning/Debugging"
> --------------------------------------------------------------------------------------------------------
>
> Key: FLINK-15698
> URL: https://issues.apache.org/jira/browse/FLINK-15698
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
> Reporter: Stephan Ewen
> Priority: Blocker
> Fix For: 1.11.0
>
>
> In order to make the configuration documentation easier to understand for users, I would propose to he these major sections.
> * Basic Setup / Getting Started
> * Common Setup Options
> * Deployment Targets (K8s, Yarn, Mesos)
> * State Backends
> * Expert/Internal
> * Deprecated Options
>
> The details of what to put into these sections is as follows:
> *Basic Setups*
> This section can be manual, does not need to be auto-generated. It should also be a bit more verbose and not just list options and descriptions.
> - "Master and TaskManager Processes"
> -- taskmanager.memory.process.size, taskmanager.memory.flink.size, jobmanager.heap.size
> -- rest.address, rest.port
> -- jobmanager.rpc.address, jobmanager.rpc.port
> - "Parallelism"
> -- taskmanager.numberOfTaskSlots
> -- parallelism.default
> - "Checkpointing"
> -- state.backend
> -- state.checkpoints.dir
> -- state.savepoints.dir
> - Directories
> -- io.tmp.dirs
> -- blob.storage.directory
> -- web.upload.dir
> - Miscellaneous
> ** web.submit.enable
> ** cluster.evenly-spread-out-slots
>
> *Setup and Common Operations*
> (These are options everyone may need at some point. They are not to alter behavior/debug/tune, but to configure the behavior and interaction with the environment)
> Hosts and Ports
> - rest.address
> - rest.port
> - rest.bind-port
> - jobmanager.rpc.address
> - jobmanager.rpc.port
> - taskmanager.host
> - taskmanager.rpc.port
> - taskmanager.data.port
> Memory Configuration
> - all the FLIP-49 memory options
> - taskmanager.debug.memory.log
> - taskmanager.debug.memory.log-interval
> Fault Tolerance
> - restart-strategy
> - restart strategy specific options
> - jobmanager.execution.failover-strategy
> - heartbeat.interval
> - heartbeat.timeout
> State Backends
> High Availability
> - High Availability (HA)
> - ZooKeeper-based HA Mode
> Security
> - SSL Settings
> - ZooKeeper Security
> - Kerberos-based Security
> Metrics
> File Systems
> History Server
> Python Runtime
> Queryable State
> ClassLoading
> - Preamble links to [https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html]
> - classloader.resolve-order
> - classloader.parent-first-patterns.default
> - classloader.parent-first-patterns.additional
>
> *Deployment Resource Management*
> - Kubernetes
> - Yarn
> - Mesos
>
> *State Backends*
> (these options are mainly for specific setups and performance tuning of the respective backends)
> - RocksDB State Backend Options
> - RocksDB Configurable Options (rename to RocksDB Native Options)
> - RocksDB Native Metrics
>
> *Debugging & Expert Setups*
> (Normally, users should not need to touch any of these options)
> Debugging Scheduling
> - slot.idle.timeout
> - slot.request.timeout
> - cluster.registration.*
> Full Rest Options
> Full JobManager Options
> Full TaskManager Options
> Resource Manager Options
> Akka Options
> Network Options
> - Netty Shuffle Environment
> - Network Communication (via Netty)
>
> *Deprecated*
> - DataSet Optimizer
> - DataSet Algorithms
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)