You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ch...@apache.org on 2020/08/05 10:26:46 UTC
[flink] branch master updated (cf8b882 -> cb0247b)
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git.
from cf8b882 [FLINK-18677][fix] Added handling of suspended or lost connections within the ZooKeeperLeaderRetrievalService.
new e306488 [hotfix][docs][config] Sort entries
new 26cd522 [hotfix][docs][config] Remove leftover files
new cb0247b [FLINK-18804][docs][config] Skip file generation for empty tables
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../generated/all_taskmanager_network_section.html | 12 +--
.../generated/file_system_configuration.html | 30 -------
.../high_availability_zookeeper_configuration.html | 96 ----------------------
.../generated/kerberos_configuration.html | 36 --------
.../netty_shuffle_environment_configuration.html | 12 +--
.../generated/network_netty_configuration.html | 54 ------------
.../generated/zoo_keeper_configuration.html | 30 -------
.../configuration/ConfigOptionsDocGenerator.java | 16 +++-
.../ConfigOptionsDocGeneratorTest.java | 10 +++
9 files changed, 34 insertions(+), 262 deletions(-)
delete mode 100644 docs/_includes/generated/file_system_configuration.html
delete mode 100644 docs/_includes/generated/high_availability_zookeeper_configuration.html
delete mode 100644 docs/_includes/generated/kerberos_configuration.html
delete mode 100644 docs/_includes/generated/network_netty_configuration.html
delete mode 100644 docs/_includes/generated/zoo_keeper_configuration.html
[flink] 02/03: [hotfix][docs][config] Remove leftover files
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 26cd5225c732ce5f95d4b422e8bd89237f130ae4
Author: Chesnay Schepler <ch...@apache.org>
AuthorDate: Mon Aug 3 13:35:21 2020 +0200
[hotfix][docs][config] Remove leftover files
---
.../generated/file_system_configuration.html | 30 -------
.../high_availability_zookeeper_configuration.html | 96 ----------------------
.../generated/kerberos_configuration.html | 36 --------
.../generated/network_netty_configuration.html | 54 ------------
.../generated/zoo_keeper_configuration.html | 30 -------
5 files changed, 246 deletions(-)
diff --git a/docs/_includes/generated/file_system_configuration.html b/docs/_includes/generated/file_system_configuration.html
deleted file mode 100644
index 055b1e8..0000000
--- a/docs/_includes/generated/file_system_configuration.html
+++ /dev/null
@@ -1,30 +0,0 @@
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Key</th>
- <th class="text-left" style="width: 15%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>fs.default-scheme</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
- <td>String</td>
- <td>The default filesystem scheme, used for paths that do not declare a scheme explicitly. May contain an authority, e.g. host:port in case of an HDFS NameNode.</td>
- </tr>
- <tr>
- <td><h5>fs.output.always-create-directory</h5></td>
- <td style="word-wrap: break-word;">false</td>
- <td>Boolean</td>
- <td>File writers running with a parallelism larger than one create a directory for the output file path and put the different result files (one per parallel writer task) into that directory. If this option is set to "true", writers with a parallelism of 1 will also create a directory and place a single result file into it. If the option is set to "false", the writer will directly create the file directly at the output path, without creating a containing directory.</td>
- </tr>
- <tr>
- <td><h5>fs.overwrite-files</h5></td>
- <td style="word-wrap: break-word;">false</td>
- <td>Boolean</td>
- <td>Specifies whether file output writers should overwrite existing files by default. Set to "true" to overwrite by default,"false" otherwise.</td>
- </tr>
- </tbody>
-</table>
diff --git a/docs/_includes/generated/high_availability_zookeeper_configuration.html b/docs/_includes/generated/high_availability_zookeeper_configuration.html
deleted file mode 100644
index 75a7b82..0000000
--- a/docs/_includes/generated/high_availability_zookeeper_configuration.html
+++ /dev/null
@@ -1,96 +0,0 @@
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Key</th>
- <th class="text-left" style="width: 15%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>high-availability.zookeeper.client.acl</h5></td>
- <td style="word-wrap: break-word;">"open"</td>
- <td>String</td>
- <td>Defines the ACL (open|creator) to be configured on ZK node. The configuration value can be set to “creator” if the ZooKeeper server configuration has the “authProvider” property mapped to use SASLAuthenticationProvider and the cluster is configured to run in secure mode (Kerberos).</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.client.connection-timeout</h5></td>
- <td style="word-wrap: break-word;">15000</td>
- <td>Integer</td>
- <td>Defines the connection timeout for ZooKeeper in ms.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.client.max-retry-attempts</h5></td>
- <td style="word-wrap: break-word;">3</td>
- <td>Integer</td>
- <td>Defines the number of connection retries before the client gives up.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.client.retry-wait</h5></td>
- <td style="word-wrap: break-word;">5000</td>
- <td>Integer</td>
- <td>Defines the pause between consecutive retries in ms.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.client.session-timeout</h5></td>
- <td style="word-wrap: break-word;">60000</td>
- <td>Integer</td>
- <td>Defines the session timeout for the ZooKeeper session in ms.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.path.checkpoint-counter</h5></td>
- <td style="word-wrap: break-word;">"/checkpoint-counter"</td>
- <td>String</td>
- <td>ZooKeeper root path (ZNode) for checkpoint counters.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.path.checkpoints</h5></td>
- <td style="word-wrap: break-word;">"/checkpoints"</td>
- <td>String</td>
- <td>ZooKeeper root path (ZNode) for completed checkpoints.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.path.jobgraphs</h5></td>
- <td style="word-wrap: break-word;">"/jobgraphs"</td>
- <td>String</td>
- <td>ZooKeeper root path (ZNode) for job graphs</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.path.latch</h5></td>
- <td style="word-wrap: break-word;">"/leaderlatch"</td>
- <td>String</td>
- <td>Defines the znode of the leader latch which is used to elect the leader.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.path.leader</h5></td>
- <td style="word-wrap: break-word;">"/leader"</td>
- <td>String</td>
- <td>Defines the znode of the leader which contains the URL to the leader and the current leader session ID.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.path.mesos-workers</h5></td>
- <td style="word-wrap: break-word;">"/mesos-workers"</td>
- <td>String</td>
- <td>The ZooKeeper root path for persisting the Mesos worker information.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.path.root</h5></td>
- <td style="word-wrap: break-word;">"/flink"</td>
- <td>String</td>
- <td>The root path under which Flink stores its entries in ZooKeeper.</td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.path.running-registry</h5></td>
- <td style="word-wrap: break-word;">"/running_job_registry/"</td>
- <td>String</td>
- <td></td>
- </tr>
- <tr>
- <td><h5>high-availability.zookeeper.quorum</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
- <td>String</td>
- <td>The ZooKeeper quorum to use, when running Flink in a high-availability mode with ZooKeeper.</td>
- </tr>
- </tbody>
-</table>
diff --git a/docs/_includes/generated/kerberos_configuration.html b/docs/_includes/generated/kerberos_configuration.html
deleted file mode 100644
index cf27917..0000000
--- a/docs/_includes/generated/kerberos_configuration.html
+++ /dev/null
@@ -1,36 +0,0 @@
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Key</th>
- <th class="text-left" style="width: 15%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>security.kerberos.login.contexts</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
- <td>String</td>
- <td>A comma-separated list of login contexts to provide the Kerberos credentials to (for example, `Client,KafkaClient` to use the credentials for ZooKeeper authentication and for Kafka authentication)</td>
- </tr>
- <tr>
- <td><h5>security.kerberos.login.keytab</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
- <td>String</td>
- <td>Absolute path to a Kerberos keytab file that contains the user credentials.</td>
- </tr>
- <tr>
- <td><h5>security.kerberos.login.principal</h5></td>
- <td style="word-wrap: break-word;">(none)</td>
- <td>String</td>
- <td>Kerberos principal name associated with the keytab.</td>
- </tr>
- <tr>
- <td><h5>security.kerberos.login.use-ticket-cache</h5></td>
- <td style="word-wrap: break-word;">true</td>
- <td>Boolean</td>
- <td>Indicates whether to read from your Kerberos ticket cache.</td>
- </tr>
- </tbody>
-</table>
diff --git a/docs/_includes/generated/network_netty_configuration.html b/docs/_includes/generated/network_netty_configuration.html
deleted file mode 100644
index 3950b56..0000000
--- a/docs/_includes/generated/network_netty_configuration.html
+++ /dev/null
@@ -1,54 +0,0 @@
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Key</th>
- <th class="text-left" style="width: 15%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>taskmanager.network.netty.client.connectTimeoutSec</h5></td>
- <td style="word-wrap: break-word;">120</td>
- <td>Integer</td>
- <td>The Netty client connection timeout.</td>
- </tr>
- <tr>
- <td><h5>taskmanager.network.netty.client.numThreads</h5></td>
- <td style="word-wrap: break-word;">-1</td>
- <td>Integer</td>
- <td>The number of Netty client threads.</td>
- </tr>
- <tr>
- <td><h5>taskmanager.network.netty.num-arenas</h5></td>
- <td style="word-wrap: break-word;">-1</td>
- <td>Integer</td>
- <td>The number of Netty arenas.</td>
- </tr>
- <tr>
- <td><h5>taskmanager.network.netty.sendReceiveBufferSize</h5></td>
- <td style="word-wrap: break-word;">0</td>
- <td>Integer</td>
- <td>The Netty send and receive buffer size. This defaults to the system buffer size (cat /proc/sys/net/ipv4/tcp_[rw]mem) and is 4 MiB in modern Linux.</td>
- </tr>
- <tr>
- <td><h5>taskmanager.network.netty.server.backlog</h5></td>
- <td style="word-wrap: break-word;">0</td>
- <td>Integer</td>
- <td>The netty server connection backlog.</td>
- </tr>
- <tr>
- <td><h5>taskmanager.network.netty.server.numThreads</h5></td>
- <td style="word-wrap: break-word;">-1</td>
- <td>Integer</td>
- <td>The number of Netty server threads.</td>
- </tr>
- <tr>
- <td><h5>taskmanager.network.netty.transport</h5></td>
- <td style="word-wrap: break-word;">"auto"</td>
- <td>String</td>
- <td>The Netty transport type, either "nio" or "epoll". The "auto" means selecting the property mode automatically based on the platform. Note that the "epoll" mode can get better performance, less GC and have more advanced features which are only available on modern Linux.</td>
- </tr>
- </tbody>
-</table>
diff --git a/docs/_includes/generated/zoo_keeper_configuration.html b/docs/_includes/generated/zoo_keeper_configuration.html
deleted file mode 100644
index 7aaaaff..0000000
--- a/docs/_includes/generated/zoo_keeper_configuration.html
+++ /dev/null
@@ -1,30 +0,0 @@
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Key</th>
- <th class="text-left" style="width: 15%">Default</th>
- <th class="text-left" style="width: 10%">Type</th>
- <th class="text-left" style="width: 55%">Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>zookeeper.sasl.disable</h5></td>
- <td style="word-wrap: break-word;">false</td>
- <td>Boolean</td>
- <td></td>
- </tr>
- <tr>
- <td><h5>zookeeper.sasl.login-context-name</h5></td>
- <td style="word-wrap: break-word;">"Client"</td>
- <td>String</td>
- <td></td>
- </tr>
- <tr>
- <td><h5>zookeeper.sasl.service-name</h5></td>
- <td style="word-wrap: break-word;">"zookeeper"</td>
- <td>String</td>
- <td></td>
- </tr>
- </tbody>
-</table>
[flink] 03/03: [FLINK-18804][docs][config] Skip file generation for
empty tables
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit cb0247b9530deaf7a6d1ce55c2dbef6317068103
Author: Chesnay Schepler <ch...@apache.org>
AuthorDate: Mon Aug 3 12:28:12 2020 +0200
[FLINK-18804][docs][config] Skip file generation for empty tables
---
.../docs/configuration/ConfigOptionsDocGenerator.java | 16 ++++++++++++----
.../configuration/ConfigOptionsDocGeneratorTest.java | 10 ++++++++++
2 files changed, 22 insertions(+), 4 deletions(-)
diff --git a/flink-docs/src/main/java/org/apache/flink/docs/configuration/ConfigOptionsDocGenerator.java b/flink-docs/src/main/java/org/apache/flink/docs/configuration/ConfigOptionsDocGenerator.java
index ac6b933..5866a92 100644
--- a/flink-docs/src/main/java/org/apache/flink/docs/configuration/ConfigOptionsDocGenerator.java
+++ b/flink-docs/src/main/java/org/apache/flink/docs/configuration/ConfigOptionsDocGenerator.java
@@ -233,6 +233,10 @@ public class ConfigOptionsDocGenerator {
ConfigGroups configGroups = optionsClass.getAnnotation(ConfigGroups.class);
List<OptionWithMetaInfo> allOptions = extractConfigOptions(optionsClass);
+ if (allOptions.isEmpty()) {
+ return Collections.emptyList();
+ }
+
List<Tuple2<ConfigGroup, String>> tables;
if (configGroups != null) {
tables = new ArrayList<>(configGroups.groups().length + 1);
@@ -240,12 +244,16 @@ public class ConfigOptionsDocGenerator {
for (ConfigGroup group : configGroups.groups()) {
List<OptionWithMetaInfo> configOptions = tree.findConfigOptions(group);
- sortOptions(configOptions);
- tables.add(Tuple2.of(group, toHtmlTable(configOptions)));
+ if (!configOptions.isEmpty()) {
+ sortOptions(configOptions);
+ tables.add(Tuple2.of(group, toHtmlTable(configOptions)));
+ }
}
List<OptionWithMetaInfo> configOptions = tree.getDefaultOptions();
- sortOptions(configOptions);
- tables.add(Tuple2.of(null, toHtmlTable(configOptions)));
+ if (!configOptions.isEmpty()) {
+ sortOptions(configOptions);
+ tables.add(Tuple2.of(null, toHtmlTable(configOptions)));
+ }
} else {
sortOptions(allOptions);
tables = Collections.singletonList(Tuple2.of(null, toHtmlTable(allOptions)));
diff --git a/flink-docs/src/test/java/org/apache/flink/docs/configuration/ConfigOptionsDocGeneratorTest.java b/flink-docs/src/test/java/org/apache/flink/docs/configuration/ConfigOptionsDocGeneratorTest.java
index c0a88aa..23fcc5f 100644
--- a/flink-docs/src/test/java/org/apache/flink/docs/configuration/ConfigOptionsDocGeneratorTest.java
+++ b/flink-docs/src/test/java/org/apache/flink/docs/configuration/ConfigOptionsDocGeneratorTest.java
@@ -46,6 +46,7 @@ import java.util.Map;
import static org.hamcrest.CoreMatchers.containsString;
import static org.hamcrest.CoreMatchers.not;
import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.empty;
import static org.junit.Assert.assertEquals;
/**
@@ -535,6 +536,15 @@ public class ConfigOptionsDocGeneratorTest {
assertEquals(expectedTable, htmlTable);
}
+ @ConfigGroups(groups = {@ConfigGroup(name = "firstGroup", keyPrefix = "first")})
+ static class EmptyConfigOptions {
+ }
+
+ @Test
+ public void testClassWithoutOptionsIsIgnored() {
+ assertThat(ConfigOptionsDocGenerator.generateTablesForClass(EmptyConfigOptions.class), empty());
+ }
+
static String getProjectRootDir() {
final String dirFromProperty = System.getProperty("rootDir");
if (dirFromProperty != null) {
[flink] 01/03: [hotfix][docs][config] Sort entries
Posted by ch...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit e30648863e853fe58e0410d7f52f1192b3fe02d4
Author: Chesnay Schepler <ch...@apache.org>
AuthorDate: Mon Aug 3 13:32:11 2020 +0200
[hotfix][docs][config] Sort entries
---
.../_includes/generated/all_taskmanager_network_section.html | 12 ++++++------
.../generated/netty_shuffle_environment_configuration.html | 12 ++++++------
2 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/docs/_includes/generated/all_taskmanager_network_section.html b/docs/_includes/generated/all_taskmanager_network_section.html
index 4e65fe8..6c2a11b 100644
--- a/docs/_includes/generated/all_taskmanager_network_section.html
+++ b/docs/_includes/generated/all_taskmanager_network_section.html
@@ -57,12 +57,6 @@
<td>The number of Netty client threads.</td>
</tr>
<tr>
- <td><h5>taskmanager.network.retries</h5></td>
- <td style="word-wrap: break-word;">0</td>
- <td>Integer</td>
- <td>The number of retry attempts for network communication. Currently it's only used for establishing input/output channel connections</td>
- </tr>
- <tr>
<td><h5>taskmanager.network.netty.num-arenas</h5></td>
<td style="word-wrap: break-word;">-1</td>
<td>Integer</td>
@@ -104,5 +98,11 @@
<td>Integer</td>
<td>Maximum backoff in milliseconds for partition requests of input channels.</td>
</tr>
+ <tr>
+ <td><h5>taskmanager.network.retries</h5></td>
+ <td style="word-wrap: break-word;">0</td>
+ <td>Integer</td>
+ <td>The number of retry attempts for network communication. Currently it's only used for establishing input/output channel connections</td>
+ </tr>
</tbody>
</table>
diff --git a/docs/_includes/generated/netty_shuffle_environment_configuration.html b/docs/_includes/generated/netty_shuffle_environment_configuration.html
index 94cb5ee..018296a 100644
--- a/docs/_includes/generated/netty_shuffle_environment_configuration.html
+++ b/docs/_includes/generated/netty_shuffle_environment_configuration.html
@@ -75,12 +75,6 @@
<td>The number of Netty client threads.</td>
</tr>
<tr>
- <td><h5>taskmanager.network.retries</h5></td>
- <td style="word-wrap: break-word;">0</td>
- <td>Integer</td>
- <td>The number of retry attempts for network communication. Currently it's only used for establishing input/output channel connections</td>
- </tr>
- <tr>
<td><h5>taskmanager.network.netty.num-arenas</h5></td>
<td style="word-wrap: break-word;">-1</td>
<td>Integer</td>
@@ -122,5 +116,11 @@
<td>Integer</td>
<td>Maximum backoff in milliseconds for partition requests of input channels.</td>
</tr>
+ <tr>
+ <td><h5>taskmanager.network.retries</h5></td>
+ <td style="word-wrap: break-word;">0</td>
+ <td>Integer</td>
+ <td>The number of retry attempts for network communication. Currently it's only used for establishing input/output channel connections</td>
+ </tr>
</tbody>
</table>