You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by gr...@apache.org on 2017/11/28 03:32:04 UTC
[3/4] flink git commit: [FLINK-7967] [config] Deprecate Hadoop
specific Flink configuration options
[FLINK-7967] [config] Deprecate Hadoop specific Flink configuration options
This closes #4946
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/fae83c0b
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/fae83c0b
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/fae83c0b
Branch: refs/heads/master
Commit: fae83c0b0c0a711d5d0f9de0b95fb6b703c4f912
Parents: 3561222
Author: zhangminglei <zm...@163.com>
Authored: Thu Nov 23 10:20:16 2017 +0800
Committer: Greg Hogan <co...@greghogan.com>
Committed: Mon Nov 27 22:30:43 2017 -0500
----------------------------------------------------------------------
docs/ops/config.md | 8 +++++---
.../org/apache/flink/configuration/ConfigConstants.java | 11 ++++++++++-
flink-dist/src/main/resources/flink-conf.yaml | 3 +++
3 files changed, 18 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/fae83c0b/docs/ops/config.md
----------------------------------------------------------------------
diff --git a/docs/ops/config.md b/docs/ops/config.md
index ed65880..6f15d01 100644
--- a/docs/ops/config.md
+++ b/docs/ops/config.md
@@ -72,8 +72,6 @@ definition. As another example, if this is set to `hdfs://localhost:9000/`, then
without explicit scheme definition, such as `/user/USERNAME/in.txt`, is going to be transformed into
`hdfs://localhost:9000/user/USERNAME/in.txt`. This scheme is used **ONLY** if no other scheme is specified (explicitly) in the user-provided `URI`.
-- `fs.hdfs.hadoopconf`: The absolute path to the Hadoop File System's (HDFS) configuration **directory** (OPTIONAL VALUE). Specifying this value allows programs to reference HDFS files using short URIs (`hdfs:///path/to/files`, without including the address and port of the NameNode in the file URI). Without this option, HDFS files can be accessed, but require fully qualified URIs like `hdfs://address:port/path/to/files`. This option also causes file writers to pick up the HDFS's default values for block sizes and replication factors. Flink will look for the "core-site.xml" and "hdfs-site.xml" files in the specified directory.
-
- `classloader.resolve-order`: Whether Flink should use a child-first `ClassLoader` when loading
user-code classes or a parent-first `ClassLoader`. Can be one of `parent-first` or `child-first`. (default: `child-first`)
@@ -246,9 +244,13 @@ Default value is the `akka.ask.timeout`.
### HDFS
+<div class="alert alert-warning">
+ <strong>Note:</strong> These keys are deprecated and it is recommended to configure the Hadoop path with the environment variable *HADOOP_CONF_DIR* instead.
+</div>
+
These parameters configure the default HDFS used by Flink. Setups that do not specify a HDFS configuration have to specify the full path to HDFS files (`hdfs://address:port/path/to/files`) Files will also be written with default HDFS parameters (block size, replication factor).
-- `fs.hdfs.hadoopconf`: The absolute path to the Hadoop configuration directory. The system will look for the "core-site.xml" and "hdfs-site.xml" files in that directory (DEFAULT: null).
+- `fs.hdfs.hadoopconf`: The absolute path to the Hadoop File System's (HDFS) configuration **directory** (OPTIONAL VALUE). Specifying this value allows programs to reference HDFS files using short URIs (`hdfs:///path/to/files`, without including the address and port of the NameNode in the file URI). Without this option, HDFS files can be accessed, but require fully qualified URIs like `hdfs://address:port/path/to/files`. This option also causes file writers to pick up the HDFS's default values for block sizes and replication factors. Flink will look for the "core-site.xml" and "hdfs-site.xml" files in the specified directory.
- `fs.hdfs.hdfsdefault`: The absolute path of Hadoop's own configuration file "hdfs-default.xml" (DEFAULT: null).
http://git-wip-us.apache.org/repos/asf/flink/blob/fae83c0b/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
----------------------------------------------------------------------
diff --git a/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java b/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
index f80bd9b..5fd7085 100644
--- a/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
+++ b/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
@@ -595,19 +595,28 @@ public final class ConfigConstants {
/**
* Path to hdfs-defaul.xml file
+ *
+ * @deprecated Use environment variable HADOOP_CONF_DIR instead.
*/
+ @Deprecated
public static final String HDFS_DEFAULT_CONFIG = "fs.hdfs.hdfsdefault";
/**
* Path to hdfs-site.xml file
+ *
+ * @deprecated Use environment variable HADOOP_CONF_DIR instead.
*/
+ @Deprecated
public static final String HDFS_SITE_CONFIG = "fs.hdfs.hdfssite";
/**
* Path to Hadoop configuration
+ *
+ * @deprecated Use environment variable HADOOP_CONF_DIR instead.
*/
+ @Deprecated
public static final String PATH_HADOOP_CONFIG = "fs.hdfs.hadoopconf";
-
+
// ------------------------ File System Behavior ------------------------
/**
http://git-wip-us.apache.org/repos/asf/flink/blob/fae83c0b/flink-dist/src/main/resources/flink-conf.yaml
----------------------------------------------------------------------
diff --git a/flink-dist/src/main/resources/flink-conf.yaml b/flink-dist/src/main/resources/flink-conf.yaml
index ddff9c8..fca97df 100644
--- a/flink-dist/src/main/resources/flink-conf.yaml
+++ b/flink-dist/src/main/resources/flink-conf.yaml
@@ -151,6 +151,9 @@ web.port: 8081
# Path to the Hadoop configuration directory.
#
+# Note: these keys are deprecated and it is recommended to configure the Hadoop
+# path with the environment variable 'HADOOP_CONF_DIR' instead.
+#
# This configuration is used when writing into HDFS. Unless specified otherwise,
# HDFS file creation will use HDFS default settings with respect to block-size,
# replication factor, etc.