You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by gr...@apache.org on 2017/11/28 03:32:04 UTC

[3/4] flink git commit: [FLINK-7967] [config] Deprecate Hadoop specific Flink configuration options

[FLINK-7967] [config] Deprecate Hadoop specific Flink configuration options

This closes #4946


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/fae83c0b
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/fae83c0b
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/fae83c0b

Branch: refs/heads/master
Commit: fae83c0b0c0a711d5d0f9de0b95fb6b703c4f912
Parents: 3561222
Author: zhangminglei <zm...@163.com>
Authored: Thu Nov 23 10:20:16 2017 +0800
Committer: Greg Hogan <co...@greghogan.com>
Committed: Mon Nov 27 22:30:43 2017 -0500

----------------------------------------------------------------------
 docs/ops/config.md                                       |  8 +++++---
 .../org/apache/flink/configuration/ConfigConstants.java  | 11 ++++++++++-
 flink-dist/src/main/resources/flink-conf.yaml            |  3 +++
 3 files changed, 18 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/fae83c0b/docs/ops/config.md
----------------------------------------------------------------------
diff --git a/docs/ops/config.md b/docs/ops/config.md
index ed65880..6f15d01 100644
--- a/docs/ops/config.md
+++ b/docs/ops/config.md
@@ -72,8 +72,6 @@ definition. As another example, if this is set to `hdfs://localhost:9000/`, then
 without explicit scheme definition, such as `/user/USERNAME/in.txt`, is going to be transformed into
 `hdfs://localhost:9000/user/USERNAME/in.txt`. This scheme is used **ONLY** if no other scheme is specified (explicitly) in the user-provided `URI`.
 
-- `fs.hdfs.hadoopconf`: The absolute path to the Hadoop File System's (HDFS) configuration **directory** (OPTIONAL VALUE). Specifying this value allows programs to reference HDFS files using short URIs (`hdfs:///path/to/files`, without including the address and port of the NameNode in the file URI). Without this option, HDFS files can be accessed, but require fully qualified URIs like `hdfs://address:port/path/to/files`. This option also causes file writers to pick up the HDFS's default values for block sizes and replication factors. Flink will look for the "core-site.xml" and "hdfs-site.xml" files in the specified directory.
-
 - `classloader.resolve-order`: Whether Flink should use a child-first `ClassLoader` when loading
 user-code classes or a parent-first `ClassLoader`. Can be one of `parent-first` or `child-first`. (default: `child-first`)
 
@@ -246,9 +244,13 @@ Default value is the `akka.ask.timeout`.
 
 ### HDFS
 
+<div class="alert alert-warning">
+  <strong>Note:</strong> These keys are deprecated and it is recommended to configure the Hadoop path with the environment variable *HADOOP_CONF_DIR* instead.
+</div>
+
 These parameters configure the default HDFS used by Flink. Setups that do not specify a HDFS configuration have to specify the full path to HDFS files (`hdfs://address:port/path/to/files`) Files will also be written with default HDFS parameters (block size, replication factor).
 
-- `fs.hdfs.hadoopconf`: The absolute path to the Hadoop configuration directory. The system will look for the "core-site.xml" and "hdfs-site.xml" files in that directory (DEFAULT: null).
+- `fs.hdfs.hadoopconf`: The absolute path to the Hadoop File System's (HDFS) configuration **directory** (OPTIONAL VALUE). Specifying this value allows programs to reference HDFS files using short URIs (`hdfs:///path/to/files`, without including the address and port of the NameNode in the file URI). Without this option, HDFS files can be accessed, but require fully qualified URIs like `hdfs://address:port/path/to/files`. This option also causes file writers to pick up the HDFS's default values for block sizes and replication factors. Flink will look for the "core-site.xml" and "hdfs-site.xml" files in the specified directory.
 
 - `fs.hdfs.hdfsdefault`: The absolute path of Hadoop's own configuration file "hdfs-default.xml" (DEFAULT: null).
 

http://git-wip-us.apache.org/repos/asf/flink/blob/fae83c0b/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
----------------------------------------------------------------------
diff --git a/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java b/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
index f80bd9b..5fd7085 100644
--- a/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
+++ b/flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java
@@ -595,19 +595,28 @@ public final class ConfigConstants {
 
 	/**
 	 * Path to hdfs-defaul.xml file
+	 *
+	 * @deprecated Use environment variable HADOOP_CONF_DIR instead.
 	 */
+	@Deprecated
 	public static final String HDFS_DEFAULT_CONFIG = "fs.hdfs.hdfsdefault";
 	
 	/**
 	 * Path to hdfs-site.xml file
+	 *
+	 * @deprecated Use environment variable HADOOP_CONF_DIR instead.
 	 */
+	@Deprecated
 	public static final String HDFS_SITE_CONFIG = "fs.hdfs.hdfssite";
 	
 	/**
 	 * Path to Hadoop configuration
+	 *
+	 * @deprecated Use environment variable HADOOP_CONF_DIR instead.
 	 */
+	@Deprecated
 	public static final String PATH_HADOOP_CONFIG = "fs.hdfs.hadoopconf";
-	
+
 	// ------------------------ File System Behavior ------------------------
 
 	/**

http://git-wip-us.apache.org/repos/asf/flink/blob/fae83c0b/flink-dist/src/main/resources/flink-conf.yaml
----------------------------------------------------------------------
diff --git a/flink-dist/src/main/resources/flink-conf.yaml b/flink-dist/src/main/resources/flink-conf.yaml
index ddff9c8..fca97df 100644
--- a/flink-dist/src/main/resources/flink-conf.yaml
+++ b/flink-dist/src/main/resources/flink-conf.yaml
@@ -151,6 +151,9 @@ web.port: 8081
 
 # Path to the Hadoop configuration directory.
 #
+# Note: these keys are deprecated and it is recommended to configure the Hadoop
+# path with the environment variable 'HADOOP_CONF_DIR' instead.
+#
 # This configuration is used when writing into HDFS. Unless specified otherwise,
 # HDFS file creation will use HDFS default settings with respect to block-size,
 # replication factor, etc.