You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ma...@apache.org on 2022/01/10 08:29:12 UTC
[flink] branch master updated: [FLINK-25224][filesystem] Bump Hadoop version to 2.8.5
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 604b128 [FLINK-25224][filesystem] Bump Hadoop version to 2.8.5
604b128 is described below
commit 604b128eb5143664deacca302e94229a24caf4f7
Author: liufangqi <10...@qq.com>
AuthorDate: Sat Jan 8 09:16:37 2022 +0800
[FLINK-25224][filesystem] Bump Hadoop version to 2.8.5
---
azure-pipelines.yml | 4 ++--
docs/content.zh/docs/deployment/filesystems/gcs.md | 2 +-
.../docs/deployment/resource-providers/yarn.md | 4 ++--
.../docs/dev/dataset/hadoop_compatibility.md | 2 +-
.../docs/connectors/dataset/formats/hadoop.md | 2 +-
.../docs/connectors/datastream/formats/hadoop.md | 2 +-
docs/content/docs/deployment/filesystems/gcs.md | 2 +-
.../docs/deployment/resource-providers/yarn.md | 4 ++--
docs/content/docs/dev/dataset/hadoop_map_reduce.md | 2 +-
.../test-scripts/common_yarn_docker.sh | 6 +++---
.../docker-hadoop-secure-cluster/Dockerfile | 2 +-
.../docker-hadoop-secure-cluster/README.md | 2 +-
flink-jepsen/src/jepsen/flink/flink.clj | 2 +-
flink-yarn-tests/README.md | 2 +-
pom.xml | 2 +-
tools/azure-pipelines/build-apache-repo.yml | 21 +++++----------------
16 files changed, 25 insertions(+), 36 deletions(-)
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index e836ac1..6fbf24e 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -77,7 +77,7 @@ stages:
vmImage: 'ubuntu-20.04'
e2e_pool_definition:
vmImage: 'ubuntu-20.04'
- environment: PROFILE="-Dhadoop.version=2.8.3 -Dinclude_hadoop_aws -Dscala-2.12 -Darchunit.freeze.store.default.allowStoreUpdate=false"
+ environment: PROFILE="-Dhadoop.version=2.8.5 -Dinclude_hadoop_aws -Dscala-2.12 -Darchunit.freeze.default.allowStoreUpdate=false"
run_end_to_end: false
container: flink-build-container
jdk: 8
@@ -94,5 +94,5 @@ stages:
- template: tools/azure-pipelines/build-python-wheels.yml
parameters:
stage_name: cron_python_wheels
- environment: PROFILE="-Dhadoop.version=2.8.3 -Dinclude_hadoop_aws -Dscala-2.12 -Darchunit.freeze.store.default.allowStoreUpdate=false"
+ environment: PROFILE="-Dhadoop.version=2.8.5 -Dinclude_hadoop_aws -Dscala-2.12 -Darchunit.freeze.store.default.allowStoreUpdate=false"
container: flink-build-container
diff --git a/docs/content.zh/docs/deployment/filesystems/gcs.md b/docs/content.zh/docs/deployment/filesystems/gcs.md
index e94108e..9574060 100644
--- a/docs/content.zh/docs/deployment/filesystems/gcs.md
+++ b/docs/content.zh/docs/deployment/filesystems/gcs.md
@@ -68,7 +68,7 @@ You must include the following jars in Flink's `lib` directory to connect Flink
</dependency>
```
-We have tested with `flink-shared-hadoop2-uber` version >= `2.8.3-1.8.3`.
+We have tested with `flink-shared-hadoop2-uber` version >= `2.8.5-1.8.3`.
You can track the latest version of the [gcs-connector hadoop 2](https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar).
### Authentication to access GCS
diff --git a/docs/content.zh/docs/deployment/resource-providers/yarn.md b/docs/content.zh/docs/deployment/resource-providers/yarn.md
index 417f5c4..bb4acbd 100644
--- a/docs/content.zh/docs/deployment/resource-providers/yarn.md
+++ b/docs/content.zh/docs/deployment/resource-providers/yarn.md
@@ -40,7 +40,7 @@ Flink can dynamically allocate and de-allocate TaskManager resources depending o
### Preparation
-This *Getting Started* section assumes a functional YARN environment, starting from version 2.4.1. YARN environments are provided most conveniently through services such as Amazon EMR, Google Cloud DataProc or products like Cloudera. [Manually setting up a YARN environment locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html) or [on a cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html) is not [...]
+This *Getting Started* section assumes a functional YARN environment, starting from version 2.8.5. YARN environments are provided most conveniently through services such as Amazon EMR, Google Cloud DataProc or products like Cloudera. [Manually setting up a YARN environment locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html) or [on a cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html) is not [...]
- Make sure your YARN cluster is ready for accepting Flink applications by running `yarn top`. It should show no error messages.
- Download a recent Flink distribution from the [download page]({{< downloads >}}) and unpack it.
@@ -219,7 +219,7 @@ Hadoop YARN 2.4.0 has a major bug (fixed in 2.5.0) preventing container restarts
### Supported Hadoop versions.
-Flink on YARN is compiled against Hadoop 2.4.1, and all Hadoop versions `>= 2.4.1` are supported, including Hadoop 3.x.
+Flink on YARN is compiled against Hadoop 2.8.5, and all Hadoop versions `>= 2.8.5` are supported, including Hadoop 3.x.
For providing Flink with the required Hadoop dependencies, we recommend setting the `HADOOP_CLASSPATH` environment variable already introduced in the [Getting Started / Preparation](#preparation) section.
diff --git a/docs/content.zh/docs/dev/dataset/hadoop_compatibility.md b/docs/content.zh/docs/dev/dataset/hadoop_compatibility.md
index 4cfc38d..18851fa 100644
--- a/docs/content.zh/docs/dev/dataset/hadoop_compatibility.md
+++ b/docs/content.zh/docs/dev/dataset/hadoop_compatibility.md
@@ -74,7 +74,7 @@ a `hadoop-client` dependency such as:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.8.3</version>
+ <version>2.8.5</version>
<scope>provided</scope>
</dependency>
```
diff --git a/docs/content/docs/connectors/dataset/formats/hadoop.md b/docs/content/docs/connectors/dataset/formats/hadoop.md
index 64702cb..b517c98 100644
--- a/docs/content/docs/connectors/dataset/formats/hadoop.md
+++ b/docs/content/docs/connectors/dataset/formats/hadoop.md
@@ -49,7 +49,7 @@ a `hadoop-client` dependency such as:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.8.3</version>
+ <version>2.8.5</version>
<scope>provided</scope>
</dependency>
```
diff --git a/docs/content/docs/connectors/datastream/formats/hadoop.md b/docs/content/docs/connectors/datastream/formats/hadoop.md
index 56553ca..e2b2c9f 100644
--- a/docs/content/docs/connectors/datastream/formats/hadoop.md
+++ b/docs/content/docs/connectors/datastream/formats/hadoop.md
@@ -50,7 +50,7 @@ a `hadoop-client` dependency such as:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.8.3</version>
+ <version>2.8.5</version>
<scope>provided</scope>
</dependency>
```
diff --git a/docs/content/docs/deployment/filesystems/gcs.md b/docs/content/docs/deployment/filesystems/gcs.md
index e94108e..9574060 100644
--- a/docs/content/docs/deployment/filesystems/gcs.md
+++ b/docs/content/docs/deployment/filesystems/gcs.md
@@ -68,7 +68,7 @@ You must include the following jars in Flink's `lib` directory to connect Flink
</dependency>
```
-We have tested with `flink-shared-hadoop2-uber` version >= `2.8.3-1.8.3`.
+We have tested with `flink-shared-hadoop2-uber` version >= `2.8.5-1.8.3`.
You can track the latest version of the [gcs-connector hadoop 2](https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar).
### Authentication to access GCS
diff --git a/docs/content/docs/deployment/resource-providers/yarn.md b/docs/content/docs/deployment/resource-providers/yarn.md
index 2426d95..1a02931 100644
--- a/docs/content/docs/deployment/resource-providers/yarn.md
+++ b/docs/content/docs/deployment/resource-providers/yarn.md
@@ -40,7 +40,7 @@ Flink can dynamically allocate and de-allocate TaskManager resources depending o
### Preparation
-This *Getting Started* section assumes a functional YARN environment, starting from version 2.4.1. YARN environments are provided most conveniently through services such as Amazon EMR, Google Cloud DataProc or products like Cloudera. [Manually setting up a YARN environment locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html) or [on a cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html) is not [...]
+This *Getting Started* section assumes a functional YARN environment, starting from version 2.8.5. YARN environments are provided most conveniently through services such as Amazon EMR, Google Cloud DataProc or products like Cloudera. [Manually setting up a YARN environment locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html) or [on a cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html) is not [...]
- Make sure your YARN cluster is ready for accepting Flink applications by running `yarn top`. It should show no error messages.
- Download a recent Flink distribution from the [download page]({{< downloads >}}) and unpack it.
@@ -231,7 +231,7 @@ Hadoop YARN 2.4.0 has a major bug (fixed in 2.5.0) preventing container restarts
### Supported Hadoop versions.
-Flink on YARN is compiled against Hadoop 2.4.1, and all Hadoop versions `>= 2.4.1` are supported, including Hadoop 3.x.
+Flink on YARN is compiled against Hadoop 2.8.5, and all Hadoop versions `>= 2.8.5` are supported, including Hadoop 3.x.
For providing Flink with the required Hadoop dependencies, we recommend setting the `HADOOP_CLASSPATH` environment variable already introduced in the [Getting Started / Preparation](#preparation) section.
diff --git a/docs/content/docs/dev/dataset/hadoop_map_reduce.md b/docs/content/docs/dev/dataset/hadoop_map_reduce.md
index 193217f..9ef907b 100644
--- a/docs/content/docs/dev/dataset/hadoop_map_reduce.md
+++ b/docs/content/docs/dev/dataset/hadoop_map_reduce.md
@@ -62,7 +62,7 @@ a `hadoop-client` dependency such as:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.8.3</version>
+ <version>2.8.5</version>
<scope>provided</scope>
</dependency>
```
diff --git a/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh b/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh
index 66ab373..602f907 100755
--- a/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh
+++ b/flink-end-to-end-tests/test-scripts/common_yarn_docker.sh
@@ -95,11 +95,11 @@ function start_hadoop_cluster() {
function build_image() {
echo "Predownloading Hadoop tarball"
- cache_path=$(get_artifact "http://archive.apache.org/dist/hadoop/common/hadoop-2.8.4/hadoop-2.8.4.tar.gz")
- ln "$cache_path" "$END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/hadoop-2.8.4.tar.gz"
+ cache_path=$(get_artifact "http://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz")
+ ln "$cache_path" "$END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/hadoop-2.8.5.tar.gz"
echo "Building Hadoop Docker container"
- docker build --build-arg HADOOP_VERSION=2.8.4 \
+ docker build --build-arg HADOOP_VERSION=2.8.5 \
-f $END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/Dockerfile \
-t flink/docker-hadoop-secure-cluster:latest \
$END_TO_END_DIR/test-scripts/docker-hadoop-secure-cluster/
diff --git a/flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/Dockerfile b/flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/Dockerfile
index cdb6c48..5aa00f1 100644
--- a/flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/Dockerfile
+++ b/flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/Dockerfile
@@ -53,7 +53,7 @@ RUN set -x \
RUN sed -i 's/^#crypto.policy=unlimited/crypto.policy=unlimited/' $JAVA_HOME/jre/lib/security/java.security
-ARG HADOOP_VERSION=2.8.4
+ARG HADOOP_VERSION=2.8.5
COPY hadoop-${HADOOP_VERSION}.tar.gz /tmp/hadoop.tar.gz
diff --git a/flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/README.md b/flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/README.md
index a9f9609..68e4c5f 100644
--- a/flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/README.md
+++ b/flink-end-to-end-tests/test-scripts/docker-hadoop-secure-cluster/README.md
@@ -12,7 +12,7 @@ Versions
--------
* JDK8
-* Hadoop 2.8.3
+* Hadoop 2.8.5
Default Environment Variables
-----------------------------
diff --git a/flink-jepsen/src/jepsen/flink/flink.clj b/flink-jepsen/src/jepsen/flink/flink.clj
index 9e17144..9c8abd3 100644
--- a/flink-jepsen/src/jepsen/flink/flink.clj
+++ b/flink-jepsen/src/jepsen/flink/flink.clj
@@ -33,7 +33,7 @@
[nemesis :as fn]]))
(def default-flink-dist-url "https://archive.apache.org/dist/flink/flink-1.6.0/flink-1.6.0-bin-hadoop28-scala_2.11.tgz")
-(def hadoop-dist-url "https://archive.apache.org/dist/hadoop/common/hadoop-2.8.3/hadoop-2.8.3.tar.gz")
+(def hadoop-dist-url "https://archive.apache.org/dist/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz")
(def kafka-dist-url "http://mirror.funkfreundelandshut.de/apache/kafka/2.2.2/kafka_2.11-2.2.2.tgz")
(def deb-zookeeper-package "3.4.9-3+deb9u2")
diff --git a/flink-yarn-tests/README.md b/flink-yarn-tests/README.md
index 998b03d..f31ff18 100644
--- a/flink-yarn-tests/README.md
+++ b/flink-yarn-tests/README.md
@@ -10,7 +10,7 @@ There are several things to consider when running these tests locally:
* Each `YARN*ITCase` will have a local working directory for resources like logs to be stored. These
working directories are located in `flink-yarn-tests/target/` (see
`find flink-yarn-tests/target -name "*.err" -or -name "*.out"` for the test's output).
-* There is a known problem causing test instabilities due to our usage of Hadoop 2.8.3 executing the
+* There is a known problem causing test instabilities due to our usage of Hadoop 2.8.5 executing the
tests. This is caused by a bug [YARN-7007](https://issues.apache.org/jira/browse/YARN-7007) that
got fixed in [Hadoop 2.8.6](https://issues.apache.org/jira/projects/YARN/versions/12344056). See
[FLINK-15534](https://issues.apache.org/jira/browse/FLINK-15534) for further details on the
diff --git a/pom.xml b/pom.xml
index b646c94..1c1c89f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -94,7 +94,7 @@ under the License.
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
- <hadoop.version>2.4.1</hadoop.version>
+ <hadoop.version>2.8.5</hadoop.version>
<!-- Need to use a user property here because the surefire
forkCount is not exposed as a property. With this we can set
it on the "mvn" commandline in travis. -->
diff --git a/tools/azure-pipelines/build-apache-repo.yml b/tools/azure-pipelines/build-apache-repo.yml
index bc7e282..924bfa6 100644
--- a/tools/azure-pipelines/build-apache-repo.yml
+++ b/tools/azure-pipelines/build-apache-repo.yml
@@ -70,7 +70,7 @@ stages:
name: Default
e2e_pool_definition:
vmImage: 'ubuntu-20.04'
- environment: PROFILE="-Dhadoop.version=2.8.3 -Dinclude_hadoop_aws -Dscala-2.12"
+ environment: PROFILE="-Dhadoop.version=2.8.5 -Dinclude_hadoop_aws -Dscala-2.12"
run_end_to_end: false
container: flink-build-container
jdk: 8
@@ -110,18 +110,7 @@ stages:
vmImage: 'ubuntu-20.04'
e2e_pool_definition:
vmImage: 'ubuntu-20.04'
- environment: PROFILE="-Dhadoop.version=2.8.3 -Dinclude_hadoop_aws -Dscala-2.12"
- run_end_to_end: true
- container: flink-build-container
- jdk: 8
- - template: jobs-template.yml
- parameters:
- stage_name: cron_hadoop241
- test_pool_definition:
- name: Default
- e2e_pool_definition:
- vmImage: 'ubuntu-20.04'
- environment: PROFILE="-Dhadoop.version=2.4.1 -Pskip-hive-tests"
+ environment: PROFILE="-Dhadoop.version=2.8.5 -Dinclude_hadoop_aws -Dscala-2.12"
run_end_to_end: true
container: flink-build-container
jdk: 8
@@ -143,7 +132,7 @@ stages:
name: Default
e2e_pool_definition:
vmImage: 'ubuntu-20.04'
- environment: PROFILE="-Dhadoop.version=2.8.3 -Dinclude_hadoop_aws -Dscala-2.12 -Djdk11 -Pjava11-target"
+ environment: PROFILE="-Dhadoop.version=2.8.5 -Dinclude_hadoop_aws -Dscala-2.12 -Djdk11 -Pjava11-target"
run_end_to_end: true
container: flink-build-container
jdk: 11
@@ -154,7 +143,7 @@ stages:
name: Default
e2e_pool_definition:
vmImage: 'ubuntu-20.04'
- environment: PROFILE="-Dhadoop.version=2.8.3 -Dinclude_hadoop_aws -Dscala-2.12 -Penable-adaptive-scheduler"
+ environment: PROFILE="-Dhadoop.version=2.8.5 -Dinclude_hadoop_aws -Dscala-2.12 -Penable-adaptive-scheduler"
run_end_to_end: true
container: flink-build-container
jdk: 8
@@ -166,5 +155,5 @@ stages:
- template: build-python-wheels.yml
parameters:
stage_name: cron_python_wheels
- environment: PROFILE="-Dhadoop.version=2.8.3 -Dinclude_hadoop_aws -Dscala-2.12"
+ environment: PROFILE="-Dhadoop.version=2.8.5 -Dinclude_hadoop_aws -Dscala-2.12"
container: flink-build-container