You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by rm...@apache.org on 2014/08/27 16:10:32 UTC
[1/2] git commit: [FLINK-1068] Remove cdh4 specific build profile.
Repository: incubator-flink
Updated Branches:
refs/heads/release-0.6.1 10b556c9e -> 8acac29ff
[FLINK-1068] Remove cdh4 specific build profile.
Users using CDH4 can now build Flink against Hadoop 2.0.0-alpha.
Project: http://git-wip-us.apache.org/repos/asf/incubator-flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-flink/commit/d3a97995
Tree: http://git-wip-us.apache.org/repos/asf/incubator-flink/tree/d3a97995
Diff: http://git-wip-us.apache.org/repos/asf/incubator-flink/diff/d3a97995
Branch: refs/heads/release-0.6.1
Commit: d3a9799593dea67df143a4d07b11e3d1882fbebf
Parents: 10b556c
Author: Robert Metzger <rm...@apache.org>
Authored: Wed Aug 27 16:07:52 2014 +0200
Committer: Robert Metzger <rm...@apache.org>
Committed: Wed Aug 27 16:07:52 2014 +0200
----------------------------------------------------------------------
docs/building.md | 39 +++++++-------------
flink-addons/flink-avro/pom.xml | 17 ---------
flink-addons/flink-hadoop-compatibility/pom.xml | 23 ------------
flink-addons/flink-spargel/pom.xml | 22 -----------
flink-dist/pom.xml | 15 --------
flink-runtime/pom.xml | 25 -------------
flink-test-utils/pom.xml | 17 ---------
flink-tests/pom.xml | 12 ------
pom.xml | 5 +++
tools/create_release_files.sh | 4 +-
10 files changed, 19 insertions(+), 160 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/docs/building.md
----------------------------------------------------------------------
diff --git a/docs/building.md b/docs/building.md
index a07b324..842dbc1 100644
--- a/docs/building.md
+++ b/docs/building.md
@@ -3,7 +3,7 @@ title: "Build Flink"
---
-In order to build Flink, you need the source code. Either download the source of a release or clone the git repository. In addition to that, you need Maven 3 and a JDK (Java Development Kit). Note that you can not build Flink with Oracle JDK 6 due to a unresolved bug in the Java compiler. It works well with OpenJDK 6 and all Java 7 and 8 compilers.
+In order to build Flink, you need the source code. Either download the source of a release or clone the git repository. In addition to that, you need Maven 3 and a JDK (Java Development Kit). Note that you can not build Flink with Oracle JDK 6 due to a unresolved bug in the Oracle Java compiler. It works well with OpenJDK 6 and all Java 7 and 8 compilers.
To clone from git, enter:
```
@@ -36,7 +36,7 @@ ERROR: The job was not successfully submitted to the nephele job manager:
There are two main versions of Hadoop that we need to differentiate:
- Hadoop 1, with all versions starting with zero or one, like 0.20, 0.23 or 1.2.1.
-- Hadoop 2 with all versions including or higher than 2.2.0.
+- Hadoop 2, with all versions starting with 2, like 2.2.0.
The main differentiation between Hadoop 1 and Hadoop 2 is the availability of Hadoop YARN (Hadoops cluster resource manager).
**To build Flink for Hadoop 2**, issue the following command:
@@ -55,11 +55,21 @@ mvn clean package -DskipTests -Dhadoop.profile=2 -Dhadoop.version=2.4.1
**To build Flink against a vendor specific Hadoop version**, issue the following command:
```
-mvn clean package -DskipTests -Pvendor-repos -Dhadoop.version=2.2.0-cdh5.0.0-beta-2
+mvn clean package -DskipTests -Pvendor-repos -Dhadoop.profile=2 -Dhadoop.version=2.2.0-cdh5.0.0-beta-2
```
The `-Pvendor-repos` activates a Maven [build profile](http://maven.apache.org/guides/introduction/introduction-to-profiles.html) that includes the repositories of popular Hadoop vendors such as Cloudera, Hortonworks, or MapR.
+**Build Flink for `hadoop2` versions before 2.2.0**
+
+Maven will automatically build Flink with its YARN client if the `-Dhadoop.profile=2` is set. But there were some changes in Hadoop versions before the 2.2.0 Hadoop release that are not supported by Flink's YARN client. Therefore, you can disable building the YARN client with the following string: `-P\!include-yarn`.
+
+So if you are building Flink for Hadoop `2.0.0-alpha`, use the following command:
+
+```bash
+-P\!include-yarn -Dhadoop.profile=2 -Dhadoop.version=2.0.0-alpha
+```
+
### Background
The builds with Maven are controlled by [properties](http://maven.apache.org/pom.html#Properties) and <a href="http://maven.apache.org/guides/introduction/introduction-to-profiles.html">build profiles</a>.
@@ -69,26 +79,3 @@ Depending on the profile, there are two Hadoop versions, set via properties. For
You can change these versions with the `hadoop-two.version` (or `hadoop-one.version`) property. For example `-Dhadoop-two.version=2.4.0`.
-
-### Example for Cloudera Hadoop 5 Beta 2
-
-
-```
-mvn -Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.2.0-cdh5.0.0-beta-2 -DskipTests package
-```
-
-The commands in detail:
-
-* `-Dhadoop.profile=2` activates the Hadoop YARN profile of Flink. This will enable all components of Flink that are compatible with Hadoop 2.2
-* `-Pvendor-repos` is adding the Maven repositories of MapR, Cloudera and Hortonworks into your Maven build.
-* `-Dhadoop.version=2.2.0-cdh5.0.0-beta-2` sets a special version of the Hadoop dependencies. Make sure that the specified Hadoop version is compatible with the profile you activated.
-
-If you want to build HDFS for Hadoop 2 without YARN, use the following parameter:
-
-```
--P!include-yarn
-```
-
-Some Cloudera versions (such as `2.0.0-cdh4.2.0`) require this, since they have a new HDFS version with the old YARN API.
-
-Please post to the _Flink mailinglist_(dev@flink.incubator.apache.org) or create an issue on [Jira]({{site.FLINK_ISSUES_URL}}), if you have issues with your YARN setup and Flink.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/flink-addons/flink-avro/pom.xml
----------------------------------------------------------------------
diff --git a/flink-addons/flink-avro/pom.xml b/flink-addons/flink-avro/pom.xml
index b95ca59..36ad445 100644
--- a/flink-addons/flink-avro/pom.xml
+++ b/flink-addons/flink-avro/pom.xml
@@ -183,23 +183,6 @@ under the License.
<maven.javadoc.skip>true</maven.javadoc.skip>
</properties>
</profile>
- <profile>
- <id>cdh4</id>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/flink-addons/flink-hadoop-compatibility/pom.xml
----------------------------------------------------------------------
diff --git a/flink-addons/flink-hadoop-compatibility/pom.xml b/flink-addons/flink-hadoop-compatibility/pom.xml
index 52c2917..746b782 100644
--- a/flink-addons/flink-hadoop-compatibility/pom.xml
+++ b/flink-addons/flink-hadoop-compatibility/pom.xml
@@ -78,28 +78,5 @@ under the License.
</dependency>
</dependencies>
</profile>
- <profile>
- <id>cdh4</id>
- <!--
- See flink-runtime/pom.xml for details
- -->
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-mapreduce-client-core</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/flink-addons/flink-spargel/pom.xml
----------------------------------------------------------------------
diff --git a/flink-addons/flink-spargel/pom.xml b/flink-addons/flink-spargel/pom.xml
index 64d3877..ee39654 100644
--- a/flink-addons/flink-spargel/pom.xml
+++ b/flink-addons/flink-spargel/pom.xml
@@ -57,26 +57,4 @@ under the License.
<scope>test</scope>
</dependency>
</dependencies>
-
- <profiles>
-
- <profile>
- <!-- Needed for the tests -->
- <id>cdh4</id>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
- </profile>
- </profiles>
</project>
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/flink-dist/pom.xml
----------------------------------------------------------------------
diff --git a/flink-dist/pom.xml b/flink-dist/pom.xml
index ac340f4..8c46149 100644
--- a/flink-dist/pom.xml
+++ b/flink-dist/pom.xml
@@ -335,21 +335,6 @@ under the License.
</plugins>
</build>
</profile>
- <profile>
- <id>cdh4</id>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- </dependencies>
- </profile>
</profiles>
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/flink-runtime/pom.xml
----------------------------------------------------------------------
diff --git a/flink-runtime/pom.xml b/flink-runtime/pom.xml
index c67162f..8aec4b8 100644
--- a/flink-runtime/pom.xml
+++ b/flink-runtime/pom.xml
@@ -224,30 +224,5 @@ under the License.
</dependency>
</dependencies>
</profile>
-
- <profile>
- <id>cdh4</id>
- <!--
-
- It seems http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/cdh4ig_topic_31.html
- that this particular CDH4 version has already separate packages for hdfs and common.
-
- To activate the cdh4 profile, use the -Pcdh4 flag. (You probably want -Pcdh4,vendor-repos)
- You also HAVE TO set the -Dhadoop.cdh4.hadoop.version property (for example to: 2.0.0-cdh4.2.0)
-
- -->
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- </dependencies>
- </profile>
</profiles>
</project>
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/flink-test-utils/pom.xml
----------------------------------------------------------------------
diff --git a/flink-test-utils/pom.xml b/flink-test-utils/pom.xml
index 328d9a4..1a0fe1b 100644
--- a/flink-test-utils/pom.xml
+++ b/flink-test-utils/pom.xml
@@ -66,21 +66,4 @@ under the License.
<version>4.11</version>
</dependency>
</dependencies>
- <profiles>
- <profile>
- <id>cdh4</id>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-hdfs</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- </dependencies>
- </profile>
- </profiles>
</project>
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/flink-tests/pom.xml
----------------------------------------------------------------------
diff --git a/flink-tests/pom.xml b/flink-tests/pom.xml
index b4b11ea..950e78e 100644
--- a/flink-tests/pom.xml
+++ b/flink-tests/pom.xml
@@ -236,16 +236,4 @@ under the License.
</plugins>
</pluginManagement>
</build>
- <profiles>
- <profile>
- <id>cdh4</id>
- <dependencies>
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <version>${hadoop.cdh4.hadoop.version}</version>
- </dependency>
- </dependencies>
- </profile>
- </profiles>
</project>
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index c9b56b6..e7702cb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -187,6 +187,11 @@ under the License.
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>${hadoop.version}</version>
+ </dependency>
</dependencies>
</dependencyManagement>
</profile>
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/d3a97995/tools/create_release_files.sh
----------------------------------------------------------------------
diff --git a/tools/create_release_files.sh b/tools/create_release_files.sh
index aa8875d..d0b0809 100755
--- a/tools/create_release_files.sh
+++ b/tools/create_release_files.sh
@@ -117,11 +117,9 @@ make_binary_release() {
}
make_binary_release "hadoop1" ""
-make_binary_release "cdh4" "-Pvendor-repos,cdh4 -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -Dhadoop.cdh4.hadoop.version=2.0.0-cdh4.2.0"
-# make_binary_release "cdh5" "-Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.3.0+cdh5.1.0+795"
+make_binary_release "hadoop200alpha" "-P\!include-yarn -Dhadoop.profile=2 -Dhadoop.version=2.0.0-alpha"
make_binary_release "hadoop2" "-Dhadoop.profile=2"
# make_binary_release "mapr4" "-Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.3.0-mapr-4.0.0-FCS"
-# make_binary_release "hdp21" "-Dhadoop.profile=2 -Pvendor-repos -Dhadoop.version=2.4.0.2.1.3.0-563"
# Copy data
[2/2] git commit: [FLINK-1067] Replaced Stratosphere by Flink in FAQ
Posted by rm...@apache.org.
[FLINK-1067] Replaced Stratosphere by Flink in FAQ
Project: http://git-wip-us.apache.org/repos/asf/incubator-flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-flink/commit/8acac29f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-flink/tree/8acac29f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-flink/diff/8acac29f
Branch: refs/heads/release-0.6.1
Commit: 8acac29ff156970ec155b0ae6ee82798b9e01746
Parents: d3a9799
Author: Fabian Hueske <fh...@gmail.com>
Authored: Tue Aug 26 23:48:52 2014 +0200
Committer: Robert Metzger <rm...@apache.org>
Committed: Wed Aug 27 16:09:56 2014 +0200
----------------------------------------------------------------------
docs/faq.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-flink/blob/8acac29f/docs/faq.md
----------------------------------------------------------------------
diff --git a/docs/faq.md b/docs/faq.md
index 694c156..60fbc5e 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -271,7 +271,7 @@ where it stores the Flink jar and configuration file.
## What kind of fault-tolerance does Flink provide?
-Stratospere can restart failed jobs. Mid-query fault tolerance will go into the
+Flink can restart failed jobs. Mid-query fault tolerance will go into the
open source project in the next versions.
## Are Hadoop-like utilities, such as Counters and the DistributedCache supported?