You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by tz...@apache.org on 2017/03/22 14:57:54 UTC
[2/2] flink git commit: [FLINK-6139] [doc] Add documentation for
running Flink on MapR
[FLINK-6139] [doc] Add documentation for running Flink on MapR
This closes #3582.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/09239ea1
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/09239ea1
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/09239ea1
Branch: refs/heads/release-1.2
Commit: 09239ea1641e06802a302d00cc80e0fc643971e5
Parents: b4d1eb4
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
Authored: Wed Mar 22 15:15:41 2017 +0800
Committer: Tzu-Li (Gordon) Tai <tz...@apache.org>
Committed: Wed Mar 22 22:57:00 2017 +0800
----------------------------------------------------------------------
docs/setup/aws.md | 2 +-
docs/setup/cluster_setup.md | 2 +-
docs/setup/gce_setup.md | 2 +-
docs/setup/mapr_setup.md | 132 +++++++++++++++++++++++++++++++++++++++
docs/setup/yarn_setup.md | 2 +-
5 files changed, 136 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/09239ea1/docs/setup/aws.md
----------------------------------------------------------------------
diff --git a/docs/setup/aws.md b/docs/setup/aws.md
index 8d04d59..b0fbefc 100644
--- a/docs/setup/aws.md
+++ b/docs/setup/aws.md
@@ -2,7 +2,7 @@
title: "Amazon Web Services (AWS)"
nav-title: AWS
nav-parent_id: deployment
-nav-pos: 10
+nav-pos: 5
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/09239ea1/docs/setup/cluster_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/cluster_setup.md b/docs/setup/cluster_setup.md
index c86e353..bcb7766 100644
--- a/docs/setup/cluster_setup.md
+++ b/docs/setup/cluster_setup.md
@@ -1,7 +1,7 @@
---
title: "Standalone Cluster"
nav-parent_id: deployment
-nav-pos: 2
+nav-pos: 1
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/09239ea1/docs/setup/gce_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/gce_setup.md b/docs/setup/gce_setup.md
index f9edfcc..ac1ad25 100644
--- a/docs/setup/gce_setup.md
+++ b/docs/setup/gce_setup.md
@@ -2,7 +2,7 @@
title: "Google Compute Engine Setup"
nav-title: Google Compute Engine
nav-parent_id: deployment
-nav-pos: 20
+nav-pos: 6
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
http://git-wip-us.apache.org/repos/asf/flink/blob/09239ea1/docs/setup/mapr_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/mapr_setup.md b/docs/setup/mapr_setup.md
new file mode 100644
index 0000000..44682e5
--- /dev/null
+++ b/docs/setup/mapr_setup.md
@@ -0,0 +1,132 @@
+---
+title: "MapR Setup"
+nav-title: MapR
+nav-parent_id: deployment
+nav-pos: 7
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This documentation provides instructions on how to prepare Flink for YARN
+executions on a [MapR](https://mapr.com/) cluster.
+
+* This will be replaced by the TOC
+{:toc}
+
+## Running Flink on YARN with MapR
+
+The instructions below assume MapR version 5.2.0. They will guide you
+to be able to start submitting [Flink on YARN]({{ site.baseurl }}/setup/yarn_setup.html)
+jobs or sessions to a MapR cluster.
+
+### Building Flink for MapR
+
+In order to run Flink on MapR, Flink needs to be built with MapR's own
+Hadoop and Zookeeper distribution. Simply build Flink using Maven with
+the following command from the project root directory:
+
+```
+mvn clean install -DskipTests -Pvendor-repos,mapr \
+ -Dhadoop.version=2.7.0-mapr-1607 \
+ -Dzookeeper.version=3.4.5-mapr-1604
+```
+
+The `vendor-repos` build profile adds MapR's repository to the build so that
+MapR's Hadoop / Zookeeper dependencies can be fetched. The `mapr` build
+profile additionally resolves some dependency clashes between MapR and
+Flink, as well as ensuring that the native MapR libraries on the cluster
+nodes are used. Both profiles must be activated.
+
+By default the `mapr` profile builds with Hadoop / Zookeeper dependencies
+for MapR version 5.2.0, so you don't need to explicitly override
+the `hadoop.version` and `zookeeper.version` properties.
+For different MapR versions, simply override these properties to appropriate
+values. The corresponding Hadoop / Zookeeper distributions for each MapR version
+can be found on MapR documentations such as
+[here](http://maprdocs.mapr.com/home/DevelopmentGuide/MavenArtifacts.html).
+
+### Job Submission Client Setup
+
+The client submitting Flink jobs to MapR also needs to be prepared with the below setups.
+
+Ensure that MapR's JAAS config file is picked up to avoid login failures:
+
+```
+export JVM_ARGS=-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf
+```
+
+Make sure that the `yarn.nodemanager.resource.cpu-vcores` property is set in `yarn-site.xml`:
+
+~~~xml
+<!-- in /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/yarn-site.xml -->
+
+<configuration>
+...
+
+<property>
+ <name>yarn.nodemanager.resource.cpu-vcores</name>
+ <value>...</value>
+</property>
+
+...
+</configuration>
+~~~
+
+Also remember to set the `YARN_CONF_DIR` or `HADOOP_CONF_DIR` environment
+variables to the path where `yarn-site.xml` is located:
+
+```
+export YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
+export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
+```
+
+Make sure that the MapR native libraries are picked up in the classpath:
+
+```
+export FLINK_CLASSPATH=/opt/mapr/lib/*
+```
+
+If you'll be starting Flink on YARN sessions with `yarn-session.sh`, the
+below is also required:
+
+```
+export CC_CLASSPATH=/opt/mapr/lib/*
+```
+
+## Running Flink with a Secured MapR Cluster
+
+*Note: In Flink 1.2.0, Flink's Kerberos authentication for YARN execution has
+a bug that forbids it to work with MapR Security. Please upgrade to later Flink
+versions in order to use Flink with a secured MapR cluster. For more details,
+please see [FLINK-5949](https://issues.apache.org/jira/browse/FLINK-5949).*
+
+Flink's [Kerberos authentication]({{ site.baseurl }}/ops/security-kerberos.html) is independent of
+[MapR's Security authentication](http://maprdocs.mapr.com/home/SecurityGuide/Configuring-MapR-Security.html).
+With the above build procedures and environment variable setups, Flink
+does not require any additional configuration to work with MapR Security.
+
+Users simply need to login by using MapR's `maprlogin` authentication
+utility. Users that haven't acquired MapR login credentials would not be
+able to submit Flink jobs, erroring with:
+
+```
+java.lang.Exception: unable to establish the security context
+Caused by: o.a.f.r.security.modules.SecurityModule$SecurityInstallException: Unable to set the Hadoop login user
+Caused by: java.io.IOException: failure to login: Unable to obtain MapR credentials
+```
http://git-wip-us.apache.org/repos/asf/flink/blob/09239ea1/docs/setup/yarn_setup.md
----------------------------------------------------------------------
diff --git a/docs/setup/yarn_setup.md b/docs/setup/yarn_setup.md
index 53423b8..3149ec2 100644
--- a/docs/setup/yarn_setup.md
+++ b/docs/setup/yarn_setup.md
@@ -2,7 +2,7 @@
title: "YARN Setup"
nav-title: YARN
nav-parent_id: deployment
-nav-pos: 3
+nav-pos: 2
---
<!--
Licensed to the Apache Software Foundation (ASF) under one