You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by zh...@apache.org on 2020/03/05 02:04:49 UTC
[submarine] branch master updated: SUBMARINE-399. Support Hadoop
2.10
This is an automated email from the ASF dual-hosted git repository.
zhouquan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 1536829 SUBMARINE-399. Support Hadoop 2.10
1536829 is described below
commit 1536829d7d92b9bc89e7222fdf097b752c13a81a
Author: Wei-Chiu Chuang <we...@cloudera.com>
AuthorDate: Tue Mar 3 15:29:35 2020 -0800
SUBMARINE-399. Support Hadoop 2.10
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://submarine.apache.org/contribution/contributions.html
### What type of PR is it?
Feature
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-399
### How should this be tested?
https://travis-ci.org/jojochuang/hadoop-submarine/builds/657551872
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes
Author: Wei-Chiu Chuang <we...@cloudera.com>
Closes #199 from jojochuang/SUBMARINE-399 and squashes the following commits:
c205e9f [Wei-Chiu Chuang] Updated QuickStart doc to mention that GPU on YARN is supported on Hadoop 2.10 and above.
90c7cbf [Wei-Chiu Chuang] Update BuildFromCode doc to include instruction for building against hadoop 2.10.
ce0dd70 [Wei-Chiu Chuang] Update travis.yml to not use hadoop-2.10 for tests.
f7b0ff3 [Wei-Chiu Chuang] Fix checkstyle.
4fd386b [Wei-Chiu Chuang] Workaround YARN-10177 by defining gpu resource in resource-types.xml.
ed0e74e [Wei-Chiu Chuang] Add the hadoop-2.10 profile
---
.travis.yml | 6 +++++
docs/development/BuildFromCode.md | 6 +++++
docs/helper/QuickStart.md | 2 +-
pom.xml | 16 +++++++++++++
submarine-all/pom.xml | 20 ++++++++++++++++
.../src/test/resources/resource-types.xml | 28 ++++++++++++++++++++++
.../commons/runtime/resource/ResourceUtils.java | 3 ++-
submarine-dist/pom.xml | 9 +++++++
8 files changed, 88 insertions(+), 2 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index d80e420..f6f7b4b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -101,6 +101,12 @@ matrix:
dist: xenial
env: PROFILE="-Phadoop-2.9" BUILD_FLAG="clean package install -DskipTests -DskipRat" TEST_FLAG="test -DskipRat" MODULES="-pl ${EXCLUDE_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},!submarine-dist" TEST_MODULES="-pl ${EXCLUDE_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_COMMON_RPC},!submarine-dist" TEST_PROJECTS=""
+ - name: Build submarine submitter on hadoop-2.10
+ language: java
+ jdk: openjdk8
+ dist: xenial
+ env: PROFILE="-Phadoop-2.10" BUILD_FLAG="clean package install -DskipTests -DskipRat" TEST_FLAG="test -DskipRat" MODULES="-pl ${EXCLUDE_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},!submarine-dist" TEST_MODULES="-pl ${EXCLUDE_K8S},${EXCLUDE_WORKBENCH},${EXCLUDE_INTERPRETER},${EXCLUDE_CLOUD},${EXCLUDE_COMMON_RPC},!submarine-dist" TEST_PROJECTS=""
+
- name: Build submarine submitter on hadoop-3.1
language: java
jdk: openjdk8
diff --git a/docs/development/BuildFromCode.md b/docs/development/BuildFromCode.md
index a9699c7..783c5f5 100644
--- a/docs/development/BuildFromCode.md
+++ b/docs/development/BuildFromCode.md
@@ -43,6 +43,12 @@ mvn clean install package -DskipTests
mvn clean install package -DskipTests -Phadoop-2.9
```
++ Create binary distribution with hadoop-2.10.x version
+
+```
+mvn clean install package -DskipTests -Phadoop-2.10
+```
+
+ Create binary distribution with hadoop-3.1.x version
```
diff --git a/docs/helper/QuickStart.md b/docs/helper/QuickStart.md
index d26eb4e..2e881db 100644
--- a/docs/helper/QuickStart.md
+++ b/docs/helper/QuickStart.md
@@ -24,7 +24,7 @@ Must:
Optional:
- [Enable YARN Service](https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html) (Only yarn 3.1.x + needs)
-- [Enable GPU on YARN 3.1.0+](https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/UsingGpus.html)
+- [Enable GPU on YARN 2.10.0+](https://hadoop.apache.org/docs/r2.10.0/hadoop-yarn/hadoop-yarn-site/UsingGpus.html)
- [Enable Docker on YARN 2.8.2+](https://hadoop.apache.org/docs/r2.8.2/hadoop-yarn/hadoop-yarn-site/DockerContainers.html)
- [Build Docker images](WriteDockerfileTF.md)
diff --git a/pom.xml b/pom.xml
index e1c4ebf..6770a99 100644
--- a/pom.xml
+++ b/pom.xml
@@ -353,6 +353,22 @@
</profile>
<profile>
+ <id>hadoop-2.10</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>2.10.0</hadoop.version>
+ <jaxb-api.version>2.2.2</jaxb-api.version>
+ <commons-compress.version>1.19</commons-compress.version>
+ <guice-servlet.version>3.0</guice-servlet.version>
+ <guice.version>3.0</guice.version>
+ <zookeeper.version>3.4.9</zookeeper.version>
+ <profile-id>hadoop-2.10</profile-id>
+ </properties>
+ </profile>
+
+ <profile>
<id>hadoop-2.9</id>
<activation>
<activeByDefault>true</activeByDefault>
diff --git a/submarine-all/pom.xml b/submarine-all/pom.xml
index f774c6a..3500950 100644
--- a/submarine-all/pom.xml
+++ b/submarine-all/pom.xml
@@ -152,6 +152,26 @@
</profile>
<profile>
+ <id>hadoop-2.10</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs-client</artifactId>
+ <version>${hadoop.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ <profile>
<id>hadoop-2.9</id>
<activation>
<activeByDefault>true</activeByDefault>
diff --git a/submarine-client/src/test/resources/resource-types.xml b/submarine-client/src/test/resources/resource-types.xml
new file mode 100644
index 0000000..331f7c6
--- /dev/null
+++ b/submarine-client/src/test/resources/resource-types.xml
@@ -0,0 +1,28 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+ <property>
+ <name>yarn.resource-types</name>
+ <value>yarn.io/gpu</value>
+ <description></description>
+ </property>
+
+</configuration>
diff --git a/submarine-commons/commons-runtime/src/main/java/org/apache/submarine/commons/runtime/resource/ResourceUtils.java b/submarine-commons/commons-runtime/src/main/java/org/apache/submarine/commons/runtime/resource/ResourceUtils.java
index dd402a0..3abb261 100644
--- a/submarine-commons/commons-runtime/src/main/java/org/apache/submarine/commons/runtime/resource/ResourceUtils.java
+++ b/submarine-commons/commons-runtime/src/main/java/org/apache/submarine/commons/runtime/resource/ResourceUtils.java
@@ -318,7 +318,8 @@ public final class ResourceUtils {
LOG.info(e.getMessage());
throw new SubmarineRuntimeException(e.getMessage(), e.getCause());
} catch (NoSuchMethodException nsme) {
- LOG.info("There is no '" + GET_RESOURCE_VALUE_METHOD + "' API in this" +
+ LOG.info("There is no '" + GET_RESOURCE_VALUE_METHOD + "' or '" +
+ REINITIALIZE_RESOURCES_METHOD + "' API in this" +
" version of YARN");
} catch (IllegalAccessException | InvocationTargetException e) {
LOG.info("Failed to invoke 'configureResourceType' method ", e);
diff --git a/submarine-dist/pom.xml b/submarine-dist/pom.xml
index 92f88e5..e10883c 100644
--- a/submarine-dist/pom.xml
+++ b/submarine-dist/pom.xml
@@ -97,6 +97,15 @@
</profile>
<profile>
+ <id>hadoop-2.10</id>
+ <activation>
+ <activeByDefault>false</activeByDefault>
+ </activation>
+ <dependencies>
+ </dependencies>
+ </profile>
+
+ <profile>
<id>hadoop-2.9</id>
<activation>
<activeByDefault>true</activeByDefault>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org