You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/01/16 06:55:00 UTC
[jira] [Commented] (BEAM-3240) Quickstart examples archetype
dependencies conflict for Apex on YARN
[ https://issues.apache.org/jira/browse/BEAM-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326812#comment-16326812 ]
ASF GitHub Bot commented on BEAM-3240:
--------------------------------------
tweise closed pull request #4402: [BEAM-3240] Fix quickstart archetypes to include expected Hadoop dependencies under Apex profile
URL: https://github.com/apache/beam/pull/4402
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
index 4a35cb14497..5051ba6a813 100644
--- a/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
+++ b/sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
@@ -44,6 +44,7 @@
<pubsub.version>@pubsub.version@</pubsub.version>
<slf4j.version>@slf4j.version@</slf4j.version>
<spark.version>@spark.version@</spark.version>
+ <hadoop.version>@hadoop.version@</hadoop.version>
<surefire-plugin.version>@surefire-plugin.version@</surefire-plugin.version>
</properties>
@@ -177,14 +178,14 @@
<scope>runtime</scope>
</dependency>
<!--
- Apex depends on httpclient version 4.3.5, project has a transitive dependency to httpclient 4.0.1 from
+ Apex depends on httpclient version 4.3.6, project has a transitive dependency to httpclient 4.0.1 from
google-http-client. Apex dependency version being specified explicitly so that it gets picked up. This
can be removed when the project no longer has a dependency on a different httpclient version.
-->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
- <version>4.3.5</version>
+ <version>4.3.6</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
@@ -193,6 +194,22 @@
</exclusion>
</exclusions>
</dependency>
+ <!--
+ Apex 3.6 is built against YARN 2.6. Version in the fat jar has to match
+ what's on the cluster, hence we need to repeat the Apex Hadoop dependencies here.
+ -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>runtime</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>runtime</scope>
+ </dependency>
</dependencies>
</profile>
diff --git a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 5aa80ff2591..f59a284e459 100644
--- a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -43,6 +43,7 @@
<pubsub.version>@pubsub.version@</pubsub.version>
<slf4j.version>@slf4j.version@</slf4j.version>
<spark.version>@spark.version@</spark.version>
+ <hadoop.version>@hadoop.version@</hadoop.version>
<surefire-plugin.version>@surefire-plugin.version@</surefire-plugin.version>
</properties>
@@ -176,14 +177,14 @@
<scope>runtime</scope>
</dependency>
<!--
- Apex depends on httpclient version 4.3.5, project has a transitive dependency to httpclient 4.0.1 from
+ Apex depends on httpclient version 4.3.6, project has a transitive dependency to httpclient 4.0.1 from
google-http-client. Apex dependency version being specified explicitly so that it gets picked up. This
can be removed when the project no longer has a dependency on a different httpclient version.
-->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
- <version>4.3.5</version>
+ <version>4.3.6</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
@@ -192,6 +193,22 @@
</exclusion>
</exclusions>
</dependency>
+ <!--
+ Apex 3.6 is built against YARN 2.6. Version in the fat jar has to match
+ what's on the cluster, hence we need to repeat the Apex Hadoop dependencies here.
+ -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>runtime</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>runtime</scope>
+ </dependency>
</dependencies>
</profile>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Quickstart examples archetype dependencies conflict for Apex on YARN
> ---------------------------------------------------------------------
>
> Key: BEAM-3240
> URL: https://issues.apache.org/jira/browse/BEAM-3240
> Project: Beam
> Issue Type: Bug
> Components: runner-apex
> Affects Versions: 2.0.0
> Reporter: Thomas Weise
> Priority: Major
>
> The archetype builds a fat jar that includes the Apex runners dependencies, which transitively include YARN 2.6 (as that's what Apex 3.6 was built against). There will be a conflict if the Hadoop version on the cluster is more recent (like 2.7.3). Project needs to include the YARN version that matches the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)