You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by "jackxu2011 (via GitHub)" <gi...@apache.org> on 2023/01/31 09:11:42 UTC
[GitHub] [linkis] jackxu2011 commented on a diff in pull request #4110: [Feat]support different hadoop version compile
jackxu2011 commented on code in PR #4110:
URL: https://github.com/apache/linkis/pull/4110#discussion_r1091635127
##########
pom.xml:
##########
@@ -1349,6 +1359,45 @@
</build>
<profiles>
+ <!-- hadoop version: mvn validate -Phadoop-3.3 ,spark still use hadoop2.7.2 by default, More details please check SPARK-23534 -->
+ <profile>
+ <id>hadoop-3.3</id>
+ <properties>
+ <hadoop.version>3.3.1</hadoop.version>
+ <curator.version>4.2.0</curator.version>
+ <hadoop-hdfs-client.artifact>hadoop-hdfs-client</hadoop-hdfs-client.artifact>
+ </properties>
+ </profile>
+ <!-- hadoop version: mvn validate -Phadoop-2.7 -->
+ <profile>
+ <id>hadoop-2.7</id>
+ <properties>
+ <hadoop.version>2.7.2</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ </properties>
+ </profile>
+ <!-- spark2-hadoop3 version:spark2.4 shade use hadoop2.7.2 by default mvn validate -Pspark-2.4-hadoop-3.3 -->
+ <profile>
+ <id>spark-2.4-hadoop-3.3</id>
Review Comment:
this profile should be moved to spark's pom.xml, and this profile should work with -Phadoop-3.3. and put the linkis-hadoop-hdfs-client-shade dependency to this profile.
##########
pom.xml:
##########
@@ -1349,6 +1359,45 @@
</build>
<profiles>
+ <!-- hadoop version: mvn validate -Phadoop-3.3 ,spark still use hadoop2.7.2 by default, More details please check SPARK-23534 -->
+ <profile>
+ <id>hadoop-3.3</id>
+ <properties>
+ <hadoop.version>3.3.1</hadoop.version>
+ <curator.version>4.2.0</curator.version>
+ <hadoop-hdfs-client.artifact>hadoop-hdfs-client</hadoop-hdfs-client.artifact>
+ </properties>
+ </profile>
+ <!-- hadoop version: mvn validate -Phadoop-2.7 -->
+ <profile>
+ <id>hadoop-2.7</id>
+ <properties>
+ <hadoop.version>2.7.2</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ </properties>
+ </profile>
+ <!-- spark2-hadoop3 version:spark2.4 shade use hadoop2.7.2 by default mvn validate -Pspark-2.4-hadoop-3.3 -->
+ <profile>
+ <id>spark-2.4-hadoop-3.3</id>
+ <properties>
+ <hadoop.version>3.3.1</hadoop.version>
+ <curator.version>4.2.0</curator.version>
+ <hadoop-hdfs-client.artifact>hadoop-hdfs-client</hadoop-hdfs-client.artifact>
Review Comment:
this properties are same with profile hadoop-3.3 can be ignore
##########
linkis-hadoop-hdfs-client-shade/dependency-reduced-pom.xml:
##########
@@ -0,0 +1,281 @@
+<?xml version="1.0" encoding="UTF-8"?>
Review Comment:
this file should be ignore
##########
linkis-engineconn-plugins/spark/pom.xml:
##########
@@ -202,13 +207,103 @@
<artifactId>linkis-rpc</artifactId>
<version>${project.version}</version>
</dependency>
-
+ <dependency>
+ <groupId>org.apache.linkis</groupId>
+ <artifactId>linkis-hadoop-hdfs-client-shade</artifactId>
+ <version>${project.version}</version>
Review Comment:
move the spark-2.4-hadoop-3.3 profile to this pom, and add the dependency to the profile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org