You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by GitBox <gi...@apache.org> on 2021/11/15 14:17:11 UTC
[GitHub] [zeppelin] zjffdu opened a new pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
zjffdu opened a new pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266
### What is this PR for?
Restructure the CI in `core.yaml`
* Split interpreters CI into 5 groups:
* jupyter-interpreter, rlang, python
* non-core interpreters (sh, markdown, etc)
* spark
* flink
* livy
* Run different profiles' spark test in one bulid. Only run it under different python versions.
* Refine the hadoop2 and hadoop3 profiles, use different hadoop profiles in integration test (`SparkIntegrationTest`, `ZeppelinSparkClusterTest`)
### What type of PR is it?
[Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5576
### How should this be tested?
* CI pass except the frontend e2e test (not related)
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? NO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r755888037
##########
File path: .github/workflows/frontend.yml
##########
@@ -47,6 +47,10 @@ jobs:
run: ./mvnw -B install -DskipTests -DskipRat -pl ${INTERPRETERS} -Phadoop2 -Pscala-2.11
- name: Run headless test
run: xvfb-run --auto-servernum --server-args="-screen 0 1024x768x24" ./mvnw verify -DskipRat -pl zeppelin-web -Phadoop2 -Pscala-2.11 -Pweb-e2e -B
+ - name: Print zeppelin logs
+ if: always()
Review comment:
Just for diagnose in case test fail, we can check the zeppelin server logs via this action
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r756643346
##########
File path: rlang/pom.xml
##########
@@ -113,6 +113,13 @@
<scope>compile</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>2.7.7</version>
Review comment:
Fixed
##########
File path: spark/interpreter/pom.xml
##########
@@ -198,10 +205,23 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.6.0</version>
+ <version>2.7.7</version>
Review comment:
Fixed
##########
File path: spark/interpreter/pom.xml
##########
@@ -198,10 +205,23 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.6.0</version>
+ <version>2.7.7</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>2.7.7</version>
Review comment:
Fixed
##########
File path: spark/spark-scala-parent/pom.xml
##########
@@ -80,6 +80,21 @@
<scope>provided</scope>
</dependency>
+ <!-- Use provided scope, hadoop dependencies are only for compilation -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>2.7.7</version>
Review comment:
Fixed
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +173,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
Fixed
##########
File path: zeppelin-interpreter-integration/src/test/java/org/apache/zeppelin/integration/ZeppelinSparkClusterTest.java
##########
@@ -154,6 +162,11 @@ private void waitForRunning(Paragraph p) {
@Test
public void scalaOutputTest() throws IOException, InterruptedException {
+ if (!isHadoopVersionMatch()) {
Review comment:
Fixed
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
Fixed
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
Fixed
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r771089274
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
Review comment:
The reason is that allow the user to control whether package hadoop in zeppelin distribution via the property `hadoop.deps.scope` defined in root pom.xml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r808841411
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
<profile>
<id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
<dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#issuecomment-1044465990
@Reamer Sorry for late response, I think all the comments are addressed now, could you help do another round of review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r756644177
##########
File path: pom.xml
##########
@@ -398,12 +401,27 @@
<version>${jettison.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
Reamer commented on pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#issuecomment-1022957585
> @Reamer Any more comments?
Yes let me review this agian.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r808838468
##########
File path: zeppelin-server/pom.xml
##########
@@ -293,30 +293,9 @@
</dependency>
<dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- <exclusions>
- <!-- Provided by javax-websocket-server-impl -->
- <exclusion>
- <groupId>org.ow2.asm</groupId>
- <artifactId>asm</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <!--test libraries-->
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <classifier>tests</classifier>
- <scope>test</scope>
- <exclusions>
- <!-- Provided by javax-websocket-server-impl -->
- <exclusion>
- <groupId>org.ow2.asm</groupId>
- <artifactId>asm</artifactId>
- </exclusion>
- </exclusions>
+ <groupId>org.apache.directory.server</groupId>
+ <artifactId>apacheds-kerberos-codec</artifactId>
+ <version>2.0.0-M15</version>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#issuecomment-977627868
@Reamer Do you have time to review it ? It make a lot changes on core CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r757000231
##########
File path: .github/workflows/core.yml
##########
@@ -441,6 +358,11 @@ jobs:
key: ${{ runner.os }}-zeppelin-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-zeppelin-
+ - name: install environment
+ run: |
+ ./mvnw install -DskipTests -DskipRat -pl livy -am -B
+ ./testing/downloadSpark.sh "2.2.0" "2.6"
Review comment:
We are using an old version of livy (0.5), it may not support newer versions of spark
##########
File path: spark/interpreter/pom.xml
##########
@@ -113,6 +113,13 @@
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
Review comment:
Fixed
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -24,7 +24,7 @@
<artifactId>zeppelin</artifactId>
<groupId>org.apache.zeppelin</groupId>
<version>0.11.0-SNAPSHOT</version>
- <relativePath>..</relativePath>
+ <relativePath>../pom.xml</relativePath>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r808842804
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r770195032
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
<profile>
<id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
Review comment:
fixed
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
<profile>
<id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r759142260
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
Review comment:
I do not see any use of `commons-io.version`, this property should be removed.
##########
File path: zeppelin-plugins/launcher/yarn/pom.xml
##########
@@ -49,6 +49,11 @@
<artifactId>hadoop-yarn-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
Review comment:
Based on the Hadoop profile in zeppelin-zengine, the Hadoop lib may or may not be present.
We should also put Hadoop profiles in this `pom.xml` and explicitly set this dependency to `provided`.
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-minicluster.artifact}</artifactId>
Review comment:
scope `test` should be added
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
Review comment:
I do not see any use of `curator.version`, this property should be removed.
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
Review comment:
I do not see any use of `curator.version`, this property should be removed.
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
Review comment:
I think we can remove this dependency because `hadoop-client-api` is a transitive dependency of `hadoop-client-runtime`.
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
Review comment:
I do not see any use of `kerberos-client.version`, this property should be removed.
##########
File path: zeppelin-plugins/notebookrepo/filesystem/pom.xml
##########
@@ -43,6 +43,10 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
</dependency>
+ <dependency>
Review comment:
Based on the Hadoop profile in zeppelin-zengine, the Hadoop lib may or may not be present.
We should also put Hadoop profiles in this `pom.xml` and explicitly set this dependency to `provided`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r757291318
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
<profile>
<id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
Review comment:
I don't see any use for this property. Where have we used it?
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
<profile>
<id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
<dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
Review comment:
I think we can remove this dependency because `hadoop-client-api` is a transitive dependency of `hadoop-client-runtime`.
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
Review comment:
The same as in `zeppelin-server`. I don't see any use for this property.
##########
File path: zeppelin-server/pom.xml
##########
@@ -293,30 +293,9 @@
</dependency>
<dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- <exclusions>
- <!-- Provided by javax-websocket-server-impl -->
- <exclusion>
- <groupId>org.ow2.asm</groupId>
- <artifactId>asm</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <!--test libraries-->
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <classifier>tests</classifier>
- <scope>test</scope>
- <exclusions>
- <!-- Provided by javax-websocket-server-impl -->
- <exclusion>
- <groupId>org.ow2.asm</groupId>
- <artifactId>asm</artifactId>
- </exclusion>
- </exclusions>
+ <groupId>org.apache.directory.server</groupId>
+ <artifactId>apacheds-kerberos-codec</artifactId>
+ <version>2.0.0-M15</version>
Review comment:
This version should be moved to a new property.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
Review comment:
I do not see any use of `curator.version`, this property should be removed.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-minicluster.artifact}</artifactId>
Review comment:
scope `test` and version `hadoop.version` should be added
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
Review comment:
`hadoop-yarn-common` is a transitive dependency of `hadoop-yarn-client`, it does not need to be defined twice.
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
Review comment:
I do not see any use of `commons-io.version`, this property should be removed.
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
Review comment:
Where do we use YARN components in zeppelin-server? In my opinion, these dependencies can be removed.
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
Review comment:
The same as in `zeppelin-server`. I don't see any use for this property.
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
Review comment:
I do not see any use of `curator.version`, this property should be removed.
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
<profile>
<id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
Review comment:
This version is hard coded in line 298. We should remove this at this point.
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
Review comment:
`hadoop-yarn-client` is declared twice.
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
Review comment:
`hadoop-client-api` is a transitive dependency of `hadoop-client-runtime`, it does not need to be defined twice.
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
Review comment:
`hadoop-yarn-common` is a transitive dependency of `hadoop-yarn-client`, it does not need to be defined twice.
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
Review comment:
`hadoop-yarn-api` is a transitive dependency of `hadoop-yarn-client`, it does not need to be defined twice.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
Review comment:
I do not see any use of `commons-io.version`, this property should be removed.
##########
File path: pom.xml
##########
@@ -398,12 +401,27 @@
<version>${jettison.version}</version>
</dependency>
+<!-- <dependency>-->
+<!-- <groupId>org.apache.hadoop</groupId>-->
+<!-- <artifactId>${hadoop-client-api.artifact}</artifactId>-->
Review comment:
Remove this block.
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
Review comment:
I see no use for this property.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
Review comment:
I do not see any use of `curator.version`, this property should be removed.
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-minicluster.artifact}</artifactId>
Review comment:
scope `test` and version `hadoop.version` should be added
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
Review comment:
Duplicate declaration.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
Review comment:
`hadoop-yarn-api` is a transitive dependency of `hadoop-yarn-client`, it does not need to be defined twice.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
Review comment:
I do not see any use of `kerberos-client.version`, this property should be removed.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
Review comment:
I think we can remove this dependency because `hadoop-client-api` is a transitive dependency of `hadoop-client-runtime`.
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
Review comment:
I see no use for this property.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
Review comment:
Hadoop libraries are only needed for the YarnUtils class. If I understand YARN correctly, the Hadoop libraries are provided by the system. Therefore, I think we can set the dependency scope to `provided` and a small comment.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
Review comment:
Hadoop libraries are only needed for the `YarnUtils` class. If I understand YARN correctly, the Hadoop libraries are provided by the system. Therefore, I think we can set the dependency scope to `provided` and a small comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#issuecomment-1022810456
@Reamer Any more comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r808846728
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-minicluster.artifact}</artifactId>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r769667126
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
Review comment:
fixed
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r808824577
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r808842127
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r769560889
##########
File path: pom.xml
##########
@@ -997,6 +1215,195 @@
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-json</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-client</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.jackrabbit</groupId>
+ <artifactId>jackrabbit-webdav</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-httpclient</groupId>
+ <artifactId>commons-httpclient</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jgit</groupId>
+ <artifactId>org.eclipse.jgit</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.jcraft</groupId>
+ <artifactId>jsch</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-compress</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xml-apis</groupId>
+ <artifactId>xml-apis</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xerces</groupId>
+ <artifactId>xercesImpl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ </exclusion>
+ <!-- using jcl-over-slf4j instead -->
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <version>${hadoop.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-json</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-client</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.jackrabbit</groupId>
+ <artifactId>jackrabbit-webdav</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-httpclient</groupId>
+ <artifactId>commons-httpclient</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jgit</groupId>
+ <artifactId>org.eclipse.jgit</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.jcraft</groupId>
+ <artifactId>jsch</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-compress</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xml-apis</groupId>
+ <artifactId>xml-apis</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xerces</groupId>
+ <artifactId>xercesImpl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ </exclusion>
+ <!-- using jcl-over-slf4j instead -->
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
Review comment:
They are different, One is `hadoop-yarn-client` another is `hadoop-yarn-api`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r808844992
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] asfgit closed pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r769666474
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
Review comment:
fixed
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
Review comment:
fixed
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
Review comment:
fixed
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +172,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-server-tests</artifactId>
+ <classifier>tests</classifier>
+ </dependency>
+
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r798274491
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
<profile>
<id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
<dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
Review comment:
Your right, but we are in the profile `hadoop3`, therefore is doesn't effect `hadoop2`
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-minicluster.artifact}</artifactId>
Review comment:
still valid
##########
File path: zeppelin-plugins/launcher/yarn/pom.xml
##########
@@ -49,6 +49,11 @@
<artifactId>hadoop-yarn-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
Review comment:
still valid
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
Review comment:
still valid
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
Review comment:
still valid
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
Review comment:
still valid
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
Review comment:
still valid
##########
File path: zeppelin-server/pom.xml
##########
@@ -293,30 +293,9 @@
</dependency>
<dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- <exclusions>
- <!-- Provided by javax-websocket-server-impl -->
- <exclusion>
- <groupId>org.ow2.asm</groupId>
- <artifactId>asm</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <!--test libraries-->
- <dependency>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- <classifier>tests</classifier>
- <scope>test</scope>
- <exclusions>
- <!-- Provided by javax-websocket-server-impl -->
- <exclusion>
- <groupId>org.ow2.asm</groupId>
- <artifactId>asm</artifactId>
- </exclusion>
- </exclusions>
+ <groupId>org.apache.directory.server</groupId>
+ <artifactId>apacheds-kerberos-codec</artifactId>
+ <version>2.0.0-M15</version>
Review comment:
Still valid.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
Review comment:
I understand that. But doesn't YARN provide the Hadoop libraries? The only use of Hadoop libraries are in the `YarnUtils` class. So I think that Zeppelin interpreter should not provide Hadoop libraries.
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
Review comment:
still valid
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-minicluster.artifact}</artifactId>
Review comment:
Still valid.
##########
File path: zeppelin-plugins/notebookrepo/filesystem/pom.xml
##########
@@ -43,6 +43,10 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
</dependency>
+ <dependency>
Review comment:
still valid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r808836041
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
+ <profile>
+ <id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
+ <dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
Review comment:
As I mentioned in the above comment, user can choose whether packaging hadoop into Zeppelin distribution. This is just like what spark did (spark has 2 distribution, one with hadoop libraries, another without hadoop libraries)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
Reamer commented on pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#issuecomment-1044802444
I will have a look at it next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r755887628
##########
File path: .github/workflows/core.yml
##########
@@ -94,97 +97,25 @@ jobs:
key: ${{ runner.os }}-zeppelin-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-zeppelin-
+ - name: install environment
+ run: ./mvnw install -DskipTests -DskipRat -am -pl .,zeppelin-interpreter,zeppelin-interpreter-shaded,${INTERPRETERS} -Pscala-2.10 -B
Review comment:
You are right, but many interpreter modules depends on scala-2.10, and seems there're no maintainers now. We can raise a discussion in community about dropping these un-maintanced interpreter modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] martin-g commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
martin-g commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r755790712
##########
File path: .github/workflows/core.yml
##########
@@ -94,97 +97,25 @@ jobs:
key: ${{ runner.os }}-zeppelin-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-zeppelin-
+ - name: install environment
+ run: ./mvnw install -DskipTests -DskipRat -am -pl .,zeppelin-interpreter,zeppelin-interpreter-shaded,${INTERPRETERS} -Pscala-2.10 -B
Review comment:
Is there a reason to still support/test Scala 2.10 ?
Most of the projects dropped support even for 2.11
##########
File path: pom.xml
##########
@@ -532,6 +550,129 @@
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>asm</groupId>
+ <artifactId>asm</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.ow2.asm</groupId>
+ <artifactId>asm</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.jboss.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey.jersey-test-framework</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey.contribs</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-compress</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
Review comment:
Should this use `${hadoop-client-runtime.artifact}` instead ?
##########
File path: pom.xml
##########
@@ -398,12 +401,27 @@
<version>${jettison.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
Review comment:
Is this dependency still needed ? It will be a duplicate of the newly added above (line 404-409)
##########
File path: spark/spark-scala-parent/pom.xml
##########
@@ -80,6 +80,21 @@
<scope>provided</scope>
</dependency>
+ <!-- Use provided scope, hadoop dependencies are only for compilation -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>2.7.7</version>
Review comment:
`${hadoop.version}` ?
##########
File path: zeppelin-interpreter-integration/src/test/java/org/apache/zeppelin/integration/ZeppelinSparkClusterTest.java
##########
@@ -154,6 +162,11 @@ private void waitForRunning(Paragraph p) {
@Test
public void scalaOutputTest() throws IOException, InterruptedException {
+ if (!isHadoopVersionMatch()) {
Review comment:
IMO this would be better by using JUnit Assumption APIs, e.g. `assumeThat(isHadoopVersionMatch(), "Hadoop version mismatch, skipping the test")`. No need of the `return;`
##########
File path: zeppelin-interpreter/pom.xml
##########
@@ -233,4 +226,86 @@
</resources>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
2.11 ?!
##########
File path: zeppelin-zengine/pom.xml
##########
@@ -317,4 +312,86 @@
</plugin>
</plugins>
</build>
+
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
2.11 ?!
##########
File path: pom.xml
##########
@@ -997,6 +1215,195 @@
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-json</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-client</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.jackrabbit</groupId>
+ <artifactId>jackrabbit-webdav</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-httpclient</groupId>
+ <artifactId>commons-httpclient</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jgit</groupId>
+ <artifactId>org.eclipse.jgit</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.jcraft</groupId>
+ <artifactId>jsch</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-compress</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xml-apis</groupId>
+ <artifactId>xml-apis</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xerces</groupId>
+ <artifactId>xercesImpl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ </exclusion>
+ <!-- using jcl-over-slf4j instead -->
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
+ <version>${hadoop.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-json</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-client</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.servlet</groupId>
+ <artifactId>servlet-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.jackrabbit</groupId>
+ <artifactId>jackrabbit-webdav</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-httpclient</groupId>
+ <artifactId>commons-httpclient</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jgit</groupId>
+ <artifactId>org.eclipse.jgit</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.jcraft</groupId>
+ <artifactId>jsch</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-compress</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xml-apis</groupId>
+ <artifactId>xml-apis</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>xerces</groupId>
+ <artifactId>xercesImpl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-all</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.jetty</groupId>
+ <artifactId>jetty-util</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ </exclusion>
+ <!-- using jcl-over-slf4j instead -->
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>${hadoop-client-runtime.artifact}</artifactId>
Review comment:
Maven could complain that this is a duplicate the one above where I asked about `hadoop-client-runtime.artifact`
##########
File path: spark/interpreter/pom.xml
##########
@@ -198,10 +205,23 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.6.0</version>
+ <version>2.7.7</version>
Review comment:
`${hadoop.version}` ?
##########
File path: rlang/pom.xml
##########
@@ -113,6 +113,13 @@
<scope>compile</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>2.7.7</version>
Review comment:
Should this use `${hadoop.version}` ?
##########
File path: spark/interpreter/pom.xml
##########
@@ -113,6 +113,13 @@
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
+ <version>2.7.7</version>
Review comment:
`${hadoop.version}` ?
##########
File path: .github/workflows/frontend.yml
##########
@@ -47,6 +47,10 @@ jobs:
run: ./mvnw -B install -DskipTests -DskipRat -pl ${INTERPRETERS} -Phadoop2 -Pscala-2.11
- name: Run headless test
run: xvfb-run --auto-servernum --server-args="-screen 0 1024x768x24" ./mvnw verify -DskipRat -pl zeppelin-web -Phadoop2 -Pscala-2.11 -Pweb-e2e -B
+ - name: Print zeppelin logs
+ if: always()
Review comment:
Why is this needed ?
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
2.11 ?!
##########
File path: spark/interpreter/pom.xml
##########
@@ -198,10 +205,23 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
- <version>2.6.0</version>
+ <version>2.7.7</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>2.7.7</version>
Review comment:
`${hadoop.version}` ?
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -203,4 +173,105 @@
</plugins>
</build>
+ <profiles>
+ <profile>
+ <id>hadoop2</id>
+
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.4</commons-io.version>
Review comment:
https://search.maven.org/artifact/commons-io/commons-io/2.11.0/jar is the latest one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
Reamer commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r755919147
##########
File path: zeppelin-interpreter-integration/pom.xml
##########
@@ -24,7 +24,7 @@
<artifactId>zeppelin</artifactId>
<groupId>org.apache.zeppelin</groupId>
<version>0.11.0-SNAPSHOT</version>
- <relativePath>..</relativePath>
+ <relativePath>../pom.xml</relativePath>
Review comment:
This line can be removed as this should be the default setting.
##########
File path: .github/workflows/core.yml
##########
@@ -441,6 +358,11 @@ jobs:
key: ${{ runner.os }}-zeppelin-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-zeppelin-
+ - name: install environment
+ run: |
+ ./mvnw install -DskipTests -DskipRat -pl livy -am -B
+ ./testing/downloadSpark.sh "2.2.0" "2.6"
Review comment:
Maybe it's time to update the Spark version to 2.4.
##########
File path: spark/interpreter/pom.xml
##########
@@ -113,6 +113,13 @@
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client</artifactId>
Review comment:
Duplicate dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #4266: [ZEPPELIN-5576] Improve Zeppelin CI
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #4266:
URL: https://github.com/apache/zeppelin/pull/4266#discussion_r769564405
##########
File path: zeppelin-server/pom.xml
##########
@@ -498,17 +477,95 @@
</zeppelin.daemon.package.base>
</properties>
</profile>
+
+ <profile>
+ <id>hadoop2</id>
+
+ <properties>
+ <hadoop.version>${hadoop2.7.version}</hadoop.version>
+ <curator.version>2.7.1</curator.version>
+ <commons-io.version>2.11</commons-io.version>
+ <!--
+ the declaration site above of these variables explains why we need to re-assign them here
+ -->
+ <hadoop-client-api.artifact>hadoop-client</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-yarn-api</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client</hadoop-client-minicluster.artifact>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-common</artifactId>
+ <version>${hadoop.version}</version>
+ <scope>${hadoop.deps.scope}</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-client</artifactId>
+ </dependency>
+ </dependencies>
+
+ </profile>
<profile>
<id>hadoop3</id>
+
+ <properties>
+ <hadoop.version>${hadoop3.2.version}</hadoop.version>
+ <curator.version>2.13.0</curator.version>
+ <kerberos-client.version>2.0.0-M15</kerberos-client.version>
+
+ <hadoop-client-api.artifact>hadoop-client-api</hadoop-client-api.artifact>
+ <hadoop-client-runtime.artifact>hadoop-client-runtime</hadoop-client-runtime.artifact>
+ <hadoop-client-minicluster.artifact>hadoop-client-minicluster</hadoop-client-minicluster.artifact>
+
+ </properties>
+
<dependencies>
+
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
Review comment:
It is only true when it is hadoop3, but not true for hadoop2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org