You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2020/11/08 23:31:58 UTC
[GitHub] [bigtop] iwasakims opened a new pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
iwasakims opened a new pull request #693:
URL: https://github.com/apache/bigtop/pull/693
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-724591807
```
[INFO] org.apache.zeppelin:zeppelin-zengine:jar:0.8.2
...
[INFO] +- org.apache.hadoop:hadoop-azure:jar:2.10.0:compile
[INFO] | \- com.microsoft.azure:azure-storage:jar:5.4.0:compile
```
Dependency tree shows that zeppelin-0.8.2 was comiled against hadoop-azure-2.10.0 and azure-storage-5.4.0 [even if the transient dependency of hadoop-azure is azure-storage-7.0.0](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/2.10.0).
If we update the hadoop.version to 2.10.1, the azure-storage version is changed to 7.0.1.
```
[INFO] org.apache.zeppelin:zeppelin-zengine:jar:0.8.2
...
[INFO] +- org.apache.hadoop:hadoop-azure:jar:2.10.1:compile
[INFO] | \- com.microsoft.azure:azure-storage:jar:7.0.1:compile
```
There are duplicate azure-storage entries in hadoop-project/pom.xml of hadoop-2.10.0.
https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-project/pom.xml#L1126-L1130
https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-project/pom.xml#L1187-L1191
My guess is that dependency conversion of maven-dependency-plugin is affected by this. The duplicate was removed by https://github.com/apache/hadoop/commit/0d4f9c778967ce0f83663c63389987335d47c3ea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-724503707
@iwasakims I also tested if other packages can be built with Hadoop 2.10.1. Most packages were successfully built, but only Zeppelin failed with the following error.
```
vagrant@debian9:~/bigtop$ ./gradlew zeppelin-clean zeppelin-pkg
(snip)
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/vagrant/bigtop/output/zeppelin/zeppelin-0.8.2/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/AzureNotebookRepo.java:[171,25] unreported exception java.net.URISyntaxException; must be caught or declared to be thrown
[ERROR] /home/vagrant/bigtop/output/zeppelin/zeppelin-0.8.2/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/AzureNotebookRepo.java:[175,26] unreported exception java.net.URISyntaxException; must be caught or declared to be thrown
[INFO] 2 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Zeppelin 0.8.2 ..................................... SUCCESS [ 7.239 s]
[INFO] Zeppelin: Interpreter Parent ....................... SUCCESS [ 1.467 s]
[INFO] Zeppelin: Interpreter .............................. SUCCESS [ 16.500 s]
[INFO] Zeppelin: Zengine .................................. FAILURE [ 17.591 s]
[INFO] Zeppelin: Display system apis ...................... SKIPPED
[INFO] Zeppelin: Groovy interpreter ....................... SKIPPED
[INFO] Zeppelin: Spark Scala Parent ....................... SKIPPED
[INFO] Zeppelin: Spark Interpreter Scala_2.10 ............. SKIPPED
[INFO] Zeppelin: Spark Interpreter Scala_2.11 ............. SKIPPED
[INFO] Zeppelin: Spark Parent ............................. SKIPPED
[INFO] Zeppelin: Spark Shims .............................. SKIPPED
[INFO] Zeppelin: Spark1 Shims ............................. SKIPPED
[INFO] Zeppelin: Spark2 Shims ............................. SKIPPED
[INFO] Zeppelin: Python interpreter ....................... SKIPPED
[INFO] Zeppelin: Spark Interpreter ........................ SKIPPED
[INFO] Zeppelin: Spark dependencies ....................... SKIPPED
[INFO] Zeppelin: Markdown interpreter ..................... SKIPPED
[INFO] Zeppelin: Angular interpreter ...................... SKIPPED
[INFO] Zeppelin: Shell interpreter ........................ SKIPPED
[INFO] Zeppelin: Livy interpreter ......................... SKIPPED
[INFO] Zeppelin: HBase interpreter ........................ SKIPPED
[INFO] Zeppelin: Apache Pig Interpreter ................... SKIPPED
[INFO] Zeppelin: JDBC interpreter ......................... SKIPPED
[INFO] Zeppelin: File System Interpreters ................. SKIPPED
[INFO] Zeppelin: Flink .................................... SKIPPED
[INFO] Zeppelin: Apache Ignite interpreter ................ SKIPPED
[INFO] Zeppelin: Kylin interpreter ........................ SKIPPED
[INFO] Zeppelin: Lens interpreter ......................... SKIPPED
[INFO] Zeppelin: Apache Cassandra interpreter ............. SKIPPED
[INFO] Zeppelin: Elasticsearch interpreter ................ SKIPPED
[INFO] Zeppelin: BigQuery interpreter ..................... SKIPPED
[INFO] Zeppelin: Alluxio interpreter ...................... SKIPPED
[INFO] Zeppelin: Scio ..................................... SKIPPED
[INFO] Zeppelin: Neo4j interpreter ........................ SKIPPED
[INFO] Zeppelin: Sap ...................................... SKIPPED
[INFO] Zeppelin: web Application .......................... SKIPPED
[INFO] Zeppelin: Server ................................... SKIPPED
[INFO] Zeppelin: Jupyter Support .......................... SKIPPED
[INFO] Zeppelin: Packaging distribution ................... SKIPPED
[INFO] Zeppelin: R Interpreter 0.8.2 ...................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 44.913 s
[INFO] Finished at: 2020-11-09T22:22:45-08:00
[INFO] ------------------------------------------------------------------------
```
It may be my environmental problem, so would you examine if it's reproducable to you?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-725709119
Merged into master and branch-1.5. Thanks @iwasakims!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims edited a comment on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
iwasakims edited a comment on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-724591807
```
[INFO] org.apache.zeppelin:zeppelin-zengine:jar:0.8.2
...
[INFO] +- org.apache.hadoop:hadoop-azure:jar:2.10.0:compile
[INFO] | \- com.microsoft.azure:azure-storage:jar:5.4.0:compile
```
Dependency tree shows that zeppelin-0.8.2 was compiled against hadoop-azure-2.10.0 and azure-storage-5.4.0 [even if the transitive dependency of hadoop-azure is azure-storage-7.0.0](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/2.10.0).
If we update the hadoop.version to 2.10.1, the azure-storage version is changed to 7.0.1.
```
[INFO] org.apache.zeppelin:zeppelin-zengine:jar:0.8.2
...
[INFO] +- org.apache.hadoop:hadoop-azure:jar:2.10.1:compile
[INFO] | \- com.microsoft.azure:azure-storage:jar:7.0.1:compile
```
There are duplicate azure-storage entries in hadoop-project/pom.xml of hadoop-2.10.0.
https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-project/pom.xml#L1126-L1130
https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-project/pom.xml#L1187-L1191
My guess is that dependency convergence of maven-dependency-plugin is affected by this. The duplicate was removed by https://github.com/apache/hadoop/commit/0d4f9c778967ce0f83663c63389987335d47c3ea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-724532379
The change of CloudFile#deleteIfExists was brought by https://github.com/Azure/azure-storage-java/commit/e3ec15cecd0615951bc1551c805ee342f578e42d on azure-cloud-storage v6.0.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn merged pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
sekikn merged pull request #693:
URL: https://github.com/apache/bigtop/pull/693
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-724641673
@sekikn I updated the PR. I think we should have the zeppelin related fix even for hadoop-2.10.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-723723538
smoke-tests of hdfs,yarn,mapreduce worked for me with the following configuration.
```
docker:
memory_limit: "8g"
image: "bigtop/puppet:trunk-centos-8"
repo: "file:///bigtop-home/output"
distro: centos
components: [hdfs,yarn,mapreduce]
enable_local_repo: true
smoke_test_components: [hdfs,yarn,mapreduce]
```
```
$ ./docker-hadoop.sh -c 3 --smoke-tests
...(snip)
BUILD SUCCESSFUL in 10m 48s
47 actionable tasks: 13 executed, 34 up-to-date
Stopped 1 worker daemon(s).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-725748993
Thanks, @sekikn .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] sekikn commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
sekikn commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-725706450
Tested newly added patch for Zeppelin. It worked as expected.
Though we're almost reaching to the 1.5.0 release, I think this upgrade is a minor one and doesn't introduce any problem as far as we tested.
It simplifies our code by removing unnecessary patches and improves Hadoop usability by adding features listed in https://hadoop.apache.org/docs/r2.10.1/index.html, so I'm +1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-724528598
@sekikn I can reproduce the issue. [HADOOP-17089](https://issues.apache.org/jira/browse/HADOOP-17089) (upgrading azure-storage-java from 7.0.0 to 7.0.1) seems to be the cause. It is a bit surprising that just bumping patch version has API incompatibility. Let me add a zeppelin side patch for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims commented on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-723795212
I got no issue on packaging and smoke-tests of Hive, HBase, Spark against Hadoop 2.10.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] iwasakims edited a comment on pull request #693: BIGTOP-3444. Bump Hadoop to 2.10.1.
Posted by GitBox <gi...@apache.org>.
iwasakims edited a comment on pull request #693:
URL: https://github.com/apache/bigtop/pull/693#issuecomment-724591807
```
[INFO] org.apache.zeppelin:zeppelin-zengine:jar:0.8.2
...
[INFO] +- org.apache.hadoop:hadoop-azure:jar:2.10.0:compile
[INFO] | \- com.microsoft.azure:azure-storage:jar:5.4.0:compile
```
Dependency tree shows that zeppelin-0.8.2 was compiled against hadoop-azure-2.10.0 and azure-storage-5.4.0 [even if the transitive dependency of hadoop-azure is azure-storage-7.0.0](https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/2.10.0).
If we update the hadoop.version to 2.10.1, the azure-storage version is changed to 7.0.1.
```
[INFO] org.apache.zeppelin:zeppelin-zengine:jar:0.8.2
...
[INFO] +- org.apache.hadoop:hadoop-azure:jar:2.10.1:compile
[INFO] | \- com.microsoft.azure:azure-storage:jar:7.0.1:compile
```
There are duplicate azure-storage entries in hadoop-project/pom.xml of hadoop-2.10.0.
https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-project/pom.xml#L1126-L1130
https://github.com/apache/hadoop/blob/rel/release-2.10.0/hadoop-project/pom.xml#L1187-L1191
My guess is that dependency conversion of maven-dependency-plugin is affected by this. The duplicate was removed by https://github.com/apache/hadoop/commit/0d4f9c778967ce0f83663c63389987335d47c3ea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org