You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2016/08/29 02:12:02 UTC
[GitHub] zeppelin pull request #1377: ZEPPELIN-1384. Spark interpreter binary compati...
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/1377
ZEPPELIN-1384. Spark interpreter binary compatibility to scala 2.10 / 2.11 broken
### What is this PR for?
As described in ZEPPELIN-1384, we may hit the following error when running zeppelin on spark 1.6 if we build zeppelin this way
```
dev/change_scala_version.sh 2.11
mvn -DskipTests -Drat.skip=true -Pscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0 -Phadoop-2.6 -Pyarn -Ppyspark -Psparkr clean package
```
```
java.lang.NoSuchMethodError: scala.runtime.VolatileByteRef.create(B)Lscala/runtime/VolatileByteRef;
at scala.xml.MetaData$.iterate$1(MetaData.scala:39)
at scala.xml.MetaData$.normalize(MetaData.scala:45)
at scala.xml.Elem.<init>(Elem.scala:99)
at org.apache.spark.ui.jobs.StagePage$$anonfun$26.apply(StagePage.scala:57)
at org.apache.spark.ui.jobs.StagePage$$anonfun$26.apply(StagePage.scala:55)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at org.apache.spark.ui.jobs.StagePage.<init>(StagePage.scala:55)
at org.apache.spark.ui.jobs.StagesTab.<init>(StagesTab.scala:34)
at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:57)
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:195)
at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:146)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:473)
at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_1(SparkInterpreter.java:440)
at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:354)
at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:137)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:743)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
```
The root cause is that scala-xml is removed from scala 2.11 to a separate library, so here we have class conflict of scala-xml api. In this PR, I make the scope of scala-xml to be provided and also make the scope of scala-library to be provided although it will be override in `ZEPPELIN_HOME/spark/pom.xml`
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1384
### How should this be tested?
Tested manually.
Use the following command to build zeppelin and then run it on spark-1.6.2
```
dev/change_scala_version.sh 2.11
mvn -DskipTests -Drat.skip=true -Pscala-2.11 -Pspark-2.0 -Dspark.version=2.0.0 -Phadoop-2.6 -Pyarn -Ppyspark -Psparkr clean package
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1384
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1377.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1377
----
commit eb075352c7ba1157154b9a35f11bfca4d7d2927b
Author: Jeff Zhang <zj...@apache.org>
Date: 2016-08-29T01:58:29Z
ZEPPELIN-1384. Spark interpreter binary compatibility to scala 2.10 / 2.11 broken
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1377: ZEPPELIN-1384. Spark interpreter binary compati...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/1377
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1377: ZEPPELIN-1384. Spark interpreter binary compatibility ...
Posted by bzz <gi...@git.apache.org>.
Github user bzz commented on the issue:
https://github.com/apache/zeppelin/pull/1377
LGTM.
CI is green as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---