You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/11/15 13:39:00 UTC
[jira] [Commented] (TOREE-420) AddDeps fails if transitive
dependency includes org.scala-lang.modules
[ https://issues.apache.org/jira/browse/TOREE-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16253448#comment-16253448 ]
ASF GitHub Bot commented on TOREE-420:
--------------------------------------
GitHub user kervel opened a pull request:
https://github.com/apache/incubator-toree/pull/146
Add --exclude option to AddDeps and exclude org.scala-lang.modules by default
This PR includes the (in my eyes valid) fix to TOREE-420 by Kalvin Chau, and extends on it by adding a way to work around other dependency conflicts by excluding specific dependencies. This is already possible in other scala-based interactive notebooks.
For example, look at https://zeppelin.apache.org/docs/0.6.2/manual/dependencymanagement.html
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Kapernikov/incubator-toree depExcludes_rebased
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-toree/pull/146.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #146
----
commit db48bf33e867c05b4bb560a3f787153d86c1c393
Author: Kalvin Chau <ka...@viasat.com>
Date: 2017-06-22T20:26:41Z
updated depdency downloaders to exclude org.scala-lang.modules to prevent scala refelection issues TOREE-420
commit 2094e0187543ee111c7ee332d95d613b53a568ef
Author: Frank Dekervel <fr...@infrabel.be>
Date: 2017-09-11T07:58:04Z
Add --exclude option to AddDeps in order to be able to exclude certain modules
The exclude syntax is follows "organisation:name" or "organisation" when everything from a certain organisation is to be excluded
----
> AddDeps fails if transitive dependency includes org.scala-lang.modules
> ----------------------------------------------------------------------
>
> Key: TOREE-420
> URL: https://issues.apache.org/jira/browse/TOREE-420
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Affects Versions: 0.2.0
> Environment: Spark 2.1.1, Scala 2.11.8, Built from source, commit id: c7b008506fa07538b42aa94b287a361a64f71ee6
> Reporter: Kalvin Chau
>
> When using the AddDeps magic, if the dependency includes org.scala-lang.modules it fails with a reflection error, this kafka depdency for example:
> {code}
> %AddDeps org.apache.kafka kafka_2.11 0.10.2.0 --transitive --trace --verbose
> {code}
> {code}
> {'buffers': [],
> 'content': {'ename': 'scala.reflect.internal.FatalError',
> 'evalue': 'object Predef does not have a member classOf',
> 'traceback': ['scala.reflect.internal.Definitions$DefinitionsClass.scala$reflect$internal$Definitions$DefinitionsClass$$fatalMissingSymbol(Definitions.scala:1186)',
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMember(Definitions.scala:1203)',
> 'scala.reflect.internal.Definitions$DefinitionsClass.getMemberMethod(Definitions.scala:1238)',
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf$lzycompute(Definitions.scala:1469)',
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.Predef_classOf(Definitions.scala:1469)',
> 'scala.reflect.internal.Definitions$DefinitionsClass$RunDefinitions.isPredefClassOf(Definitions.scala:1459)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdent$2(Typers.scala:4885)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typedIdentOrWildcard$1(Typers.scala:4908)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5340)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5360)',
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5472)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5480)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)',
> 'scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)',
> 'scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)',
> 'scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)',
> 'scala.tools.nsc.interpreter.ReplGlobal$$anon$1$$anon$2.typed(ReplGlobal.scala:36)',
> 'scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)',
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)',
> 'scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:440)',
> 'scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:431)',
> 'scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:440)',
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)',
> 'scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:93)',
> {code}
> However, the kafka 2.10 version does not fail:
> {code}
> %AddDeps org.apache.kafka kafka_2.10 0.10.2.0 --transitive --trace --verbose
> {code}
> After inspecting the pom files for each of the versions, the only difference was the dependency on
> {code}
> <dependency>
> <groupId>org.scala-lang.modules</groupId>
> <artifactId>scala-parser-combinators_2.11</artifactId>
> <version>1.0.4</version>
> <scope>compile</scope>
> {code}
> We should exclude the org.scala-lang.modules organization as well.
> Including org.scala-lang.modules as a exclusion fixes this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)