[jira] [Comment Edited] (TOREE-478) Problems with dependencies

["antonkulaga (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TOREE-478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567370#comment-16567370 ] 

antonkulaga edited comment on TOREE-478 at 8/2/18 6:59 PM:
-----------------------------------------------------------

I also have it with many others notebooks, like:
I have a dependency problem

After I fetch

%AddDeps comp.bio.aging adam-playground_2.11 0.0.12 --repository https://dl.bintray.com/comp-bio-aging/main/

I do not get any classes from it in apache-toree (I use it inside latest all-spark-notebook docker container)
I get something like:

import comp.bio.aging.playground.extras.uniprot._

val idmapping =  spark.readTypedTSV[UniprotMapping](az("/indexes/uniprot/idmapping_selected.tab"))

Name: Unknown Error
Message: <console>:145: error: not found: value comp
       import comp.bio.aging.playground.extras.uniprot._
              ^
<console>:146: error: value readTypedTSV is not a member of org.apache.spark.sql.SparkSession
       val idmapping =  spark.readTypedTSV[UniprotMapping](az("/indexes/uniprot/idmapping_selected.tab"))
                              ^
<console>:146: error: not found: type UniprotMapping
       val idmapping =  spark.readTypedTSV[UniprotMapping](az("/indexes/uniprot/idmapping_selected.tab"))
                                           ^
StackTrace:


was (Author: antonkulaga):
I also have it with many others notebooks, like:


> Problems with dependencies
> --------------------------
>
>                 Key: TOREE-478
>                 URL: https://issues.apache.org/jira/browse/TOREE-478
>             Project: TOREE
>          Issue Type: Bug
>    Affects Versions: 0.2.0
>         Environment: Linux Mint, docker container based on latest jupyter/all-spark-notebook https://github.com/antonkulaga/bigdata-docker/blob/master/containers/spark-ml-notebook/Dockerfile
>            Reporter: antonkulaga
>            Priority: Major
>
> Whenever I try to import some libs, like:
> %AddDeps org.bdgenomics.adam adam-core-spark2_2.11 0.24.0 --transitive
> %AddDeps comp.bio.aging adam-playground_2.11 0.0.13 --repository https://dl.bintray.com/comp-bio-aging/main/
> And then start using them in a cell:
> import org.bdgenomics.adam.rdd.ADAMContext._
> val genage = sc.loadFasta("/path/to/my/fasta/file"))
> I get:
> lastException = null
> Name: org.apache.spark.SparkException
> Message: Job aborted due to stage failure: ClassNotFound with classloader: scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@4c2a056b
> StackTrace:   at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1590)
>   at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1589)
>   at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1589)
>   at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
>   at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
>   at scala.Option.foreach(Option.scala:257)
>   at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
>   at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1823)
>   at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1772)
>   at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1761)
>   at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>   at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
>   at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
>   at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
>   at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
>   at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
>   at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:939)
>   at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>   at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
>   at org.apache.spark.rdd.RDD.collect(RDD.scala:938)
>   at org.bdgenomics.adam.rdd.contig.NucleotideContigFragmentRDD$.apply(NucleotideContigFragmentRDD.scala:95)
>   at org.bdgenomics.adam.rdd.ADAMContext$$anonfun$loadFasta$1.apply(ADAMContext.scala:2488)
>   at org.bdgenomics.adam.rdd.ADAMContext$$anonfun$loadFasta$1.apply(ADAMContext.scala:2472)
>   at scala.Option.fold(Option.scala:158)
>   at org.apache.spark.rdd.Timer.time(Timer.scala:48)
>   at org.bdgenomics.adam.rdd.ADAMContext.loadFasta(ADAMContext.scala:2472)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)