You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Krishna Kalyan (JIRA)" <ji...@apache.org> on 2016/06/28 08:56:57 UTC
[jira] [Comment Edited] (SPARK-16055) sparkR.init() can not load
sparkPackages when executing an R file
[ https://issues.apache.org/jira/browse/SPARK-16055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352051#comment-15352051 ]
Krishna Kalyan edited comment on SPARK-16055 at 6/28/16 8:55 AM:
-----------------------------------------------------------------
Hi [~shivaram]
log stack trace below (Spark 1.6) - Every thing works fine / Unable to replicate the issue
https://gist.github.com/krishnakalyan3/78eaf61e4c53fe8b6ed26c7133ae653e
I even tried with spark 1.5.
log stack trace below (Spark 1.5) - Every thing works fine / Unable to replicate the issue
https://gist.github.com/krishnakalyan3/4a433cc854def9cb13925b431bd2dfd2
Could you please let me know how to proceed.
Thanks,
Krishna
was (Author: krishnakalyan3):
Hi [~shivaram]
(Spark 1.6) - Could replicate the issue with the error above.
log stack trace below (Spark 1.5) - Every thing works fine / Unable to replicate the issue
https://gist.github.com/krishnakalyan3/4a433cc854def9cb13925b431bd2dfd2
Could you please help me understand why there is a problem for version 1.6 and for 1.5 every thing works fine.
Thanks,
Krishna
> sparkR.init() can not load sparkPackages when executing an R file
> -----------------------------------------------------------------
>
> Key: SPARK-16055
> URL: https://issues.apache.org/jira/browse/SPARK-16055
> Project: Spark
> Issue Type: Brainstorming
> Components: SparkR
> Affects Versions: 1.6.1
> Reporter: Sun Rui
> Priority: Minor
>
> This is an issue reported in the Spark user mailing list. Refer to http://comments.gmane.org/gmane.comp.lang.scala.spark.user/35742
> This issue does not occur in an interactive SparkR session, while it does occur when executing an R file.
> The following example code can be put into an R file to reproduce this issue:
> {code}
> .libPaths(c("/home/user/spark-1.6.1-bin-hadoop2.6/R/lib",.libPaths()))
> Sys.setenv(SPARK_HOME="/home/user/spark-1.6.1-bin-hadoop2.6")
> library("SparkR")
> sc <- sparkR.init(sparkPackages = "com.databricks:spark-csv_2.11:1.4.0")
> sqlContext <- sparkRSQL.init(sc)
> df <- read.df(sqlContext, "file:///home/user/spark-1.6.1-bin-hadoop2.6/data/mllib/sample_tree_data.csv","csv")
> showDF(df)
> {code}
> The error message is as such:
> {panel}
> 16/06/19 15:48:56 ERROR RBackendHandler: loadDF on org.apache.spark.sql.api.r.SQLUtils failed
> Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
> java.lang.ClassNotFoundException: Failed to find data source: csv. Please find packages at http://spark-packages.org
> at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77)
> at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
> at org.apache.spark.sql.api.r.SQLUtils$.loadDF(SQLUtils.scala:160)
> at org.apache.spark.sql.api.r.SQLUtils.loadDF(SQLUtils.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:141)
> at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala
> Calls: read.df -> callJStatic -> invokeJava
> Execution halted
> {panel}
> The reason behind this is that in case you execute an R file, the R backend launches before the R interpreter, so there is no opportunity for packages specified with ‘sparkPackages’ to be processed.
> This JIRA issue is to track this issue. An appropriate solution is to be discussed. Maybe documentation the limitation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org