You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by "poplav (JIRA)" <ji...@apache.org> on 2016/05/25 19:25:12 UTC
[jira] [Commented] (TOREE-315) SparkR interpreter hangs when
running on Spark 1.6.1
[ https://issues.apache.org/jira/browse/TOREE-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300694#comment-15300694 ]
poplav commented on TOREE-315:
------------------------------
[~gurvinder], I believe your are hitting an issue where markSuccess on the sparkr_runner is hung.
https://github.com/apache/incubator-toree/blob/master/sparkr-interpreter/src/main/resources/kernelR/sparkr_runner.R#L116 is this not completing for you or is it something else? As an example, so you can see in the notebook that it received code, evaluated it, but it hangs on marking it as success? I'm trying to figure out if the issue was introduced when manually upgrading the sparkR fork or from this markSuccess issue.
> SparkR interpreter hangs when running on Spark 1.6.1
> ----------------------------------------------------
>
> Key: TOREE-315
> URL: https://issues.apache.org/jira/browse/TOREE-315
> Project: TOREE
> Issue Type: Bug
> Affects Versions: 0.1.0
> Environment: Spark 1.6.1
> Jupyter notebook 4.2.0
> R 3.2.2
> Toree 0.1.0.dev7
> Reporter: Gurvinder
> Priority: Minor
> Fix For: 0.1.0
>
>
> Create notebook with scala interpreter and once notebook started with spark context, running SparkR magic as
> %%SparkR sc
> cause notebook to hang and no further execution of any cells. Need to restart the notebook. Issue seems to be caused by patching/updating install SparkR package in spark directory (/usr/local/spark/R/libs/SparkR , in my case). Here is the diff from default installed SparkR and updated one
> diff SparkR/DESCRIPTION SparkR.bak/DESCRIPTION
> 4c4
> < Version: 1.6.1
> ---
> > Version: 1.5.0
> 16,18c16,17
> < 'functions.R' 'mllib.R' 'serialize.R' 'sparkR.R' 'stats.R'
> < 'types.R' 'utils.R'
> < Built: R 3.1.1; ; 2016-02-27 04:45:19 UTC; unix
> ---
> > 'functions.R' 'mllib.R' 'serialize.R' 'sparkR.R' 'utils.R'
> > Built: R 3.2.1; ; 2015-10-01 18:38:50 UTC; unix
> Only in SparkR: INDEX
> Common subdirectories: SparkR/Meta and SparkR.bak/Meta
> diff SparkR/NAMESPACE SparkR.bak/NAMESPACE
> 12a13,22
> > # Needed exports for runner
> > export("sparkR.connect")
> > export("isInstanceOf")
> > export("callJMethod")
> > export("callJStatic")
> > export("newJObject")
> > export("removeJObject")
> > export("isRemoveMethod")
> > export("invokeJava")
> >
> 26,27d35
> < "as.data.frame",
> < "attach",
> 30,33d37
> < "colnames",
> < "colnames<-",
> < "coltypes",
> < "coltypes<-",
> 36,37d39
> < "cov",
> < "corr",
> 49d50
> < "freqItems",
> Also once the default version is patched, running sparkR from terminal also give this error which worked earlier
> Launching java with spark-submit command /usr/local/spark/bin/spark-submit "sparkr-shell" /tmp/RtmpbfR7Ea/backend_port27b11cd7f3f
> 16/05/17 18:26:02 ERROR RBackendHandler: createSparkContext on org.apache.spark.api.r.RRDD failed
> Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)