You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Meethu Mathew <me...@flytxt.com> on 2018/01/03 07:05:03 UTC
More than 2 notebooks in R failing with error sparkr intrepreter not responding
Hi,
I have met with a strange issue in running R notebooks in zeppelin(0.7.2).
Spark intrepreter is in per note Scoped mode and spark version is 1.6.2
Please find the steps below to reproduce the issue:
1. Create a notebook (Note1) and run any r code in a paragraph. I ran the
following code.
> %r
>
> rdf <- data.frame(c(1,2,3,4))
>
> colnames(rdf) <- c("myCol")
>
> sdf <- createDataFrame(sqlContext, rdf)
>
> withColumn(sdf, "newCol", sdf$myCol * 2.0)
>
>
2. Create another notebook (Note2) and run any r code in a paragraph. I
ran the same code as above.
Till now everything works fine.
3. Create third notebook (Note3) and run any r code in a paragraph. I ran
the same code. This notebook fails with the error
> org.apache.zeppelin.interpreter.InterpreterException: sparkr is not
> responding
What I understood from the analysis is that the process created for
sparkr interpreter is not getting killed properly and this makes every
third model to throw an error while executing. The process will be killed
on restarting the sparkr interpreter and another 2 models could be executed
successfully. ie, For every third model run using the sparkr interpreter,
the error is thrown. We suspect this as a limitation with zeppelin.
Please help to solve this issue
Regards,
Meethu Mathew
Re: More than 2 notebooks in R failing with error sparkr intrepreter
not responding
Posted by Meethu Mathew <me...@flytxt.com>.
Hi Jeff,
PFB the interpreter log.
INFO [2018-01-03 12:10:05,960] ({pool-2-thread-9}
Logging.scala[logInfo]:58) - Starting HTTP Server
INFO [2018-01-03 12:10:05,961] ({pool-2-thread-9}
Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT
INFO [2018-01-03 12:10:05,963] ({pool-2-thread-9}
AbstractConnector.java[doStart]:338) - Started SocketConnector@0.0.0.0:58989
INFO [2018-01-03 12:10:05,963] ({pool-2-thread-9}
Logging.scala[logInfo]:58) - Successfully started service 'HTTP class
server' on port 58989.
INFO [2018-01-03 12:10:06,094] ({dispatcher-event-loop-1}
Logging.scala[logInfo]:58) - Removed broadcast_1_piece0 on localhost:42453
in memory (size: 854.0 B, free: 511.1 MB)
INFO [2018-01-03 12:10:07,049] ({pool-2-thread-9}
ZeppelinR.java[createRScript]:353) - File
/tmp/zeppelin_sparkr-5046601627391341672.R created
ERROR [2018-01-03 12:10:17,051] ({pool-2-thread-9} Job.java[run]:188) - Job
failed
*org.apache.zeppelin.interpreter.InterpreterException: sparkr is not
responding *
R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
....
....
> args <- commandArgs(trailingOnly = TRUE)
> hashCode <- as.integer(args[1])
> port <- as.integer(args[2])
> libPath <- args[3]
> version <- as.integer(args[4])
> rm(args)
>
> print(paste("Port ", toString(port)))
[1]
"Port 58063"
> print(paste("LibPath ", libPath))
[1]
"LibPath /home/meethu/spark-1.6.1-bin-hadoop2.6/R/lib"
>
> .libPaths(c(file.path(libPath), .libPaths()))
> library(SparkR)
Attaching package: ‘SparkR’
The following objects are masked from ‘package:stats’:
cov, filter, lag, na.omit, predict, sd, var
The following objects are masked from ‘package:base’:
colnames, colnames<-, endsWith, intersect, rank, rbind, sample,
startsWith, subset, summary, table, transform
> SparkR:::connectBackend("localhost", port, 6000)
A connection with
description "->localhost:58063"
class
"sockconn"
mode "wb"
text "binary"
opened "opened"
can read "yes"
can write "yes"
>
> # scStartTime is needed by R/pkg/R/sparkR.R
> assign(".scStartTime", as.integer(Sys.time()), envir =
SparkR:::.sparkREnv)
> # getZeppelinR
> *.zeppelinR = SparkR:::callJStatic("org.apache.zeppelin.spark.ZeppelinR",
"getZeppelinR", hashCode)*
at
org.apache.zeppelin.spark.ZeppelinR.waitForRScriptInitialized(ZeppelinR.java:285)
at org.apache.zeppelin.spark.ZeppelinR.request(ZeppelinR.java:227)
at org.apache.zeppelin.spark.ZeppelinR.eval(ZeppelinR.java:176)
at org.apache.zeppelin.spark.ZeppelinR.open(ZeppelinR.java:165)
at
org.apache.zeppelin.spark.SparkRInterpreter.open(SparkRInterpreter.java:90)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:491)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [2018-01-03 12:10:17,070] ({pool-2-thread-9}
SchedulerFactory.java[jobFinished]:137) - Job
remoteInterpretJob_1514961605951 finished by scheduler
org.apache.zeppelin.spark.SparkRInterpreter392022746
INFO [2018-01-03 12:39:22,664] ({Spark Context Cleaner}
Logging.scala[logInfo]:58) - Cleaned accumulator 2
PFB the output of the command *ps -ef | grep /usr/lib/R/bin/exec/R*
meethu 6647 6470 0 12:09 pts/1 00:00:00 /usr/lib/R/bin/exec/R
--no-save --no-restore -f /tmp/zeppelin_sparkr-1100854828050763213.R --args
214655664 58063 /home/meethu/spark-1.6.1-bin-hadoop2.6/R/lib 10601
meethu 6701 6470 0 12:09 pts/1 00:00:00 /usr/lib/R/bin/exec/R
--no-save --no-restore -f /tmp/zeppelin_sparkr-4152305170353311178.R --args
1642312173 58063 /home/meethu/spark-1.6.1-bin-hadoop2.6/R/lib 10601
meethu 6745 6470 0 12:10 pts/1 00:00:00 /usr/lib/R/bin/exec/R
--no-save --no-restore -f /tmp/zeppelin_sparkr-5046601627391341672.R --args
1158632477 58063 /home/meethu/spark-1.6.1-bin-hadoop2.6/R/lib 10601
Regards,
Meethu Mathew
On Wed, Jan 3, 2018 at 12:56 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
> Could you check the interpreter log ?
>
> Meethu Mathew <me...@flytxt.com>于2018年1月3日周三 下午3:05写道：
>
>> Hi,
>>
>> I have met with a strange issue in running R notebooks in zeppelin(0.7.2).
>> Spark intrepreter is in per note Scoped mode and spark version is 1.6.2
>>
>> Please find the steps below to reproduce the issue:
>> 1. Create a notebook (Note1) and run any r code in a paragraph. I ran the
>> following code.
>>
>>> %r
>>>
>>> rdf <- data.frame(c(1,2,3,4))
>>>
>>> colnames(rdf) <- c("myCol")
>>>
>>> sdf <- createDataFrame(sqlContext, rdf)
>>>
>>> withColumn(sdf, "newCol", sdf$myCol * 2.0)
>>>
>>>
>> 2. Create another notebook (Note2) and run any r code in a paragraph. I
>> ran the same code as above.
>>
>> Till now everything works fine.
>>
>> 3. Create third notebook (Note3) and run any r code in a paragraph. I
>> ran the same code. This notebook fails with the error
>>
>>> org.apache.zeppelin.interpreter.InterpreterException: sparkr is not
>>> responding
>>
>>
>> What I understood from the analysis is that the process created for
>> sparkr interpreter is not getting killed properly and this makes every
>> third model to throw an error while executing. The process will be killed
>> on restarting the sparkr interpreter and another 2 models could be executed
>> successfully. ie, For every third model run using the sparkr interpreter,
>> the error is thrown. We suspect this as a limitation with zeppelin.
>>
>> Please help to solve this issue
>>
>> Regards,
>>
>>
>> Meethu Mathew
>>
>>
Re: More than 2 notebooks in R failing with error sparkr intrepreter
not responding
Posted by Jeff Zhang <zj...@gmail.com>.
Could you check the interpreter log ?
Meethu Mathew <me...@flytxt.com>于2018年1月3日周三 下午3:05写道：
> Hi,
>
> I have met with a strange issue in running R notebooks in zeppelin(0.7.2).
> Spark intrepreter is in per note Scoped mode and spark version is 1.6.2
>
> Please find the steps below to reproduce the issue:
> 1. Create a notebook (Note1) and run any r code in a paragraph. I ran the
> following code.
>
>> %r
>>
>> rdf <- data.frame(c(1,2,3,4))
>>
>> colnames(rdf) <- c("myCol")
>>
>> sdf <- createDataFrame(sqlContext, rdf)
>>
>> withColumn(sdf, "newCol", sdf$myCol * 2.0)
>>
>>
> 2. Create another notebook (Note2) and run any r code in a paragraph. I
> ran the same code as above.
>
> Till now everything works fine.
>
> 3. Create third notebook (Note3) and run any r code in a paragraph. I ran
> the same code. This notebook fails with the error
>
>> org.apache.zeppelin.interpreter.InterpreterException: sparkr is not
>> responding
>
>
> What I understood from the analysis is that the process created for
> sparkr interpreter is not getting killed properly and this makes every
> third model to throw an error while executing. The process will be killed
> on restarting the sparkr interpreter and another 2 models could be executed
> successfully. ie, For every third model run using the sparkr interpreter,
> the error is thrown. We suspect this as a limitation with zeppelin.
>
> Please help to solve this issue
>
> Regards,
>
>
> Meethu Mathew
>
>