You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sandeep Khurana <sa...@infoworks.io> on 2016/01/22 12:05:15 UTC
SparkR works from command line but not from rstudio
Hello
I installed spark in a folder. I start bin/sparkR on console. Then I
execute below command and all work fine. I can see the data as well.
hivecontext <<- sparkRHive.init(sc) ;
df <- loadDF(hivecontext, "/someHdfsPath", "orc")
showDF(df)
But when I give same to rstudio, it throws the error mentioned below
rstudio code
============
Sys.setenv(SPARK_HOME="/home/myname/spark-1.6.0-bin-hadoop2.6")
.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
library(SparkR)
sc <- sparkR.init(master="local")
hivecontext <<- sparkRHive.init(sc) ;
df <- loadDF(hivecontext, "/someHdfsPath", "orc")
print("showing df now")
showDF(df)
Error thrown from rstudio
===================
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
for more info.Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties16/01/22 06:00:12 ERROR
RBackendHandler: createSparkContext on org.apache.spark.api.r.RRDD
failedError in invokeJava(isStatic = TRUE, className, methodName, ...)
:
What is different in rstudio than sparkR shell ? Should I change any
setting to make it work in rstudio ?
Re: SparkR works from command line but not from rstudio
Posted by Sandeep Khurana <sa...@infoworks.io>.
Resolved this issue after reinstalling r, rstudio. Had issues with earlier
installation.
On Jan 22, 2016 6:48 PM, "Sandeep Khurana" <sa...@infoworks.io> wrote:
> This problem is fixed by restarting R from R studio. Now see
>
> 16/01/22 08:08:38 INFO HiveMetaStore: No user is added in admin role, since config is empty16/01/22 08:08:38 ERROR RBackendHandler: <init> on org.apache.spark.sql.hive.HiveContext failedError in value[[3L]](cond) : Spark SQL is not built with Hive support
>
>
> in rstudio while running same code and hive-site.xml is present in the .
> It works in sparkR shell.
>
> Any ideas?
>
> On Fri, Jan 22, 2016 at 4:35 PM, Sandeep Khurana <sa...@infoworks.io>
> wrote:
>
>> Hello
>>
>> I installed spark in a folder. I start bin/sparkR on console. Then I
>> execute below command and all work fine. I can see the data as well.
>>
>> hivecontext <<- sparkRHive.init(sc) ;
>> df <- loadDF(hivecontext, "/someHdfsPath", "orc")
>> showDF(df)
>>
>>
>> But when I give same to rstudio, it throws the error mentioned below
>>
>> rstudio code
>> ============
>> Sys.setenv(SPARK_HOME="/home/myname/spark-1.6.0-bin-hadoop2.6")
>> .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
>> library(SparkR)
>>
>> sc <- sparkR.init(master="local")
>> hivecontext <<- sparkRHive.init(sc) ;
>> df <- loadDF(hivecontext, "/someHdfsPath", "orc")
>> print("showing df now")
>> showDF(df)
>>
>> Error thrown from rstudio
>> ===================
>>
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties16/01/22 06:00:12 ERROR RBackendHandler: createSparkContext on org.apache.spark.api.r.RRDD failedError in invokeJava(isStatic = TRUE, className, methodName, ...) :
>>
>>
>>
>> What is different in rstudio than sparkR shell ? Should I change any
>> setting to make it work in rstudio ?
>>
>>
>>
>
>
> --
> Architect
> Infoworks.io
> http://Infoworks.io
>
Re: SparkR works from command line but not from rstudio
Posted by Sandeep Khurana <sa...@infoworks.io>.
This problem is fixed by restarting R from R studio. Now see
16/01/22 08:08:38 INFO HiveMetaStore: No user is added in admin role,
since config is empty16/01/22 08:08:38 ERROR RBackendHandler: <init>
on org.apache.spark.sql.hive.HiveContext failedError in
value[[3L]](cond) : Spark SQL is not built with Hive support
in rstudio while running same code and hive-site.xml is present in the .
It works in sparkR shell.
Any ideas?
On Fri, Jan 22, 2016 at 4:35 PM, Sandeep Khurana <sa...@infoworks.io>
wrote:
> Hello
>
> I installed spark in a folder. I start bin/sparkR on console. Then I
> execute below command and all work fine. I can see the data as well.
>
> hivecontext <<- sparkRHive.init(sc) ;
> df <- loadDF(hivecontext, "/someHdfsPath", "orc")
> showDF(df)
>
>
> But when I give same to rstudio, it throws the error mentioned below
>
> rstudio code
> ============
> Sys.setenv(SPARK_HOME="/home/myname/spark-1.6.0-bin-hadoop2.6")
> .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
> library(SparkR)
>
> sc <- sparkR.init(master="local")
> hivecontext <<- sparkRHive.init(sc) ;
> df <- loadDF(hivecontext, "/someHdfsPath", "orc")
> print("showing df now")
> showDF(df)
>
> Error thrown from rstudio
> ===================
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties16/01/22 06:00:12 ERROR RBackendHandler: createSparkContext on org.apache.spark.api.r.RRDD failedError in invokeJava(isStatic = TRUE, className, methodName, ...) :
>
>
>
> What is different in rstudio than sparkR shell ? Should I change any
> setting to make it work in rstudio ?
>
>
>
--
Architect
Infoworks.io
http://Infoworks.io