You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Simone Pallotta <si...@mail.polimi.it> on 2017/10/04 09:37:53 UTC
different behaviour linux/Unix vs windows when load spark context in
scala method called from R function using rscala package
In my R code, I am using rscala package to bridge to a scala method. in scala method I have initialized a spark context to be used later.
R code:
s <- scala(classpath = "<path to jar>", heap.maximum = "4g")
assign("WrappeR",s$.it.wrapper.r.Wrapper)
WrappeR$init()
where init is a scala function and Wrapper is scala class name (see below).
I created a scala object (Wrapper) with init function
scala code:
package it.wrapper.r
object Wrapper {
var Spark_context: SparkContext = _
def init(): Unit = {
val spark_conf = new SparkConf()
.setMaster("local[*]")
.setAppName("GMQL-R")
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.executor.memory", "6g")
.set("spark.driver.memory", "2g")
Spark_context = SparkContext.getOrCreate(spark_conf)
}
}
I removed the following code after Spark_context because is not useful.
this code works without flaws under OS X and Linux. Of course window is not gentle as other OSes. I checked out my project under windows (I have installed scala 2.11.8, java 1.8 and create environment variable JAVA_HOME and SCALA_HOME that are mandatory for running rscala package in R, I did the same in OS X/Linux)
when I run the jar file as standalone application it works fine under windows, but when I invoke the same function from R it fails giving this error:
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
at org.apache.hadoop.security.Groups.<init>(Groups.java:55)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:235)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
at `enter code here`org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:295)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:145)
at it.polimi.genomics.r.Wrapper$.initGMQL(Wrapper.scala:98)
at $line4.$read$$iw$$iw$$anonfun$1.apply$mcV$sp(<console>:19)
at $line4.$read$$iw$$iw$$anonfun$1.apply(<console>:16)
at $line4.$read$$iw$$iw$$anonfun$1.apply(<console>:16)
... 46 more
Caused by: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1746)
... 62 more
java.lang.reflect.InvocationTargetException
java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.hadoop.security.GroupMappingServiceProvider
Does anyone has already seen this kind of behavior?