Error while calling udf Spark submit

[Divya Gehlot <di...@gmail.com>]

Hi,
I am on Spark 1.6.1
I am getting below error when I am trying to call UDF in my spark Dataframe
column
UDF
/* get the train line */
val deriveLineFunc :(String => String) = (str:String) => {
val build_key = str.split(",").toList
val getValue  = if(build_key.length > 1)
MapBrdcst.value.getOrElse((build_key(0),build_key(1)),("no","no")) else
("no","no")
val (value:String,_) = getValue
value
}

Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 0 in stage 12.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 12.0 (TID 621,
ip-xxx-xx-xx-xx.ap-southeast-1.compute.internal):
java.lang.ExceptionInInitializerError



Any clue ,what went wrong


Thanks,

Divya

Re: Error while calling udf Spark submit

[Marco Mistroni <mm...@gmail.com>]

Not enough info. But u can try same code in spark shell and get hold of the
exception
Hth

On 8 Sep 2016 11:16 am, "Divya Gehlot" <di...@gmail.com> wrote:

> Hi,
> I am on Spark 1.6.1
> I am getting below error when I am trying to call UDF in my spark
> Dataframe column
> UDF
> /* get the train line */
> val deriveLineFunc :(String => String) = (str:String) => {
> val build_key = str.split(",").toList
> val getValue  = if(build_key.length > 1) MapBrdcst.value.getOrElse((
> build_key(0),build_key(1)),("no","no")) else ("no","no")
> val (value:String,_) = getValue
> value
> }
>
> Exception in thread "main" org.apache.spark.SparkException: Job aborted
> due to stage failure: Task 0 in stage 12.0 failed 4 times, most recent
> failure: Lost task 0.3 in stage 12.0 (TID 621, ip-xxx-xx-xx-xx.ap-southeast-1.compute.internal):
> java.lang.ExceptionInInitializerError
>
>
>
> Any clue ,what went wrong
>
>
> Thanks,
>
> Divya
>
>
>
>