You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Soheila S." <so...@gmail.com> on 2017/04/14 11:32:42 UTC

Parameter in FlatMap function

Hello all,
Can someone help me to solve the following fundamental problem?


I have a JavaRDD and as a flatMap method, I call a new instance of a class
which implements FlatMapFunction. This class has a constructor method and a
call method. In constructor method, I set the values for "List" variables
which I need them in call method.

In local run, there is no problem and it works correctly, but when I run it
on cluster (using YARN) it doesn't know them and the variables are null. (I
get Null pointer exception)

Any solution, idea or hint will be really appreciated,


All the best,
Soheila

Re: Parameter in FlatMap function

Posted by Ankur Srivastava <an...@gmail.com>.
You should instead broadcast your list and then use the broadcast variable
in the flatmap function.

Thanks
Ankur

On Fri, Apr 14, 2017 at 4:32 AM, Soheila S. <so...@gmail.com> wrote:

> Hello all,
> Can someone help me to solve the following fundamental problem?
>
>
> I have a JavaRDD and as a flatMap method, I call a new instance of a class
> which implements FlatMapFunction. This class has a constructor method and a
> call method. In constructor method, I set the values for "List" variables
> which I need them in call method.
>
> In local run, there is no problem and it works correctly, but when I run
> it on cluster (using YARN) it doesn't know them and the variables are null.
> (I get Null pointer exception)
>
> Any solution, idea or hint will be really appreciated,
>
>
> All the best,
> Soheila
>