You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by klrmowse <kl...@gmail.com> on 2018/04/07 17:07:14 UTC
[Spark 2.x Core] Adding to ArrayList inside rdd.foreach()
it gives null pointer exception...
is there a workaround for adding to an arrayList during .foreach of an rdd?
thank you
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()
Posted by Gourav Sengupta <go...@gmail.com>.
Hi,
in case the key value store is large then can you give redis a try? SPARK
does work quite well with redis.
Regards,
Gourav Sengupta
On Sat, Apr 7, 2018 at 6:29 PM, klrmowse <kl...@gmail.com> wrote:
> okie, well...
>
> i'm working with a pair rdd <key, iterable(value)>
>
> i need to extract the values and store them somehow (maybe a simple
> Array??), which i later parallelize and reuse
>
> since adding to a list is a no-no, what, if any, are the other options?
> (Java Spark, btw)
>
>
>
> thanks
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()
Posted by Bryan Jeffrey <br...@gmail.com>.
You can just call rdd.flatMap(_._2).collect
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: klrmowse <kl...@gmail.com>
Sent: Saturday, April 7, 2018 1:29:34 PM
To: user@spark.apache.org
Subject: Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()
okie, well...
i'm working with a pair rdd <key, iterable(value)>
i need to extract the values and store them somehow (maybe a simple
Array??), which i later parallelize and reuse
since adding to a list is a no-no, what, if any, are the other options?
(Java Spark, btw)
thanks
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()
Posted by klrmowse <kl...@gmail.com>.
okie, well...
i'm working with a pair rdd <key, iterable(value)>
i need to extract the values and store them somehow (maybe a simple
Array??), which i later parallelize and reuse
since adding to a list is a no-no, what, if any, are the other options?
(Java Spark, btw)
thanks
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: [Spark 2.x Core] Adding to ArrayList inside rdd.foreach()
Posted by Jörn Franke <jo...@gmail.com>.
What are you trying to achieve ? You should not use global variables in a spark application. Especially not adding to a list - that makes in most cases no sense.
If you want to put everything into a file then you should repartition to 1 .
> On 7. Apr 2018, at 19:07, klrmowse <kl...@gmail.com> wrote:
>
> it gives null pointer exception...
>
> is there a workaround for adding to an arrayList during .foreach of an rdd?
>
>
> thank you
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org