You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Krishna <re...@gmail.com> on 2016/03/19 00:55:20 UTC

Saving intermediate results in mapPartitions

Hi,

I've a situation where the number of elements output by each partition from
mapPartitions don't fit into the RAM even with the lowest number of rows in
the partition (there is a hard lower limit on this value). What's the best
way to address this problem? During the mapPartition phase, is there a way
to convert intermediate results to a DF and save to a database? Rows saved
to database don't need to be part of the output results from mapPartitions.

Re: Saving intermediate results in mapPartitions

Posted by Enrico Rotundo <en...@gmail.com>.

Try to set MEMORY_AND_DISK as RDD’s storage persistence level.
http://spark.apache.org/docs/latest/programming-guide.html#rdd-persistence <http://spark.apache.org/docs/latest/programming-guide.html#rdd-persistence>
> On 19 Mar 2016, at 00:55, Krishna <re...@gmail.com> wrote:
> 
> Hi,
> 
> I've a situation where the number of elements output by each partition from mapPartitions don't fit into the RAM even with the lowest number of rows in the partition (there is a hard lower limit on this value). What's the best way to address this problem? During the mapPartition phase, is there a way to convert intermediate results to a DF and save to a database? Rows saved to database don't need to be part of the output results from mapPartitions.
> 
>