You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Eugene Morozov <fa...@list.ru> on 2015/08/12 16:06:42 UTC

Does Spark optimization might miss to run transformation?

Hi!

I’d like to complete action (store / print smth) inside of transformation (map or mapPartitions). This approach has some flaws, but there is a question. Might it happen that Spark will optimise (RDD or DataFrame) processing so that my mapPartitions simply won’t happen?

--
Eugene Morozov
fathersson@list.ru

Re: Does Spark optimization might miss to run transformation?

Posted by Michael Armbrust <mi...@databricks.com>.

-dev

If you want to guarantee the side effects happen you should use foreach or
foreachPartitions.  A `take`, for example, might only evaluate a subset of
the partitions until it find enough results.

On Wed, Aug 12, 2015 at 7:06 AM, Eugene Morozov <fa...@list.ru> wrote:

> Hi!
>
> I’d like to complete action (store / print smth) inside of transformation
> (map or mapPartitions). This approach has some flaws, but there is a
> question. Might it happen that Spark will optimise (RDD or DataFrame)
> processing so that my mapPartitions simply won’t happen?
>
> --
> Eugene Morozov
> fathersson@list.ru
>
>
>
>
>