You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by ankits <an...@gmail.com> on 2016/10/31 21:05:55 UTC
Re: Issue with repartition and cache
Hi,
Did you ever figure this one out? I'm seeing the same behavior:
Calling cache() after a repartition() makes Spark cache the version of the
RDD BEFORE the repartition, which means a shuffle everytime it is accessed..
However, calling cache before the repartition() seems to work fine, the
cached version has the new partitioning applied.
In summary, these 2 patterns dont seem to work as expected:
-------
repartition()
cache()
--------
repartition()
cache()
count()
---------
But this works:
cache()
repartition()
Very strange..
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Issue-with-repartition-and-cache-tp10235p19664.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org