You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Usman Ghani <us...@platfora.com> on 2014/03/04 07:45:44 UTC
Shuffle Files
Where on the filesystem does spark write the shuffle files?
Re: Shuffle Files
Posted by Aniket Mokashi <an...@gmail.com>.
>From BlockManager code + ShuffleMapTask code, it writes under
spark.local.dir or java.io.tmpdir.
val diskBlockManager = new DiskBlockManager(shuffleBlockManager,
conf.get("spark.local.dir", System.getProperty("java.io.tmpdir")))
On Mon, Mar 3, 2014 at 10:45 PM, Usman Ghani <us...@platfora.com> wrote:
> Where on the filesystem does spark write the shuffle files?
>
--
"...:::Aniket:::... Quetzalco@tl"