You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Usman Ghani <us...@platfora.com> on 2014/03/04 07:45:44 UTC

Shuffle Files

Where on the filesystem does spark write the shuffle files?

Re: Shuffle Files

Posted by Aniket Mokashi <an...@gmail.com>.
>From BlockManager code + ShuffleMapTask code, it writes under
spark.local.dir or java.io.tmpdir.

val diskBlockManager = new DiskBlockManager(shuffleBlockManager,
    conf.get("spark.local.dir",  System.getProperty("java.io.tmpdir")))




On Mon, Mar 3, 2014 at 10:45 PM, Usman Ghani <us...@platfora.com> wrote:

> Where on the filesystem does spark write the shuffle files?
>



-- 
"...:::Aniket:::... Quetzalco@tl"