You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Vaibhav Kulkarni <va...@gmail.com> on 2018/08/15 15:30:12 UTC

Shuffle uses Direct Memory Buffer even after setting "spark.shuffle.io.preferDirectBufs = false"

Hi,


I am using Standalone Spark 2.3 and have a question regarding Shuffle.
Going by the documentation, default Shuffle behaviour is to use Direct
Memory buffers. But, even after I set the following parameter, I notice
Shuffle still uses Direct Memory buffers.


spark.shuffle.io.preferDirectBufs = false


Is this a bug ? How can I disable use of Direct Memory for Shuffle


Thanks,

Vaibhav Kulkarni