You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Niu Zhaojie <nz...@gmail.com> on 2018/04/10 04:35:06 UTC

pyspark.daemon exhaust a lot of memory

Hi All,

We are running spark 2.1.1 on Hadoop YARN 2.6.5.

We found the pyspark.daemon process consume more than 300GB memory.

However, according to
https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals, the
daemon process shouldn't have this problem.

Also, we find the daemon process is forked by the container process,
obviously it already beyonds the container memory limit, why YARN doesn't
kill this container?

-- 
*Regards,*
*Zhaojie*