You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Lantao Jin <ji...@gmail.com> on 2016/09/30 09:13:02 UTC

Lots of spark-assembly jars localized to /usercache/username/filecache directory

Hi,
Our Spark is deployed on YARN and I found there were lots of spark-assembly
jars in the Spark heavy user filecache directory (aka
/usercache/username/filecache), and you know the assembly jar is bigger
than 100 MB before Spark v2. So all of them take 26GB (1/4 reserved
space) in most of Datanodes. And that's only one Spark heavy user. We have
lot's of spark heavy user.


BTW, I didn't put the assembly jar to HDFS. It is in $SPARK_HOME/lib and
submitted every time.

Best regards
Alan