You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Egor Pahomov <pa...@gmail.com> on 2014/09/25 14:18:48 UTC

java.io.FileNotFoundException in usercache

I work with spark on unstable cluster with bad administration.
I started get

14/09/25 15:29:56 ERROR storage.DiskBlockObjectWriter: Uncaught
exception while reverting partial writes to file
/local/hd2/yarn/local/usercache/epahomov/appcache/application_1411219858924_15501/spark-local-20140925151931-a4c3/3a/shuffle_4_30_174

java.io.FileNotFoundException:
/local/hd2/yarn/local/usercache/epahomov/appcache/application_1411219858924_15501/spark-local-20140925151931-a4c3/3a/shuffle_4_30_174
(No such file or directory)

couple days ago. After this error spark context shuted down. I'm are that
there are some problems with "distributed cache on cluster", some people
add too much data in it.

I totally don't understand what's going on, but willing to undertand deeply.

1) Does spark somehow rely on yarn localization mechanizm?
2) What is directory "usercache" about?
3) Is there a quick way to go around of problem?
4) Isn't shutting spark context is overreaction on this error?


-- 



*Sincerely yoursEgor PakhomovDeveloper, Yandex*