You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sutanu Das <sd...@att.com> on 2016/02/19 16:15:50 UTC
Spark stream job is take up /TMP with 100%
We have a Spark steaming job and when running in LOCAL mode, it takes up /TMP at 100% and Fails with error below, this doesn't happen in YARN Mode but in YARN we have performance issues.
How can I re-direct Spark Local Subffle from /TMP to /other_filesystem_location (where we have lots of space) ?
Please help us, thanks again community
16/02/19 09:22:52 ERROR Executor: Exception in task 1.0 in stage 4336.0 (TID 470495)
java.io.FileNotFoundException: /tmp/spark-348b864b-1f78-4109-8649-6ebf32fca3ff/blockmgr-a4aeabb9-f52a-460c-98e7-ec57b523055b/2f/temp_shuffle_900e97fe-b9d3-4d28-b59b-68954a8d81d5 (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
Re: Spark stream job is take up /TMP with 100%
Posted by Holden Karau <ho...@pigscanfly.ca>.
Thats a good question, you can find most of what you are looking for in the
configuration guide at
http://spark.apache.org/docs/latest/configuration.html - you probably want
to change the spark.local.dir to point to your scratch directory. Out of
interest what problems have you been seeing with YARN that you haven't in
local mode?
On Fri, Feb 19, 2016 at 7:15 AM, Sutanu Das <sd...@att.com> wrote:
> We have a Spark steaming job and when running in LOCAL mode, it takes up
> /TMP at 100% and Fails with error below, this doesn’t happen in YARN Mode
> but in YARN we have performance issues.
>
>
>
> How can I re-direct Spark Local Subffle from /TMP to
> /other_filesystem_location (where we have lots of space) ?
>
>
>
>
>
> Please help us, thanks again community
>
>
>
>
>
>
>
> 16/02/19 09:22:52 ERROR Executor: Exception in task 1.0 in stage 4336.0
> (TID 470495)
>
> java.io.FileNotFoundException: */tmp/spark-348b864b-1f78-4109-8649-6ebf32fca3ff/blockmgr-a4aeabb9-f52a-460c-98e7-ec57b523055b/2f/temp_shuffle_900e97fe-b9d3-4d28-b59b-68954a8d81d5
> (No such file or directory)*
>
> at java.io.FileOutputStream.open0(Native Method)
>
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
>
> at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
>
>
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau