You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by gaurav sharma <sh...@gmail.com> on 2015/07/11 19:48:17 UTC

Worker dies with java.io.IOException: Stream closed

Hi All,

I am facing this issue in my production environment.

My worker dies by throwing this exception.
But i see the space is available on all the partitions on my disk
I did NOT see any abrupt increase in DIsk IO, which might have choked the
executor to write on to the stderr file.

But still my worker dies, this is not happening on all my workers, it's one
machine that is performing this way.
Could you please help me debug if it is happening because i am doing
something wrong, or some issue from hardware/OS perspective, that i can
debug and fix.


15/07/11 18:05:45 ERROR Worker: RECEIVED SIGNAL 1: SIGHUP
15/07/11 18:05:45 INFO ExecutorRunner: Killing process!
15/07/11 18:05:45 ERROR FileAppender: Error writing stream to file
/opt/spark-1.4.0-bin-hadoop2.6/work/app-20150710162005-0001/16517/stderr
java.io.IOException: Stream closed
        at
java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at java.io.FilterInputStream.read(FilterInputStream.java:107)
        at
org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
        at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
        at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
        at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
        at
org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
        at
org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
15/07/11 18:05:46 INFO Utils: Shutdown hook called
15/07/11 18:05:46 INFO Utils: Deleting directory
/tmp/spark-f269acd9-3ab0-4b3c-843c-bcf2e8c2669f
15/07/11 18:05:46 INFO Worker: Executor app-20150710162005-0001/16517
finished with state EXITED message Command exited with code 129 exitStatus
129

Re: Worker dies with java.io.IOException: Stream closed

Posted by gaurav sharma <sh...@gmail.com>.
the logs i pasted are from worker logs only,

spark does have permission to write into /opt, its not like the worker is
not able to start........it runs perfectly for days, but then abruptly dies.

and its not always this machine, sometimes its some other machine. It
happens once in a while, but whne it happens, the problem persists for at
least a day, no matter how many times i restart the worker, it again dies
with the same exception


On Sun, Jul 12, 2015 at 12:42 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:

> Can you dig a bit more in the worker logs? Also make sure that spark has
> permission to write to /opt/ on that machine as its one machine always
> throwing up.
>
> Thanks
> Best Regards
>
> On Sat, Jul 11, 2015 at 11:18 PM, gaurav sharma <sh...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I am facing this issue in my production environment.
>>
>> My worker dies by throwing this exception.
>> But i see the space is available on all the partitions on my disk
>> I did NOT see any abrupt increase in DIsk IO, which might have choked the
>> executor to write on to the stderr file.
>>
>> But still my worker dies, this is not happening on all my workers, it's
>> one machine that is performing this way.
>> Could you please help me debug if it is happening because i am doing
>> something wrong, or some issue from hardware/OS perspective, that i can
>> debug and fix.
>>
>>
>> 15/07/11 18:05:45 ERROR Worker: RECEIVED SIGNAL 1: SIGHUP
>> 15/07/11 18:05:45 INFO ExecutorRunner: Killing process!
>> 15/07/11 18:05:45 ERROR FileAppender: Error writing stream to file
>> /opt/spark-1.4.0-bin-hadoop2.6/work/app-20150710162005-0001/16517/stderr
>> java.io.IOException: Stream closed
>>         at
>> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
>>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
>>         at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>         at java.io.FilterInputStream.read(FilterInputStream.java:107)
>>         at
>> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
>>         at
>> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
>>         at
>> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>>         at
>> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>>         at
>> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
>>         at
>> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
>> 15/07/11 18:05:46 INFO Utils: Shutdown hook called
>> 15/07/11 18:05:46 INFO Utils: Deleting directory
>> /tmp/spark-f269acd9-3ab0-4b3c-843c-bcf2e8c2669f
>> 15/07/11 18:05:46 INFO Worker: Executor app-20150710162005-0001/16517
>> finished with state EXITED message Command exited with code 129 exitStatus
>> 129
>>
>>
>>
>>
>>
>

Re: Worker dies with java.io.IOException: Stream closed

Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Can you dig a bit more in the worker logs? Also make sure that spark has
permission to write to /opt/ on that machine as its one machine always
throwing up.

Thanks
Best Regards

On Sat, Jul 11, 2015 at 11:18 PM, gaurav sharma <sh...@gmail.com>
wrote:

> Hi All,
>
> I am facing this issue in my production environment.
>
> My worker dies by throwing this exception.
> But i see the space is available on all the partitions on my disk
> I did NOT see any abrupt increase in DIsk IO, which might have choked the
> executor to write on to the stderr file.
>
> But still my worker dies, this is not happening on all my workers, it's
> one machine that is performing this way.
> Could you please help me debug if it is happening because i am doing
> something wrong, or some issue from hardware/OS perspective, that i can
> debug and fix.
>
>
> 15/07/11 18:05:45 ERROR Worker: RECEIVED SIGNAL 1: SIGHUP
> 15/07/11 18:05:45 INFO ExecutorRunner: Killing process!
> 15/07/11 18:05:45 ERROR FileAppender: Error writing stream to file
> /opt/spark-1.4.0-bin-hadoop2.6/work/app-20150710162005-0001/16517/stderr
> java.io.IOException: Stream closed
>         at
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>         at java.io.FilterInputStream.read(FilterInputStream.java:107)
>         at
> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
>         at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
>         at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>         at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>         at
> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
>         at
> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
> 15/07/11 18:05:46 INFO Utils: Shutdown hook called
> 15/07/11 18:05:46 INFO Utils: Deleting directory
> /tmp/spark-f269acd9-3ab0-4b3c-843c-bcf2e8c2669f
> 15/07/11 18:05:46 INFO Worker: Executor app-20150710162005-0001/16517
> finished with state EXITED message Command exited with code 129 exitStatus
> 129
>
>
>
>
>