You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sachit Murarka <co...@gmail.com> on 2021/01/31 13:08:27 UTC

EOF Exception Spark Structured Streams - Kubernetes

Hi Users,

I am running Spark application on Kubernetes and getting the following
exception in the driver pod. Though it is not affecting the output.

This exception is coming every 5 minutes and this is a structured streaming
job.

Could anyone please advise ?

21/01/29 06:33:15 WARN WatchConnectionManager: Exec
Failurejava.io.EOFException at
okio.RealBufferedSource.require(RealBufferedSource.java:61) at
okio.RealBufferedSource.readByte(RealBufferedSource.java:74) at
okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117) at
okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at
okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at
okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at
okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)21/01/29 06:38:16 WARN
WatchConnectionManager: Exec Failure


Kind Regards,
Sachit Murarka

Re: EOF Exception Spark Structured Streams - Kubernetes

Posted by Sachit Murarka <co...@gmail.com>.

Hi Prashant,

Yes I understand that will be available from Spark 3.0.2 . But I am using
Spark3.0.1

Could you please suggest any way to mitigate this exception?

Will it cause any issues from this exception in the spark application?

Thanks
Sachit

On Mon, 1 Feb 2021, 13:57 Prashant Sharma, <sc...@gmail.com> wrote:

> Hi Sachit,
>
> The fix verison on that JIRA says 3.0.2, so this fix is not yet released.
> Soon, there will be a 3.1.1 release, in the meantime you can try out the
> 3.1.1-rc which also has the fix and let us know your findings.
>
> Thanks,
>
>
> On Mon, Feb 1, 2021 at 10:24 AM Sachit Murarka <co...@gmail.com>
> wrote:
>
>> Following is the related JIRA , Can someone pls check
>>
>> https://issues.apache.org/jira/browse/SPARK-24266
>>
>> I am using 3.0.1 , It says fixed in 3.0.0 and 3.1.0 . Could you please
>> suggest what can be done to avoid this?
>>
>> Kind Regards,
>> Sachit Murarka
>>
>>
>> On Sun, Jan 31, 2021 at 6:38 PM Sachit Murarka <co...@gmail.com>
>> wrote:
>>
>>> Hi Users,
>>>
>>> I am running Spark application on Kubernetes and getting the following
>>> exception in the driver pod. Though it is not affecting the output.
>>>
>>> This exception is coming every 5 minutes and this is a structured
>>> streaming job.
>>>
>>> Could anyone please advise ?
>>>
>>> 21/01/29 06:33:15 WARN WatchConnectionManager: Exec
>>> Failurejava.io.EOFException at
>>> okio.RealBufferedSource.require(RealBufferedSource.java:61) at
>>> okio.RealBufferedSource.readByte(RealBufferedSource.java:74) at
>>> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117) at
>>> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
>>> at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at
>>> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at
>>> okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at
>>> okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> at java.base/java.lang.Thread.run(Thread.java:834)21/01/29 06:38:16 WARN
>>> WatchConnectionManager: Exec Failure
>>>
>>>
>>> Kind Regards,
>>> Sachit Murarka
>>>
>>

Re: EOF Exception Spark Structured Streams - Kubernetes

Posted by Prashant Sharma <sc...@gmail.com>.

Hi Sachit,

The fix verison on that JIRA says 3.0.2, so this fix is not yet released.
Soon, there will be a 3.1.1 release, in the meantime you can try out the
3.1.1-rc which also has the fix and let us know your findings.

Thanks,


On Mon, Feb 1, 2021 at 10:24 AM Sachit Murarka <co...@gmail.com>
wrote:

> Following is the related JIRA , Can someone pls check
>
> https://issues.apache.org/jira/browse/SPARK-24266
>
> I am using 3.0.1 , It says fixed in 3.0.0 and 3.1.0 . Could you please
> suggest what can be done to avoid this?
>
> Kind Regards,
> Sachit Murarka
>
>
> On Sun, Jan 31, 2021 at 6:38 PM Sachit Murarka <co...@gmail.com>
> wrote:
>
>> Hi Users,
>>
>> I am running Spark application on Kubernetes and getting the following
>> exception in the driver pod. Though it is not affecting the output.
>>
>> This exception is coming every 5 minutes and this is a structured
>> streaming job.
>>
>> Could anyone please advise ?
>>
>> 21/01/29 06:33:15 WARN WatchConnectionManager: Exec
>> Failurejava.io.EOFException at
>> okio.RealBufferedSource.require(RealBufferedSource.java:61) at
>> okio.RealBufferedSource.readByte(RealBufferedSource.java:74) at
>> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117) at
>> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
>> at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at
>> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at
>> okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at
>> okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:834)21/01/29 06:38:16 WARN
>> WatchConnectionManager: Exec Failure
>>
>>
>> Kind Regards,
>> Sachit Murarka
>>
>

Re: EOF Exception Spark Structured Streams - Kubernetes

Posted by Sachit Murarka <co...@gmail.com>.

Following is the related JIRA , Can someone pls check

https://issues.apache.org/jira/browse/SPARK-24266

I am using 3.0.1 , It says fixed in 3.0.0 and 3.1.0 . Could you please
suggest what can be done to avoid this?

Kind Regards,
Sachit Murarka


On Sun, Jan 31, 2021 at 6:38 PM Sachit Murarka <co...@gmail.com>
wrote:

> Hi Users,
>
> I am running Spark application on Kubernetes and getting the following
> exception in the driver pod. Though it is not affecting the output.
>
> This exception is coming every 5 minutes and this is a structured
> streaming job.
>
> Could anyone please advise ?
>
> 21/01/29 06:33:15 WARN WatchConnectionManager: Exec
> Failurejava.io.EOFException at
> okio.RealBufferedSource.require(RealBufferedSource.java:61) at
> okio.RealBufferedSource.readByte(RealBufferedSource.java:74) at
> okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117) at
> okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
> at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) at
> okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) at
> okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at
> okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)21/01/29 06:38:16 WARN
> WatchConnectionManager: Exec Failure
>
>
> Kind Regards,
> Sachit Murarka
>