You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Ophir Cohen <op...@gmail.com> on 2015/07/10 09:38:41 UTC
Re: NPE exception in Spark interpreter log + broken pipe exception
once awhile in the Zeppelin log
Hi Guys,
I'm raising this issue again.
My problem is that when I'm using Zeppelin on top yarn the Spark web-ui
does not work.
Actually I get java.lang.NullPointerException in the intereprtor log (see
below).
Tyring to access the web ui - it redirects to <url>:4040/jobs and returns:
HTTP ERROR: 500
Problem accessing /jobs/. Reason:
Server Error
Powered by Jetty://
Zepplein 0.5.0, on yarn, Hadoop 2.4, Spark 1.3.1
BTW
On spark shell the web ui opens as expected.
Any help will appricated!
10x
The error:
WARN [2015-07-10 07:36:13,456] ({qtp1043111551-65}
ServletHandler.java[doHandle]:561) - /jobs/
java.lang.NullPointerException
at
org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at
org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at
org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at
org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at
org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at
org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.spark-project.jetty.server.Server.handle(Server.java:370)
at
org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at
org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at
org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at
org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at
org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at
org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at
org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at
org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
On Tue, Jun 23, 2015 at 12:35 PM, Ophir Cohen <op...@gmail.com> wrote:
> Hi Moon and thanks for the answers, this mailing list is defiantly one of
> the responsive mailing list I have encountered.
>
> About the NPE, Zeppelin continue working as expected after the exception.
> Actually, this exception raised WHEN I'm trying to access the web ui.
> Also, it raised once awhile without anyone human that tries to access it
> (can it be that the YARN cluster or Zeppelin server tries to access the web
> ui for some reason??).
>
> It does not affects the functionality it affects my ability to monitor the
> Spark jobs and debug problems.
> BTW
> When running Spark shell the web UI works as excepted.
>
>
>
> On Tue, Jun 23, 2015 at 6:29 AM, moon soo Lee <mo...@apache.org> wrote:
>
>> Hi,
>>
>> Really appreciate for sharing the problem.
>> About the NPE, when do you get the exception? Are you getting the
>> exception when you trying to access Spark application ui?
>> If it is, any spark job (like .count(), .collect(), sparksql, etc) runs
>> without error?
>>
>> Thanks!
>> moon
>>
>> On Mon, Jun 22, 2015 at 5:19 AM Ophir Cohen <op...@gmail.com> wrote:
>>
>>> OK,
>>> I think we can seperate the issues.
>>> Let's start from the NPE. For some reason (is that on perpuse?) when
>>> Zeppelin starts Spark it seems that it does not open the web ui.
>>> Trying to access port 4041 (this is the one it started on) yield the
>>> above NPE with the "WARN [2015-06-22 09:22:37,265] ({qtp622396559-74}
>>> ServletHandler.java[doHandle]:561) - /jobs/
>>> java.lang.NullPointerException" message.
>>>
>>> Any ideas?
>>>
>>> On Mon, Jun 22, 2015 at 12:34 PM, Ophir Cohen <op...@gmail.com> wrote:
>>>
>>>> Hi Guys and great work!
>>>> In our company we decided to relay on Zeppelin as our tool to access
>>>> and visualize our data.
>>>> I'm working with Spark 1.3.1 on top of Hadoop yarn 2.4
>>>>
>>>> Lately we started to see many NullPointerExcpetion on the Spark
>>>> interpreter log (see below).
>>>> Sometimes it does not affect Zeppelin but sometimes we start to see
>>>> 'can't get status' error caused by broken pipe, this time on the
>>>> *zeppelin* log file (see below).
>>>> When the latest error starts - it stacks the Spark interpretor and it
>>>> enforce us to restart Zeppelin.
>>>>
>>>> I couldn't find when and why it's happening, it just happened once
>>>> awhile...
>>>> Any help will be more than appreciated!
>>>> 10x!
>>>>
>>>> *The NPE on zeppelin-interpreter-spark log file:*
>>>> WARN [2015-06-22 09:22:37,265] ({qtp622396559-74}
>>>> ServletHandler.java[doHandle]:561) - /jobs/
>>>> java.lang.NullPointerException
>>>> at
>>>> org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
>>>> at
>>>> org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
>>>> at
>>>> org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
>>>> at
>>>> org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
>>>> at
>>>> org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
>>>> at
>>>> org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>>> at
>>>> org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>>> at
>>>> org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>>> at org.spark-project.jetty.server.Server.handle(Server.java:370)
>>>> at
>>>> org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
>>>> at
>>>> org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
>>>> at
>>>> org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
>>>> at
>>>> org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644)
>>>> at
>>>> org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>>> at
>>>> org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>>>> at
>>>> org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
>>>> at
>>>> org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>>>> at
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>> at
>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> *The broken pipe on zeppelin log file:*
>>>> ERROR [2015-06-21 13:10:31,368] ({Thread-120}
>>>> RemoteScheduler.java[getStatus]:245) - Can't get status information
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.SocketException: Broken pipe
>>>> at
>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>>> at
>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
>>>> at
>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.send_getStatus(RemoteInterpreterService.java:319)
>>>> at
>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getStatus(RemoteInterpreterService.java:311)
>>>> at
>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.getStatus(RemoteScheduler.java:232)
>>>> at
>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.run(RemoteScheduler.java:183)
>>>> Caused by: java.net.SocketException: Broken pipe
>>>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>>>> at
>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>>> at
>>>> java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>>>> at
>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>> at
>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>> at
>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>>> ... 5 more
>>>>
>>>>
>>>
>
Re: NPE exception in Spark interpreter log + broken pipe exception
once awhile in the Zeppelin log
Posted by moon soo Lee <mo...@apache.org>.
Hi,
Could you create an issue if you keep seeing the problem?
I tried by i could access :4040/jobs without error
Thanks,
moon
On Fri, Jul 10, 2015 at 12:39 AM Ophir Cohen <op...@gmail.com> wrote:
> Hi Guys,
> I'm raising this issue again.
>
> My problem is that when I'm using Zeppelin on top yarn the Spark web-ui
> does not work.
> Actually I get java.lang.NullPointerException in the intereprtor log (see
> below).
>
> Tyring to access the web ui - it redirects to <url>:4040/jobs and returns:
> HTTP ERROR: 500
>
> Problem accessing /jobs/. Reason:
>
> Server Error
> Powered by Jetty://
>
> Zepplein 0.5.0, on yarn, Hadoop 2.4, Spark 1.3.1
>
> BTW
> On spark shell the web ui opens as expected.
> Any help will appricated!
> 10x
>
> The error:
> WARN [2015-07-10 07:36:13,456] ({qtp1043111551-65}
> ServletHandler.java[doHandle]:561) - /jobs/
>
> java.lang.NullPointerException
> at
> org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
> at
> org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
> at
> org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
> at
> org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
> at
> org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
> at
> org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.spark-project.jetty.server.Server.handle(Server.java:370)
> at
> org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
> at
> org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
> at
> org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
> at
> org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at
> org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at
> org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> at
> org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> On Tue, Jun 23, 2015 at 12:35 PM, Ophir Cohen <op...@gmail.com> wrote:
>
>> Hi Moon and thanks for the answers, this mailing list is defiantly one of
>> the responsive mailing list I have encountered.
>>
>> About the NPE, Zeppelin continue working as expected after the exception.
>> Actually, this exception raised WHEN I'm trying to access the web ui.
>> Also, it raised once awhile without anyone human that tries to access it
>> (can it be that the YARN cluster or Zeppelin server tries to access the web
>> ui for some reason??).
>>
>> It does not affects the functionality it affects my ability to monitor
>> the Spark jobs and debug problems.
>> BTW
>> When running Spark shell the web UI works as excepted.
>>
>>
>>
>> On Tue, Jun 23, 2015 at 6:29 AM, moon soo Lee <mo...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> Really appreciate for sharing the problem.
>>> About the NPE, when do you get the exception? Are you getting the
>>> exception when you trying to access Spark application ui?
>>> If it is, any spark job (like .count(), .collect(), sparksql, etc) runs
>>> without error?
>>>
>>> Thanks!
>>> moon
>>>
>>> On Mon, Jun 22, 2015 at 5:19 AM Ophir Cohen <op...@gmail.com> wrote:
>>>
>>>> OK,
>>>> I think we can seperate the issues.
>>>> Let's start from the NPE. For some reason (is that on perpuse?) when
>>>> Zeppelin starts Spark it seems that it does not open the web ui.
>>>> Trying to access port 4041 (this is the one it started on) yield the
>>>> above NPE with the "WARN [2015-06-22 09:22:37,265] ({qtp622396559-74}
>>>> ServletHandler.java[doHandle]:561) - /jobs/
>>>> java.lang.NullPointerException" message.
>>>>
>>>> Any ideas?
>>>>
>>>> On Mon, Jun 22, 2015 at 12:34 PM, Ophir Cohen <op...@gmail.com> wrote:
>>>>
>>>>> Hi Guys and great work!
>>>>> In our company we decided to relay on Zeppelin as our tool to access
>>>>> and visualize our data.
>>>>> I'm working with Spark 1.3.1 on top of Hadoop yarn 2.4
>>>>>
>>>>> Lately we started to see many NullPointerExcpetion on the Spark
>>>>> interpreter log (see below).
>>>>> Sometimes it does not affect Zeppelin but sometimes we start to see
>>>>> 'can't get status' error caused by broken pipe, this time on the
>>>>> *zeppelin* log file (see below).
>>>>> When the latest error starts - it stacks the Spark interpretor and it
>>>>> enforce us to restart Zeppelin.
>>>>>
>>>>> I couldn't find when and why it's happening, it just happened once
>>>>> awhile...
>>>>> Any help will be more than appreciated!
>>>>> 10x!
>>>>>
>>>>> *The NPE on zeppelin-interpreter-spark log file:*
>>>>> WARN [2015-06-22 09:22:37,265] ({qtp622396559-74}
>>>>> ServletHandler.java[doHandle]:561) - /jobs/
>>>>> java.lang.NullPointerException
>>>>> at
>>>>> org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
>>>>> at
>>>>> org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
>>>>> at
>>>>> org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
>>>>> at
>>>>> org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
>>>>> at
>>>>> org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
>>>>> at
>>>>> org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>>>> at
>>>>> org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>>>> at
>>>>> org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>>>> at org.spark-project.jetty.server.Server.handle(Server.java:370)
>>>>> at
>>>>> org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
>>>>> at
>>>>> org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
>>>>> at
>>>>> org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
>>>>> at
>>>>> org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644)
>>>>> at
>>>>> org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>>>> at
>>>>> org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>>>>> at
>>>>> org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
>>>>> at
>>>>> org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>>>>> at
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>>>> at
>>>>> org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> *The broken pipe on zeppelin log file:*
>>>>> ERROR [2015-06-21 13:10:31,368] ({Thread-120}
>>>>> RemoteScheduler.java[getStatus]:245) - Can't get status information
>>>>> org.apache.thrift.transport.TTransportException:
>>>>> java.net.SocketException: Broken pipe
>>>>> at
>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>>>> at
>>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
>>>>> at
>>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.send_getStatus(RemoteInterpreterService.java:319)
>>>>> at
>>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getStatus(RemoteInterpreterService.java:311)
>>>>> at
>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.getStatus(RemoteScheduler.java:232)
>>>>> at
>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.run(RemoteScheduler.java:183)
>>>>> Caused by: java.net.SocketException: Broken pipe
>>>>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>>>>> at
>>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
>>>>> at
>>>>> java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>>>>> at
>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>> at
>>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>>> at
>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>>>> ... 5 more
>>>>>
>>>>>
>>>>
>>
>