You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by James Srinivasan <ja...@gmail.com> on 2019/09/02 18:39:29 UTC
Re: Any way to diagnose TTransportException errors?
Is there a chance you have differing versions of libthrift on your
classpath? I've seen errors (admittedly not looking like yours) for that
reason?
James
On Sun, 1 Sep 2019 at 00:57, Paul Brenner <pb...@placeiq.com> wrote:
> We are using a recent build post 0.81 so 0.82?
>
> Restarting the interpreter helps in 75% of the cases, but the remaining
> few just won’t recover until we shut everything down and restart all if
> zeppelin. I suspect there is some other process that we aren’t identifying
> that needs killing.
>
>
>
> PAUL BRENNER
>
> Head of Data Science
> pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq
> <https://www.linkedin.com/company/placeiq/>
> [image: Experian Makes Strategic Investment In PlaceIQ]
> <https://www.placeiq.com/2019/08/experian-makes-strategic-investment-in-placeiq/>
> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider]
> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00>
> On Aug 31, 2019, 7:50 PM -0400, Jeff Zhang <zj...@gmail.com>, wrote:
>
> Which version do you use ? And does restarting interpreter work for you ?
>
> Paul Brenner <pb...@placeiq.com> 于2019年8月30日周五 下午11:45写道：
>
> We run into the error pasted below semi-regularly and often the only
> solution is to do a full restart of zeppelin to get the user/notebook back
> up and running. I’m at a loss of where to even look to try to debug this or
> address it. Anyone have any ideas or any way to point me in the right
> direction?
>
> This is what the notebook shows:
> org.apache.thrift.transport.TTransportException at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437) at
> org.apache.zeppelin.scheduler.Job.run(Job.java:188) at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> This is what I pulled from the log:
>
> ERROR [2019-08-30 15:41:56,923] ({pool-2-thread-24} Job.java[run]:190) -
> Job failed
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> ... 11 more
> INFO [2019-08-30 15:41:56,954] ({Exec Default Executor}
> RemoteInterpreterManagedProcess.java[onProcessComplete]:243) - Interpreter
> process exited 0
> DEBUG [2019-08-30 15:41:56,957] ({pool-2-thread-24}
> RemoteScheduler.java[run]:332) - Job ABORT, 20190821-204527_149556857,
> org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> ERROR [2019-08-30 15:41:56,959] ({pool-2-thread-24}
> NotebookServer.java[afterStatusChange]:2308) - Error
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> ... 11 more
> WARN [2019-08-30 15:41:56,959] ({pool-2-thread-24}
> NotebookServer.java[afterStatusChange]:2316) - Job
> 20190821-204527_149556857 is finished, status: ERROR, exception:
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException, result: %text
> org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> PAUL BRENNER
>
> Head of Data Science
> pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq
> <https://www.linkedin.com/company/placeiq/>
> [image: Experian Makes Strategic Investment In PlaceIQ]
> <https://www.placeiq.com/2019/08/experian-makes-strategic-investment-in-placeiq/>
> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider]
> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
Re: Any way to diagnose TTransportException errors?
Posted by Jeff Zhang <zj...@gmail.com>.
Paul,
Do you use 0.8.x ? We have shaded thrift in 0.9, if this is due to thrift,
then it should work in 0.9
James Srinivasan <ja...@gmail.com> 于2019年9月5日周四 上午3:43写道：
> I had to use 0.9.1 due to using Accumulo 1.7 (which requires 0.9.1) in
> some extra JARs I wanted to use with Spark. In the end, I built Zeppelin
> specifying 0.9.1 and it worked for me.
>
> On Wed, 4 Sep 2019 at 16:38, Paul Brenner <pb...@placeiq.com> wrote:
>
>> It looks like we do have two versions:
>>
>> ./zeppelin/spark/spark-dependencies/target/spark-2.4.0/pom.xml:
>> <libthrift.version>0.9.3</libthrift.version>
>>
>> ./zeppelin/pom.xml: <libthrift.version>0.9.2</libthrift.version>
>>
>> I’m always hesitant to rebuild blindly with changes. Did you align your
>> versions to 0.9.3 and have success? Can anyone weigh in on this discrepancy?
>> 11:33 AM <https://placeiq.slack.com/archives/CBZG184MN/p1567611216003200>
>> ./zeppelin/pom.xml: <libthrift.version>0.9.2</libthrift.version>
>>
>>
>>
>> PAUL BRENNER
>>
>> Head of Data Science
>> pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
>> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq
>> <https://www.linkedin.com/company/placeiq/>
>> [image: Experian Makes Strategic Investment In PlaceIQ]
>> <https://www.placeiq.com/2019/08/experian-makes-strategic-investment-in-placeiq/>
>> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider]
>> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00>
>> On Sep 4, 2019, 7:57 AM -0400, James Srinivasan <
>> james.srinivasan@gmail.com>, wrote:
>>
>> Is there a chance you have differing versions of libthrift on your
>> classpath? I've seen errors (admittedly not looking like yours) for that
>> reason?
>>
>> James
>>
>> On Sun, 1 Sep 2019 at 00:57, Paul Brenner <pb...@placeiq.com> wrote:
>>
>> We are using a recent build post 0.81 so 0.82?
>>
>> Restarting the interpreter helps in 75% of the cases, but the remaining
>> few just won’t recover until we shut everything down and restart all if
>> zeppelin. I suspect there is some other process that we aren’t identifying
>> that needs killing.
>>
>>
>>
>> PAUL BRENNER
>>
>> Head of Data Science
>> pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
>> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq
>> <https://www.linkedin.com/company/placeiq/>
>> [image: Experian Makes Strategic Investment In PlaceIQ]
>> <https://www.placeiq.com/2019/08/experian-makes-strategic-investment-in-placeiq/>
>> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider]
>> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00>
>> On Aug 31, 2019, 7:50 PM -0400, Jeff Zhang <zj...@gmail.com>, wrote:
>>
>> Which version do you use ? And does restarting interpreter work for you ?
>>
>> Paul Brenner <pb...@placeiq.com> 于2019年8月30日周五 下午11:45写道：
>>
>> We run into the error pasted below semi-regularly and often the only
>> solution is to do a full restart of zeppelin to get the user/notebook back
>> up and running. I’m at a loss of where to even look to try to debug this or
>> address it. Anyone have any ideas or any way to point me in the right
>> direction?
>>
>> This is what the notebook shows:
>> org.apache.thrift.transport.TTransportException at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at
>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437) at
>> org.apache.zeppelin.scheduler.Job.run(Job.java:188) at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> This is what I pulled from the log:
>>
>> ERROR [2019-08-30 15:41:56,923] ({pool-2-thread-24} Job.java[run]:190) -
>> Job failed
>> java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
>> at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.thrift.transport.TTransportException
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
>> ... 11 more
>> INFO [2019-08-30 15:41:56,954] ({Exec Default Executor}
>> RemoteInterpreterManagedProcess.java[onProcessComplete]:243) - Interpreter
>> process exited 0
>> DEBUG [2019-08-30 15:41:56,957] ({pool-2-thread-24}
>> RemoteScheduler.java[run]:332) - Job ABORT, 20190821-204527_149556857,
>> org.apache.thrift.transport.TTransportException
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
>> at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> ERROR [2019-08-30 15:41:56,959] ({pool-2-thread-24}
>> NotebookServer.java[afterStatusChange]:2308) - Error
>> java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
>> at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.thrift.transport.TTransportException
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
>> ... 11 more
>> WARN [2019-08-30 15:41:56,959] ({pool-2-thread-24}
>> NotebookServer.java[afterStatusChange]:2316) - Job
>> 20190821-204527_149556857 is finished, status: ERROR, exception:
>> java.lang.RuntimeException:
>> org.apache.thrift.transport.TTransportException, result: %text
>> org.apache.thrift.transport.TTransportException
>> at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
>> at
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
>> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
>> at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>> PAUL BRENNER
>>
>> Head of Data Science
>> pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
>> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq
>> <https://www.linkedin.com/company/placeiq/>
>> [image: Experian Makes Strategic Investment In PlaceIQ]
>> <https://www.placeiq.com/2019/08/experian-makes-strategic-investment-in-placeiq/>
>> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider]
>> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00>
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>>
--
Best Regards
Jeff Zhang
Re: Any way to diagnose TTransportException errors?
Posted by Paul Brenner <pb...@placeiq.com>.
Thanks all, I will try 0.9 and report back.
PAUL BRENNER
Head of Data Science
pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
twitter @placeiq linkedin /placeiq
On Sep 5, 2019, 5:52 AM -0400, James Srinivasan <ja...@gmail.com>, wrote:
> I had to use 0.9.1 due to using Accumulo 1.7 (which requires 0.9.1) in some extra JARs I wanted to use with Spark. In the end, I built Zeppelin specifying 0.9.1 and it worked for me.
>
> > On Wed, 4 Sep 2019 at 16:38, Paul Brenner <pb...@placeiq.com> wrote:
> > > It looks like we do have two versions:
> > >
> > > ./zeppelin/spark/spark-dependencies/target/spark-2.4.0/pom.xml: <libthrift.version>0.9.3</libthrift.version>
> > >
> > > ./zeppelin/pom.xml: <libthrift.version>0.9.2</libthrift.version>
> > >
> > > I’m always hesitant to rebuild blindly with changes. Did you align your versions to 0.9.3 and have success? Can anyone weigh in on this discrepancy?
> > > 11:33 AM
> > > ./zeppelin/pom.xml: <libthrift.version>0.9.2</libthrift.version>
> > >
> > >
> > >
> > > PAUL BRENNER
> > > Head of Data Science
> > > pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> > > twitter @placeiq linkedin /placeiq
> > > On Sep 4, 2019, 7:57 AM -0400, James Srinivasan <ja...@gmail.com>, wrote:
> > > > Is there a chance you have differing versions of libthrift on your classpath? I've seen errors (admittedly not looking like yours) for that reason?
> > > >
> > > > James
> > > >
> > > > > On Sun, 1 Sep 2019 at 00:57, Paul Brenner <pb...@placeiq.com> wrote:
> > > > > > We are using a recent build post 0.81 so 0.82?
> > > > > >
> > > > > > Restarting the interpreter helps in 75% of the cases, but the remaining few just won’t recover until we shut everything down and restart all if zeppelin. I suspect there is some other process that we aren’t identifying that needs killing.
> > > > > >
> > > > > >
> > > > > >
> > > > > > PAUL BRENNER
> > > > > > Head of Data Science
> > > > > > pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> > > > > > twitter @placeiq linkedin /placeiq
> > > > > > On Aug 31, 2019, 7:50 PM -0400, Jeff Zhang <zj...@gmail.com>, wrote:
> > > > > > > Which version do you use ? And does restarting interpreter work for you ?
> > > > > > >
> > > > > > > > Paul Brenner <pb...@placeiq.com> 于2019年8月30日周五 下午11:45写道：
> > > > > > > > > We run into the error pasted below semi-regularly and often the only solution is to do a full restart of zeppelin to get the user/notebook back up and running. I’m at a loss of where to even look to try to debug this or address it. Anyone have any ideas or any way to point me in the right direction?
> > > > > > > > >
> > > > > > > > > This is what the notebook shows:
> > > > > > > > > org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437) at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This is what I pulled from the log:
> > > > > > > > >
> > > > > > > > > ERROR [2019-08-30 15:41:56,923] ({pool-2-thread-24} Job.java[run]:190) - Job failed
> > > > > > > > > java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > > > > > > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > > > > > > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > > > > > > > at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > > > > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > > Caused by: org.apache.thrift.transport.TTransportException
> > > > > > > > > at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > > > > > > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > > > > > > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > > > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > > > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > > > > > > > ... 11 more
> > > > > > > > > INFO [2019-08-30 15:41:56,954] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:243) - Interpreter process exited 0
> > > > > > > > > DEBUG [2019-08-30 15:41:56,957] ({pool-2-thread-24} RemoteScheduler.java[run]:332) - Job ABORT, 20190821-204527_149556857, org.apache.thrift.transport.TTransportException
> > > > > > > > > at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > > > > > > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > > > > > > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > > > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > > > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > > > > > > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > > > > > > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > > > > > > > at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > > > > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > >
> > > > > > > > > ERROR [2019-08-30 15:41:56,959] ({pool-2-thread-24} NotebookServer.java[afterStatusChange]:2308) - Error
> > > > > > > > > java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > > > > > > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > > > > > > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > > > > > > > at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > > > > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > > Caused by: org.apache.thrift.transport.TTransportException
> > > > > > > > > at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > > > > > > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > > > > > > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > > > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > > > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > > > > > > > ... 11 more
> > > > > > > > > WARN [2019-08-30 15:41:56,959] ({pool-2-thread-24} NotebookServer.java[afterStatusChange]:2316) - Job 20190821-204527_149556857 is finished, status: ERROR, exception: java.lang.RuntimeException: org.apache.thrift.transport.TTransportException, result: %text org.apache.thrift.transport.TTransportException
> > > > > > > > > at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > > > > > > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > > > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > > > > > > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > > > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > > > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > > > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > > > > > > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > > > > > > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > > > > > > > at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > > > > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > PAUL BRENNER
> > > > > > > > > Head of Data Science
> > > > > > > > > pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> > > > > > > > > twitter @placeiq linkedin /placeiq
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best Regards
> > > > > > >
> > > > > > > Jeff Zhang
Re: Any way to diagnose TTransportException errors?
Posted by James Srinivasan <ja...@gmail.com>.
I had to use 0.9.1 due to using Accumulo 1.7 (which requires 0.9.1) in some
extra JARs I wanted to use with Spark. In the end, I built Zeppelin
specifying 0.9.1 and it worked for me.
On Wed, 4 Sep 2019 at 16:38, Paul Brenner <pb...@placeiq.com> wrote:
> It looks like we do have two versions:
>
> ./zeppelin/spark/spark-dependencies/target/spark-2.4.0/pom.xml:
> <libthrift.version>0.9.3</libthrift.version>
>
> ./zeppelin/pom.xml: <libthrift.version>0.9.2</libthrift.version>
>
> I’m always hesitant to rebuild blindly with changes. Did you align your
> versions to 0.9.3 and have success? Can anyone weigh in on this discrepancy?
> 11:33 AM <https://placeiq.slack.com/archives/CBZG184MN/p1567611216003200>
> ./zeppelin/pom.xml: <libthrift.version>0.9.2</libthrift.version>
>
>
>
> PAUL BRENNER
>
> Head of Data Science
> pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq
> <https://www.linkedin.com/company/placeiq/>
> [image: Experian Makes Strategic Investment In PlaceIQ]
> <https://www.placeiq.com/2019/08/experian-makes-strategic-investment-in-placeiq/>
> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider]
> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00>
> On Sep 4, 2019, 7:57 AM -0400, James Srinivasan <
> james.srinivasan@gmail.com>, wrote:
>
> Is there a chance you have differing versions of libthrift on your
> classpath? I've seen errors (admittedly not looking like yours) for that
> reason?
>
> James
>
> On Sun, 1 Sep 2019 at 00:57, Paul Brenner <pb...@placeiq.com> wrote:
>
> We are using a recent build post 0.81 so 0.82?
>
> Restarting the interpreter helps in 75% of the cases, but the remaining
> few just won’t recover until we shut everything down and restart all if
> zeppelin. I suspect there is some other process that we aren’t identifying
> that needs killing.
>
>
>
> PAUL BRENNER
>
> Head of Data Science
> pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq
> <https://www.linkedin.com/company/placeiq/>
> [image: Experian Makes Strategic Investment In PlaceIQ]
> <https://www.placeiq.com/2019/08/experian-makes-strategic-investment-in-placeiq/>
> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider]
> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00>
> On Aug 31, 2019, 7:50 PM -0400, Jeff Zhang <zj...@gmail.com>, wrote:
>
> Which version do you use ? And does restarting interpreter work for you ?
>
> Paul Brenner <pb...@placeiq.com> 于2019年8月30日周五 下午11:45写道：
>
> We run into the error pasted below semi-regularly and often the only
> solution is to do a full restart of zeppelin to get the user/notebook back
> up and running. I’m at a loss of where to even look to try to debug this or
> address it. Anyone have any ideas or any way to point me in the right
> direction?
>
> This is what the notebook shows:
> org.apache.thrift.transport.TTransportException at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437) at
> org.apache.zeppelin.scheduler.Job.run(Job.java:188) at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> This is what I pulled from the log:
>
> ERROR [2019-08-30 15:41:56,923] ({pool-2-thread-24} Job.java[run]:190) -
> Job failed
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> ... 11 more
> INFO [2019-08-30 15:41:56,954] ({Exec Default Executor}
> RemoteInterpreterManagedProcess.java[onProcessComplete]:243) - Interpreter
> process exited 0
> DEBUG [2019-08-30 15:41:56,957] ({pool-2-thread-24}
> RemoteScheduler.java[run]:332) - Job ABORT, 20190821-204527_149556857,
> org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> ERROR [2019-08-30 15:41:56,959] ({pool-2-thread-24}
> NotebookServer.java[afterStatusChange]:2308) - Error
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> ... 11 more
> WARN [2019-08-30 15:41:56,959] ({pool-2-thread-24}
> NotebookServer.java[afterStatusChange]:2316) - Job
> 20190821-204527_149556857 is finished, status: ERROR, exception:
> java.lang.RuntimeException:
> org.apache.thrift.transport.TTransportException, result: %text
> org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> at
> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> PAUL BRENNER
>
> Head of Data Science
> pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> twitter @placeiq <https://twitter.com/PlaceIQ> linkedin /placeiq
> <https://www.linkedin.com/company/placeiq/>
> [image: Experian Makes Strategic Investment In PlaceIQ]
> <https://www.placeiq.com/2019/08/experian-makes-strategic-investment-in-placeiq/>
> [image: PlaceIQ: Oracle Data Cloud Premier Data Provider]
> <http://go.placeiq.com/WC01DPG0l0N3Gq000000e00>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
Re: Any way to diagnose TTransportException errors?
Posted by Paul Brenner <pb...@placeiq.com>.
It looks like we do have two versions:
./zeppelin/spark/spark-dependencies/target/spark-2.4.0/pom.xml: <libthrift.version>0.9.3</libthrift.version>
./zeppelin/pom.xml: <libthrift.version>0.9.2</libthrift.version>
I’m always hesitant to rebuild blindly with changes. Did you align your versions to 0.9.3 and have success? Can anyone weigh in on this discrepancy?
11:33 AM
./zeppelin/pom.xml: <libthrift.version>0.9.2</libthrift.version>
PAUL BRENNER
Head of Data Science
pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
twitter @placeiq linkedin /placeiq
On Sep 4, 2019, 7:57 AM -0400, James Srinivasan <ja...@gmail.com>, wrote:
> Is there a chance you have differing versions of libthrift on your classpath? I've seen errors (admittedly not looking like yours) for that reason?
>
> James
>
> > On Sun, 1 Sep 2019 at 00:57, Paul Brenner <pb...@placeiq.com> wrote:
> > > We are using a recent build post 0.81 so 0.82?
> > >
> > > Restarting the interpreter helps in 75% of the cases, but the remaining few just won’t recover until we shut everything down and restart all if zeppelin. I suspect there is some other process that we aren’t identifying that needs killing.
> > >
> > >
> > >
> > > PAUL BRENNER
> > > Head of Data Science
> > > pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> > > twitter @placeiq linkedin /placeiq
> > > On Aug 31, 2019, 7:50 PM -0400, Jeff Zhang <zj...@gmail.com>, wrote:
> > > > Which version do you use ? And does restarting interpreter work for you ?
> > > >
> > > > > Paul Brenner <pb...@placeiq.com> 于2019年8月30日周五 下午11:45写道：
> > > > > > We run into the error pasted below semi-regularly and often the only solution is to do a full restart of zeppelin to get the user/notebook back up and running. I’m at a loss of where to even look to try to debug this or address it. Anyone have any ideas or any way to point me in the right direction?
> > > > > >
> > > > > > This is what the notebook shows:
> > > > > > org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437) at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
> > > > > >
> > > > > >
> > > > > > This is what I pulled from the log:
> > > > > >
> > > > > > ERROR [2019-08-30 15:41:56,923] ({pool-2-thread-24} Job.java[run]:190) - Job failed
> > > > > > java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > > > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > > > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > > > > at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > Caused by: org.apache.thrift.transport.TTransportException
> > > > > > at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > > > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > > > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > > > > ... 11 more
> > > > > > INFO [2019-08-30 15:41:56,954] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:243) - Interpreter process exited 0
> > > > > > DEBUG [2019-08-30 15:41:56,957] ({pool-2-thread-24} RemoteScheduler.java[run]:332) - Job ABORT, 20190821-204527_149556857, org.apache.thrift.transport.TTransportException
> > > > > > at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > > > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > > > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > > > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > > > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > > > > at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > >
> > > > > > ERROR [2019-08-30 15:41:56,959] ({pool-2-thread-24} NotebookServer.java[afterStatusChange]:2308) - Error
> > > > > > java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > > > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > > > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > > > > at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > Caused by: org.apache.thrift.transport.TTransportException
> > > > > > at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > > > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > > > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > > > > ... 11 more
> > > > > > WARN [2019-08-30 15:41:56,959] ({pool-2-thread-24} NotebookServer.java[afterStatusChange]:2316) - Job 20190821-204527_149556857 is finished, status: ERROR, exception: java.lang.RuntimeException: org.apache.thrift.transport.TTransportException, result: %text org.apache.thrift.transport.TTransportException
> > > > > > at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> > > > > > at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> > > > > > at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> > > > > > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:274)
> > > > > > at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:258)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:233)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter$4.call(RemoteInterpreter.java:229)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:135)
> > > > > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:228)
> > > > > > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:437)
> > > > > > at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> > > > > > at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> > > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > >
> > > > > >
> > > > > >
> > > > > > PAUL BRENNER
> > > > > > Head of Data Science
> > > > > > pbrenner@placeiq.com | (217) 390-3033 | www.placeiq.com
> > > > > > twitter @placeiq linkedin /placeiq
> > > >
> > > >
> > > > --
> > > > Best Regards
> > > >
> > > > Jeff Zhang