You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by 小野圭二 <on...@gmail.com> on 2018/03/02 05:32:39 UTC
Thrift server for call back will start. Port:
Hi there,
I am so confusing now, let me know why if You know this, please.
The env:
Mac: High Sierra ver.10.13.3
Zep: 0.7.3
Spark: 2.2.0
Java: 1.8
notebook script: %pyspark print 'aaa' (just it for test)
In this env, i got the error in log as
ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
Indeed, before running notebook, RemoteInterpreterManagedProcess.java
prepares a Port for callback to Thrift.
INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) -
Thrift server for callback will start. Port: 63039
I guess, the reason why the error was Zeppelin could not attache to the
Thrift PORT.
And my confusion is, when i run the same module on CentOS, it worked fine.
CentOS: 7
Zep: same
Spark: same
Java: same
notebook script: same
And there was no the Thift port description in the log file. From this
consequence, the fine working was cause of did not use Thrift port on
CentOS.
Do You know actually there is different behavior between Mac and CentOS?
And i am so appreciated if You know the workaround on Mac.
Best,
Ono
Re: Thrift server for call back will start. Port:
Posted by 小野圭二 <on...@gmail.com>.
Combinations: SPARK_HOME set in 'O'->point to external spark , no '-'
->head to internal spark lib
spark interpreter setting 〇 〇 - - 〇 - -
zeppelin -env.sh - - - - 〇 〇 〇
environment parameters - 〇 〇 - 〇 〇 -
Work Fine:〇 NG:× 〇 〇 〇 〇 × × ×
The consequences are zeppelin-env.sh has a top priority. And if you want to
use external Spark on Mac, you should not set SPARK_HOME in
zeppelin-env.sh. Set it as environment parameters or in spark interpreter.
Cheers,
Ono Keiji
2018-03-06 12:23 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
> If you set SPARK_HOME in spark interpreter setting, then please remove
> SPARK_HOME in spark-env.sh, otherwise you do will hit weird issue.
>
>
>
>
> 小野圭二 <on...@gmail.com>于2018年3月6日周二 上午10:56写道：
>
>> Hi Jeff,
>>
>> I did set SPARK_HOME to use external spark at first. And that was fine
>> work on CentOS, but not on Mac.
>> Do you mean try it again?
>>
>> 2018-03-06 11:12 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>
>>> >>> Seems like it is not good chemistry between Spark and Zeppelin on
>>> Mac. From the reason that the consequence of removing the environment
>>> parameters, Zeppelin headed to own spark lib.
>>>
>>> Could you specify SPARK_HOME in your spark interpreter to use external
>>> spark instead of zeppelin's own spark lib ?
>>>
>>>
>>>
>>>
>>> 小野圭二 <on...@gmail.com>于2018年3月6日周二 上午10:06写道：
>>>
>>>> I wonder why, but it seems work fine, and would like to share how to
>>>> fix it.
>>>>
>>>> First of all, this issue happened just on MAC.
>>>> [THE ENV WAS]
>>>> macOS High Sierra v 10.13.3
>>>> cpu core i5
>>>> mem 8GB
>>>> Zeppelin 0.8.0 snapshot
>>>> Spark 2.2.0
>>>> [ISSUE]
>>>> see this previous mails
>>>> [SOLUTION]
>>>> ref. http://apache-zeppelin-users-incubating-mailing-list.
>>>> 75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter-
>>>> process-failed-td4743.html
>>>> According to the above ref, i removed 'SPARK_HOME',
>>>> 'SPARK_SUBMIT_OPTIONS' from my environment parameters and
>>>> conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin,
>>>> that's all.
>>>> [TO DO]
>>>> Seems like it is not good chemistry between Spark and Zeppelin on
>>>> Mac. From the reason that the consequence of removing the environment
>>>> parameters, Zeppelin headed to own spark lib.
>>>> Maybe should write this in the document as caution.
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Ono Keiji
>>>>
>>>>
>>>>
>>>>
>>>> 2018-03-06 9:15 GMT+09:00 小野圭二 <on...@gmail.com>:
>>>>
>>>>> Nothing changed. I guess this property does not effect cause of
>>>>> failing to call the interpreter.
>>>>> i challenged it as log4j.......zeppelin=DEBUG, but no way.
>>>>>
>>>>> 2018-03-05 18:24 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>
>>>>>>
>>>>>> Try to add the following to log4j.properties for more log
>>>>>>
>>>>>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>>>>>>
>>>>>>
>>>>>>
>>>>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午5:13写道：
>>>>>>
>>>>>>> Actually, there was not created the interpreter log file as said
>>>>>>> 'Cannot run interpreter' in the log. And it was run on local machine as
>>>>>>> MAC, i mean stand alone. i stopped every firewall to make it easy. And
>>>>>>> again,this happened just on Mac. On CentOS, it works fine.
>>>>>>> That makes me confusion. :-b
>>>>>>>
>>>>>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>>>
>>>>>>>>
>>>>>>>> Could you check the interpreter log ? The issue of not be able to
>>>>>>>> connecting to thrift port could happen due to network issue, like firewall
>>>>>>>> or you use VPN, etc.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I show the all log on this error.There are 2 type error. The 1 is
>>>>>>>>> at the first run. The 2 is at the second run. Seems like could not
>>>>>>>>> established to Thrift callback port 51348 that was set at the first run.
>>>>>>>>> First of all, does it need to establish the callback port of
>>>>>>>>> Thrift server?
>>>>>>>>> The notebook script was just
>>>>>>>>> %pyspark
>>>>>>>>> print 'aaa'
>>>>>>>>> Any advice appreciate.
>>>>>>>>>
>>>>>>>>> 1.
>>>>>>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>>>>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109)
>>>>>>>>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>>>>>>>>> interpreter.remote.RemoteInterpreter-shared_session INFO
>>>>>>>>> [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) -
>>>>>>>>> run paragraph 20180302-135047_99102973 using pyspark
>>>>>>>>> RemoteInterpreter_org.apache.zeppelin.spark.
>>>>>>>>> PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936]
>>>>>>>>> ({pool-2-thread-2} ManagedInterpreterGroup.java[
>>>>>>>>> getOrCreateInterpreterProcess]:57) - Create InterperterProcess
>>>>>>>>> for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05
>>>>>>>>> 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796)
>>>>>>>>> - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05
>>>>>>>>> 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123)
>>>>>>>>> - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>>>>>>> 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130)
>>>>>>>>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>>>>>>>>> 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205)
>>>>>>>>> - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>>>>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>>>>>>>>> 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>>>>>>>>> -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>>>>>>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) -
>>>>>>>>> Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058]
>>>>>>>>> ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.
>>>>>>>>> interpreter.InterpreterException: Cannot run interpreter at
>>>>>>>>> org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreterManagedProcess.start(
>>>>>>>>> RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.
>>>>>>>>> interpreter.remote.RemoteInterpreter.
>>>>>>>>> getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>>>>>>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>>>>>>>>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>>>>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05
>>>>>>>>> 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219)
>>>>>>>>> - Error org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> Cannot run interpreter at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreterManagedProcess.start(
>>>>>>>>> RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.
>>>>>>>>> interpreter.remote.RemoteInterpreter.
>>>>>>>>> getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>>>>>>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>>>>>>>>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>>>>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05
>>>>>>>>> 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227)
>>>>>>>>> - Job 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>>>>>> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
>>>>>>>>> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving
>>>>>>>>> note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
>>>>>>>>> SchedulerFactory.java[jobFinished]:115) - Job
>>>>>>>>> 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.
>>>>>>>>> interpreter.remote.RemoteInterpreter-shared_session
>>>>>>>>>
>>>>>>>>> 2.
>>>>>>>>> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
>>>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>>>>> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
>>>>>>>>> SchedulerFactory.java[jobStarted]:109) - Job
>>>>>>>>> 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>>>>>>>>> interpreter.remote.RemoteInterpreter-shared_session
>>>>>>>>> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
>>>>>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>>>>>> pyspark RemoteInterpreter_org.apache.zeppelin.spark.
>>>>>>>>> PySparkInterpreter_shared_session
>>>>>>>>> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2}
>>>>>>>>> Job.java[run]:184) - Job failed
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreterProcess.callRemoteFunction(
>>>>>>>>> RemoteInterpreterProcess.java:152)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
>>>>>>>>> RemoteInterpreter.java:126)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.
>>>>>>>>> java:423)
>>>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>>>>>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>>>>>>>>> RemoteScheduler.java:307)
>>>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.
>>>>>>>>> call(Executors.java:511)
>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.
>>>>>>>>> java:180)
>>>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>>>> ThreadPoolExecutor.java:1149)
>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>>>> ThreadPoolExecutor.java:624)
>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>>>> ClientFactory.java:53)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>>>> ClientFactory.java:37)
>>>>>>>>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>>>>>>>>> BasePooledObjectFactory.java:60)
>>>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.create(
>>>>>>>>> GenericObjectPool.java:861)
>>>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>>>>>>> GenericObjectPool.java:435)
>>>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>>>>>>> GenericObjectPool.java:363)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:
>>>>>>>>> 73)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreterProcess.callRemoteFunction(
>>>>>>>>> RemoteInterpreterProcess.java:144)
>>>>>>>>> ... 13 more
>>>>>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>>>> ClientFactory.java:51)
>>>>>>>>> ... 20 more
>>>>>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>>>>>> (Connection refused)
>>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>>>>> at java.net.AbstractPlainSocketImpl.doConnect(
>>>>>>>>> AbstractPlainSocketImpl.java:350)
>>>>>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(
>>>>>>>>> AbstractPlainSocketImpl.java:206)
>>>>>>>>> at java.net.AbstractPlainSocketImpl.connect(
>>>>>>>>> AbstractPlainSocketImpl.java:188)
>>>>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>>>>>> ... 21 more
>>>>>>>>> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
>>>>>>>>> NotebookServer.java[afterStatusChange]:2219) - Error
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreterProcess.callRemoteFunction(
>>>>>>>>> RemoteInterpreterProcess.java:152)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
>>>>>>>>> RemoteInterpreter.java:126)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.
>>>>>>>>> java:423)
>>>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>>>>>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>>>>>>>>> RemoteScheduler.java:307)
>>>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.
>>>>>>>>> call(Executors.java:511)
>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.
>>>>>>>>> java:180)
>>>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>>>> ThreadPoolExecutor.java:1149)
>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>>>> ThreadPoolExecutor.java:624)
>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>>>> ClientFactory.java:53)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>>>> ClientFactory.java:37)
>>>>>>>>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>>>>>>>>> BasePooledObjectFactory.java:60)
>>>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.create(
>>>>>>>>> GenericObjectPool.java:861)
>>>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>>>>>>> GenericObjectPool.java:435)
>>>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>>>>>>> GenericObjectPool.java:363)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:
>>>>>>>>> 73)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>>>> RemoteInterpreterProcess.callRemoteFunction(
>>>>>>>>> RemoteInterpreterProcess.java:144)
>>>>>>>>> ... 13 more
>>>>>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>>>> ClientFactory.java:51)
>>>>>>>>> ... 20 more
>>>>>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>>>>>> (Connection refused)
>>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>>>>> at java.net.AbstractPlainSocketImpl.doConnect(
>>>>>>>>> AbstractPlainSocketImpl.java:350)
>>>>>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(
>>>>>>>>> AbstractPlainSocketImpl.java:206)
>>>>>>>>> at java.net.AbstractPlainSocketImpl.connect(
>>>>>>>>> AbstractPlainSocketImpl.java:188)
>>>>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>>>>>> ... 21 more
>>>>>>>>> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
>>>>>>>>> NotebookServer.java[afterStatusChange]:2227) - Job
>>>>>>>>> 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused), result:
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
>>>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>>>>> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
>>>>>>>>> SchedulerFactory.java[jobFinished]:115) - Job
>>>>>>>>> 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.
>>>>>>>>> interpreter.remote.RemoteInterpreter-shared_session
>>>>>>>>>
>>>>>>>>> 2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Hi Jeff,
>>>>>>>>>>
>>>>>>>>>> No one graved 63039 so far on Mac.
>>>>>>>>>> Yes, this issue has been reproduced as changing the port NO
>>>>>>>>>> randomly.
>>>>>>>>>>
>>>>>>>>>> Ono
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It is possible that 63039 is used by other processes, as Mac
>>>>>>>>>>> can not detect whether this port is used or not from my experience.
>>>>>>>>>>>
>>>>>>>>>>> And can you reproduce this issue consistently ? Because each
>>>>>>>>>>> time zeppelin will chose a random port as long as you don't specify the
>>>>>>>>>>> portRange, so if you reproduce it consistently, it might due to other
>>>>>>>>>>> issues.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>>>>>>>>>
>>>>>>>>>>>> Hi there,
>>>>>>>>>>>>
>>>>>>>>>>>> I am so confusing now, let me know why if You know this,
>>>>>>>>>>>> please.
>>>>>>>>>>>>
>>>>>>>>>>>> The env:
>>>>>>>>>>>> Mac: High Sierra ver.10.13.3
>>>>>>>>>>>> Zep: 0.7.3
>>>>>>>>>>>> Spark: 2.2.0
>>>>>>>>>>>> Java: 1.8
>>>>>>>>>>>> notebook script: %pyspark print 'aaa' (just
>>>>>>>>>>>> it for test)
>>>>>>>>>>>>
>>>>>>>>>>>> In this env, i got the error in log as
>>>>>>>>>>>>
>>>>>>>>>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>>>>>>>>>>> prepares a Port for callback to Thrift.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>>>>>>>>>>>> - Thrift server for callback will start. Port: 63039
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I guess, the reason why the error was Zeppelin could not
>>>>>>>>>>>> attache to the Thrift PORT.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And my confusion is, when i run the same module on CentOS, it
>>>>>>>>>>>> worked fine.
>>>>>>>>>>>> CentOS: 7
>>>>>>>>>>>> Zep: same
>>>>>>>>>>>> Spark: same
>>>>>>>>>>>> Java: same
>>>>>>>>>>>> notebook script: same
>>>>>>>>>>>>
>>>>>>>>>>>> And there was no the Thift port description in the log file.
>>>>>>>>>>>> From this consequence, the fine working was cause of did not use Thrift
>>>>>>>>>>>> port on CentOS.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Do You know actually there is different behavior between Mac
>>>>>>>>>>>> and CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Ono
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>
Re: Thrift server for call back will start. Port:
Posted by Jeff Zhang <zj...@gmail.com>.
If you set SPARK_HOME in spark interpreter setting, then please remove
SPARK_HOME in spark-env.sh, otherwise you do will hit weird issue.
小野圭二 <on...@gmail.com>于2018年3月6日周二 上午10:56写道：
> Hi Jeff,
>
> I did set SPARK_HOME to use external spark at first. And that was fine
> work on CentOS, but not on Mac.
> Do you mean try it again?
>
> 2018-03-06 11:12 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
>> >>> Seems like it is not good chemistry between Spark and Zeppelin on
>> Mac. From the reason that the consequence of removing the environment
>> parameters, Zeppelin headed to own spark lib.
>>
>> Could you specify SPARK_HOME in your spark interpreter to use external
>> spark instead of zeppelin's own spark lib ?
>>
>>
>>
>>
>> 小野圭二 <on...@gmail.com>于2018年3月6日周二 上午10:06写道：
>>
>>> I wonder why, but it seems work fine, and would like to share how to fix
>>> it.
>>>
>>> First of all, this issue happened just on MAC.
>>> [THE ENV WAS]
>>> macOS High Sierra v 10.13.3
>>> cpu core i5
>>> mem 8GB
>>> Zeppelin 0.8.0 snapshot
>>> Spark 2.2.0
>>> [ISSUE]
>>> see this previous mails
>>> [SOLUTION]
>>> ref.
>>> http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter-process-failed-td4743.html
>>> According to the above ref, i removed 'SPARK_HOME',
>>> 'SPARK_SUBMIT_OPTIONS' from my environment parameters and
>>> conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin,
>>> that's all.
>>> [TO DO]
>>> Seems like it is not good chemistry between Spark and Zeppelin on
>>> Mac. From the reason that the consequence of removing the environment
>>> parameters, Zeppelin headed to own spark lib.
>>> Maybe should write this in the document as caution.
>>>
>>>
>>> Cheers,
>>>
>>> Ono Keiji
>>>
>>>
>>>
>>>
>>> 2018-03-06 9:15 GMT+09:00 小野圭二 <on...@gmail.com>:
>>>
>>>> Nothing changed. I guess this property does not effect cause of failing
>>>> to call the interpreter.
>>>> i challenged it as log4j.......zeppelin=DEBUG, but no way.
>>>>
>>>> 2018-03-05 18:24 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>
>>>>>
>>>>> Try to add the following to log4j.properties for more log
>>>>>
>>>>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>>>>>
>>>>>
>>>>>
>>>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午5:13写道：
>>>>>
>>>>>> Actually, there was not created the interpreter log file as said
>>>>>> 'Cannot run interpreter' in the log. And it was run on local machine as
>>>>>> MAC, i mean stand alone. i stopped every firewall to make it easy. And
>>>>>> again,this happened just on Mac. On CentOS, it works fine.
>>>>>> That makes me confusion. :-b
>>>>>>
>>>>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>>
>>>>>>>
>>>>>>> Could you check the interpreter log ? The issue of not be able to
>>>>>>> connecting to thrift port could happen due to network issue, like firewall
>>>>>>> or you use VPN, etc.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I show the all log on this error.There are 2 type error. The 1 is
>>>>>>>> at the first run. The 2 is at the second run. Seems like could not
>>>>>>>> established to Thrift callback port 51348 that was set at the first run.
>>>>>>>> First of all, does it need to establish the callback port of Thrift
>>>>>>>> server?
>>>>>>>> The notebook script was just
>>>>>>>> %pyspark
>>>>>>>> print 'aaa'
>>>>>>>> Any advice appreciate.
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
>>>>>>>> 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) -
>>>>>>>> Job 20180302-135047_99102973 started by scheduler
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>>>>> pyspark
>>>>>>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>>>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
>>>>>>>> InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
>>>>>>>> [2018-03-05 13:11:14,010] ({pool-2-thread-2}
>>>>>>>> InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not
>>>>>>>> found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
>>>>>>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
>>>>>>>> Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>>>>>> 13:11:14,023] ({pool-2-thread-2}
>>>>>>>> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
>>>>>>>> callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049]
>>>>>>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run
>>>>>>>> interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>>>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10,
>>>>>>>> -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g,
>>>>>>>> spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>>>>>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter
>>>>>>>> process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2}
>>>>>>>> Job.java[run]:184) - Job failed
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>>>>> interpreter at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>>>>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
>>>>>>>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>>>>> interpreter at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>>>>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075]
>>>>>>>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job
>>>>>>>> 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>>>>> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
>>>>>>>> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>>>> INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
>>>>>>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>>>>>>> finished by scheduler
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>>>>>
>>>>>>>> 2.
>>>>>>>> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
>>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>>>> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
>>>>>>>> SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973
>>>>>>>> started by scheduler
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>>>>> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
>>>>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>>>>> pyspark
>>>>>>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>>>>>> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2}
>>>>>>>> Job.java[run]:184) - Job failed
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>>>> at
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>>>>>>>> at
>>>>>>>> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>>>>>>>> at
>>>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>>>>>>>> at
>>>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>>>>>>>> at
>>>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>>>>>> ... 13 more
>>>>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>>>>>>>> ... 20 more
>>>>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>>>>> (Connection refused)
>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>>>> at java.net
>>>>>>>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>>>>>>> at java.net
>>>>>>>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>>>>>>> at java.net
>>>>>>>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>>>>> ... 21 more
>>>>>>>> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
>>>>>>>> NotebookServer.java[afterStatusChange]:2219) - Error
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>>>> at
>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>>>>>>>> at
>>>>>>>> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>>>>>>>> at
>>>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>>>>>>>> at
>>>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>>>>>>>> at
>>>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>>>>>> ... 13 more
>>>>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>>>>> at
>>>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>>>>>>>> ... 20 more
>>>>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>>>>> (Connection refused)
>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>>>> at java.net
>>>>>>>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>>>>>>> at java.net
>>>>>>>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>>>>>>> at java.net
>>>>>>>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>>>>> ... 21 more
>>>>>>>> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
>>>>>>>> NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973
>>>>>>>> is finished, status: ERROR, exception:
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused (Connection refused), result:
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
>>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>>>> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
>>>>>>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>>>>>>> finished by scheduler
>>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>>>>>
>>>>>>>> 2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
>>>>>>>>
>>>>>>>>> Hi Jeff,
>>>>>>>>>
>>>>>>>>> No one graved 63039 so far on Mac.
>>>>>>>>> Yes, this issue has been reproduced as changing the port NO
>>>>>>>>> randomly.
>>>>>>>>>
>>>>>>>>> Ono
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It is possible that 63039 is used by other processes, as Mac can
>>>>>>>>>> not detect whether this port is used or not from my experience.
>>>>>>>>>>
>>>>>>>>>> And can you reproduce this issue consistently ? Because each time
>>>>>>>>>> zeppelin will chose a random port as long as you don't specify the
>>>>>>>>>> portRange, so if you reproduce it consistently, it might due to other
>>>>>>>>>> issues.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>>>>>>>>
>>>>>>>>>>> Hi there,
>>>>>>>>>>>
>>>>>>>>>>> I am so confusing now, let me know why if You know this, please.
>>>>>>>>>>>
>>>>>>>>>>> The env:
>>>>>>>>>>> Mac: High Sierra ver.10.13.3
>>>>>>>>>>> Zep: 0.7.3
>>>>>>>>>>> Spark: 2.2.0
>>>>>>>>>>> Java: 1.8
>>>>>>>>>>> notebook script: %pyspark print 'aaa' (just
>>>>>>>>>>> it for test)
>>>>>>>>>>>
>>>>>>>>>>> In this env, i got the error in log as
>>>>>>>>>>>
>>>>>>>>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>>>>>>>>
>>>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Indeed, before running notebook,
>>>>>>>>>>> RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> INFO ({pool-2-thread-1}
>>>>>>>>>>> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
>>>>>>>>>>> callback will start. Port: 63039
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I guess, the reason why the error was Zeppelin could not attache
>>>>>>>>>>> to the Thrift PORT.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> And my confusion is, when i run the same module on CentOS, it
>>>>>>>>>>> worked fine.
>>>>>>>>>>> CentOS: 7
>>>>>>>>>>> Zep: same
>>>>>>>>>>> Spark: same
>>>>>>>>>>> Java: same
>>>>>>>>>>> notebook script: same
>>>>>>>>>>>
>>>>>>>>>>> And there was no the Thift port description in the log file.
>>>>>>>>>>> From this consequence, the fine working was cause of did not use Thrift
>>>>>>>>>>> port on CentOS.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Do You know actually there is different behavior between Mac and
>>>>>>>>>>> CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Ono
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>
Re: Thrift server for call back will start. Port:
Posted by 小野圭二 <on...@gmail.com>.
Hi Jeff,
I did set SPARK_HOME to use external spark at first. And that was fine work
on CentOS, but not on Mac.
Do you mean try it again?
2018-03-06 11:12 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
> >>> Seems like it is not good chemistry between Spark and Zeppelin on
> Mac. From the reason that the consequence of removing the environment
> parameters, Zeppelin headed to own spark lib.
>
> Could you specify SPARK_HOME in your spark interpreter to use external
> spark instead of zeppelin's own spark lib ?
>
>
>
>
> 小野圭二 <on...@gmail.com>于2018年3月6日周二 上午10:06写道：
>
>> I wonder why, but it seems work fine, and would like to share how to fix
>> it.
>>
>> First of all, this issue happened just on MAC.
>> [THE ENV WAS]
>> macOS High Sierra v 10.13.3
>> cpu core i5
>> mem 8GB
>> Zeppelin 0.8.0 snapshot
>> Spark 2.2.0
>> [ISSUE]
>> see this previous mails
>> [SOLUTION]
>> ref. http://apache-zeppelin-users-incubating-mailing-list.
>> 75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter-
>> process-failed-td4743.html
>> According to the above ref, i removed 'SPARK_HOME',
>> 'SPARK_SUBMIT_OPTIONS' from my environment parameters and
>> conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin,
>> that's all.
>> [TO DO]
>> Seems like it is not good chemistry between Spark and Zeppelin on Mac.
>> From the reason that the consequence of removing the environment
>> parameters, Zeppelin headed to own spark lib.
>> Maybe should write this in the document as caution.
>>
>>
>> Cheers,
>>
>> Ono Keiji
>>
>>
>>
>>
>> 2018-03-06 9:15 GMT+09:00 小野圭二 <on...@gmail.com>:
>>
>>> Nothing changed. I guess this property does not effect cause of failing
>>> to call the interpreter.
>>> i challenged it as log4j.......zeppelin=DEBUG, but no way.
>>>
>>> 2018-03-05 18:24 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>
>>>>
>>>> Try to add the following to log4j.properties for more log
>>>>
>>>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>>>>
>>>>
>>>>
>>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午5:13写道：
>>>>
>>>>> Actually, there was not created the interpreter log file as said
>>>>> 'Cannot run interpreter' in the log. And it was run on local machine as
>>>>> MAC, i mean stand alone. i stopped every firewall to make it easy. And
>>>>> again,this happened just on Mac. On CentOS, it works fine.
>>>>> That makes me confusion. :-b
>>>>>
>>>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>
>>>>>>
>>>>>> Could you check the interpreter log ? The issue of not be able to
>>>>>> connecting to thrift port could happen due to network issue, like firewall
>>>>>> or you use VPN, etc.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>>>>> the first run. The 2 is at the second run. Seems like could not established
>>>>>>> to Thrift callback port 51348 that was set at the first run.
>>>>>>> First of all, does it need to establish the callback port of Thrift
>>>>>>> server?
>>>>>>> The notebook script was just
>>>>>>> %pyspark
>>>>>>> print 'aaa'
>>>>>>> Any advice appreciate.
>>>>>>>
>>>>>>> 1.
>>>>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109)
>>>>>>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>>>>>>> interpreter.remote.RemoteInterpreter-shared_session INFO
>>>>>>> [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) -
>>>>>>> run paragraph 20180302-135047_99102973 using pyspark
>>>>>>> RemoteInterpreter_org.apache.zeppelin.spark.
>>>>>>> PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936]
>>>>>>> ({pool-2-thread-2} ManagedInterpreterGroup.java[
>>>>>>> getOrCreateInterpreterProcess]:57) - Create InterperterProcess for
>>>>>>> InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010]
>>>>>>> ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796)
>>>>>>> - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05
>>>>>>> 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123)
>>>>>>> - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>>>>> 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130)
>>>>>>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>>>>>>> 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205)
>>>>>>> - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>>>>>>> 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>>>>>>> -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>>>>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) -
>>>>>>> Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058]
>>>>>>> ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.
>>>>>>> interpreter.InterpreterException: Cannot run interpreter at
>>>>>>> org.apache.zeppelin.interpreter.remote.
>>>>>>> RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>>>> getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>>>>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>>>>>>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05
>>>>>>> 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219)
>>>>>>> - Error org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> Cannot run interpreter at org.apache.zeppelin.interpreter.remote.
>>>>>>> RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>>>> getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>>>>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>>>>>>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05
>>>>>>> 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227)
>>>>>>> - Job 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>>>> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
>>>>>>> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving
>>>>>>> note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
>>>>>>> SchedulerFactory.java[jobFinished]:115) - Job
>>>>>>> 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.
>>>>>>> interpreter.remote.RemoteInterpreter-shared_session
>>>>>>>
>>>>>>> 2.
>>>>>>> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>>> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
>>>>>>> SchedulerFactory.java[jobStarted]:109) - Job
>>>>>>> 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>>>>>>> interpreter.remote.RemoteInterpreter-shared_session
>>>>>>> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
>>>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>>>> pyspark RemoteInterpreter_org.apache.zeppelin.spark.
>>>>>>> PySparkInterpreter_shared_session
>>>>>>> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2}
>>>>>>> Job.java[run]:184) - Job failed
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>>>>> callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>> RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
>>>>>>> RemoteInterpreter.java:126)
>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>>>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>>>>>>> RemoteScheduler.java:307)
>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:511)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1149)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:624)
>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>> ClientFactory.java:53)
>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>> ClientFactory.java:37)
>>>>>>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>>>>>>> BasePooledObjectFactory.java:60)
>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.create(
>>>>>>> GenericObjectPool.java:861)
>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>>>>> GenericObjectPool.java:435)
>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>>>>> GenericObjectPool.java:363)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>>>>> getClient(RemoteInterpreterProcess.java:73)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>>>>> callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>>>>> ... 13 more
>>>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>> ClientFactory.java:51)
>>>>>>> ... 20 more
>>>>>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>>>>>> refused)
>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>>> at java.net.AbstractPlainSocketImpl.doConnect(
>>>>>>> AbstractPlainSocketImpl.java:350)
>>>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(
>>>>>>> AbstractPlainSocketImpl.java:206)
>>>>>>> at java.net.AbstractPlainSocketImpl.connect(
>>>>>>> AbstractPlainSocketImpl.java:188)
>>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>>>> ... 21 more
>>>>>>> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
>>>>>>> NotebookServer.java[afterStatusChange]:2219) - Error
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>>>>> callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>> RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
>>>>>>> RemoteInterpreter.java:126)
>>>>>>> at org.apache.zeppelin.interpreter.remote.
>>>>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>>>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>>>>>>> RemoteScheduler.java:307)
>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:511)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1149)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:624)
>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>> ClientFactory.java:53)
>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>> ClientFactory.java:37)
>>>>>>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>>>>>>> BasePooledObjectFactory.java:60)
>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.create(
>>>>>>> GenericObjectPool.java:861)
>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>>>>> GenericObjectPool.java:435)
>>>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>>>>> GenericObjectPool.java:363)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>>>>> getClient(RemoteInterpreterProcess.java:73)
>>>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>>>>> callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>>>>> ... 13 more
>>>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>>>> ClientFactory.java:51)
>>>>>>> ... 20 more
>>>>>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>>>>>> refused)
>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>>> at java.net.AbstractPlainSocketImpl.doConnect(
>>>>>>> AbstractPlainSocketImpl.java:350)
>>>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(
>>>>>>> AbstractPlainSocketImpl.java:206)
>>>>>>> at java.net.AbstractPlainSocketImpl.connect(
>>>>>>> AbstractPlainSocketImpl.java:188)
>>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>>>> ... 21 more
>>>>>>> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
>>>>>>> NotebookServer.java[afterStatusChange]:2227) - Job
>>>>>>> 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused (Connection refused), result:
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>>> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
>>>>>>> SchedulerFactory.java[jobFinished]:115) - Job
>>>>>>> 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.
>>>>>>> interpreter.remote.RemoteInterpreter-shared_session
>>>>>>>
>>>>>>> 2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
>>>>>>>
>>>>>>>> Hi Jeff,
>>>>>>>>
>>>>>>>> No one graved 63039 so far on Mac.
>>>>>>>> Yes, this issue has been reproduced as changing the port NO
>>>>>>>> randomly.
>>>>>>>>
>>>>>>>> Ono
>>>>>>>>
>>>>>>>>
>>>>>>>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> It is possible that 63039 is used by other processes, as Mac can
>>>>>>>>> not detect whether this port is used or not from my experience.
>>>>>>>>>
>>>>>>>>> And can you reproduce this issue consistently ? Because each time
>>>>>>>>> zeppelin will chose a random port as long as you don't specify the
>>>>>>>>> portRange, so if you reproduce it consistently, it might due to other
>>>>>>>>> issues.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>>>>>>>
>>>>>>>>>> Hi there,
>>>>>>>>>>
>>>>>>>>>> I am so confusing now, let me know why if You know this, please.
>>>>>>>>>>
>>>>>>>>>> The env:
>>>>>>>>>> Mac: High Sierra ver.10.13.3
>>>>>>>>>> Zep: 0.7.3
>>>>>>>>>> Spark: 2.2.0
>>>>>>>>>> Java: 1.8
>>>>>>>>>> notebook script: %pyspark print 'aaa' (just it
>>>>>>>>>> for test)
>>>>>>>>>>
>>>>>>>>>> In this env, i got the error in log as
>>>>>>>>>>
>>>>>>>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>>>>>>>
>>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>>>>>>>>> prepares a Port for callback to Thrift.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>>>>>>>>>> - Thrift server for callback will start. Port: 63039
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I guess, the reason why the error was Zeppelin could not attache
>>>>>>>>>> to the Thrift PORT.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And my confusion is, when i run the same module on CentOS, it
>>>>>>>>>> worked fine.
>>>>>>>>>> CentOS: 7
>>>>>>>>>> Zep: same
>>>>>>>>>> Spark: same
>>>>>>>>>> Java: same
>>>>>>>>>> notebook script: same
>>>>>>>>>>
>>>>>>>>>> And there was no the Thift port description in the log file. From
>>>>>>>>>> this consequence, the fine working was cause of did not use Thrift port on
>>>>>>>>>> CentOS.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Do You know actually there is different behavior between Mac and
>>>>>>>>>> CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Ono
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>
>>
Re: Thrift server for call back will start. Port:
Posted by Jeff Zhang <zj...@gmail.com>.
>>> Seems like it is not good chemistry between Spark and Zeppelin on Mac.
From the reason that the consequence of removing the environment
parameters, Zeppelin headed to own spark lib.
Could you specify SPARK_HOME in your spark interpreter to use external
spark instead of zeppelin's own spark lib ?
小野圭二 <on...@gmail.com>于2018年3月6日周二 上午10:06写道：
> I wonder why, but it seems work fine, and would like to share how to fix
> it.
>
> First of all, this issue happened just on MAC.
> [THE ENV WAS]
> macOS High Sierra v 10.13.3
> cpu core i5
> mem 8GB
> Zeppelin 0.8.0 snapshot
> Spark 2.2.0
> [ISSUE]
> see this previous mails
> [SOLUTION]
> ref.
> http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter-process-failed-td4743.html
> According to the above ref, i removed 'SPARK_HOME',
> 'SPARK_SUBMIT_OPTIONS' from my environment parameters and
> conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin,
> that's all.
> [TO DO]
> Seems like it is not good chemistry between Spark and Zeppelin on Mac.
> From the reason that the consequence of removing the environment
> parameters, Zeppelin headed to own spark lib.
> Maybe should write this in the document as caution.
>
>
> Cheers,
>
> Ono Keiji
>
>
>
>
> 2018-03-06 9:15 GMT+09:00 小野圭二 <on...@gmail.com>:
>
>> Nothing changed. I guess this property does not effect cause of failing
>> to call the interpreter.
>> i challenged it as log4j.......zeppelin=DEBUG, but no way.
>>
>> 2018-03-05 18:24 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>
>>>
>>> Try to add the following to log4j.properties for more log
>>>
>>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>>>
>>>
>>>
>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午5:13写道：
>>>
>>>> Actually, there was not created the interpreter log file as said
>>>> 'Cannot run interpreter' in the log. And it was run on local machine as
>>>> MAC, i mean stand alone. i stopped every firewall to make it easy. And
>>>> again,this happened just on Mac. On CentOS, it works fine.
>>>> That makes me confusion. :-b
>>>>
>>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>
>>>>>
>>>>> Could you check the interpreter log ? The issue of not be able to
>>>>> connecting to thrift port could happen due to network issue, like firewall
>>>>> or you use VPN, etc.
>>>>>
>>>>>
>>>>>
>>>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>>>> the first run. The 2 is at the second run. Seems like could not established
>>>>>> to Thrift callback port 51348 that was set at the first run.
>>>>>> First of all, does it need to establish the callback port of Thrift
>>>>>> server?
>>>>>> The notebook script was just
>>>>>> %pyspark
>>>>>> print 'aaa'
>>>>>> Any advice appreciate.
>>>>>>
>>>>>> 1.
>>>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
>>>>>> 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) -
>>>>>> Job 20180302-135047_99102973 started by scheduler
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>>> pyspark
>>>>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
>>>>>> InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
>>>>>> [2018-03-05 13:11:14,010] ({pool-2-thread-2}
>>>>>> InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not
>>>>>> found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
>>>>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
>>>>>> Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>>>> 13:11:14,023] ({pool-2-thread-2}
>>>>>> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
>>>>>> callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049]
>>>>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run
>>>>>> interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10,
>>>>>> -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g,
>>>>>> spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>>>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter
>>>>>> process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2}
>>>>>> Job.java[run]:184) - Job failed
>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>>> interpreter at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
>>>>>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>>> interpreter at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075]
>>>>>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job
>>>>>> 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>>> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
>>>>>> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>> INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
>>>>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>>>>> finished by scheduler
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>>>
>>>>>> 2.
>>>>>> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
>>>>>> SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973
>>>>>> started by scheduler
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>>> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
>>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>>> pyspark
>>>>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>>>> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184)
>>>>>> - Job failed
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>>> at
>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>>>>>> at
>>>>>> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>>>>>> at
>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>>>>>> at
>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>>>>>> at
>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>>>> ... 13 more
>>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>>>>>> ... 20 more
>>>>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>>>>> refused)
>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>> at java.net
>>>>>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>>>>> at java.net
>>>>>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>>>>> at java.net
>>>>>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>>> ... 21 more
>>>>>> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
>>>>>> NotebookServer.java[afterStatusChange]:2219) - Error
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>>> at
>>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>>>>>> at
>>>>>> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>>>>>> at
>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>>>>>> at
>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>>>>>> at
>>>>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>>>> ... 13 more
>>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>>> at
>>>>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>>>>>> ... 20 more
>>>>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>>>>> refused)
>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>> at java.net
>>>>>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>>>>> at java.net
>>>>>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>>>>> at java.net
>>>>>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>>> ... 21 more
>>>>>> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
>>>>>> NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973
>>>>>> is finished, status: ERROR, exception:
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused), result:
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>>> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
>>>>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>>>>> finished by scheduler
>>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>>>
>>>>>> 2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
>>>>>>
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> No one graved 63039 so far on Mac.
>>>>>>> Yes, this issue has been reproduced as changing the port NO randomly.
>>>>>>>
>>>>>>> Ono
>>>>>>>
>>>>>>>
>>>>>>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>>>
>>>>>>>>
>>>>>>>> It is possible that 63039 is used by other processes, as Mac can
>>>>>>>> not detect whether this port is used or not from my experience.
>>>>>>>>
>>>>>>>> And can you reproduce this issue consistently ? Because each time
>>>>>>>> zeppelin will chose a random port as long as you don't specify the
>>>>>>>> portRange, so if you reproduce it consistently, it might due to other
>>>>>>>> issues.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>>>>>>
>>>>>>>>> Hi there,
>>>>>>>>>
>>>>>>>>> I am so confusing now, let me know why if You know this, please.
>>>>>>>>>
>>>>>>>>> The env:
>>>>>>>>> Mac: High Sierra ver.10.13.3
>>>>>>>>> Zep: 0.7.3
>>>>>>>>> Spark: 2.2.0
>>>>>>>>> Java: 1.8
>>>>>>>>> notebook script: %pyspark print 'aaa' (just it
>>>>>>>>> for test)
>>>>>>>>>
>>>>>>>>> In this env, i got the error in log as
>>>>>>>>>
>>>>>>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>>>>>>
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Indeed, before running notebook,
>>>>>>>>> RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> INFO ({pool-2-thread-1}
>>>>>>>>> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
>>>>>>>>> callback will start. Port: 63039
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I guess, the reason why the error was Zeppelin could not attache
>>>>>>>>> to the Thrift PORT.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> And my confusion is, when i run the same module on CentOS, it
>>>>>>>>> worked fine.
>>>>>>>>> CentOS: 7
>>>>>>>>> Zep: same
>>>>>>>>> Spark: same
>>>>>>>>> Java: same
>>>>>>>>> notebook script: same
>>>>>>>>>
>>>>>>>>> And there was no the Thift port description in the log file. From
>>>>>>>>> this consequence, the fine working was cause of did not use Thrift port on
>>>>>>>>> CentOS.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Do You know actually there is different behavior between Mac and
>>>>>>>>> CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Ono
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
>
Re: Thrift server for call back will start. Port:
Posted by 小野圭二 <on...@gmail.com>.
I wonder why, but it seems work fine, and would like to share how to fix it.
First of all, this issue happened just on MAC.
[THE ENV WAS]
macOS High Sierra v 10.13.3
cpu core i5
mem 8GB
Zeppelin 0.8.0 snapshot
Spark 2.2.0
[ISSUE]
see this previous mails
[SOLUTION]
ref.
http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter-process-failed-td4743.html
According to the above ref, i removed 'SPARK_HOME',
'SPARK_SUBMIT_OPTIONS' from my environment parameters and
conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin,
that's all.
[TO DO]
Seems like it is not good chemistry between Spark and Zeppelin on Mac.
From the reason that the consequence of removing the environment
parameters, Zeppelin headed to own spark lib.
Maybe should write this in the document as caution.
Cheers,
Ono Keiji
2018-03-06 9:15 GMT+09:00 小野圭二 <on...@gmail.com>:
> Nothing changed. I guess this property does not effect cause of failing to
> call the interpreter.
> i challenged it as log4j.......zeppelin=DEBUG, but no way.
>
> 2018-03-05 18:24 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
>>
>> Try to add the following to log4j.properties for more log
>>
>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>>
>>
>>
>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午5:13写道：
>>
>>> Actually, there was not created the interpreter log file as said 'Cannot
>>> run interpreter' in the log. And it was run on local machine as MAC, i mean
>>> stand alone. i stopped every firewall to make it easy. And again,this
>>> happened just on Mac. On CentOS, it works fine.
>>> That makes me confusion. :-b
>>>
>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>
>>>>
>>>> Could you check the interpreter log ? The issue of not be able to
>>>> connecting to thrift port could happen due to network issue, like firewall
>>>> or you use VPN, etc.
>>>>
>>>>
>>>>
>>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>>>
>>>>> Hi,
>>>>>
>>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>>> the first run. The 2 is at the second run. Seems like could not established
>>>>> to Thrift callback port 51348 that was set at the first run.
>>>>> First of all, does it need to establish the callback port of Thrift
>>>>> server?
>>>>> The notebook script was just
>>>>> %pyspark
>>>>> print 'aaa'
>>>>> Any advice appreciate.
>>>>>
>>>>> 1.
>>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109)
>>>>> - Job 20180302-135047_99102973 started by scheduler
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>> pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) -
>>>>> Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process
>>>>> WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2}
>>>>> InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip
>>>>> is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
>>>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
>>>>> Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>>> 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130)
>>>>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>>>>> 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205)
>>>>> - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>>>>> 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>>>>> -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) -
>>>>> Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058]
>>>>> ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>> interpreter at org.apache.zeppelin.interprete
>>>>> r.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.get
>>>>> OrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> internal_create(RemoteInterpreter.java:153) at
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> open(RemoteInterpreter.java:126) at org.apache.zeppelin.interprete
>>>>> r.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180) at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>>> tureTask.run(ScheduledThreadPoolExecutor.java:293) at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05
>>>>> 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219)
>>>>> - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot
>>>>> run interpreter at org.apache.zeppelin.interprete
>>>>> r.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.get
>>>>> OrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> internal_create(RemoteInterpreter.java:153) at
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> open(RemoteInterpreter.java:126) at org.apache.zeppelin.interprete
>>>>> r.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180) at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>>> tureTask.run(ScheduledThreadPoolExecutor.java:293) at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05
>>>>> 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227)
>>>>> - Job 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>>> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
>>>>> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving
>>>>> note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
>>>>> SchedulerFactory.java[jobFinished]:115) - Job
>>>>> 20180302-135047_99102973 finished by scheduler
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-
>>>>> shared_session
>>>>>
>>>>> 2.
>>>>> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
>>>>> SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973
>>>>> started by scheduler org.apache.zeppelin.interprete
>>>>> r.remote.RemoteInterpreter-shared_session
>>>>> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>> pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpret
>>>>> er_shared_session
>>>>> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184)
>>>>> - Job failed
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.thrift.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProc
>>>>> ess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> internal_create(RemoteInterpreter.java:154)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> open(RemoteInterpreter.java:126)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> getFormType(RemoteInterpreter.java:265)
>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>>>>> RemoteScheduler.java:307)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(
>>>>> Executors.java:511)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>>> Executor.java:1149)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>>> lExecutor.java:624)
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.thrift.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>> ClientFactory.java:53)
>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>> ClientFactory.java:37)
>>>>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>>>>> BasePooledObjectFactory.java:60)
>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.create(Gener
>>>>> icObjectPool.java:861)
>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject
>>>>> (GenericObjectPool.java:435)
>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject
>>>>> (GenericObjectPool.java:363)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProc
>>>>> ess.getClient(RemoteInterpreterProcess.java:73)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProc
>>>>> ess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>>> ... 13 more
>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>> ClientFactory.java:51)
>>>>> ... 20 more
>>>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>>>> refused)
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock
>>>>> etImpl.java:350)
>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPl
>>>>> ainSocketImpl.java:206)
>>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket
>>>>> Impl.java:188)
>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>> ... 21 more
>>>>> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
>>>>> NotebookServer.java[afterStatusChange]:2219) - Error
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.thrift.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProc
>>>>> ess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> internal_create(RemoteInterpreter.java:154)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> open(RemoteInterpreter.java:126)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>>> getFormType(RemoteInterpreter.java:265)
>>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>>>>> RemoteScheduler.java:307)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(
>>>>> Executors.java:511)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>>> Executor.java:1149)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>>> lExecutor.java:624)
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.thrift.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>> ClientFactory.java:53)
>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>> ClientFactory.java:37)
>>>>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>>>>> BasePooledObjectFactory.java:60)
>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.create(Gener
>>>>> icObjectPool.java:861)
>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject
>>>>> (GenericObjectPool.java:435)
>>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject
>>>>> (GenericObjectPool.java:363)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProc
>>>>> ess.getClient(RemoteInterpreterProcess.java:73)
>>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProc
>>>>> ess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>>> ... 13 more
>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>>> ClientFactory.java:51)
>>>>> ... 20 more
>>>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>>>> refused)
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock
>>>>> etImpl.java:350)
>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPl
>>>>> ainSocketImpl.java:206)
>>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket
>>>>> Impl.java:188)
>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>> at java.net.Socket.connect(Socket.java:589)
>>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>>> ... 21 more
>>>>> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
>>>>> NotebookServer.java[afterStatusChange]:2227) - Job
>>>>> 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.thrift.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused (Connection refused), result:
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.thrift.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>>> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
>>>>> SchedulerFactory.java[jobFinished]:115) - Job
>>>>> 20180302-135047_99102973 finished by scheduler
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-
>>>>> shared_session
>>>>>
>>>>> 2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
>>>>>
>>>>>> Hi Jeff,
>>>>>>
>>>>>> No one graved 63039 so far on Mac.
>>>>>> Yes, this issue has been reproduced as changing the port NO randomly.
>>>>>>
>>>>>> Ono
>>>>>>
>>>>>>
>>>>>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>>
>>>>>>>
>>>>>>> It is possible that 63039 is used by other processes, as Mac can
>>>>>>> not detect whether this port is used or not from my experience.
>>>>>>>
>>>>>>> And can you reproduce this issue consistently ? Because each time
>>>>>>> zeppelin will chose a random port as long as you don't specify the
>>>>>>> portRange, so if you reproduce it consistently, it might due to other
>>>>>>> issues.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>>>>>
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> I am so confusing now, let me know why if You know this, please.
>>>>>>>>
>>>>>>>> The env:
>>>>>>>> Mac: High Sierra ver.10.13.3
>>>>>>>> Zep: 0.7.3
>>>>>>>> Spark: 2.2.0
>>>>>>>> Java: 1.8
>>>>>>>> notebook script: %pyspark print 'aaa' (just it
>>>>>>>> for test)
>>>>>>>>
>>>>>>>> In this env, i got the error in log as
>>>>>>>>
>>>>>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>>>>>
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>>
>>>>>>>>
>>>>>>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>>>>>>> prepares a Port for callback to Thrift.
>>>>>>>>
>>>>>>>>
>>>>>>>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>>>>>>>> - Thrift server for callback will start. Port: 63039
>>>>>>>>
>>>>>>>>
>>>>>>>> I guess, the reason why the error was Zeppelin could not attache to
>>>>>>>> the Thrift PORT.
>>>>>>>>
>>>>>>>>
>>>>>>>> And my confusion is, when i run the same module on CentOS, it
>>>>>>>> worked fine.
>>>>>>>> CentOS: 7
>>>>>>>> Zep: same
>>>>>>>> Spark: same
>>>>>>>> Java: same
>>>>>>>> notebook script: same
>>>>>>>>
>>>>>>>> And there was no the Thift port description in the log file. From
>>>>>>>> this consequence, the fine working was cause of did not use Thrift port on
>>>>>>>> CentOS.
>>>>>>>>
>>>>>>>>
>>>>>>>> Do You know actually there is different behavior between Mac and
>>>>>>>> CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Ono
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>
Re: Thrift server for call back will start. Port:
Posted by 小野圭二 <on...@gmail.com>.
Nothing changed. I guess this property does not effect cause of failing to
call the interpreter.
i challenged it as log4j.......zeppelin=DEBUG, but no way.
2018-03-05 18:24 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
> Try to add the following to log4j.properties for more log
>
> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>
>
>
> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午5:13写道：
>
>> Actually, there was not created the interpreter log file as said 'Cannot
>> run interpreter' in the log. And it was run on local machine as MAC, i mean
>> stand alone. i stopped every firewall to make it easy. And again,this
>> happened just on Mac. On CentOS, it works fine.
>> That makes me confusion. :-b
>>
>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>
>>>
>>> Could you check the interpreter log ? The issue of not be able to
>>> connecting to thrift port could happen due to network issue, like firewall
>>> or you use VPN, etc.
>>>
>>>
>>>
>>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>>
>>>> Hi,
>>>>
>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>> the first run. The 2 is at the second run. Seems like could not established
>>>> to Thrift callback port 51348 that was set at the first run.
>>>> First of all, does it need to establish the callback port of Thrift
>>>> server?
>>>> The notebook script was just
>>>> %pyspark
>>>> print 'aaa'
>>>> Any advice appreciate.
>>>>
>>>> 1.
>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109)
>>>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>>>> interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05
>>>> 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run
>>>> paragraph 20180302-135047_99102973 using pyspark
>>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) -
>>>> Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process
>>>> WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796)
>>>> - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05
>>>> 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123)
>>>> - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>> 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130)
>>>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>>>> 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205)
>>>> - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>>>> 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>>>> -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) -
>>>> Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058]
>>>> ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.
>>>> interpreter.InterpreterException: Cannot run interpreter at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProces
>>>> s.start(RemoteInterpreterManagedProcess.java:220) at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>> getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>>>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>> at org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$
>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05
>>>> 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219)
>>>> - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot
>>>> run interpreter at org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>> getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>>>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>> at org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$
>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05
>>>> 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227)
>>>> - Job 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>>> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
>>>> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving
>>>> note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
>>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>>> finished by scheduler org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreter-shared_session
>>>>
>>>> 2.
>>>> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
>>>> SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973
>>>> started by scheduler org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreter-shared_session
>>>> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>> pyspark RemoteInterpreter_org.apache.zeppelin.spark.
>>>> PySparkInterpreter_shared_session
>>>> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) -
>>>> Job failed
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>> callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>> create(RemoteInterpreter.java:154)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
>>>> RemoteInterpreter.java:126)
>>>> at org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>>>> RemoteScheduler.java:307)
>>>> at java.util.concurrent.Executors$RunnableAdapter.
>>>> call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> ThreadPoolExecutor.java:1149)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>> ClientFactory.java:53)
>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>> ClientFactory.java:37)
>>>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>>>> BasePooledObjectFactory.java:60)
>>>> at org.apache.commons.pool2.impl.GenericObjectPool.create(
>>>> GenericObjectPool.java:861)
>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>> GenericObjectPool.java:435)
>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>> GenericObjectPool.java:363)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>> getClient(RemoteInterpreterProcess.java:73)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>> callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>> ... 13 more
>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>> ClientFactory.java:51)
>>>> ... 20 more
>>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>>> refused)
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at java.net.AbstractPlainSocketImpl.doConnect(
>>>> AbstractPlainSocketImpl.java:350)
>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(
>>>> AbstractPlainSocketImpl.java:206)
>>>> at java.net.AbstractPlainSocketImpl.connect(
>>>> AbstractPlainSocketImpl.java:188)
>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>> at java.net.Socket.connect(Socket.java:589)
>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>> ... 21 more
>>>> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219)
>>>> - Error
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>> callRemoteFunction(RemoteInterpreterProcess.java:152)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>> create(RemoteInterpreter.java:154)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
>>>> RemoteInterpreter.java:126)
>>>> at org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>>>> RemoteScheduler.java:307)
>>>> at java.util.concurrent.Executors$RunnableAdapter.
>>>> call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> ThreadPoolExecutor.java:1149)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>> ClientFactory.java:53)
>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>> ClientFactory.java:37)
>>>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>>>> BasePooledObjectFactory.java:60)
>>>> at org.apache.commons.pool2.impl.GenericObjectPool.create(
>>>> GenericObjectPool.java:861)
>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>> GenericObjectPool.java:435)
>>>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>>>> GenericObjectPool.java:363)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>> getClient(RemoteInterpreterProcess.java:73)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>>>> callRemoteFunction(RemoteInterpreterProcess.java:144)
>>>> ... 13 more
>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>>>> ClientFactory.java:51)
>>>> ... 20 more
>>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>>> refused)
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at java.net.AbstractPlainSocketImpl.doConnect(
>>>> AbstractPlainSocketImpl.java:350)
>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(
>>>> AbstractPlainSocketImpl.java:206)
>>>> at java.net.AbstractPlainSocketImpl.connect(
>>>> AbstractPlainSocketImpl.java:188)
>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>> at java.net.Socket.connect(Socket.java:589)
>>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>>> ... 21 more
>>>> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227)
>>>> - Job 20180302-135047_99102973 is finished, status: ERROR, exception:
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused (Connection refused), result:
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>>> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
>>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>>> finished by scheduler org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreter-shared_session
>>>>
>>>> 2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
>>>>
>>>>> Hi Jeff,
>>>>>
>>>>> No one graved 63039 so far on Mac.
>>>>> Yes, this issue has been reproduced as changing the port NO randomly.
>>>>>
>>>>> Ono
>>>>>
>>>>>
>>>>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>>
>>>>>>
>>>>>> It is possible that 63039 is used by other processes, as Mac can not
>>>>>> detect whether this port is used or not from my experience.
>>>>>>
>>>>>> And can you reproduce this issue consistently ? Because each time
>>>>>> zeppelin will chose a random port as long as you don't specify the
>>>>>> portRange, so if you reproduce it consistently, it might due to other
>>>>>> issues.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>>>>
>>>>>>> Hi there,
>>>>>>>
>>>>>>> I am so confusing now, let me know why if You know this, please.
>>>>>>>
>>>>>>> The env:
>>>>>>> Mac: High Sierra ver.10.13.3
>>>>>>> Zep: 0.7.3
>>>>>>> Spark: 2.2.0
>>>>>>> Java: 1.8
>>>>>>> notebook script: %pyspark print 'aaa' (just it
>>>>>>> for test)
>>>>>>>
>>>>>>> In this env, i got the error in log as
>>>>>>>
>>>>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>>>>
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>>
>>>>>>>
>>>>>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>>>>>> prepares a Port for callback to Thrift.
>>>>>>>
>>>>>>>
>>>>>>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>>>>>>> - Thrift server for callback will start. Port: 63039
>>>>>>>
>>>>>>>
>>>>>>> I guess, the reason why the error was Zeppelin could not attache to
>>>>>>> the Thrift PORT.
>>>>>>>
>>>>>>>
>>>>>>> And my confusion is, when i run the same module on CentOS, it worked
>>>>>>> fine.
>>>>>>> CentOS: 7
>>>>>>> Zep: same
>>>>>>> Spark: same
>>>>>>> Java: same
>>>>>>> notebook script: same
>>>>>>>
>>>>>>> And there was no the Thift port description in the log file. From
>>>>>>> this consequence, the fine working was cause of did not use Thrift port on
>>>>>>> CentOS.
>>>>>>>
>>>>>>>
>>>>>>> Do You know actually there is different behavior between Mac and
>>>>>>> CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Ono
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
Re: Thrift server for call back will start. Port:
Posted by Jeff Zhang <zj...@gmail.com>.
Try to add the following to log4j.properties for more log
log4j.logger.org.apache.zeppelin.interpreter=DEBUG
小野圭二 <on...@gmail.com>于2018年3月5日周一 下午5:13写道：
> Actually, there was not created the interpreter log file as said 'Cannot
> run interpreter' in the log. And it was run on local machine as MAC, i mean
> stand alone. i stopped every firewall to make it easy. And again,this
> happened just on Mac. On CentOS, it works fine.
> That makes me confusion. :-b
>
> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
>>
>> Could you check the interpreter log ? The issue of not be able to
>> connecting to thrift port could happen due to network issue, like firewall
>> or you use VPN, etc.
>>
>>
>>
>> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>
>>> Hi,
>>>
>>> I show the all log on this error.There are 2 type error. The 1 is at the
>>> first run. The 2 is at the second run. Seems like could not established to
>>> Thrift callback port 51348 that was set at the first run.
>>> First of all, does it need to establish the callback port of Thrift
>>> server?
>>> The notebook script was just
>>> %pyspark
>>> print 'aaa'
>>> Any advice appreciate.
>>>
>>> 1.
>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
>>> 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) -
>>> Job 20180302-135047_99102973 started by scheduler
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>> pyspark
>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
>>> InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
>>> [2018-03-05 13:11:14,010] ({pool-2-thread-2}
>>> InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not
>>> found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
>>> Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>> 13:11:14,023] ({pool-2-thread-2}
>>> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
>>> callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049]
>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run
>>> interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10,
>>> -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g,
>>> spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter
>>> process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2}
>>> Job.java[run]:184) - Job failed
>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>> interpreter at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
>>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>> interpreter at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075]
>>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job
>>> 20180302-135047_99102973 is finished, status: ERROR, exception:
>>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>>> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
>>> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>> INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>> finished by scheduler
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>
>>> 2.
>>> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
>>> SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973
>>> started by scheduler
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>> pyspark
>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) -
>>> Job failed
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>> at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>>> at
>>> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>> ... 13 more
>>> Caused by: org.apache.thrift.transport.TTransportException:
>>> java.net.ConnectException: Connection refused (Connection refused)
>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>>> ... 20 more
>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>> refused)
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net
>>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>> at java.net
>>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>> at java.net
>>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:589)
>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>> ... 21 more
>>> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
>>> NotebookServer.java[afterStatusChange]:2219) - Error
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>>> at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>>> at
>>> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>>> at
>>> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
>>> ... 13 more
>>> Caused by: org.apache.thrift.transport.TTransportException:
>>> java.net.ConnectException: Connection refused (Connection refused)
>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>>> at
>>> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>>> ... 20 more
>>> Caused by: java.net.ConnectException: Connection refused (Connection
>>> refused)
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net
>>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>> at java.net
>>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>> at java.net
>>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:589)
>>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>>> ... 21 more
>>> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
>>> NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973
>>> is finished, status: ERROR, exception:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused), result:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
>>> Connection refused (Connection refused)
>>> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>>> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
>>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>>> finished by scheduler
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>
>>> 2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
>>>
>>>> Hi Jeff,
>>>>
>>>> No one graved 63039 so far on Mac.
>>>> Yes, this issue has been reproduced as changing the port NO randomly.
>>>>
>>>> Ono
>>>>
>>>>
>>>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>>
>>>>>
>>>>> It is possible that 63039 is used by other processes, as Mac can not
>>>>> detect whether this port is used or not from my experience.
>>>>>
>>>>> And can you reproduce this issue consistently ? Because each time
>>>>> zeppelin will chose a random port as long as you don't specify the
>>>>> portRange, so if you reproduce it consistently, it might due to other
>>>>> issues.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> I am so confusing now, let me know why if You know this, please.
>>>>>>
>>>>>> The env:
>>>>>> Mac: High Sierra ver.10.13.3
>>>>>> Zep: 0.7.3
>>>>>> Spark: 2.2.0
>>>>>> Java: 1.8
>>>>>> notebook script: %pyspark print 'aaa' (just it for
>>>>>> test)
>>>>>>
>>>>>> In this env, i got the error in log as
>>>>>>
>>>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>>>
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>>
>>>>>>
>>>>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>>>>> prepares a Port for callback to Thrift.
>>>>>>
>>>>>>
>>>>>> INFO ({pool-2-thread-1}
>>>>>> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
>>>>>> callback will start. Port: 63039
>>>>>>
>>>>>>
>>>>>> I guess, the reason why the error was Zeppelin could not attache to
>>>>>> the Thrift PORT.
>>>>>>
>>>>>>
>>>>>> And my confusion is, when i run the same module on CentOS, it worked
>>>>>> fine.
>>>>>> CentOS: 7
>>>>>> Zep: same
>>>>>> Spark: same
>>>>>> Java: same
>>>>>> notebook script: same
>>>>>>
>>>>>> And there was no the Thift port description in the log file. From
>>>>>> this consequence, the fine working was cause of did not use Thrift port on
>>>>>> CentOS.
>>>>>>
>>>>>>
>>>>>> Do You know actually there is different behavior between Mac and
>>>>>> CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Ono
>>>>>>
>>>>>
>>>>
>>>
>
Re: Thrift server for call back will start. Port:
Posted by 小野圭二 <on...@gmail.com>.
Actually, there was not created the interpreter log file as said 'Cannot
run interpreter' in the log. And it was run on local machine as MAC, i mean
stand alone. i stopped every firewall to make it easy. And again,this
happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b
2018-03-05 14:06 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
> Could you check the interpreter log ? The issue of not be able to
> connecting to thrift port could happen due to network issue, like firewall
> or you use VPN, etc.
>
>
>
> 小野圭二 <on...@gmail.com>于2018年3月5日周一 下午12:32写道：
>
>> Hi,
>>
>> I show the all log on this error.There are 2 type error. The 1 is at the
>> first run. The 2 is at the second run. Seems like could not established to
>> Thrift callback port 51348 that was set at the first run.
>> First of all, does it need to establish the callback port of Thrift
>> server?
>> The notebook script was just
>> %pyspark
>> print 'aaa'
>> Any advice appreciate.
>>
>> 1.
>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
>> 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109)
>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05
>> 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run
>> paragraph 20180302-135047_99102973 using pyspark
>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
>> InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
>> [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796)
>> - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05
>> 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123)
>> - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>> 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130)
>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>> 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205)
>> - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>> 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>> -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) -
>> Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058]
>> ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.
>> interpreter.InterpreterException: Cannot run interpreter at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProces
>> s.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
>> ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.
>> ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>> interpreter at org.apache.zeppelin.interpreter.remote.
>> RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>> getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
>> ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.
>> ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075]
>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job
>> 20180302-135047_99102973 is finished, status: ERROR, exception:
>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
>> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving
>> note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
>> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
>> finished by scheduler org.apache.zeppelin.interpreter.remote.
>> RemoteInterpreter-shared_session
>>
>> 2.
>> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109)
>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter-shared_session
>> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>> pyspark RemoteInterpreter_org.apache.zeppelin.spark.
>> PySparkInterpreter_shared_session
>> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) -
>> Job failed
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>> callRemoteFunction(RemoteInterpreterProcess.java:152)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>> create(RemoteInterpreter.java:154)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
>> RemoteInterpreter.java:126)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(
>> RemoteInterpreter.java:265)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>> RemoteScheduler.java:307)
>> at java.util.concurrent.Executors$RunnableAdapter.
>> call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>> ClientFactory.java:53)
>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>> ClientFactory.java:37)
>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>> BasePooledObjectFactory.java:60)
>> at org.apache.commons.pool2.impl.GenericObjectPool.create(
>> GenericObjectPool.java:861)
>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>> GenericObjectPool.java:435)
>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>> GenericObjectPool.java:363)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>> getClient(RemoteInterpreterProcess.java:73)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>> callRemoteFunction(RemoteInterpreterProcess.java:144)
>> ... 13 more
>> Caused by: org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>> ClientFactory.java:51)
>> ... 20 more
>> Caused by: java.net.ConnectException: Connection refused (Connection
>> refused)
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.AbstractPlainSocketImpl.doConnect(
>> AbstractPlainSocketImpl.java:350)
>> at java.net.AbstractPlainSocketImpl.connectToAddress(
>> AbstractPlainSocketImpl.java:206)
>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
>> 188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:589)
>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>> ... 21 more
>> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219)
>> - Error
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>> callRemoteFunction(RemoteInterpreterProcess.java:152)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>> create(RemoteInterpreter.java:154)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(
>> RemoteInterpreter.java:126)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(
>> RemoteInterpreter.java:265)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(
>> RemoteScheduler.java:307)
>> at java.util.concurrent.Executors$RunnableAdapter.
>> call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>> ClientFactory.java:53)
>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>> ClientFactory.java:37)
>> at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(
>> BasePooledObjectFactory.java:60)
>> at org.apache.commons.pool2.impl.GenericObjectPool.create(
>> GenericObjectPool.java:861)
>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>> GenericObjectPool.java:435)
>> at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(
>> GenericObjectPool.java:363)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>> getClient(RemoteInterpreterProcess.java:73)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.
>> callRemoteFunction(RemoteInterpreterProcess.java:144)
>> ... 13 more
>> Caused by: org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>> at org.apache.zeppelin.interpreter.remote.ClientFactory.create(
>> ClientFactory.java:51)
>> ... 20 more
>> Caused by: java.net.ConnectException: Connection refused (Connection
>> refused)
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.AbstractPlainSocketImpl.doConnect(
>> AbstractPlainSocketImpl.java:350)
>> at java.net.AbstractPlainSocketImpl.connectToAddress(
>> AbstractPlainSocketImpl.java:206)
>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
>> 188)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:589)
>> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>> ... 21 more
>> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227)
>> - Job 20180302-135047_99102973 is finished, status: ERROR, exception:
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused), result:
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
>> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115)
>> - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter-shared_session
>>
>> 2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
>>
>>> Hi Jeff,
>>>
>>> No one graved 63039 so far on Mac.
>>> Yes, this issue has been reproduced as changing the port NO randomly.
>>>
>>> Ono
>>>
>>>
>>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>>
>>>>
>>>> It is possible that 63039 is used by other processes, as Mac can not
>>>> detect whether this port is used or not from my experience.
>>>>
>>>> And can you reproduce this issue consistently ? Because each time
>>>> zeppelin will chose a random port as long as you don't specify the
>>>> portRange, so if you reproduce it consistently, it might due to other
>>>> issues.
>>>>
>>>>
>>>>
>>>>
>>>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>>
>>>>> Hi there,
>>>>>
>>>>> I am so confusing now, let me know why if You know this, please.
>>>>>
>>>>> The env:
>>>>> Mac: High Sierra ver.10.13.3
>>>>> Zep: 0.7.3
>>>>> Spark: 2.2.0
>>>>> Java: 1.8
>>>>> notebook script: %pyspark print 'aaa' (just it for
>>>>> test)
>>>>>
>>>>> In this env, i got the error in log as
>>>>>
>>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>>
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>>> org.apache.thrift.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>>
>>>>>
>>>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>>>> prepares a Port for callback to Thrift.
>>>>>
>>>>>
>>>>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>>>>> - Thrift server for callback will start. Port: 63039
>>>>>
>>>>>
>>>>> I guess, the reason why the error was Zeppelin could not attache to
>>>>> the Thrift PORT.
>>>>>
>>>>>
>>>>> And my confusion is, when i run the same module on CentOS, it worked
>>>>> fine.
>>>>> CentOS: 7
>>>>> Zep: same
>>>>> Spark: same
>>>>> Java: same
>>>>> notebook script: same
>>>>>
>>>>> And there was no the Thift port description in the log file. From this
>>>>> consequence, the fine working was cause of did not use Thrift port on
>>>>> CentOS.
>>>>>
>>>>>
>>>>> Do You know actually there is different behavior between Mac and
>>>>> CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>>
>>>>>
>>>>> Best,
>>>>> Ono
>>>>>
>>>>
>>>
>>
Re: Thrift server for call back will start. Port:
Posted by Jeff Zhang <zj...@gmail.com>.
Could you check the interpreter log ? The issue of not be able to
connecting to thrift port could happen due to network issue, like firewall
or you use VPN, etc.
小野圭二 <on...@gmail.com>于2018年3月5日周一 下午12:32写道：
> Hi,
>
> I show the all log on this error.There are 2 type error. The 1 is at the
> first run. The 2 is at the second run. Seems like could not established to
> Thrift callback port 51348 that was set at the first run.
> First of all, does it need to establish the callback port of Thrift
> server?
> The notebook script was just
> %pyspark
> print 'aaa'
> Any advice appreciate.
>
> 1.
> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
> 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) -
> Job 20180302-135047_99102973 started by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
> pyspark
> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
> InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
> [2018-03-05 13:11:14,010] ({pool-2-thread-2}
> InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not
> found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
> Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
> 13:11:14,023] ({pool-2-thread-2}
> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
> callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049]
> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run
> interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10,
> -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g,
> spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter
> process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2}
> Job.java[run]:184) - Job failed
> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
> interpreter at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
> interpreter at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075]
> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job
> 20180302-135047_99102973 is finished, status: ERROR, exception:
> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
> interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
> ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
> INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
> finished by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>
> 2.
> INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
> INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
> SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973
> started by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
> INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
> pyspark
> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
> ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) -
> Job failed
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused)
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
> at
> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
> at
> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
> ... 13 more
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused (Connection refused)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
> ... 20 more
> Caused by: java.net.ConnectException: Connection refused (Connection
> refused)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
> ... 21 more
> ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
> NotebookServer.java[afterStatusChange]:2219) - Error
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused)
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
> at
> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
> at
> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
> ... 13 more
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused (Connection refused)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
> ... 20 more
> Caused by: java.net.ConnectException: Connection refused (Connection
> refused)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
> ... 21 more
> WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
> NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973
> is finished, status: ERROR, exception:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused), result:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused)
> INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
> INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
> SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
> finished by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>
> 2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
>
>> Hi Jeff,
>>
>> No one graved 63039 so far on Mac.
>> Yes, this issue has been reproduced as changing the port NO randomly.
>>
>> Ono
>>
>>
>> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>
>>>
>>> It is possible that 63039 is used by other processes, as Mac can not
>>> detect whether this port is used or not from my experience.
>>>
>>> And can you reproduce this issue consistently ? Because each time
>>> zeppelin will chose a random port as long as you don't specify the
>>> portRange, so if you reproduce it consistently, it might due to other
>>> issues.
>>>
>>>
>>>
>>>
>>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>>
>>>> Hi there,
>>>>
>>>> I am so confusing now, let me know why if You know this, please.
>>>>
>>>> The env:
>>>> Mac: High Sierra ver.10.13.3
>>>> Zep: 0.7.3
>>>> Spark: 2.2.0
>>>> Java: 1.8
>>>> notebook script: %pyspark print 'aaa' (just it for
>>>> test)
>>>>
>>>> In this env, i got the error in log as
>>>>
>>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>>
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused (Connection refused)
>>>>
>>>>
>>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>>> prepares a Port for callback to Thrift.
>>>>
>>>>
>>>> INFO ({pool-2-thread-1}
>>>> RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
>>>> callback will start. Port: 63039
>>>>
>>>>
>>>> I guess, the reason why the error was Zeppelin could not attache to the
>>>> Thrift PORT.
>>>>
>>>>
>>>> And my confusion is, when i run the same module on CentOS, it worked
>>>> fine.
>>>> CentOS: 7
>>>> Zep: same
>>>> Spark: same
>>>> Java: same
>>>> notebook script: same
>>>>
>>>> And there was no the Thift port description in the log file. From this
>>>> consequence, the fine working was cause of did not use Thrift port on
>>>> CentOS.
>>>>
>>>>
>>>> Do You know actually there is different behavior between Mac and
>>>> CentOS? And i am so appreciated if You know the workaround on Mac.
>>>>
>>>>
>>>> Best,
>>>> Ono
>>>>
>>>
>>
>
Re: Thrift server for call back will start. Port:
Posted by 小野圭二 <on...@gmail.com>.
Hi,
I show the all log on this error.There are 2 type error. The 1 is at the
first run. The 2 is at the second run. Seems like could not established to
Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift
server?
The notebook script was just
%pyspark
print 'aaa'
Any advice appreciate.
1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) -
Job 20180302-135047_99102973 started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
pyspark
RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
[2018-03-05 13:11:14,010] ({pool-2-thread-2}
InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not
found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
13:11:14,023] ({pool-2-thread-2}
RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for
callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049]
({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run
interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
-d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10,
-p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g,
spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter
process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2}
Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: Cannot run
interpreter at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: Cannot run
interpreter at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075]
({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job
20180302-135047_99102973 is finished, status: ERROR, exception:
org.apache.zeppelin.interpreter.InterpreterException: Cannot run
interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103]
({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2}
SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
finished by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13}
VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2}
SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973
started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2}
Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
pyspark
RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job
failed
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2}
NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973
is finished, status: ERROR, exception:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused), result:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
finished by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
2018-03-02 15:07 GMT+09:00 小野圭二 <on...@gmail.com>:
> Hi Jeff,
>
> No one graved 63039 so far on Mac.
> Yes, this issue has been reproduced as changing the port NO randomly.
>
> Ono
>
>
> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
>>
>> It is possible that 63039 is used by other processes, as Mac can not
>> detect whether this port is used or not from my experience.
>>
>> And can you reproduce this issue consistently ? Because each time
>> zeppelin will chose a random port as long as you don't specify the
>> portRange, so if you reproduce it consistently, it might due to other
>> issues.
>>
>>
>>
>>
>> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>
>>> Hi there,
>>>
>>> I am so confusing now, let me know why if You know this, please.
>>>
>>> The env:
>>> Mac: High Sierra ver.10.13.3
>>> Zep: 0.7.3
>>> Spark: 2.2.0
>>> Java: 1.8
>>> notebook script: %pyspark print 'aaa' (just it for
>>> test)
>>>
>>> In this env, i got the error in log as
>>>
>>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>>
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> org.apache.thrift.transport.TTransportException:
>>> java.net.ConnectException: Connection refused (Connection refused)
>>>
>>>
>>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>>> prepares a Port for callback to Thrift.
>>>
>>>
>>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>>> - Thrift server for callback will start. Port: 63039
>>>
>>>
>>> I guess, the reason why the error was Zeppelin could not attache to the
>>> Thrift PORT.
>>>
>>>
>>> And my confusion is, when i run the same module on CentOS, it worked
>>> fine.
>>> CentOS: 7
>>> Zep: same
>>> Spark: same
>>> Java: same
>>> notebook script: same
>>>
>>> And there was no the Thift port description in the log file. From this
>>> consequence, the fine working was cause of did not use Thrift port on
>>> CentOS.
>>>
>>>
>>> Do You know actually there is different behavior between Mac and CentOS?
>>> And i am so appreciated if You know the workaround on Mac.
>>>
>>>
>>> Best,
>>> Ono
>>>
>>
>
Re: Thrift server for call back will start. Port:
Posted by 小野圭二 <on...@gmail.com>.
Hi Jeff,
No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.
Ono
2018-03-02 14:49 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
> It is possible that 63039 is used by other processes, as Mac can not
> detect whether this port is used or not from my experience.
>
> And can you reproduce this issue consistently ? Because each time zeppelin
> will chose a random port as long as you don't specify the portRange, so if
> you reproduce it consistently, it might due to other issues.
>
>
>
>
> 小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
>
>> Hi there,
>>
>> I am so confusing now, let me know why if You know this, please.
>>
>> The env:
>> Mac: High Sierra ver.10.13.3
>> Zep: 0.7.3
>> Spark: 2.2.0
>> Java: 1.8
>> notebook script: %pyspark print 'aaa' (just it for
>> test)
>>
>> In this env, i got the error in log as
>>
>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>>
>>
>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>> prepares a Port for callback to Thrift.
>>
>>
>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>> - Thrift server for callback will start. Port: 63039
>>
>>
>> I guess, the reason why the error was Zeppelin could not attache to the
>> Thrift PORT.
>>
>>
>> And my confusion is, when i run the same module on CentOS, it worked fine.
>> CentOS: 7
>> Zep: same
>> Spark: same
>> Java: same
>> notebook script: same
>>
>> And there was no the Thift port description in the log file. From this
>> consequence, the fine working was cause of did not use Thrift port on
>> CentOS.
>>
>>
>> Do You know actually there is different behavior between Mac and CentOS?
>> And i am so appreciated if You know the workaround on Mac.
>>
>>
>> Best,
>> Ono
>>
>
Re: Thrift server for call back will start. Port:
Posted by Jeff Zhang <zj...@gmail.com>.
It is possible that 63039 is used by other processes, as Mac can not detect
whether this port is used or not from my experience.
And can you reproduce this issue consistently ? Because each time zeppelin
will chose a random port as long as you don't specify the portRange, so if
you reproduce it consistently, it might due to other issues.
小野圭二 <on...@gmail.com>于2018年3月2日周五 下午1:32写道：
> Hi there,
>
> I am so confusing now, let me know why if You know this, please.
>
> The env:
> Mac: High Sierra ver.10.13.3
> Zep: 0.7.3
> Spark: 2.2.0
> Java: 1.8
> notebook script: %pyspark print 'aaa' (just it for test)
>
> In this env, i got the error in log as
>
> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused)
>
>
> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
> prepares a Port for callback to Thrift.
>
>
> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) -
> Thrift server for callback will start. Port: 63039
>
>
> I guess, the reason why the error was Zeppelin could not attache to the
> Thrift PORT.
>
>
> And my confusion is, when i run the same module on CentOS, it worked fine.
> CentOS: 7
> Zep: same
> Spark: same
> Java: same
> notebook script: same
>
> And there was no the Thift port description in the log file. From this
> consequence, the fine working was cause of did not use Thrift port on
> CentOS.
>
>
> Do You know actually there is different behavior between Mac and CentOS?
> And i am so appreciated if You know the workaround on Mac.
>
>
> Best,
> Ono
>