You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Sarthak Sharma <sa...@media.net> on 2018/11/19 11:41:41 UTC
Runtime Exception while trying to connect to remote spark
interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)
Hi,
We already have a zeppelin-0.7.3 setup which runs fine and is in use
currently but we are looking into the yarn cluster mode support for spark
interpreter in zeppelin-0.8. I've built it from source from *branch-0.8 (As
of Nov-15) *and am facing the following issues intermittently in some of
the spark interpreters while trying to use spark-sql on it.
*18/11/19 10:04:07 INFO yarn.Client: Submitting application
application_1542587655772_35129 to ResourceManager*
*18/11/19 10:04:07 INFO impl.YarnClientImpl: Submitted application
application_1542587655772_35129*
*18/11/19 10:04:08 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:08 INFO yarn.Client:*
* client token: N/A*
* diagnostics: N/A*
* ApplicationMaster host: N/A*
* ApplicationMaster RPC port: -1*
* queue: root.zep*
* start time: 1542621847537*
* final status: UNDEFINED*
* tracking
URL: http://resource-manager-addr/proxy/application_1542587655772_35129/
<http://c8-auto-hadoop-service-1.srv.media.net:8088/proxy/application_1542587655772_35129/>*
* user: sarthak.sh*
*18/11/19 10:04:09 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:10 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:11 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:12 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:13 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:14 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:15 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:16 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:17 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:18 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:19 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:20 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:21 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:22 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:23 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:24 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:25 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:26 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:27 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:28 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:29 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:30 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:31 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:32 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:33 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:34 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:35 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:36 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:37 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:38 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:39 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:40 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:41 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:42 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:43 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:44 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:45 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:46 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:47 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:48 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:49 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:50 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:51 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:52 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:53 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:54 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:55 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:56 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:57 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:58 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:04:59 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:00 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:01 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:02 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:03 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:04 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:05 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:06 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:07 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:08 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:09 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:10 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:11 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:12 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:13 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:14 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:15 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:16 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:17 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:18 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:19 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:20 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:21 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:22 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:23 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:24 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:25 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:26 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:27 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:28 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:29 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:30 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:31 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:32 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:33 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:34 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:35 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:36 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:37 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:38 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:39 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:40 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:41 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:42 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:43 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:44 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:45 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:46 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:47 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:48 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:49 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:50 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:51 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:52 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:53 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:54 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:55 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:56 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:57 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:58 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
*18/11/19 10:05:59 INFO yarn.Client: Application report for
application_1542587655772_35129 (state: ACCEPTED)*
* at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:205)*
* at
org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:64)*
* at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:111)*
* at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:164)*
* at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:132)*
* at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:299)*
* at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:407)*
* at org.apache.zeppelin.scheduler.Job.run(Job.java:188)*
* at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)*
* at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)*
* at java.util.concurrent.FutureTask.run(FutureTask.java:266)*
* at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)*
* at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)*
* at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*
* at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*
* at java.lang.Thread.run(Thread.java:748)*
Any further submit to this interpreter will give null pointer exceptions
due to the absence of an interpreter process.
It looks like the interpreter driver process while getting submitted to
yarn, is stuck in ACCEPTED state because of which we're not able to connect
to the remote interpreter process. This happens even if there are resources
on the cluster in yarn.
Also I've tried increasing the *zeppelin.interpreter.connect.timeout *but
that didn't help since the application is stuck in ACCEPTED state
indefinitely and there are no logs available too.
It'll be great if you can point me to something that can help. Also please
do let me know if any configuration files are required for debugging this.
Thanks and Regards
*Sarthak Sharma*
DevOps Engineer, Media.Net
+918002228376 | sarthak.sh@media.net
<http://en-gb.facebook.com/people/Sarthak-Sharma/100006006014244>
<http://in.linkedin.com/in/sarthaksharma96>
Re: Runtime Exception while trying to connect to remote spark
interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)
Posted by Xun Liu <ne...@163.com>.
hi，Sarthak Sharma
You can be at the zeppelin server,
Run ./bin/spark-submit --class org.apache.spark.examples.SparkPi,
Test it to see if there is a problem with the spark runtime environment on the zeppelin server.
> 在 2018年11月20日，下午5:39，Sarthak Sharma <sa...@media.net> 写道：
>
> Is it similar to an existing bug related to the interpreter processes getting stuck ? (wherein the workaround is to kill the application on yarn, restart the interpreter from the interface and then try resubmitting the query again).
> The problem in this case is that it is intermittently happening on some spark interpreters randomly. And since the driver app is not scheduled on yarn, there are no logs available to figure out the reason for this issue.
>
> Thanks and Regards
>
> Sarthak Sharma
> DevOps Engineer, Media.Net
> +918002228376 <tel:+918002228376> | sarthak.sh@media.net <ma...@media.net>
> <http://en-gb.facebook.com/people/Sarthak-Sharma/100006006014244> <http://in.linkedin.com/in/sarthaksharma96>
>
>
> On Tue, Nov 20, 2018 at 2:22 PM Jeff Zhang <zjffdu@gmail.com <ma...@gmail.com>> wrote:
> If zeppelin.interpreter.connect.timeout is reached, but the yarn app is still in ACCEPTED state, then this should be a bug. The yarn app should be killed it it can not be created in the timeout threashold
>
> Sarthak Sharma <sarthak.sh@media.net <ma...@media.net>> 于2018年11月20日周二 下午4:47写道：
> Hey,
>
> Like you mentioned, I'm already using the spark.yarn.queue parameter, hence I know which yarn queue it is getting scheduled in and this queue has resources available for applications since other apps are also getting scheduled there.
> However, assuming the queue does NOT have resources for it to schedule within the given time frame causing it to throw an exception after the zeppelin.interpreter.connect.timeout is reached, the application should in any case get scheduled eventually which is not the case here. Interpreter driver process remains stuck in ACCEPTED state. Is there a change in the way it is implemented in this version ? Since we never experienced this on the previous one (zeppelin-0.7.3) where drivers would get scheduled eventually in their respective queues.
>
> On Tue, Nov 20, 2018, 7:29 AM Xun Liu <neliuxun@163.com <ma...@163.com> wrote:
> HI，Sarthak Sharma
>
> The log shows that the task submitted by spark-submmit has been waiting for execution in the queue of YARN. Is there no resource for the queue of YARN?
> You can specify a queue with resources in the spark interpreter via the spark.yarn.queue parameter.
>
>
>> 在 2018年11月19日，下午7:41，Sarthak Sharma <sarthak.sh@media.net <ma...@media.net>> 写道：
>>
>> Hi,
>>
>> We already have a zeppelin-0.7.3 setup which runs fine and is in use currently but we are looking into the yarn cluster mode support for spark interpreter in zeppelin-0.8. I've built it from source from branch-0.8 (As of Nov-15) and am facing the following issues intermittently in some of the spark interpreters while trying to use spark-sql on it.
>>
>> 18/11/19 10:04:07 INFO yarn.Client: Submitting application application_1542587655772_35129 to ResourceManager
>> 18/11/19 10:04:07 INFO impl.YarnClientImpl: Submitted application application_1542587655772_35129
>> 18/11/19 10:04:08 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:08 INFO yarn.Client:
>> client token: N/A
>> diagnostics: N/A
>> ApplicationMaster host: N/A
>> ApplicationMaster RPC port: -1
>> queue: root.zep
>> start time: 1542621847537
>> final status: UNDEFINED
>> tracking URL: http://resource-manager-addr/proxy/application_1542587655772_35129/ <http://c8-auto-hadoop-service-1.srv.media.net:8088/proxy/application_1542587655772_35129/>
>> user: sarthak.sh
>> 18/11/19 10:04:09 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:10 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:11 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:12 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:13 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:14 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:15 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:16 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:17 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:18 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:19 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:20 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:21 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:22 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:23 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:24 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:25 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:26 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:27 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:28 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:29 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:30 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:31 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:32 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:33 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:34 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:35 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:36 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:37 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:38 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:39 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:40 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:41 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:42 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:43 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:44 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:45 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:46 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:47 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:48 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:49 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:50 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:51 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:52 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:53 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:54 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:55 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:56 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:57 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:58 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:04:59 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:00 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:01 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:02 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:03 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:04 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:05 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:06 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:07 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:08 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:09 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:10 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:11 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:12 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:13 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:14 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:15 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:16 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:17 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:18 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:19 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:20 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:21 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:22 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:23 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:24 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:25 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:26 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:27 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:28 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:29 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:30 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:31 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:32 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:33 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:34 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:35 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:36 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:37 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:38 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:39 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:40 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:41 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:42 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:43 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:44 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:45 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:46 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:47 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:48 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:49 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:50 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:51 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:52 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:53 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:54 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:55 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:56 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:57 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:58 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>> 18/11/19 10:05:59 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>>
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:205)
>> at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:64)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:111)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:164)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:132)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:299)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:407)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
>> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> Any further submit to this interpreter will give null pointer exceptions due to the absence of an interpreter process.
>> It looks like the interpreter driver process while getting submitted to yarn, is stuck in ACCEPTED state because of which we're not able to connect to the remote interpreter process. This happens even if there are resources on the cluster in yarn.
>> Also I've tried increasing the zeppelin.interpreter.connect.timeout but that didn't help since the application is stuck in ACCEPTED state indefinitely and there are no logs available too.
>> It'll be great if you can point me to something that can help. Also please do let me know if any configuration files are required for debugging this.
>>
>>
>> Thanks and Regards
>>
>>
>> Sarthak Sharma
>> DevOps Engineer, Media.Net <http://media.net/>
>> +918002228376 <tel:+918002228376> | sarthak.sh@media.net <ma...@media.net>
>> <http://en-gb.facebook.com/people/Sarthak-Sharma/100006006014244> <http://in.linkedin.com/in/sarthaksharma96>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
Re: Runtime Exception while trying to connect to remote spark
interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)
Posted by Sarthak Sharma <sa...@media.net>.
Is it similar to an existing bug related to the interpreter processes
getting stuck ? (wherein the workaround is to kill the application on yarn,
restart the interpreter from the interface and then try resubmitting the
query again).
The problem in this case is that it is intermittently happening on some
spark interpreters randomly. And since the driver app is not scheduled on
yarn, there are no logs available to figure out the reason for this issue.
Thanks and Regards
*Sarthak Sharma*
DevOps Engineer, Media.Net
+918002228376 | sarthak.sh@media.net
<http://en-gb.facebook.com/people/Sarthak-Sharma/100006006014244>
<http://in.linkedin.com/in/sarthaksharma96>
On Tue, Nov 20, 2018 at 2:22 PM Jeff Zhang <zj...@gmail.com> wrote:
> If *zeppelin.interpreter.connect.timeout *is reached, but the yarn app is
> still in ACCEPTED state, then this should be a bug. The yarn app should be
> killed it it can not be created in the timeout threashold
>
> Sarthak Sharma <sa...@media.net> 于2018年11月20日周二 下午4:47写道：
>
>> Hey,
>>
>> Like you mentioned, I'm already using the *spark.yarn.queue* parameter,
>> hence I know which yarn queue it is getting scheduled in and this queue has
>> resources available for applications since other apps are also getting
>> scheduled there.
>> However, assuming the queue does NOT have resources for it to schedule
>> within the given time frame causing it to throw an exception after the *zeppelin.interpreter.connect.timeout
>> *is reached, the application should in any case get scheduled eventually
>> which is not the case here. Interpreter driver process remains stuck in
>> ACCEPTED state. Is there a change in the way it is implemented in this
>> version ? Since we never experienced this on the previous one
>> (zeppelin-0.7.3) where drivers would get scheduled eventually in their
>> respective queues.
>>
>> On Tue, Nov 20, 2018, 7:29 AM Xun Liu <neliuxun@163.com wrote:
>>
>>> HI，Sarthak Sharma
>>>
>>> The log shows that the task submitted by spark-submmit has been waiting
>>> for execution in the queue of YARN. Is there no resource for the queue of
>>> YARN?
>>> You can specify a queue with resources in the spark interpreter via the
>>> spark.yarn.queue parameter.
>>>
>>>
>>> 在 2018年11月19日，下午7:41，Sarthak Sharma <sa...@media.net> 写道：
>>>
>>> Hi,
>>>
>>> We already have a zeppelin-0.7.3 setup which runs fine and is in use
>>> currently but we are looking into the yarn cluster mode support for spark
>>> interpreter in zeppelin-0.8. I've built it from source from *branch-0.8
>>> (As of Nov-15) *and am facing the following issues intermittently in
>>> some of the spark interpreters while trying to use spark-sql on it.
>>>
>>> *18/11/19 10:04:07 INFO yarn.Client: Submitting application
>>> application_1542587655772_35129 to ResourceManager*
>>> *18/11/19 10:04:07 INFO impl.YarnClientImpl: Submitted application
>>> application_1542587655772_35129*
>>> *18/11/19 10:04:08 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:08 INFO yarn.Client:*
>>> * client token: N/A*
>>> * diagnostics: N/A*
>>> * ApplicationMaster host: N/A*
>>> * ApplicationMaster RPC port: -1*
>>> * queue: root.zep*
>>> * start time: 1542621847537*
>>> * final status: UNDEFINED*
>>> * tracking
>>> URL: http://resource-manager-addr/proxy/application_1542587655772_35129/
>>> <http://c8-auto-hadoop-service-1.srv.media.net:8088/proxy/application_1542587655772_35129/>*
>>> * user: sarthak.sh*
>>> *18/11/19 10:04:09 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:10 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:11 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:12 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:13 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:14 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:15 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:16 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:17 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:18 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:19 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:20 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:21 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:22 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:23 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:24 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:25 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:26 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:27 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:28 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:29 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:30 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:31 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:32 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:33 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:34 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:35 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:36 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:37 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:38 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:39 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:40 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:41 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:42 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:43 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:44 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:45 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:46 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:47 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:48 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:49 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:50 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:51 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:52 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:53 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:54 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:55 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:56 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:57 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:58 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:04:59 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:00 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:01 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:02 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:03 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:04 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:05 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:06 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:07 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:08 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:09 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:10 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:11 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:12 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:13 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:14 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:15 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:16 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:17 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:18 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:19 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:20 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:21 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:22 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:23 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:24 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:25 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:26 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:27 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:28 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:29 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:30 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:31 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:32 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:33 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:34 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:35 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:36 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:37 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:38 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:39 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:40 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:41 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:42 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:43 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:44 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:45 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:46 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:47 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:48 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:49 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:50 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:51 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:52 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:53 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:54 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:55 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:56 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:57 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:58 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>> *18/11/19 10:05:59 INFO yarn.Client: Application report for
>>> application_1542587655772_35129 (state: ACCEPTED)*
>>>
>>> * at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:205)*
>>> * at
>>> org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:64)*
>>> * at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:111)*
>>> * at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:164)*
>>> * at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:132)*
>>> * at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:299)*
>>> * at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:407)*
>>> * at org.apache.zeppelin.scheduler.Job.run(Job.java:188)*
>>> * at
>>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)*
>>> * at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)*
>>> * at java.util.concurrent.FutureTask.run(FutureTask.java:266)*
>>> * at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)*
>>> * at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)*
>>> * at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*
>>> * at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*
>>> * at java.lang.Thread.run(Thread.java:748)*
>>>
>>> Any further submit to this interpreter will give null pointer exceptions
>>> due to the absence of an interpreter process.
>>> It looks like the interpreter driver process while getting submitted to
>>> yarn, is stuck in ACCEPTED state because of which we're not able to connect
>>> to the remote interpreter process. This happens even if there are resources
>>> on the cluster in yarn.
>>> Also I've tried increasing the *zeppelin.interpreter.connect.timeout *but
>>> that didn't help since the application is stuck in ACCEPTED state
>>> indefinitely and there are no logs available too.
>>> It'll be great if you can point me to something that can help. Also
>>> please do let me know if any configuration files are required for debugging
>>> this.
>>>
>>>
>>> Thanks and Regards
>>>
>>>
>>> *Sarthak Sharma*
>>> DevOps Engineer, Media.Net <http://media.net/>
>>> +918002228376 | sarthak.sh@media.net
>>> <http://en-gb.facebook.com/people/Sarthak-Sharma/100006006014244>
>>> <http://in.linkedin.com/in/sarthaksharma96>
>>>
>>>
>>>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Runtime Exception while trying to connect to remote spark
interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)
Posted by Jeff Zhang <zj...@gmail.com>.
If *zeppelin.interpreter.connect.timeout *is reached, but the yarn app is
still in ACCEPTED state, then this should be a bug. The yarn app should be
killed it it can not be created in the timeout threashold
Sarthak Sharma <sa...@media.net> 于2018年11月20日周二 下午4:47写道：
> Hey,
>
> Like you mentioned, I'm already using the *spark.yarn.queue* parameter,
> hence I know which yarn queue it is getting scheduled in and this queue has
> resources available for applications since other apps are also getting
> scheduled there.
> However, assuming the queue does NOT have resources for it to schedule
> within the given time frame causing it to throw an exception after the *zeppelin.interpreter.connect.timeout
> *is reached, the application should in any case get scheduled eventually
> which is not the case here. Interpreter driver process remains stuck in
> ACCEPTED state. Is there a change in the way it is implemented in this
> version ? Since we never experienced this on the previous one
> (zeppelin-0.7.3) where drivers would get scheduled eventually in their
> respective queues.
>
> On Tue, Nov 20, 2018, 7:29 AM Xun Liu <neliuxun@163.com wrote:
>
>> HI，Sarthak Sharma
>>
>> The log shows that the task submitted by spark-submmit has been waiting
>> for execution in the queue of YARN. Is there no resource for the queue of
>> YARN?
>> You can specify a queue with resources in the spark interpreter via the
>> spark.yarn.queue parameter.
>>
>>
>> 在 2018年11月19日，下午7:41，Sarthak Sharma <sa...@media.net> 写道：
>>
>> Hi,
>>
>> We already have a zeppelin-0.7.3 setup which runs fine and is in use
>> currently but we are looking into the yarn cluster mode support for spark
>> interpreter in zeppelin-0.8. I've built it from source from *branch-0.8
>> (As of Nov-15) *and am facing the following issues intermittently in
>> some of the spark interpreters while trying to use spark-sql on it.
>>
>> *18/11/19 10:04:07 INFO yarn.Client: Submitting application
>> application_1542587655772_35129 to ResourceManager*
>> *18/11/19 10:04:07 INFO impl.YarnClientImpl: Submitted application
>> application_1542587655772_35129*
>> *18/11/19 10:04:08 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:08 INFO yarn.Client:*
>> * client token: N/A*
>> * diagnostics: N/A*
>> * ApplicationMaster host: N/A*
>> * ApplicationMaster RPC port: -1*
>> * queue: root.zep*
>> * start time: 1542621847537*
>> * final status: UNDEFINED*
>> * tracking
>> URL: http://resource-manager-addr/proxy/application_1542587655772_35129/
>> <http://c8-auto-hadoop-service-1.srv.media.net:8088/proxy/application_1542587655772_35129/>*
>> * user: sarthak.sh*
>> *18/11/19 10:04:09 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:10 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:11 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:12 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:13 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:14 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:15 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:16 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:17 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:18 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:19 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:20 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:21 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:22 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:23 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:24 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:25 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:26 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:27 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:28 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:29 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:30 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:31 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:32 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:33 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:34 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:35 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:36 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:37 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:38 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:39 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:40 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:41 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:42 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:43 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:44 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:45 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:46 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:47 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:48 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:49 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:50 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:51 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:52 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:53 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:54 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:55 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:56 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:57 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:58 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:04:59 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:00 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:01 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:02 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:03 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:04 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:05 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:06 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:07 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:08 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:09 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:10 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:11 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:12 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:13 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:14 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:15 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:16 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:17 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:18 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:19 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:20 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:21 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:22 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:23 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:24 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:25 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:26 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:27 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:28 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:29 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:30 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:31 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:32 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:33 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:34 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:35 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:36 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:37 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:38 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:39 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:40 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:41 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:42 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:43 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:44 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:45 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:46 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:47 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:48 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:49 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:50 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:51 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:52 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:53 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:54 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:55 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:56 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:57 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:58 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>> *18/11/19 10:05:59 INFO yarn.Client: Application report for
>> application_1542587655772_35129 (state: ACCEPTED)*
>>
>> * at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:205)*
>> * at
>> org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:64)*
>> * at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:111)*
>> * at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:164)*
>> * at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:132)*
>> * at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:299)*
>> * at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:407)*
>> * at org.apache.zeppelin.scheduler.Job.run(Job.java:188)*
>> * at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)*
>> * at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)*
>> * at java.util.concurrent.FutureTask.run(FutureTask.java:266)*
>> * at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)*
>> * at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)*
>> * at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*
>> * at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*
>> * at java.lang.Thread.run(Thread.java:748)*
>>
>> Any further submit to this interpreter will give null pointer exceptions
>> due to the absence of an interpreter process.
>> It looks like the interpreter driver process while getting submitted to
>> yarn, is stuck in ACCEPTED state because of which we're not able to connect
>> to the remote interpreter process. This happens even if there are resources
>> on the cluster in yarn.
>> Also I've tried increasing the *zeppelin.interpreter.connect.timeout *but
>> that didn't help since the application is stuck in ACCEPTED state
>> indefinitely and there are no logs available too.
>> It'll be great if you can point me to something that can help. Also
>> please do let me know if any configuration files are required for debugging
>> this.
>>
>>
>> Thanks and Regards
>>
>>
>> *Sarthak Sharma*
>> DevOps Engineer, Media.Net <http://media.net/>
>> +918002228376 | sarthak.sh@media.net
>> <http://en-gb.facebook.com/people/Sarthak-Sharma/100006006014244>
>> <http://in.linkedin.com/in/sarthaksharma96>
>>
>>
>>
--
Best Regards
Jeff Zhang
Re: Runtime Exception while trying to connect to remote spark
interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)
Posted by Sarthak Sharma <sa...@media.net>.
Hey,
Like you mentioned, I'm already using the *spark.yarn.queue* parameter,
hence I know which yarn queue it is getting scheduled in and this queue has
resources available for applications since other apps are also getting
scheduled there.
However, assuming the queue does NOT have resources for it to schedule
within the given time frame causing it to throw an exception after the
*zeppelin.interpreter.connect.timeout
*is reached, the application should in any case get scheduled eventually
which is not the case here. Interpreter driver process remains stuck in
ACCEPTED state. Is there a change in the way it is implemented in this
version ? Since we never experienced this on the previous one
(zeppelin-0.7.3) where drivers would get scheduled eventually in their
respective queues.
On Tue, Nov 20, 2018, 7:29 AM Xun Liu <neliuxun@163.com wrote:
> HI，Sarthak Sharma
>
> The log shows that the task submitted by spark-submmit has been waiting
> for execution in the queue of YARN. Is there no resource for the queue of
> YARN?
> You can specify a queue with resources in the spark interpreter via the
> spark.yarn.queue parameter.
>
>
> 在 2018年11月19日，下午7:41，Sarthak Sharma <sa...@media.net> 写道：
>
> Hi,
>
> We already have a zeppelin-0.7.3 setup which runs fine and is in use
> currently but we are looking into the yarn cluster mode support for spark
> interpreter in zeppelin-0.8. I've built it from source from *branch-0.8
> (As of Nov-15) *and am facing the following issues intermittently in some
> of the spark interpreters while trying to use spark-sql on it.
>
> *18/11/19 10:04:07 INFO yarn.Client: Submitting application
> application_1542587655772_35129 to ResourceManager*
> *18/11/19 10:04:07 INFO impl.YarnClientImpl: Submitted application
> application_1542587655772_35129*
> *18/11/19 10:04:08 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:08 INFO yarn.Client:*
> * client token: N/A*
> * diagnostics: N/A*
> * ApplicationMaster host: N/A*
> * ApplicationMaster RPC port: -1*
> * queue: root.zep*
> * start time: 1542621847537*
> * final status: UNDEFINED*
> * tracking
> URL: http://resource-manager-addr/proxy/application_1542587655772_35129/
> <http://c8-auto-hadoop-service-1.srv.media.net:8088/proxy/application_1542587655772_35129/>*
> * user: sarthak.sh*
> *18/11/19 10:04:09 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:10 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:11 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:12 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:13 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:14 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:15 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:16 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:17 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:18 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:19 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:20 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:21 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:22 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:23 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:24 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:25 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:26 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:27 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:28 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:29 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:30 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:31 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:32 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:33 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:34 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:35 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:36 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:37 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:38 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:39 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:40 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:41 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:42 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:43 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:44 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:45 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:46 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:47 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:48 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:49 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:50 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:51 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:52 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:53 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:54 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:55 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:56 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:57 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:58 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:04:59 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:00 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:01 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:02 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:03 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:04 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:05 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:06 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:07 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:08 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:09 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:10 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:11 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:12 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:13 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:14 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:15 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:16 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:17 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:18 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:19 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:20 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:21 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:22 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:23 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:24 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:25 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:26 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:27 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:28 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:29 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:30 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:31 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:32 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:33 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:34 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:35 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:36 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:37 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:38 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:39 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:40 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:41 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:42 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:43 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:44 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:45 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:46 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:47 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:48 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:49 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:50 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:51 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:52 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:53 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:54 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:55 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:56 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:57 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:58 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
> *18/11/19 10:05:59 INFO yarn.Client: Application report for
> application_1542587655772_35129 (state: ACCEPTED)*
>
> * at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:205)*
> * at
> org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:64)*
> * at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:111)*
> * at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:164)*
> * at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:132)*
> * at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:299)*
> * at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:407)*
> * at org.apache.zeppelin.scheduler.Job.run(Job.java:188)*
> * at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)*
> * at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)*
> * at java.util.concurrent.FutureTask.run(FutureTask.java:266)*
> * at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)*
> * at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)*
> * at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*
> * at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*
> * at java.lang.Thread.run(Thread.java:748)*
>
> Any further submit to this interpreter will give null pointer exceptions
> due to the absence of an interpreter process.
> It looks like the interpreter driver process while getting submitted to
> yarn, is stuck in ACCEPTED state because of which we're not able to connect
> to the remote interpreter process. This happens even if there are resources
> on the cluster in yarn.
> Also I've tried increasing the *zeppelin.interpreter.connect.timeout *but
> that didn't help since the application is stuck in ACCEPTED state
> indefinitely and there are no logs available too.
> It'll be great if you can point me to something that can help. Also please
> do let me know if any configuration files are required for debugging this.
>
>
> Thanks and Regards
>
>
> *Sarthak Sharma*
> DevOps Engineer, Media.Net <http://media.net/>
> +918002228376 | sarthak.sh@media.net
> <http://en-gb.facebook.com/people/Sarthak-Sharma/100006006014244>
> <http://in.linkedin.com/in/sarthaksharma96>
>
>
>
Re: Runtime Exception while trying to connect to remote spark
interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)
Posted by Xun Liu <ne...@163.com>.
HI，Sarthak Sharma
The log shows that the task submitted by spark-submmit has been waiting for execution in the queue of YARN. Is there no resource for the queue of YARN?
You can specify a queue with resources in the spark interpreter via the spark.yarn.queue parameter.
> 在 2018年11月19日，下午7:41，Sarthak Sharma <sarthak.sh@media.net <ma...@media.net>> 写道：
>
> Hi,
>
> We already have a zeppelin-0.7.3 setup which runs fine and is in use currently but we are looking into the yarn cluster mode support for spark interpreter in zeppelin-0.8. I've built it from source from branch-0.8 (As of Nov-15) and am facing the following issues intermittently in some of the spark interpreters while trying to use spark-sql on it.
>
> 18/11/19 10:04:07 INFO yarn.Client: Submitting application application_1542587655772_35129 to ResourceManager
> 18/11/19 10:04:07 INFO impl.YarnClientImpl: Submitted application application_1542587655772_35129
> 18/11/19 10:04:08 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:08 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: root.zep
> start time: 1542621847537
> final status: UNDEFINED
> tracking URL: http://resource-manager-addr/proxy/application_1542587655772_35129/ <http://c8-auto-hadoop-service-1.srv.media.net:8088/proxy/application_1542587655772_35129/>
> user: sarthak.sh
> 18/11/19 10:04:09 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:10 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:11 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:12 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:13 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:14 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:15 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:16 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:17 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:18 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:19 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:20 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:21 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:22 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:23 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:24 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:25 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:26 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:27 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:28 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:29 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:30 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:31 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:32 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:33 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:34 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:35 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:36 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:37 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:38 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:39 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:40 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:41 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:42 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:43 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:44 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:45 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:46 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:47 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:48 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:49 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:50 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:51 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:52 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:53 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:54 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:55 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:56 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:57 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:58 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:04:59 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:00 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:01 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:02 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:03 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:04 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:05 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:06 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:07 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:08 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:09 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:10 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:11 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:12 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:13 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:14 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:15 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:16 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:17 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:18 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:19 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:20 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:21 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:22 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:23 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:24 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:25 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:26 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:27 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:28 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:29 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:30 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:31 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:32 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:33 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:34 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:35 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:36 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:37 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:38 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:39 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:40 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:41 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:42 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:43 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:44 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:45 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:46 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:47 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:48 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:49 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:50 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:51 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:52 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:53 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:54 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:55 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:56 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:57 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:58 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
> 18/11/19 10:05:59 INFO yarn.Client: Application report for application_1542587655772_35129 (state: ACCEPTED)
>
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:205)
> at org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:64)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:111)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:164)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:132)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:299)
> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:407)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
> at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> Any further submit to this interpreter will give null pointer exceptions due to the absence of an interpreter process.
> It looks like the interpreter driver process while getting submitted to yarn, is stuck in ACCEPTED state because of which we're not able to connect to the remote interpreter process. This happens even if there are resources on the cluster in yarn.
> Also I've tried increasing the zeppelin.interpreter.connect.timeout but that didn't help since the application is stuck in ACCEPTED state indefinitely and there are no logs available too.
> It'll be great if you can point me to something that can help. Also please do let me know if any configuration files are required for debugging this.
>
>
> Thanks and Regards
>
>
> Sarthak Sharma
> DevOps Engineer, Media.Net <http://media.net/>
> +918002228376 <tel:+918002228376> | sarthak.sh@media.net <ma...@media.net>
> <http://en-gb.facebook.com/people/Sarthak-Sharma/100006006014244> <http://in.linkedin.com/in/sarthaksharma96>