You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Francis Lau <fr...@smartsheet.com> on 2015/07/30 00:47:56 UTC
Zeppelin on AWS EMR: Issue 2 - Zeppelin returns error "Yarn
application has already ended! It might have been killed or unable to launch
application master"
I ran into this next issue now. I ran a very simple Python command - print
date time, and I got the following error "org.apache.spark.SparkException:
Yarn application has already ended!"
Has anyone seen this error before? I have not done any additional
configuration Zeppelin, am I missing something in the configs?
Thanks
Francis
*Command in Zeppelin Notebook*
%pyspark
import datetime
print "Start Time: " + str(datetime.datetime.now())
*Error*
org.apache.spark.SparkException: Yarn application has already ended! It
might have been killed or unable to launch application master.
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:113)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:59)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
at
org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:301)
at
org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:423)
at
org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
at
org.apache.zeppelin.spark.PySparkInterpreter.getSparkInterpreter(PySparkInterpreter.java:353)
at
org.apache.zeppelin.spark.PySparkInterpreter.getJavaSparkContext(PySparkInterpreter.java:374)
at
org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:140)
at
org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:92)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:276)
at org.apache.zeppelin.scheduler.Job.run(Job.java:170)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Re: Zeppelin on AWS EMR: Issue 2 - Zeppelin returns error "Yarn
application has already ended! It might have been killed or unable to launch
application master"
Posted by Francis Lau <fr...@smartsheet.com>.
I looked at the log "zeppelin-interpreter-spark-hadoop-ip-172-31-57-88.log"
and here are the errors it shows. Has anyone seen the errors I highlighted
below?
Francis
INFO [2015-07-29 23:11:34,543] ({pool-2-thread-6}
Logging.scala[logInfo]:59) - Application report for
application_1438184471093_0008 (state: FAILED)
INFO [2015-07-29 23:11:34,543] ({pool-2-thread-6}
Logging.scala[logInfo]:59) -
client token: N/A
*diagnostics: Application application_XXX_0008 failed 2 times due to AM
Container for appattempt_XXX exited with exitCode: -1000*
For more detailed output, check application tracking page:
http://ip-XXX.ec2.internal:20888/proxy/application_1438184471093_0008/Then,
click on links to logs of each attempt.
*Diagnostics: java.io.IOException: Resource
file:/home/hadoop/zeppelin/interpreter/spark/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar
changed on src filesystem (expected 1438184142000, was 1438184225000*
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1438211490529
final status: FAILED
tracking URL: http://ip-XXX.ec2.internal:8088/cluster/app/application_XXX
user: hadoop
ERROR [2015-07-29 23:11:34,543] ({pool-2-thread-6} Job.java[run]:183) - Job
failed
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.spark.SparkException: Yarn application has already ended! It
might have been killed or unable to launch application master.
at
org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:76)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:92)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:276)
at org.apache.zeppelin.scheduler.Job.run(Job.java:170)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zeppelin.interpreter.InterpreterException:
org.apache.spark.SparkException: Yarn application has already ended! It
might have been killed or unable to launch application master.
at
org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:76)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
at
org.apache.zeppelin.spark.PySparkInterpreter.getSparkInterpreter(PySparkInterpreter.java:353)
at
org.apache.zeppelin.spark.PySparkInterpreter.getJavaSparkContext(PySparkInterpreter.java:374)
at
org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:140)
at
org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
... 12 more
Caused by: org.apache.spark.SparkException: Yarn application has already
ended! It might have been killed or unable to launch application master.
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:113)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:59)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
at
org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:301)
at
org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:423)
at
org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
... 17 more
INFO [2015-07-29 23:11:34,544] ({pool-2-thread-6}
SchedulerFactory.java[jobFinished]:138) - Job remoteInterpretJob_XXX
finished by scheduler interpreter_XXX
Application failed 2 times due to AM Container for exited with exitCode:
-1000
On Wed, Jul 29, 2015 at 3:47 PM, Francis Lau <fr...@smartsheet.com>
wrote:
> I ran into this next issue now. I ran a very simple Python command - print
> date time, and I got the following error "org.apache.spark.SparkException:
> Yarn application has already ended!"
>
> Has anyone seen this error before? I have not done any additional
> configuration Zeppelin, am I missing something in the configs?
>
> Thanks
>
> Francis
>
> *Command in Zeppelin Notebook*
> %pyspark
> import datetime
> print "Start Time: " + str(datetime.datetime.now())
>
>
> *Error*
> org.apache.spark.SparkException: Yarn application has already ended! It
> might have been killed or unable to launch application master.
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:113)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:59)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
> at
> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:301)
> at
> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:146)
> at
> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:423)
> at
> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
> at
> org.apache.zeppelin.spark.PySparkInterpreter.getSparkInterpreter(PySparkInterpreter.java:353)
> at
> org.apache.zeppelin.spark.PySparkInterpreter.getJavaSparkContext(PySparkInterpreter.java:374)
> at
> org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:140)
> at
> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:74)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:92)
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:276)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:170)
> at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:118)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
--
*Francis Lau* | *Smartsheet*
Senior Director of Product Intelligence
*c* 425-830-3889 (call/text)
francis.lau@smartsheet.com <ja...@smartsheet.com>