You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Sam Brougher (JIRA)" <ji...@apache.org> on 2019/04/05 16:04:00 UTC
[jira] [Created] (LIVY-586) When a batch fails on startup, Livy
continues to report the batch as "starting", even though it has failed
Sam Brougher created LIVY-586:
---------------------------------
Summary: When a batch fails on startup, Livy continues to report the batch as "starting", even though it has failed
Key: LIVY-586
URL: https://issues.apache.org/jira/browse/LIVY-586
Project: Livy
Issue Type: Bug
Components: Batch
Affects Versions: 0.5.0
Environment: AWS EMR, Livy submits batches to YARN in cluster mode
Reporter: Sam Brougher
When starting a Livy batch, I accidentally provided it a jar location in S3 that did not exist. Livy then continued to report that the job was "starting", even though it had clearly failed.
stdout:
{code:java}
2019-04-05 11:24:18,149 [main] WARN org.apache.hadoop.util.NativeCodeLoader [appName=] [jobId=] [clusterId=] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Warning: Skip remote jar s3://dev-dp-local/jars/develop-fix/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar.
2019-04-05 11:24:19,152 [main] INFO org.apache.hadoop.yarn.client.RMProxy [appName=] [jobId=] [clusterId=] - Connecting to ResourceManager at ip-10-25-30-127.dev.cainc.internal/10.25.30.127:8032
2019-04-05 11:24:19,453 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Requesting a new application from cluster with 6 NodeManagers
2019-04-05 11:24:19,532 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Verifying our application has not requested more than the maximum memory capability of the cluster (54272 MB per container)
2019-04-05 11:24:19,533 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Will allocate AM container, with 9011 MB memory including 819 MB overhead
2019-04-05 11:24:19,534 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Setting up container launch context for our AM
2019-04-05 11:24:19,537 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Setting up the launch environment for our AM container
2019-04-05 11:24:19,549 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Preparing resources for our AM container
2019-04-05 11:24:21,059 [main] WARN org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2019-04-05 11:24:23,790 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Uploading resource file:/mnt/tmp/spark-b4e4a760-77a3-4554-a3f3-c3f82675d865/__spark_libs__3639879082942366045.zip -> hdfs://ip-10-25-30-127.dev.cainc.internal:8020/user/livy/.sparkStaging/application_1554234858331_0222/__spark_libs__3639879082942366045.zip
2019-04-05 11:24:26,817 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Uploading resource s3://dev-dp-local/jars/develop-fix/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar -> hdfs://ip-10-25-30-127.dev.cainc.internal:8020/user/livy/.sparkStaging/application_1554234858331_0222/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar
2019-04-05 11:24:26,940 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Deleted staging directory hdfs://ip-10-25-30-127.dev.cainc.internal:8020/user/livy/.sparkStaging/application_1554234858331_0222
Exception in thread "main" java.io.FileNotFoundException: No such file or directory 's3://dev-dp-local/jars/develop-fix/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar'
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:805)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.getFileStatus(EmrFileSystem.java:536)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:356)
at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:478)
at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10.apply(Client.scala:577)
at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10.apply(Client.scala:576)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:576)
at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:869)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1152)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2019-04-05 11:24:26,964 [pool-1-thread-1] INFO org.apache.spark.util.ShutdownHookManager [appName=] [jobId=] [clusterId=] - Shutdown hook called
2019-04-05 11:24:26,965 [pool-1-thread-1] INFO org.apache.spark.util.ShutdownHookManager [appName=] [jobId=] [clusterId=] - Deleting directory /mnt/tmp/spark-aa8e8eff-ca2c-4358-a24f-19eb3863ef8f
2019-04-05 11:24:26,966 [pool-1-thread-1] INFO org.apache.spark.util.ShutdownHookManager [appName=] [jobId=] [clusterId=] - Deleting directory /mnt/tmp/spark-b4e4a760-77a3-4554-a3f3-c3f82675d865
{code}
stderror is empty
YARN Diagnostics eventually warns that the tag for the batch can't be found after 900 seconds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)