You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Naksh Arora (Jira)" <ji...@apache.org> on 2020/10/23 09:42:00 UTC
[jira] [Updated] (LIVY-798) Facing intermittent issue while submit
job to Livy
[ https://issues.apache.org/jira/browse/LIVY-798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Naksh Arora updated LIVY-798:
-----------------------------
Description:
While submitting spark jobs to Apache Livy via Airflow, we are intermittently getting the below Exception. Can someone please help in extracting the root cause for this:
ERROR utils.SparkYarnApp: Error whiling refreshing YARN state
java.lang.IllegalStateException: No YARN application is found with tag livy-batch-82850-44ej2bgm in 300 seconds. This may be because 1) spark-submit fail to submit application to YARN; or 2) YARN cluster doesn't have enough resources to start the application in time. Please check Livy log and YARN log to know the details.
at org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:206)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:267)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:264)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:264)
at org.apache.livy.Utils$$anon$1.run(Utils.scala:97)
INFO utils.LineBufferedStream: Warning: Skip remote jar <jar_path>
Exception in thread "Thread-57893" java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:46)
was:
While submitting spark jobs to Apache Livy via Airflow, we are intermittently getting the below Exception. Can someone please help in extracting the root cause for this:
2020-10-23 06:05:55,422 ERROR utils.SparkYarnApp: Error whiling refreshing YARN state
java.lang.IllegalStateException: No YARN application is found with tag livy-batch-82850-44ej2bgm in 300 seconds. This may be because 1) spark-submit fail to submit application to YARN; or 2) YARN cluster doesn't have enough resources to start the application in time. Please check Livy log and YARN log to know the details.
at org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:206)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:267)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:264)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:264)
at org.apache.livy.Utils$$anon$1.run(Utils.scala:97)
2020-10-23 06:05:55,528 INFO utils.LineBufferedStream: Warning: Skip remote jar <jar_path>
Exception in thread "Thread-57893" java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:46)
> Facing intermittent issue while submit job to Livy
> --------------------------------------------------
>
> Key: LIVY-798
> URL: https://issues.apache.org/jira/browse/LIVY-798
> Project: Livy
> Issue Type: Bug
> Reporter: Naksh Arora
> Priority: Major
>
> While submitting spark jobs to Apache Livy via Airflow, we are intermittently getting the below Exception. Can someone please help in extracting the root cause for this:
> ERROR utils.SparkYarnApp: Error whiling refreshing YARN state
> java.lang.IllegalStateException: No YARN application is found with tag livy-batch-82850-44ej2bgm in 300 seconds. This may be because 1) spark-submit fail to submit application to YARN; or 2) YARN cluster doesn't have enough resources to start the application in time. Please check Livy log and YARN log to know the details.
> at org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:206)
> at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:267)
> at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:264)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:264)
> at org.apache.livy.Utils$$anon$1.run(Utils.scala:97)
> INFO utils.LineBufferedStream: Warning: Skip remote jar <jar_path>
> Exception in thread "Thread-57893" java.io.IOException: Stream closed
> at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:46)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)