You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Ruslan Dautkhanov (JIRA)" <ji...@apache.org> on 2019/02/04 17:12:00 UTC
[jira] [Comment Edited] (LIVY-541) Multiple Livy servers submitting
to Yarn results in LivyException: Session is finished ... No YARN
application is found with tag livy-session-197-uveqmqyj in 300 seconds.
Please check your cluster status, it is may be very busy
[ https://issues.apache.org/jira/browse/LIVY-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760029#comment-16760029 ]
Ruslan Dautkhanov edited comment on LIVY-541 at 2/4/19 5:11 PM:
----------------------------------------------------------------
It might be a one-liner change here
[https://github.com/apache/incubator-livy/blob/56c76bc2d4563593edce062a563603fe63e5a431/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L99]
Change
|builderProperties.getOrElseUpdate("spark.app.name", s"livy-session-$id")|
to
builderProperties.getOrElseUpdate("spark.app.name", apptag)
I don't thing there is a drawback for doing this..
A better approach might be to add a configuration knob `livy.yarn.session.prefix` so you could specify prefixes for different Livy servers and they wouldn't overlap.
was (Author: tagar):
It might be a one-liner change here
[https://github.com/apache/incubator-livy/blob/56c76bc2d4563593edce062a563603fe63e5a431/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L99]
Change
|builderProperties.getOrElseUpdate("spark.app.name", s"livy-session-$id")|
to
builderProperties.getOrElseUpdate("spark.app.name", apptag)
I don't thing there is a drawback for doing there..
A better approach might be to add a configuration knob `livy.yarn.session.prefix` so you could specify prefixes for different Livy servers and they wouldn't overlap.
> Multiple Livy servers submitting to Yarn results in LivyException: Session is finished ... No YARN application is found with tag livy-session-197-uveqmqyj in 300 seconds. Please check your cluster status, it is may be very busy
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: LIVY-541
> URL: https://issues.apache.org/jira/browse/LIVY-541
> Project: Livy
> Issue Type: Bug
> Components: Server
> Affects Versions: 0.5.0
> Environment: Hortonworks HDP 2.6
> Reporter: Hari Sekhon
> Priority: Critical
>
> It appears Livy doesn't differentiate sessions properly in Yarn causing errors when running multiple Livy servers behind a load balancer for HA / performance scaling on the same Hadoop cluster.
> Each livy server uses monotonically incrementing session IDs with a random suffix but it appears that the random suffix isn't passed to Yarn which results in the following errors on the Livy server which is further behind in session numbers because it appears to find the session with the same number has already finished (submitted earlier by a different user on another Livy server as seen in Yarn RM UI):
> {code:java}
> org.apache.zeppelin.livy.LivyException: Session 197 is finished, appId: null, log: [ at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2887), at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2904), at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511), at java.util.concurrent.FutureTask.run(FutureTask.java:266), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617), at java.lang.Thread.run(Thread.java:748),
> YARN Diagnostics: , java.lang.Exception: No YARN application is found with tag livy-session-197-uveqmqyj in 300 seconds. Please check your cluster status, it is may be very busy., org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:182) org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:239) org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:236) scala.Option.getOrElse(Option.scala:120) org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:236) org.apache.livy.Utils$$anon$1.run(Utils.scala:94)]
> at org.apache.zeppelin.livy.BaseLivyInterpreter.createSession(BaseLivyInterpreter.java:300)
> at org.apache.zeppelin.livy.BaseLivyInterpreter.initLivySession(BaseLivyInterpreter.java:184)
> at org.apache.zeppelin.livy.LivySharedInterpreter.open(LivySharedInterpreter.java:57)
> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivySharedInterpreter(BaseLivyInterpreter.java:165)
> at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:139)
> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493)
> at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)