You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by "nileshrathi345 (via GitHub)" <gi...@apache.org> on 2026/06/23 10:30:40 UTC
[PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
nileshrathi345 opened a new pull request, #529:
URL: https://github.com/apache/livy/pull/529
## What changes were proposed in this pull request?
This PR introduces support for configuring a default YARN queue at the Livy server level via a new configuration property: `livy.spark.yarn.queue.`
Previously, if a client did not explicitly specify a YARN queue parameter (queue) in their session creation payload, the session would fall back to the global Hadoop cluster's default YARN queue. This change allows cluster administrators to isolate Livy-generated workloads into a specific default queue without requiring end-users to pass it manually in every API request.
## How was this patch tested?
This patch has been thoroughly validated using both newly added automated unit tests and manual API validation:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3485092364
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala:
##########
@@ -137,6 +138,7 @@ object BatchSession extends Logging {
m.owner,
m.proxyUser,
sessionStore,
+ None,
Review Comment:
It might be possible to remove this and change the unit test in BatchSessionSpec to verify that the queue is set via the mockApp. I am not entirely sure though, but if that is the case this PR could be further simplified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3464803010
##########
server/src/main/scala/org/apache/livy/LivyConf.scala:
##########
@@ -41,6 +41,7 @@ object LivyConf {
val TEST_MODE = ClientConf.TEST_MODE
+ val SPARK_YARN_QUEUE = Entry("livy.spark.yarn.queue", null)
Review Comment:
thanks..!!
Documented config to `livy.conf.template`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on PR #529:
URL: https://github.com/apache/livy/pull/529#issuecomment-4778918128
@ArnavBalyan, @gyogal and @roczei could you please take a look when possible thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3469743119
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSessionServlet.scala:
##########
@@ -33,6 +33,7 @@ case class BatchSessionView(
state: String,
appId: Option[String],
appInfo: AppInfo,
+ queue: Option[String],
Review Comment:
I think this change is only needed if we want to be able to return which queue the batch job has been submitted into. If so, the Batch object may need to be updated at https://livy.apache.org/docs/latest/rest-api.html
But this is something LIVY-1059 does not currently mention, which is only about defaulting the value for the queue parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3482757368
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala:
##########
@@ -137,6 +138,7 @@ object BatchSession extends Logging {
m.owner,
m.proxyUser,
sessionStore,
+ None,
Review Comment:
Currently, queue is primarily used during session creation and tracking. However, passing None during recover means that if the Livy server restarts, any recovered batch session will permanently lose its original queue context.
To maintain that have added to the state in previous version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on PR #529:
URL: https://github.com/apache/livy/pull/529#issuecomment-4811258730
@samuhasavak5 could you please take a look and approve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3469731412
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala:
##########
@@ -40,6 +40,7 @@ case class BatchRecoveryMetadata(
appTag: String,
owner: String,
proxyUser: Option[String],
+ queue: Option[String],
Review Comment:
Why does the recovery state store save the queue the batch has been started with? Do we want to be able to show it after a Livy server restart? I think this may be independent new functionality not covered by LIVY-1059 (which is only about allowing the user to set a default YARN queue)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3472216894
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSessionServlet.scala:
##########
@@ -33,6 +33,7 @@ case class BatchSessionView(
state: String,
appId: Option[String],
appInfo: AppInfo,
+ queue: Option[String],
Review Comment:
As per previous comment now No queue field has been added to BatchRecoveryMetadata, nothing is persisted to the state store, and the REST API Batch object is unchanged. The only observable behaviour change from LIVY-1059 is that batch.queue returns the resolved value during a live session created with the default conf fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3462800737
##########
server/src/main/scala/org/apache/livy/LivyConf.scala:
##########
@@ -41,6 +41,7 @@ object LivyConf {
val TEST_MODE = ClientConf.TEST_MODE
+ val SPARK_YARN_QUEUE = Entry("livy.spark.yarn.queue", null)
Review Comment:
Can we add documentation for this config to `livy.conf.template` too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3472208270
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala:
##########
@@ -40,6 +40,7 @@ case class BatchRecoveryMetadata(
appTag: String,
owner: String,
proxyUser: Option[String],
+ queue: Option[String],
Review Comment:
Thanks for the clarification. Agreed that persisting queue in BatchRecoveryMetadata is out of scope for LIVY-1059.
The reason queue appears in recover() is purely a constructor consistency concern — since BatchSession now accepts queue as a constructor parameter (added in create() to store the resolved value for the lifetime of the session), recover() must also pass a value to satisfy that constructor. In this case, can now pass **None** OR explicitly, meaning no queue is stored or exposed for recovered sessions, which aligns with point that recovery does not need to know or show the original queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal merged PR #529:
URL: https://github.com/apache/livy/pull/529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3472208270
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala:
##########
@@ -40,6 +40,7 @@ case class BatchRecoveryMetadata(
appTag: String,
owner: String,
proxyUser: Option[String],
+ queue: Option[String],
Review Comment:
Thanks for the clarification. Agreed that persisting queue in BatchRecoveryMetadata is out of scope for LIVY-1059.
The reason queue appears in recover() is purely a constructor consistency concern — since BatchSession now accepts queue as a constructor parameter (added in create() to store the resolved value for the lifetime of the session), recover() must also pass a value to satisfy that constructor. In this case, can now pass **None** explicitly, meaning no queue is stored or exposed for recovered sessions, which aligns with point that recovery does not need to know or show the original queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3482434316
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala:
##########
@@ -137,6 +138,7 @@ object BatchSession extends Logging {
m.owner,
m.proxyUser,
sessionStore,
+ None,
Review Comment:
Now I understand why the queue was added to the state store in the previous version, it was so that this value could be populated correctly. Is the queue in BatchSessions used by anything else other than the unit test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3482434316
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala:
##########
@@ -137,6 +138,7 @@ object BatchSession extends Logging {
m.owner,
m.proxyUser,
sessionStore,
+ None,
Review Comment:
I think I now understand why the queue was added to the state store in the previous version, it was so that this value could be populated correctly. Is the queue in BatchSessions used by anything else other than the unit test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on code in PR #529:
URL: https://github.com/apache/livy/pull/529#discussion_r3496619688
##########
server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala:
##########
@@ -137,6 +138,7 @@ object BatchSession extends Logging {
m.owner,
m.proxyUser,
sessionStore,
+ None,
Review Comment:
I have removed the parameter from BatchSession.recover and updated the unit tests in BatchSessionSpec to verify that the queue configuration is correctly resolved via the mockApp lifecycle instead. This keeps the recovery metadata schema unchanged and simplifies the PR as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org