You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Akshat Bordia (Jira)" <ji...@apache.org> on 2021/09/08 04:44:00 UTC
[jira] [Commented] (LIVY-866) Optimize Yarn GetApplications Query
to prevent additional load on Yarn and Livy
[ https://issues.apache.org/jira/browse/LIVY-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411686#comment-17411686 ]
Akshat Bordia commented on LIVY-866:
------------------------------------
Create PR for review: [[LIVY-866] Optimizing Yarn GetApplications Query to prevent additional load on Yarn and Livy by akshatb1 · Pull Request #327 · apache/incubator-livy (github.com)|https://github.com/apache/incubator-livy/pull/327].
> Optimize Yarn GetApplications Query to prevent additional load on Yarn and Livy
> -------------------------------------------------------------------------------
>
> Key: LIVY-866
> URL: https://issues.apache.org/jira/browse/LIVY-866
> Project: Livy
> Issue Type: Improvement
> Components: Server
> Reporter: Akshat Bordia
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently Livy queries Yarn applications by applicationType : Spark. This will put heavy load on Yarn clusters if there are thousands or more Spark application in all states (running, finished, failed, queued etc.).
> A better approach would be to query the applications by tags in addition to job type since Livy only needs to track application with certain application tags. However, YarnClient does not expose any API to query applications by tags.
>
> We can extend the YarnClientImpl which would allow us to use protected rmClient in YarnClient and we can implement getApplications method with GetApplicationsRequest where we can include tags.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)