[jira] [Resolved] (LIVY-1072) Reduce Kubernetes API load in SparkKubernetesApp polling

["Gyorgy Gal (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LIVY-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gyorgy Gal resolved LIVY-1072.
------------------------------
    Fix Version/s: 1.0.0
         Assignee: Soumyadeep Roy
       Resolution: Fixed

PR: https://github.com/apache/livy/pull/545

> Reduce Kubernetes API load in SparkKubernetesApp polling
> --------------------------------------------------------
>
>                 Key: LIVY-1072
>                 URL: https://issues.apache.org/jira/browse/LIVY-1072
>             Project: Livy
>          Issue Type: Improvement
>          Components: Server
>            Reporter: Soumyadeep Roy
>            Assignee: Soumyadeep Roy
>            Priority: Major
>             Fix For: 1.0.0
>
>          Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> SparkKubernetesApp.getApplicationReport currently issues a single LIST returning every pod tagged with the session's app tag. On large jobs this deserializes hundreds of executor pods into memory on every poll cycle, even though application state is derived only from the driver pod.
> Proposed changes:
> - Narrow the driver lookup to spark-role=driver at the API server so the LIST returns a single pod.
> - Gate the separate executor LIST behind a new config flag livy.server.kubernetes.executor-tracking.enabled (default true, so existing deployments see no behavior change). Operators running large clusters can disable it to skip the executor LIST entirely. When disabled, per-executor log URLs and per-executor entries in session diagnostics are omitted; driver state tracking is unaffected.
> PR: https://github.com/apache/livy/pull/545



--
This message was sent by Atlassian Jira
(v8.20.10#820010)