You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Soumyadeep Roy (Jira)" <ji...@apache.org> on 2026/08/21 06:04:00 UTC

[jira] [Created] (LIVY-1072) Reduce Kubernetes API load in SparkKubernetesApp polling

Soumyadeep Roy created LIVY-1072:
------------------------------------

             Summary: Reduce Kubernetes API load in SparkKubernetesApp polling
                 Key: LIVY-1072
                 URL: https://issues.apache.org/jira/browse/LIVY-1072
             Project: Livy
          Issue Type: Improvement
          Components: Server
            Reporter: Soumyadeep Roy


SparkKubernetesApp.getApplicationReport currently issues a single LIST returning every pod tagged with the session's app tag. On large jobs this deserializes hundreds of executor pods into memory on every poll cycle, even though application state is derived only from the driver pod.

Proposed changes:
- Narrow the driver lookup to spark-role=driver at the API server so the LIST returns a single pod.
- Gate the separate executor LIST behind a new config flag livy.server.kubernetes.executor-tracking.enabled (default true, so existing deployments see no behavior change). Operators running large clusters can disable it to skip the executor LIST entirely. When disabled, per-executor log URLs and per-executor entries in session diagnostics are omitted; driver state tracking is unaffected.

PR: https://github.com/apache/livy/pull/545



--
This message was sent by Atlassian Jira
(v8.20.10#820010)