You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Soumyadeep Roy (Jira)" <ji...@apache.org> on 2026/08/21 06:04:00 UTC
[jira] [Created] (LIVY-1072) Reduce Kubernetes API load in SparkKubernetesApp polling
Soumyadeep Roy created LIVY-1072:
------------------------------------
Summary: Reduce Kubernetes API load in SparkKubernetesApp polling
Key: LIVY-1072
URL: https://issues.apache.org/jira/browse/LIVY-1072
Project: Livy
Issue Type: Improvement
Components: Server
Reporter: Soumyadeep Roy
SparkKubernetesApp.getApplicationReport currently issues a single LIST returning every pod tagged with the session's app tag. On large jobs this deserializes hundreds of executor pods into memory on every poll cycle, even though application state is derived only from the driver pod.
Proposed changes:
- Narrow the driver lookup to spark-role=driver at the API server so the LIST returns a single pod.
- Gate the separate executor LIST behind a new config flag livy.server.kubernetes.executor-tracking.enabled (default true, so existing deployments see no behavior change). Operators running large clusters can disable it to skip the executor LIST entirely. When disabled, per-executor log URLs and per-executor entries in session diagnostics are omitted; driver state tracking is unaffected.
PR: https://github.com/apache/livy/pull/545
--
This message was sent by Atlassian Jira
(v8.20.10#820010)