[jira] [Commented] (SPARK-38668) Spark on Kubernetes: add separate pod watcher service to reduce pressure on K8s API server

["Yikun Jiang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-38668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533177#comment-17533177 ] 

Yikun Jiang commented on SPARK-38668:
-------------------------------------

Related fix:

[https://github.com/apache/spark/pull/36433]

SPARK-36462

> Spark on Kubernetes: add separate pod watcher service to reduce pressure on K8s API server
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-38668
>                 URL: https://issues.apache.org/jira/browse/SPARK-38668
>             Project: Spark
>          Issue Type: New Feature
>          Components: Kubernetes
>    Affects Versions: 3.2.1
>            Reporter: BoYang
>            Priority: Major
>
> Spark driver will listen to all pods events to manage its executor pods. This will cause pressure on Kubernetes API server in a large cluster, because there will be many drivers connect to the API server and watch for the pods.
>  
> An alternative is to have a separate service to listen and watch all pod events. Then each Spark driver only connects to that service to get pod events. This will reduce the load on Kubernetes API server.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org