[I] [FEATURE] Reduce Kubernetes informer watch events per Kyuubi server [kyuubi]

["ruanwenjun (via GitHub)" <gi...@apache.org>]

ruanwenjun opened a new issue, #7756:
URL: https://github.com/apache/kyuubi/issues/7756

   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   ### Search before asking
   
   - [x] I have searched the existing issues and found no issue covering owner-scoped Kubernetes informer watches.
   
   ### Describe the feature
   
   When multiple Kyuubi server instances monitor the same Kubernetes context and namespace, every instance currently watches all Spark Driver Pods carrying `kyuubi-unique-tag`. When Spark UI URLs use Services, every instance also watches all Services in the namespace.
   
   As a result, each matching Pod or Service event is delivered to every Kyuubi server. The API server watch fan-out, network traffic, client deserialization, informer cache updates, audit logging, and application-state processing grow with the number of server replicas.
   
   Add an optional owner-scoped watch mode. When enabled, each Kyuubi server should:
   
   1. Derive a stable owner value from its hostname.
   2. Add the owner label to Spark Driver Pods and Driver Services submitted by that server.
   3. Filter its Pod and Service informers by the owner label.
   
   The option should be disabled by default to preserve existing behavior. Deployments enabling it must provide unique hostnames that remain stable across server restarts, such as StatefulSet Pod hostnames.
   
   ### Motivation
   
   Some Kyuubi clusters only need each server to observe applications it submitted. In these deployments, broadcasting every Driver Pod and Service event to every server adds unnecessary Kubernetes API server and Kyuubi client load.
   
   Owner-scoped watches keep the number of watch connections unchanged, but reduce event delivery and initial list response sizes by partitioning resources across Kyuubi servers.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org