You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yalin Dogu Sahin (Jira)" <ji...@apache.org> on 2026/08/10 12:51:00 UTC

[jira] [Comment Edited] (SPARK-58691) NetworkPolicy generated for standalone SparkCluster workers blocks Prometheus metrics scraping

    [ https://issues.apache.org/jira/browse/SPARK-58691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103373#comment-18103373 ] 

Yalin Dogu Sahin edited comment on SPARK-58691 at 8/10/26 12:50 PM:
--------------------------------------------------------------------

PR: https://github.com/apache/spark-kubernetes-operator/pull/794 — adds a port-scoped ingress rule to the worker NetworkPolicy so metrics
  scrapers aren't blocked, without loosening the RPC/shuffle/block-manager ports.


was (Author: JIRAUSER313566):
PR is available https://github.com/apache/spark-kubernetes-operator/pull/794

> NetworkPolicy generated for standalone SparkCluster workers blocks Prometheus metrics scraping
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58691
>                 URL: https://issues.apache.org/jira/browse/SPARK-58691
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: kubernetes-operator-1.0.0
>            Reporter: Yalin Dogu Sahin
>            Priority: Minor
>              Labels: pull-request-available
>
> h2. Problem
>   For a standalone {{SparkCluster}} resource, the operator generates a worker {{NetworkPolicy}} ({{SparkClusterResourceSpec#buildWorkerNetworkPolicy}}) whose ingress rule only admits traffic from pods carrying the
>   cluster label ({{spark.operator/spark-cluster-name: <cluster>}}) or the driver label ({{spark-role: driver}}). No {{ports}} or {{namespaceSelector}} are specified.
>   Because a Prometheus scraper (e.g. Prometheus Operator's ServiceMonitor/PodMonitor) runs as an arbitrary pod without either label, its connection to a worker is dropped. As a result, *worker Prometheus metrics are
>   un-scrapable in standalone mode*.
>   * The worker web UI / Prometheus servlet runs on *port 8081* ({{/metrics/prometheus}}), but the worker pod is inside the NetworkPolicy-restricted set, so the scrape is rejected / times out.
>   * The *master has no {{NetworkPolicy}} at all*, so master metrics on *8080* remain reachable — the failure is specific to workers.
>   This is the same label-based allow-list mechanism addressed for *block-manager / task-result* traffic in [SPARK-58649|https://issues.apache.org/jira/browse/SPARK-58649], but here it surfaces as blocked metrics scraping rather than task failures.
>   h2. Reproduction
>   # Deploy a {{SparkCluster}} (master + worker) and a Prometheus/ServiceMonitor targeting worker pods.
>   # Attempt to scrape {{<worker-pod>:8081/metrics/prometheus}}.
>   # The scrape fails because the worker pod is not an allowed ingress source in the {{NetworkPolicy}} named {{<cluster>-worker}}.
>   h2. Evidence
>   || Policy element || Value ||
>   | Name | {{<cluster>-worker}} |
>   | Worker podSelector | {{spark-role: worker}} AND {{spark.operator/spark-cluster-name: <cluster>}} |
>   | Allowed ingress {{from}} | (a) pods with {{spark.operator/spark-cluster-name: <cluster>}}, (b) pods with {{spark-role: driver}} |
>   | Ports rule | none (all ports, but only to the {{from}} sources above) |
>   | Master | no {{NetworkPolicy}} — unrestricted |
>   h2. Suggested approach (for discussion)
>   A few options; maintainers' input is welcome:
>   # *Configurable / default-open* — make the worker NetworkPolicy ingress configurable in {{ClusterSpec}} (e.g. an explicit list of scraper-side labels/namespaces, or a {{networkPolicy.enabled}} toggle), preserving
>   current behavior by default.
>   # *Add a scrape source* — extend the {{from}} list with a well-known metrics-scraper label and/or a {{namespaceSelector}} for the monitoring namespace.
>   # *Port-scoped carve-out* — keep the allow-list but add an explicit {{ports}} ingress rule for 8081 so the intent is explicit and auditable.
>   Happy to prepare a patch once the preferred approach is agreed.
>   h2. Additional context (verified against code on main)
>   * The operator has no Prometheus/metrics configuration for Spark pods today. {{MetricsSystem}}/{{MetricsService}} expose only the operator's own metrics. The worker/master PrometheusServlet on 8081/8080 is not
>   wired up by the operator — it only exists if the user enables Spark's built-in PrometheusServlet via their own {{spark.metrics.conf}}.
>   * {{ClusterSpec}} currently has *no* {{networkPolicy}} (and no metrics) field. So a fully configurable approach (option 1) introduces new API surface and requires regenerating the CRD; options 2 and 3 (label/port
>   carve-out) are self-contained in the worker NetworkPolicy builder and need no CRD change.
>   * The core issue is unchanged: the worker NetworkPolicy's ingress allow-list (cluster label + {{spark-role: driver}}, no ports) excludes arbitrary scraper pods.
>   h2. Related
>   * [SPARK-55099|https://issues.apache.org/jira/browse/SPARK-55099] — introduced the {{NetworkPolicy}} for {{SparkCluster}}. 
>   * [SPARK-58649|https://issues.apache.org/jira/browse/SPARK-58649] — the same NetworkPolicy blocking worker ingress for attached drivers. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org