You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yalin Dogu Sahin (Jira)" <ji...@apache.org> on 2026/08/10 11:31:00 UTC
[jira] [Updated] (SPARK-58691) NetworkPolicy generated for standalone SparkCluster workers blocks Prometheus metrics scraping
[ https://issues.apache.org/jira/browse/SPARK-58691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yalin Dogu Sahin updated SPARK-58691:
-------------------------------------
Description:
h2. Summary
The {{NetworkPolicy}} generated for standalone {{SparkCluster}} workers blocks Prometheus metrics scraping.
h2. Problem
For a standalone {{SparkCluster}} resource, the operator generates a worker {{NetworkPolicy}} ({{{}SparkClusterResourceSpec#buildWorkerNetworkPolicy{}}}) whose ingress rule only admits traffic from pods carrying the
cluster label ({{{}spark.operator/spark-cluster-name: <cluster>{}}}) or the driver label ({{{}spark-role: driver{}}}). No {{ports}} or {{namespaceSelector}} are specified.
Because a Prometheus scraper (e.g. Prometheus Operator's ServiceMonitor/PodMonitor) runs as an arbitrary pod without either label, its connection to a worker is dropped. As a result, *worker Prometheus metrics are
un-scrapable in standalone mode*.
* The worker web UI / Prometheus servlet runs on *port 8081* ({{{}/metrics/prometheus{}}}), but the worker pod is inside the NetworkPolicy-restricted set, so the scrape is rejected / times out.
* The {*}master has no {{NetworkPolicy}} at all{*}, so master metrics on *8080* remain reachable — the failure is specific to workers.
This is the same label-based allow-list mechanism addressed for *block-manager / task-result* traffic in [[SPARK-58649||http://example.com/]https://issues.apache.org/jira/browse/SPARK-58649 []|http://example.com/], but here it surfaces as blocked metrics scraping rather than task failures.
h2. Reproduction
# Deploy a {{SparkCluster}} (master + worker) and a Prometheus/ServiceMonitor targeting worker pods.
# Attempt to scrape {{{}<worker-pod>:8081/metrics/prometheus{}}}.
# The scrape fails because the worker pod is not an allowed ingress source in the {{NetworkPolicy}} named {{{}<cluster>-worker{}}}.
h2. Evidence
||Policy element||Value||
|Name|{{<cluster>-worker}}|
|Worker podSelector|{{spark-role: worker}} AND {{spark.operator/spark-cluster-name: <cluster>}}|
|Allowed ingress {{from}}|(a) pods with {{{}spark.operator/spark-cluster-name: <cluster>{}}}, (b) pods with {{spark-role: driver}}|
|Ports rule|none (all ports, but only to the {{from}} sources above)|
|Master|no {{NetworkPolicy}} — unrestricted|
h2. Suggested approach (for discussion)
A few options; maintainers' input is welcome:
# *Configurable / default-open* — make the worker NetworkPolicy ingress configurable in {{ClusterSpec}} (e.g. an explicit list of scraper-side labels/namespaces, or a {{networkPolicy.enabled}} toggle), preserving
current behavior by default.
# *Add a scrape source* — extend the {{from}} list with a well-known metrics-scraper label and/or a {{namespaceSelector}} for the monitoring namespace.
# *Port-scoped carve-out* — keep the allow-list but add an explicit {{ports}} ingress rule for 8081 so the intent is explicit and auditable.
Happy to prepare a patch once the preferred approach is agreed.
h2. Related
* SPARK-55099 — introduced the {{NetworkPolicy}} for {{{}SparkCluster{}}}. https://issues.apache.org/jira/browse/SPARK-55099
* SPARK-58649 — the same NetworkPolicy blocking worker ingress for attached drivers. https://issues.apache.org/jira/browse/SPARK-58649
was:
h2. Summary
The {{NetworkPolicy}} generated for standalone {{SparkCluster}} workers blocks Prometheus metrics scraping.
h2. Problem
For a standalone {{SparkCluster}} resource, the operator generates a worker {{NetworkPolicy}} ({{{}SparkClusterResourceSpec#buildWorkerNetworkPolicy{}}}) whose ingress rule only admits traffic from pods carrying the
cluster label ({{{}spark.operator/spark-cluster-name: <cluster>{}}}) or the driver label ({{{}spark-role: driver{}}}). No {{ports}} or {{namespaceSelector}} are specified.
Because a Prometheus scraper (e.g. Prometheus Operator's ServiceMonitor/PodMonitor) runs as an arbitrary pod without either label, its connection to a worker is dropped. As a result, *worker Prometheus metrics are
un-scrapable in standalone mode*.
* The worker web UI / Prometheus servlet runs on *port 8081* ({{{}/metrics/prometheus{}}}), but the worker pod is inside the NetworkPolicy-restricted set, so the scrape is rejected / times out.
* The {*}master has no {{NetworkPolicy}} at all{*}, so master metrics on *8080* remain reachable — the failure is specific to workers.
This is the same label-based allow-list mechanism addressed for *block-manager / task-result* traffic in [SPARK-58649|http://example.com]{-}{-}, but here it surfaces as blocked metrics scraping rather than task failures.
h2. Reproduction
# Deploy a {{SparkCluster}} (master + worker) and a Prometheus/ServiceMonitor targeting worker pods.
# Attempt to scrape {{{}<worker-pod>:8081/metrics/prometheus{}}}.
# The scrape fails because the worker pod is not an allowed ingress source in the {{NetworkPolicy}} named {{{}<cluster>-worker{}}}.
h2. Evidence
||Policy element||Value||
|Name|{{<cluster>-worker}}|
|Worker podSelector|{{spark-role: worker}} AND {{spark.operator/spark-cluster-name: <cluster>}}|
|Allowed ingress {{from}}|(a) pods with {{{}spark.operator/spark-cluster-name: <cluster>{}}}, (b) pods with {{spark-role: driver}}|
|Ports rule|none (all ports, but only to the {{from}} sources above)|
|Master|no {{NetworkPolicy}} — unrestricted|
h2. Suggested approach (for discussion)
A few options; maintainers' input is welcome:
# *Configurable / default-open* — make the worker NetworkPolicy ingress configurable in {{ClusterSpec}} (e.g. an explicit list of scraper-side labels/namespaces, or a {{networkPolicy.enabled}} toggle), preserving
current behavior by default.
# *Add a scrape source* — extend the {{from}} list with a well-known metrics-scraper label and/or a {{namespaceSelector}} for the monitoring namespace.
# *Port-scoped carve-out* — keep the allow-list but add an explicit {{ports}} ingress rule for 8081 so the intent is explicit and auditable.
Happy to prepare a patch once the preferred approach is agreed.
h2. Related
* SPARK-55099 — introduced the {{NetworkPolicy}} for {{{}SparkCluster{}}}. https://issues.apache.org/jira/browse/SPARK-55099
* SPARK-58649 — the same NetworkPolicy blocking worker ingress for attached drivers. https://issues.apache.org/jira/browse/SPARK-58649
> NetworkPolicy generated for standalone SparkCluster workers blocks Prometheus metrics scraping
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-58691
> URL: https://issues.apache.org/jira/browse/SPARK-58691
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: kubernetes-operator-1.0.0
> Reporter: Yalin Dogu Sahin
> Priority: Minor
>
> h2. Summary
> The {{NetworkPolicy}} generated for standalone {{SparkCluster}} workers blocks Prometheus metrics scraping.
> h2. Problem
> For a standalone {{SparkCluster}} resource, the operator generates a worker {{NetworkPolicy}} ({{{}SparkClusterResourceSpec#buildWorkerNetworkPolicy{}}}) whose ingress rule only admits traffic from pods carrying the
> cluster label ({{{}spark.operator/spark-cluster-name: <cluster>{}}}) or the driver label ({{{}spark-role: driver{}}}). No {{ports}} or {{namespaceSelector}} are specified.
> Because a Prometheus scraper (e.g. Prometheus Operator's ServiceMonitor/PodMonitor) runs as an arbitrary pod without either label, its connection to a worker is dropped. As a result, *worker Prometheus metrics are
> un-scrapable in standalone mode*.
> * The worker web UI / Prometheus servlet runs on *port 8081* ({{{}/metrics/prometheus{}}}), but the worker pod is inside the NetworkPolicy-restricted set, so the scrape is rejected / times out.
> * The {*}master has no {{NetworkPolicy}} at all{*}, so master metrics on *8080* remain reachable — the failure is specific to workers.
> This is the same label-based allow-list mechanism addressed for *block-manager / task-result* traffic in [[SPARK-58649||http://example.com/]https://issues.apache.org/jira/browse/SPARK-58649 []|http://example.com/], but here it surfaces as blocked metrics scraping rather than task failures.
> h2. Reproduction
> # Deploy a {{SparkCluster}} (master + worker) and a Prometheus/ServiceMonitor targeting worker pods.
> # Attempt to scrape {{{}<worker-pod>:8081/metrics/prometheus{}}}.
> # The scrape fails because the worker pod is not an allowed ingress source in the {{NetworkPolicy}} named {{{}<cluster>-worker{}}}.
> h2. Evidence
> ||Policy element||Value||
> |Name|{{<cluster>-worker}}|
> |Worker podSelector|{{spark-role: worker}} AND {{spark.operator/spark-cluster-name: <cluster>}}|
> |Allowed ingress {{from}}|(a) pods with {{{}spark.operator/spark-cluster-name: <cluster>{}}}, (b) pods with {{spark-role: driver}}|
> |Ports rule|none (all ports, but only to the {{from}} sources above)|
> |Master|no {{NetworkPolicy}} — unrestricted|
> h2. Suggested approach (for discussion)
> A few options; maintainers' input is welcome:
> # *Configurable / default-open* — make the worker NetworkPolicy ingress configurable in {{ClusterSpec}} (e.g. an explicit list of scraper-side labels/namespaces, or a {{networkPolicy.enabled}} toggle), preserving
> current behavior by default.
> # *Add a scrape source* — extend the {{from}} list with a well-known metrics-scraper label and/or a {{namespaceSelector}} for the monitoring namespace.
> # *Port-scoped carve-out* — keep the allow-list but add an explicit {{ports}} ingress rule for 8081 so the intent is explicit and auditable.
> Happy to prepare a patch once the preferred approach is agreed.
> h2. Related
> * SPARK-55099 — introduced the {{NetworkPolicy}} for {{{}SparkCluster{}}}. https://issues.apache.org/jira/browse/SPARK-55099
> * SPARK-58649 — the same NetworkPolicy blocking worker ingress for attached drivers. https://issues.apache.org/jira/browse/SPARK-58649
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org