You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/08/01 22:01:04 UTC

[PR] [SPARK-58495] Use 7077 for `spark` named container port in master StatefulSet [spark-kubernetes-operator]

dongjoon-hyun opened a new pull request, #781:
URL: https://github.com/apache/spark-kubernetes-operator/pull/781

### What changes were proposed in this pull request?

This PR aims to use `7077` for the `spark` named container port in the master StatefulSet of `SparkClusterResourceSpec`, instead of `7070`.

### Why are the changes needed?

The `spark` named container port declaration was inconsistent with the actual master RPC port. The Spark standalone master listens on `7077` (the default of `SPARK_MASTER_PORT`), and all other references agree on `7077`:

- The master Service exposes port `7077` with `targetPort: spark` (a named port reference).
- Workers connect via `spark://<name>-master-svc:7077`.

Currently, this mismatch is harmless because the master Service is headless (`ClusterIP: None`), so workers resolve the pod IP via DNS and connect directly to `7077` without any Service port translation, and `containerPort` itself is a non-enforcing declaration. However, the stale value would surface as a real failure if:

1. A user overrides `masterSpec.serviceSpec` with a non-headless Service type, where `targetPort: spark` would resolve to `7070` and break connections.
2. Any Endpoints/EndpointSlice consumer (e.g., Prometheus service discovery, service mesh) reads the resolved port.
3. A NetworkPolicy referencing the named port is added on the master side.

### Does this PR introduce _any_ user-facing change?

No, for the default headless master Service. The generated master pod spec now declares the correct port, which only affects configurations that resolve the `spark` named port.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Fable 5

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58495] Use 7077 for `spark` named container port in master StatefulSet [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #781:
URL: https://github.com/apache/spark-kubernetes-operator/pull/781#issuecomment-5171451646

   Thank you, @peter-toth !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58495] Use 7077 for `spark` named container port in master StatefulSet [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #781:
URL: https://github.com/apache/spark-kubernetes-operator/pull/781#issuecomment-5154859310

   Could you review his PR when you have some time, @viirya ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58495] Use 7077 for `spark` named container port in master StatefulSet [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #781:
URL: https://github.com/apache/spark-kubernetes-operator/pull/781#issuecomment-5162382541

   Merged to main


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58495] Use 7077 for `spark` named container port in master StatefulSet [spark-kubernetes-operator]

Posted by "peter-toth (via GitHub)" <gi...@apache.org>.

peter-toth commented on PR #781:
URL: https://github.com/apache/spark-kubernetes-operator/pull/781#issuecomment-5163726694

   Late LGTM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58495] Use 7077 for `spark` named container port in master StatefulSet [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #781:
URL: https://github.com/apache/spark-kubernetes-operator/pull/781#issuecomment-5153904168

   All tests passed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58495] Use 7077 for `spark` named container port in master StatefulSet [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun closed pull request #781: [SPARK-58495] Use 7077 for `spark` named container port in master StatefulSet
URL: https://github.com/apache/spark-kubernetes-operator/pull/781


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58495] Use 7077 for `spark` named container port in master StatefulSet [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #781:
URL: https://github.com/apache/spark-kubernetes-operator/pull/781#issuecomment-5155829115

   Thank you, @viirya !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org