You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/10/07 05:07:15 UTC
[PR] [SPARK-60020] Fix `KubernetesMetricsInterceptor` exceptions for `namespaces` paths and out-of-range response codes [spark-kubernetes-operator]
dongjoon-hyun opened a new pull request, #947:
URL: https://github.com/apache/spark-kubernetes-operator/pull/947
### What changes were proposed in this pull request?
This PR aims to fix two robustness bugs of `KubernetesMetricsInterceptor`.
- `parseNamespaceScopedResource` returns `Optional.empty()` unless the request path has `namespaces/<namespace>/<resource>` segments. It matches the `namespaces` path segment instead of the first `namespaces` substring.
- `updateResponseMetrics` marks a response code group meter, `1xx` to `5xx`, only when the response code belongs to one of them.
### Why are the changes needed?
`parseNamespaceScopedResource` assumes that `<namespace>/<resource>` always follows the first `namespaces` substring of the path. It throws for the following paths.
| Path | Request | Exception |
| --- | --- | --- |
| `/api/v1/namespaces` | List or watch of namespaces | `StringIndexOutOfBoundsException` |
| `/api/v1/namespaces/spark-system` | Get of a namespace | `ArrayIndexOutOfBoundsException` |
| `/apis/rbac.authorization.k8s.io/v1/clusterroles/namespaces-reader` | Get of a cluster-scoped resource whose name contains `namespaces` | `ArrayIndexOutOfBoundsException` |
Since fabric8 7.8.0 `StandardHttpClient` calls `Interceptor.before(...)` synchronously, the exception fails the request itself before it is sent, e.g. `client.namespaces().list()` throws `StringIndexOutOfBoundsException`. The operator does not call these APIs today, so this bug is latent.
`updateResponseMetrics` uses `response.code() / 100 - 1` as the index of the group meters. Netty accepts any non-negative status code, e.g. `HTTP/1.1 999` from a proxy, and fabric8 passes it to `Interceptor.after(...)`. For a code outside of `100` to `599`, `responseCodeGroupMeters.get(...)` throws `IndexOutOfBoundsException` on the Vert.x event loop. fabric8 calls `after(...)` of all interceptors in one loop of a `thenAccept` callback whose future it drops, so the exception silently skips `after(...)` of the interceptors registered after this one.
Both bugs exist in the latest release.
- https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0 (2026-07-26)
### Does this PR introduce _any_ user-facing change?
No. The metrics of the requests which the operator sends are unchanged. For a response code outside of `100` to `599`, `kubernetes.client.http.response` and `kubernetes.client.http.response.<code>` are recorded as before, without the exception.
### How was this patch tested?
Pass the CIs with the newly added test cases. Without this fix, `testParseNamespaceScopedResource` fails with `StringIndexOutOfBoundsException` and `testResponseCodeGroups` fails with `IndexOutOfBoundsException`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60020] Fix `KubernetesMetricsInterceptor` exceptions for `namespaces` paths and out-of-range response codes [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #947:
URL: https://github.com/apache/spark-kubernetes-operator/pull/947#issuecomment-6041431692
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60020] Fix `KubernetesMetricsInterceptor` exceptions for `namespaces` paths and out-of-range response codes [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #947:
URL: https://github.com/apache/spark-kubernetes-operator/pull/947#issuecomment-6041421602
Thank you again, @peter-toth !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60020] Fix `KubernetesMetricsInterceptor` exceptions for `namespaces` paths and out-of-range response codes [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #947:
URL: https://github.com/apache/spark-kubernetes-operator/pull/947#discussion_r4204342231
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/source/KubernetesMetricsInterceptor.java:
##########
@@ -217,14 +221,14 @@ private Meter getMeterByResponseCode(int code) {
* Parses the given path to extract namespace-scoped resource information.
*
* @param path The request path.
- * @return An Optional containing a Pair of namespace and resource name, or empty if not found.
+ * @return An Optional containing a Pair of namespace and resource name, or empty if the path has
+ * no {@code namespaces/<namespace>/<resource>} segments.
*/
public Optional<Pair<String, String>> parseNamespaceScopedResource(String path) {
- if (path.contains(NAMESPACES)) {
- int index = path.indexOf(NAMESPACES) + NAMESPACES.length();
- String namespaceAndResources = path.substring(index + 1);
- String[] parts = namespaceAndResources.split("/");
- return Optional.of(Pair.of(parts[0], parts[1]));
+ List<String> segments = Arrays.asList(path.split("/"));
Review Comment:
**Finding 1.** The description says "The operator does not call these APIs today, so this bug is latent." On `main` it does, for the third row of the table. With `spark.kubernetes.operator.kueue.enabled`, these calls GET a cluster-scoped object by a name which the user picks:
- `KueueWorkloadPriority.java:95` reads the `WorkloadPriorityClass` of the `kueue.x-k8s.io/priority-class` label.
- `KueueWorkloadPriority.java:141` reads the `PriorityClass` of a pod template.
- `KueueWorkloadUtils.java:591` reads each `ResourceFlavor` which Kueue assigned.
The client metrics are on by default. So a flavor named `all-namespaces` gives the path `/apis/kueue.x-k8s.io/v1beta2/resourceflavors/all-namespaces`, and the old code throws `StringIndexOutOfBoundsException` for it, like for `/api/v1/namespaces`. Neither `setPriority` nor `holdForAdmission` catches that type. So it reaches the `catch (Exception e)` at `AppInitStep.java:154` and `ClusterInitStep.java:183`, and the application or cluster ends in `SchedulingFailure`. I traced this by reading and didn't run it.
Please say in the description that the fix also covers these Kueue lookups on `main`. 1.0.0 has no Kueue integration, so the bug stays latent there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60020] Fix `KubernetesMetricsInterceptor` exceptions for `namespaces` paths and out-of-range response codes [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #947:
URL: https://github.com/apache/spark-kubernetes-operator/pull/947#discussion_r4208738569
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/source/KubernetesMetricsInterceptor.java:
##########
@@ -217,14 +221,14 @@ private Meter getMeterByResponseCode(int code) {
* Parses the given path to extract namespace-scoped resource information.
*
* @param path The request path.
- * @return An Optional containing a Pair of namespace and resource name, or empty if not found.
+ * @return An Optional containing a Pair of namespace and resource name, or empty if the path has
+ * no {@code namespaces/<namespace>/<resource>} segments.
*/
public Optional<Pair<String, String>> parseNamespaceScopedResource(String path) {
- if (path.contains(NAMESPACES)) {
- int index = path.indexOf(NAMESPACES) + NAMESPACES.length();
- String namespaceAndResources = path.substring(index + 1);
- String[] parts = namespaceAndResources.split("/");
- return Optional.of(Pair.of(parts[0], parts[1]));
+ List<String> segments = Arrays.asList(path.split("/"));
Review Comment:
Thank you, @peter-toth. You're right. I also ran the three Kueue lookups with the name `all-namespaces` and the parsing of the base. `KueueWorkloadUtils.requestAdmission`, for `WorkloadPriorityClass` and `PriorityClass`, and `KueueWorkloadUtils.resolvePodSetFlavors` throw the raw `StringIndexOutOfBoundsException`, while they throw the expected not-found exceptions with this PR. I updated the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60020] Fix `KubernetesMetricsInterceptor` exceptions for `namespaces` paths and out-of-range response codes [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #947: [SPARK-60020] Fix `KubernetesMetricsInterceptor` exceptions for `namespaces` paths and out-of-range response codes
URL: https://github.com/apache/spark-kubernetes-operator/pull/947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org