You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiangzho (via GitHub)" <gi...@apache.org> on 2026/08/28 22:09:21 UTC
[PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
jiangzho opened a new pull request, #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806
### What changes were proposed in this pull request?
When ReconcilerUtils.getOrCreateSecondaryResource retries a failed secondary-resource creation, it now honors the server's requested retry delay instead of always using our own exponential backoff:
- BackoffUtils.getRetryAfterMillis reads KubernetesClientException.getStatus().getDetails().getRetryAfterSeconds() — the field the Kubernetes API server mirrors from the HTTP Retry-After header into the response body (fabric8's client doesn't expose the raw header itself). When present and positive, BackoffUtils.backoffSleep sleeps for exactly that duration; otherwise it falls back to the existing jittered exponential backoff, unchanged.
- ReconcilerUtils.shouldBackoffBeforeRetry now also returns true whenever a Retry-After value is present, regardless of status code — previously only 409 and 429 triggered a backoff sleep, so a transient 503 carrying retryAfterSeconds was retried immediately, ignoring the server's hint.
### Why are the changes needed?
The server is in the best position to say how long to wait before retrying a throttled or overloaded request. Always using our own backoff schedule can retry too
soon (extending throttling) or wait longer than necessary, and previously we ignored the hint entirely for non-429/409 errors like 503.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added BackoffUtilsTest covering getRetryAfterMillis (present/absent/non-positive/no-status cases) and computeBackoffDelay bounds. Added two tests in
ReconcilerUtilsTest: retriesCreateOn429AndSucceedsHonoringRetryAfter (429 with retryAfterSeconds=1) and retriesCreateOnTransient5xxHonoringRetryAfterWhenPresent
(503 with retryAfterSeconds=1), both asserting the retry succeeds and waited at least 1s.
### Was this patch authored or co-authored using generative AI tooling?
Co-Authored-by: Claude Code (claude-sonnet-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5531054214
Got it. Let me investigate the failure to unblock you, @jiangzho .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #806: [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors
URL: https://github.com/apache/spark-kubernetes-operator/pull/806
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5531248663
The main branch is recovered. Please re-trigger your GitHub Actions CI on your fork. As you see in the following, the community member cannot re-trigger your CIs because it's owned by only you.
<img width="1286" height="909" alt="Screenshot 2026-09-03 at 12 49 03" src="https://github.com/user-attachments/assets/968e1010-ad83-4855-83da-a6c03fac7285" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5458562112
Thank you for working on this, @jiangzho . The direction looks good to me — honoring the server-provided `retryAfterSeconds` is the right thing to do. I have one blocker and one design concern.
**1. `retriesCreateOnTransient5xxHonoringRetryAfterWhenPresent` actually fails (blocker)**
I ran the PR branch locally and this test fails at the timing assertion:
```
ReconcilerUtilsTest > retriesCreateOnTransient5xxHonoringRetryAfterWhenPresent() FAILED
org.opentest4j.AssertionFailedError at ReconcilerUtilsTest.java:189
```
The test never reaches the backoff sleep. Since 503 is a transient error, the catch block in `getOrCreateSecondaryResource` performs the GET fallback **before** the backoff sleep, and the test stubs the second `get()` to return the pod — so the method returns immediately from inside the catch block (elapsed ≈ 0ms < 1000ms). To exercise the intended path (503 + Retry-After → sleep → retry), the second `get()` should return `null` and the second `create()` should succeed.
**2. The server-provided delay is not capped**
`getRetryAfterMillis` feeds `Thread.sleep` directly. The existing exponential backoff is capped by `API_SECONDARY_RESOURCE_CREATE_MAX_BACKOFF_MILLIS`, but the server-provided value is trusted without a bound. Since this sleep runs on the reconciler thread, an API server (or an intermediate proxy) responding with e.g. `retryAfterSeconds: 3600` would block reconciliation for an hour. I'd suggest applying the existing cap, e.g. `Math.min(retryAfterMillis, maxBackoffMs)`.
**Minor**: the two new `ReconcilerUtilsTest` tests each sleep a real 1s, adding ~2s to the unit test suite. Probably unavoidable given `retryAfterSeconds` has whole-second granularity, but worth noting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5533368806
The build workflow is happy new after merging from main, it just not reports back the succeeded results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5532917755
It seems that another CI outage is caused by ASF Infra team at `Gradle` GitHub Actions pattern. I'm digging it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5531121892
I made a PR to unblock this PR. It's a Minikube issue. @jiangzho .
- https://github.com/apache/spark-kubernetes-operator/pull/807
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5533915116
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5458568214
BTW, it seems that this PR is under active development via force-pushing . In that case, please use `Draft` status, @jiangzho .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5458574194
I converted this PR to `Draft` status. Please convert this back to normal when the PR is ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5472749879
Thanks @dongjoon-hyun for the review!
Sorry I was working on the flaky test and did a few force push, it should not be ready for review.
I also added cap as suggested so a server or intermediate proxy can't stall the reconciler thread beyond our configured ceiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5532948720
I made another PR to comply ASF Infra policy change.
- https://github.com/apache/spark-kubernetes-operator/pull/808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5529062497
Could you re-trigger the CIs, @jiangzho ?
<img width="627" height="984" alt="Screenshot 2026-09-03 at 09 48 59" src="https://github.com/user-attachments/assets/a2d5b552-a7c7-4164-b59f-5706952718db" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5533916868
Thank you, @jiangzho !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59092] Honor Retry-After when retrying secondary resource creation on Kubernetes API errors [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on PR #806:
URL: https://github.com/apache/spark-kubernetes-operator/pull/806#issuecomment-5530090832
Hi Dongjoon - I tried a few times but these are constantly failing with `moby/buildkit:buildx-stable-1` 404. Since I don't have write access to retrigger checks on this repo directly, I'll stop spinning cycles on my fork — could you retrigger the checks here when you get a chance? If it's still failing, it's likely a broader GitHub Actions/Docker Hub issue worth flagging separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org