[jira] [Commented] (CASSSIDECAR-465) Improve HTTP 429 handling across sidecar server and client

["Arjun Ashok (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSSIDECAR-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100923#comment-18100923 ] 

Arjun Ashok commented on CASSSIDECAR-465:
-----------------------------------------

*Prerequisite details:*

The plan is to add a 429 case to {{BasicRetryPolicy}} that reads {{Retry-After}} and waits before retrying.

Currently {{RequestExecutor.schedule()}} ({{{}client/.../RequestExecutor.java:369-375{}}}) schedules the delayed retry and runs it immediately anyway on the next line, unconditionally. So the retry fires right away regardless of any delay {{BasicRetryPolicy}} computes, plus a redundant second retry.

Net effect if we only fix {{{}BasicRetryPolicy{}}}: instead of failing after one try, the client fires a burst of retries almost instantly, with no real wait, straight back at a sidecar that is at capacity.

Note: this also affects the 503 path today.

> Improve HTTP 429 handling across sidecar server and client
> ----------------------------------------------------------
>
>                 Key: CASSSIDECAR-465
>                 URL: https://issues.apache.org/jira/browse/CASSSIDECAR-465
>             Project: Sidecar for Apache Cassandra
>          Issue Type: Bug
>          Components: Bulk Analytics, Client
>            Reporter: Arjun Ashok
>            Priority: Major
>
> *Update (see comments):* _fixing this requires also fixing a bug in {{RequestExecutor.schedule()}} that causes every delayed retry (including the existing 503 {{Retry-After}} path) to fire immediately in addition to firing after the delay. This is a prerequisite for this ticket's fix; see comment for details._
> The sidecar server returns HTTP 429 in two cases: when the concurrent upload limit is exceeded (SSTable uploads) and when the stream request rate limit is exhausted (SSTable component streaming). In both cases the client has no meaningful retry behavior for this response.
> BasicRetryPolicy, which all retry policy implementations extend, handles 503 SERVICE_UNAVAILABLE specially by parsing the Retry-After response header and applying backoff before retrying. HTTP 429 falls into the generic 4xx catch-all instead, which either retries immediately on a different host with no backoff, likely hitting the same limit again or fails permanently if no alternative host is available.
> Gap on server side: SSTableUploadHandler does not include a Retry-After header in its 429 response, unlike the streaming path which does. This means even with a client-side fix, upload callers cannot benefit from server-indicated retry timing.
> Expected behavior:
> BasicRetryPolicy should handle 429 analogously to 503, parsing Retry-After if present and otherwise applying exponential backoff before retrying. Since all retry policies extend BasicRetryPolicy, this fix applies uniformly across all APIs. Additionally, SSTableUploadHandler should include a Retry-After header in its 429 response so the client can back off for an appropriate duration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org