You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "arloliu (via GitHub)" <gi...@apache.org> on 2026/05/17 08:01:06 UTC

[PR] Cap RequestErrUnprepared retry recursion on query and batch paths [cassandra-gocql-driver]

arloliu opened a new pull request, #1952:
URL: https://github.com/apache/cassandra-gocql-driver/pull/1952

   ## Summary
   
   `Conn.executeQuery` and `Conn.executeBatch` both respond to a `*RequestErrUnprepared` by evicting the prepared-statement cache entry and recursing on themselves with no upper bound. When the server persistently re-reports the same statement as unprepared after re-prepare — for example a coordinator thrashing its prepared-statement cache under high statement cardinality, or a misbehaving proxy/fork — the recursion never terminates. The goroutine stack eventually exceeds `runtime.SetMaxStack` (1 GiB by default), at which point Go's runtime crashes the **entire process** with an unrecoverable stack-overflow throw. No `recover()` can intercept it.
   
   The failure mode is reachable from any prepared-statement workload; the batch path is on the hot write path. No attacker is required — organic Cassandra prep-cache thrash is sufficient.
   
   This PR caps re-prepare retries on both paths at 5 and surfaces a descriptive error wrapping the underlying `*RequestErrUnprepared` so `errors.As` continues to work.
   
   ## Approach
   
   - New unexported helpers `executeQueryWithUnprepRetries` / `executeBatchWithUnprepRetries` take an `unprepAttempt int` and increment it on each retry.
   - Existing entry points (`executeQuery`, `executeBatch`) become thin wrappers that start the counter at 0.
   - When the counter reaches `maxUnprepRetries = 5`, the iter is returned with `err = fmt.Errorf("…after N re-prepare attempts: %w", serverErr)`.
   - Behavior is a strict superset of the prior code: queries and batches that succeed within 5 attempts behave exactly as before. Only the pathological no-progress case is changed.
   
   ## Tests
   
   Test fake-server (`conn_test.go`) gains:
   - an `always-unprep` `opPrepare` case returning `id=99`,
   - an `opBatch` arm that replies `ErrCodeUnprepared` when any statement carries `id=99`,
   - case-insensitive verb-trimming in the `opPrepare` query-name parser (`select|insert|update|delete`) so DML in batches can reach the `always-unprep` case.
   
   `unprep_retry_test.go` (new):
   - `TestExecuteQuery_UnprepRetryIsCapped` drives the always-unprep path through `Query.Exec`.
   - `TestExecuteBatch_UnprepRetryIsCapped` drives it through `Session.ExecuteBatch`.
   - Each asserts: no infinite recursion, error mentions `"re-prepare attempts"`, `errors.As(err, &*RequestErrUnprepared)` succeeds, and the server received exactly `maxUnprepRetries + 1` prepare/execute (resp. prepare/batch) pairs.
   
   ## Test plan
   
   - [x] `make check` clean
   - [x] `make test-unit` green (both new tests pass)
   - [ ] CI on GitHub Actions
   
   ## Notes
   
   No CASSGO ticket attached; happy to file one and amend the commit / CHANGELOG entry if the committer prefers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSGO-133 Cap RequestErrUnprepared retry recursion on query and batch paths [cassandra-gocql-driver]

Posted by "worryg0d (via GitHub)" <gi...@apache.org>.
worryg0d commented on PR #1952:
URL: https://github.com/apache/cassandra-gocql-driver/pull/1952#issuecomment-5118500481

   > I think that surfacing the unprepared error to the retry policy is not the right approach. The prepare flow is something that should be internal to the driver, it's mostly a protocol thing and the user in theory doesn't even know or care about prepare semantics.
   
   You're right, gocql implicitly prepares queries, so it makes sense. Interesting that RequestErrUnprepared is never returned to the users, but the type is exposed.
   
   Then what we can do is handle it in the preparing goroutine in Conn.prepareStatement method, and return this error when it hits the cap


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Cap RequestErrUnprepared retry recursion on query and batch paths [cassandra-gocql-driver]

Posted by "worryg0d (via GitHub)" <gi...@apache.org>.
worryg0d commented on PR #1952:
URL: https://github.com/apache/cassandra-gocql-driver/pull/1952#issuecomment-4999967903

   I created CASSGO-133 for this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSGO-133 Cap RequestErrUnprepared retry recursion on query and batch paths [cassandra-gocql-driver]

Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on PR #1952:
URL: https://github.com/apache/cassandra-gocql-driver/pull/1952#issuecomment-5117830007

   Is this something that you actually ran into in the real world? No other CQL driver has a cap on the number of prepare retries and I don't think I've ever seen someone report an issue with this. Even if the application is using prepared statements in the worst way possible the prepare should eventually succeed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] CASSGO-133 Cap RequestErrUnprepared retry recursion on query and batch paths [cassandra-gocql-driver]

Posted by "joao-r-reis (via GitHub)" <gi...@apache.org>.
joao-r-reis commented on PR #1952:
URL: https://github.com/apache/cassandra-gocql-driver/pull/1952#issuecomment-5118035576

   There is one thing that is different about gocql and it's the fact that the unprepared retries happen in a recursive manner which can lead to the stack growing slowly over time until that request eventually succeeds.
   
   I think that surfacing the unprepared error to the retry policy is not the right approach. The prepare flow is something that should be internal to the driver, it's mostly a protocol thing and the user in theory doesn't even know or care about prepare semantics.
   
   I'm not opposed to capping the number of prepare retries even though this has never been needed on other drivers but this should be an internal driver thing and the retry policy should not be involved in my opinion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Cap RequestErrUnprepared retry recursion on query and batch paths [cassandra-gocql-driver]

Posted by "worryg0d (via GitHub)" <gi...@apache.org>.
worryg0d commented on code in PR #1952:
URL: https://github.com/apache/cassandra-gocql-driver/pull/1952#discussion_r3594830122


##########
conn.go:
##########
@@ -1746,9 +1759,18 @@ func (c *Conn) executeQuery(ctx context.Context, q *internalQuery) *Iter {
 		// is not consistent with regards to its schema.
 		return iter
 	case *RequestErrUnprepared:
+		if unprepAttempt >= maxUnprepRetries {
+			// Pathological re-prepare loop (server-side cache evicting
+			// our prepared statement between every attempt). Bail with
+			// the underlying server error rather than recursing
+			// indefinitely.
+			iter.err = fmt.Errorf("gocql: failed to execute prepared statement after %d re-prepare attempts: %w",
+				unprepAttempt+1, x)
+			return iter

Review Comment:
   Previously, `RequestErrUnprepared` was never returned to the users, but this patch changes that, so I'm concerned about inconsistencies in error wrapping here. Similar error types are just returned as is without any wrapping, so it is possible to type-assert for the desired error type:
   ```go
     err := gocql.Query("...").ExecContext(ctx)
     if err != nil {
       if syntaxErr, ok := err.(*RequestErrSyntax); ok {
         ...
       }
     }
   ```
   
   However, having this error wrapped makes type-assertion for `*RequestErrUnprepared` impossible, but it still would work with `errors.As`, so I'm unsure if this is problematic or not for the users



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org