You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Jaydeep Chovatia <ch...@gmail.com> on 2024/01/20 03:17:20 UTC

Race condition in QueryProcessor::prepare API

Hi,

Today, in our production, we came across the following scenario:

   1. We have 100 nodes of the Cassandra cluster on 4.0.6, and our client
   uses PreparedStatement, say, "*SELECT * FROM T1 WHERE PK=?*"
   2. We applied a schema change to add a *regular* column, "*ALTER TABLE
   T1 ADD COLUMN c1 UUID"*
   3. Around 20 (out of 100) of the Cassandra nodes started throwing the
   error: "*Successfully prepared, but could not find prepared statement
   for* "  (Code path: QueryEvents.java
   <https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/cql3/QueryEvents.java#L225C80-L225C145>
   )
   4. The error continued for around 2 hours, and only restarting those 20
   nodes resolved the issue.

Our current hypothesis is that there is a race condition in the
QueryProcessor::prepare
<https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L575>
API in that one thread is evicting the prepared statements while the other
is adding, which is never-ending. A similar hypothesis has been mentioned
in a ticket in 2022: https://issues.apache.org/jira/browse/CASSANDRA-17401

Has anyone ever experienced this? Are there any quick pointers on what
could have gone wrong?

Thanks in advance!

Jaydeep

Re: Race condition in QueryProcessor::prepare API

Posted by Jaydeep Chovatia <ch...@gmail.com>.

I think this is a regression introduced as part of CASSANDRA-17248
<https://issues.apache.org/jira/browse/CASSANDRA-17248>, the following code
got introduced in QueryProcessor.java
<https://github.com/apache/cassandra/commit/242f7f9b18db77bce36c9bba00b2acda4ff3209e#r137491766>
since C* 3.0.26.

// Make sure the missing one is going to be eventually re-prepared
evictPrepared(hashWithKeyspace);
evictPrepared(hashWithoutKeyspace);

The above-mentioned code was not present in C* 3.0.25
<https://github.com/apache/cassandra/blob/cassandra-3.0.25/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L391>
.

Alex, could you please take a look at it?

Jaydeep

On Fri, Jan 19, 2024 at 7:17 PM Jaydeep Chovatia <ch...@gmail.com>
wrote:

> Hi,
>
> Today, in our production, we came across the following scenario:
>
>    1. We have 100 nodes of the Cassandra cluster on 4.0.6, and our client
>    uses PreparedStatement, say, "*SELECT * FROM T1 WHERE PK=?*"
>    2. We applied a schema change to add a *regular* column, "*ALTER TABLE
>    T1 ADD COLUMN c1 UUID"*
>    3. Around 20 (out of 100) of the Cassandra nodes started throwing the
>    error: "*Successfully prepared, but could not find prepared statement
>    for* "  (Code path: QueryEvents.java
>    <https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/cql3/QueryEvents.java#L225C80-L225C145>
>    )
>    4. The error continued for around 2 hours, and only restarting those
>    20 nodes resolved the issue.
>
> Our current hypothesis is that there is a race condition in the
> QueryProcessor::prepare
> <https://github.com/apache/cassandra/blob/cassandra-4.0/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L575>
> API in that one thread is evicting the prepared statements while the other
> is adding, which is never-ending. A similar hypothesis has been mentioned
> in a ticket in 2022: https://issues.apache.org/jira/browse/CASSANDRA-17401
>
> Has anyone ever experienced this? Are there any quick pointers on what
> could have gone wrong?
>
> Thanks in advance!
>
> Jaydeep
>