[jira] [Commented] (CASSANDRA-18433) Row cache inconsistency issue: A read can put stale data into row cache in a race condition

["Huapeng Yuan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17709516#comment-17709516 ] 

Huapeng Yuan commented on CASSANDRA-18433:
------------------------------------------

From source code, 3.11.14 should still have same issue. Sorry, I don't have such test to reproduce the issue. It is a race condition and depends on some timing. In our production system, it happened 1 or 2 times per hour.

 

The line has issue:
In SinglePartitionReadCommand::getThroughCache:
 CacheService.instance.rowCache.replace(key, sentinel, toCache);
 
Suppose it should insert only if the sentinel exists and matches. Should not insert if the sentinel had been deleted by invalidation from another thread.
 

> Row cache inconsistency issue: A read can put stale data into row cache in a race condition
> -------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-18433
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-18433
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local/Caching
>            Reporter: Huapeng Yuan
>            Priority: Normal
>             Fix For: 3.11.x
>
>
> We found the issue in our production system which has the version 3.11.6.  When we did an update and then read immediately after update successfully, we may read the stale data sometimes.  Same issue for writeAll + readOne consistency and writeQuorm+readQuorum. The issue is gone once we disabled the row cache.
> The config for row cache: 
> caching = \{'keys': 'ALL', 'rows_per_partition': 'ALL'}
>  
> After some investigations, we think there is a race condition during read/write path. Problems:
> When two threads are reading and writing the same partition (for example, two rows with same partition key) at same time, the read thread may load the stale data into row cache for the row which is being updated.
> {{}}
> {panel:title=The steps of write-thread inserting a row to partition p}
> {{W-Step }}{{{}1{}}}{{{}: inserts the value v1 to memtable.{}}}
> {{W-Step }}{{{}2{}}}{{{}: invalidates the row cache using partition key.{}}}
> {panel}
> {{}}
> {panel:title=The steps of read-thread reading a row from partition p}
> {{R-Step }}{{{}1{}}}{{{}: Checks row cache and finds whether the row is not present in cache. If not, goes to '{}}}{{{}R-Step {}}}{{{}2'{}}}{{{}.{}}}
> {{R-Step }}{{{}2{}}}{{{}: Insert a sentinel (timestamp) as the row value into row cache to tell other read threads should skip the row cache.{}}}
> {{R-Step }}{{{}3{}}}{{{}: Read from storage layer and get value v0 which can be older than v1.{}}}
> {{R-Step }}{{{}4{}}}{{{}: Insert v0 to row cache {}}}{{for}} {{the row by checking }}{{if}} {{the row doesn't exist or it has the same sentinel. *The inconsistency is caused by this step. Should not insert the stale value if the sentinel doesn't exist in row cache any more.*}}
> {panel}
> {{}}
> {panel:title=The sequence to reproduce the issue}
> {{R-Step }}{{1}}
> {{R-Step }}{{2}}
> {{R-Step }}{{3}}
> {{W-Step }}{{1}}
> {{W-Step }}{{2}}
> {{R-Step }}{{4}}
> {panel}
> {{}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org