[jira] [Updated] (CASSANDRA-21681) Compression dictionary distribution can silently leave a node on a stale dictionary

["Stefan Miklosovic (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-21681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stefan Miklosovic updated CASSANDRA-21681:
------------------------------------------
     Bug Category: Parent values: Availability(12983)
       Complexity: Normal
    Discovered By: User Report
    Fix Version/s: 6.0-alpha3
         Severity: Normal
         Assignee: Stefan Miklosovic
           Status: Open  (was: Triage Needed)

> Compression dictionary distribution can silently leave a node on a stale dictionary
> -----------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-21681
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21681
>             Project: Apache Cassandra
>          Issue Type: Bug
>          Components: Feature/Compression
>            Reporter: Stefan Miklosovic
>            Assignee: Stefan Miklosovic
>            Priority: Normal
>             Fix For: 6.0-alpha3
>
>
> h2. Problem
>  
> Dictionaries live in \{{system_distributed.compression_dictionaries}} and propagate via a
> fire-and-forget \{{CompressionDictionaryUpdateMessage}}. Every read/write used
> {\{ConsistencyLevel.ONE}} and every failure along the path is silent: the write may land on one replica ahead of the (unordered) notification; a recipient reading at ONE can hit a replica without the row yet and get nothing; \{{retrieveCompressionDictionary}} swallows exceptions as \{{null}} unlogged; and \{{CompressionDictionaryEventHandler}} feeds that \{{null}} to \{{cache.add()}} (a no-op).
>  
> The node then keeps its old dictionary, silently, until the periodic refresh (\{{compression_dictionary_refresh_interval}}, default 3600s) - so after a workload shift it can write an hour of SSTables with a stale dictionary while its peers have moved on
>  
> h2. Changes
>  
> # *ONE -> QUORUM* for dictionary store/retrieve/delete/list in \{{SystemDistributedKeyspace}}: a QUORUM write cannot be missed by a later QUORUM read, removing the race rather than papering over
> it. (View-build queries and the \{{ANY}} write elsewhere are deliberately untouched.)
> # *Failed reads are logged*: the retrieves now WARN on a genuine read failure - otherwise an unavailable replica is indistinguishable from a table with no dictionary.
> # *No \{{null}} to the cache*: on notify-but-can't-read, \{{CompressionDictionaryEventHandler}} WARNs
> and returns instead of calling \{{cache.add(null)}}, stating that the node keeps its current
> dictionary until the next refresh.
>  
> h2. Empty result != failure
>  
> {\{UntypedResultSet.one()}} throws on an empty result set, which is the normal state for any table whose dictionary has not been stored yet. Routing that through (2)'s catch would fire the WARN on every refresh/eval tick and bury the real failures it exists to surface (measured: 7 spurious \{{IllegalStateException}} warnings in 36 min on one table). The retrieves therefore check \{{isEmpty()}} and return \{{null}} on the normal path, leaving the catch for genuine failures.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org