[jira] [Created] (CASSANDRA-21681) Compression dictionary distribution can silently leave a node on a stale dictionary

["Stefan Miklosovic (Jira)" <ji...@apache.org>]

Stefan Miklosovic created CASSANDRA-21681:
---------------------------------------------

             Summary: Compression dictionary distribution can silently leave a node on a stale dictionary
                 Key: CASSANDRA-21681
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21681
             Project: Apache Cassandra
          Issue Type: Bug
          Components: Feature/Compression
            Reporter: Stefan Miklosovic


h2. Problem
 
Dictionaries live in \{{system_distributed.compression_dictionaries}} and propagate via a
fire-and-forget \{{CompressionDictionaryUpdateMessage}}. Every read/write used
{\{ConsistencyLevel.ONE}} and every failure along the path is silent: the write may land on one replica ahead of the (unordered) notification; a recipient reading at ONE can hit a replica without the row yet and get nothing; \{{retrieveCompressionDictionary}} swallows exceptions as \{{null}} unlogged; and \{{CompressionDictionaryEventHandler}} feeds that \{{null}} to \{{cache.add()}} (a no-op).
 
The node then keeps its old dictionary, silently, until the periodic refresh (\{{compression_dictionary_refresh_interval}}, default 3600s) - so after a workload shift it can write an hour of SSTables with a stale dictionary while its peers have moved on
 
h2. Changes
 
# *ONE -> QUORUM* for dictionary store/retrieve/delete/list in \{{SystemDistributedKeyspace}}: a QUORUM write cannot be missed by a later QUORUM read, removing the race rather than papering over
it. (View-build queries and the \{{ANY}} write elsewhere are deliberately untouched.)
# *Failed reads are logged*: the retrieves now WARN on a genuine read failure - otherwise an unavailable replica is indistinguishable from a table with no dictionary.
# *No \{{null}} to the cache*: on notify-but-can't-read, \{{CompressionDictionaryEventHandler}} WARNs
and returns instead of calling \{{cache.add(null)}}, stating that the node keeps its current
dictionary until the next refresh.
 
h2. Empty result != failure
 
{\{UntypedResultSet.one()}} throws on an empty result set, which is the normal state for any table whose dictionary has not been stored yet. Routing that through (2)'s catch would fire the WARN on every refresh/eval tick and bury the real failures it exists to surface (measured: 7 spurious \{{IllegalStateException}} warnings in 36 min on one table). The retrieves therefore check \{{isEmpty()}} and return \{{null}} on the normal path, leaving the catch for genuine failures.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org