Blocking read repair giving consistent data but not repairing existing data

[Bhuvan Rawal <bh...@gmail.com>]

Hi,

We are using dsc 3.0.3 on total of *6 Nodes*,* 2 DC's, 3 Node Each, RF-3*
so every node has complete data. Now we are facing a situation on a table
with 1 partition key, 2 clustering columns and 4 normal columns.

Out of the 6 5 nodes has a single value and Partition key, 2 clustering key
for a row but 3 other normal values are null.

When doing consistency level all query we get complete view of the row and
in the tracing output it says that inconsistency found in digest and read
repair is sent out to the nodes.
<*Exact error in tracing : Digest mismatch*:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(-9222324572206777856, 53444c393233363439393233)
(c421efa89ea3435c153617a34c08f396 vs 51a7e02f9e5e93520f56541ed6730558>

But on doing another read with reduced consistency the output received is
not repaired.

We are speculating that the node which is having complete view of the row
was down when delete happened on that row for more than 3 hrs (default hint
window).  We had not enabled read repair within gc grace period so possibly
the 3 deleted cells have come alive on that node, but in that case:

1. if the consistency level all query is giving complete row view then why
isnt it reflected on other nodes.
2. *read_repair_chance* is 0.0 but *dclocal_read_repair_chance* = 0.1 on
the table  (default configurations) and I tried the query with Local one on
all the servers to fulfill that probability (35-40 times on every server).

Therefore what could be the possibility that blocking and non blocking read
repair is not working? Could Full read repair fix it? Possible bug in
dsc3.0.3 which could be fixed in later version?

Any assistance on this will be welcome as this appears to be one off
scenario. I can provide complete cqlsh tracing log for consistency
local_one read query and consistency all query if required.

C*eers,
Bhuvan

Re: Blocking read repair giving consistent data but not repairing existing data

[Michael Semb Wever <mc...@apache.org>]

> We are using dsc 3.0.3 on total of *6 Nodes*,* 2 DC's, 3 Node Each, RF-3*
> so every node has complete data. Now we are facing a situation on a table
> with 1 partition key, 2 clustering columns and 4 normal columns.
> 
> Out of the 6 5 nodes has a single value and Partition key, 2 clustering key
> for a row but 3 other normal values are null.
> 
> When doing consistency level all query we get complete view of the row and
> in the tracing output it says that inconsistency found in digest and read
> repair is sent out to the nodes.
> <*Exact error in tracing : Digest mismatch*:
> org.apache.cassandra.service.DigestMismatchException: Mismatch for key
> DecoratedKey(-9222324572206777856, 53444c393233363439393233)
> (c421efa89ea3435c153617a34c08f396 vs 51a7e02f9e5e93520f56541ed6730558>
> 
> But on doing another read with reduced consistency the output received is
> not repaired.


FTR:

if range tombstones are involved here: ie manual deletes over a range of clustering keys within a partition key; then it could be https://issues.apache.org/jira/browse/CASSANDRA-13340

regards,
Mick

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org