You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Artur Siekielski <ar...@vhex.net> on 2017/10/26 10:19:59 UTC
Getting DigestMismatchExceptions despite setting read repair chances
to zero
Hi,
we have one table for which reads and writes are done with CL=ONE. The
table contains counters. We wanted to disable async read repair for the
table (to lessen cluster load and to avoid DigestMismatchExceptions in
debug.log). After altering the table with read_repair_chance=0,
dclocal_read_repair_chance=0, the Digest exceptions still happen:
DEBUG [ReadRepairStage:92] 2017-10-26 10:00:02,798 ReadCallback.java:242
- Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(5238932067721150894, 7da6f64695d74899a91bd691321de534)
(33f950054869a91d1ea225eae342499a vs 70d054183b9b001de5f71139aa65b8d9)
at
org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_141]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_141]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141]
I have verified that the DecoratedKey arguments are for a row from the
altered table (by checking token() of the partition key).
Shouldn't the settings disable AsyncRepairRunner for the table?
Is there an option to disable async read repair globally, or each table
must be altered?
Cassandra version: 3.11.1
Thanks,
Artur
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Getting DigestMismatchExceptions despite setting read repair chances to zero
Posted by Jeff Jirsa <jj...@gmail.com>.
> On Oct 27, 2017, at 3:08 AM, Artur Siekielski <ar...@vhex.net> wrote:
>
> I noticed that the DecoratedKey printed in the stack trace can be for a different table. The arguments are a token and a partition key and they can be the same for multiple tables. Is there a way to know for which table the DigestMismatchException happens?
>
No, the read repair stats we provide are not per table, so if it’s not in the log, it’s not apparent. Feel free to open a jira to ask for it to be added to the log message.
> Can the AsyncRepairRunner be triggered if read and writes for all other tables are done with CL=LOCAL_QUORUM (RF=3)? I assumed in that case async read repair is not done even if dclocal_read_repair_chance > 0. Could it be that the async repair runs for that case and it's executed faster than the background syncing to meet RF=3?
>
Very likely. Async repair runner can be triggered if either (dclocal_)read_repair_chance is > 0. If you write at local_quorum, reads can definitely race (and even in that case, some writes can be dropped by load shedding or missed during network hiccups or GC pauses).
>
>> On 10/26/2017 12:19 PM, Artur Siekielski wrote:
>> Hi,
>>
>> we have one table for which reads and writes are done with CL=ONE. The table contains counters. We wanted to disable async read repair for the table (to lessen cluster load and to avoid DigestMismatchExceptions in debug.log). After altering the table with read_repair_chance=0, dclocal_read_repair_chance=0, the Digest exceptions still happen:
>>
>>
>> DEBUG [ReadRepairStage:92] 2017-10-26 10:00:02,798 ReadCallback.java:242 - Digest mismatch:
>> org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(5238932067721150894, 7da6f64695d74899a91bd691321de534) (33f950054869a91d1ea225eae342499a vs 70d054183b9b001de5f71139aa65b8d9)
>> at org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1]
>> at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233) ~[apache-cassandra-3.11.1.jar:3.11.1]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_141]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141]
>> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
>> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141]
>>
>>
>> I have verified that the DecoratedKey arguments are for a row from the altered table (by checking token() of the partition key).
>>
>> Shouldn't the settings disable AsyncRepairRunner for the table?
>>
>> Is there an option to disable async read repair globally, or each table must be altered?
>>
>> Cassandra version: 3.11.1
>>
>>
>> Thanks,
>> Artur
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Getting DigestMismatchExceptions despite setting read repair
chances to zero
Posted by Artur Siekielski <ar...@vhex.net>.
I noticed that the DecoratedKey printed in the stack trace can be for a
different table. The arguments are a token and a partition key and they
can be the same for multiple tables. Is there a way to know for which
table the DigestMismatchException happens?
Can the AsyncRepairRunner be triggered if read and writes for all other
tables are done with CL=LOCAL_QUORUM (RF=3)? I assumed in that case
async read repair is not done even if dclocal_read_repair_chance > 0.
Could it be that the async repair runs for that case and it's executed
faster than the background syncing to meet RF=3?
On 10/26/2017 12:19 PM, Artur Siekielski wrote:
> Hi,
>
> we have one table for which reads and writes are done with CL=ONE. The
> table contains counters. We wanted to disable async read repair for
> the table (to lessen cluster load and to avoid
> DigestMismatchExceptions in debug.log). After altering the table with
> read_repair_chance=0, dclocal_read_repair_chance=0, the Digest
> exceptions still happen:
>
>
> DEBUG [ReadRepairStage:92] 2017-10-26 10:00:02,798
> ReadCallback.java:242 - Digest mismatch:
> org.apache.cassandra.service.DigestMismatchException: Mismatch for key
> DecoratedKey(5238932067721150894, 7da6f64695d74899a91bd691321de534)
> (33f950054869a91d1ea225eae342499a vs 70d054183b9b001de5f71139aa65b8d9)
> at
> org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_141]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_141]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.1.jar:3.11.1]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141]
>
>
> I have verified that the DecoratedKey arguments are for a row from the
> altered table (by checking token() of the partition key).
>
> Shouldn't the settings disable AsyncRepairRunner for the table?
>
> Is there an option to disable async read repair globally, or each
> table must be altered?
>
> Cassandra version: 3.11.1
>
>
> Thanks,
> Artur
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Getting DigestMismatchExceptions despite setting read repair
chances to zero
Posted by Artur Siekielski <ar...@vhex.net>.
It was set to the default 99PERCENTILE, I changed it to NONE but the
exceptions are still logged (for the same table). I'm assuming node
restarts are not required for that ALTER.
On 10/26/2017 05:13 PM, Jeff Jirsa wrote:
> Is speculative retry enabled?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Getting DigestMismatchExceptions despite setting read repair chances to zero
Posted by Jeff Jirsa <jj...@gmail.com>.
Is speculative retry enabled?
--
Jeff Jirsa
> On Oct 26, 2017, at 3:19 AM, Artur Siekielski <ar...@vhex.net> wrote:
>
> Hi,
>
> we have one table for which reads and writes are done with CL=ONE. The table contains counters. We wanted to disable async read repair for the table (to lessen cluster load and to avoid DigestMismatchExceptions in debug.log). After altering the table with read_repair_chance=0, dclocal_read_repair_chance=0, the Digest exceptions still happen:
>
>
> DEBUG [ReadRepairStage:92] 2017-10-26 10:00:02,798 ReadCallback.java:242 - Digest mismatch:
> org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(5238932067721150894, 7da6f64695d74899a91bd691321de534) (33f950054869a91d1ea225eae342499a vs 70d054183b9b001de5f71139aa65b8d9)
> at org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233) ~[apache-cassandra-3.11.1.jar:3.11.1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_141]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141]
>
>
> I have verified that the DecoratedKey arguments are for a row from the altered table (by checking token() of the partition key).
>
> Shouldn't the settings disable AsyncRepairRunner for the table?
>
> Is there an option to disable async read repair globally, or each table must be altered?
>
> Cassandra version: 3.11.1
>
>
> Thanks,
> Artur
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org