You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Vineet G H <vg...@gmail.com> on 2018/04/09 18:43:37 UTC
Cassandra Hints file corruption
Hello All,
We have a 14 node Cassandra cluster 3.11.1. For some odd reason
intermittently we see the following error
ERROR [HintsDispatcher:1] 2018-04-06 16:26:44,423
CassandraDaemon.java:228 - Exception in thread
Thread[HintsDispatcher:1,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Digest
mismatch exception
at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:298)
~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
The jar in question has the patch from bug
https://issues.apache.org/jira/browse/CASSANDRA-13696
We are able get past the issue by running truncatehints
1. Could these new manifestation of the issue? Probably not related to bug above
2. Are there any tools which dump hints file content?
3. What are the implication of truncatehints? Sounds like there could
be data loss, but we have quoram for writes and reads, which means we
should enough replicas to reconstruct the data.
I am gathering more evidence on the issue would be happy to work with devs
Regards,
Vineet
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Cassandra Hints file corruption
Posted by Vineet G H <vg...@gmail.com>.
Yes, the commit log show that we built it out f919cf4a4 which we used
later locally to build it.
I realize that using release artifact is suggested. We tried even
3.11.1 ( official release) and where able to reproduce this issue on
14 node cluster
On Mon, Apr 9, 2018 at 12:11 PM, Michael Shuler <mi...@pbandjelly.org> wrote:
> On 04/09/2018 01:43 PM, Vineet G H wrote:
>> Hello All,
>>
>> We have a 14 node Cassandra cluster 3.11.1. For some odd reason
>> intermittently we see the following error
>>
>> ERROR [HintsDispatcher:1] 2018-04-06 16:26:44,423
>> CassandraDaemon.java:228 - Exception in thread
>> Thread[HintsDispatcher:1,1,main]
>> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest
>> mismatch exception
>> at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:298)
>> ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
>
> 3.11.1-SNAPSHOT? This could be any commit between the 3.11.0 and 3.11.1
> releases (usually). If you built this yourself, what commit sha is your
> SNAPSHOT jar from and does the git log show it includes commit f919cf4a4?
>
> Generally, using a release artifact is highly suggested, since everyone
> knows the code state of the release. No one but yourself can have any
> reasonable knowledge of where your cluster is running at code-wise.
>
>> The jar in question has the patch from bug
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-13696
>>
>> We are able get past the issue by running truncatehints
>>
>> 1. Could these new manifestation of the issue? Probably not related to bug above
>> 2. Are there any tools which dump hints file content?
>> 3. What are the implication of truncatehints? Sounds like there could
>> be data loss, but we have quoram for writes and reads, which means we
>> should enough replicas to reconstruct the data.
>>
>> I am gathering more evidence on the issue would be happy to work with devs
>>
>> Regards,
>> Vineet
>
> --
> Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Cassandra Hints file corruption
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 04/09/2018 01:43 PM, Vineet G H wrote:
> Hello All,
>
> We have a 14 node Cassandra cluster 3.11.1. For some odd reason
> intermittently we see the following error
>
> ERROR [HintsDispatcher:1] 2018-04-06 16:26:44,423
> CassandraDaemon.java:228 - Exception in thread
> Thread[HintsDispatcher:1,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest
> mismatch exception
> at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:298)
> ~[apache-cassandra-3.11.1.jar:3.11.1-SNAPSHOT]
3.11.1-SNAPSHOT? This could be any commit between the 3.11.0 and 3.11.1
releases (usually). If you built this yourself, what commit sha is your
SNAPSHOT jar from and does the git log show it includes commit f919cf4a4?
Generally, using a release artifact is highly suggested, since everyone
knows the code state of the release. No one but yourself can have any
reasonable knowledge of where your cluster is running at code-wise.
> The jar in question has the patch from bug
>
> https://issues.apache.org/jira/browse/CASSANDRA-13696
>
> We are able get past the issue by running truncatehints
>
> 1. Could these new manifestation of the issue? Probably not related to bug above
> 2. Are there any tools which dump hints file content?
> 3. What are the implication of truncatehints? Sounds like there could
> be data loss, but we have quoram for writes and reads, which means we
> should enough replicas to reconstruct the data.
>
> I am gathering more evidence on the issue would be happy to work with devs
>
> Regards,
> Vineet
--
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org