You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by pwozniak <pw...@man.poznan.pl> on 2022/07/21 13:52:20 UTC
Wrong Consistency level seems to be used
Hi,
we have the following code (java driver):
cluster =Cluster.builder().addContactPoints(contactPoints).withPort(port)
.withProtocolVersion(ProtocolVersion.V3)
.withQueryOptions(new QueryOptions()
.setConsistencyLevel(ConsistencyLevel.QUORUM))
.withTimestampGenerator(new AtomicMonotonicTimestampGenerator())
.withCredentials(userName, password).build();
session =cluster.connect(keyspaceName);
where ConsistencyLevel.QUORUM is our default consistency level. But we
keep receiving the following exceptions:
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra
timeout during read query at consistency ALL (3 responses were required
but only 2 replica responded)
Why the consistency level is ALL in there? Availability of our cluster
is reduced because of that. We verified all our source code and haven't
found places where ALL is set.
We also did heap dump and found only ConsistencyLevel.QUORUM there.
Regards,
Pawel
Re: Wrong Consistency level seems to be used
Posted by Jim Shaw <jx...@gmail.com>.
My experience to debug this kind of issue is to turn on trace. The nice
thing in cassandra is:
you can turn on trace only on 1 node and with a small percentage, i.e.
nodetool settraceprobability 0.05 --- only run on 1 node.
Hope it helps.
Regards,
James
On Thu, Jul 21, 2022 at 2:50 PM Tolbert, Andy <x...@andrewtolbert.com> wrote:
> I'd bet the JIRA that Paul is pointing to is likely what's happening
> here. I'd look for read repair errors in your system logs or in your
> metrics (if you have easy access to them).
>
> There are operations that can happen during the course of a query
> being executed that may happen at different CLs, atomic batch log
> timeouts (CL TWO I think?) and read repair came to my mind (especially
> for CL ALL) that can make the timeout/unavailable exceptions include a
> different CL. I also remember some DSE features causing this as well
> (rbac, auditing, graph and solr stuff). In newer versions of C* the
> errors may be more specific or a warning may come along with it
> depending on what is failing.
>
> Thanks,
> Andy
>
Re: Wrong Consistency level seems to be used
Posted by "Tolbert, Andy" <x...@andrewtolbert.com>.
I'd bet the JIRA that Paul is pointing to is likely what's happening
here. I'd look for read repair errors in your system logs or in your
metrics (if you have easy access to them).
There are operations that can happen during the course of a query
being executed that may happen at different CLs, atomic batch log
timeouts (CL TWO I think?) and read repair came to my mind (especially
for CL ALL) that can make the timeout/unavailable exceptions include a
different CL. I also remember some DSE features causing this as well
(rbac, auditing, graph and solr stuff). In newer versions of C* the
errors may be more specific or a warning may come along with it
depending on what is failing.
Thanks,
Andy
Re: Wrong Consistency level seems to be used
Posted by Paul Chandler <pa...@redshots.com>.
I came across this problem a few years ago, and had long conversations with Datastax support about it.
In my case it turns out that the error message is misleading and I was pointed to the ticket: https://issues.apache.org/jira/browse/CASSANDRA-14715
I con’t remember much about it now, but see if that ticket applies to your experience.
Thanks
Paul Chandler
> On 21 Jul 2022, at 15:12, pwozniak <pw...@man.poznan.pl> wrote:
>
> Yes, I did it. Nothing like this in my code. Consistency level is set only in one place (shown below).
>
>
>
> On 7/21/22 4:08 PM, manish khandelwal wrote:
>> Consistency can also be set on a statement basis. So please check in your code that you might be setting consistency 'ALL' for some queries.
>>
>> On Thu, Jul 21, 2022 at 7:23 PM pwozniak <pwozniak@man.poznan.pl <ma...@man.poznan.pl>> wrote:
>> Hi,
>>
>> we have the following code (java driver):
>>
>> cluster = Cluster.builder().addContactPoints(contactPoints).withPort(port)
>> .withProtocolVersion(ProtocolVersion.V3)
>> .withQueryOptions(new QueryOptions()
>> .setConsistencyLevel(ConsistencyLevel.QUORUM))
>> .withTimestampGenerator(new AtomicMonotonicTimestampGenerator())
>> .withCredentials(userName, password).build();
>>
>> session = cluster.connect(keyspaceName);
>>
>> where ConsistencyLevel.QUORUM is our default consistency level. But we keep receiving the following exceptions:
>>
>>
>>
>> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded)
>>
>>
>>
>> Why the consistency level is ALL in there? Availability of our cluster is reduced because of that. We verified all our source code and haven't found places where ALL is set.
>> We also did heap dump and found only ConsistencyLevel.QUORUM there.
>>
>>
>>
>> Regards,
>>
>> Pawel
>>
Re: Wrong Consistency level seems to be used
Posted by pwozniak <pw...@man.poznan.pl>.
Yes, I did it. Nothing like this in my code. Consistency level is set
only in one place (shown below).
On 7/21/22 4:08 PM, manish khandelwal wrote:
> Consistency can also be set on a statement basis. So please check in
> your code that you might be setting consistency 'ALL' for some queries.
>
> On Thu, Jul 21, 2022 at 7:23 PM pwozniak <pwozniak@man.poznan.pl
> <ma...@man.poznan.pl>> wrote:
>
> Hi,
>
> we have the following code (java driver):
>
> cluster =Cluster.builder().addContactPoints(contactPoints).withPort(port)
> .withProtocolVersion(ProtocolVersion.V3)
> .withQueryOptions(new QueryOptions()
> .setConsistencyLevel(ConsistencyLevel.QUORUM))
> .withTimestampGenerator(new AtomicMonotonicTimestampGenerator())
> .withCredentials(userName, password).build();
>
> session =cluster.connect(keyspaceName);
>
> where ConsistencyLevel.QUORUM is our default consistency level.
> But we keep receiving the following exceptions:
>
>
> com.datastax.driver.core.exceptions.ReadTimeoutException:
> Cassandra timeout during read query at consistency ALL (3
> responses were required but only 2 replica responded)
>
>
> Why the consistency level is ALL in there? Availability of our
> cluster is reduced because of that. We verified all our source
> code and haven't found places where ALL is set.
> We also did heap dump and found only ConsistencyLevel.QUORUM there.
>
>
> Regards,
>
> Pawel
>
Re: Wrong Consistency level seems to be used
Posted by manish khandelwal <ma...@gmail.com>.
Consistency can also be set on a statement basis. So please check in your
code that you might be setting consistency 'ALL' for some queries.
On Thu, Jul 21, 2022 at 7:23 PM pwozniak <pw...@man.poznan.pl> wrote:
> Hi,
>
> we have the following code (java driver):
>
> cluster = Cluster.builder().addContactPoints(contactPoints).withPort(port)
> .withProtocolVersion(ProtocolVersion.V3)
> .withQueryOptions(new QueryOptions()
> .setConsistencyLevel(ConsistencyLevel.QUORUM))
> .withTimestampGenerator(new AtomicMonotonicTimestampGenerator())
> .withCredentials(userName, password).build();
> session = cluster.connect(keyspaceName);
>
>
> where ConsistencyLevel.QUORUM is our default consistency level. But we
> keep receiving the following exceptions:
>
>
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra
> timeout during read query at consistency ALL (3 responses were required but
> only 2 replica responded)
>
>
> Why the consistency level is ALL in there? Availability of our cluster is
> reduced because of that. We verified all our source code and haven't found
> places where ALL is set.
> We also did heap dump and found only ConsistencyLevel.QUORUM there.
>
>
> Regards,
>
> Pawel
>
Re: Wrong Consistency level seems to be used
Posted by Bowen Song via user <us...@cassandra.apache.org>.
It doesn't make any sense to see consistency level ALL if the code is
not explicitly using it. My best guess is somewhere in the code the
consistency level was overridden.
On 21/07/2022 14:52, pwozniak wrote:
>
> Hi,
>
> we have the following code (java driver):
>
> cluster =Cluster.builder().addContactPoints(contactPoints).withPort(port)
> .withProtocolVersion(ProtocolVersion.V3)
> .withQueryOptions(new QueryOptions()
> .setConsistencyLevel(ConsistencyLevel.QUORUM))
> .withTimestampGenerator(new AtomicMonotonicTimestampGenerator())
> .withCredentials(userName, password).build();
>
> session =cluster.connect(keyspaceName);
>
> where ConsistencyLevel.QUORUM is our default consistency level. But we
> keep receiving the following exceptions:
>
>
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra
> timeout during read query at consistency ALL (3 responses were
> required but only 2 replica responded)
>
>
> Why the consistency level is ALL in there? Availability of our cluster
> is reduced because of that. We verified all our source code and
> haven't found places where ALL is set.
> We also did heap dump and found only ConsistencyLevel.QUORUM there.
>
>
> Regards,
>
> Pawel
>