You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by srinivasarao daruna <sr...@gmail.com> on 2017/03/16 16:31:52 UTC
Issue with Cassandra consistency in results
Hi Team,
We are struggling with a problem related to cassandra counts, after backup
and restore of the cluster. Aaron Morton has suggested to send this to user
list, so some one of the list will be able to help me.
We are have a rest api to talk to cassandra and one of our query which
fetches count is creating problems for us.
We have done backup and restore and copied all the data to new cluster. We
have done nodetool refresh on the tables, and did the nodetool repair as
well.
However, one of our key API call is returning inconsistent results. The
result count is 0 in the first call and giving the actual values for later
calls. The query frequency is bit high and failure rate has also raised
considerably.
1) The count query has partition keys in it. Didnt see any read timeout or
any errors from api logs.
2) This is how our code of creating session looks.
val poolingOptions = new PoolingOptions
poolingOptions
.setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
.setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
.setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
.setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
val builtCluster = clusterBuilder.withCredentials(username, password)
.withPoolingOptions(poolingOptions)
.build()
val cassandraSession = builtCluster.get.connect()
val preparedStatement =
cassandraSession.prepare(statement).setConsistencyLevel(ConsistencyLevel.QUORUM)
cassandraSession.execute(preparedStatement.bind(args :_*))
Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND
date_column_of_clustering_key>=?
3) Cluster configuration:
6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
machine is equipped with 16 Cores and 64 GB Ram.
Replication factor is 3, and write consistency is ONE and read
consistency is QUORUM.
4) cassandra is never down on any machine
5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
6) nodetool tpstats shows no read failures, and no other failures.
7) Do not see any other issues from system.log of cassandra. We just see
few warnings as below.
Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103 -
88 prepared statements discarded in the last minute because cache limit
reached (32 MB)
The first api call returns 0 and the api calls later gives right values.
Please let me know, if any other details needed.
Could you please have a look at this issue once and kindly give me your
inputs? This issue literally broke the confidence on Cassandra from our
business team.
Your inputs will be really helpful.
Thank You,
Regards,
Srini
Re: Issue with Cassandra consistency in results
Posted by srinivasarao daruna <sr...@gmail.com>.
Would switching to select partition_key instead of select count(*) help me
any way ?
I know that, Logically they both are same.. but just asking out of
desperation. Is it worth a shot?
On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
Replication factor is 3, and write consistency is ONE and read
consistency is QUORUM.
That combination is not gonna work well:
*Write succeeds to NODE A but fails on node B,C*
*Read goes to NODE B, C*
If you can tolerate some temporary inaccuracy you can use QUORUM but may
still have the situation where
Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
Read succeeds on node B and C at timestamp 1
If you need fully race condition free counts I'm afraid you need to use
SERIAL or LOCAL_SERIAL (for in DC only accuracy)
On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <sree.srinu38@gmail.com
> wrote:
> Replication strategy is SimpleReplicationStrategy.
>
> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>
> I was worried that CL=ALL have more read latency and read failures. But
> won't rule out trying it.
>
> Should I switch select count (*) to select partition_key column? Would
> that be of any help.?
>
>
> Thank you
> Regards
> Srini
>
> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <ar...@gmail.com>
> wrote:
>
> What are your replication strategy and snitch settings?
>
> Have you tried doing a read at CL=ALL? If it's an actual inconsistency
> issue (missing data), this should cause the correct results to be returned.
> You'll need to run a repair to fix the inconsistencies.
>
> If all the data is actually there, you might have one or several nodes
> that aren't identifying the correct replicas.
>
> Arvydas
>
>
>
> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
> sree.srinu38@gmail.com> wrote:
>
>> Hi Team,
>>
>> We are struggling with a problem related to cassandra counts, after
>> backup and restore of the cluster. Aaron Morton has suggested to send this
>> to user list, so some one of the list will be able to help me.
>>
>> We are have a rest api to talk to cassandra and one of our query which
>> fetches count is creating problems for us.
>>
>> We have done backup and restore and copied all the data to new cluster.
>> We have done nodetool refresh on the tables, and did the nodetool repair as
>> well.
>>
>> However, one of our key API call is returning inconsistent results. The
>> result count is 0 in the first call and giving the actual values for later
>> calls. The query frequency is bit high and failure rate has also raised
>> considerably.
>>
>> 1) The count query has partition keys in it. Didnt see any read timeout
>> or any errors from api logs.
>>
>> 2) This is how our code of creating session looks.
>>
>> val poolingOptions = new PoolingOptions
>> poolingOptions
>> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
>> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
>> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
>> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>>
>> val builtCluster = clusterBuilder.withCredentials(username, password)
>> .withPoolingOptions(poolingOptions)
>> .build()
>> val cassandraSession = builtCluster.get.connect()
>>
>> val preparedStatement = cassandraSession.prepare(state
>> ment).setConsistencyLevel(ConsistencyLevel.QUORUM)
>> cassandraSession.execute(preparedStatement.bind(args :_*))
>>
>> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
>> text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND
>> date_column_of_clustering_key>=?
>>
>> 3) Cluster configuration:
>>
>> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
>> machine is equipped with 16 Cores and 64 GB Ram.
>>
>> Replication factor is 3, and write consistency is ONE and read
>> consistency is QUORUM.
>>
>> 4) cassandra is never down on any machine
>>
>> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>>
>> 6) nodetool tpstats shows no read failures, and no other failures.
>>
>> 7) Do not see any other issues from system.log of cassandra. We just see
>> few warnings as below.
>>
>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>> 1.000MiB
>> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103
>> - 88 prepared statements discarded in the last minute because cache limit
>> reached (32 MB)
>> The first api call returns 0 and the api calls later gives right values.
>>
>> Please let me know, if any other details needed.
>> Could you please have a look at this issue once and kindly give me your
>> inputs? This issue literally broke the confidence on Cassandra from our
>> business team.
>>
>> Your inputs will be really helpful.
>>
>> Thank You,
>> Regards,
>> Srini
>>
>
>
>
--
Thanks,
Ryan Svihla
RE: Issue with Cassandra consistency in results
Posted by "Durity, Sean R" <SE...@homedepot.com>.
There have been many instances of supposed inconsistency noted on this list if nodes do not have the same system time. Make sure you have a matching clock on all nodes (ntp or similar).
Sean Durity
From: Shubham Jaju [mailto:shubham@vassarlabs.com]
Sent: Tuesday, March 21, 2017 9:58 PM
To: user@cassandra.apache.org
Subject: Re: Issue with Cassandra consistency in results
Hi
This issue used to appear with me . What I figured in my case was
1. I had 3 machines
2. Inserted the data with ONE consistency (i.e there is no guarantee that data was propagated to remaining nodes , cassandra is supposed to take care of that).
3. Later I figured also that one of machines has different hard disk space compared to other two (less and data size was more ) . i.e it was not able to contain whole set of data.
So I think in above cases ( 2,3 )if you will query you will get different results as nodes are not in sync.
nodetool repair should solve this problem. But it takes more time if you have more data.
Check if this solves you problem.
Regards
Shubham Jaju
On Wed, Mar 22, 2017 at 12:23 AM, srinivasarao daruna <sr...@gmail.com>> wrote:
The same issue is appearing in CQL Shell as well.
1) Entered into cqlsh
2) SET CONSISTENCY QUORUM;
3) Ran a select * with partition key in where cluase.
First result gave 0 records,
and Next records gave results.
Its really freaking out us at the moment. And nothing in debug.log or system.log.
Thank You,
Regards,
Srini
On Fri, Mar 17, 2017 at 2:33 AM, daemeon reiydelle <da...@gmail.com>> wrote:
The prep is needed. If I recall correctly it must remain in cache for the query to complete. I don't have the docs to dig out the yaml parm to adjust query cache. I had run into the problem stress testing a smallish cluster with many queries at once.
Do you have a sense of how many distinct queries are hitting the cluster at peak?
If many clients, how do you balance the connection load or do you always hit the same node?
sent from my mobile
Daemeon Reiydelle
skype daemeon.c.m.reiydelle
USA 415.501.0198
On Mar 16, 2017 3:25 PM, "srinivasarao daruna" <sr...@gmail.com>> wrote:
Hi reiydelle,
I cannot confirm the range as the volume of data is huge and the query frequency is also high.
If the cache is the cause of issue, can we increase cache size or is there solution to avoid dropped prep statements.?
Thank You,
Regards,
Srini
On Thu, Mar 16, 2017 at 2:13 PM, daemeon reiydelle <da...@gmail.com>> wrote:
The discard due to oom is causing the zero returned. I would guess a cache miss problem of some sort, but not sure. Are you using row, index, etc. caches? Are you seeing the failed prep statement on random nodes (duh, nodes that have the relevant data ranges)?
.......
Daemeon C.M. Reiydelle
USA (+1) 415.501.0198<tel:+1%20415-501-0198>
London (+44) (0) 20 8144 9872<tel:+44%2020%208144%209872>
On Thu, Mar 16, 2017 at 10:56 AM, Ryan Svihla <rs...@foundev.pro>> wrote:
Depends actually, restore just restores what's there, so if only one node had a copy of the data then only one node had a copy of the data meaning quorum will still be wrong sometimes.
On Thu, Mar 16, 2017 at 1:53 PM, Arvydas Jonusonis <ar...@gmail.com>> wrote:
If the data was written at ONE, consistency is not guaranteed. ..but considering you just restored the cluster, there's a good chance something else is off.
On Thu, Mar 16, 2017 at 18:19 srinivasarao daruna <sr...@gmail.com>> wrote:
Want to make read and write QUORUM as well.
On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro>> wrote:
Replication factor is 3, and write consistency is ONE and read consistency is QUORUM.
That combination is not gonna work well:
Write succeeds to NODE A but fails on node B,C
Read goes to NODE B, C
If you can tolerate some temporary inaccuracy you can use QUORUM but may still have the situation where
Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
Read succeeds on node B and C at timestamp 1
If you need fully race condition free counts I'm afraid you need to use SERIAL or LOCAL_SERIAL (for in DC only accuracy)
On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <sr...@gmail.com>> wrote:
Replication strategy is SimpleReplicationStrategy.
Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
I was worried that CL=ALL have more read latency and read failures. But won't rule out trying it.
Should I switch select count (*) to select partition_key column? Would that be of any help.?
Thank you
Regards
Srini
On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <ar...@gmail.com>> wrote:
What are your replication strategy and snitch settings?
Have you tried doing a read at CL=ALL? If it's an actual inconsistency issue (missing data), this should cause the correct results to be returned. You'll need to run a repair to fix the inconsistencies.
If all the data is actually there, you might have one or several nodes that aren't identifying the correct replicas.
Arvydas
On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <sr...@gmail.com>> wrote:
Hi Team,
We are struggling with a problem related to cassandra counts, after backup and restore of the cluster. Aaron Morton has suggested to send this to user list, so some one of the list will be able to help me.
We are have a rest api to talk to cassandra and one of our query which fetches count is creating problems for us.
We have done backup and restore and copied all the data to new cluster. We have done nodetool refresh on the tables, and did the nodetool repair as well.
However, one of our key API call is returning inconsistent results. The result count is 0 in the first call and giving the actual values for later calls. The query frequency is bit high and failure rate has also raised considerably.
1) The count query has partition keys in it. Didnt see any read timeout or any errors from api logs.
2) This is how our code of creating session looks.
val poolingOptions = new PoolingOptions
poolingOptions
.setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
.setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
.setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
.setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
val builtCluster = clusterBuilder.withCredentials(username, password)
.withPoolingOptions(poolingOptions)
.build()
val cassandraSession = builtCluster.get.connect()
val preparedStatement = cassandraSession.prepare(statement).setConsistencyLevel(ConsistencyLevel.QUORUM)
cassandraSession.execute(preparedStatement.bind(args :_*))
Query: SELECT count(*) FROM table_name WHERE parition_column=? AND text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND date_column_of_clustering_key>=?
3) Cluster configuration:
6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each machine is equipped with 16 Cores and 64 GB Ram.
Replication factor is 3, and write consistency is ONE and read consistency is QUORUM.
4) cassandra is never down on any machine
5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
6) nodetool tpstats shows no read failures, and no other failures.
7) Do not see any other issues from system.log of cassandra. We just see few warnings as below.
Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103 - 88 prepared statements discarded in the last minute because cache limit reached (32 MB)
The first api call returns 0 and the api calls later gives right values.
Please let me know, if any other details needed.
Could you please have a look at this issue once and kindly give me your inputs? This issue literally broke the confidence on Cassandra from our business team.
Your inputs will be really helpful.
Thank You,
Regards,
Srini
--
Thanks,
Ryan Svihla
--
Thanks,
Ryan Svihla
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: Issue with Cassandra consistency in results
Posted by Shubham Jaju <sh...@vassarlabs.com>.
Hi
This issue used to appear with me . What I figured in my case was
1. I had 3 machines
2. Inserted the data with ONE consistency (i.e there is no guarantee that
data was propagated to remaining nodes , cassandra is supposed to take care
of that).
3. Later I figured also that one of machines has different hard disk
space compared to other two (less and data size was more ) . i.e it was not
able to contain whole set of data.
So I think in above cases ( 2,3 )if you will query you will get different
results as nodes are not in sync.
nodetool repair should solve this problem. But it takes more time if you
have more data.
Check if this solves you problem.
Regards
Shubham Jaju
On Wed, Mar 22, 2017 at 12:23 AM, srinivasarao daruna <
sree.srinu38@gmail.com> wrote:
> The same issue is appearing in CQL Shell as well.
>
> 1) Entered into cqlsh
> 2) SET CONSISTENCY QUORUM;
> 3) Ran a select * with partition key in where cluase.
>
> First result gave 0 records,
> and Next records gave results.
>
> Its really freaking out us at the moment. And nothing in debug.log or
> system.log.
>
> Thank You,
> Regards,
> Srini
>
> On Fri, Mar 17, 2017 at 2:33 AM, daemeon reiydelle <da...@gmail.com>
> wrote:
>
>> The prep is needed. If I recall correctly it must remain in cache for the
>> query to complete. I don't have the docs to dig out the yaml parm to adjust
>> query cache. I had run into the problem stress testing a smallish cluster
>> with many queries at once.
>>
>> Do you have a sense of how many distinct queries are hitting the cluster
>> at peak?
>>
>> If many clients, how do you balance the connection load or do you always
>> hit the same node?
>>
>>
>> sent from my mobile
>> Daemeon Reiydelle
>> skype daemeon.c.m.reiydelle
>> USA 415.501.0198
>>
>> On Mar 16, 2017 3:25 PM, "srinivasarao daruna" <sr...@gmail.com>
>> wrote:
>>
>>> Hi reiydelle,
>>>
>>> I cannot confirm the range as the volume of data is huge and the query
>>> frequency is also high.
>>> If the cache is the cause of issue, can we increase cache size or is
>>> there solution to avoid dropped prep statements.?
>>>
>>>
>>>
>>>
>>>
>>>
>>> Thank You,
>>> Regards,
>>> Srini
>>>
>>> On Thu, Mar 16, 2017 at 2:13 PM, daemeon reiydelle <da...@gmail.com>
>>> wrote:
>>>
>>>> The discard due to oom is causing the zero returned. I would guess a
>>>> cache miss problem of some sort, but not sure. Are you using row, index,
>>>> etc. caches? Are you seeing the failed prep statement on random nodes (duh,
>>>> nodes that have the relevant data ranges)?
>>>>
>>>>
>>>> *.......*
>>>>
>>>>
>>>>
>>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <+1%20415-501-0198>London
>>>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>>>
>>>> On Thu, Mar 16, 2017 at 10:56 AM, Ryan Svihla <rs...@foundev.pro> wrote:
>>>>
>>>>> Depends actually, restore just restores what's there, so if only one
>>>>> node had a copy of the data then only one node had a copy of the data
>>>>> meaning quorum will still be wrong sometimes.
>>>>>
>>>>> On Thu, Mar 16, 2017 at 1:53 PM, Arvydas Jonusonis <
>>>>> arvydas.jonusonis@gmail.com> wrote:
>>>>>
>>>>>> If the data was written at ONE, consistency is not guaranteed. ..but
>>>>>> considering you just restored the cluster, there's a good chance something
>>>>>> else is off.
>>>>>>
>>>>>> On Thu, Mar 16, 2017 at 18:19 srinivasarao daruna <
>>>>>> sree.srinu38@gmail.com> wrote:
>>>>>>
>>>>>>> Want to make read and write QUORUM as well.
>>>>>>>
>>>>>>>
>>>>>>> On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
>>>>>>>
>>>>>>> Replication factor is 3, and write consistency is ONE and
>>>>>>> read consistency is QUORUM.
>>>>>>>
>>>>>>> That combination is not gonna work well:
>>>>>>>
>>>>>>> *Write succeeds to NODE A but fails on node B,C*
>>>>>>>
>>>>>>> *Read goes to NODE B, C*
>>>>>>>
>>>>>>> If you can tolerate some temporary inaccuracy you can use QUORUM but
>>>>>>> may still have the situation where
>>>>>>>
>>>>>>> Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
>>>>>>> Read succeeds on node B and C at timestamp 1
>>>>>>>
>>>>>>> If you need fully race condition free counts I'm afraid you need to
>>>>>>> use SERIAL or LOCAL_SERIAL (for in DC only accuracy)
>>>>>>>
>>>>>>> On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <
>>>>>>> sree.srinu38@gmail.com> wrote:
>>>>>>>
>>>>>>> Replication strategy is SimpleReplicationStrategy.
>>>>>>>
>>>>>>> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>>>>>>>
>>>>>>> I was worried that CL=ALL have more read latency and read failures.
>>>>>>> But won't rule out trying it.
>>>>>>>
>>>>>>> Should I switch select count (*) to select partition_key column?
>>>>>>> Would that be of any help.?
>>>>>>>
>>>>>>>
>>>>>>> Thank you
>>>>>>> Regards
>>>>>>> Srini
>>>>>>>
>>>>>>> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <
>>>>>>> arvydas.jonusonis@gmail.com> wrote:
>>>>>>>
>>>>>>> What are your replication strategy and snitch settings?
>>>>>>>
>>>>>>> Have you tried doing a read at CL=ALL? If it's an actual
>>>>>>> inconsistency issue (missing data), this should cause the correct results
>>>>>>> to be returned. You'll need to run a repair to fix the inconsistencies.
>>>>>>>
>>>>>>> If all the data is actually there, you might have one or several
>>>>>>> nodes that aren't identifying the correct replicas.
>>>>>>>
>>>>>>> Arvydas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
>>>>>>> sree.srinu38@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Team,
>>>>>>>
>>>>>>> We are struggling with a problem related to cassandra counts, after
>>>>>>> backup and restore of the cluster. Aaron Morton has suggested to send this
>>>>>>> to user list, so some one of the list will be able to help me.
>>>>>>>
>>>>>>> We are have a rest api to talk to cassandra and one of our query
>>>>>>> which fetches count is creating problems for us.
>>>>>>>
>>>>>>> We have done backup and restore and copied all the data to new
>>>>>>> cluster. We have done nodetool refresh on the tables, and did the nodetool
>>>>>>> repair as well.
>>>>>>>
>>>>>>> However, one of our key API call is returning inconsistent results.
>>>>>>> The result count is 0 in the first call and giving the actual values for
>>>>>>> later calls. The query frequency is bit high and failure rate has also
>>>>>>> raised considerably.
>>>>>>>
>>>>>>> 1) The count query has partition keys in it. Didnt see any read
>>>>>>> timeout or any errors from api logs.
>>>>>>>
>>>>>>> 2) This is how our code of creating session looks.
>>>>>>>
>>>>>>> val poolingOptions = new PoolingOptions
>>>>>>> poolingOptions
>>>>>>> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
>>>>>>> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
>>>>>>> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
>>>>>>> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>>>>>>>
>>>>>>> val builtCluster = clusterBuilder.withCredentials(username,
>>>>>>> password)
>>>>>>> .withPoolingOptions(poolingOptions)
>>>>>>> .build()
>>>>>>> val cassandraSession = builtCluster.get.connect()
>>>>>>>
>>>>>>> val preparedStatement = cassandraSession.prepare(state
>>>>>>> ment).setConsistencyLevel(ConsistencyLevel.QUORUM)
>>>>>>> cassandraSession.execute(preparedStatement.bind(args :_*))
>>>>>>>
>>>>>>> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
>>>>>>> text_column_of_clustering_key=? AND date_column_of_clustering_key<=?
>>>>>>> AND date_column_of_clustering_key>=?
>>>>>>>
>>>>>>> 3) Cluster configuration:
>>>>>>>
>>>>>>> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
>>>>>>> machine is equipped with 16 Cores and 64 GB Ram.
>>>>>>>
>>>>>>> Replication factor is 3, and write consistency is ONE and
>>>>>>> read consistency is QUORUM.
>>>>>>>
>>>>>>> 4) cassandra is never down on any machine
>>>>>>>
>>>>>>> 5) Using cassandra-driver-core artifact with 3.1.1 version in the
>>>>>>> api.
>>>>>>>
>>>>>>> 6) nodetool tpstats shows no read failures, and no other failures.
>>>>>>>
>>>>>>> 7) Do not see any other issues from system.log of cassandra. We just
>>>>>>> see few warnings as below.
>>>>>>>
>>>>>>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>>>>>>> 1.000MiB
>>>>>>> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141
>>>>>>> QueryProcessor.java:103 - 88 prepared statements discarded in the last
>>>>>>> minute because cache limit reached (32 MB)
>>>>>>> The first api call returns 0 and the api calls later gives right
>>>>>>> values.
>>>>>>>
>>>>>>> Please let me know, if any other details needed.
>>>>>>> Could you please have a look at this issue once and kindly give me
>>>>>>> your inputs? This issue literally broke the confidence on Cassandra from
>>>>>>> our business team.
>>>>>>>
>>>>>>> Your inputs will be really helpful.
>>>>>>>
>>>>>>> Thank You,
>>>>>>> Regards,
>>>>>>> Srini
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ryan Svihla
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> Ryan Svihla
>>>>>
>>>>>
>>>>
>>>
>
Re: Issue with Cassandra consistency in results
Posted by srinivasarao daruna <sr...@gmail.com>.
The same issue is appearing in CQL Shell as well.
1) Entered into cqlsh
2) SET CONSISTENCY QUORUM;
3) Ran a select * with partition key in where cluase.
First result gave 0 records,
and Next records gave results.
Its really freaking out us at the moment. And nothing in debug.log or
system.log.
Thank You,
Regards,
Srini
On Fri, Mar 17, 2017 at 2:33 AM, daemeon reiydelle <da...@gmail.com>
wrote:
> The prep is needed. If I recall correctly it must remain in cache for the
> query to complete. I don't have the docs to dig out the yaml parm to adjust
> query cache. I had run into the problem stress testing a smallish cluster
> with many queries at once.
>
> Do you have a sense of how many distinct queries are hitting the cluster
> at peak?
>
> If many clients, how do you balance the connection load or do you always
> hit the same node?
>
>
> sent from my mobile
> Daemeon Reiydelle
> skype daemeon.c.m.reiydelle
> USA 415.501.0198
>
> On Mar 16, 2017 3:25 PM, "srinivasarao daruna" <sr...@gmail.com>
> wrote:
>
>> Hi reiydelle,
>>
>> I cannot confirm the range as the volume of data is huge and the query
>> frequency is also high.
>> If the cache is the cause of issue, can we increase cache size or is
>> there solution to avoid dropped prep statements.?
>>
>>
>>
>>
>>
>>
>> Thank You,
>> Regards,
>> Srini
>>
>> On Thu, Mar 16, 2017 at 2:13 PM, daemeon reiydelle <da...@gmail.com>
>> wrote:
>>
>>> The discard due to oom is causing the zero returned. I would guess a
>>> cache miss problem of some sort, but not sure. Are you using row, index,
>>> etc. caches? Are you seeing the failed prep statement on random nodes (duh,
>>> nodes that have the relevant data ranges)?
>>>
>>>
>>> *.......*
>>>
>>>
>>>
>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <+1%20415-501-0198>London
>>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>>
>>> On Thu, Mar 16, 2017 at 10:56 AM, Ryan Svihla <rs...@foundev.pro> wrote:
>>>
>>>> Depends actually, restore just restores what's there, so if only one
>>>> node had a copy of the data then only one node had a copy of the data
>>>> meaning quorum will still be wrong sometimes.
>>>>
>>>> On Thu, Mar 16, 2017 at 1:53 PM, Arvydas Jonusonis <
>>>> arvydas.jonusonis@gmail.com> wrote:
>>>>
>>>>> If the data was written at ONE, consistency is not guaranteed. ..but
>>>>> considering you just restored the cluster, there's a good chance something
>>>>> else is off.
>>>>>
>>>>> On Thu, Mar 16, 2017 at 18:19 srinivasarao daruna <
>>>>> sree.srinu38@gmail.com> wrote:
>>>>>
>>>>>> Want to make read and write QUORUM as well.
>>>>>>
>>>>>>
>>>>>> On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
>>>>>>
>>>>>> Replication factor is 3, and write consistency is ONE and
>>>>>> read consistency is QUORUM.
>>>>>>
>>>>>> That combination is not gonna work well:
>>>>>>
>>>>>> *Write succeeds to NODE A but fails on node B,C*
>>>>>>
>>>>>> *Read goes to NODE B, C*
>>>>>>
>>>>>> If you can tolerate some temporary inaccuracy you can use QUORUM but
>>>>>> may still have the situation where
>>>>>>
>>>>>> Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
>>>>>> Read succeeds on node B and C at timestamp 1
>>>>>>
>>>>>> If you need fully race condition free counts I'm afraid you need to
>>>>>> use SERIAL or LOCAL_SERIAL (for in DC only accuracy)
>>>>>>
>>>>>> On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <
>>>>>> sree.srinu38@gmail.com> wrote:
>>>>>>
>>>>>> Replication strategy is SimpleReplicationStrategy.
>>>>>>
>>>>>> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>>>>>>
>>>>>> I was worried that CL=ALL have more read latency and read failures.
>>>>>> But won't rule out trying it.
>>>>>>
>>>>>> Should I switch select count (*) to select partition_key column?
>>>>>> Would that be of any help.?
>>>>>>
>>>>>>
>>>>>> Thank you
>>>>>> Regards
>>>>>> Srini
>>>>>>
>>>>>> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <
>>>>>> arvydas.jonusonis@gmail.com> wrote:
>>>>>>
>>>>>> What are your replication strategy and snitch settings?
>>>>>>
>>>>>> Have you tried doing a read at CL=ALL? If it's an actual
>>>>>> inconsistency issue (missing data), this should cause the correct results
>>>>>> to be returned. You'll need to run a repair to fix the inconsistencies.
>>>>>>
>>>>>> If all the data is actually there, you might have one or several
>>>>>> nodes that aren't identifying the correct replicas.
>>>>>>
>>>>>> Arvydas
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
>>>>>> sree.srinu38@gmail.com> wrote:
>>>>>>
>>>>>> Hi Team,
>>>>>>
>>>>>> We are struggling with a problem related to cassandra counts, after
>>>>>> backup and restore of the cluster. Aaron Morton has suggested to send this
>>>>>> to user list, so some one of the list will be able to help me.
>>>>>>
>>>>>> We are have a rest api to talk to cassandra and one of our query
>>>>>> which fetches count is creating problems for us.
>>>>>>
>>>>>> We have done backup and restore and copied all the data to new
>>>>>> cluster. We have done nodetool refresh on the tables, and did the nodetool
>>>>>> repair as well.
>>>>>>
>>>>>> However, one of our key API call is returning inconsistent results.
>>>>>> The result count is 0 in the first call and giving the actual values for
>>>>>> later calls. The query frequency is bit high and failure rate has also
>>>>>> raised considerably.
>>>>>>
>>>>>> 1) The count query has partition keys in it. Didnt see any read
>>>>>> timeout or any errors from api logs.
>>>>>>
>>>>>> 2) This is how our code of creating session looks.
>>>>>>
>>>>>> val poolingOptions = new PoolingOptions
>>>>>> poolingOptions
>>>>>> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
>>>>>> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
>>>>>> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
>>>>>> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>>>>>>
>>>>>> val builtCluster = clusterBuilder.withCredentials(username, password)
>>>>>> .withPoolingOptions(poolingOptions)
>>>>>> .build()
>>>>>> val cassandraSession = builtCluster.get.connect()
>>>>>>
>>>>>> val preparedStatement = cassandraSession.prepare(state
>>>>>> ment).setConsistencyLevel(ConsistencyLevel.QUORUM)
>>>>>> cassandraSession.execute(preparedStatement.bind(args :_*))
>>>>>>
>>>>>> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
>>>>>> text_column_of_clustering_key=? AND date_column_of_clustering_key<=?
>>>>>> AND date_column_of_clustering_key>=?
>>>>>>
>>>>>> 3) Cluster configuration:
>>>>>>
>>>>>> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
>>>>>> machine is equipped with 16 Cores and 64 GB Ram.
>>>>>>
>>>>>> Replication factor is 3, and write consistency is ONE and
>>>>>> read consistency is QUORUM.
>>>>>>
>>>>>> 4) cassandra is never down on any machine
>>>>>>
>>>>>> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>>>>>>
>>>>>> 6) nodetool tpstats shows no read failures, and no other failures.
>>>>>>
>>>>>> 7) Do not see any other issues from system.log of cassandra. We just
>>>>>> see few warnings as below.
>>>>>>
>>>>>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>>>>>> 1.000MiB
>>>>>> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141
>>>>>> QueryProcessor.java:103 - 88 prepared statements discarded in the last
>>>>>> minute because cache limit reached (32 MB)
>>>>>> The first api call returns 0 and the api calls later gives right
>>>>>> values.
>>>>>>
>>>>>> Please let me know, if any other details needed.
>>>>>> Could you please have a look at this issue once and kindly give me
>>>>>> your inputs? This issue literally broke the confidence on Cassandra from
>>>>>> our business team.
>>>>>>
>>>>>> Your inputs will be really helpful.
>>>>>>
>>>>>> Thank You,
>>>>>> Regards,
>>>>>> Srini
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Thanks,
>>>>>> Ryan Svihla
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Ryan Svihla
>>>>
>>>>
>>>
>>
Re: Issue with Cassandra consistency in results
Posted by daemeon reiydelle <da...@gmail.com>.
The prep is needed. If I recall correctly it must remain in cache for the
query to complete. I don't have the docs to dig out the yaml parm to adjust
query cache. I had run into the problem stress testing a smallish cluster
with many queries at once.
Do you have a sense of how many distinct queries are hitting the cluster at
peak?
If many clients, how do you balance the connection load or do you always
hit the same node?
sent from my mobile
Daemeon Reiydelle
skype daemeon.c.m.reiydelle
USA 415.501.0198
On Mar 16, 2017 3:25 PM, "srinivasarao daruna" <sr...@gmail.com>
wrote:
> Hi reiydelle,
>
> I cannot confirm the range as the volume of data is huge and the query
> frequency is also high.
> If the cache is the cause of issue, can we increase cache size or is there
> solution to avoid dropped prep statements.?
>
>
>
>
>
>
> Thank You,
> Regards,
> Srini
>
> On Thu, Mar 16, 2017 at 2:13 PM, daemeon reiydelle <da...@gmail.com>
> wrote:
>
>> The discard due to oom is causing the zero returned. I would guess a
>> cache miss problem of some sort, but not sure. Are you using row, index,
>> etc. caches? Are you seeing the failed prep statement on random nodes (duh,
>> nodes that have the relevant data ranges)?
>>
>>
>> *.......*
>>
>>
>>
>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <+1%20415-501-0198>London
>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>
>> On Thu, Mar 16, 2017 at 10:56 AM, Ryan Svihla <rs...@foundev.pro> wrote:
>>
>>> Depends actually, restore just restores what's there, so if only one
>>> node had a copy of the data then only one node had a copy of the data
>>> meaning quorum will still be wrong sometimes.
>>>
>>> On Thu, Mar 16, 2017 at 1:53 PM, Arvydas Jonusonis <
>>> arvydas.jonusonis@gmail.com> wrote:
>>>
>>>> If the data was written at ONE, consistency is not guaranteed. ..but
>>>> considering you just restored the cluster, there's a good chance something
>>>> else is off.
>>>>
>>>> On Thu, Mar 16, 2017 at 18:19 srinivasarao daruna <
>>>> sree.srinu38@gmail.com> wrote:
>>>>
>>>>> Want to make read and write QUORUM as well.
>>>>>
>>>>>
>>>>> On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
>>>>>
>>>>> Replication factor is 3, and write consistency is ONE and read
>>>>> consistency is QUORUM.
>>>>>
>>>>> That combination is not gonna work well:
>>>>>
>>>>> *Write succeeds to NODE A but fails on node B,C*
>>>>>
>>>>> *Read goes to NODE B, C*
>>>>>
>>>>> If you can tolerate some temporary inaccuracy you can use QUORUM but
>>>>> may still have the situation where
>>>>>
>>>>> Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
>>>>> Read succeeds on node B and C at timestamp 1
>>>>>
>>>>> If you need fully race condition free counts I'm afraid you need to
>>>>> use SERIAL or LOCAL_SERIAL (for in DC only accuracy)
>>>>>
>>>>> On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <
>>>>> sree.srinu38@gmail.com> wrote:
>>>>>
>>>>> Replication strategy is SimpleReplicationStrategy.
>>>>>
>>>>> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>>>>>
>>>>> I was worried that CL=ALL have more read latency and read failures.
>>>>> But won't rule out trying it.
>>>>>
>>>>> Should I switch select count (*) to select partition_key column? Would
>>>>> that be of any help.?
>>>>>
>>>>>
>>>>> Thank you
>>>>> Regards
>>>>> Srini
>>>>>
>>>>> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <
>>>>> arvydas.jonusonis@gmail.com> wrote:
>>>>>
>>>>> What are your replication strategy and snitch settings?
>>>>>
>>>>> Have you tried doing a read at CL=ALL? If it's an actual inconsistency
>>>>> issue (missing data), this should cause the correct results to be returned.
>>>>> You'll need to run a repair to fix the inconsistencies.
>>>>>
>>>>> If all the data is actually there, you might have one or several nodes
>>>>> that aren't identifying the correct replicas.
>>>>>
>>>>> Arvydas
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
>>>>> sree.srinu38@gmail.com> wrote:
>>>>>
>>>>> Hi Team,
>>>>>
>>>>> We are struggling with a problem related to cassandra counts, after
>>>>> backup and restore of the cluster. Aaron Morton has suggested to send this
>>>>> to user list, so some one of the list will be able to help me.
>>>>>
>>>>> We are have a rest api to talk to cassandra and one of our query which
>>>>> fetches count is creating problems for us.
>>>>>
>>>>> We have done backup and restore and copied all the data to new
>>>>> cluster. We have done nodetool refresh on the tables, and did the nodetool
>>>>> repair as well.
>>>>>
>>>>> However, one of our key API call is returning inconsistent results.
>>>>> The result count is 0 in the first call and giving the actual values for
>>>>> later calls. The query frequency is bit high and failure rate has also
>>>>> raised considerably.
>>>>>
>>>>> 1) The count query has partition keys in it. Didnt see any read
>>>>> timeout or any errors from api logs.
>>>>>
>>>>> 2) This is how our code of creating session looks.
>>>>>
>>>>> val poolingOptions = new PoolingOptions
>>>>> poolingOptions
>>>>> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
>>>>> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
>>>>> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
>>>>> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>>>>>
>>>>> val builtCluster = clusterBuilder.withCredentials(username, password)
>>>>> .withPoolingOptions(poolingOptions)
>>>>> .build()
>>>>> val cassandraSession = builtCluster.get.connect()
>>>>>
>>>>> val preparedStatement = cassandraSession.prepare(state
>>>>> ment).setConsistencyLevel(ConsistencyLevel.QUORUM)
>>>>> cassandraSession.execute(preparedStatement.bind(args :_*))
>>>>>
>>>>> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
>>>>> text_column_of_clustering_key=? AND date_column_of_clustering_key<=?
>>>>> AND date_column_of_clustering_key>=?
>>>>>
>>>>> 3) Cluster configuration:
>>>>>
>>>>> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
>>>>> machine is equipped with 16 Cores and 64 GB Ram.
>>>>>
>>>>> Replication factor is 3, and write consistency is ONE and read
>>>>> consistency is QUORUM.
>>>>>
>>>>> 4) cassandra is never down on any machine
>>>>>
>>>>> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>>>>>
>>>>> 6) nodetool tpstats shows no read failures, and no other failures.
>>>>>
>>>>> 7) Do not see any other issues from system.log of cassandra. We just
>>>>> see few warnings as below.
>>>>>
>>>>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>>>>> 1.000MiB
>>>>> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141
>>>>> QueryProcessor.java:103 - 88 prepared statements discarded in the last
>>>>> minute because cache limit reached (32 MB)
>>>>> The first api call returns 0 and the api calls later gives right
>>>>> values.
>>>>>
>>>>> Please let me know, if any other details needed.
>>>>> Could you please have a look at this issue once and kindly give me
>>>>> your inputs? This issue literally broke the confidence on Cassandra from
>>>>> our business team.
>>>>>
>>>>> Your inputs will be really helpful.
>>>>>
>>>>> Thank You,
>>>>> Regards,
>>>>> Srini
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>> Ryan Svihla
>>>>>
>>>>>
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Ryan Svihla
>>>
>>>
>>
>
Re: Issue with Cassandra consistency in results
Posted by srinivasarao daruna <sr...@gmail.com>.
Hi reiydelle,
I cannot confirm the range as the volume of data is huge and the query
frequency is also high.
If the cache is the cause of issue, can we increase cache size or is there
solution to avoid dropped prep statements.?
Thank You,
Regards,
Srini
On Thu, Mar 16, 2017 at 2:13 PM, daemeon reiydelle <da...@gmail.com>
wrote:
> The discard due to oom is causing the zero returned. I would guess a cache
> miss problem of some sort, but not sure. Are you using row, index, etc.
> caches? Are you seeing the failed prep statement on random nodes (duh,
> nodes that have the relevant data ranges)?
>
>
> *.......*
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <+1%20415-501-0198>London
> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>
> On Thu, Mar 16, 2017 at 10:56 AM, Ryan Svihla <rs...@foundev.pro> wrote:
>
>> Depends actually, restore just restores what's there, so if only one node
>> had a copy of the data then only one node had a copy of the data meaning
>> quorum will still be wrong sometimes.
>>
>> On Thu, Mar 16, 2017 at 1:53 PM, Arvydas Jonusonis <
>> arvydas.jonusonis@gmail.com> wrote:
>>
>>> If the data was written at ONE, consistency is not guaranteed. ..but
>>> considering you just restored the cluster, there's a good chance something
>>> else is off.
>>>
>>> On Thu, Mar 16, 2017 at 18:19 srinivasarao daruna <
>>> sree.srinu38@gmail.com> wrote:
>>>
>>>> Want to make read and write QUORUM as well.
>>>>
>>>>
>>>> On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
>>>>
>>>> Replication factor is 3, and write consistency is ONE and read
>>>> consistency is QUORUM.
>>>>
>>>> That combination is not gonna work well:
>>>>
>>>> *Write succeeds to NODE A but fails on node B,C*
>>>>
>>>> *Read goes to NODE B, C*
>>>>
>>>> If you can tolerate some temporary inaccuracy you can use QUORUM but
>>>> may still have the situation where
>>>>
>>>> Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
>>>> Read succeeds on node B and C at timestamp 1
>>>>
>>>> If you need fully race condition free counts I'm afraid you need to use
>>>> SERIAL or LOCAL_SERIAL (for in DC only accuracy)
>>>>
>>>> On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <
>>>> sree.srinu38@gmail.com> wrote:
>>>>
>>>> Replication strategy is SimpleReplicationStrategy.
>>>>
>>>> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>>>>
>>>> I was worried that CL=ALL have more read latency and read failures. But
>>>> won't rule out trying it.
>>>>
>>>> Should I switch select count (*) to select partition_key column? Would
>>>> that be of any help.?
>>>>
>>>>
>>>> Thank you
>>>> Regards
>>>> Srini
>>>>
>>>> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <
>>>> arvydas.jonusonis@gmail.com> wrote:
>>>>
>>>> What are your replication strategy and snitch settings?
>>>>
>>>> Have you tried doing a read at CL=ALL? If it's an actual inconsistency
>>>> issue (missing data), this should cause the correct results to be returned.
>>>> You'll need to run a repair to fix the inconsistencies.
>>>>
>>>> If all the data is actually there, you might have one or several nodes
>>>> that aren't identifying the correct replicas.
>>>>
>>>> Arvydas
>>>>
>>>>
>>>>
>>>> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
>>>> sree.srinu38@gmail.com> wrote:
>>>>
>>>> Hi Team,
>>>>
>>>> We are struggling with a problem related to cassandra counts, after
>>>> backup and restore of the cluster. Aaron Morton has suggested to send this
>>>> to user list, so some one of the list will be able to help me.
>>>>
>>>> We are have a rest api to talk to cassandra and one of our query which
>>>> fetches count is creating problems for us.
>>>>
>>>> We have done backup and restore and copied all the data to new cluster.
>>>> We have done nodetool refresh on the tables, and did the nodetool repair as
>>>> well.
>>>>
>>>> However, one of our key API call is returning inconsistent results. The
>>>> result count is 0 in the first call and giving the actual values for later
>>>> calls. The query frequency is bit high and failure rate has also raised
>>>> considerably.
>>>>
>>>> 1) The count query has partition keys in it. Didnt see any read timeout
>>>> or any errors from api logs.
>>>>
>>>> 2) This is how our code of creating session looks.
>>>>
>>>> val poolingOptions = new PoolingOptions
>>>> poolingOptions
>>>> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
>>>> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
>>>> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
>>>> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>>>>
>>>> val builtCluster = clusterBuilder.withCredentials(username, password)
>>>> .withPoolingOptions(poolingOptions)
>>>> .build()
>>>> val cassandraSession = builtCluster.get.connect()
>>>>
>>>> val preparedStatement = cassandraSession.prepare(state
>>>> ment).setConsistencyLevel(ConsistencyLevel.QUORUM)
>>>> cassandraSession.execute(preparedStatement.bind(args :_*))
>>>>
>>>> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
>>>> text_column_of_clustering_key=? AND date_column_of_clustering_key<=?
>>>> AND date_column_of_clustering_key>=?
>>>>
>>>> 3) Cluster configuration:
>>>>
>>>> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
>>>> machine is equipped with 16 Cores and 64 GB Ram.
>>>>
>>>> Replication factor is 3, and write consistency is ONE and read
>>>> consistency is QUORUM.
>>>>
>>>> 4) cassandra is never down on any machine
>>>>
>>>> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>>>>
>>>> 6) nodetool tpstats shows no read failures, and no other failures.
>>>>
>>>> 7) Do not see any other issues from system.log of cassandra. We just
>>>> see few warnings as below.
>>>>
>>>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>>>> 1.000MiB
>>>> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141
>>>> QueryProcessor.java:103 - 88 prepared statements discarded in the last
>>>> minute because cache limit reached (32 MB)
>>>> The first api call returns 0 and the api calls later gives right values.
>>>>
>>>> Please let me know, if any other details needed.
>>>> Could you please have a look at this issue once and kindly give me your
>>>> inputs? This issue literally broke the confidence on Cassandra from our
>>>> business team.
>>>>
>>>> Your inputs will be really helpful.
>>>>
>>>> Thank You,
>>>> Regards,
>>>> Srini
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>> Ryan Svihla
>>>>
>>>>
>>
>>
>> --
>>
>> Thanks,
>> Ryan Svihla
>>
>>
>
Re: Issue with Cassandra consistency in results
Posted by daemeon reiydelle <da...@gmail.com>.
The discard due to oom is causing the zero returned. I would guess a cache
miss problem of some sort, but not sure. Are you using row, index, etc.
caches? Are you seeing the failed prep statement on random nodes (duh,
nodes that have the relevant data ranges)?
*.......*
*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*
On Thu, Mar 16, 2017 at 10:56 AM, Ryan Svihla <rs...@foundev.pro> wrote:
> Depends actually, restore just restores what's there, so if only one node
> had a copy of the data then only one node had a copy of the data meaning
> quorum will still be wrong sometimes.
>
> On Thu, Mar 16, 2017 at 1:53 PM, Arvydas Jonusonis <
> arvydas.jonusonis@gmail.com> wrote:
>
>> If the data was written at ONE, consistency is not guaranteed. ..but
>> considering you just restored the cluster, there's a good chance something
>> else is off.
>>
>> On Thu, Mar 16, 2017 at 18:19 srinivasarao daruna <sr...@gmail.com>
>> wrote:
>>
>>> Want to make read and write QUORUM as well.
>>>
>>>
>>> On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
>>>
>>> Replication factor is 3, and write consistency is ONE and read
>>> consistency is QUORUM.
>>>
>>> That combination is not gonna work well:
>>>
>>> *Write succeeds to NODE A but fails on node B,C*
>>>
>>> *Read goes to NODE B, C*
>>>
>>> If you can tolerate some temporary inaccuracy you can use QUORUM but may
>>> still have the situation where
>>>
>>> Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
>>> Read succeeds on node B and C at timestamp 1
>>>
>>> If you need fully race condition free counts I'm afraid you need to use
>>> SERIAL or LOCAL_SERIAL (for in DC only accuracy)
>>>
>>> On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <
>>> sree.srinu38@gmail.com> wrote:
>>>
>>> Replication strategy is SimpleReplicationStrategy.
>>>
>>> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>>>
>>> I was worried that CL=ALL have more read latency and read failures. But
>>> won't rule out trying it.
>>>
>>> Should I switch select count (*) to select partition_key column? Would
>>> that be of any help.?
>>>
>>>
>>> Thank you
>>> Regards
>>> Srini
>>>
>>> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <
>>> arvydas.jonusonis@gmail.com> wrote:
>>>
>>> What are your replication strategy and snitch settings?
>>>
>>> Have you tried doing a read at CL=ALL? If it's an actual inconsistency
>>> issue (missing data), this should cause the correct results to be returned.
>>> You'll need to run a repair to fix the inconsistencies.
>>>
>>> If all the data is actually there, you might have one or several nodes
>>> that aren't identifying the correct replicas.
>>>
>>> Arvydas
>>>
>>>
>>>
>>> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
>>> sree.srinu38@gmail.com> wrote:
>>>
>>> Hi Team,
>>>
>>> We are struggling with a problem related to cassandra counts, after
>>> backup and restore of the cluster. Aaron Morton has suggested to send this
>>> to user list, so some one of the list will be able to help me.
>>>
>>> We are have a rest api to talk to cassandra and one of our query which
>>> fetches count is creating problems for us.
>>>
>>> We have done backup and restore and copied all the data to new cluster.
>>> We have done nodetool refresh on the tables, and did the nodetool repair as
>>> well.
>>>
>>> However, one of our key API call is returning inconsistent results. The
>>> result count is 0 in the first call and giving the actual values for later
>>> calls. The query frequency is bit high and failure rate has also raised
>>> considerably.
>>>
>>> 1) The count query has partition keys in it. Didnt see any read timeout
>>> or any errors from api logs.
>>>
>>> 2) This is how our code of creating session looks.
>>>
>>> val poolingOptions = new PoolingOptions
>>> poolingOptions
>>> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
>>> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
>>> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
>>> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>>>
>>> val builtCluster = clusterBuilder.withCredentials(username, password)
>>> .withPoolingOptions(poolingOptions)
>>> .build()
>>> val cassandraSession = builtCluster.get.connect()
>>>
>>> val preparedStatement = cassandraSession.prepare(state
>>> ment).setConsistencyLevel(ConsistencyLevel.QUORUM)
>>> cassandraSession.execute(preparedStatement.bind(args :_*))
>>>
>>> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
>>> text_column_of_clustering_key=? AND date_column_of_clustering_key<=?
>>> AND date_column_of_clustering_key>=?
>>>
>>> 3) Cluster configuration:
>>>
>>> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
>>> machine is equipped with 16 Cores and 64 GB Ram.
>>>
>>> Replication factor is 3, and write consistency is ONE and read
>>> consistency is QUORUM.
>>>
>>> 4) cassandra is never down on any machine
>>>
>>> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>>>
>>> 6) nodetool tpstats shows no read failures, and no other failures.
>>>
>>> 7) Do not see any other issues from system.log of cassandra. We just see
>>> few warnings as below.
>>>
>>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>>> 1.000MiB
>>> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103
>>> - 88 prepared statements discarded in the last minute because cache limit
>>> reached (32 MB)
>>> The first api call returns 0 and the api calls later gives right values.
>>>
>>> Please let me know, if any other details needed.
>>> Could you please have a look at this issue once and kindly give me your
>>> inputs? This issue literally broke the confidence on Cassandra from our
>>> business team.
>>>
>>> Your inputs will be really helpful.
>>>
>>> Thank You,
>>> Regards,
>>> Srini
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Ryan Svihla
>>>
>>>
>
>
> --
>
> Thanks,
> Ryan Svihla
>
>
AW: Issue with Cassandra consistency in results
Posted by j....@enercast.de.
Hi,
doing a quick scan over the thread two things that came into my mind:
Frist, did the restore copy the sstables to the right machines back? Node As data to node A and so on?
Second, did you run full repairs on every node? Not just incremental ones which now is the default?
Also a look into debug.log is an option.
If all done already, nevermind.
Gesendet von meinem Windows 10 Phone
Von: Ryan Svihla
Gesendet: Donnerstag, 16. März 2017 18:57
An: user
Betreff: Re: Issue with Cassandra consistency in results
Depends actually, restore just restores what's there, so if only one node had a copy of the data then only one node had a copy of the data meaning quorum will still be wrong sometimes.
On Thu, Mar 16, 2017 at 1:53 PM, Arvydas Jonusonis <ar...@gmail.com> wrote:
If the data was written at ONE, consistency is not guaranteed. ..but considering you just restored the cluster, there's a good chance something else is off.
On Thu, Mar 16, 2017 at 18:19 srinivasarao daruna <sr...@gmail.com> wrote:
Want to make read and write QUORUM as well.
On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
Replication factor is 3, and write consistency is ONE and read consistency is QUORUM.
That combination is not gonna work well:
Write succeeds to NODE A but fails on node B,C
Read goes to NODE B, C
If you can tolerate some temporary inaccuracy you can use QUORUM but may still have the situation where
Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
Read succeeds on node B and C at timestamp 1
If you need fully race condition free counts I'm afraid you need to use SERIAL or LOCAL_SERIAL (for in DC only accuracy)
On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <sr...@gmail.com> wrote:
Replication strategy is SimpleReplicationStrategy.
Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
I was worried that CL=ALL have more read latency and read failures. But won't rule out trying it.
Should I switch select count (*) to select partition_key column? Would that be of any help.?
Thank you
Regards
Srini
On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <ar...@gmail.com> wrote:
What are your replication strategy and snitch settings?
Have you tried doing a read at CL=ALL? If it's an actual inconsistency issue (missing data), this should cause the correct results to be returned. You'll need to run a repair to fix the inconsistencies.
If all the data is actually there, you might have one or several nodes that aren't identifying the correct replicas.
Arvydas
On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <sr...@gmail.com> wrote:
Hi Team,
We are struggling with a problem related to cassandra counts, after backup and restore of the cluster. Aaron Morton has suggested to send this to user list, so some one of the list will be able to help me.
We are have a rest api to talk to cassandra and one of our query which fetches count is creating problems for us.
We have done backup and restore and copied all the data to new cluster. We have done nodetool refresh on the tables, and did the nodetool repair as well.
However, one of our key API call is returning inconsistent results. The result count is 0 in the first call and giving the actual values for later calls. The query frequency is bit high and failure rate has also raised considerably.
1) The count query has partition keys in it. Didnt see any read timeout or any errors from api logs.
2) This is how our code of creating session looks.
val poolingOptions = new PoolingOptions
poolingOptions
.setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
.setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
.setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
.setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
val builtCluster = clusterBuilder.withCredentials(username, password)
.withPoolingOptions(poolingOptions)
.build()
val cassandraSession = builtCluster.get.connect()
val preparedStatement = cassandraSession.prepare(statement).setConsistencyLevel(ConsistencyLevel.QUORUM)
cassandraSession.execute(preparedStatement.bind(args :_*))
Query: SELECT count(*) FROM table_name WHERE parition_column=? AND text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND date_column_of_clustering_key>=?
3) Cluster configuration:
6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each machine is equipped with 16 Cores and 64 GB Ram.
Replication factor is 3, and write consistency is ONE and read consistency is QUORUM.
4) cassandra is never down on any machine
5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
6) nodetool tpstats shows no read failures, and no other failures.
7) Do not see any other issues from system.log of cassandra. We just see few warnings as below.
Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103 - 88 prepared statements discarded in the last minute because cache limit reached (32 MB)
The first api call returns 0 and the api calls later gives right values.
Please let me know, if any other details needed.
Could you please have a look at this issue once and kindly give me your inputs? This issue literally broke the confidence on Cassandra from our business team.
Your inputs will be really helpful.
Thank You,
Regards,
Srini
--
Thanks,
Ryan Svihla
--
Thanks,
Ryan Svihla
Re: Issue with Cassandra consistency in results
Posted by Ryan Svihla <rs...@foundev.pro>.
Depends actually, restore just restores what's there, so if only one node
had a copy of the data then only one node had a copy of the data meaning
quorum will still be wrong sometimes.
On Thu, Mar 16, 2017 at 1:53 PM, Arvydas Jonusonis <
arvydas.jonusonis@gmail.com> wrote:
> If the data was written at ONE, consistency is not guaranteed. ..but
> considering you just restored the cluster, there's a good chance something
> else is off.
>
> On Thu, Mar 16, 2017 at 18:19 srinivasarao daruna <sr...@gmail.com>
> wrote:
>
>> Want to make read and write QUORUM as well.
>>
>>
>> On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
>>
>> Replication factor is 3, and write consistency is ONE and read
>> consistency is QUORUM.
>>
>> That combination is not gonna work well:
>>
>> *Write succeeds to NODE A but fails on node B,C*
>>
>> *Read goes to NODE B, C*
>>
>> If you can tolerate some temporary inaccuracy you can use QUORUM but may
>> still have the situation where
>>
>> Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
>> Read succeeds on node B and C at timestamp 1
>>
>> If you need fully race condition free counts I'm afraid you need to use
>> SERIAL or LOCAL_SERIAL (for in DC only accuracy)
>>
>> On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <
>> sree.srinu38@gmail.com> wrote:
>>
>> Replication strategy is SimpleReplicationStrategy.
>>
>> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>>
>> I was worried that CL=ALL have more read latency and read failures. But
>> won't rule out trying it.
>>
>> Should I switch select count (*) to select partition_key column? Would
>> that be of any help.?
>>
>>
>> Thank you
>> Regards
>> Srini
>>
>> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <
>> arvydas.jonusonis@gmail.com> wrote:
>>
>> What are your replication strategy and snitch settings?
>>
>> Have you tried doing a read at CL=ALL? If it's an actual inconsistency
>> issue (missing data), this should cause the correct results to be returned.
>> You'll need to run a repair to fix the inconsistencies.
>>
>> If all the data is actually there, you might have one or several nodes
>> that aren't identifying the correct replicas.
>>
>> Arvydas
>>
>>
>>
>> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
>> sree.srinu38@gmail.com> wrote:
>>
>> Hi Team,
>>
>> We are struggling with a problem related to cassandra counts, after
>> backup and restore of the cluster. Aaron Morton has suggested to send this
>> to user list, so some one of the list will be able to help me.
>>
>> We are have a rest api to talk to cassandra and one of our query which
>> fetches count is creating problems for us.
>>
>> We have done backup and restore and copied all the data to new cluster.
>> We have done nodetool refresh on the tables, and did the nodetool repair as
>> well.
>>
>> However, one of our key API call is returning inconsistent results. The
>> result count is 0 in the first call and giving the actual values for later
>> calls. The query frequency is bit high and failure rate has also raised
>> considerably.
>>
>> 1) The count query has partition keys in it. Didnt see any read timeout
>> or any errors from api logs.
>>
>> 2) This is how our code of creating session looks.
>>
>> val poolingOptions = new PoolingOptions
>> poolingOptions
>> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
>> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
>> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
>> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>>
>> val builtCluster = clusterBuilder.withCredentials(username, password)
>> .withPoolingOptions(poolingOptions)
>> .build()
>> val cassandraSession = builtCluster.get.connect()
>>
>> val preparedStatement = cassandraSession.prepare(statement).
>> setConsistencyLevel(ConsistencyLevel.QUORUM)
>> cassandraSession.execute(preparedStatement.bind(args :_*))
>>
>> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
>> text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND
>> date_column_of_clustering_key>=?
>>
>> 3) Cluster configuration:
>>
>> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
>> machine is equipped with 16 Cores and 64 GB Ram.
>>
>> Replication factor is 3, and write consistency is ONE and read
>> consistency is QUORUM.
>>
>> 4) cassandra is never down on any machine
>>
>> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>>
>> 6) nodetool tpstats shows no read failures, and no other failures.
>>
>> 7) Do not see any other issues from system.log of cassandra. We just see
>> few warnings as below.
>>
>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>> 1.000MiB
>> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103
>> - 88 prepared statements discarded in the last minute because cache limit
>> reached (32 MB)
>> The first api call returns 0 and the api calls later gives right values.
>>
>> Please let me know, if any other details needed.
>> Could you please have a look at this issue once and kindly give me your
>> inputs? This issue literally broke the confidence on Cassandra from our
>> business team.
>>
>> Your inputs will be really helpful.
>>
>> Thank You,
>> Regards,
>> Srini
>>
>>
>>
>>
>>
>>
>> --
>>
>> Thanks,
>> Ryan Svihla
>>
>>
--
Thanks,
Ryan Svihla
Re: Issue with Cassandra consistency in results
Posted by Arvydas Jonusonis <ar...@gmail.com>.
If the data was written at ONE, consistency is not guaranteed. ..but
considering you just restored the cluster, there's a good chance something
else is off.
On Thu, Mar 16, 2017 at 18:19 srinivasarao daruna <sr...@gmail.com>
wrote:
> Want to make read and write QUORUM as well.
>
>
> On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
>
> Replication factor is 3, and write consistency is ONE and read
> consistency is QUORUM.
>
> That combination is not gonna work well:
>
> *Write succeeds to NODE A but fails on node B,C*
>
> *Read goes to NODE B, C*
>
> If you can tolerate some temporary inaccuracy you can use QUORUM but may
> still have the situation where
>
> Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
> Read succeeds on node B and C at timestamp 1
>
> If you need fully race condition free counts I'm afraid you need to use
> SERIAL or LOCAL_SERIAL (for in DC only accuracy)
>
> On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <
> sree.srinu38@gmail.com> wrote:
>
> Replication strategy is SimpleReplicationStrategy.
>
> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>
> I was worried that CL=ALL have more read latency and read failures. But
> won't rule out trying it.
>
> Should I switch select count (*) to select partition_key column? Would
> that be of any help.?
>
>
> Thank you
> Regards
> Srini
>
> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <ar...@gmail.com>
> wrote:
>
> What are your replication strategy and snitch settings?
>
> Have you tried doing a read at CL=ALL? If it's an actual inconsistency
> issue (missing data), this should cause the correct results to be returned.
> You'll need to run a repair to fix the inconsistencies.
>
> If all the data is actually there, you might have one or several nodes
> that aren't identifying the correct replicas.
>
> Arvydas
>
>
>
> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
> sree.srinu38@gmail.com> wrote:
>
> Hi Team,
>
> We are struggling with a problem related to cassandra counts, after backup
> and restore of the cluster. Aaron Morton has suggested to send this to user
> list, so some one of the list will be able to help me.
>
> We are have a rest api to talk to cassandra and one of our query which
> fetches count is creating problems for us.
>
> We have done backup and restore and copied all the data to new cluster. We
> have done nodetool refresh on the tables, and did the nodetool repair as
> well.
>
> However, one of our key API call is returning inconsistent results. The
> result count is 0 in the first call and giving the actual values for later
> calls. The query frequency is bit high and failure rate has also raised
> considerably.
>
> 1) The count query has partition keys in it. Didnt see any read timeout or
> any errors from api logs.
>
> 2) This is how our code of creating session looks.
>
> val poolingOptions = new PoolingOptions
> poolingOptions
> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>
> val builtCluster = clusterBuilder.withCredentials(username, password)
> .withPoolingOptions(poolingOptions)
> .build()
> val cassandraSession = builtCluster.get.connect()
>
> val preparedStatement =
> cassandraSession.prepare(statement).setConsistencyLevel(ConsistencyLevel.QUORUM)
> cassandraSession.execute(preparedStatement.bind(args :_*))
>
> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
> text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND
> date_column_of_clustering_key>=?
>
> 3) Cluster configuration:
>
> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
> machine is equipped with 16 Cores and 64 GB Ram.
>
> Replication factor is 3, and write consistency is ONE and read
> consistency is QUORUM.
>
> 4) cassandra is never down on any machine
>
> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>
> 6) nodetool tpstats shows no read failures, and no other failures.
>
> 7) Do not see any other issues from system.log of cassandra. We just see
> few warnings as below.
>
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
> 1.000MiB
> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103 -
> 88 prepared statements discarded in the last minute because cache limit
> reached (32 MB)
> The first api call returns 0 and the api calls later gives right values.
>
> Please let me know, if any other details needed.
> Could you please have a look at this issue once and kindly give me your
> inputs? This issue literally broke the confidence on Cassandra from our
> business team.
>
> Your inputs will be really helpful.
>
> Thank You,
> Regards,
> Srini
>
>
>
>
>
>
> --
>
> Thanks,
> Ryan Svihla
>
>
Re: Issue with Cassandra consistency in results
Posted by srinivasarao daruna <sr...@gmail.com>.
Want to make read and write QUORUM as well.
On Mar 16, 2017 1:09 PM, "Ryan Svihla" <rs...@foundev.pro> wrote:
Replication factor is 3, and write consistency is ONE and read
consistency is QUORUM.
That combination is not gonna work well:
*Write succeeds to NODE A but fails on node B,C*
*Read goes to NODE B, C*
If you can tolerate some temporary inaccuracy you can use QUORUM but may
still have the situation where
Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
Read succeeds on node B and C at timestamp 1
If you need fully race condition free counts I'm afraid you need to use
SERIAL or LOCAL_SERIAL (for in DC only accuracy)
On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <sree.srinu38@gmail.com
> wrote:
> Replication strategy is SimpleReplicationStrategy.
>
> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>
> I was worried that CL=ALL have more read latency and read failures. But
> won't rule out trying it.
>
> Should I switch select count (*) to select partition_key column? Would
> that be of any help.?
>
>
> Thank you
> Regards
> Srini
>
> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <ar...@gmail.com>
> wrote:
>
> What are your replication strategy and snitch settings?
>
> Have you tried doing a read at CL=ALL? If it's an actual inconsistency
> issue (missing data), this should cause the correct results to be returned.
> You'll need to run a repair to fix the inconsistencies.
>
> If all the data is actually there, you might have one or several nodes
> that aren't identifying the correct replicas.
>
> Arvydas
>
>
>
> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
> sree.srinu38@gmail.com> wrote:
>
>> Hi Team,
>>
>> We are struggling with a problem related to cassandra counts, after
>> backup and restore of the cluster. Aaron Morton has suggested to send this
>> to user list, so some one of the list will be able to help me.
>>
>> We are have a rest api to talk to cassandra and one of our query which
>> fetches count is creating problems for us.
>>
>> We have done backup and restore and copied all the data to new cluster.
>> We have done nodetool refresh on the tables, and did the nodetool repair as
>> well.
>>
>> However, one of our key API call is returning inconsistent results. The
>> result count is 0 in the first call and giving the actual values for later
>> calls. The query frequency is bit high and failure rate has also raised
>> considerably.
>>
>> 1) The count query has partition keys in it. Didnt see any read timeout
>> or any errors from api logs.
>>
>> 2) This is how our code of creating session looks.
>>
>> val poolingOptions = new PoolingOptions
>> poolingOptions
>> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
>> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
>> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
>> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>>
>> val builtCluster = clusterBuilder.withCredentials(username, password)
>> .withPoolingOptions(poolingOptions)
>> .build()
>> val cassandraSession = builtCluster.get.connect()
>>
>> val preparedStatement = cassandraSession.prepare(state
>> ment).setConsistencyLevel(ConsistencyLevel.QUORUM)
>> cassandraSession.execute(preparedStatement.bind(args :_*))
>>
>> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
>> text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND
>> date_column_of_clustering_key>=?
>>
>> 3) Cluster configuration:
>>
>> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
>> machine is equipped with 16 Cores and 64 GB Ram.
>>
>> Replication factor is 3, and write consistency is ONE and read
>> consistency is QUORUM.
>>
>> 4) cassandra is never down on any machine
>>
>> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>>
>> 6) nodetool tpstats shows no read failures, and no other failures.
>>
>> 7) Do not see any other issues from system.log of cassandra. We just see
>> few warnings as below.
>>
>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>> 1.000MiB
>> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103
>> - 88 prepared statements discarded in the last minute because cache limit
>> reached (32 MB)
>> The first api call returns 0 and the api calls later gives right values.
>>
>> Please let me know, if any other details needed.
>> Could you please have a look at this issue once and kindly give me your
>> inputs? This issue literally broke the confidence on Cassandra from our
>> business team.
>>
>> Your inputs will be really helpful.
>>
>> Thank You,
>> Regards,
>> Srini
>>
>
>
>
--
Thanks,
Ryan Svihla
Re: Issue with Cassandra consistency in results
Posted by Ryan Svihla <rs...@foundev.pro>.
Replication factor is 3, and write consistency is ONE and read
consistency is QUORUM.
That combination is not gonna work well:
*Write succeeds to NODE A but fails on node B,C*
*Read goes to NODE B, C*
If you can tolerate some temporary inaccuracy you can use QUORUM but may
still have the situation where
Write succeeds on node A a timestamp 1, B succeeds at timestamp 2
Read succeeds on node B and C at timestamp 1
If you need fully race condition free counts I'm afraid you need to use
SERIAL or LOCAL_SERIAL (for in DC only accuracy)
On Thu, Mar 16, 2017 at 1:04 PM, srinivasarao daruna <sree.srinu38@gmail.com
> wrote:
> Replication strategy is SimpleReplicationStrategy.
>
> Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
>
> I was worried that CL=ALL have more read latency and read failures. But
> won't rule out trying it.
>
> Should I switch select count (*) to select partition_key column? Would
> that be of any help.?
>
>
> Thank you
> Regards
> Srini
>
> On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <ar...@gmail.com>
> wrote:
>
> What are your replication strategy and snitch settings?
>
> Have you tried doing a read at CL=ALL? If it's an actual inconsistency
> issue (missing data), this should cause the correct results to be returned.
> You'll need to run a repair to fix the inconsistencies.
>
> If all the data is actually there, you might have one or several nodes
> that aren't identifying the correct replicas.
>
> Arvydas
>
>
>
> On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <
> sree.srinu38@gmail.com> wrote:
>
>> Hi Team,
>>
>> We are struggling with a problem related to cassandra counts, after
>> backup and restore of the cluster. Aaron Morton has suggested to send this
>> to user list, so some one of the list will be able to help me.
>>
>> We are have a rest api to talk to cassandra and one of our query which
>> fetches count is creating problems for us.
>>
>> We have done backup and restore and copied all the data to new cluster.
>> We have done nodetool refresh on the tables, and did the nodetool repair as
>> well.
>>
>> However, one of our key API call is returning inconsistent results. The
>> result count is 0 in the first call and giving the actual values for later
>> calls. The query frequency is bit high and failure rate has also raised
>> considerably.
>>
>> 1) The count query has partition keys in it. Didnt see any read timeout
>> or any errors from api logs.
>>
>> 2) This is how our code of creating session looks.
>>
>> val poolingOptions = new PoolingOptions
>> poolingOptions
>> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
>> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
>> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
>> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>>
>> val builtCluster = clusterBuilder.withCredentials(username, password)
>> .withPoolingOptions(poolingOptions)
>> .build()
>> val cassandraSession = builtCluster.get.connect()
>>
>> val preparedStatement = cassandraSession.prepare(state
>> ment).setConsistencyLevel(ConsistencyLevel.QUORUM)
>> cassandraSession.execute(preparedStatement.bind(args :_*))
>>
>> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
>> text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND
>> date_column_of_clustering_key>=?
>>
>> 3) Cluster configuration:
>>
>> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
>> machine is equipped with 16 Cores and 64 GB Ram.
>>
>> Replication factor is 3, and write consistency is ONE and read
>> consistency is QUORUM.
>>
>> 4) cassandra is never down on any machine
>>
>> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>>
>> 6) nodetool tpstats shows no read failures, and no other failures.
>>
>> 7) Do not see any other issues from system.log of cassandra. We just see
>> few warnings as below.
>>
>> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
>> 1.000MiB
>> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103
>> - 88 prepared statements discarded in the last minute because cache limit
>> reached (32 MB)
>> The first api call returns 0 and the api calls later gives right values.
>>
>> Please let me know, if any other details needed.
>> Could you please have a look at this issue once and kindly give me your
>> inputs? This issue literally broke the confidence on Cassandra from our
>> business team.
>>
>> Your inputs will be really helpful.
>>
>> Thank You,
>> Regards,
>> Srini
>>
>
>
>
--
Thanks,
Ryan Svihla
Re: Issue with Cassandra consistency in results
Posted by srinivasarao daruna <sr...@gmail.com>.
Replication strategy is SimpleReplicationStrategy.
Smith is : EC2 snitch. As we deployed cluster on EC2 instances.
I was worried that CL=ALL have more read latency and read failures. But
won't rule out trying it.
Should I switch select count (*) to select partition_key column? Would that
be of any help.?
Thank you
Regards
Srini
On Mar 16, 2017 12:46 PM, "Arvydas Jonusonis" <ar...@gmail.com>
wrote:
What are your replication strategy and snitch settings?
Have you tried doing a read at CL=ALL? If it's an actual inconsistency
issue (missing data), this should cause the correct results to be returned.
You'll need to run a repair to fix the inconsistencies.
If all the data is actually there, you might have one or several nodes that
aren't identifying the correct replicas.
Arvydas
On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <sree.srinu38@gmail.com
> wrote:
> Hi Team,
>
> We are struggling with a problem related to cassandra counts, after backup
> and restore of the cluster. Aaron Morton has suggested to send this to user
> list, so some one of the list will be able to help me.
>
> We are have a rest api to talk to cassandra and one of our query which
> fetches count is creating problems for us.
>
> We have done backup and restore and copied all the data to new cluster. We
> have done nodetool refresh on the tables, and did the nodetool repair as
> well.
>
> However, one of our key API call is returning inconsistent results. The
> result count is 0 in the first call and giving the actual values for later
> calls. The query frequency is bit high and failure rate has also raised
> considerably.
>
> 1) The count query has partition keys in it. Didnt see any read timeout or
> any errors from api logs.
>
> 2) This is how our code of creating session looks.
>
> val poolingOptions = new PoolingOptions
> poolingOptions
> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>
> val builtCluster = clusterBuilder.withCredentials(username, password)
> .withPoolingOptions(poolingOptions)
> .build()
> val cassandraSession = builtCluster.get.connect()
>
> val preparedStatement = cassandraSession.prepare(state
> ment).setConsistencyLevel(ConsistencyLevel.QUORUM)
> cassandraSession.execute(preparedStatement.bind(args :_*))
>
> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
> text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND
> date_column_of_clustering_key>=?
>
> 3) Cluster configuration:
>
> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
> machine is equipped with 16 Cores and 64 GB Ram.
>
> Replication factor is 3, and write consistency is ONE and read
> consistency is QUORUM.
>
> 4) cassandra is never down on any machine
>
> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>
> 6) nodetool tpstats shows no read failures, and no other failures.
>
> 7) Do not see any other issues from system.log of cassandra. We just see
> few warnings as below.
>
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
> 1.000MiB
> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103 -
> 88 prepared statements discarded in the last minute because cache limit
> reached (32 MB)
> The first api call returns 0 and the api calls later gives right values.
>
> Please let me know, if any other details needed.
> Could you please have a look at this issue once and kindly give me your
> inputs? This issue literally broke the confidence on Cassandra from our
> business team.
>
> Your inputs will be really helpful.
>
> Thank You,
> Regards,
> Srini
>
Re: Issue with Cassandra consistency in results
Posted by Arvydas Jonusonis <ar...@gmail.com>.
What are your replication strategy and snitch settings?
Have you tried doing a read at CL=ALL? If it's an actual inconsistency
issue (missing data), this should cause the correct results to be returned.
You'll need to run a repair to fix the inconsistencies.
If all the data is actually there, you might have one or several nodes that
aren't identifying the correct replicas.
Arvydas
On Thu, Mar 16, 2017 at 5:31 PM, srinivasarao daruna <sree.srinu38@gmail.com
> wrote:
> Hi Team,
>
> We are struggling with a problem related to cassandra counts, after backup
> and restore of the cluster. Aaron Morton has suggested to send this to user
> list, so some one of the list will be able to help me.
>
> We are have a rest api to talk to cassandra and one of our query which
> fetches count is creating problems for us.
>
> We have done backup and restore and copied all the data to new cluster. We
> have done nodetool refresh on the tables, and did the nodetool repair as
> well.
>
> However, one of our key API call is returning inconsistent results. The
> result count is 0 in the first call and giving the actual values for later
> calls. The query frequency is bit high and failure rate has also raised
> considerably.
>
> 1) The count query has partition keys in it. Didnt see any read timeout or
> any errors from api logs.
>
> 2) This is how our code of creating session looks.
>
> val poolingOptions = new PoolingOptions
> poolingOptions
> .setCoreConnectionsPerHost(HostDistance.LOCAL, 4)
> .setMaxConnectionsPerHost(HostDistance.LOCAL, 10)
> .setCoreConnectionsPerHost(HostDistance.REMOTE, 4)
> .setMaxConnectionsPerHost( HostDistance.REMOTE, 10)
>
> val builtCluster = clusterBuilder.withCredentials(username, password)
> .withPoolingOptions(poolingOptions)
> .build()
> val cassandraSession = builtCluster.get.connect()
>
> val preparedStatement = cassandraSession.prepare(statement).
> setConsistencyLevel(ConsistencyLevel.QUORUM)
> cassandraSession.execute(preparedStatement.bind(args :_*))
>
> Query: SELECT count(*) FROM table_name WHERE parition_column=? AND
> text_column_of_clustering_key=? AND date_column_of_clustering_key<=? AND
> date_column_of_clustering_key>=?
>
> 3) Cluster configuration:
>
> 6 Machines: 3 seeds, we are using apache cassandra 3.9 version. Each
> machine is equipped with 16 Cores and 64 GB Ram.
>
> Replication factor is 3, and write consistency is ONE and read
> consistency is QUORUM.
>
> 4) cassandra is never down on any machine
>
> 5) Using cassandra-driver-core artifact with 3.1.1 version in the api.
>
> 6) nodetool tpstats shows no read failures, and no other failures.
>
> 7) Do not see any other issues from system.log of cassandra. We just see
> few warnings as below.
>
> Maximum memory usage reached (512.000MiB), cannot allocate chunk of
> 1.000MiB
> WARN [ScheduledTasks:1] 2017-03-14 14:58:37,141 QueryProcessor.java:103 -
> 88 prepared statements discarded in the last minute because cache limit
> reached (32 MB)
> The first api call returns 0 and the api calls later gives right values.
>
> Please let me know, if any other details needed.
> Could you please have a look at this issue once and kindly give me your
> inputs? This issue literally broke the confidence on Cassandra from our
> business team.
>
> Your inputs will be really helpful.
>
> Thank You,
> Regards,
> Srini
>