You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Aditya Shetty <ad...@snapdeal.com> on 2015/06/25 12:12:25 UTC
Inconsistent behavior during read
Hi
I have a 3 node cassandra cluster with a replication factor of 2. I have a
basic column family which I am reading by primary key. Here is the CF
structure:
CREATE TABLE reviews_platform.object_stats (
object_owner_id int,
object_type int,
object_id text,
num_of_reviews int, PRIMARY KEY ((object_owner_id, object_type),
object_id)
);
However, when I query this column family by primary key, I sometimes get no
records, even though the record exists. The same query after some time
gives a proper response.
*Query: select * from reviews_platform.object_stats whereobject_id =
'674813358269'and object_type = 0and object_owner_id = 0;*
Note that I am querying using the java driver.
--
Aditya Shetty
*Lead Engineer*
*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,
SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
Download Our App [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
Re: Inconsistent behavior during read
Posted by Aditya Shetty <ad...@snapdeal.com>.
Hi
If this problem was because of data inconsistencies, it should have been
very rare. However, I am seeing this happen very often (almost 50 % of the
times). Statistically, this should be very unlikely if the number of
replication failures are small.
On Thu, Jun 25, 2015 at 11:55 PM, Tyler Hobbs <ty...@datastax.com> wrote:
>
> On Thu, Jun 25, 2015 at 1:00 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> [1] or read repair set to 100% combined with a full scan of all data...
>> which no one does...
>
>
> And this is only true if "full scan" means reading every partition
> individually. Reads of partition ranges (or a range slice, in old Thrift
> terms) don't do read repair.
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>
>
--
Aditya Shetty
*Lead Engineer*
*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,
SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
Download Our App [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
Re: Inconsistent behavior during read
Posted by Tyler Hobbs <ty...@datastax.com>.
On Thu, Jun 25, 2015 at 1:00 PM, Robert Coli <rc...@eventbrite.com> wrote:
> [1] or read repair set to 100% combined with a full scan of all data...
> which no one does...
And this is only true if "full scan" means reading every partition
individually. Reads of partition ranges (or a range slice, in old Thrift
terms) don't do read repair.
--
Tyler Hobbs
DataStax <http://datastax.com/>
Re: Inconsistent behavior during read
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Jun 25, 2015 at 5:14 AM, Jack Krupansky <ja...@gmail.com>
wrote:
> Hinted handoff - which is what provides eventual consistency - can time
> out and be discarded/lost if the cluster is under heavy load or encounters
> poor network connectivity or nodes are down for too long, which is what
> requires running repair. That's why quorum is the recommended cl for both
> write and read for true consistency.
>
As a minor quibble, hinted handoff "provides" nothing, for the reasons you
say : delivery is not guaranteed.
The only guarantor of consistency is repair. [1]
=Rob
[1] or read repair set to 100% combined with a full scan of all data...
which no one does...
Re: [MASSMAIL]Re: Inconsistent behavior during read
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
Quorum will give you strong consistency, but if you're using RF=2 you're
going to have issues, as Quorum on RF=2 = CL=ALL. You'll want to use RF=3
to make sure you can tolerate failure of a node, otherwise a single node
going down will result in unanswerable queries.
On Thu, Jun 25, 2015 at 6:37 AM Marcos Ortiz <ml...@uci.cu> wrote:
> Regards, Aditya. I´m agree with Jack here. In our tests here with read
> and writes in Cassandra (version 2.1.5), we played with several CL, and
> QUORUM is the best for us.
>
>
> On 25/06/15 08:14, Jack Krupansky wrote:
>
> Hinted handoff - which is what provides eventual consistency - can time
> out and be discarded/lost if the cluster is under heavy load or encounters
> poor network connectivity or nodes are down for too long, which is what
> requires running repair. That's why quorum is the recommended cl for both
> write and read for true consistency.
>
> -- Jack Krupansky
>
> On Thu, Jun 25, 2015 at 7:51 AM, Alain RODRIGUEZ <ar...@gmail.com>
> wrote:
>
>> Excepted if the node failed to take the write and you have no Hinted
>> Handoff (or for some reason they also failed).
>>
>> Have you tried at QUORUM or even ALL, this would force a synchronous
>> read repair. You can also try to repair directly.
>>
>> Hope this will help,
>>
>> C*heers,
>>
>> Alain
>>
>> 2015-06-25 13:34 GMT+02:00 Aditya Shetty <ad...@snapdeal.com>:
>>
>>> I am using consistency one for both. However, the writes had happened a
>>> few days before, so it does not look like an issue of eventual consistency.
>>>
>>> On Thu, Jun 25, 2015 at 3:59 PM, Perica Milošević <
>>> perica.milosevic@gmail.com> wrote:
>>>
>>>> Which ConsistencyLevel do you use for writing and reading of the data?
>>>>
>>>> Cheers,
>>>> Perica
>>>>
>>>>
>>>> On Thu, Jun 25, 2015 at 12:12 PM, Aditya Shetty <
>>>> aditya.shetty@snapdeal.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I have a 3 node cassandra cluster with a replication factor of 2. I
>>>>> have a basic column family which I am reading by primary key. Here is the
>>>>> CF structure:
>>>>>
>>>>> CREATE TABLE reviews_platform.object_stats (
>>>>>
>>>>> object_owner_id int,
>>>>>
>>>>> object_type int,
>>>>>
>>>>> object_id text,
>>>>> num_of_reviews int, PRIMARY KEY ((object_owner_id,
>>>>> object_type), object_id)
>>>>>
>>>>> );
>>>>> However, when I query this column family by primary key, I sometimes
>>>>> get no records, even though the record exists. The same query after some
>>>>> time gives a proper response.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Query: select * from reviews_platform.object_stats where object_id =
>>>>> '674813358269' and object_type = 0 and object_owner_id = 0; *
>>>>> Note that I am querying using the java driver.
>>>>>
>>>>> --
>>>>>
>>>>> Aditya Shetty
>>>>> *Lead Engineer*
>>>>>
>>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>>> 2nd FLOOR, WEST WING,
>>>>>
>>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>>
>>>>> Download Our App [image: A]
>>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>>> [image: A]
>>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>>> [image: W]
>>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Aditya Shetty
>>> *Lead Engineer*
>>>
>>> *M*: +91 7022423545 <%2B91%207022423545>, *T*: 080 46603000 *EXT*: 4417
>>> 2nd FLOOR, WEST WING,
>>>
>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>
>>> Download Our App [image: A]
>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>> [image: A]
>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>> [image: W]
>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>
>>
>>
>
> --
> Marcos Ortiz <http://about.me/marcosortiz>, Sr. Product Manager (Data
> Infrastructure) at UCI
> @marcosluis2186 <http://twitter.com/marcosluis2186>
>
>
>
Re: [MASSMAIL]Re: Inconsistent behavior during read
Posted by Marcos Ortiz <ml...@uci.cu>.
Regards, Aditya. I´m agree with Jack here. In our tests here with read
and writes in Cassandra (version 2.1.5), we played with several CL, and
QUORUM is the best for us.
On 25/06/15 08:14, Jack Krupansky wrote:
> Hinted handoff - which is what provides eventual consistency - can
> time out and be discarded/lost if the cluster is under heavy load or
> encounters poor network connectivity or nodes are down for too long,
> which is what requires running repair. That's why quorum is the
> recommended cl for both write and read for true consistency.
>
> -- Jack Krupansky
>
> On Thu, Jun 25, 2015 at 7:51 AM, Alain RODRIGUEZ <arodrime@gmail.com
> <ma...@gmail.com>> wrote:
>
> Excepted if the node failed to take the write and you have no
> Hinted Handoff (or for some reason they also failed).
>
> Have you tried at QUORUM or even ALL, this would force a
> synchronous read repair. You can also try to repair directly.
>
> Hope this will help,
>
> C*heers,
>
> Alain
>
> 2015-06-25 13:34 GMT+02:00 Aditya Shetty
> <aditya.shetty@snapdeal.com <ma...@snapdeal.com>>:
>
> I am using consistency one for both. However, the writes had
> happened a few days before, so it does not look like an issue
> of eventual consistency.
>
> On Thu, Jun 25, 2015 at 3:59 PM, Perica Milošević
> <perica.milosevic@gmail.com
> <ma...@gmail.com>> wrote:
>
> Which ConsistencyLevel do you use for writing and reading
> of the data?
>
> Cheers,
> Perica
>
>
> On Thu, Jun 25, 2015 at 12:12 PM, Aditya Shetty
> <aditya.shetty@snapdeal.com
> <ma...@snapdeal.com>> wrote:
>
> Hi
>
> I have a 3 node cassandra cluster with a replication
> factor of 2. I have a basic column family which I am
> reading by primary key. Here is the CF structure:
>
> CREATE TABLE reviews_platform.object_stats (
>
> object_owner_id int,
>
> object_type int,
>
> object_id text,
>
> num_of_reviews int, PRIMARY KEY
> ((object_owner_id, object_type), object_id)
>
> );
>
> However, when I query this column family by primary
> key, I sometimes get no records, even though the
> record exists. The same query after some time gives a
> proper response.
>
> /Query:
> select * from reviews_platform.object_stats where
> object_id = '674813358269'
> and object_type = 0
> and object_owner_id = 0;
> /
> Note that I am querying using the java driver.
>
> --
>
>
> Aditya Shetty
> *Lead Engineer*
>
>
>
> *M*: +91 7022423545 <tel:%2B91%207022423545>, *T*: 080
> 46603000 *EXT*: 4417
> 2^nd FLOOR, WEST WING,
>
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
>
>
>
> Download Our App
> A
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>
> A
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>
> W
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
>
>
>
>
>
> --
>
>
> Aditya Shetty
> *Lead Engineer*
>
>
>
> *M*: +91 7022423545 <tel:%2B91%207022423545>, *T*: 080
> 46603000 *EXT*: 4417
> 2^nd FLOOR, WEST WING,
>
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
>
>
>
> Download Our App
> A
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>
> A
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>
> W
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
>
>
>
--
Marcos Ortiz <http://about.me/marcosortiz>, Sr. Product Manager (Data
Infrastructure) at UCI
@marcosluis2186 <http://twitter.com/marcosluis2186>
Re: Restore Snapshots
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi Jean,
Glad to hear it worked this way.
Some other people provided (and continue providing) similar help to me,
just trying to give back to the community as much as I received from it.
See you around.
Alain
2015-06-26 8:44 GMT+02:00 Jean Tremblay <je...@zen-innovations.com>:
> Good morning,
> Alain, thank you so much. This is exactly what I needed.
>
> In my test I had a node which had for whatever reason the directory
> containing my data corrupted. I keep in a separate folder my snapshots.
>
> Here are the steps I took to recover my sick node:
>
> 0) Cassandra is stopped on my sick node.
> 1) I wiped out my data directory. My snapshots were kept outside this
> directory.
> 2) I modified my Cassandra.yaml. I added auto_bootstrap: false .This is to
> make sure that my node does not synch with the others.
> 3) I restarted Cassandra. This step created a basic structure for my new
> data directory.
> 4) I did the command: nodetool resetlocalschema. This recreated all the
> folders for my cf.
> 5) I stopped Cassandra on my node.
> 6) I copied my snapshot in the right location. I actually hard linked
> them, this is very fast.
> 7) I restarted Cassandra.
>
> That's it.
>
> Thank you SO MUCH ALAIN for your support. You really helped me a lot.
>
> On 25 Jun,2015, at 18:37, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>
> Hi Jean,
>
> Answers in line to be sure to be exhaustive:
>
> - how can I restore the data directory structure in order to copy my
> snapshots at the right position?
> --> making a script to do it and testing it I would say. basically under
> any table repo you have a "snapshots/snapshot_name" directory (snapshot_name
> is timestamp if not specified off the top of my head..) and then your
> sstables.
>
> - is it possible to recreate the schema on one node?
> --> The easiest way that come to my mind is to set "auto_bootstrap: false"
> on a node not already in the ring. If you have trouble with the schema of a
> node in the ring run a "nodetool resetlocalschema"
>
> - how can I avoid the node from streaming from the other nodes?
> --> See above (auto_bootstrap: false). BTW, option might not be present
> at all, just add it.
>
> - must I also have the snapshot of the system tables in order to restore
> a node from only the snapshot of my tables?
> --> just you user table. Yet remember that snapshot is per node and as
> such you will just have part of the data this node use to hold. meaning
> that if the new node have different tokens, there will be unused data +
> missing data for sure.
>
> Basically when a node is down I use to remove it, repair the cluster,
> and bootstap it (auto_bootstrap: true). Streams are part of Cassandra. I
> accept that. An other solution would be to "replace" the node -->
> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html
>
>
> C*heers,
>
> Alain
>
> 2015-06-25 17:07 GMT+02:00 Jean Tremblay <
> jean.tremblay@zen-innovations.com>:
>
>> Hi,
>>
>> I am testing snapshot restore procedures in case of a major catastrophe
>> on our cluster. I’m using Cassandra 2.1.7 with RF:3
>>
>> The scenario that I am trying to solve is how to quickly get one node
>> back to work after its disk failed and lost all its data assuming that the
>> only thing I have is its snapshots.
>>
>> The procedure that I’m following is the one explained here:
>> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html
>>
>> I can do a snapshot that is straight forward.
>> My problem is in the restore of the snapshot.
>>
>> If I restart Cassandra with an empty data directory the node will
>> bootstrap.
>> Bootstrap is very nice, since it recreate the schema and reload the data
>> from its neighbour.
>> But this is quite heavy traffic and quite a slow process.
>>
>> My questions are:
>>
>> - how can I restore the data directory structure in order to copy my
>> snapshots at the right position?
>> - is it possible to recreate the schema on one node?
>> - how can I avoid the node from streaming from the other nodes?
>> - must I also have the snapshot of the system tables in order to restore
>> a node from only the snapshot of my tables?
>>
>> Thanks for your comments.
>>
>> Jean
>>
>>
>>
>>
>>
>
Re: Restore Snapshots
Posted by Jean Tremblay <je...@zen-innovations.com>.
Good morning,
Alain, thank you so much. This is exactly what I needed.
In my test I had a node which had for whatever reason the directory containing my data corrupted. I keep in a separate folder my snapshots.
Here are the steps I took to recover my sick node:
0) Cassandra is stopped on my sick node.
1) I wiped out my data directory. My snapshots were kept outside this directory.
2) I modified my Cassandra.yaml. I added auto_bootstrap: false .This is to make sure that my node does not synch with the others.
3) I restarted Cassandra. This step created a basic structure for my new data directory.
4) I did the command: nodetool resetlocalschema. This recreated all the folders for my cf.
5) I stopped Cassandra on my node.
6) I copied my snapshot in the right location. I actually hard linked them, this is very fast.
7) I restarted Cassandra.
That's it.
Thank you SO MUCH ALAIN for your support. You really helped me a lot.
On 25 Jun,2015, at 18:37, Alain RODRIGUEZ <ar...@gmail.com>> wrote:
Hi Jean,
Answers in line to be sure to be exhaustive:
- how can I restore the data directory structure in order to copy my snapshots at the right position?
--> making a script to do it and testing it I would say. basically under any table repo you have a "snapshots/snapshot_name" directory (snapshot_name is timestamp if not specified off the top of my head..) and then your sstables.
- is it possible to recreate the schema on one node?
--> The easiest way that come to my mind is to set "auto_bootstrap: false" on a node not already in the ring. If you have trouble with the schema of a node in the ring run a "nodetool resetlocalschema"
- how can I avoid the node from streaming from the other nodes?
--> See above (auto_bootstrap: false). BTW, option might not be present at all, just add it.
- must I also have the snapshot of the system tables in order to restore a node from only the snapshot of my tables?
--> just you user table. Yet remember that snapshot is per node and as such you will just have part of the data this node use to hold. meaning that if the new node have different tokens, there will be unused data + missing data for sure.
Basically when a node is down I use to remove it, repair the cluster, and bootstap it (auto_bootstrap: true). Streams are part of Cassandra. I accept that. An other solution would be to "replace" the node --> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html
C*heers,
Alain
2015-06-25 17:07 GMT+02:00 Jean Tremblay <je...@zen-innovations.com>>:
Hi,
I am testing snapshot restore procedures in case of a major catastrophe on our cluster. I'm using Cassandra 2.1.7 with RF:3
The scenario that I am trying to solve is how to quickly get one node back to work after its disk failed and lost all its data assuming that the only thing I have is its snapshots.
The procedure that I'm following is the one explained here: http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html
I can do a snapshot that is straight forward.
My problem is in the restore of the snapshot.
If I restart Cassandra with an empty data directory the node will bootstrap.
Bootstrap is very nice, since it recreate the schema and reload the data from its neighbour.
But this is quite heavy traffic and quite a slow process.
My questions are:
- how can I restore the data directory structure in order to copy my snapshots at the right position?
- is it possible to recreate the schema on one node?
- how can I avoid the node from streaming from the other nodes?
- must I also have the snapshot of the system tables in order to restore a node from only the snapshot of my tables?
Thanks for your comments.
Jean
Re: Restore Snapshots
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi Jean,
Answers in line to be sure to be exhaustive:
- how can I restore the data directory structure in order to copy my
snapshots at the right position?
--> making a script to do it and testing it I would say. basically under
any table repo you have a "snapshots/snapshot_name" directory (snapshot_name
is timestamp if not specified off the top of my head..) and then your
sstables.
- is it possible to recreate the schema on one node?
--> The easiest way that come to my mind is to set "auto_bootstrap: false"
on a node not already in the ring. If you have trouble with the schema of a
node in the ring run a "nodetool resetlocalschema"
- how can I avoid the node from streaming from the other nodes?
--> See above (auto_bootstrap: false). BTW, option might not be present at
all, just add it.
- must I also have the snapshot of the system tables in order to restore a
node from only the snapshot of my tables?
--> just you user table. Yet remember that snapshot is per node and as such
you will just have part of the data this node use to hold. meaning that if
the new node have different tokens, there will be unused data + missing
data for sure.
Basically when a node is down I use to remove it, repair the cluster, and
bootstap it (auto_bootstrap: true). Streams are part of Cassandra. I accept
that. An other solution would be to "replace" the node -->
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html
C*heers,
Alain
2015-06-25 17:07 GMT+02:00 Jean Tremblay <je...@zen-innovations.com>
:
> Hi,
>
> I am testing snapshot restore procedures in case of a major catastrophe
> on our cluster. I’m using Cassandra 2.1.7 with RF:3
>
> The scenario that I am trying to solve is how to quickly get one node
> back to work after its disk failed and lost all its data assuming that the
> only thing I have is its snapshots.
>
> The procedure that I’m following is the one explained here:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html
>
> I can do a snapshot that is straight forward.
> My problem is in the restore of the snapshot.
>
> If I restart Cassandra with an empty data directory the node will
> bootstrap.
> Bootstrap is very nice, since it recreate the schema and reload the data
> from its neighbour.
> But this is quite heavy traffic and quite a slow process.
>
> My questions are:
>
> - how can I restore the data directory structure in order to copy my
> snapshots at the right position?
> - is it possible to recreate the schema on one node?
> - how can I avoid the node from streaming from the other nodes?
> - must I also have the snapshot of the system tables in order to restore a
> node from only the snapshot of my tables?
>
> Thanks for your comments.
>
> Jean
>
>
>
>
>
Restore Snapshots
Posted by Jean Tremblay <je...@zen-innovations.com>.
Hi,
I am testing snapshot restore procedures in case of a major catastrophe on our cluster. I’m using Cassandra 2.1.7 with RF:3
The scenario that I am trying to solve is how to quickly get one node back to work after its disk failed and lost all its data assuming that the only thing I have is its snapshots.
The procedure that I’m following is the one explained here: http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html
I can do a snapshot that is straight forward.
My problem is in the restore of the snapshot.
If I restart Cassandra with an empty data directory the node will bootstrap.
Bootstrap is very nice, since it recreate the schema and reload the data from its neighbour.
But this is quite heavy traffic and quite a slow process.
My questions are:
- how can I restore the data directory structure in order to copy my snapshots at the right position?
- is it possible to recreate the schema on one node?
- how can I avoid the node from streaming from the other nodes?
- must I also have the snapshot of the system tables in order to restore a node from only the snapshot of my tables?
Thanks for your comments.
Jean
Re: Inconsistent behavior during read
Posted by Jack Krupansky <ja...@gmail.com>.
Hinted handoff - which is what provides eventual consistency - can time out
and be discarded/lost if the cluster is under heavy load or encounters poor
network connectivity or nodes are down for too long, which is what requires
running repair. That's why quorum is the recommended cl for both write and
read for true consistency.
-- Jack Krupansky
On Thu, Jun 25, 2015 at 7:51 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> Excepted if the node failed to take the write and you have no Hinted
> Handoff (or for some reason they also failed).
>
> Have you tried at QUORUM or even ALL, this would force a synchronous read
> repair. You can also try to repair directly.
>
> Hope this will help,
>
> C*heers,
>
> Alain
>
> 2015-06-25 13:34 GMT+02:00 Aditya Shetty <ad...@snapdeal.com>:
>
>> I am using consistency one for both. However, the writes had happened a
>> few days before, so it does not look like an issue of eventual consistency.
>>
>> On Thu, Jun 25, 2015 at 3:59 PM, Perica Milošević <
>> perica.milosevic@gmail.com> wrote:
>>
>>> Which ConsistencyLevel do you use for writing and reading of the data?
>>>
>>> Cheers,
>>> Perica
>>>
>>>
>>> On Thu, Jun 25, 2015 at 12:12 PM, Aditya Shetty <
>>> aditya.shetty@snapdeal.com> wrote:
>>>
>>>> Hi
>>>>
>>>> I have a 3 node cassandra cluster with a replication factor of 2. I
>>>> have a basic column family which I am reading by primary key. Here is the
>>>> CF structure:
>>>>
>>>> CREATE TABLE reviews_platform.object_stats (
>>>>
>>>> object_owner_id int,
>>>>
>>>> object_type int,
>>>>
>>>> object_id text,
>>>> num_of_reviews int, PRIMARY KEY ((object_owner_id, object_type),
>>>> object_id)
>>>>
>>>> );
>>>> However, when I query this column family by primary key, I sometimes
>>>> get no records, even though the record exists. The same query after some
>>>> time gives a proper response.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Query: select * from reviews_platform.object_stats whereobject_id =
>>>> '674813358269'and object_type = 0and object_owner_id = 0;*
>>>> Note that I am querying using the java driver.
>>>>
>>>> --
>>>>
>>>> Aditya Shetty
>>>> *Lead Engineer*
>>>>
>>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>>> 2nd FLOOR, WEST WING,
>>>>
>>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>>
>>>> Download Our App [image: A]
>>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>>> [image: A]
>>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>>> [image: W]
>>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>>
>>>
>>>
>>
>>
>> --
>>
>> Aditya Shetty
>> *Lead Engineer*
>>
>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>> 2nd FLOOR, WEST WING,
>>
>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>
>> Download Our App [image: A]
>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>> [image: A]
>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>> [image: W]
>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>
>
>
Re: Inconsistent behavior during read
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Excepted if the node failed to take the write and you have no Hinted
Handoff (or for some reason they also failed).
Have you tried at QUORUM or even ALL, this would force a synchronous read
repair. You can also try to repair directly.
Hope this will help,
C*heers,
Alain
2015-06-25 13:34 GMT+02:00 Aditya Shetty <ad...@snapdeal.com>:
> I am using consistency one for both. However, the writes had happened a
> few days before, so it does not look like an issue of eventual consistency.
>
> On Thu, Jun 25, 2015 at 3:59 PM, Perica Milošević <
> perica.milosevic@gmail.com> wrote:
>
>> Which ConsistencyLevel do you use for writing and reading of the data?
>>
>> Cheers,
>> Perica
>>
>>
>> On Thu, Jun 25, 2015 at 12:12 PM, Aditya Shetty <
>> aditya.shetty@snapdeal.com> wrote:
>>
>>> Hi
>>>
>>> I have a 3 node cassandra cluster with a replication factor of 2. I have
>>> a basic column family which I am reading by primary key. Here is the CF
>>> structure:
>>>
>>> CREATE TABLE reviews_platform.object_stats (
>>>
>>> object_owner_id int,
>>>
>>> object_type int,
>>>
>>> object_id text,
>>> num_of_reviews int, PRIMARY KEY ((object_owner_id, object_type),
>>> object_id)
>>>
>>> );
>>> However, when I query this column family by primary key, I sometimes get
>>> no records, even though the record exists. The same query after some time
>>> gives a proper response.
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Query: select * from reviews_platform.object_stats whereobject_id =
>>> '674813358269'and object_type = 0and object_owner_id = 0;*
>>> Note that I am querying using the java driver.
>>>
>>> --
>>>
>>> Aditya Shetty
>>> *Lead Engineer*
>>>
>>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>>> 2nd FLOOR, WEST WING,
>>>
>>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>>
>>> Download Our App [image: A]
>>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>>> [image: A]
>>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>>> [image: W]
>>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>>
>>
>>
>
>
> --
>
> Aditya Shetty
> *Lead Engineer*
>
> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
> 2nd FLOOR, WEST WING,
>
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
> Download Our App [image: A]
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> [image: A]
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> [image: W]
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>
Re: Inconsistent behavior during read
Posted by Aditya Shetty <ad...@snapdeal.com>.
I am using consistency one for both. However, the writes had happened a few
days before, so it does not look like an issue of eventual consistency.
On Thu, Jun 25, 2015 at 3:59 PM, Perica Milošević <
perica.milosevic@gmail.com> wrote:
> Which ConsistencyLevel do you use for writing and reading of the data?
>
> Cheers,
> Perica
>
>
> On Thu, Jun 25, 2015 at 12:12 PM, Aditya Shetty <
> aditya.shetty@snapdeal.com> wrote:
>
>> Hi
>>
>> I have a 3 node cassandra cluster with a replication factor of 2. I have
>> a basic column family which I am reading by primary key. Here is the CF
>> structure:
>>
>> CREATE TABLE reviews_platform.object_stats (
>>
>> object_owner_id int,
>>
>> object_type int,
>>
>> object_id text,
>> num_of_reviews int, PRIMARY KEY ((object_owner_id, object_type),
>> object_id)
>>
>> );
>> However, when I query this column family by primary key, I sometimes get
>> no records, even though the record exists. The same query after some time
>> gives a proper response.
>>
>>
>>
>>
>>
>>
>> *Query: select * from reviews_platform.object_stats whereobject_id =
>> '674813358269'and object_type = 0and object_owner_id = 0;*
>> Note that I am querying using the java driver.
>>
>> --
>>
>> Aditya Shetty
>> *Lead Engineer*
>>
>> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
>> 2nd FLOOR, WEST WING,
>>
>> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>>
>> Download Our App [image: A]
>> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
>> [image: A]
>> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
>> [image: W]
>> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>>
>
>
--
Aditya Shetty
*Lead Engineer*
*M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
2nd FLOOR, WEST WING,
SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
Download Our App [image: A]
<https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
[image: A]
<https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
[image: W]
<http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
Re: Inconsistent behavior during read
Posted by Perica Milošević <pe...@gmail.com>.
Which ConsistencyLevel do you use for writing and reading of the data?
Cheers,
Perica
On Thu, Jun 25, 2015 at 12:12 PM, Aditya Shetty <ad...@snapdeal.com>
wrote:
> Hi
>
> I have a 3 node cassandra cluster with a replication factor of 2. I have a
> basic column family which I am reading by primary key. Here is the CF
> structure:
>
> CREATE TABLE reviews_platform.object_stats (
>
> object_owner_id int,
>
> object_type int,
>
> object_id text,
> num_of_reviews int, PRIMARY KEY ((object_owner_id, object_type),
> object_id)
>
> );
> However, when I query this column family by primary key, I sometimes get
> no records, even though the record exists. The same query after some time
> gives a proper response.
>
>
>
>
>
>
> *Query: select * from reviews_platform.object_stats whereobject_id =
> '674813358269'and object_type = 0and object_owner_id = 0;*
> Note that I am querying using the java driver.
>
> --
>
> Aditya Shetty
> *Lead Engineer*
>
> *M*: +91 7022423545, *T*: 080 46603000 *EXT*: 4417
> 2nd FLOOR, WEST WING,
>
> SALARPURIA SUPREME, MARATHAHALLI, BENGALURU
>
> Download Our App [image: A]
> <https://play.google.com/store/apps/details?id=com.snapdeal.main&utm_source=mobileAppLp&utm_campaign=android>
> [image: A]
> <https://itunes.apple.com/in/app/snapdeal-mobile-shopping/id721124909?ls=1&mt=8&utm_source=mobileAppLp&utm_campaign=ios>
> [image: W]
> <http://www.windowsphone.com/en-in/store/app/snapdeal/ee17fccf-40d0-4a59-80a3-04da47a5553f>
>