You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Flavien Charlon <fl...@gmail.com> on 2015/01/19 22:50:48 UTC
How do replica become out of sync
Hi,
When writing to Cassandra using CL = Quorum (or anything less than ALL), is
it correct to say that Cassandra tries to write to all the replica, but
only waits for Quorum?
If so, what can cause some replica to become out of sync when they're all
online?
Thanks
Flavien
Re: How do replica become out of sync
Posted by Flavien Charlon <fl...@gmail.com>.
Quite a few, see here: http://pastebin.com/SMnprHdp. In total about 3,000
ranges across the 3 nodes.
This is with vnodes disabled. It was at least an order of magnitude worse
when we had it enabled.
Flavien
On 20 January 2015 at 22:22, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Jan 19, 2015 at 5:44 PM, Flavien Charlon <
> flavien.charlon@gmail.com> wrote:
>
>> Thanks Andi. The reason I was asking is that even though my nodes have
>> been 100% available and no write has been rejected, when running an
>> incremental repair, the logs still indicate that some ranges are out of
>> sync (which then results in large amounts of compaction), how can this be
>> possible?
>>
>
> This is most likely, as you conjecture, due to slight differences between
> nodes at the time of Merkle Tree calculation.
>
> How many rows differ?
>
> =Rob
>
>
Re: How do replica become out of sync
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Jan 19, 2015 at 5:44 PM, Flavien Charlon <fl...@gmail.com>
wrote:
> Thanks Andi. The reason I was asking is that even though my nodes have
> been 100% available and no write has been rejected, when running an
> incremental repair, the logs still indicate that some ranges are out of
> sync (which then results in large amounts of compaction), how can this be
> possible?
>
This is most likely, as you conjecture, due to slight differences between
nodes at the time of Merkle Tree calculation.
How many rows differ?
=Rob
Re: sharding vs what cassandra does
Posted by Nagesh <na...@gmail.com>.
Sharding is a type of database partitioning. The sweet spot of cassandra is
to supporting fast random reads. This is achieved by grouping data based on
a partition key and replicate to different nodes. The querying should be in
such a way to look up data of one partition at a time.
Grouping data based on a partition key is a type of sharding.
On Tue, Jan 20, 2015 at 11:11 AM, Mohammed Guller <mo...@glassbeam.com>
wrote:
> Partitioning is similar to sharding.
>
>
>
> Mohammed
>
>
>
> *From:* Adaryl "Bob" Wakefield, MBA [mailto:adaryl.wakefield@hotmail.com]
> *Sent:* Monday, January 19, 2015 8:28 PM
> *To:* user@cassandra.apache.org
> *Subject:* sharding vs what cassandra does
>
>
>
> It’s my understanding that the way Cassandra replicates data across nodes
> is NOT sharding. Can someone provide a better explanation or correct my
> understanding?
>
> B.
>
--
Thanks,
Nageswara Rao.V
*"The LORD reigns"*
RE: sharding vs what cassandra does
Posted by Mohammed Guller <mo...@glassbeam.com>.
Partitioning is similar to sharding.
Mohammed
From: Adaryl "Bob" Wakefield, MBA [mailto:adaryl.wakefield@hotmail.com]
Sent: Monday, January 19, 2015 8:28 PM
To: user@cassandra.apache.org
Subject: sharding vs what cassandra does
It’s my understanding that the way Cassandra replicates data across nodes is NOT sharding. Can someone provide a better explanation or correct my understanding?
B.
RE: sharding vs what cassandra does
Posted by Job Thomas <jo...@suntecgroup.com>.
Hi,
If we think it the perspective of column family (table), its rows are split into different nodes(Sharding) based on ring concept in Cassandra.
But the core unit of data storage (rows) id not spit across nodes, only copy is maintained in different rows. All column associated to a single row (based on row key) is stored in a single node , but Cassandra
keep its copy (based on our configuration) another nodes for fault tolerance.
Hadoop(HDFS) does data sharding and replication . It split a file into multiple pieces and also copy these pieces into multiple nodes for fault tolerance.
Thanks & Regards
Job M Thomas| Platform & Technology
SunTec Business Solutions Pvt. Ltd
Mob : 7560885748
________________________________
From: Adaryl "Bob" Wakefield, MBA [mailto:adaryl.wakefield@hotmail.com]
Sent: Tue 1/20/2015 9:57 AM
To: user@cassandra.apache.org
Subject: sharding vs what cassandra does
It’s my understanding that the way Cassandra replicates data across nodes is NOT sharding. Can someone provide a better explanation or correct my understanding?
B.
sharding vs what cassandra does
Posted by "Adaryl \"Bob\" Wakefield, MBA" <ad...@hotmail.com>.
It’s my understanding that the way Cassandra replicates data across nodes is NOT sharding. Can someone provide a better explanation or correct my understanding?
B.
Re: How do replica become out of sync
Posted by Flavien Charlon <fl...@gmail.com>.
Thanks Andi. The reason I was asking is that even though my nodes have been
100% available and no write has been rejected, when running an incremental
repair, the logs still indicate that some ranges are out of sync (which
then results in large amounts of compaction), how can this be possible?
I have found this (http://stackoverflow.com/a/20928922/980059) which seems
to indicate this could be because in parallel repairs, the merkle trees are
computed at different times, which results in repair thinking some ranges
are out of sync if the data has changed in the meantime.
Is that accurate? Does sequential repair alleviate this issue (since it
uses snapshots)?
Thanks
Flavien
On 19 January 2015 at 23:57, Andreas Finke <An...@solvians.com>
wrote:
> Hi,
>
>
> right, QUORUM means that data is written to all replicas but the
> coordinator waits for QUORUM responses before returning back to client. If
> a replica is out of sync due to network or internal issue than consistency
> is ensured through:
>
> - HintedHandoff (Automatically
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_about_hh_c.html
> )
> - ReadRepair (Automatically
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dmlClientRequestsRead.html
> )
> - nodetool repair (Manually
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_repair_nodes_c.html
> )
>
> Regards
> Andi
> ------------------------------
> *From:* Flavien Charlon [flavien.charlon@gmail.com]
> *Sent:* 19 January 2015 22:50
> *To:* user@cassandra.apache.org
> *Subject:* How do replica become out of sync
>
> Hi,
>
> When writing to Cassandra using CL = Quorum (or anything less than ALL),
> is it correct to say that Cassandra tries to write to all the replica, but
> only waits for Quorum?
>
> If so, what can cause some replica to become out of sync when they're
> all online?
>
> Thanks
> Flavien
>
RE: How do replica become out of sync
Posted by Andreas Finke <An...@solvians.com>.
Hi,
right, QUORUM means that data is written to all replicas but the coordinator waits for QUORUM responses before returning back to client. If a replica is out of sync due to network or internal issue than consistency is ensured through:
- HintedHandoff (Automatically http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_about_hh_c.html)
- ReadRepair (Automatically http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dmlClientRequestsRead.html)
- nodetool repair (Manually http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_repair_nodes_c.html)
Regards
Andi
________________________________
From: Flavien Charlon [flavien.charlon@gmail.com]
Sent: 19 January 2015 22:50
To: user@cassandra.apache.org
Subject: How do replica become out of sync
Hi,
When writing to Cassandra using CL = Quorum (or anything less than ALL), is it correct to say that Cassandra tries to write to all the replica, but only waits for Quorum?
If so, what can cause some replica to become out of sync when they're all online?
Thanks
Flavien