You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Zhong Li <zl...@voxeo.com> on 2010/08/17 17:39:47 UTC
data deleted came back after 9 days.
Hi All,
We have strange issue here.
We have 10 nodes cross 5 datacenters. Today I found a strange thing.
On one node, few data deleted came back after 8-9 days.
The data saved on a node and retrieved/deleted on another node in a
remote datacenter. The CF is a super column.
What is possible causing this?
Thanks,
Zhong Li
Re: data deleted came back after 9 days.
Posted by Jonathan Ellis <jb...@gmail.com>.
Corrected, thanks. (Better would be to edit the wiki yourself, of course. :)
On Tue, Aug 17, 2010 at 2:58 PM, Jeremy Dunck <jd...@gmail.com> wrote:
> On Tue, Aug 17, 2010 at 2:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> It doesn't have to be disconnected more than GC grace seconds to cause
>> what you are seeing, it just has to be disconnected at all (thus
>> missing delete commands).
>>
>> Thus you need to be running repair more often than gcgrace, or
>> confident that read repair will handle it for you (which clearly is
>> not the case for you :). see
>> http://wiki.apache.org/cassandra/Operations
>
> FWIW, the docs there say:
> "Remember though that if a node is down longer than your configured
> GCGraceSeconds (default: 10 days), it could have missed remove
> operations permanently"
>
> So that's probably a source of misunderstanding.
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: data deleted came back after 9 days.
Posted by Jonathan Ellis <jb...@gmail.com>.
Actually, tombstones are read repaired too -- as long as they are not
expired. But nodetool repair is much less error-prone than relying on
RR and your memory of what deletes you issued.
Either way, you'd need to increase GCGraceSeconds first to make the
tombstones un-expired first.
On Wed, Aug 18, 2010 at 12:43 AM, Benjamin Black <b...@b3k.us> wrote:
> On Tue, Aug 17, 2010 at 7:49 PM, Zhong Li <zl...@voxeo.com> wrote:
>> Those data were inserted one node, then deleted on a remote node in less
>> than 2 seconds. So it is very possible some node lost tombstone when
>> connection lost.
>> My question, is a ConstencyLevel.ALL read can retrieve lost tombstone back
>> instead of repair?
>>
>
> No. Read repair does not replay operations. You must run nodetool repair.
>
>
> b
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: data deleted came back after 9 days.
Posted by Benjamin Black <b...@b3k.us>.
On Tue, Aug 17, 2010 at 7:49 PM, Zhong Li <zl...@voxeo.com> wrote:
> Those data were inserted one node, then deleted on a remote node in less
> than 2 seconds. So it is very possible some node lost tombstone when
> connection lost.
> My question, is a ConstencyLevel.ALL read can retrieve lost tombstone back
> instead of repair?
>
No. Read repair does not replay operations. You must run nodetool repair.
b
Re: data deleted came back after 9 days.
Posted by Zhong Li <zl...@voxeo.com>.
Those data were inserted one node, then deleted on a remote node in
less than 2 seconds. So it is very possible some node lost tombstone
when connection lost.
My question, is a ConstencyLevel.ALL read can retrieve lost tombstone
back instead of repair?
On Aug 17, 2010, at 4:11 PM, Ned Wolpert wrote:
> (gurus, please check my logic here... I'm trying to validate my
> understanding of this situation.)
>
> Isn't the issue that while a server was disconnected, a delete could
> have occurred, and thus the disconnected server never got the
> 'tombstone'?
> (http://wiki.apache.org/cassandra/DistributedDeletes) When it comes
> back, only after it receives the delete request will the data be
> deleted from the reconnected server. I do not think this happens
> automatically when the server rejoins the cluster, but requires the
> manual repair command.
>
> From my understanding, if the consistency level is greater then the
> number of servers missing that tombstone, you'll get the correct
> data. If its less, then you 'could' get the right or wrong answer.
> So the issue is how often do you need to run repair? If you have a
> ReplicationFactor=3, and you use ConstencyLevel.QUORUM, (2
> responses) then you need to run it after one server fails just to be
> sure. If you can handle some tolerance for this, you can wait a bit
> more before running the repair.
>
> On Tue, Aug 17, 2010 at 12:58 PM, Jeremy Dunck <jd...@gmail.com>
> wrote:
> On Tue, Aug 17, 2010 at 2:49 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> > It doesn't have to be disconnected more than GC grace seconds to
> cause
> > what you are seeing, it just has to be disconnected at all (thus
> > missing delete commands).
> >
> > Thus you need to be running repair more often than gcgrace, or
> > confident that read repair will handle it for you (which clearly is
> > not the case for you :). see
> > http://wiki.apache.org/cassandra/Operations
>
> FWIW, the docs there say:
> "Remember though that if a node is down longer than your configured
> GCGraceSeconds (default: 10 days), it could have missed remove
> operations permanently"
>
> So that's probably a source of misunderstanding.
>
>
>
> --
> Virtually, Ned Wolpert
>
> "Settle thy studies, Faustus, and begin..." --Marlowe
Re: data deleted came back after 9 days.
Posted by Jonathan Ellis <jb...@gmail.com>.
Best practice is to schedule repair more often than GCGraceSeconds,
say weekly, rather than doing it manually when you notice the FD mark
someone dead.
On Tue, Aug 17, 2010 at 3:11 PM, Ned Wolpert <ne...@imemories.com> wrote:
> (gurus, please check my logic here... I'm trying to validate my
> understanding of this situation.)
> Isn't the issue that while a server was disconnected, a delete could have
> occurred, and thus the disconnected server never got the 'tombstone'?
> (http://wiki.apache.org/cassandra/DistributedDeletes) When it comes back,
> only after it receives the delete request will the data be deleted from the
> reconnected server. I do not think this happens automatically when the
> server rejoins the cluster, but requires the manual repair command.
> From my understanding, if the consistency level is greater then the number
> of servers missing that tombstone, you'll get the correct data. If its less,
> then you 'could' get the right or wrong answer. So the issue is how often do
> you need to run repair? If you have a ReplicationFactor=3, and you use
> ConstencyLevel.QUORUM, (2 responses) then you need to run it after one
> server fails just to be sure. If you can handle some tolerance for this, you
> can wait a bit more before running the repair.
> On Tue, Aug 17, 2010 at 12:58 PM, Jeremy Dunck <jd...@gmail.com> wrote:
>>
>> On Tue, Aug 17, 2010 at 2:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> > It doesn't have to be disconnected more than GC grace seconds to cause
>> > what you are seeing, it just has to be disconnected at all (thus
>> > missing delete commands).
>> >
>> > Thus you need to be running repair more often than gcgrace, or
>> > confident that read repair will handle it for you (which clearly is
>> > not the case for you :). see
>> > http://wiki.apache.org/cassandra/Operations
>>
>> FWIW, the docs there say:
>> "Remember though that if a node is down longer than your configured
>> GCGraceSeconds (default: 10 days), it could have missed remove
>> operations permanently"
>>
>> So that's probably a source of misunderstanding.
>
>
>
> --
> Virtually, Ned Wolpert
>
> "Settle thy studies, Faustus, and begin..." --Marlowe
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: data deleted came back after 9 days.
Posted by Ned Wolpert <ne...@imemories.com>.
(gurus, please check my logic here... I'm trying to validate my
understanding of this situation.)
Isn't the issue that while a server was disconnected, a delete could have
occurred, and thus the disconnected server never got the 'tombstone'?
(http://wiki.apache.org/cassandra/DistributedDeletes) When it comes back,
only after it receives the delete request will the data be deleted from the
reconnected server. I do not think this happens automatically when the
server rejoins the cluster, but requires the manual repair command.
>From my understanding, if the consistency level is greater then the number
of servers missing that tombstone, you'll get the correct data. If its less,
then you 'could' get the right or wrong answer. So the issue is how often do
you need to run repair? If you have a ReplicationFactor=3, and you use
ConstencyLevel.QUORUM, (2 responses) then you need to run it after one
server fails just to be sure. If you can handle some tolerance for this, you
can wait a bit more before running the repair.
On Tue, Aug 17, 2010 at 12:58 PM, Jeremy Dunck <jd...@gmail.com> wrote:
> On Tue, Aug 17, 2010 at 2:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> > It doesn't have to be disconnected more than GC grace seconds to cause
> > what you are seeing, it just has to be disconnected at all (thus
> > missing delete commands).
> >
> > Thus you need to be running repair more often than gcgrace, or
> > confident that read repair will handle it for you (which clearly is
> > not the case for you :). see
> > http://wiki.apache.org/cassandra/Operations
>
> FWIW, the docs there say:
> "Remember though that if a node is down longer than your configured
> GCGraceSeconds (default: 10 days), it could have missed remove
> operations permanently"
>
> So that's probably a source of misunderstanding.
>
--
Virtually, Ned Wolpert
"Settle thy studies, Faustus, and begin..." --Marlowe
Re: data deleted came back after 9 days.
Posted by Jeremy Dunck <jd...@gmail.com>.
On Tue, Aug 17, 2010 at 2:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> It doesn't have to be disconnected more than GC grace seconds to cause
> what you are seeing, it just has to be disconnected at all (thus
> missing delete commands).
>
> Thus you need to be running repair more often than gcgrace, or
> confident that read repair will handle it for you (which clearly is
> not the case for you :). see
> http://wiki.apache.org/cassandra/Operations
FWIW, the docs there say:
"Remember though that if a node is down longer than your configured
GCGraceSeconds (default: 10 days), it could have missed remove
operations permanently"
So that's probably a source of misunderstanding.
Re: data deleted came back after 9 days.
Posted by Jonathan Ellis <jb...@gmail.com>.
It doesn't have to be disconnected more than GC grace seconds to cause
what you are seeing, it just has to be disconnected at all (thus
missing delete commands).
Thus you need to be running repair more often than gcgrace, or
confident that read repair will handle it for you (which clearly is
not the case for you :). see
http://wiki.apache.org/cassandra/Operations
On Tue, Aug 17, 2010 at 11:35 AM, Zhong Li <zl...@voxeo.com> wrote:
> <GCGraceSeconds>864000</GCGraceSeconds>
> It is default 10 days.
>
> I checked all system.log, all nodes are connected, although not all the
> time, but they reconnected after a few minutes. None of node disconnected
> more than GC grace seconds.
>
> Best,
>
> On Aug 17, 2010, at 11:53 AM, Peter Schuller wrote:
>
>>>> We have 10 nodes cross 5 datacenters. Today I found a strange thing. On
>>>> one node, few data deleted came back after 8-9 days.
>>>>
>>>> The data saved on a node and retrieved/deleted on another node in a
>>>> remote
>>>> datacenter. The CF is a super column.
>>>>
>>>> What is possible causing this?
>>
>> What is your GC grace seconds set to? Is it lower than 8-9 days, and
>> is it possible one or more nodes were disconnected from the remainder
>> of the cluster for a period longer than the GC grace seconds?
>>
>> See: http://wiki.apache.org/cassandra/DistributedDeletes
>>
>> --
>> / Peter Schuller
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: data deleted came back after 9 days.
Posted by Zhong Li <zl...@voxeo.com>.
<GCGraceSeconds>864000</GCGraceSeconds>
It is default 10 days.
I checked all system.log, all nodes are connected, although not all
the time, but they reconnected after a few minutes. None of node
disconnected more than GC grace seconds.
Best,
On Aug 17, 2010, at 11:53 AM, Peter Schuller wrote:
>>> We have 10 nodes cross 5 datacenters. Today I found a strange
>>> thing. On
>>> one node, few data deleted came back after 8-9 days.
>>>
>>> The data saved on a node and retrieved/deleted on another node in
>>> a remote
>>> datacenter. The CF is a super column.
>>>
>>> What is possible causing this?
>
> What is your GC grace seconds set to? Is it lower than 8-9 days, and
> is it possible one or more nodes were disconnected from the remainder
> of the cluster for a period longer than the GC grace seconds?
>
> See: http://wiki.apache.org/cassandra/DistributedDeletes
>
> --
> / Peter Schuller
Re: data deleted came back after 9 days.
Posted by Peter Schuller <pe...@infidyne.com>.
>> We have 10 nodes cross 5 datacenters. Today I found a strange thing. On
>> one node, few data deleted came back after 8-9 days.
>>
>> The data saved on a node and retrieved/deleted on another node in a remote
>> datacenter. The CF is a super column.
>>
>> What is possible causing this?
What is your GC grace seconds set to? Is it lower than 8-9 days, and
is it possible one or more nodes were disconnected from the remainder
of the cluster for a period longer than the GC grace seconds?
See: http://wiki.apache.org/cassandra/DistributedDeletes
--
/ Peter Schuller
Re: data deleted came back after 9 days.
Posted by Zhong Li <zl...@voxeo.com>.
Cassandra version is 0.6.3
On Aug 17, 2010, at 11:39 AM, Zhong Li wrote:
> Hi All,
>
> We have strange issue here.
>
> We have 10 nodes cross 5 datacenters. Today I found a strange
> thing. On one node, few data deleted came back after 8-9 days.
>
> The data saved on a node and retrieved/deleted on another node in a
> remote datacenter. The CF is a super column.
>
> What is possible causing this?
>
> Thanks,
>
> Zhong Li
>
>