You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Hiller, Dean" <De...@nrel.gov> on 2012/10/01 19:32:12 UTC
read-repair and deletes / forgotten deletes
I know there is a 10 day limit if you have a node out of the cluster where you better be running read-repair or you end up with forgotten deletes, but what about on a clean cluster with all nodes always available? Shouldn't the deletes eventually take place or does one have to keep running read-repair manually all the time(god, I hope not as that should be automated for deletes).
I am seeing the empty rows in the CLI right now like so…(and would prefer they go away on their own)…
RowKey: post:10079376916:b2:4045302966737879216
-------------------
RowKey: post:10079386137:b2:3120637961487430473
-------------------
RowKey: post:10079377944:b2:5118918738052923761
-------------------
RowKey: post:10079378212:b2:1449128414219430571
-------------------
RowKey: post:10079386010:b2:8567843818586569179
If not, someone has to already have an automated project for this, anyone know of one??
Thanks,
Dean
Re: read-repair and deletes / forgotten deletes
Posted by Aaron Turner <sy...@gmail.com>.
inline...
On Mon, Oct 1, 2012 at 7:46 PM, Hiller, Dean <De...@nrel.gov> wrote:
> Thanks, (actually new it was configurable) BUT what I don't get is why I
> have to run a repair. IF all nodes became consistent on the delete, it
> should not be possible to get a forgotten delete, correct. The forgotten
> delete will only occur if I have a node down and out for 10 days and it
> comes back online because by then the nodes no longer have the delete
> anymore and the new node has data so getting to a consistent state the
> node with data would win.
>
> Soooo, if I run repair say every 20 days, isn't it true, I would have no
> problems as long as I did not have a node outage?
Basically if you know for certain that you have 100% uptime for all
your nodes and you haven't lost any updates due to overload/etc then
you don't need to run repair. You run repair to ensure that all the
tombstones are replicated to all the necessary nodes prior to gc_grace
so that the data doesn't come back from the dead after a compaction.
Generally speaking it's a lot safer/easier to just always run repair.
> And most importantly, does anyone know of an automated tool for running
> repairs every X days(this should really be an automated/schedulable
> thing)???
I use a cron job. It's a good idea to use the '-pr' flag btw. Also,
you only need to run repair against CF's which actually have deletes.
--
Aaron Turner
http://synfin.net/ Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
-- Benjamin Franklin
"carpe diem quam minimum credula postero"
Re: read-repair and deletes / forgotten deletes
Posted by "Hiller, Dean" <De...@nrel.gov>.
Oh, and I have been reading Aaron Mortan's article here
http://thelastpickle.com/2011/05/15/Deletes-and-Tombstones/
On 10/1/12 12:46 PM, "Hiller, Dean" <De...@nrel.gov> wrote:
>Thanks, (actually new it was configurable) BUT what I don't get is why I
>have to run a repair. IF all nodes became consistent on the delete, it
>should not be possible to get a forgotten delete, correct. The forgotten
>delete will only occur if I have a node down and out for 10 days and it
>comes back online because by then the nodes no longer have the delete
>anymore and the new node has data so getting to a consistent state the
>node with data would win.
>
>Soooo, if I run repair say every 20 days, isn't it true, I would have no
>problems as long as I did not have a node outage?
>
>And most importantly, does anyone know of an automated tool for running
>repairs every X days(this should really be an automated/schedulable
>thing)???
>
>Thanks,
>Dean
>
>On 10/1/12 11:57 AM, "Aaron Turner" <sy...@gmail.com> wrote:
>
>>the 10 days is actually configurable... look into gc_grace.
>>
>>Basically, you always need to run repair once per/gc_grace period.
>>You won't see empty/deleted rows go away until they're compacted away.
>>
>>On Mon, Oct 1, 2012 at 6:32 PM, Hiller, Dean <De...@nrel.gov>
>>wrote:
>>> I know there is a 10 day limit if you have a node out of the cluster
>>>where you better be running read-repair or you end up with forgotten
>>>deletes, but what about on a clean cluster with all nodes always
>>>available? Shouldn't the deletes eventually take place or does one have
>>>to keep running read-repair manually all the time(god, I hope not as
>>>that should be automated for deletes).
>>>
>>> I am seeing the empty rows in the CLI right now like soŠ(and would
>>>prefer they go away on their own)Š
>>>
>>> RowKey: post:10079376916:b2:4045302966737879216
>>> -------------------
>>> RowKey: post:10079386137:b2:3120637961487430473
>>> -------------------
>>> RowKey: post:10079377944:b2:5118918738052923761
>>> -------------------
>>> RowKey: post:10079378212:b2:1449128414219430571
>>> -------------------
>>> RowKey: post:10079386010:b2:8567843818586569179
>>>
>>> If not, someone has to already have an automated project for this,
>>>anyone know of one??
>>>
>>> Thanks,
>>> Dean
>>>
>>
>>
>>
>>--
>>Aaron Turner
>>http://synfin.net/ Twitter: @synfinatic
>>http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix &
>>Windows
>>Those who would give up essential Liberty, to purchase a little temporary
>>Safety, deserve neither Liberty nor Safety.
>> -- Benjamin Franklin
>>"carpe diem quam minimum credula postero"
>
Re: read-repair and deletes / forgotten deletes
Posted by "Hiller, Dean" <De...@nrel.gov>.
Thanks, (actually new it was configurable) BUT what I don't get is why I
have to run a repair. IF all nodes became consistent on the delete, it
should not be possible to get a forgotten delete, correct. The forgotten
delete will only occur if I have a node down and out for 10 days and it
comes back online because by then the nodes no longer have the delete
anymore and the new node has data so getting to a consistent state the
node with data would win.
Soooo, if I run repair say every 20 days, isn't it true, I would have no
problems as long as I did not have a node outage?
And most importantly, does anyone know of an automated tool for running
repairs every X days(this should really be an automated/schedulable
thing)???
Thanks,
Dean
On 10/1/12 11:57 AM, "Aaron Turner" <sy...@gmail.com> wrote:
>the 10 days is actually configurable... look into gc_grace.
>
>Basically, you always need to run repair once per/gc_grace period.
>You won't see empty/deleted rows go away until they're compacted away.
>
>On Mon, Oct 1, 2012 at 6:32 PM, Hiller, Dean <De...@nrel.gov> wrote:
>> I know there is a 10 day limit if you have a node out of the cluster
>>where you better be running read-repair or you end up with forgotten
>>deletes, but what about on a clean cluster with all nodes always
>>available? Shouldn't the deletes eventually take place or does one have
>>to keep running read-repair manually all the time(god, I hope not as
>>that should be automated for deletes).
>>
>> I am seeing the empty rows in the CLI right now like soŠ(and would
>>prefer they go away on their own)Š
>>
>> RowKey: post:10079376916:b2:4045302966737879216
>> -------------------
>> RowKey: post:10079386137:b2:3120637961487430473
>> -------------------
>> RowKey: post:10079377944:b2:5118918738052923761
>> -------------------
>> RowKey: post:10079378212:b2:1449128414219430571
>> -------------------
>> RowKey: post:10079386010:b2:8567843818586569179
>>
>> If not, someone has to already have an automated project for this,
>>anyone know of one??
>>
>> Thanks,
>> Dean
>>
>
>
>
>--
>Aaron Turner
>http://synfin.net/ Twitter: @synfinatic
>http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix &
>Windows
>Those who would give up essential Liberty, to purchase a little temporary
>Safety, deserve neither Liberty nor Safety.
> -- Benjamin Franklin
>"carpe diem quam minimum credula postero"
Re: read-repair and deletes / forgotten deletes
Posted by Aaron Turner <sy...@gmail.com>.
the 10 days is actually configurable... look into gc_grace.
Basically, you always need to run repair once per/gc_grace period.
You won't see empty/deleted rows go away until they're compacted away.
On Mon, Oct 1, 2012 at 6:32 PM, Hiller, Dean <De...@nrel.gov> wrote:
> I know there is a 10 day limit if you have a node out of the cluster where you better be running read-repair or you end up with forgotten deletes, but what about on a clean cluster with all nodes always available? Shouldn't the deletes eventually take place or does one have to keep running read-repair manually all the time(god, I hope not as that should be automated for deletes).
>
> I am seeing the empty rows in the CLI right now like so…(and would prefer they go away on their own)…
>
> RowKey: post:10079376916:b2:4045302966737879216
> -------------------
> RowKey: post:10079386137:b2:3120637961487430473
> -------------------
> RowKey: post:10079377944:b2:5118918738052923761
> -------------------
> RowKey: post:10079378212:b2:1449128414219430571
> -------------------
> RowKey: post:10079386010:b2:8567843818586569179
>
> If not, someone has to already have an automated project for this, anyone know of one??
>
> Thanks,
> Dean
>
--
Aaron Turner
http://synfin.net/ Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
-- Benjamin Franklin
"carpe diem quam minimum credula postero"