You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Héctor Izquierdo Seliva <iz...@strands.com> on 2011/04/19 17:36:30 UTC
Tombstones and memtable_operations
Hi everyone. I've configured in one of my column families
memtable_operations = 0.02 and started deleting keys. I have already
deleted 54k, but there hasn't been any flush of the memtable. Memory
keeps pilling up and eventually nodes start to do stop-the-world GCs. Is
this the way this is supposed to work or have I done something wrong?
Thanks!
Re: Tombstones and memtable_operations
Posted by Héctor Izquierdo Seliva <iz...@strands.com>.
El mié, 20-04-2011 a las 23:00 +1200, aaron morton escribió:
> Looks like a bug, I've added a patch
> here https://issues.apache.org/jira/browse/CASSANDRA-2519
>
>
> Aaron
>
That was fast! Thanks Aaron
Re: Tombstones and memtable_operations
Posted by aaron morton <aa...@thelastpickle.com>.
Looks like a bug, I've added a patch here https://issues.apache.org/jira/browse/CASSANDRA-2519
Aaron
On 20 Apr 2011, at 13:15, aaron morton wrote:
> Thats what I was looking for, thanks.
>
> At first glance the behaviour looks inconsistent, we count the number of columns in the delete mutation. But when deleting a row the column count is zero. I'll try to take a look later.
>
> In the mean time you can force a memtable via JConsole, navigate down to the CF and look for the forceFlush() operation.
>
> Aaron
> On 20 Apr 2011, at 09:39, Héctor Izquierdo Seliva wrote:
>
>> El mié, 20-04-2011 a las 09:08 +1200, aaron morton escribió:
>>> Yes, I saw that.
>>>
>>> Wanted to know what "issue deletes through pelops" means so I can work out what command it's sending to cassandra and hopefully I don't waste my time looking in the wrong place.
>>>
>>> Aaron
>>>
>>
>> Oh, sorry. Didn't get what you were asking. I use this code:
>>
>> RowDeletor deletor = Pelops.createRowDeletor(keySpace);
>> deletor.deleteRow(cf, rowId, ConsistencyLevel.QUORUM);
>>
>> which seems to be calling
>> org.apache.cassandra.thrift.Cassandra.Client.remove.
>>
>> I hope this is useful
>>
>>
>
Re: Tombstones and memtable_operations
Posted by aaron morton <aa...@thelastpickle.com>.
Thats what I was looking for, thanks.
At first glance the behaviour looks inconsistent, we count the number of columns in the delete mutation. But when deleting a row the column count is zero. I'll try to take a look later.
In the mean time you can force a memtable via JConsole, navigate down to the CF and look for the forceFlush() operation.
Aaron
On 20 Apr 2011, at 09:39, Héctor Izquierdo Seliva wrote:
> El mié, 20-04-2011 a las 09:08 +1200, aaron morton escribió:
>> Yes, I saw that.
>>
>> Wanted to know what "issue deletes through pelops" means so I can work out what command it's sending to cassandra and hopefully I don't waste my time looking in the wrong place.
>>
>> Aaron
>>
>
> Oh, sorry. Didn't get what you were asking. I use this code:
>
> RowDeletor deletor = Pelops.createRowDeletor(keySpace);
> deletor.deleteRow(cf, rowId, ConsistencyLevel.QUORUM);
>
> which seems to be calling
> org.apache.cassandra.thrift.Cassandra.Client.remove.
>
> I hope this is useful
>
>
Re: Tombstones and memtable_operations
Posted by Héctor Izquierdo Seliva <iz...@strands.com>.
El mié, 20-04-2011 a las 09:08 +1200, aaron morton escribió:
> Yes, I saw that.
>
> Wanted to know what "issue deletes through pelops" means so I can work out what command it's sending to cassandra and hopefully I don't waste my time looking in the wrong place.
>
> Aaron
>
Oh, sorry. Didn't get what you were asking. I use this code:
RowDeletor deletor = Pelops.createRowDeletor(keySpace);
deletor.deleteRow(cf, rowId, ConsistencyLevel.QUORUM);
which seems to be calling
org.apache.cassandra.thrift.Cassandra.Client.remove.
I hope this is useful
Re: Tombstones and memtable_operations
Posted by aaron morton <aa...@thelastpickle.com>.
Yes, I saw that.
Wanted to know what "issue deletes through pelops" means so I can work out what command it's sending to cassandra and hopefully I don't waste my time looking in the wrong place.
Aaron
On 20 Apr 2011, at 09:04, Héctor Izquierdo Seliva wrote:
> I poste it a couple of messages back, but here it is again:
>
> I'm using 0.7.4. I have a file with all the row keys I have to delete
> (around 100 million) and I just go through the file and issue deletes
> through pelops. Should I manually issue flushes with a cron every x
> time?
>
Re: Tombstones and memtable_operations
Posted by Héctor Izquierdo Seliva <iz...@strands.com>.
I poste it a couple of messages back, but here it is again:
I'm using 0.7.4. I have a file with all the row keys I have to delete
(around 100 million) and I just go through the file and issue deletes
through pelops. Should I manually issue flushes with a cron every x
time?
Re: Tombstones and memtable_operations
Posted by aaron morton <aa...@thelastpickle.com>.
How do you do the deletes ?
Aaron
On 20 Apr 2011, at 08:39, Héctor Izquierdo Seliva wrote:
> El mar, 19-04-2011 a las 23:33 +0300, shimi escribió:
>> You can use memtable_flush_after_mins instead of the cron
>>
>>
>> Shimi
>>
>
> Good point! I'll try that.
>
> Wouldn't it be better to count a delete as a one column operation so it
> contributes to flush by operations?
>
>> 2011/4/19 Héctor Izquierdo Seliva <iz...@strands.com>
>>
>> El mié, 20-04-2011 a las 08:16 +1200, aaron morton escribió:
>>> I think their may be an issue here, we are counting the
>> number of columns in the operation. When deleting an entire
>> row we do not have a column count.
>>>
>>> Can you let us know what version you are using and how you
>> are doing the delete ?
>>>
>>> Thanks
>>> Aaron
>>>
>>
>>
>> I'm using 0.7.4. I have a file with all the row keys I have to
>> delete
>> (around 100 million) and I just go through the file and issue
>> deletes
>> through pelops.
>>
>> Should I manually issue flushes with a cron every x time?
>>
>>
>>> On 20 Apr 2011, at 04:21, Héctor Izquierdo Seliva wrote:
>>>
>>>> Ok, I've read about gc grace seconds, but i'm not sure I
>> understand it
>>>> fully. Untill gc grace seconds have passed, and there is a
>> compaction,
>>>> the tombstones live in memory? I have to delete 100
>> million rows and my
>>>> insert rate is very low, so I don't have a lot of
>> compactions. What
>>>> should I do in this case? Lower the major compaction
>> threshold and
>>>> memtable_operations to some very low number?
>>>>
>>>> Thanks
>>>>
>>>> El mar, 19-04-2011 a las 17:36 +0200, Héctor Izquierdo
>> Seliva escribió:
>>>>> Hi everyone. I've configured in one of my column families
>>>>> memtable_operations = 0.02 and started deleting keys. I
>> have already
>>>>> deleted 54k, but there hasn't been any flush of the
>> memtable. Memory
>>>>> keeps pilling up and eventually nodes start to do
>> stop-the-world GCs. Is
>>>>> this the way this is supposed to work or have I done
>> something wrong?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>>
>>
>
>
Re: Tombstones and memtable_operations
Posted by Héctor Izquierdo Seliva <iz...@strands.com>.
El mar, 19-04-2011 a las 23:33 +0300, shimi escribió:
> You can use memtable_flush_after_mins instead of the cron
>
>
> Shimi
>
Good point! I'll try that.
Wouldn't it be better to count a delete as a one column operation so it
contributes to flush by operations?
> 2011/4/19 Héctor Izquierdo Seliva <iz...@strands.com>
>
> El mié, 20-04-2011 a las 08:16 +1200, aaron morton escribió:
> > I think their may be an issue here, we are counting the
> number of columns in the operation. When deleting an entire
> row we do not have a column count.
> >
> > Can you let us know what version you are using and how you
> are doing the delete ?
> >
> > Thanks
> > Aaron
> >
>
>
> I'm using 0.7.4. I have a file with all the row keys I have to
> delete
> (around 100 million) and I just go through the file and issue
> deletes
> through pelops.
>
> Should I manually issue flushes with a cron every x time?
>
>
> > On 20 Apr 2011, at 04:21, Héctor Izquierdo Seliva wrote:
> >
> > > Ok, I've read about gc grace seconds, but i'm not sure I
> understand it
> > > fully. Untill gc grace seconds have passed, and there is a
> compaction,
> > > the tombstones live in memory? I have to delete 100
> million rows and my
> > > insert rate is very low, so I don't have a lot of
> compactions. What
> > > should I do in this case? Lower the major compaction
> threshold and
> > > memtable_operations to some very low number?
> > >
> > > Thanks
> > >
> > > El mar, 19-04-2011 a las 17:36 +0200, Héctor Izquierdo
> Seliva escribió:
> > >> Hi everyone. I've configured in one of my column families
> > >> memtable_operations = 0.02 and started deleting keys. I
> have already
> > >> deleted 54k, but there hasn't been any flush of the
> memtable. Memory
> > >> keeps pilling up and eventually nodes start to do
> stop-the-world GCs. Is
> > >> this the way this is supposed to work or have I done
> something wrong?
> > >>
> > >> Thanks!
> > >>
> > >
> > >
> >
>
>
>
>
>
Re: Tombstones and memtable_operations
Posted by shimi <sh...@gmail.com>.
You can use memtable_flush_after_mins instead of the cron
Shimi
2011/4/19 Héctor Izquierdo Seliva <iz...@strands.com>
>
> El mié, 20-04-2011 a las 08:16 +1200, aaron morton escribió:
> > I think their may be an issue here, we are counting the number of columns
> in the operation. When deleting an entire row we do not have a column count.
> >
> > Can you let us know what version you are using and how you are doing the
> delete ?
> >
> > Thanks
> > Aaron
> >
>
> I'm using 0.7.4. I have a file with all the row keys I have to delete
> (around 100 million) and I just go through the file and issue deletes
> through pelops.
>
> Should I manually issue flushes with a cron every x time?
>
> > On 20 Apr 2011, at 04:21, Héctor Izquierdo Seliva wrote:
> >
> > > Ok, I've read about gc grace seconds, but i'm not sure I understand it
> > > fully. Untill gc grace seconds have passed, and there is a compaction,
> > > the tombstones live in memory? I have to delete 100 million rows and my
> > > insert rate is very low, so I don't have a lot of compactions. What
> > > should I do in this case? Lower the major compaction threshold and
> > > memtable_operations to some very low number?
> > >
> > > Thanks
> > >
> > > El mar, 19-04-2011 a las 17:36 +0200, Héctor Izquierdo Seliva escribió:
> > >> Hi everyone. I've configured in one of my column families
> > >> memtable_operations = 0.02 and started deleting keys. I have already
> > >> deleted 54k, but there hasn't been any flush of the memtable. Memory
> > >> keeps pilling up and eventually nodes start to do stop-the-world GCs.
> Is
> > >> this the way this is supposed to work or have I done something wrong?
> > >>
> > >> Thanks!
> > >>
> > >
> > >
> >
>
>
>
Re: Tombstones and memtable_operations
Posted by Héctor Izquierdo Seliva <iz...@strands.com>.
El mié, 20-04-2011 a las 08:16 +1200, aaron morton escribió:
> I think their may be an issue here, we are counting the number of columns in the operation. When deleting an entire row we do not have a column count.
>
> Can you let us know what version you are using and how you are doing the delete ?
>
> Thanks
> Aaron
>
I'm using 0.7.4. I have a file with all the row keys I have to delete
(around 100 million) and I just go through the file and issue deletes
through pelops.
Should I manually issue flushes with a cron every x time?
> On 20 Apr 2011, at 04:21, Héctor Izquierdo Seliva wrote:
>
> > Ok, I've read about gc grace seconds, but i'm not sure I understand it
> > fully. Untill gc grace seconds have passed, and there is a compaction,
> > the tombstones live in memory? I have to delete 100 million rows and my
> > insert rate is very low, so I don't have a lot of compactions. What
> > should I do in this case? Lower the major compaction threshold and
> > memtable_operations to some very low number?
> >
> > Thanks
> >
> > El mar, 19-04-2011 a las 17:36 +0200, Héctor Izquierdo Seliva escribió:
> >> Hi everyone. I've configured in one of my column families
> >> memtable_operations = 0.02 and started deleting keys. I have already
> >> deleted 54k, but there hasn't been any flush of the memtable. Memory
> >> keeps pilling up and eventually nodes start to do stop-the-world GCs. Is
> >> this the way this is supposed to work or have I done something wrong?
> >>
> >> Thanks!
> >>
> >
> >
>
Re: Tombstones and memtable_operations
Posted by aaron morton <aa...@thelastpickle.com>.
I think their may be an issue here, we are counting the number of columns in the operation. When deleting an entire row we do not have a column count.
Can you let us know what version you are using and how you are doing the delete ?
Thanks
Aaron
On 20 Apr 2011, at 04:21, Héctor Izquierdo Seliva wrote:
> Ok, I've read about gc grace seconds, but i'm not sure I understand it
> fully. Untill gc grace seconds have passed, and there is a compaction,
> the tombstones live in memory? I have to delete 100 million rows and my
> insert rate is very low, so I don't have a lot of compactions. What
> should I do in this case? Lower the major compaction threshold and
> memtable_operations to some very low number?
>
> Thanks
>
> El mar, 19-04-2011 a las 17:36 +0200, Héctor Izquierdo Seliva escribió:
>> Hi everyone. I've configured in one of my column families
>> memtable_operations = 0.02 and started deleting keys. I have already
>> deleted 54k, but there hasn't been any flush of the memtable. Memory
>> keeps pilling up and eventually nodes start to do stop-the-world GCs. Is
>> this the way this is supposed to work or have I done something wrong?
>>
>> Thanks!
>>
>
>
Re: Tombstones and memtable_operations
Posted by Héctor Izquierdo Seliva <iz...@strands.com>.
Ok, I've read about gc grace seconds, but i'm not sure I understand it
fully. Untill gc grace seconds have passed, and there is a compaction,
the tombstones live in memory? I have to delete 100 million rows and my
insert rate is very low, so I don't have a lot of compactions. What
should I do in this case? Lower the major compaction threshold and
memtable_operations to some very low number?
Thanks
El mar, 19-04-2011 a las 17:36 +0200, Héctor Izquierdo Seliva escribió:
> Hi everyone. I've configured in one of my column families
> memtable_operations = 0.02 and started deleting keys. I have already
> deleted 54k, but there hasn't been any flush of the memtable. Memory
> keeps pilling up and eventually nodes start to do stop-the-world GCs. Is
> this the way this is supposed to work or have I done something wrong?
>
> Thanks!
>