You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Iulia Zidaru <iu...@1and1.ro> on 2011/03/08 13:30:59 UTC
Performance degradation when deleting columns
Hi,
We have some problems when performing a large amounts of deletes.
We are using ASF HBase 0.90 with cloudera distribution for
HDFS(cdh3b3)). We store in HBase the inverted index of some documents.
We get a constant throughput in inserting documents and in scanning the
table, but we have problems when some deletes are performed.
We did some tests (10 000) with the following operations:
- scan some rows (a few: 2-3 rows)
- delete some columns for the previous scanned rows(one column per test
deleted)
- add some columns for the previous scanned rows
We got a huge degradation in performance in both scan and deletes.
- scan took from 5 to 15 milliseconds (for first 10 000)
- delete took from 4 to 13 milliseconds
and it continued to decrease. After 30000 operations a scan took 45
milliseconds and a delete 16 milliseconds.
The put operation was (almost) constant in time (3.99 ms to 4.5ms after
30000 tests).
On long time tests, we had the same performance degradation, but it
seems that at some point the performance is up again. It might be a
major compaction, a disk flush or what?
In which way deletes affect the scan operation? How can we minimize
these effects? Do you know what operation will put the database in the
optimal state?
Thank you,
Iulia
--
Iulia Zidaru
Java Developer
1&1 Internet AG - Bucharest/Romania - Web Components Romania
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
iulia.zidaru@1and1.ro
0040 31 223 9153
Re: Performance degradation when deleting columns
Posted by Iulia Zidaru <iu...@1and1.ro>.
Thank you J-D. These operations are more clear to me now.
On 03/10/2011 12:58 AM, Jean-Daniel Cryans wrote:
> There's seems to be quite a few questions in your email, I'll try to
> answer all of them but pardon me if I miss any of them.
>
> J-D
>
> On Wed, Mar 9, 2011 at 1:33 AM, Iulia Zidaru<iu...@1and1.ro> wrote:
>> Thank you very much J-D. Your replay is very useful because we are working
>> to change the delete scenario and we have to understand what happens inside
>> HBase because it's impossible to entirely change deletes with put
>> operations.
>> The numbers are small because I did some tests on my local machine. On our
>> testing cluster we have much higher values, but the performance degrade is
>> still present. And, yes, we are hitting the same region over and over again.
>>
>> In my understanding things happen like this:
>>
>> Test 1(delete column1)
>> - load in MemStore the latest version for the row
>> - mark column 1 as deleted(insert a tombstone in MemStore)
> Nope. MemStore only gets new Puts and Deletes. There's something
> called the Block Cache and this is where the data is loaded from HDFS
> (unless it's already present). So it loads the blocks from HDFS, it
> may read from many files, and finally figures which is the value it is
> deleting and puts the tombstone in the MemStore.
>
>> Test 2(delete column2)
>> - load in MemStore the latest version for the row
>> - mark column 1 and column 2 as deleted(insert a tombstone in MemStore)
> Same comment, also the tombstone for column should already be there
> (unless there was a flush).
>
>>
>> Every scan should also avoid columns marked as deleted, so it has more and
>> more columns to avoid. Is this true?
> It's one way to put it... what does happen is that the MemStore
> _grows_ when you delete data and it will have more and more discarding
> to do as you add tombstones.
>
>> What is not entirely clear is what to do to have a good scan performance
>> again.
> Good question, it would need more experimentation or stop deleting so many rows.
>
>> I see here http://outerthought.org/blog/465-ot.html that there are some
>> operations on regions: flush, compaction and major compaction.
>> Deletes in the file system are performed only on a major compaction.
>> In which way the deleted rows are loaded into MemStore after a flush or a
>> minor compaction. Are they loaded with a tombstone or they are not loaded at
>> all?
> The tombstones are loaded yes, and they are deleted once a major
> compaction materializes the full row in memory and figures what can be
> discarded. HBase works just like Bigtable regarding deletes, it may be
> clearer in that paper: http://labs.google.com/papers/bigtable.html
>
>> Thank you,
>> Iulia
--
Iulia Zidaru
Java Developer
1&1 Internet AG - Bucharest/Romania - Web Components Romania
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
iulia.zidaru@1and1.ro
0040 31 223 9153
Re: Performance degradation when deleting columns
Posted by Jean-Daniel Cryans <jd...@apache.org>.
There's seems to be quite a few questions in your email, I'll try to
answer all of them but pardon me if I miss any of them.
J-D
On Wed, Mar 9, 2011 at 1:33 AM, Iulia Zidaru <iu...@1and1.ro> wrote:
> Thank you very much J-D. Your replay is very useful because we are working
> to change the delete scenario and we have to understand what happens inside
> HBase because it's impossible to entirely change deletes with put
> operations.
> The numbers are small because I did some tests on my local machine. On our
> testing cluster we have much higher values, but the performance degrade is
> still present. And, yes, we are hitting the same region over and over again.
>
> In my understanding things happen like this:
>
> Test 1(delete column1)
> - load in MemStore the latest version for the row
> - mark column 1 as deleted(insert a tombstone in MemStore)
Nope. MemStore only gets new Puts and Deletes. There's something
called the Block Cache and this is where the data is loaded from HDFS
(unless it's already present). So it loads the blocks from HDFS, it
may read from many files, and finally figures which is the value it is
deleting and puts the tombstone in the MemStore.
>
> Test 2(delete column2)
> - load in MemStore the latest version for the row
> - mark column 1 and column 2 as deleted(insert a tombstone in MemStore)
Same comment, also the tombstone for column should already be there
(unless there was a flush).
>
>
> Every scan should also avoid columns marked as deleted, so it has more and
> more columns to avoid. Is this true?
It's one way to put it... what does happen is that the MemStore
_grows_ when you delete data and it will have more and more discarding
to do as you add tombstones.
>
> What is not entirely clear is what to do to have a good scan performance
> again.
Good question, it would need more experimentation or stop deleting so many rows.
> I see here http://outerthought.org/blog/465-ot.html that there are some
> operations on regions: flush, compaction and major compaction.
> Deletes in the file system are performed only on a major compaction.
> In which way the deleted rows are loaded into MemStore after a flush or a
> minor compaction. Are they loaded with a tombstone or they are not loaded at
> all?
The tombstones are loaded yes, and they are deleted once a major
compaction materializes the full row in memory and figures what can be
discarded. HBase works just like Bigtable regarding deletes, it may be
clearer in that paper: http://labs.google.com/papers/bigtable.html
>
> Thank you,
> Iulia
Re: Performance degradation when deleting columns
Posted by Iulia Zidaru <iu...@1and1.ro>.
Thank you very much J-D. Your replay is very useful because we are
working to change the delete scenario and we have to understand what
happens inside HBase because it's impossible to entirely change deletes
with put operations.
The numbers are small because I did some tests on my local machine. On
our testing cluster we have much higher values, but the performance
degrade is still present. And, yes, we are hitting the same region over
and over again.
In my understanding things happen like this:
Test 1(delete column1)
- load in MemStore the latest version for the row
- mark column 1 as deleted(insert a tombstone in MemStore)
Test 2(delete column2)
- load in MemStore the latest version for the row
- mark column 1 and column 2 as deleted(insert a tombstone in MemStore)
Every scan should also avoid columns marked as deleted, so it has more
and more columns to avoid. Is this true?
What is not entirely clear is what to do to have a good scan performance
again.
I see here http://outerthought.org/blog/465-ot.html that there are some
operations on regions: flush, compaction and major compaction.
Deletes in the file system are performed only on a major compaction.
In which way the deleted rows are loaded into MemStore after a flush or
a minor compaction. Are they loaded with a tombstone or they are not
loaded at all?
Thank you,
Iulia
On 03/08/2011 08:35 PM, Jean-Daniel Cryans wrote:
> That's a weird use case Iulia ;)
>
> So those numbers are pretty small, for all I know you could be hitting
> the same region over and over again... so each run of the test
> influences the next one.
>
> In any case, deletes are special operations where it has the do a Get
> for each column that's specified in order to find the latest insert
> before inserting a tombstone in the MemStore. This pretty expensive
> and can generate a lot of churn in the block cache. It may be worth
> more investigation tho, Ryan might have something to add about that.
>
> A Put operation is really just an insert in the MemStore, it doesn't
> read from any file, so it's supposed to be faster than a Delete.
>
> J-D
>
> On Tue, Mar 8, 2011 at 4:30 AM, Iulia Zidaru<iu...@1and1.ro> wrote:
>> Hi,
>> We have some problems when performing a large amounts of deletes.
>>
>> We are using ASF HBase 0.90 with cloudera distribution for HDFS(cdh3b3)). We
>> store in HBase the inverted index of some documents. We get a constant
>> throughput in inserting documents and in scanning the table, but we have
>> problems when some deletes are performed.
>>
>> We did some tests (10 000) with the following operations:
>> - scan some rows (a few: 2-3 rows)
>> - delete some columns for the previous scanned rows(one column per test
>> deleted)
>> - add some columns for the previous scanned rows
>>
>> We got a huge degradation in performance in both scan and deletes.
>> - scan took from 5 to 15 milliseconds (for first 10 000)
>> - delete took from 4 to 13 milliseconds
>> and it continued to decrease. After 30000 operations a scan took 45
>> milliseconds and a delete 16 milliseconds.
>>
>> The put operation was (almost) constant in time (3.99 ms to 4.5ms after
>> 30000 tests).
>>
>> On long time tests, we had the same performance degradation, but it seems
>> that at some point the performance is up again. It might be a major
>> compaction, a disk flush or what?
>>
>> In which way deletes affect the scan operation? How can we minimize these
>> effects? Do you know what operation will put the database in the optimal
>> state?
>>
>> Thank you,
>> Iulia
>>
>>
>>
>> --
>> Iulia Zidaru
>> Java Developer
>>
>> 1&1 Internet AG - Bucharest/Romania - Web Components Romania
>> 18 Mircea Eliade St
>> Sect 1, Bucharest
>> RO Bucharest, 012015
>> iulia.zidaru@1and1.ro
>> 0040 31 223 9153
>>
>>
>>
--
Iulia Zidaru
Java Developer
1&1 Internet AG - Bucharest/Romania - Web Components Romania
18 Mircea Eliade St
Sect 1, Bucharest
RO Bucharest, 012015
iulia.zidaru@1and1.ro
0040 31 223 9153
Re: Performance degradation when deleting columns
Posted by Jean-Daniel Cryans <jd...@apache.org>.
That's a weird use case Iulia ;)
So those numbers are pretty small, for all I know you could be hitting
the same region over and over again... so each run of the test
influences the next one.
In any case, deletes are special operations where it has the do a Get
for each column that's specified in order to find the latest insert
before inserting a tombstone in the MemStore. This pretty expensive
and can generate a lot of churn in the block cache. It may be worth
more investigation tho, Ryan might have something to add about that.
A Put operation is really just an insert in the MemStore, it doesn't
read from any file, so it's supposed to be faster than a Delete.
J-D
On Tue, Mar 8, 2011 at 4:30 AM, Iulia Zidaru <iu...@1and1.ro> wrote:
> Hi,
> We have some problems when performing a large amounts of deletes.
>
> We are using ASF HBase 0.90 with cloudera distribution for HDFS(cdh3b3)). We
> store in HBase the inverted index of some documents. We get a constant
> throughput in inserting documents and in scanning the table, but we have
> problems when some deletes are performed.
>
> We did some tests (10 000) with the following operations:
> - scan some rows (a few: 2-3 rows)
> - delete some columns for the previous scanned rows(one column per test
> deleted)
> - add some columns for the previous scanned rows
>
> We got a huge degradation in performance in both scan and deletes.
> - scan took from 5 to 15 milliseconds (for first 10 000)
> - delete took from 4 to 13 milliseconds
> and it continued to decrease. After 30000 operations a scan took 45
> milliseconds and a delete 16 milliseconds.
>
> The put operation was (almost) constant in time (3.99 ms to 4.5ms after
> 30000 tests).
>
> On long time tests, we had the same performance degradation, but it seems
> that at some point the performance is up again. It might be a major
> compaction, a disk flush or what?
>
> In which way deletes affect the scan operation? How can we minimize these
> effects? Do you know what operation will put the database in the optimal
> state?
>
> Thank you,
> Iulia
>
>
>
> --
> Iulia Zidaru
> Java Developer
>
> 1&1 Internet AG - Bucharest/Romania - Web Components Romania
> 18 Mircea Eliade St
> Sect 1, Bucharest
> RO Bucharest, 012015
> iulia.zidaru@1and1.ro
> 0040 31 223 9153
>
>
>