You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Peter Rietzler <pe...@smarter-ecommerce.com> on 2009/07/17 07:49:00 UTC
Batched Gets and Deletes by Id / Performance Issues ?
Hi
First of all thanks for your great work! We are enjoying using HBase in our
project :)
One of our HBase tables uses UUIDs for keys. The character of the data is
random access and we are indexing the rows with an external indexer. The
access characteristic is much like you would do with a traditional SQL
statement - Issue a query to the index and then get all associated rows
using the list of returned ids (typically from a few 100 up to around
100.000 rows ...). The same applies for deletion.
While batch updating is available I miss an opportunity to batch fetch and
batch delete rows by id. I currently use HTable.getRow() and
HTable.deleteAll() for these operations - however - both of the operations
perform pretty poor for more than a couple of 100 rows (I guess it's because
an IPC call needs to be send for each call...). Unfortunately scanning is
not an issue for this use case.
Is it planned to provide batch random access batch fetches ? Something like
HTable.getRows(list of ids) and HTable.deleteAll(list of ids) would be
perfect :)
Cheers,
Peter
--
View this message in context: http://www.nabble.com/Batched-Gets-and-Deletes-by-Id---Performance-Issues---tp24528949p24528949.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Batched Gets and Deletes by Id / Performance Issues ?
Posted by stack <st...@duboce.net>.
HBASE-1574 is the issue for batch deletes. Do you want to have a go at
making a patch to add this batching? (if your patch includes tests, it'll go
in quicker).
St.Ack
On Fri, Jul 17, 2009 at 12:21 AM, Peter Rietzler <
peter.rietzler@smarter-ecommerce.com> wrote:
>
> Thanks - Basically our IDS are just postfixed with UUIDs to get them unique
> -
> we use a prefix to group data that is often read together - so this would
> be
> a great performance improvement for us!
>
>
>
> Ryan Rawson wrote:
> >
> > hey,
> >
> > there is no API calls right now, but no strong reasons why there could
> > not be. Not likely for 0.20.0, but maybe for .1, and more likely for
> > 0.21.
> >
> > Batching is only as effective as how clustered your keys on regions
> > are for your calls. If you are only seeing 1 key per region per
> > batch, you aren't getting any efficiencies. If you have a deeper
> > density, yeah you are going to get better performance.
> >
> >
> > On Thu, Jul 16, 2009 at 10:49 PM, Peter
> > Rietzler<pe...@smarter-ecommerce.com> wrote:
> >>
> >> Hi
> >>
> >> First of all thanks for your great work! We are enjoying using HBase in
> >> our
> >> project :)
> >>
> >> One of our HBase tables uses UUIDs for keys. The character of the data
> is
> >> random access and we are indexing the rows with an external indexer. The
> >> access characteristic is much like you would do with a traditional SQL
> >> statement - Issue a query to the index and then get all associated rows
> >> using the list of returned ids (typically from a few 100 up to around
> >> 100.000 rows ...). The same applies for deletion.
> >>
> >> While batch updating is available I miss an opportunity to batch fetch
> >> and
> >> batch delete rows by id. I currently use HTable.getRow() and
> >> HTable.deleteAll() for these operations - however - both of the
> >> operations
> >> perform pretty poor for more than a couple of 100 rows (I guess it's
> >> because
> >> an IPC call needs to be send for each call...). Unfortunately scanning
> is
> >> not an issue for this use case.
> >>
> >> Is it planned to provide batch random access batch fetches ? Something
> >> like
> >> HTable.getRows(list of ids) and HTable.deleteAll(list of ids) would be
> >> perfect :)
> >>
> >> Cheers,
> >> Peter
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Batched-Gets-and-Deletes-by-Id---Performance-Issues---tp24528949p24528949.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Batched-Gets-and-Deletes-by-Id---Performance-Issues---tp24528949p24529783.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
Re: Batched Gets and Deletes by Id / Performance Issues ?
Posted by Peter Rietzler <pe...@smarter-ecommerce.com>.
Thanks - Basically our IDS are just postfixed with UUIDs to get them unique -
we use a prefix to group data that is often read together - so this would be
a great performance improvement for us!
Ryan Rawson wrote:
>
> hey,
>
> there is no API calls right now, but no strong reasons why there could
> not be. Not likely for 0.20.0, but maybe for .1, and more likely for
> 0.21.
>
> Batching is only as effective as how clustered your keys on regions
> are for your calls. If you are only seeing 1 key per region per
> batch, you aren't getting any efficiencies. If you have a deeper
> density, yeah you are going to get better performance.
>
>
> On Thu, Jul 16, 2009 at 10:49 PM, Peter
> Rietzler<pe...@smarter-ecommerce.com> wrote:
>>
>> Hi
>>
>> First of all thanks for your great work! We are enjoying using HBase in
>> our
>> project :)
>>
>> One of our HBase tables uses UUIDs for keys. The character of the data is
>> random access and we are indexing the rows with an external indexer. The
>> access characteristic is much like you would do with a traditional SQL
>> statement - Issue a query to the index and then get all associated rows
>> using the list of returned ids (typically from a few 100 up to around
>> 100.000 rows ...). The same applies for deletion.
>>
>> While batch updating is available I miss an opportunity to batch fetch
>> and
>> batch delete rows by id. I currently use HTable.getRow() and
>> HTable.deleteAll() for these operations - however - both of the
>> operations
>> perform pretty poor for more than a couple of 100 rows (I guess it's
>> because
>> an IPC call needs to be send for each call...). Unfortunately scanning is
>> not an issue for this use case.
>>
>> Is it planned to provide batch random access batch fetches ? Something
>> like
>> HTable.getRows(list of ids) and HTable.deleteAll(list of ids) would be
>> perfect :)
>>
>> Cheers,
>> Peter
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Batched-Gets-and-Deletes-by-Id---Performance-Issues---tp24528949p24528949.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Batched-Gets-and-Deletes-by-Id---Performance-Issues---tp24528949p24529783.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Batched Gets and Deletes by Id / Performance Issues ?
Posted by Ryan Rawson <ry...@gmail.com>.
hey,
there is no API calls right now, but no strong reasons why there could
not be. Not likely for 0.20.0, but maybe for .1, and more likely for
0.21.
Batching is only as effective as how clustered your keys on regions
are for your calls. If you are only seeing 1 key per region per
batch, you aren't getting any efficiencies. If you have a deeper
density, yeah you are going to get better performance.
On Thu, Jul 16, 2009 at 10:49 PM, Peter
Rietzler<pe...@smarter-ecommerce.com> wrote:
>
> Hi
>
> First of all thanks for your great work! We are enjoying using HBase in our
> project :)
>
> One of our HBase tables uses UUIDs for keys. The character of the data is
> random access and we are indexing the rows with an external indexer. The
> access characteristic is much like you would do with a traditional SQL
> statement - Issue a query to the index and then get all associated rows
> using the list of returned ids (typically from a few 100 up to around
> 100.000 rows ...). The same applies for deletion.
>
> While batch updating is available I miss an opportunity to batch fetch and
> batch delete rows by id. I currently use HTable.getRow() and
> HTable.deleteAll() for these operations - however - both of the operations
> perform pretty poor for more than a couple of 100 rows (I guess it's because
> an IPC call needs to be send for each call...). Unfortunately scanning is
> not an issue for this use case.
>
> Is it planned to provide batch random access batch fetches ? Something like
> HTable.getRows(list of ids) and HTable.deleteAll(list of ids) would be
> perfect :)
>
> Cheers,
> Peter
>
>
> --
> View this message in context: http://www.nabble.com/Batched-Gets-and-Deletes-by-Id---Performance-Issues---tp24528949p24528949.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>