You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by sauloricci <in...@gmail.com> on 2017/10/07 16:59:26 UTC
Strategies to query efficiently in HBase
I've designed HBase table where my rowkey is driven to the queries that I
need to run on it. Following the best practices I included a hashed portion
as prefix of my rowkeys, trying to spread the rows as much uniform as
possible across my table regions.
I'm concerned about the following scenario:
I got billions of rows in my HBase table split between 3 regions. I'm using
this table to provide data for a REST API, and because that, it needs to
provide the rows as fast as possible.
Unfortunately, I'm hitting the famous timeout error Failed to get result
within timeout, timeout=60000ms even if I invoke either count "table_name"
in the hbase shell or for any other scan including startrow and stoprow
parameters.It seems scanning by ranges using the rowkey is sweeping the
whole hbase table.
Considering this scenario I'd like to ask 2 questions:
1) Is there any strategy to really retrieve range of rows as fast as
possible without hitting the timeout error? I really appreciate any pointers
here.
2) As I'm new in HBase, do you think HBase is the best or at least adequate
solution to provide fast retrieval data to a REST API, considering the
search are in fact scans by range of rowkeys?
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
Re: Strategies to query efficiently in HBase
Posted by sauloricci <in...@gmail.com>.
Hi spats,
actually I got better performance approaching like this:
I split the table in more regions(50 per node) and got 5 region servers
instead of just 3 ones. Each region server now contains the double of the
original cores and the disk is SSD. So I got significant improvement.
I have in the end basically 250 regions spread in 5 region servers.
Do you think this is a reasonable approach that people uses in HBase? It
seems splitting more across more machines helped me significantly.
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
Re: Strategies to query efficiently in HBase
Posted by sudhir patil <sp...@gmail.com>.
Your row key looks good, only part not sure is why ur appending with 0x you
could have just appended number to string. But even in the current format
it should work.
Can you try scan with very small limit? to check if timeout is because of
large data you fetching or issue is something else.
On Tue, Oct 10, 2017 at 2:33 AM, sauloricci <in...@gmail.com> wrote:
> Hi spats,
>
> thank you for your message.
>
>
> Let me clarify a mistake I mentioned in my original post: I have 11 regions
> for that billions of rows table from 3 regions servers actually.
>
> I'm gonna answer your questions bellow:
>
> 1>
> This is the pattern of my rowkeys: "<hash>#<field_1>#<field_2>#
> 0x<field_3>",
> where <hash> is the sha2 of <field_1>
>
> I wanna to retrieve rows whose start keys as
> "<hash>#<field_1>#<field_2>#0x00000" and stoprows as
> "<hash>#<field_1>#<field_2>#0xfffff0", where the fields <hash>, <field_1>,
> <field_2> in both start and stop rows are the same. Just <field_3> is
> different.
>
> What do you think about this pattern? I hashed the most significant part of
> the rowkey and just changed the less significant part when searching the
> ranges I want.
>
> Thank you
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html
>
Re: Strategies to query efficiently in HBase
Posted by sauloricci <in...@gmail.com>.
Hi spats,
thank you for your message.
Let me clarify a mistake I mentioned in my original post: I have 11 regions
for that billions of rows table from 3 regions servers actually.
I'm gonna answer your questions bellow:
1>
This is the pattern of my rowkeys: "<hash>#<field_1>#<field_2>#0x<field_3>",
where <hash> is the sha2 of <field_1>
I wanna to retrieve rows whose start keys as
"<hash>#<field_1>#<field_2>#0x00000" and stoprows as
"<hash>#<field_1>#<field_2>#0xfffff0", where the fields <hash>, <field_1>,
<field_2> in both start and stop rows are the same. Just <field_3> is
different.
What do you think about this pattern? I hashed the most significant part of
the rowkey and just changed the less significant part when searching the
ranges I want.
Thank you
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
Re: Strategies to query efficiently in HBase
Posted by sudhir patil <sp...@gmail.com>.
1> surprising that your range scan times out. Could you share some insert
row keys(after hashing) and start & stop rowkeys you are using to
scan/query? Row keys design is very important, it has to match your
retrieval pattern.
2> Yes, its Hbase is best as long you are doing range scan or to get key
value pairs.
On Sun, Oct 8, 2017 at 1:16 AM, Ted Yu <yu...@gmail.com> wrote:
> What's the value for hbase.hregion.max.filesize ?
> By default it is 10GB.
>
> bq. any other scan including startrow and stoprow
>
> Did the start / stop rows encompass the 3 regions ?
>
> Please also see http://hbase.apache.org/book.html#ops.date.tiered
>
> Cheers
>
> On Sat, Oct 7, 2017 at 9:59 AM, sauloricci <in...@gmail.com> wrote:
>
> > I've designed HBase table where my rowkey is driven to the queries that I
> > need to run on it. Following the best practices I included a hashed
> portion
> > as prefix of my rowkeys, trying to spread the rows as much uniform as
> > possible across my table regions.
> >
> > I'm concerned about the following scenario:
> >
> > I got billions of rows in my HBase table split between 3 regions. I'm
> using
> > this table to provide data for a REST API, and because that, it needs to
> > provide the rows as fast as possible.
> >
> > Unfortunately, I'm hitting the famous timeout error Failed to get result
> > within timeout, timeout=60000ms even if I invoke either count
> "table_name"
> > in the hbase shell or for any other scan including startrow and stoprow
> > parameters.It seems scanning by ranges using the rowkey is sweeping the
> > whole hbase table.
> >
> > Considering this scenario I'd like to ask 2 questions:
> >
> > 1) Is there any strategy to really retrieve range of rows as fast as
> > possible without hitting the timeout error? I really appreciate any
> > pointers
> > here.
> >
> > 2) As I'm new in HBase, do you think HBase is the best or at least
> adequate
> > solution to provide fast retrieval data to a REST API, considering the
> > search are in fact scans by range of rowkeys?
> >
> >
> >
> > --
> > Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> > f4020416.html
> >
>
Re: Strategies to query efficiently in HBase
Posted by sauloricci <in...@gmail.com>.
Hi Ted Yu-3,
yes, my hbase.hregion.max.filesize is set to 10GB as you mentioned.
Let me clarify a mistake I mentioned in my original post: I have 11 regions
for that billions of rows table from 3 regions servers actually.
The start/stop rows should most of times encompass just only one region, but
there are cases where I want to make broader search and they will encompass
more than just one region for that table.
What do you think about this scenario?
Thank you
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
Re: Strategies to query efficiently in HBase
Posted by Ted Yu <yu...@gmail.com>.
What's the value for hbase.hregion.max.filesize ?
By default it is 10GB.
bq. any other scan including startrow and stoprow
Did the start / stop rows encompass the 3 regions ?
Please also see http://hbase.apache.org/book.html#ops.date.tiered
Cheers
On Sat, Oct 7, 2017 at 9:59 AM, sauloricci <in...@gmail.com> wrote:
> I've designed HBase table where my rowkey is driven to the queries that I
> need to run on it. Following the best practices I included a hashed portion
> as prefix of my rowkeys, trying to spread the rows as much uniform as
> possible across my table regions.
>
> I'm concerned about the following scenario:
>
> I got billions of rows in my HBase table split between 3 regions. I'm using
> this table to provide data for a REST API, and because that, it needs to
> provide the rows as fast as possible.
>
> Unfortunately, I'm hitting the famous timeout error Failed to get result
> within timeout, timeout=60000ms even if I invoke either count "table_name"
> in the hbase shell or for any other scan including startrow and stoprow
> parameters.It seems scanning by ranges using the rowkey is sweeping the
> whole hbase table.
>
> Considering this scenario I'd like to ask 2 questions:
>
> 1) Is there any strategy to really retrieve range of rows as fast as
> possible without hitting the timeout error? I really appreciate any
> pointers
> here.
>
> 2) As I'm new in HBase, do you think HBase is the best or at least adequate
> solution to provide fast retrieval data to a REST API, considering the
> search are in fact scans by range of rowkeys?
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html
>