You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Matt Davies <ma...@mattdavies.net> on 2011/05/05 23:26:43 UTC
Considerations for using HBase in User Facing applications
Afternoon everyone,
I am researching what the best practice is for using HBase in user facing
applications. I do not know all of the applications that will be ported to
use HBase, but they do share common characteristics such as
- simple key / value data. Not serving large files ATM. Perhaps a couple
columns in a single column family
- very tall tables
- hundreds of millions of rows
- need millisecond access times for a single row
- random access
- maintain very, very good query times while loading in new data
The quick choice would be to use something like memcache or Redis, but the
data is growing faster than the memory of a single box or even few boxes.
We also have a significant investment in Hadoop technologies so keeping
HBase prime seems to make a lot of sense.
So, some questions:
1. do you find that having a single HBase cluster to serve all applications
vs smaller clusters to serve application specific data is better?
2. In the real world do people hook API's directly to HBase or is there some
caching layer that is used?
3. I remember hearing people like StumbleUpon use different clusters for
analytics vs customer apps. Is this still best practice?
4. Anyone using MSLAB's to reduce GC pauses in production? Experiences /
landmines?
5. What other considerations have you found when hooking HBase up for
user-facing applications?
Thanks in advance and I'd love to hear some bragging!
-Matt
Re: Considerations for using HBase in User Facing applications
Posted by Xavier Stevens <xs...@mozilla.com>.
1.) We use a different cluster for each app. I don't know if this is
best practice or not to be honest. We just wanted to separate downtime
and potential damage for each application.
2.) We usually use the HBase APIs directly. Having said that, we
recently started working on a new service. We were looking at Redis and
Hazelcast. We chose Hazelcast because we could do a MapStore
implementation easily that would persist the data to HBase.
Here's the implementation that we have so far:
https://github.com/xstevens/bagheera/blob/master/src/java/com/mozilla/bagheera/hazelcast/persistence/HBaseMapStore.java
We are really only using this as a memory staging area before we do
batch puts into HBase.
3.) The StumbleUpon guys have helped us a lot and we have a setup
similar to them. As of last night we're now using HBase replication to a
secondary "analysis" cluster.
4.) Can't really speak for MSLAB yet. We're using it on our analysis
cluster as a testing ground before using it in production. Todd's blog
posts were pretty thorough on the topic though.
Cheers,
-Xavier
On 5/5/11 2:26 PM, Matt Davies wrote:
> Afternoon everyone,
>
> I am researching what the best practice is for using HBase in user facing
> applications. I do not know all of the applications that will be ported to
> use HBase, but they do share common characteristics such as
>
> - simple key / value data. Not serving large files ATM. Perhaps a couple
> columns in a single column family
> - very tall tables
> - hundreds of millions of rows
> - need millisecond access times for a single row
> - random access
> - maintain very, very good query times while loading in new data
>
>
> The quick choice would be to use something like memcache or Redis, but the
> data is growing faster than the memory of a single box or even few boxes.
> We also have a significant investment in Hadoop technologies so keeping
> HBase prime seems to make a lot of sense.
>
> So, some questions:
>
> 1. do you find that having a single HBase cluster to serve all applications
> vs smaller clusters to serve application specific data is better?
> 2. In the real world do people hook API's directly to HBase or is there some
> caching layer that is used?
> 3. I remember hearing people like StumbleUpon use different clusters for
> analytics vs customer apps. Is this still best practice?
> 4. Anyone using MSLAB's to reduce GC pauses in production? Experiences /
> landmines?
> 5. What other considerations have you found when hooking HBase up for
> user-facing applications?
>
> Thanks in advance and I'd love to hear some bragging!
>
> -Matt
>
Re: A question about unit test
Posted by Gaojinchao <ga...@huawei.com>.
Unlucky. It can't reproduce. It happened last night.
It seems some threads for freeze or locking.
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2011年5月6日 10:59
收件人: user@hbase.apache.org
抄送: Chenjian
主题: Re: A question about unit test
Can you enable -X switch ?
Also, 1 minute into running the test, you can use jstack to see what the
main thread is doing.
On Thu, May 5, 2011 at 7:47 PM, Gaojinchao <ga...@huawei.com> wrote:
> A test case was timeout.
>
> Who has the experience for this.
> Thanks
>
> <message priority="info"><![CDATA[[exec] Tests run: 3, Failures: 0,
> Errors: 0, Skipped: 0, Time elapsed: 0.101 sec]]></message>
> <message priority="info"><![CDATA[[exec] Running
> org.apache.hadoop.hbase.TestInfoServers]]></message>
> <message priority="info"><![CDATA[[exec]
> killed.]]></message>
> <message priority="info"><![CDATA[[exec] [INFO]
> ------------------------------------------------------------------------]]></message>
> <message priority="info"><![CDATA[[exec] [INFO] BUILD
> FAILURE]]></message>
> <message priority="info"><![CDATA[[exec] [INFO]
> ------------------------------------------------------------------------]]></message>
> <message priority="info"><![CDATA[[exec] [INFO] Total time:
> 1:15:10.277s]]></message>
> <message priority="info"><![CDATA[[exec] [INFO] Finished at:
> Fri May 06 04:52:47 GMT+08:00 2011]]></message>
> <message priority="info"><![CDATA[[exec] [INFO] Final
> Memory: 53M/949M]]></message>
> <message priority="info"><![CDATA[[exec] [INFO]
> ------------------------------------------------------------------------]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] Failed to
> execute goal org.apache.maven.plugins:maven-surefire-plugin:2.4.3:test
> (default-test) on project hbase: Error while executing forked tests.; nested
> exception is
> org.apache.maven.surefire.booter.shade.org.codehaus.plexus.util.cli.CommandLineException:
> Error while executing external command, process killed. Process timeout out
> after 900 seconds -> [Help 1]]]></message>
> <message priority="info"><![CDATA[[exec]
> [ERROR]]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] To see the
> full stack trace of the errors, re-run Maven with the -e
> switch.]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] Re-run
> Maven using the -X switch to enable full debug logging.]]></message>
> <message priority="info"><![CDATA[[exec]
> [ERROR]]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] For more
> information about the errors and possible solutions, please read the
> following articles:]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException]
> ]></message>
> <message priority="info"><![CDATA[[echo] llt
> success.]]></message>
> <message priority="info"><![CDATA[llt.report:]]></message>
>
> The log is half
>
> 2011-05-06 04:38:15,466 DEBUG [main-EventThread] zookeeper.ZKUtil(1107):
> master:56634-0x12fc1e1c2fa0000 Retrieved 54 byte(s) of data from znode
> /hbase/unassigned/1028785192 and set watcher; region=.META.,,1,
> server=linux1.site,48768,1304627889113, state=RS_ZK_REGION_OPENING
> 2011-05-06 04:38:15,466 DEBUG [main-EventThread]
> master.AssignmentManager(371): Handling transition=RS_ZK_REGION_OPENING,
> server=linux1.site,48768,13046278891 // The log is half
>
Re: A question about unit test
Posted by Ted Yu <yu...@gmail.com>.
Can you enable -X switch ?
Also, 1 minute into running the test, you can use jstack to see what the
main thread is doing.
On Thu, May 5, 2011 at 7:47 PM, Gaojinchao <ga...@huawei.com> wrote:
> A test case was timeout.
>
> Who has the experience for this.
> Thanks
>
> <message priority="info"><![CDATA[[exec] Tests run: 3, Failures: 0,
> Errors: 0, Skipped: 0, Time elapsed: 0.101 sec]]></message>
> <message priority="info"><![CDATA[[exec] Running
> org.apache.hadoop.hbase.TestInfoServers]]></message>
> <message priority="info"><![CDATA[[exec]
> killed.]]></message>
> <message priority="info"><![CDATA[[exec] [INFO]
> ------------------------------------------------------------------------]]></message>
> <message priority="info"><![CDATA[[exec] [INFO] BUILD
> FAILURE]]></message>
> <message priority="info"><![CDATA[[exec] [INFO]
> ------------------------------------------------------------------------]]></message>
> <message priority="info"><![CDATA[[exec] [INFO] Total time:
> 1:15:10.277s]]></message>
> <message priority="info"><![CDATA[[exec] [INFO] Finished at:
> Fri May 06 04:52:47 GMT+08:00 2011]]></message>
> <message priority="info"><![CDATA[[exec] [INFO] Final
> Memory: 53M/949M]]></message>
> <message priority="info"><![CDATA[[exec] [INFO]
> ------------------------------------------------------------------------]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] Failed to
> execute goal org.apache.maven.plugins:maven-surefire-plugin:2.4.3:test
> (default-test) on project hbase: Error while executing forked tests.; nested
> exception is
> org.apache.maven.surefire.booter.shade.org.codehaus.plexus.util.cli.CommandLineException:
> Error while executing external command, process killed. Process timeout out
> after 900 seconds -> [Help 1]]]></message>
> <message priority="info"><![CDATA[[exec]
> [ERROR]]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] To see the
> full stack trace of the errors, re-run Maven with the -e
> switch.]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] Re-run
> Maven using the -X switch to enable full debug logging.]]></message>
> <message priority="info"><![CDATA[[exec]
> [ERROR]]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] For more
> information about the errors and possible solutions, please read the
> following articles:]]></message>
> <message priority="info"><![CDATA[[exec] [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException]
> ]></message>
> <message priority="info"><![CDATA[[echo] llt
> success.]]></message>
> <message priority="info"><![CDATA[llt.report:]]></message>
>
> The log is half
>
> 2011-05-06 04:38:15,466 DEBUG [main-EventThread] zookeeper.ZKUtil(1107):
> master:56634-0x12fc1e1c2fa0000 Retrieved 54 byte(s) of data from znode
> /hbase/unassigned/1028785192 and set watcher; region=.META.,,1,
> server=linux1.site,48768,1304627889113, state=RS_ZK_REGION_OPENING
> 2011-05-06 04:38:15,466 DEBUG [main-EventThread]
> master.AssignmentManager(371): Handling transition=RS_ZK_REGION_OPENING,
> server=linux1.site,48768,13046278891 // The log is half
>
A question about unit test
Posted by Gaojinchao <ga...@huawei.com>.
A test case was timeout.
Who has the experience for this.
Thanks
<message priority="info"><![CDATA[[exec] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.101 sec]]></message>
<message priority="info"><![CDATA[[exec] Running org.apache.hadoop.hbase.TestInfoServers]]></message>
<message priority="info"><![CDATA[[exec] killed.]]></message>
<message priority="info"><![CDATA[[exec] [INFO] ------------------------------------------------------------------------]]></message>
<message priority="info"><![CDATA[[exec] [INFO] BUILD FAILURE]]></message>
<message priority="info"><![CDATA[[exec] [INFO] ------------------------------------------------------------------------]]></message>
<message priority="info"><![CDATA[[exec] [INFO] Total time: 1:15:10.277s]]></message>
<message priority="info"><![CDATA[[exec] [INFO] Finished at: Fri May 06 04:52:47 GMT+08:00 2011]]></message>
<message priority="info"><![CDATA[[exec] [INFO] Final Memory: 53M/949M]]></message>
<message priority="info"><![CDATA[[exec] [INFO] ------------------------------------------------------------------------]]></message>
<message priority="info"><![CDATA[[exec] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.4.3:test (default-test) on project hbase: Error while executing forked tests.; nested exception is org.apache.maven.surefire.booter.shade.org.codehaus.plexus.util.cli.CommandLineException: Error while executing external command, process killed. Process timeout out after 900 seconds -> [Help 1]]]></message>
<message priority="info"><![CDATA[[exec] [ERROR]]]></message>
<message priority="info"><![CDATA[[exec] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.]]></message>
<message priority="info"><![CDATA[[exec] [ERROR] Re-run Maven using the -X switch to enable full debug logging.]]></message>
<message priority="info"><![CDATA[[exec] [ERROR]]]></message>
<message priority="info"><![CDATA[[exec] [ERROR] For more information about the errors and possible solutions, please read the following articles:]]></message>
<message priority="info"><![CDATA[[exec] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException]]></message>
<message priority="info"><![CDATA[[echo] llt success.]]></message>
<message priority="info"><![CDATA[llt.report:]]></message>
The log is half
2011-05-06 04:38:15,466 DEBUG [main-EventThread] zookeeper.ZKUtil(1107): master:56634-0x12fc1e1c2fa0000 Retrieved 54 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, server=linux1.site,48768,1304627889113, state=RS_ZK_REGION_OPENING
2011-05-06 04:38:15,466 DEBUG [main-EventThread] master.AssignmentManager(371): Handling transition=RS_ZK_REGION_OPENING, server=linux1.site,48768,13046278891 // The log is half
Re: Considerations for using HBase in User Facing applications
Posted by Patrick Angeles <pa...@cloudera.com>.
Inline...
On Thu, May 5, 2011 at 5:26 PM, Matt Davies <ma...@mattdavies.net> wrote:
> Afternoon everyone,
>
> I am researching what the best practice is for using HBase in user facing
> applications. I do not know all of the applications that will be ported to
> use HBase, but they do share common characteristics such as
>
> - simple key / value data. Not serving large files ATM. Perhaps a couple
> columns in a single column family
> - very tall tables
> - hundreds of millions of rows
> - need millisecond access times for a single row
> - random access
> - maintain very, very good query times while loading in new data
>
>
> The quick choice would be to use something like memcache or Redis, but the
> data is growing faster than the memory of a single box or even few boxes.
> We also have a significant investment in Hadoop technologies so keeping
> HBase prime seems to make a lot of sense.
>
> So, some questions:
>
> 1. do you find that having a single HBase cluster to serve all applications
> vs smaller clusters to serve application specific data is better?
>
In general, you get higher utilization with the former, but you can better
tune for a specific application with the latter. Also consider that a single
20-node cluster running 2 apps will balance load and tolerate node failures
better than two 10-node clusters running one app each. Another consideration
is upgrades and maintenance. You might have one app out of 10 that needs the
shiny new HBase v1.3.37 hotness, but you'll have to bring them all down if
they're served out of the same cluster.
> 2. In the real world do people hook API's directly to HBase or is there
> some
> caching layer that is used?
>
We see both in the real world. I think unless you a priori have a good idea
of what to cache, it's better to rely on HBase's block cache. One severe
limitation on block cache, though, that is that you can only make it so big,
because large heaps will make you prone to long GC pauses (even with MSLAB,
you will GC eventually). However, the OS does a pretty good job
compensating, because it will use up available memory to cache the
filesystem.
> 3. I remember hearing people like StumbleUpon use different clusters for
> analytics vs customer apps. Is this still best practice?
This is good practice in RDBMS-land, and I think it still holds, assuming
you can afford 2 clusters. The former is read-mostly and the latter is
transactional, and some tuning parameters like memstore/blockcache ratios
are cluster wide.
> 4. Anyone using MSLAB's to reduce GC pauses in production? Experiences /
> landmines?
> 5. What other considerations have you found when hooking HBase up for
> user-facing applications?
>
> Thanks in advance and I'd love to hear some bragging!
>
> -Matt
>