You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Tao Xie <xi...@gmail.com> on 2010/09/10 04:32:35 UTC
ycsb test on hbase
hi, all
I use YCSB to measure the insert/read latency of hbase.
I found there will be 0 records inserted in up to 10 seconds during the
insertion procedure.
See the following result at 1514 second. I want to know why this occurs. Is
this due to compaction?
And I also want to know why the ops/sec varies all the time. Seems no a
stable time.
Thanks.
2010-09-10 00:07:29,608 1484 sec: 28786280 operations; 23475.3 current
ops/sec; [INSERT AverageLatency(ms)=8.81]
2010-09-10 00:07:39,610 1494 sec: 28842632 operations; 5634.07 current
ops/sec; [INSERT AverageLatency(ms)=6.68]
2010-09-10 00:07:49,612 1504 sec: 28964728 operations; 12207.16 current
ops/sec; [INSERT AverageLatency(ms)=7.68]
2010-09-10 00:07:59,614 1514 sec: 28964728 operations; 0 current ops/sec;
2010-09-10 00:08:10,778 1525 sec: 29130475 operations; 14846.56 current
ops/sec; [INSERT AverageLatency(ms)=24.45]
2010-09-10 00:08:20,782 1535 sec: 29606967 operations; 47630.15 current
ops/sec; [INSERT AverageLatency(ms)=12.64]
2010-09-10 00:08:30,784 1545 sec: 29908624 operations; 30159.67 current
ops/sec; [INSERT AverageLatency(ms)=0.12]
2010-09-10 00:08:40,786 1555 sec: 30016632 operations; 10798.64 current
ops/sec; [INSERT AverageLatency(ms)=5.66]
Re: what is the unit of the TTL in hbase and how often hbase remove expired regions?
Posted by Jinsong Hu <ji...@hotmail.com>.
I realized that there is a problem. I read some comments and realized that
the TTL is used when the compaction is executed.
that means if there is some regions that has older data then TTL , and no
new data is written to that region, then the old data will never be removed
, because no compaction will ever happen for those data after they are
stored.
Unfortunately, this is a fairly common situation for log data. Once they are
written, they are seldom touched again, hence there will no compaction
against those data. It would be nice to use TTL to retire them, but because
there is no compaction
for those old regions, the data will never be retired.
I wonder if the above scenario is correct and if it is, is there a solution
for this other than periodically issue a client side
delete request ?
Jimmy
--------------------------------------------------
From: "Jonathan Gray" <jg...@facebook.com>
Sent: Monday, September 13, 2010 6:06 PM
To: <us...@hbase.apache.org>
Subject: RE: what is the unit of the TTL in hbase and how often hbase remove
expired regions?
> The unit is seconds as outlined in HColumnDescriptor. It's a little
> confusing because server-side everything is milliseconds. On the server,
> it is converted from the user-configured seconds to milliseconds.
>
> Also, HBase will never expire "regions", rather it expires individual
> versions of cells according to their timestamps.
>
> This is not enforced periodically, it is actually enforced constantly, so
> you should *never see* an expired cell. This does not mean it does not
> still exist on disk, it means it will not be visible in user queries. On
> a major compaction (default every 24 hours) HBase will actually delete the
> expired cells.
>
> JG
>
>> -----Original Message-----
>> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
>> Sent: Monday, September 13, 2010 3:52 PM
>> To: user@hbase.apache.org
>> Subject: what is the unit of the TTL in hbase and how often hbase
>> remove expired regions?
>>
>> Hi,
>> I want to find out what is the unit for TTL in hbase. I googled
>> around and
>> found some people say it is microsecond.
>> and I thought it was millisecond as that is java default. Then I
>> searched
>> hbase code and saw some test code treating
>> the unit to be seconds.
>> I used a TTL=600000. if the unit is millisecond, then that means 10
>> minute. However, I continue to insert records into
>> this table, and found that the regions older than 10 minutes are not
>> removed.
>> The question I have is , what is the unit for TTL. and the second
>> question
>> is, how often hbase checks all regions and
>> remove expired regions.
>>
>> Jimmy
>
>
RE: what is the unit of the TTL in hbase and how often hbase remove
expired regions?
Posted by Jonathan Gray <jg...@facebook.com>.
The unit is seconds as outlined in HColumnDescriptor. It's a little confusing because server-side everything is milliseconds. On the server, it is converted from the user-configured seconds to milliseconds.
Also, HBase will never expire "regions", rather it expires individual versions of cells according to their timestamps.
This is not enforced periodically, it is actually enforced constantly, so you should *never see* an expired cell. This does not mean it does not still exist on disk, it means it will not be visible in user queries. On a major compaction (default every 24 hours) HBase will actually delete the expired cells.
JG
> -----Original Message-----
> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
> Sent: Monday, September 13, 2010 3:52 PM
> To: user@hbase.apache.org
> Subject: what is the unit of the TTL in hbase and how often hbase
> remove expired regions?
>
> Hi,
> I want to find out what is the unit for TTL in hbase. I googled
> around and
> found some people say it is microsecond.
> and I thought it was millisecond as that is java default. Then I
> searched
> hbase code and saw some test code treating
> the unit to be seconds.
> I used a TTL=600000. if the unit is millisecond, then that means 10
> minute. However, I continue to insert records into
> this table, and found that the regions older than 10 minutes are not
> removed.
> The question I have is , what is the unit for TTL. and the second
> question
> is, how often hbase checks all regions and
> remove expired regions.
>
> Jimmy
what is the unit of the TTL in hbase and how often hbase remove expired regions?
Posted by Jinsong Hu <ji...@hotmail.com>.
Hi,
I want to find out what is the unit for TTL in hbase. I googled around and
found some people say it is microsecond.
and I thought it was millisecond as that is java default. Then I searched
hbase code and saw some test code treating
the unit to be seconds.
I used a TTL=600000. if the unit is millisecond, then that means 10
minute. However, I continue to insert records into
this table, and found that the regions older than 10 minutes are not
removed.
The question I have is , what is the unit for TTL. and the second question
is, how often hbase checks all regions and
remove expired regions.
Jimmy
Re: ycsb test on hbase
Posted by Todd Lipcon <to...@cloudera.com>.
Opened HBASE-2981
On Fri, Sep 10, 2010 at 1:27 AM, Jeff Hammerbacher <ha...@cloudera.com>wrote:
> > We've been brainstorming some ideas to "smooth out" these performance
> > lapses, so instead of getting a 10 second period of unavailability, you
> get
> > a 30 second period of slower performance, which is usually preferable.
> >
>
> Where is this brainstorming taking place? Could we open a JIRA issue to
> capture the brainstorming in public and searchable fashion?
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: ycsb test on hbase
Posted by Jeff Whiting <je...@qualtrics.com>.
I've updated the jira with a patch to trunk a few days ago. Feel free to try it out if you want.
~Jeff
On 9/10/2010 10:44 AM, Jeff Whiting wrote:
> We were having the exact same problem when we were doing our own load testing with hbase. We
> found that a memstore would reach its hbase.hstore.blockingStoreFiles limit or its
> hbase.hregion.memstore.block.multiplier. Hitting either of those limits prevents writes to the
> specific region and the client would have to pause until a compaction could come through and clean
> stuff up.
>
> However the biggest problem is that there would be a descent size compaction queue, we'd hit one
> of those limits, and then get put on the *back* of the queue and would have to wait *minutes*
> before it finally got to do the compaction we needed to stop the blocking. I created a jira to
> address the issue HBASE-2646. There is a patch in the jira for 0.20.4 that creates a priority
> compaction queue that greatly helped our problem. In fact we saw little to no pausing after
> applying the patch. In the comments of the jira you can see some of the settings we used to
> prevent the problem without the patch.
>
> Apparently there is some work going on to do concurrent priority compaction (Jonathan Gray has
> been working on it) but I haven't seen anything yet in hbase and don't know the time line. My
> personal opinion is that we should integrate the patch into trunk and use it until the more
> advanced compactions are implemented.
>
> ~Jeff
>
> On 9/10/2010 2:27 AM, Jeff Hammerbacher wrote:
>>> We've been brainstorming some ideas to "smooth out" these performance
>>> lapses, so instead of getting a 10 second period of unavailability, you get
>>> a 30 second period of slower performance, which is usually preferable.
>>>
>> Where is this brainstorming taking place? Could we open a JIRA issue to
>> capture the brainstorming in public and searchable fashion?
>>
>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
master shows the region but regionserver doesn't show. Why can this happen ?
Posted by Jinsong Hu <ji...@hotmail.com>.
Hi,
I have noticed that my hbase client is complaining that
Trying to contact region server m0002036.ppops.net:60020 for region
HEARTBEAT_CLUSTER,,1279930490584.a84b7160318e91ef51c7efd215cd7e46., row
'time\x09weekly\x092010W30', but failed after 10 attempts.
I then go to the hbase master web page and it shows that this region is
there and served by one of the region server. however,
I clicked on the link for that regionserver and found that this region is
not listed on the served regions. I then searched the log on this region
server and I found the regionserver actually closed this region:
2010-09-13 13:16:58,602 INFO org.apache.hadoop.hbase.regionserver.Store:
Complet
ed major compaction of 4 file(s) in fields of
HEARTBEAT_CLUSTER,,1279930490584.a
84b7160318e91ef51c7efd215cd7e46.; new storefile is
hdfs://d-namenodes1.cloud.ppo
ps.net:8020/hbase/HEARTBEAT_CLUSTER/a84b7160318e91ef51c7efd215cd7e46/fields/8534
678064854849666; store size is 102.6m
2010-09-13 13:16:58,602 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compa
ction completed on region
HEARTBEAT_CLUSTER,,1279930490584.a84b7160318e91ef51c7e
fd215cd7e46. in 46sec
2010-09-13 16:24:31,981 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer
: Closing region
HEARTBEAT_CLUSTER,,1279930490584.a84b7160318e91ef51c7efd215cd7e
46.
2010-09-13 16:24:31,982 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Clos
ing HEARTBEAT_CLUSTER,,1279930490584.a84b7160318e91ef51c7efd215cd7e46.:
disablin
g compactions & flushes
2010-09-13 16:24:32,028 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Upda
tes disabled for region, no outstanding scanners on
HEARTBEAT_CLUSTER,,127993049
0584.a84b7160318e91ef51c7efd215cd7e46.
2010-09-13 16:24:32,031 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
No m
ore row locks outstanding on region
HEARTBEAT_CLUSTER,,1279930490584.a84b7160318
e91ef51c7efd215cd7e46.
2010-09-13 16:24:32,057 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Close
d HEARTBEAT_CLUSTER,,1279930490584.a84b7160318e91ef51c7efd215cd7e46.
That means nobody can access this table any more. How could this happen ? I
checked HDFS and confirmed the files for this
table is there. I am using the CDH3 beta 2 hbase 0.89 version. I end up
having to restart all regionservers to get this region online.
Jimmy.
RE: ycsb test on hbase
Posted by Jonathan Gray <jg...@facebook.com>.
Prioritized compactions and multi-threaded compactions are being addressed with the move to the ExecutorService/EventHandler way of doing things.
Unfortunately stabilizing the new master (and fixing up brokenness in old master) has taken priority over that task.
I would say there's a 50% chance that stuff gets in to 0.90 and a 90% chance it is done by the end of October.
Part of the issue is that we need to fix other stuff at the same time like the issues addressed in HBASE-2375. It's feasible to just add prioritization to compactions and I'm going to race to implement that in late September if I don't have time to address the larger issues at hand.
If anyone wants to work with me on this stuff and has time over the next few weeks, let me know and we can work together on it.
JG
> -----Original Message-----
> From: Jinsong Hu [mailto:jinsong_hu@hotmail.com]
> Sent: Friday, September 10, 2010 10:50 AM
> To: user@hbase.apache.org
> Subject: Re: ycsb test on hbase
>
> When I did the test, I found the CPU went to 100% for the compaction
> thread.
> while other cores in the
> OS is not fully utilized. I checked the hbase code and found there is
> only
> one single thread that does
> compaction. That needs to be changed. I found that 0.21 version of
> hbase
> does have this plan to
> use multiple thread to do so, but it appears it will take a long time
> for us
> to get there.
>
> Another issue that I found with hbase is that when the number of region
> reaches around 1000 for each
> regionserver, the regionserver begin to shutdown by itself for various
> reasons. most of the time
> IO issue with data node. occasionally session expiration issue with
> zookeeper. This is true regardless
> of what key I use for the table.
>
> Jimmy.
>
> --------------------------------------------------
> From: "Jeff Whiting" <je...@qualtrics.com>
> Sent: Friday, September 10, 2010 9:44 AM
> To: <us...@hbase.apache.org>
> Subject: Re: ycsb test on hbase
>
> > We were having the exact same problem when we were doing our own
> load
> > testing with hbase. We found that a memstore would reach its
> > hbase.hstore.blockingStoreFiles limit or its
> > hbase.hregion.memstore.block.multiplier. Hitting either of those
> limits
> > prevents writes to the specific region and the client would have to
> pause
> > until a compaction could come through and clean stuff up.
> >
> > However the biggest problem is that there would be a descent size
> > compaction queue, we'd hit one of those limits, and then get put on
> the
> > *back* of the queue and would have to wait *minutes* before it
> finally got
> > to do the compaction we needed to stop the blocking. I created a jira
> to
> > address the issue HBASE-2646. There is a patch in the jira for
> 0.20.4
> > that creates a priority compaction queue that greatly helped our
> problem.
> > In fact we saw little to no pausing after applying the patch. In the
> > comments of the jira you can see some of the settings we used to
> prevent
> > the problem without the patch.
> >
> > Apparently there is some work going on to do concurrent priority
> > compaction (Jonathan Gray has been working on it) but I haven't seen
> > anything yet in hbase and don't know the time line. My personal
> opinion
> > is that we should integrate the patch into trunk and use it until the
> more
> > advanced compactions are implemented.
> >
> > ~Jeff
> >
> > On 9/10/2010 2:27 AM, Jeff Hammerbacher wrote:
> >>> We've been brainstorming some ideas to "smooth out" these
> performance
> >>> lapses, so instead of getting a 10 second period of unavailability,
> you
> >>> get
> >>> a 30 second period of slower performance, which is usually
> preferable.
> >>>
> >> Where is this brainstorming taking place? Could we open a JIRA issue
> to
> >> capture the brainstorming in public and searchable fashion?
> >>
> >
> > --
> > Jeff Whiting
> > Qualtrics Senior Software Engineer
> > jeffw@qualtrics.com
> >
> >
Re: ycsb test on hbase
Posted by Jinsong Hu <ji...@hotmail.com>.
When I did the test, I found the CPU went to 100% for the compaction thread.
while other cores in the
OS is not fully utilized. I checked the hbase code and found there is only
one single thread that does
compaction. That needs to be changed. I found that 0.21 version of hbase
does have this plan to
use multiple thread to do so, but it appears it will take a long time for us
to get there.
Another issue that I found with hbase is that when the number of region
reaches around 1000 for each
regionserver, the regionserver begin to shutdown by itself for various
reasons. most of the time
IO issue with data node. occasionally session expiration issue with
zookeeper. This is true regardless
of what key I use for the table.
Jimmy.
--------------------------------------------------
From: "Jeff Whiting" <je...@qualtrics.com>
Sent: Friday, September 10, 2010 9:44 AM
To: <us...@hbase.apache.org>
Subject: Re: ycsb test on hbase
> We were having the exact same problem when we were doing our own load
> testing with hbase. We found that a memstore would reach its
> hbase.hstore.blockingStoreFiles limit or its
> hbase.hregion.memstore.block.multiplier. Hitting either of those limits
> prevents writes to the specific region and the client would have to pause
> until a compaction could come through and clean stuff up.
>
> However the biggest problem is that there would be a descent size
> compaction queue, we'd hit one of those limits, and then get put on the
> *back* of the queue and would have to wait *minutes* before it finally got
> to do the compaction we needed to stop the blocking. I created a jira to
> address the issue HBASE-2646. There is a patch in the jira for 0.20.4
> that creates a priority compaction queue that greatly helped our problem.
> In fact we saw little to no pausing after applying the patch. In the
> comments of the jira you can see some of the settings we used to prevent
> the problem without the patch.
>
> Apparently there is some work going on to do concurrent priority
> compaction (Jonathan Gray has been working on it) but I haven't seen
> anything yet in hbase and don't know the time line. My personal opinion
> is that we should integrate the patch into trunk and use it until the more
> advanced compactions are implemented.
>
> ~Jeff
>
> On 9/10/2010 2:27 AM, Jeff Hammerbacher wrote:
>>> We've been brainstorming some ideas to "smooth out" these performance
>>> lapses, so instead of getting a 10 second period of unavailability, you
>>> get
>>> a 30 second period of slower performance, which is usually preferable.
>>>
>> Where is this brainstorming taking place? Could we open a JIRA issue to
>> capture the brainstorming in public and searchable fashion?
>>
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
>
Re: ycsb test on hbase
Posted by Jeff Whiting <je...@qualtrics.com>.
We were having the exact same problem when we were doing our own load testing with hbase. We
found that a memstore would reach its hbase.hstore.blockingStoreFiles limit or its
hbase.hregion.memstore.block.multiplier. Hitting either of those limits prevents writes to the
specific region and the client would have to pause until a compaction could come through and clean
stuff up.
However the biggest problem is that there would be a descent size compaction queue, we'd hit one of
those limits, and then get put on the *back* of the queue and would have to wait *minutes* before it
finally got to do the compaction we needed to stop the blocking. I created a jira to address the
issue HBASE-2646. There is a patch in the jira for 0.20.4 that creates a priority compaction queue
that greatly helped our problem. In fact we saw little to no pausing after applying the patch. In
the comments of the jira you can see some of the settings we used to prevent the problem without the
patch.
Apparently there is some work going on to do concurrent priority compaction (Jonathan Gray has been
working on it) but I haven't seen anything yet in hbase and don't know the time line. My personal
opinion is that we should integrate the patch into trunk and use it until the more advanced
compactions are implemented.
~Jeff
On 9/10/2010 2:27 AM, Jeff Hammerbacher wrote:
>> We've been brainstorming some ideas to "smooth out" these performance
>> lapses, so instead of getting a 10 second period of unavailability, you get
>> a 30 second period of slower performance, which is usually preferable.
>>
> Where is this brainstorming taking place? Could we open a JIRA issue to
> capture the brainstorming in public and searchable fashion?
>
--
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com
Re: ycsb test on hbase
Posted by Jeff Hammerbacher <ha...@cloudera.com>.
> We've been brainstorming some ideas to "smooth out" these performance
> lapses, so instead of getting a 10 second period of unavailability, you get
> a 30 second period of slower performance, which is usually preferable.
>
Where is this brainstorming taking place? Could we open a JIRA issue to
capture the brainstorming in public and searchable fashion?
Re: ycsb test on hbase
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Tao,
Your results match up with what I see running YCSB as well.
The lapse in write throughput happens when one of the regions is splitting
or has reached its "blocking" threshold. Essentially there's a short period
of time when it can't take writes, and thus clients will back up waiting on
it. Because YCSB is doing entirely random writes, after a couple of seconds
all of the threads will have hit the "stalled" region and throughput will
drop to 0 until that region becomes available again.
We've been brainstorming some ideas to "smooth out" these performance
lapses, so instead of getting a 10 second period of unavailability, you get
a 30 second period of slower performance, which is usually preferable.
Thanks
-Todd
On Thu, Sep 9, 2010 at 10:04 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Is there anything specific you'd like to work on or improve? I see
> numbers, but that doesn't really mean anything. For example, do you
> have more heap that the size of those regions? If not, it may be
> garbage collecting like mad? Are the machines swapping? Is the CPU
> even contented? How many machines do you have? (please don't answer,
> those are just examples of reasons why a back and forth on the mailing
> list is probably going to take a long time and get us nowhere)
>
> If you are just trying to get better throughput for the sake of it,
> then you should probably look around the documentation (like the FAQ)
> and search the mailing for advices. Look at what's happening in the
> region server logs and try to see where it's blocking, it's usually
> pretty obvious. Also try using the most recent HBase version.
>
> If you're trying to see how fast you can import your data set
> (providing that you have one), then instead of a 100% writes YCSB I
> would recommend instead using HFOF:
> http://hbase.apache.org/docs/r0.89.20100726/bulk-loads.html
>
> J-D
>
> On Thu, Sep 9, 2010 at 9:47 PM, Tao Xie <xi...@gmail.com> wrote:
> > You mean "hbase.hregion.max.filesize", I set it to 2G. But I see more
> > waiting time and lower ops. I have no real use case for intensive write.
> I
> > just use YCSB to do a performance test.
> >
> > 2010-09-10 11:20:42,796 332 sec: 2437024 operations; 7509.85 current
> > ops/sec; [INSERT AverageLatency(ms)=35.09]
> > 2010-09-10 11:20:52,801 342 sec: 2437024 operations; 0 current ops/sec;
> > 2010-09-10 11:21:02,806 352 sec: 2441720 operations; 469.37 current
> > ops/sec; [INSERT AverageLatency(ms)=21.37]
> > 2010-09-10 11:21:12,807 362 sec: 2451112 operations; 939.11 current
> > ops/sec; [INSERT AverageLatency(ms)=40.12]
> > 2010-09-10 11:21:22,809 372 sec: 2569586 operations; 11845.03 current
> > ops/sec; [INSERT AverageLatency(ms)=31.16]
> > 2010-09-10 11:21:32,811 382 sec: 2714088 operations; 14447.31 current
> > ops/sec; [INSERT AverageLatency(ms)=33.44]
> > 2010-09-10 11:21:42,814 392 sec: 2718784 operations; 469.46 current
> > ops/sec; [INSERT AverageLatency(ms)=34.64]
> > 2010-09-10 11:21:52,815 402 sec: 2779832 operations; 6104.19 current
> > ops/sec; [INSERT AverageLatency(ms)=36.46]
> > 2010-09-10 11:22:02,817 412 sec: 2930104 operations; 15024.2 current
> > ops/sec; [INSERT AverageLatency(ms)=38.77]
> > 2010-09-10 11:22:12,819 422 sec: 3009936 operations; 7981.6 current
> > ops/sec; [INSERT AverageLatency(ms)=43.41]
> > 2010-09-10 11:22:22,821 432 sec: 3009936 operations; 0 current ops/sec;
> > 2010-09-10 11:22:32,823 442 sec: 3009936 operations; 0 current ops/sec;
> > 2010-09-10 11:22:42,825 452 sec: 3144094 operations; 13413.12 current
> > ops/sec; [INSERT AverageLatency(ms)=56.82]
> > 2010-09-10 11:22:52,827 462 sec: 3310480 operations; 16635.27 current
> > ops/sec; [INSERT AverageLatency(ms)=34.46]
> > 2010-09-10 11:23:02,829 472 sec: 3338656 operations; 2817.04 current
> > ops/sec; [INSERT AverageLatency(ms)=20.91]
> > 2010-09-10 11:23:12,831 482 sec: 3338656 operations; 0 current ops/sec;
> > 2010-09-10 11:23:22,832 492 sec: 3438535 operations; 9986.9 current
> > ops/sec; [INSERT AverageLatency(ms)=26.74]
> > 2010-09-10 11:23:35,600 505 sec: 3566729 operations; 10040.26 current
> > ops/sec; [INSERT AverageLatency(ms)=27.53]
> > 2010-09-10 11:23:45,601 515 sec: 3620416 operations; 5368.16 current
> > ops/sec; [INSERT AverageLatency(ms)=48.66]
> > 2010-09-10 11:23:55,603 525 sec: 3620416 operations; 0 current ops/sec;
> > 2010-09-10 11:24:05,605 535 sec: 3620416 operations; 0 current ops/sec;
> > 2010-09-10 11:24:15,607 545 sec: 3648592 operations; 2817.04 current
> > ops/sec; [INSERT AverageLatency(ms)=52.15]
> >
> > 2010/9/10 Jean-Daniel Cryans <jd...@apache.org>
> >
> >> If you have a very heavy write load (like YCSB when only inserting),
> >> then you really have to tune HBase for that kind of workload since
> >> it's not the "normal" use case. Setting MAX_FILESIZE really high
> >> (1-2GB) and even pre-splitting the table when creating it (available
> >> in 0.89) will help.
> >>
> >> Most of the time spent waiting is due to splitting and blocking due to
> >> either MemStores growing over their max size and the global MemStore
> >> size limit being reached. It's kinda rough and could probably be
> >> "smoother", but do you really have a use case that requires it or just
> >> poking?
> >>
> >> J-D
> >>
> >> On Thu, Sep 9, 2010 at 7:32 PM, Tao Xie <xi...@gmail.com>
> wrote:
> >> > hi, all
> >> > I use YCSB to measure the insert/read latency of hbase.
> >> > I found there will be 0 records inserted in up to 10 seconds during
> the
> >> > insertion procedure.
> >> > See the following result at 1514 second. I want to know why this
> occurs.
> >> Is
> >> > this due to compaction?
> >> > And I also want to know why the ops/sec varies all the time. Seems no
> a
> >> > stable time.
> >> > Thanks.
> >> >
> >> > 2010-09-10 00:07:29,608 1484 sec: 28786280 operations; 23475.3
> current
> >> > ops/sec; [INSERT AverageLatency(ms)=8.81]
> >> > 2010-09-10 00:07:39,610 1494 sec: 28842632 operations; 5634.07
> current
> >> > ops/sec; [INSERT AverageLatency(ms)=6.68]
> >> > 2010-09-10 00:07:49,612 1504 sec: 28964728 operations; 12207.16
> current
> >> > ops/sec; [INSERT AverageLatency(ms)=7.68]
> >> > 2010-09-10 00:07:59,614 1514 sec: 28964728 operations; 0 current
> >> ops/sec;
> >> > 2010-09-10 00:08:10,778 1525 sec: 29130475 operations; 14846.56
> current
> >> > ops/sec; [INSERT AverageLatency(ms)=24.45]
> >> > 2010-09-10 00:08:20,782 1535 sec: 29606967 operations; 47630.15
> current
> >> > ops/sec; [INSERT AverageLatency(ms)=12.64]
> >> > 2010-09-10 00:08:30,784 1545 sec: 29908624 operations; 30159.67
> current
> >> > ops/sec; [INSERT AverageLatency(ms)=0.12]
> >> > 2010-09-10 00:08:40,786 1555 sec: 30016632 operations; 10798.64
> current
> >> > ops/sec; [INSERT AverageLatency(ms)=5.66]
> >> >
> >>
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: ycsb test on hbase
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Is there anything specific you'd like to work on or improve? I see
numbers, but that doesn't really mean anything. For example, do you
have more heap that the size of those regions? If not, it may be
garbage collecting like mad? Are the machines swapping? Is the CPU
even contented? How many machines do you have? (please don't answer,
those are just examples of reasons why a back and forth on the mailing
list is probably going to take a long time and get us nowhere)
If you are just trying to get better throughput for the sake of it,
then you should probably look around the documentation (like the FAQ)
and search the mailing for advices. Look at what's happening in the
region server logs and try to see where it's blocking, it's usually
pretty obvious. Also try using the most recent HBase version.
If you're trying to see how fast you can import your data set
(providing that you have one), then instead of a 100% writes YCSB I
would recommend instead using HFOF:
http://hbase.apache.org/docs/r0.89.20100726/bulk-loads.html
J-D
On Thu, Sep 9, 2010 at 9:47 PM, Tao Xie <xi...@gmail.com> wrote:
> You mean "hbase.hregion.max.filesize", I set it to 2G. But I see more
> waiting time and lower ops. I have no real use case for intensive write. I
> just use YCSB to do a performance test.
>
> 2010-09-10 11:20:42,796 332 sec: 2437024 operations; 7509.85 current
> ops/sec; [INSERT AverageLatency(ms)=35.09]
> 2010-09-10 11:20:52,801 342 sec: 2437024 operations; 0 current ops/sec;
> 2010-09-10 11:21:02,806 352 sec: 2441720 operations; 469.37 current
> ops/sec; [INSERT AverageLatency(ms)=21.37]
> 2010-09-10 11:21:12,807 362 sec: 2451112 operations; 939.11 current
> ops/sec; [INSERT AverageLatency(ms)=40.12]
> 2010-09-10 11:21:22,809 372 sec: 2569586 operations; 11845.03 current
> ops/sec; [INSERT AverageLatency(ms)=31.16]
> 2010-09-10 11:21:32,811 382 sec: 2714088 operations; 14447.31 current
> ops/sec; [INSERT AverageLatency(ms)=33.44]
> 2010-09-10 11:21:42,814 392 sec: 2718784 operations; 469.46 current
> ops/sec; [INSERT AverageLatency(ms)=34.64]
> 2010-09-10 11:21:52,815 402 sec: 2779832 operations; 6104.19 current
> ops/sec; [INSERT AverageLatency(ms)=36.46]
> 2010-09-10 11:22:02,817 412 sec: 2930104 operations; 15024.2 current
> ops/sec; [INSERT AverageLatency(ms)=38.77]
> 2010-09-10 11:22:12,819 422 sec: 3009936 operations; 7981.6 current
> ops/sec; [INSERT AverageLatency(ms)=43.41]
> 2010-09-10 11:22:22,821 432 sec: 3009936 operations; 0 current ops/sec;
> 2010-09-10 11:22:32,823 442 sec: 3009936 operations; 0 current ops/sec;
> 2010-09-10 11:22:42,825 452 sec: 3144094 operations; 13413.12 current
> ops/sec; [INSERT AverageLatency(ms)=56.82]
> 2010-09-10 11:22:52,827 462 sec: 3310480 operations; 16635.27 current
> ops/sec; [INSERT AverageLatency(ms)=34.46]
> 2010-09-10 11:23:02,829 472 sec: 3338656 operations; 2817.04 current
> ops/sec; [INSERT AverageLatency(ms)=20.91]
> 2010-09-10 11:23:12,831 482 sec: 3338656 operations; 0 current ops/sec;
> 2010-09-10 11:23:22,832 492 sec: 3438535 operations; 9986.9 current
> ops/sec; [INSERT AverageLatency(ms)=26.74]
> 2010-09-10 11:23:35,600 505 sec: 3566729 operations; 10040.26 current
> ops/sec; [INSERT AverageLatency(ms)=27.53]
> 2010-09-10 11:23:45,601 515 sec: 3620416 operations; 5368.16 current
> ops/sec; [INSERT AverageLatency(ms)=48.66]
> 2010-09-10 11:23:55,603 525 sec: 3620416 operations; 0 current ops/sec;
> 2010-09-10 11:24:05,605 535 sec: 3620416 operations; 0 current ops/sec;
> 2010-09-10 11:24:15,607 545 sec: 3648592 operations; 2817.04 current
> ops/sec; [INSERT AverageLatency(ms)=52.15]
>
> 2010/9/10 Jean-Daniel Cryans <jd...@apache.org>
>
>> If you have a very heavy write load (like YCSB when only inserting),
>> then you really have to tune HBase for that kind of workload since
>> it's not the "normal" use case. Setting MAX_FILESIZE really high
>> (1-2GB) and even pre-splitting the table when creating it (available
>> in 0.89) will help.
>>
>> Most of the time spent waiting is due to splitting and blocking due to
>> either MemStores growing over their max size and the global MemStore
>> size limit being reached. It's kinda rough and could probably be
>> "smoother", but do you really have a use case that requires it or just
>> poking?
>>
>> J-D
>>
>> On Thu, Sep 9, 2010 at 7:32 PM, Tao Xie <xi...@gmail.com> wrote:
>> > hi, all
>> > I use YCSB to measure the insert/read latency of hbase.
>> > I found there will be 0 records inserted in up to 10 seconds during the
>> > insertion procedure.
>> > See the following result at 1514 second. I want to know why this occurs.
>> Is
>> > this due to compaction?
>> > And I also want to know why the ops/sec varies all the time. Seems no a
>> > stable time.
>> > Thanks.
>> >
>> > 2010-09-10 00:07:29,608 1484 sec: 28786280 operations; 23475.3 current
>> > ops/sec; [INSERT AverageLatency(ms)=8.81]
>> > 2010-09-10 00:07:39,610 1494 sec: 28842632 operations; 5634.07 current
>> > ops/sec; [INSERT AverageLatency(ms)=6.68]
>> > 2010-09-10 00:07:49,612 1504 sec: 28964728 operations; 12207.16 current
>> > ops/sec; [INSERT AverageLatency(ms)=7.68]
>> > 2010-09-10 00:07:59,614 1514 sec: 28964728 operations; 0 current
>> ops/sec;
>> > 2010-09-10 00:08:10,778 1525 sec: 29130475 operations; 14846.56 current
>> > ops/sec; [INSERT AverageLatency(ms)=24.45]
>> > 2010-09-10 00:08:20,782 1535 sec: 29606967 operations; 47630.15 current
>> > ops/sec; [INSERT AverageLatency(ms)=12.64]
>> > 2010-09-10 00:08:30,784 1545 sec: 29908624 operations; 30159.67 current
>> > ops/sec; [INSERT AverageLatency(ms)=0.12]
>> > 2010-09-10 00:08:40,786 1555 sec: 30016632 operations; 10798.64 current
>> > ops/sec; [INSERT AverageLatency(ms)=5.66]
>> >
>>
>
Re: ycsb test on hbase
Posted by Tao Xie <xi...@gmail.com>.
You mean "hbase.hregion.max.filesize", I set it to 2G. But I see more
waiting time and lower ops. I have no real use case for intensive write. I
just use YCSB to do a performance test.
2010-09-10 11:20:42,796 332 sec: 2437024 operations; 7509.85 current
ops/sec; [INSERT AverageLatency(ms)=35.09]
2010-09-10 11:20:52,801 342 sec: 2437024 operations; 0 current ops/sec;
2010-09-10 11:21:02,806 352 sec: 2441720 operations; 469.37 current
ops/sec; [INSERT AverageLatency(ms)=21.37]
2010-09-10 11:21:12,807 362 sec: 2451112 operations; 939.11 current
ops/sec; [INSERT AverageLatency(ms)=40.12]
2010-09-10 11:21:22,809 372 sec: 2569586 operations; 11845.03 current
ops/sec; [INSERT AverageLatency(ms)=31.16]
2010-09-10 11:21:32,811 382 sec: 2714088 operations; 14447.31 current
ops/sec; [INSERT AverageLatency(ms)=33.44]
2010-09-10 11:21:42,814 392 sec: 2718784 operations; 469.46 current
ops/sec; [INSERT AverageLatency(ms)=34.64]
2010-09-10 11:21:52,815 402 sec: 2779832 operations; 6104.19 current
ops/sec; [INSERT AverageLatency(ms)=36.46]
2010-09-10 11:22:02,817 412 sec: 2930104 operations; 15024.2 current
ops/sec; [INSERT AverageLatency(ms)=38.77]
2010-09-10 11:22:12,819 422 sec: 3009936 operations; 7981.6 current
ops/sec; [INSERT AverageLatency(ms)=43.41]
2010-09-10 11:22:22,821 432 sec: 3009936 operations; 0 current ops/sec;
2010-09-10 11:22:32,823 442 sec: 3009936 operations; 0 current ops/sec;
2010-09-10 11:22:42,825 452 sec: 3144094 operations; 13413.12 current
ops/sec; [INSERT AverageLatency(ms)=56.82]
2010-09-10 11:22:52,827 462 sec: 3310480 operations; 16635.27 current
ops/sec; [INSERT AverageLatency(ms)=34.46]
2010-09-10 11:23:02,829 472 sec: 3338656 operations; 2817.04 current
ops/sec; [INSERT AverageLatency(ms)=20.91]
2010-09-10 11:23:12,831 482 sec: 3338656 operations; 0 current ops/sec;
2010-09-10 11:23:22,832 492 sec: 3438535 operations; 9986.9 current
ops/sec; [INSERT AverageLatency(ms)=26.74]
2010-09-10 11:23:35,600 505 sec: 3566729 operations; 10040.26 current
ops/sec; [INSERT AverageLatency(ms)=27.53]
2010-09-10 11:23:45,601 515 sec: 3620416 operations; 5368.16 current
ops/sec; [INSERT AverageLatency(ms)=48.66]
2010-09-10 11:23:55,603 525 sec: 3620416 operations; 0 current ops/sec;
2010-09-10 11:24:05,605 535 sec: 3620416 operations; 0 current ops/sec;
2010-09-10 11:24:15,607 545 sec: 3648592 operations; 2817.04 current
ops/sec; [INSERT AverageLatency(ms)=52.15]
2010/9/10 Jean-Daniel Cryans <jd...@apache.org>
> If you have a very heavy write load (like YCSB when only inserting),
> then you really have to tune HBase for that kind of workload since
> it's not the "normal" use case. Setting MAX_FILESIZE really high
> (1-2GB) and even pre-splitting the table when creating it (available
> in 0.89) will help.
>
> Most of the time spent waiting is due to splitting and blocking due to
> either MemStores growing over their max size and the global MemStore
> size limit being reached. It's kinda rough and could probably be
> "smoother", but do you really have a use case that requires it or just
> poking?
>
> J-D
>
> On Thu, Sep 9, 2010 at 7:32 PM, Tao Xie <xi...@gmail.com> wrote:
> > hi, all
> > I use YCSB to measure the insert/read latency of hbase.
> > I found there will be 0 records inserted in up to 10 seconds during the
> > insertion procedure.
> > See the following result at 1514 second. I want to know why this occurs.
> Is
> > this due to compaction?
> > And I also want to know why the ops/sec varies all the time. Seems no a
> > stable time.
> > Thanks.
> >
> > 2010-09-10 00:07:29,608 1484 sec: 28786280 operations; 23475.3 current
> > ops/sec; [INSERT AverageLatency(ms)=8.81]
> > 2010-09-10 00:07:39,610 1494 sec: 28842632 operations; 5634.07 current
> > ops/sec; [INSERT AverageLatency(ms)=6.68]
> > 2010-09-10 00:07:49,612 1504 sec: 28964728 operations; 12207.16 current
> > ops/sec; [INSERT AverageLatency(ms)=7.68]
> > 2010-09-10 00:07:59,614 1514 sec: 28964728 operations; 0 current
> ops/sec;
> > 2010-09-10 00:08:10,778 1525 sec: 29130475 operations; 14846.56 current
> > ops/sec; [INSERT AverageLatency(ms)=24.45]
> > 2010-09-10 00:08:20,782 1535 sec: 29606967 operations; 47630.15 current
> > ops/sec; [INSERT AverageLatency(ms)=12.64]
> > 2010-09-10 00:08:30,784 1545 sec: 29908624 operations; 30159.67 current
> > ops/sec; [INSERT AverageLatency(ms)=0.12]
> > 2010-09-10 00:08:40,786 1555 sec: 30016632 operations; 10798.64 current
> > ops/sec; [INSERT AverageLatency(ms)=5.66]
> >
>
Re: ycsb test on hbase
Posted by Jean-Daniel Cryans <jd...@apache.org>.
If you have a very heavy write load (like YCSB when only inserting),
then you really have to tune HBase for that kind of workload since
it's not the "normal" use case. Setting MAX_FILESIZE really high
(1-2GB) and even pre-splitting the table when creating it (available
in 0.89) will help.
Most of the time spent waiting is due to splitting and blocking due to
either MemStores growing over their max size and the global MemStore
size limit being reached. It's kinda rough and could probably be
"smoother", but do you really have a use case that requires it or just
poking?
J-D
On Thu, Sep 9, 2010 at 7:32 PM, Tao Xie <xi...@gmail.com> wrote:
> hi, all
> I use YCSB to measure the insert/read latency of hbase.
> I found there will be 0 records inserted in up to 10 seconds during the
> insertion procedure.
> See the following result at 1514 second. I want to know why this occurs. Is
> this due to compaction?
> And I also want to know why the ops/sec varies all the time. Seems no a
> stable time.
> Thanks.
>
> 2010-09-10 00:07:29,608 1484 sec: 28786280 operations; 23475.3 current
> ops/sec; [INSERT AverageLatency(ms)=8.81]
> 2010-09-10 00:07:39,610 1494 sec: 28842632 operations; 5634.07 current
> ops/sec; [INSERT AverageLatency(ms)=6.68]
> 2010-09-10 00:07:49,612 1504 sec: 28964728 operations; 12207.16 current
> ops/sec; [INSERT AverageLatency(ms)=7.68]
> 2010-09-10 00:07:59,614 1514 sec: 28964728 operations; 0 current ops/sec;
> 2010-09-10 00:08:10,778 1525 sec: 29130475 operations; 14846.56 current
> ops/sec; [INSERT AverageLatency(ms)=24.45]
> 2010-09-10 00:08:20,782 1535 sec: 29606967 operations; 47630.15 current
> ops/sec; [INSERT AverageLatency(ms)=12.64]
> 2010-09-10 00:08:30,784 1545 sec: 29908624 operations; 30159.67 current
> ops/sec; [INSERT AverageLatency(ms)=0.12]
> 2010-09-10 00:08:40,786 1555 sec: 30016632 operations; 10798.64 current
> ops/sec; [INSERT AverageLatency(ms)=5.66]
>