You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2009/03/20 20:16:50 UTC
[jira] Commented: (HBASE-1008) [performance] The replay of logs on
server crash takes way too long
[ https://issues.apache.org/jira/browse/HBASE-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683995#action_12683995 ]
Jean-Daniel Cryans commented on HBASE-1008:
-------------------------------------------
A multi-threaded version that I run at openplaces was able to process 33 logs in a "record" time and our job didn't even failed like it usually does :
{quote}
2009-03-20 15:06:24,047 INFO org.apache.hadoop.hbase.regionserver.HLog: Splitting 33 log(s) in hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020
2009-03-20 15:06:24,047 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 1 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237560687378
2009-03-20 15:06:24,106 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Adding queue for entities,,1236805004423
2009-03-20 15:06:25,443 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100006 entries
2009-03-20 15:06:25,459 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 2 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237560790320
2009-03-20 15:06:25,879 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Adding queue for hbase_types,426,1225564254435
2009-03-20 15:06:27,101 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100867 entries
2009-03-20 15:06:27,103 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 3 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561649939
2009-03-20 15:06:28,694 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 101754 entries
2009-03-20 15:06:28,696 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 4 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561658514
2009-03-20 15:06:33,324 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 332220 entries
2009-03-20 15:06:33,327 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 5 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561669181
2009-03-20 15:06:38,707 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 349439 entries
2009-03-20 15:06:38,711 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 6 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561688463
2009-03-20 15:06:40,922 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 207909 entries
2009-03-20 15:06:40,925 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 7 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561698495
2009-03-20 15:06:42,048 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 102829 entries
2009-03-20 15:06:42,050 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 8 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561703659
2009-03-20 15:06:44,199 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 204528 entries
2009-03-20 15:06:44,201 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 9 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561715064
2009-03-20 15:06:46,875 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 225964 entries
2009-03-20 15:06:46,878 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 10 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561726289
2009-03-20 15:06:47,885 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100645 entries
2009-03-20 15:06:47,887 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 11 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561793535
2009-03-20 15:06:49,198 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 105605 entries
2009-03-20 15:06:49,222 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 12 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561854543
2009-03-20 15:06:50,227 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100363 entries
2009-03-20 15:06:50,229 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 13 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561941125
2009-03-20 15:06:51,305 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100648 entries
2009-03-20 15:06:51,307 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 14 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561953452
2009-03-20 15:06:53,111 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 159954 entries
2009-03-20 15:06:53,113 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 15 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237561986701
2009-03-20 15:06:54,450 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 101025 entries
2009-03-20 15:06:54,452 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 16 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562003837
2009-03-20 15:06:55,717 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100449 entries
2009-03-20 15:06:55,719 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 17 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562016248
2009-03-20 15:06:56,682 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 101244 entries
2009-03-20 15:06:56,699 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 18 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562049500
2009-03-20 15:06:57,749 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100274 entries
2009-03-20 15:06:57,751 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 19 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562060231
2009-03-20 15:06:59,012 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 111015 entries
2009-03-20 15:06:59,014 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 20 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562127374
2009-03-20 15:06:59,999 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100373 entries
2009-03-20 15:07:00,001 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 21 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562177943
2009-03-20 15:07:01,001 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100213 entries
2009-03-20 15:07:01,003 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 22 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562277537
2009-03-20 15:07:04,116 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 192782 entries
2009-03-20 15:07:04,119 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 23 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562304890
2009-03-20 15:07:05,774 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 101842 entries
2009-03-20 15:07:05,776 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 24 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562315409
2009-03-20 15:07:06,843 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 104371 entries
2009-03-20 15:07:06,845 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 25 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562321140
2009-03-20 15:07:08,213 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 102252 entries
2009-03-20 15:07:08,215 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 26 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562342084
2009-03-20 15:07:09,371 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100330 entries
2009-03-20 15:07:09,373 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 27 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562347414
2009-03-20 15:07:12,583 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 258947 entries
2009-03-20 15:07:12,585 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 28 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562678929
2009-03-20 15:07:13,926 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100082 entries
2009-03-20 15:07:13,928 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 29 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562683228
2009-03-20 15:07:15,216 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100115 entries
2009-03-20 15:07:15,218 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 30 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562708153
2009-03-20 15:07:16,418 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100030 entries
2009-03-20 15:07:16,420 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 31 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562734111
2009-03-20 15:07:17,783 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100054 entries
2009-03-20 15:07:17,785 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 32 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562748277
2009-03-20 15:07:19,902 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Pushed 100116 entries
2009-03-20 15:07:19,904 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Splitting 33 of 33: hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020/hlog.dat.1237562763336
2009-03-20 15:07:36,115 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Thread got 49699 to process
2009-03-20 15:07:36,114 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Thread got 4357600 to process
2009-03-20 15:07:36,168 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Creating new log file writer for path hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/entities/1200514131/oldlogfile.log and region entities,,1236805004423
2009-03-20 15:07:36,199 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Creating new log file writer for path hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/entities/862114639/oldlogfile.log and region hbase_types,426,1225564254435
2009-03-20 15:07:36,599 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Applied 49699 total edits to hbase_types,426,1225564254435 in 484ms
2009-03-20 15:07:44,734 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Cache hit for row <> in tableName .META.: location server 192.168.1.109:62020, location region name .META.,,1
2009-03-20 15:07:51,633 DEBUG org.apache.hadoop.hbase.regionserver.HLog: Applied 4357600 total edits to entities,,1236805004423 in 15517ms
2009-03-20 15:07:51,633 INFO org.apache.hadoop.hbase.regionserver.HLog: Took 87586ms
2009-03-20 15:07:51,650 INFO org.apache.hadoop.hbase.regionserver.HLog: log file splitting completed for hdfs://factory01.lab.mtl:9200/hbase/amsterdam_factory/log_192.168.1.111_1237511553894_62020
{quote}
> [performance] The replay of logs on server crash takes way too long
> -------------------------------------------------------------------
>
> Key: HBASE-1008
> URL: https://issues.apache.org/jira/browse/HBASE-1008
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Priority: Blocker
> Fix For: 0.20.0
>
>
> Watching recovery from a crash on streamy.com where there were 1048 logs and repay is running at rate of about 20 seconds each. Meantime these regions are not online. This is way too long to wait on recovery for a live site. Marking critical. Performance related so priority and in 0.20.0.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.