[jira] Commented: (HBASE-674) memcache size unreliable

["Billy Pearson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609705#action_12609705 ] 

Billy Pearson commented on HBASE-674:
-------------------------------------

I thank this should be a blocker for 2.0 as it causes problems on long running jobs and region server that have been online for a while.

What I am seeing is the last line of the flush like the below log lines has the memcache size on the end after the flush. this grows with every flush that has data in it. 

The problem comes up with the region server thanks the memcache is > then hbase.hregion.memcache.flush.size then it starts flushing back to back with little or nothing 
to flush when it starts doing this on my servers the region server hangs and is needed to be restarted after killing it with a kill -9 pid.

Also a side note I do see flushes back to back on region server that only have one or two regions I see a memcache flush then 1-3 more of the same region back to back then its ok but this seams to go away after the region server host more then 4 region.

Logs from a few days ago:
{code}
2008-06-25 21:02:20,632 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 10604ms, sequence id=1973821, 17.0m
2008-06-25 21:02:20,633 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 1ms, sequence id=1973822, 17.0m
2008-06-25 21:02:21,478 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 842ms, sequence id=1973832, 17.4m
2008-06-25 21:02:22,896 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 1408ms, sequence id=1976711, 17.8m
2008-06-25 21:11:20,979 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 16988ms, sequence id=3827578, 52.0m
hbase restarted
2008-06-25 21:16:18,004 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 10336ms, sequence id=4817365, 17.0m
2008-06-25 21:16:18,408 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 404ms, sequence id=4817378, 17.0m
2008-06-25 21:16:19,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 1421ms, sequence id=4817952, 17.7m
2008-06-25 21:19:01,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 9692ms, sequence id=5661153, 32.8m
2008-06-25 21:19:01,821 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 308ms, sequence id=5661169, 32.8m
2008-06-25 21:19:04,261 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 2439ms, sequence id=5661523, 33.4m
2008-06-25 21:19:05,317 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 1052ms, sequence id=5666391, 33.5m
2008-06-25 21:49:21,616 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 14214ms, sequence id=7415697, 64.7m
2008-06-25 21:49:22,369 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 738ms, sequence id=7415798, 64.7m
2008-06-25 21:49:22,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214427652459 in 4ms, sequence id=7415800, 64.7m
hbase restarted
2008-06-25 22:54:33,665 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 11168ms, sequence id=9238125, 19.5m
2008-06-25 22:54:34,248 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 582ms, sequence id=9241589, 19.7m
2008-06-25 22:54:34,989 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 740ms, sequence id=9242882, 19.8m
2008-06-25 22:54:35,749 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 759ms, sequence id=9244773, 19.8m
2008-06-25 22:54:36,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 869ms, sequence id=9247214, 19.8m
2008-06-25 22:54:37,657 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 1031ms, sequence id=9249946, 20.0m
2008-06-25 23:14:29,857 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 13549ms, sequence id=12776517, 53.0m
hbase restart
2008-06-25 23:21:25,013 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 13706ms, sequence id=14457037, 20.2m
2008-06-25 23:21:25,508 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 494ms, sequence id=14474826, 20.2m
2008-06-25 23:21:26,353 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 844ms, sequence id=14475821, 20.3m
2008-06-25 23:21:27,208 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 854ms, sequence id=14478027, 20.3m
2008-06-25 23:21:27,816 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 607ms, sequence id=14481138, 20.3m
2008-06-25 23:21:28,609 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 791ms, sequence id=14483523, 20.3m
2008-06-25 23:21:29,234 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 623ms, sequence id=14485701, 20.3m
2008-06-25 23:21:31,459 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 2223ms, sequence id=14487619, 20.6m
2008-06-25 23:21:32,546 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 1079ms, sequence id=14496978, 20.8m
2008-06-25 23:24:36,701 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 3432ms, sequence id=15103902, 27.4m
hbase restarted
2008-06-25 23:28:18,288 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214448561806 in 1ms, sequence id=15103936, 0.0
2008-06-25 23:28:19,151 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214454498285 in 0ms, sequence id=15103937, 0.0
2008-06-25 23:45:43,021 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214454498285 in 14039ms, sequence id=19210706, 23.6m
2008-06-25 23:45:43,285 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214454498285 in 264ms, sequence id=19226471, 23.6m
2008-06-25 23:45:44,016 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214454498285 in 730ms, sequence id=19227141, 23.6m
2008-06-25 23:45:45,481 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214454498285 in 1464ms, sequence id=19229633, 23.7m
2008-06-25 23:45:46,474 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214454498285 in 990ms, sequence id=19234660, 23.8m
2008-06-25 23:47:56,208 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,com.wallbuilders.www%2FLIBissuesArticles.asp%3Fid%3D45%3Ahttp467123 in 759ms, sequence id=19961148, 21.7m
2008-06-25 23:55:07,552 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush for region webdata,,1214454498285 in 5795ms, sequence id=22333011, 41.4m
hbase restarted
{code}

> memcache size unreliable
> ------------------------
>
>                 Key: HBASE-674
>                 URL: https://issues.apache.org/jira/browse/HBASE-674
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.1.2
>            Reporter: stack
>         Attachments: 674-v2.patch, 674.patch
>
>
> Multiple updates against same row/column/ts will be seen as increments to cache size on insert but when we then play the memcache at flush time, we'll only see the most recent entry and decrement the memcache size by whatever its size; memcache will be off.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.