You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/01/21 08:06:59 UTC
[jira] Created: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
--------------------------------------------------------------------------------------------------------------------------------------------------
Key: HBASE-1143
URL: https://issues.apache.org/jira/browse/HBASE-1143
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716044#action_12716044 ]
Jean-Daniel Cryans commented on HBASE-1143:
-------------------------------------------
With 16k memcache I saw maybe 2-3 flushes per minute during the 7 minutes import. Each took anywhere between 50ms and 500ms. Compactions were keeping up since they are very fast.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-1143.
---------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Closing this issue. Will add a patch to help migration later.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711373#action_12711373 ]
stack commented on HBASE-1143:
------------------------------
I still see this. Easy to reproduce. Once numbers go erratic, don't seem to come back.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716752#action_12716752 ]
stack commented on HBASE-1143:
------------------------------
Backported the above fix which makes .META. schema editable and J-D's setting of memcache size to 16k on .META.:
{code}
Index: src/java/org/apache/hadoop/hbase/HTableDescriptor.java
===================================================================
--- src/java/org/apache/hadoop/hbase/HTableDescriptor.java (revision 782130)
+++ src/java/org/apache/hadoop/hbase/HTableDescriptor.java (working copy)
@@ -111,6 +111,8 @@
for(HColumnDescriptor descriptor : families) {
this.families.put(Bytes.mapKey(descriptor.getName()), descriptor);
}
+ // TODO: Fix this hard-coding.
+ setMemcacheFlushSize(16 * 1024);
}
/**
@@ -687,4 +689,4 @@
new HColumnDescriptor(HConstants.COLUMN_FAMILY_HISTORIAN,
HConstants.ALL_VERSIONS, HColumnDescriptor.CompressionType.NONE,
false, false, Integer.MAX_VALUE, HConstants.WEEK_IN_SECONDS, false)});
-}
{code}
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716043#action_12716043 ]
stack commented on HBASE-1143:
------------------------------
Lets try it. 16k rather than 12k just because seems less 'odd'. How many flushes were there at any one time? Was compaction keeping up?
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716411#action_12716411 ]
stack commented on HBASE-1143:
------------------------------
I added this as part of J-D's change:
{code}
Index: src/java/org/apache/hadoop/hbase/client/HConnectionManager.java
===================================================================
--- src/java/org/apache/hadoop/hbase/client/HConnectionManager.java (revision 781854)
+++ src/java/org/apache/hadoop/hbase/client/HConnectionManager.java (working copy)
@@ -461,7 +461,7 @@
return new UnmodifyableHTableDescriptor(HTableDescriptor.ROOT_TABLEDESC);
}
if (Bytes.equals(tableName, HConstants.META_TABLE_NAME)) {
- return new UnmodifyableHTableDescriptor(HTableDescriptor.META_TABLEDESC);
+ return HTableDescriptor.META_TABLEDESC;
}
HTableDescriptorFinder finder = new HTableDescriptorFinder(tableName);
MetaScanner.metaScan(conf, finder);
{code}
Makes it so you can edit the .META. schema so you can change memcache flushsize on the .META. table from shell.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1143:
--------------------------------------
Assignee: Jean-Daniel Cryans
I'm taking the issue. I'll test very low memcache size on .META. and .ROOT. and see how it goes under massive uploads.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715659#action_12715659 ]
Jonathan Gray commented on HBASE-1143:
--------------------------------------
+1 on flushing/compating META all the time until we have appends and can flush every 1 edit.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1143:
-------------------------
Fix Version/s: 0.20.0
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716296#action_12716296 ]
Jean-Daniel Cryans commented on HBASE-1143:
-------------------------------------------
Tested this morning on a 4 nodes cluster the current trunk (which includes my commit of yesterday). I first did a PE randomWrite 5 followed by a PE randomWrite 10. Flushes happened every 2 minutes or so and store files quickly got compacted when there were more than 6. So I think 16k is just great, it won't throttle huge imports and will save some lost edits.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715658#action_12715658 ]
Jean-Daniel Cryans commented on HBASE-1143:
-------------------------------------------
I tried this. What I see after killing the .META. holding RS is that we lose a bunch of edits so we only see an old version of the .META. table coming back. That version, if you killed your RS not long after the cluster startup, is a snapshot of .META. on the last shutdown. What I saw is a bunch of "Current assignment of ...", just like a fresh start!
Now I restarted HBase and tested again but I first flushed .META. and, guess what, everything was fine. IMO we should flush/compact .META. like crazy until we are able to not lose edits. That region is always small so it seconds < 1 sec to do.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716036#action_12716036 ]
Jean-Daniel Cryans commented on HBASE-1143:
-------------------------------------------
I tried different values for the memcache max size. Between each test of randomWrite 4 on a single node (cluster wasn't available), I did a shutdown of HBase + deleted the directory in HDFS. Here are the results:
8k memcache first try
ELAPSED_TIME=423519
second try
ELAPSED_TIME=405319
16k memcache first try
ELAPSED_TIME=437552
second try
ELAPSED_TIME=444659
1M memcache first try
ELAPSED_TIME=423519
second try
ELAPSED_TIME=432052
12M memcache first try
ELAPSED_TIME=450118
second try
ELAPSED_TIME=428693
I did also try with 1k memcache but it was complete nonsense since it was flushing every 5 seconds. I think a sane value would be 12k so we get to flush more often but not too much. Could make this configurable.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1143) region count erratic in master UI
(kill server hosting root or meta and see how count goes awry).... make
sure you have a bunch of reions in there
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716381#action_12716381 ]
stack commented on HBASE-1143:
------------------------------
So, just had this happen on our cluster. Need to backport something for 0.19.4.
> region count erratic in master UI (kill server hosting root or meta and see how count goes awry).... make sure you have a bunch of reions in there
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1143
> URL: https://issues.apache.org/jira/browse/HBASE-1143
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.20.0
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.