You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by st...@duboce.net on 2010/08/31 09:02:35 UTC
Review Request: hbase-2692 Master rewrite and cleanup for 0.90
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/750/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
See the issue for an overview.
Here is a bit of a synopsis:
+ Region in transition data structure is now kept out in zookeeper to facilitate master failover and to do away with race conditions that resulted in double assignment of regions
+ Refactored ZK usage
+ Open and close of regions as well as server shutdown handling and table transitions are not done in Executors; config. says how much parallellism to run with
+ New load balancer; features include figuring out the plan on startup and then assigning out all regions in the one assignment.
+ Most of what passed over the heartbeating mechanism has now moved to go via zk or the master directly invokes rpc to close/open regions rather than wait on heartbeat to come around
Bulk of this patch by Karthik Ranganathan and Jon Gray
This addresses bugs 0.90, Master, and, cleanup, for, hbase-2692, and rewrite.
http://issues.apache.org/jira/browse/0.90
http://issues.apache.org/jira/browse/Master
http://issues.apache.org/jira/browse/and
http://issues.apache.org/jira/browse/cleanup
http://issues.apache.org/jira/browse/for
http://issues.apache.org/jira/browse/hbase-2692
http://issues.apache.org/jira/browse/rewrite
Diffs
-----
.gitignore a8a9adc
BRANCH_CHANGES.txt PRE-CREATION
BRANCH_NOTES.txt PRE-CREATION
BRANCH_TODO.txt PRE-CREATION
conf/hbase-site.xml.psuedo-distributed.template a960eb7
pom.xml 2d3d75a
src/main/java/org/apache/hadoop/hbase/Abortable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/Chore.java 25bf1a0
src/main/java/org/apache/hadoop/hbase/ClusterStatus.java d102643
src/main/java/org/apache/hadoop/hbase/ColumnNameParseException.java 943c1e3
src/main/java/org/apache/hadoop/hbase/DroppedSnapshotException.java 5ddfd0b
src/main/java/org/apache/hadoop/hbase/HBaseConfTool.java 5b9ad3f
src/main/java/org/apache/hadoop/hbase/HConstants.java bfaa4a1
src/main/java/org/apache/hadoop/hbase/HMerge.java 12b4b7b
src/main/java/org/apache/hadoop/hbase/HMsg.java 9c311bb
src/main/java/org/apache/hadoop/hbase/HRegionInfo.java ee94690
src/main/java/org/apache/hadoop/hbase/HRegionLocation.java 722991b
src/main/java/org/apache/hadoop/hbase/HServerAddress.java 763eca2
src/main/java/org/apache/hadoop/hbase/HServerInfo.java 3457f9e
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 0d57270
src/main/java/org/apache/hadoop/hbase/InvalidFamilyOperationException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/LeaseException.java c48cc7f
src/main/java/org/apache/hadoop/hbase/LeaseListener.java 54b3452
src/main/java/org/apache/hadoop/hbase/Leases.java e502cc4
src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/MiniZooKeeperCluster.java ff9ce2b
src/main/java/org/apache/hadoop/hbase/NotAllMetaRegionsOnlineException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/PleaseHoldException.java 5f63de4
src/main/java/org/apache/hadoop/hbase/Server.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/Stoppable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/UnknownRegionException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/UnknownRowLockException.java 8cb3985
src/main/java/org/apache/hadoop/hbase/ValueOverMaxLengthException.java 383c9db
src/main/java/org/apache/hadoop/hbase/WritableComparator.java 8faa189
src/main/java/org/apache/hadoop/hbase/YouAreDeadException.java 4089a08
src/main/java/org/apache/hadoop/hbase/ZooKeeperConnectionException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/avro/AvroServer.java d8cd00a
src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 8b01aa0
src/main/java/org/apache/hadoop/hbase/client/HBaseFsck.java 19cdec1
src/main/java/org/apache/hadoop/hbase/client/HConnection.java 8b3b6a4
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java ac2b99c
src/main/java/org/apache/hadoop/hbase/client/HTable.java cd60eeb
src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 5f8b909
src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedException.java b7cfc78
src/main/java/org/apache/hadoop/hbase/client/ServerConnectionManager.java dbb7f57
src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/executor/HBaseEventHandler.java d4f4318
src/main/java/org/apache/hadoop/hbase/executor/HBaseExecutorService.java b5f8987
src/main/java/org/apache/hadoop/hbase/executor/NamedThreadFactory.java 87ea97d
src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionEventData.java 1b39de9
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 20860d6
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 9873172
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPCProtocolVersion.java d4bcbed
src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java bd48a4b
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1157fe1
src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/AddColumn.java 686521b
src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/BaseScanner.java 2deea4a
src/main/java/org/apache/hadoop/hbase/master/ChangeTableState.java c26bb56
src/main/java/org/apache/hadoop/hbase/master/ColumnOperation.java c8206ac
src/main/java/org/apache/hadoop/hbase/master/DeadServer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/DeleteColumn.java 1d58668
src/main/java/org/apache/hadoop/hbase/master/HMaster.java 4735304
src/main/java/org/apache/hadoop/hbase/master/InvalidColumnNameException.java 797fa3a
src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java 1c33831
src/main/java/org/apache/hadoop/hbase/master/LogsCleaner.java 9d1a8b8
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/MasterServices.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/MetaRegion.java 8d9a2db
src/main/java/org/apache/hadoop/hbase/master/MetaScanner.java e6434ba
src/main/java/org/apache/hadoop/hbase/master/ModifyColumn.java 2099444
src/main/java/org/apache/hadoop/hbase/master/ModifyTableMeta.java c985da9
src/main/java/org/apache/hadoop/hbase/master/NotAllMetaRegionsOnlineException.java 0e0ae27
src/main/java/org/apache/hadoop/hbase/master/ProcessRegionClose.java b94e887
src/main/java/org/apache/hadoop/hbase/master/ProcessRegionOpen.java 89e9915
src/main/java/org/apache/hadoop/hbase/master/ProcessRegionStatusChange.java b55c4f5
src/main/java/org/apache/hadoop/hbase/master/ProcessServerShutdown.java 2c1a1d8
src/main/java/org/apache/hadoop/hbase/master/RegionManager.java 6c6b98e
src/main/java/org/apache/hadoop/hbase/master/RegionServerOperation.java affd304
src/main/java/org/apache/hadoop/hbase/master/RegionServerOperationListener.java d221110
src/main/java/org/apache/hadoop/hbase/master/RegionServerOperationQueue.java d128370
src/main/java/org/apache/hadoop/hbase/master/RetryableMetaOperation.java 7b66785
src/main/java/org/apache/hadoop/hbase/master/RootScanner.java 7547928
src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 558ff10
src/main/java/org/apache/hadoop/hbase/master/TableDelete.java 1153e62
src/main/java/org/apache/hadoop/hbase/master/TableOperation.java 928d607
src/main/java/org/apache/hadoop/hbase/master/ZKMasterAddressWatcher.java 7b00819
src/main/java/org/apache/hadoop/hbase/master/ZKUnassignedWatcher.java acd23d1
src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/MasterCloseRegionHandler.java 4c850c3
src/main/java/org/apache/hadoop/hbase/master/handler/MasterOpenRegionHandler.java ee6b92e
src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/handler/TotesHRegionInfo.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java a78eea2
src/main/java/org/apache/hadoop/hbase/regionserver/CompactionRequestor.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java 38ac209
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java a692125
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8356d64
src/main/java/org/apache/hadoop/hbase/regionserver/LeaseException.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/LeaseListener.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 7b8fa56
src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java b45ed02
src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java 62f6eab
src/main/java/org/apache/hadoop/hbase/regionserver/RSZookeeperUpdater.java 310b063
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java 719c59d
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 3507c0d
src/main/java/org/apache/hadoop/hbase/regionserver/Stoppable.java 1fd7146
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 107d641
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java 1c13639
src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseMetaHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRootHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenMetaHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRootHandler.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 20a535c
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java 3036be3
src/main/java/org/apache/hadoop/hbase/regionserver/wal/LogActionsListener.java 110d35d
src/main/java/org/apache/hadoop/hbase/regionserver/wal/LogEntryVisitor.java 3518fba
src/main/java/org/apache/hadoop/hbase/regionserver/wal/LogRollListener.java 76514c1
src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeperWrapper.java 1007aeb
src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java 4d4b00a
src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 52cb8e8
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java 3bed8bb
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 32508de
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java f6e72dd
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java 8046b73
src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java 441917b
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 5cf3481
src/main/java/org/apache/hadoop/hbase/util/HBaseConfTool.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/util/HMerge.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java 280b91d
src/main/java/org/apache/hadoop/hbase/util/Merge.java c78110e
src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java 4481b12
src/main/java/org/apache/hadoop/hbase/util/Sleeper.java e5b4a5f
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/HQuorumPeer.java 71f1ab5
src/main/java/org/apache/hadoop/hbase/zookeeper/MetaNodeTracker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerTracker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKServerTool.java 94806bd
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableDisable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperListener.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java 3256ac9
src/main/resources/hbase-webapps/master/master.jsp 2c01d46
src/main/resources/hbase-webapps/master/table.jsp 35ad8a4
src/main/resources/hbase-webapps/master/zk.jsp 3a0c9f7
src/main/resources/hbase-webapps/regionserver/regionserver.jsp efc509c
src/site/resources/images/replication_overview.png 47d7b4c
src/test/java/org/apache/hadoop/hbase/BROKE_TODO_FIX_TestAcidGuarantees.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/HBaseClusterTestCase.java 500ab70
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java fadee21
src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 9c49e36
src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 1d7320c
src/test/java/org/apache/hadoop/hbase/TestFullLogReconstruction.java aad84ee
src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/TestHMsg.java 2b75f06
src/test/java/org/apache/hadoop/hbase/TestMergeMeta.java 611589b
src/test/java/org/apache/hadoop/hbase/TestMergeTable.java ec37f9b
src/test/java/org/apache/hadoop/hbase/TestMultiParallelPut.java 1727930
src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 23ed1f6
src/test/java/org/apache/hadoop/hbase/TestSerialization.java 9810385
src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java 4c59ea2
src/test/java/org/apache/hadoop/hbase/avro/TestAvroServer.java 3e293b6
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java c9b78b9
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java fc7b2af
src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 3818892
src/test/java/org/apache/hadoop/hbase/master/OOMEHMaster.java 607a2f0
src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/master/TestDeadServer.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/master/TestKillingServersFromMaster.java 8ced96d
src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java 8b7f60f
src/test/java/org/apache/hadoop/hbase/master/TestMaster.java 01ae0d2
src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java 8848b3d
src/test/java/org/apache/hadoop/hbase/master/TestMasterWithDisabling.java 91d4d08
src/test/java/org/apache/hadoop/hbase/master/TestROOTAssignment.java 0125134
src/test/java/org/apache/hadoop/hbase/master/TestRegionManager.java 7206e12
src/test/java/org/apache/hadoop/hbase/master/TestRegionServerOperationQueue.java bf63e1a
src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java 4584e52
src/test/java/org/apache/hadoop/hbase/master/TestServerManager.java d017e03
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java d0a9403
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java 4fb4c65
src/test/java/org/apache/hadoop/hbase/regionserver/DisabledTestRegionServerExit.java e90df70
src/test/java/org/apache/hadoop/hbase/regionserver/OOMERegionServer.java fd1d1bf
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java d0b84cc
src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java a245d97
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 0b47975
src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedSequenceFileLogWriter.java 7c740d3
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java ad8f9e5
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogActionsListener.java 6b03c4c
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 20ea8ee
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java c982662
src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java e1f4f98
src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 31cc680
src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java dc4f458
src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java b497b9a
src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java 2813db7
src/test/java/org/apache/hadoop/hbase/util/TestBytes.java 418fc1e
src/test/java/org/apache/hadoop/hbase/util/TestMergeMeta.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java PRE-CREATION
src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 3d6f0ec
src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java f0d90c0
src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java PRE-CREATION
Diff: http://review.cloudera.org/r/750/diff
Testing
-------
Near all unit tests pass. Still working through them.
Thanks,
stack