You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2018/08/27 21:58:00 UTC
[jira] [Resolved] (HBASE-21120) MoveRegionProcedure makes no
progress; goes to STUCK
[ https://issues.apache.org/jira/browse/HBASE-21120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-21120.
---------------------------
Resolution: Fixed
Assignee: stack
Fix Version/s: 2.0.2
2.1.1
Pushed to branch-2.0 and branch-2.1 which is where HBASE-21078 went.
With this patch, cluster tests get their robustness back.
> MoveRegionProcedure makes no progress; goes to STUCK
> ----------------------------------------------------
>
> Key: HBASE-21120
> URL: https://issues.apache.org/jira/browse/HBASE-21120
> Project: HBase
> Issue Type: Bug
> Components: amv2
> Reporter: stack
> Assignee: stack
> Priority: Major
> Fix For: 2.1.1, 2.0.2
>
> Attachments: HBASE-21120.apply-in-reverse.txt
>
>
> This is tip of branch-2.0. I'm running cluster tests. I've seen this twice now where a slowDeterministic schedules a move procedure just after startup and we go dead-in-the-water never going beyond schedule of the unassign.We even schedule subsequent moves but they go no where either.
> {code}
> ...
> 2018-08-26 10:19:30,352 INFO [PEWorker-14] procedure2.ProcedureExecutor: Finished pid=286, ppid=260, state=SUCCESS; MoveRegionProcedure hri=fb79d389cf324aaf3ad4b686d3ad21d5, source=ve0530.halxg.cloudera.com,16020,1535303391139, destination=ve0530.halxg.cloudera.com,16020,1535303391139 in 10.8100sec
> ...
> // MASTER KILLED
> ....
> 2018-08-26 10:21:10,585 INFO [Thread-19] assignment.RegionStateStore: Load hbase:meta entry region=fb79d389cf324aaf3ad4b686d3ad21d5, regionState=OPEN, lastHost=ve0530.halxg.cloudera.com,16020,1535303391139, regionLocation=ve0530.halxg.cloudera.com,16020,1535303391139, openSeqNum=42095
> ...
> 2018-08-26 10:26:15,531 INFO [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16000] master.HMaster: Client=stack//10.17.240.20 move hri=fb79d389cf324aaf3ad4b686d3ad21d5, source=ve0530.halxg.cloudera.com,16020,1535303391139, destination=ve0534.halxg.cloudera.com,16020,1535303812180, running balancer
> 2018-08-26 10:26:16,635 INFO [PEWorker-13] procedure.MasterProcedureScheduler: pid=413, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=fb79d389cf324aaf3ad4b686d3ad21d5, source=ve0530.halxg.cloudera.com,16020,1535303391139, destination=ve0534.halxg.cloudera.com,16020,1535303812180 checking lock on fb79d389cf324aaf3ad4b686d3ad21d5
> 2018-08-26 10:26:16,896 INFO [PEWorker-14] procedure2.ProcedureExecutor: Finished pid=405, state=SUCCESS; ServerCrashProcedure server=ve0540.halxg.cloudera.com,16020,1535303391120, splitWal=true, meta=true in 25.8450sec
> 2018-08-26 10:26:16,896 INFO [PEWorker-13] procedure.MasterProcedureScheduler: pid=414, ppid=413, state=RUNNABLE:REGION_TRANSITION_QUEUE; UnassignProcedure table=IntegrationTestBigLinkedList, region=fb79d389cf324aaf3ad4b686d3ad21d5, server=ve0530.halxg.cloudera.com,16020,1535303391139 checking lock on fb79d389cf324aaf3ad4b686d3ad21d5
> 2018-08-26 10:26:29,308 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16000] master.ServerManager: Registering regionserver=ve0540.halxg.cloudera.com,16020,1535304386779 2018-08-26 10:26:29,342 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [ve0540.halxg.cloudera.com,16020,1535304386779] 2018-08-26 10:27:03,820 INFO [ReadOnlyZKClient-ve0524.halxg.cloudera.com:2222@0x1da09819] zookeeper.ZooKeeper: Session: 0x16577368c840054 closed
> 2018-08-26 10:27:03,820 INFO [ReadOnlyZKClient-ve0524.halxg.cloudera.com:2222@0x1da09819-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x16577368c840054 2018-08-26 10:27:16,411 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16000] master.HMaster: Client=stack//10.17.240.20 move hri=fb79d389cf324aaf3ad4b686d3ad21d5, source=ve0530.halxg.cloudera.com,16020,1535303391139, destination=ve0534.halxg.cloudera.com,16020,1535303812180, running balancer 2018-08-26 10:27:17,001 INFO [PEWorker-13] procedure.MasterProcedureScheduler: pid=415, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=fb79d389cf324aaf3ad4b686d3ad21d5, source=ve0530.halxg
> .cloudera.com,16020,1535303391139, destination=ve0534.halxg.cloudera.com,16020,1535303812180 checking lock on fb79d389cf324aaf3ad4b686d3ad21d5
> ...
> {code}
> Looking in UI, it says the below which has NO mention of pid=414 ... like it disappeared!!!
> {code}
> 413 WAITING stack MoveRegionProcedure hri=fb79d389cf324aaf3ad4b686d3ad21d5, source=ve0530.halxg.cloudera.com,16020,1535303391139, destination=ve0534.halxg.cloudera.com,16020,1535303812180 Sun Aug 26 10:26:15 PDT 2018 Sun Aug 26 10:26:16 PDT 2018 [ { state => [ '1', '2' ] }, { regionId => '1535303851257', tableName => { namespace => 'ZGVmYXVsdA==', qualifier => 'SW50ZWdyYXRpb25UZXN0QmlnTGlua2VkTGlzdA==' }, startKey => 'mGGOysI+5tUhxwXKql8zRg==', endKey => 'nnnnnnnnnnA=', offline => 'false', split => 'false', replicaId => '0' }, { sourceServer => { hostName => 've0530.halxg.cloudera.com', port => '16020', startCode => '1535303391139' }, destinationServer => { hostName => 've0534.halxg.cloudera.com', port => '16020', startCode => '1535303812180' } } ]
> 415 RUNNABLE stack MoveRegionProcedure hri=fb79d389cf324aaf3ad4b686d3ad21d5, source=ve0530.halxg.cloudera.com,16020,1535303391139, destination=ve0534.halxg.cloudera.com,16020,1535303812180 Sun Aug 26 10:27:16 PDT 2018 Sun Aug 26 10:27:16 PDT 2018 [ {}, { regionId => '1535303851257', tableName => { namespace => 'ZGVmYXVsdA==', qualifier => 'SW50ZWdyYXRpb25UZXN0QmlnTGlua2VkTGlzdA==' }, startKey => 'mGGOysI+5tUhxwXKql8zRg==', endKey => 'nnnnnnnnnnA=', offline => 'false', split => 'false', replicaId => '0' }, { sourceServer => { hostName => 've0530.halxg.cloudera.com', port => '16020', startCode => '1535303391139' }, destinationServer => { hostName => 've0534.halxg.cloudera.com', port => '16020', startCode => '1535303812180' } } ]
> {code}
> Looking at locks in UI, it says that pid=413 has the exclusive lock on this region...
> {code}
> REGION: fb79d389cf324aaf3ad4b686d3ad21d5
> Lock type: EXCLUSIVE
> Owner procedure: { ID => '413', PARENT_ID => '-1', STATE => 'WAITING', OWNER => 'stack', TYPE => 'MoveRegionProcedure hri=fb79d389cf324aaf3ad4b686d3ad21d5, source=ve0530.halxg.cloudera.com,16020,1535303391139, destination=ve0534.halxg.cloudera.com,16020,1535303812180', START_TIME => 'Sun Aug 26 10:26:15 PDT 2018', LAST_UPDATE => 'Sun Aug 26 10:26:16 PDT 2018', PARAMETERS => [ { state => [ '1', '2' ] }, { regionId => '1535303851257', tableName => { namespace => 'ZGVmYXVsdA==', qualifier => 'SW50ZWdyYXRpb25UZXN0QmlnTGlua2VkTGlzdA==' }, startKey => 'mGGOysI+5tUhxwXKql8zRg==', endKey => 'nnnnnnnnnnA=', offline => 'false', split => 'false', replicaId => '0' }, { sourceServer => { hostName => 've0530.halxg.cloudera.com', port => '16020', startCode => '1535303391139' }, destinationServer => { hostName => 've0534.halxg.cloudera.com', port => '16020', startCode => '1535303812180' } } ] }
> {code}
> Is it that the child is not picking up the lock? Let me look at the WALs to see what story with pid=414 is....
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)