You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (Jira)" <ji...@apache.org> on 2019/09/19 22:16:00 UTC
[jira] [Resolved] (HBASE-20824) IllegalArgumentException:
state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS
SplitTableRegionProcedure
[ https://issues.apache.org/jira/browse/HBASE-20824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-20824.
---------------------------
Resolution: Won't Fix
Resolving old issue about condition we probably fixed long time ago.
> IllegalArgumentException: state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS SplitTableRegionProcedure
> ------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-20824
> URL: https://issues.apache.org/jira/browse/HBASE-20824
> Project: HBase
> Issue Type: Bug
> Components: amv2
> Reporter: stack
> Priority: Major
>
> Running tests to check HBASE-20796 STUCK RIT though region successfully assigned, I came across this case.
> Log is throwing out these:
> {code}
> STUCK Region-In-Transition rit=CLOSED, location=ve0536.X.Y.Z.com,16020,1530233509248, table=IntegrationTestBigLinkedList, region=18643f59d7db3689eeb133062ae4bc4f
> {code}
> Backup up I got this:
> {code}
> 2018-06-28 17:57:32,744 INFO [PEWorker-3] procedure.MasterProcedureScheduler: pid=2716, state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=IntegrationTestBigLinkedList, parent=18643f59d7db3689eeb133062ae4bc4f, daughterA=6b0d2a0d22113c529dd49fc923a35ad9, daughterB=be72b5916a6a11d9ddce7d8af24f129f checking lock on 18643f59d7db3689eeb133062ae4bc4f
> 2018-06-28 17:57:32,745 WARN [PEWorker-3] procedure2.ProcedureExecutor: Worker terminating UNNATURALLY null java.lang.IllegalArgumentException: pid=2716, state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=IntegrationTestBigLinkedList, parent=18643f59d7db3689eeb133062ae4bc4f, daughterA=6b0d2a0d22113c529dd49fc923a35ad9, daughterB=be72b5916a6a11d9ddce7d8af24f129f at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:134) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1449) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {code}
> Just before this I see the below:
> {code}
> 2018-06-28 17:57:32,604 WARN [master/ve0524:16000.Chore.1] master.CatalogJanitor: Failed initial scan of catalog table org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.isInMaintenanceMode(HMaster.java:2823) at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:281) at org.apache.hadoop.hbase.master.CatalogJanitor.initialChore(CatalogJanitor.java:81)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
> {code}
> We are failing this check
> Preconditions.checkArgument(procedure.getState() == ProcedureState.RUNNABLE,
> procedure.toString());
> Oh, almost in the same millisecond, we get this:
> 2018-06-28 17:57:32,743 INFO [master/ve0524:16000] master.HMaster: Master has completed initialization 13.246sec
> Is it a race? The RUNNING state has not been set yet?
> Here was what was in meta. We'd just loaded the region:
> 2018-06-28 17:57:32,443 INFO [master/ve0524:16000] assignment.RegionStateStore: Load hbase:meta entry region=18643f59d7db3689eeb133062ae4bc4f, regionState=CLOSED, lastHost=ve0536.halxg.cloudera.com,16020,1530233509248, regionLocation=ve0536.X.Y.Z,16020,1530233509248, openSeqNum=488250
> And before the crash, last thing was
> 2018-06-28 17:57:08,609 INFO [PEWorker-16] assignment.RegionStateStore: pid=2717 updating hbase:meta row=18643f59d7db3689eeb133062ae4bc4f, regionState=CLOSED
> Before that...
> 2018-06-28 17:57:05,457 INFO [PEWorker-10] procedure.MasterProcedureScheduler: pid=2717, ppid=2716, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=IntegrationTestBigLinkedList, region=18643f59d7db3689eeb133062ae4bc4f, server=ve0536.X.Y.Z.com,16020,1530233509248 checking lock on 18643f59d7db3689eeb133062ae4bc4f
> pid=2716 is the split.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)