You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2018/06/29 17:17:00 UTC
[jira] [Created] (HBASE-20824) IllegalArgumentException:
state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS
SplitTableRegionProcedure
stack created HBASE-20824:
-----------------------------
Summary: IllegalArgumentException: state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS SplitTableRegionProcedure
Key: HBASE-20824
URL: https://issues.apache.org/jira/browse/HBASE-20824
Project: HBase
Issue Type: Bug
Components: amv2
Reporter: stack
Running tests to check HBASE-20796 STUCK RIT though region successfully assigned, I came across this case.
Log is throwing out these:
{code}
STUCK Region-In-Transition rit=CLOSED, location=ve0536.X.Y.Z.com,16020,1530233509248, table=IntegrationTestBigLinkedList, region=18643f59d7db3689eeb133062ae4bc4f
{code}
Backup up I got this:
{code}
2018-06-28 17:57:32,744 INFO [PEWorker-3] procedure.MasterProcedureScheduler: pid=2716, state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=IntegrationTestBigLinkedList, parent=18643f59d7db3689eeb133062ae4bc4f, daughterA=6b0d2a0d22113c529dd49fc923a35ad9, daughterB=be72b5916a6a11d9ddce7d8af24f129f checking lock on 18643f59d7db3689eeb133062ae4bc4f
2018-06-28 17:57:32,745 WARN [PEWorker-3] procedure2.ProcedureExecutor: Worker terminating UNNATURALLY null java.lang.IllegalArgumentException: pid=2716, state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=IntegrationTestBigLinkedList, parent=18643f59d7db3689eeb133062ae4bc4f, daughterA=6b0d2a0d22113c529dd49fc923a35ad9, daughterB=be72b5916a6a11d9ddce7d8af24f129f at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:134) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1449) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
{code}
Just before this I see the below:
{code}
2018-06-28 17:57:32,604 WARN [master/ve0524:16000.Chore.1] master.CatalogJanitor: Failed initial scan of catalog table org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.isInMaintenanceMode(HMaster.java:2823) at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:281) at org.apache.hadoop.hbase.master.CatalogJanitor.initialChore(CatalogJanitor.java:81)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
{code}
We are failing this check
Preconditions.checkArgument(procedure.getState() == ProcedureState.RUNNABLE,
procedure.toString());
Oh, almost in the same millisecond, we get this:
2018-06-28 17:57:32,743 INFO [master/ve0524:16000] master.HMaster: Master has completed initialization 13.246sec
Is it a race? The RUNNING state has not been set yet?
Here was what was in meta. We'd just loaded the region:
2018-06-28 17:57:32,443 INFO [master/ve0524:16000] assignment.RegionStateStore: Load hbase:meta entry region=18643f59d7db3689eeb133062ae4bc4f, regionState=CLOSED, lastHost=ve0536.halxg.cloudera.com,16020,1530233509248, regionLocation=ve0536.X.Y.Z,16020,1530233509248, openSeqNum=488250
And before the crash, last thing was
2018-06-28 17:57:08,609 INFO [PEWorker-16] assignment.RegionStateStore: pid=2717 updating hbase:meta row=18643f59d7db3689eeb133062ae4bc4f, regionState=CLOSED
Before that...
2018-06-28 17:57:05,457 INFO [PEWorker-10] procedure.MasterProcedureScheduler: pid=2717, ppid=2716, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=IntegrationTestBigLinkedList, region=18643f59d7db3689eeb133062ae4bc4f, server=ve0536.X.Y.Z.com,16020,1530233509248 checking lock on 18643f59d7db3689eeb133062ae4bc4f
pid=2716 is the split.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)