You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by gi...@apache.org on 2018/05/03 14:47:27 UTC
[01/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Repository: hbase-site
Updated Branches:
refs/heads/asf-site 18642a472 -> 2d03f412f
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/downloads.html
----------------------------------------------------------------------
diff --git a/downloads.html b/downloads.html
index ef70d10..099207f 100644
--- a/downloads.html
+++ b/downloads.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Apache HBase Downloads</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -366,7 +366,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/export_control.html
----------------------------------------------------------------------
diff --git a/export_control.html b/export_control.html
index 3f86c8c..762a633 100644
--- a/export_control.html
+++ b/export_control.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase –
Export Control
@@ -331,7 +331,7 @@ for more details.</p>
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/index.html
----------------------------------------------------------------------
diff --git a/index.html b/index.html
index 6fd3abe..a2cff89 100644
--- a/index.html
+++ b/index.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Apache HBase™ Home</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -409,7 +409,7 @@ Apache HBase is an open-source, distributed, versioned, non-relational database
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/integration.html
----------------------------------------------------------------------
diff --git a/integration.html b/integration.html
index 6f85deb..595dcba 100644
--- a/integration.html
+++ b/integration.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – CI Management</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -291,7 +291,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/issue-tracking.html
----------------------------------------------------------------------
diff --git a/issue-tracking.html b/issue-tracking.html
index 3c6e64d..13aacbb 100644
--- a/issue-tracking.html
+++ b/issue-tracking.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Issue Management</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -288,7 +288,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/license.html
----------------------------------------------------------------------
diff --git a/license.html b/license.html
index 9d5b520..2f9f8fa 100644
--- a/license.html
+++ b/license.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Project Licenses</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -491,7 +491,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/mail-lists.html
----------------------------------------------------------------------
diff --git a/mail-lists.html b/mail-lists.html
index 950ff77..f41f35f 100644
--- a/mail-lists.html
+++ b/mail-lists.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Project Mailing Lists</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -341,7 +341,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/metrics.html
----------------------------------------------------------------------
diff --git a/metrics.html b/metrics.html
index 8de636d..fa63090 100644
--- a/metrics.html
+++ b/metrics.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase –
Apache HBase (TM) Metrics
@@ -459,7 +459,7 @@ export HBASE_REGIONSERVER_OPTS="$HBASE_JMX_OPTS -Dcom.sun.management.jmxrem
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/old_news.html
----------------------------------------------------------------------
diff --git a/old_news.html b/old_news.html
index b510a41..399e924 100644
--- a/old_news.html
+++ b/old_news.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase –
Old Apache HBase (TM) News
@@ -440,7 +440,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/plugin-management.html
----------------------------------------------------------------------
diff --git a/plugin-management.html b/plugin-management.html
index 380cf98..04d0689 100644
--- a/plugin-management.html
+++ b/plugin-management.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Project Plugin Management</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -440,7 +440,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/plugins.html
----------------------------------------------------------------------
diff --git a/plugins.html b/plugins.html
index 8927d88..6f5cc89 100644
--- a/plugins.html
+++ b/plugins.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Project Plugins</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -375,7 +375,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/poweredbyhbase.html
----------------------------------------------------------------------
diff --git a/poweredbyhbase.html b/poweredbyhbase.html
index 27647a0..e85334d 100644
--- a/poweredbyhbase.html
+++ b/poweredbyhbase.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Powered By Apache HBase</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -769,7 +769,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/project-info.html
----------------------------------------------------------------------
diff --git a/project-info.html b/project-info.html
index 4b11639..78c3f33 100644
--- a/project-info.html
+++ b/project-info.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Project Information</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -335,7 +335,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/project-reports.html
----------------------------------------------------------------------
diff --git a/project-reports.html b/project-reports.html
index d66991b..cc392fa 100644
--- a/project-reports.html
+++ b/project-reports.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Generated Reports</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -305,7 +305,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/project-summary.html
----------------------------------------------------------------------
diff --git a/project-summary.html b/project-summary.html
index 1b49bdc..a3a750f 100644
--- a/project-summary.html
+++ b/project-summary.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Project Summary</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -331,7 +331,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/pseudo-distributed.html
----------------------------------------------------------------------
diff --git a/pseudo-distributed.html b/pseudo-distributed.html
index 6d3329c..65fad55 100644
--- a/pseudo-distributed.html
+++ b/pseudo-distributed.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase –
Running Apache HBase (TM) in pseudo-distributed mode
@@ -308,7 +308,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/replication.html
----------------------------------------------------------------------
diff --git a/replication.html b/replication.html
index 9fa89b8..093a598 100644
--- a/replication.html
+++ b/replication.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase –
Apache HBase (TM) Replication
@@ -303,7 +303,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/resources.html
----------------------------------------------------------------------
diff --git a/resources.html b/resources.html
index eb5d64d..ff2f521 100644
--- a/resources.html
+++ b/resources.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Other Apache HBase (TM) Resources</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -331,7 +331,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/source-repository.html
----------------------------------------------------------------------
diff --git a/source-repository.html b/source-repository.html
index 6d0109f..e9389cf 100644
--- a/source-repository.html
+++ b/source-repository.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Source Code Management</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -299,7 +299,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/sponsors.html
----------------------------------------------------------------------
diff --git a/sponsors.html b/sponsors.html
index 373e8d9..3d91cbb 100644
--- a/sponsors.html
+++ b/sponsors.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Apache HBase Sponsors</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -333,7 +333,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/supportingprojects.html
----------------------------------------------------------------------
diff --git a/supportingprojects.html b/supportingprojects.html
index 201f15e..458e3db 100644
--- a/supportingprojects.html
+++ b/supportingprojects.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Supporting Projects</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -520,7 +520,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/team-list.html
----------------------------------------------------------------------
diff --git a/team-list.html b/team-list.html
index 1e38067..c70decc 100644
--- a/team-list.html
+++ b/team-list.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Project Team</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -730,7 +730,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/testdevapidocs/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html
----------------------------------------------------------------------
diff --git a/testdevapidocs/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html b/testdevapidocs/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html
index 06ab617..6c55d3d 100644
--- a/testdevapidocs/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html
+++ b/testdevapidocs/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html
@@ -109,7 +109,7 @@ var activeTableTab = "activeTableTab";
<li class="blockList">
<hr>
<br>
-<pre>public class <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.52">TestAssignmentManagerMetrics</a>
+<pre>public class <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.53">TestAssignmentManagerMetrics</a>
extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html?is-external=true" title="class or interface in java.lang">Object</a></pre>
</li>
</ul>
@@ -233,7 +233,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>CLASS_RULE</h4>
-<pre>public static final <a href="../../../../../org/apache/hadoop/hbase/HBaseClassTestRule.html" title="class in org.apache.hadoop.hbase">HBaseClassTestRule</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.55">CLASS_RULE</a></pre>
+<pre>public static final <a href="../../../../../org/apache/hadoop/hbase/HBaseClassTestRule.html" title="class in org.apache.hadoop.hbase">HBaseClassTestRule</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.56">CLASS_RULE</a></pre>
</li>
</ul>
<a name="LOG">
@@ -242,7 +242,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>LOG</h4>
-<pre>private static final org.slf4j.Logger <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.58">LOG</a></pre>
+<pre>private static final org.slf4j.Logger <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.59">LOG</a></pre>
</li>
</ul>
<a name="METRICS_HELPER">
@@ -251,7 +251,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>METRICS_HELPER</h4>
-<pre>private static final <a href="../../../../../org/apache/hadoop/hbase/test/MetricsAssertHelper.html" title="interface in org.apache.hadoop.hbase.test">MetricsAssertHelper</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.59">METRICS_HELPER</a></pre>
+<pre>private static final <a href="../../../../../org/apache/hadoop/hbase/test/MetricsAssertHelper.html" title="interface in org.apache.hadoop.hbase.test">MetricsAssertHelper</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.60">METRICS_HELPER</a></pre>
</li>
</ul>
<a name="CLUSTER">
@@ -260,7 +260,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>CLUSTER</h4>
-<pre>private static <a href="../../../../../org/apache/hadoop/hbase/MiniHBaseCluster.html" title="class in org.apache.hadoop.hbase">MiniHBaseCluster</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.62">CLUSTER</a></pre>
+<pre>private static <a href="../../../../../org/apache/hadoop/hbase/MiniHBaseCluster.html" title="class in org.apache.hadoop.hbase">MiniHBaseCluster</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.63">CLUSTER</a></pre>
</li>
</ul>
<a name="MASTER">
@@ -269,7 +269,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>MASTER</h4>
-<pre>private static org.apache.hadoop.hbase.master.HMaster <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.63">MASTER</a></pre>
+<pre>private static org.apache.hadoop.hbase.master.HMaster <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.64">MASTER</a></pre>
</li>
</ul>
<a name="TEST_UTIL">
@@ -278,7 +278,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>TEST_UTIL</h4>
-<pre>private static <a href="../../../../../org/apache/hadoop/hbase/HBaseTestingUtility.html" title="class in org.apache.hadoop.hbase">HBaseTestingUtility</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.64">TEST_UTIL</a></pre>
+<pre>private static <a href="../../../../../org/apache/hadoop/hbase/HBaseTestingUtility.html" title="class in org.apache.hadoop.hbase">HBaseTestingUtility</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.65">TEST_UTIL</a></pre>
</li>
</ul>
<a name="MSG_INTERVAL">
@@ -287,7 +287,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>MSG_INTERVAL</h4>
-<pre>private static final int <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.65">MSG_INTERVAL</a></pre>
+<pre>private static final int <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.66">MSG_INTERVAL</a></pre>
<dl>
<dt><span class="seeLabel">See Also:</span></dt>
<dd><a href="../../../../../constant-values.html#org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.MSG_INTERVAL">Constant Field Values</a></dd>
@@ -300,7 +300,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockListLast">
<li class="blockList">
<h4>name</h4>
-<pre>public org.junit.rules.TestName <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.68">name</a></pre>
+<pre>public org.junit.rules.TestName <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.69">name</a></pre>
</li>
</ul>
</li>
@@ -317,7 +317,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockListLast">
<li class="blockList">
<h4>TestAssignmentManagerMetrics</h4>
-<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.52">TestAssignmentManagerMetrics</a>()</pre>
+<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.53">TestAssignmentManagerMetrics</a>()</pre>
</li>
</ul>
</li>
@@ -334,7 +334,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>startCluster</h4>
-<pre>public static void <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.71">startCluster</a>()
+<pre>public static void <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.72">startCluster</a>()
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Exception.html?is-external=true" title="class or interface in java.lang">Exception</a></pre>
<dl>
<dt><span class="throwsLabel">Throws:</span></dt>
@@ -348,7 +348,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>after</h4>
-<pre>public static void <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.100">after</a>()
+<pre>public static void <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.103">after</a>()
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Exception.html?is-external=true" title="class or interface in java.lang">Exception</a></pre>
<dl>
<dt><span class="throwsLabel">Throws:</span></dt>
@@ -362,7 +362,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockListLast">
<li class="blockList">
<h4>testRITAssignmentManagerMetrics</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.106">testRITAssignmentManagerMetrics</a>()
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html#line.109">testRITAssignmentManagerMetrics</a>()
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Exception.html?is-external=true" title="class or interface in java.lang">Exception</a></pre>
<dl>
<dt><span class="throwsLabel">Throws:</span></dt>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/testdevapidocs/src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html
----------------------------------------------------------------------
diff --git a/testdevapidocs/src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html b/testdevapidocs/src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html
index fba1d10..f83e181 100644
--- a/testdevapidocs/src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html
+++ b/testdevapidocs/src-html/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.html
@@ -41,128 +41,134 @@
<span class="sourceLineNo">033</span>import org.apache.hadoop.hbase.client.Table;<a name="line.33"></a>
<span class="sourceLineNo">034</span>import org.apache.hadoop.hbase.client.TableDescriptor;<a name="line.34"></a>
<span class="sourceLineNo">035</span>import org.apache.hadoop.hbase.client.TableDescriptorBuilder;<a name="line.35"></a>
-<span class="sourceLineNo">036</span>import org.apache.hadoop.hbase.master.assignment.AssignmentManager;<a name="line.36"></a>
-<span class="sourceLineNo">037</span>import org.apache.hadoop.hbase.test.MetricsAssertHelper;<a name="line.37"></a>
-<span class="sourceLineNo">038</span>import org.apache.hadoop.hbase.testclassification.MasterTests;<a name="line.38"></a>
-<span class="sourceLineNo">039</span>import org.apache.hadoop.hbase.testclassification.MediumTests;<a name="line.39"></a>
-<span class="sourceLineNo">040</span>import org.apache.hadoop.hbase.util.Bytes;<a name="line.40"></a>
-<span class="sourceLineNo">041</span>import org.junit.AfterClass;<a name="line.41"></a>
-<span class="sourceLineNo">042</span>import org.junit.BeforeClass;<a name="line.42"></a>
-<span class="sourceLineNo">043</span>import org.junit.ClassRule;<a name="line.43"></a>
-<span class="sourceLineNo">044</span>import org.junit.Rule;<a name="line.44"></a>
-<span class="sourceLineNo">045</span>import org.junit.Test;<a name="line.45"></a>
-<span class="sourceLineNo">046</span>import org.junit.experimental.categories.Category;<a name="line.46"></a>
-<span class="sourceLineNo">047</span>import org.junit.rules.TestName;<a name="line.47"></a>
-<span class="sourceLineNo">048</span>import org.slf4j.Logger;<a name="line.48"></a>
-<span class="sourceLineNo">049</span>import org.slf4j.LoggerFactory;<a name="line.49"></a>
-<span class="sourceLineNo">050</span><a name="line.50"></a>
-<span class="sourceLineNo">051</span>@Category({ MasterTests.class, MediumTests.class })<a name="line.51"></a>
-<span class="sourceLineNo">052</span>public class TestAssignmentManagerMetrics {<a name="line.52"></a>
-<span class="sourceLineNo">053</span><a name="line.53"></a>
-<span class="sourceLineNo">054</span> @ClassRule<a name="line.54"></a>
-<span class="sourceLineNo">055</span> public static final HBaseClassTestRule CLASS_RULE =<a name="line.55"></a>
-<span class="sourceLineNo">056</span> HBaseClassTestRule.forClass(TestAssignmentManagerMetrics.class);<a name="line.56"></a>
-<span class="sourceLineNo">057</span><a name="line.57"></a>
-<span class="sourceLineNo">058</span> private static final Logger LOG = LoggerFactory.getLogger(TestAssignmentManagerMetrics.class);<a name="line.58"></a>
-<span class="sourceLineNo">059</span> private static final MetricsAssertHelper METRICS_HELPER = CompatibilityFactory<a name="line.59"></a>
-<span class="sourceLineNo">060</span> .getInstance(MetricsAssertHelper.class);<a name="line.60"></a>
-<span class="sourceLineNo">061</span><a name="line.61"></a>
-<span class="sourceLineNo">062</span> private static MiniHBaseCluster CLUSTER;<a name="line.62"></a>
-<span class="sourceLineNo">063</span> private static HMaster MASTER;<a name="line.63"></a>
-<span class="sourceLineNo">064</span> private static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();<a name="line.64"></a>
-<span class="sourceLineNo">065</span> private static final int MSG_INTERVAL = 1000;<a name="line.65"></a>
-<span class="sourceLineNo">066</span><a name="line.66"></a>
-<span class="sourceLineNo">067</span> @Rule<a name="line.67"></a>
-<span class="sourceLineNo">068</span> public TestName name = new TestName();<a name="line.68"></a>
-<span class="sourceLineNo">069</span><a name="line.69"></a>
-<span class="sourceLineNo">070</span> @BeforeClass<a name="line.70"></a>
-<span class="sourceLineNo">071</span> public static void startCluster() throws Exception {<a name="line.71"></a>
-<span class="sourceLineNo">072</span> LOG.info("Starting cluster");<a name="line.72"></a>
-<span class="sourceLineNo">073</span> Configuration conf = TEST_UTIL.getConfiguration();<a name="line.73"></a>
-<span class="sourceLineNo">074</span><a name="line.74"></a>
-<span class="sourceLineNo">075</span> // Disable sanity check for coprocessor<a name="line.75"></a>
-<span class="sourceLineNo">076</span> conf.setBoolean("hbase.table.sanity.checks", false);<a name="line.76"></a>
-<span class="sourceLineNo">077</span><a name="line.77"></a>
-<span class="sourceLineNo">078</span> // set RIT stuck warning threshold to a small value<a name="line.78"></a>
-<span class="sourceLineNo">079</span> conf.setInt(HConstants.METRICS_RIT_STUCK_WARNING_THRESHOLD, 20);<a name="line.79"></a>
-<span class="sourceLineNo">080</span><a name="line.80"></a>
-<span class="sourceLineNo">081</span> // set msgInterval to 1 second<a name="line.81"></a>
-<span class="sourceLineNo">082</span> conf.setInt("hbase.regionserver.msginterval", MSG_INTERVAL);<a name="line.82"></a>
-<span class="sourceLineNo">083</span><a name="line.83"></a>
-<span class="sourceLineNo">084</span> // set tablesOnMaster to none<a name="line.84"></a>
-<span class="sourceLineNo">085</span> conf.set("hbase.balancer.tablesOnMaster", "none");<a name="line.85"></a>
-<span class="sourceLineNo">086</span><a name="line.86"></a>
-<span class="sourceLineNo">087</span> // set client sync wait timeout to 5sec<a name="line.87"></a>
-<span class="sourceLineNo">088</span> conf.setInt("hbase.client.sync.wait.timeout.msec", 5000);<a name="line.88"></a>
-<span class="sourceLineNo">089</span> conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 1);<a name="line.89"></a>
-<span class="sourceLineNo">090</span> conf.setInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT, 2500);<a name="line.90"></a>
-<span class="sourceLineNo">091</span> // set a small interval for updating rit metrics<a name="line.91"></a>
-<span class="sourceLineNo">092</span> conf.setInt(AssignmentManager.RIT_CHORE_INTERVAL_MSEC_CONF_KEY, MSG_INTERVAL);<a name="line.92"></a>
-<span class="sourceLineNo">093</span><a name="line.93"></a>
-<span class="sourceLineNo">094</span> TEST_UTIL.startMiniCluster(1);<a name="line.94"></a>
-<span class="sourceLineNo">095</span> CLUSTER = TEST_UTIL.getHBaseCluster();<a name="line.95"></a>
-<span class="sourceLineNo">096</span> MASTER = CLUSTER.getMaster();<a name="line.96"></a>
-<span class="sourceLineNo">097</span> }<a name="line.97"></a>
-<span class="sourceLineNo">098</span><a name="line.98"></a>
-<span class="sourceLineNo">099</span> @AfterClass<a name="line.99"></a>
-<span class="sourceLineNo">100</span> public static void after() throws Exception {<a name="line.100"></a>
-<span class="sourceLineNo">101</span> LOG.info("AFTER {} <= IS THIS NULL?", TEST_UTIL);<a name="line.101"></a>
-<span class="sourceLineNo">102</span> TEST_UTIL.shutdownMiniCluster();<a name="line.102"></a>
-<span class="sourceLineNo">103</span> }<a name="line.103"></a>
-<span class="sourceLineNo">104</span><a name="line.104"></a>
-<span class="sourceLineNo">105</span> @Test<a name="line.105"></a>
-<span class="sourceLineNo">106</span> public void testRITAssignmentManagerMetrics() throws Exception {<a name="line.106"></a>
-<span class="sourceLineNo">107</span> final TableName TABLENAME = TableName.valueOf(name.getMethodName());<a name="line.107"></a>
-<span class="sourceLineNo">108</span> final byte[] FAMILY = Bytes.toBytes("family");<a name="line.108"></a>
-<span class="sourceLineNo">109</span> try (Table table = TEST_UTIL.createTable(TABLENAME, FAMILY)){<a name="line.109"></a>
-<span class="sourceLineNo">110</span> final byte[] row = Bytes.toBytes("row");<a name="line.110"></a>
-<span class="sourceLineNo">111</span> final byte[] qualifier = Bytes.toBytes("qualifier");<a name="line.111"></a>
-<span class="sourceLineNo">112</span> final byte[] value = Bytes.toBytes("value");<a name="line.112"></a>
-<span class="sourceLineNo">113</span><a name="line.113"></a>
-<span class="sourceLineNo">114</span> Put put = new Put(row);<a name="line.114"></a>
-<span class="sourceLineNo">115</span> put.addColumn(FAMILY, qualifier, value);<a name="line.115"></a>
-<span class="sourceLineNo">116</span> table.put(put);<a name="line.116"></a>
-<span class="sourceLineNo">117</span><a name="line.117"></a>
-<span class="sourceLineNo">118</span> // Sleep 3 seconds, wait for doMetrics chore catching up<a name="line.118"></a>
-<span class="sourceLineNo">119</span> Thread.sleep(MSG_INTERVAL * 3);<a name="line.119"></a>
+<span class="sourceLineNo">036</span>import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;<a name="line.36"></a>
+<span class="sourceLineNo">037</span>import org.apache.hadoop.hbase.master.assignment.AssignmentManager;<a name="line.37"></a>
+<span class="sourceLineNo">038</span>import org.apache.hadoop.hbase.test.MetricsAssertHelper;<a name="line.38"></a>
+<span class="sourceLineNo">039</span>import org.apache.hadoop.hbase.testclassification.MasterTests;<a name="line.39"></a>
+<span class="sourceLineNo">040</span>import org.apache.hadoop.hbase.testclassification.MediumTests;<a name="line.40"></a>
+<span class="sourceLineNo">041</span>import org.apache.hadoop.hbase.util.Bytes;<a name="line.41"></a>
+<span class="sourceLineNo">042</span>import org.junit.AfterClass;<a name="line.42"></a>
+<span class="sourceLineNo">043</span>import org.junit.BeforeClass;<a name="line.43"></a>
+<span class="sourceLineNo">044</span>import org.junit.ClassRule;<a name="line.44"></a>
+<span class="sourceLineNo">045</span>import org.junit.Rule;<a name="line.45"></a>
+<span class="sourceLineNo">046</span>import org.junit.Test;<a name="line.46"></a>
+<span class="sourceLineNo">047</span>import org.junit.experimental.categories.Category;<a name="line.47"></a>
+<span class="sourceLineNo">048</span>import org.junit.rules.TestName;<a name="line.48"></a>
+<span class="sourceLineNo">049</span>import org.slf4j.Logger;<a name="line.49"></a>
+<span class="sourceLineNo">050</span>import org.slf4j.LoggerFactory;<a name="line.50"></a>
+<span class="sourceLineNo">051</span><a name="line.51"></a>
+<span class="sourceLineNo">052</span>@Category({ MasterTests.class, MediumTests.class })<a name="line.52"></a>
+<span class="sourceLineNo">053</span>public class TestAssignmentManagerMetrics {<a name="line.53"></a>
+<span class="sourceLineNo">054</span><a name="line.54"></a>
+<span class="sourceLineNo">055</span> @ClassRule<a name="line.55"></a>
+<span class="sourceLineNo">056</span> public static final HBaseClassTestRule CLASS_RULE =<a name="line.56"></a>
+<span class="sourceLineNo">057</span> HBaseClassTestRule.forClass(TestAssignmentManagerMetrics.class);<a name="line.57"></a>
+<span class="sourceLineNo">058</span><a name="line.58"></a>
+<span class="sourceLineNo">059</span> private static final Logger LOG = LoggerFactory.getLogger(TestAssignmentManagerMetrics.class);<a name="line.59"></a>
+<span class="sourceLineNo">060</span> private static final MetricsAssertHelper METRICS_HELPER = CompatibilityFactory<a name="line.60"></a>
+<span class="sourceLineNo">061</span> .getInstance(MetricsAssertHelper.class);<a name="line.61"></a>
+<span class="sourceLineNo">062</span><a name="line.62"></a>
+<span class="sourceLineNo">063</span> private static MiniHBaseCluster CLUSTER;<a name="line.63"></a>
+<span class="sourceLineNo">064</span> private static HMaster MASTER;<a name="line.64"></a>
+<span class="sourceLineNo">065</span> private static HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();<a name="line.65"></a>
+<span class="sourceLineNo">066</span> private static final int MSG_INTERVAL = 1000;<a name="line.66"></a>
+<span class="sourceLineNo">067</span><a name="line.67"></a>
+<span class="sourceLineNo">068</span> @Rule<a name="line.68"></a>
+<span class="sourceLineNo">069</span> public TestName name = new TestName();<a name="line.69"></a>
+<span class="sourceLineNo">070</span><a name="line.70"></a>
+<span class="sourceLineNo">071</span> @BeforeClass<a name="line.71"></a>
+<span class="sourceLineNo">072</span> public static void startCluster() throws Exception {<a name="line.72"></a>
+<span class="sourceLineNo">073</span> LOG.info("Starting cluster");<a name="line.73"></a>
+<span class="sourceLineNo">074</span> Configuration conf = TEST_UTIL.getConfiguration();<a name="line.74"></a>
+<span class="sourceLineNo">075</span><a name="line.75"></a>
+<span class="sourceLineNo">076</span> // Disable sanity check for coprocessor<a name="line.76"></a>
+<span class="sourceLineNo">077</span> conf.setBoolean("hbase.table.sanity.checks", false);<a name="line.77"></a>
+<span class="sourceLineNo">078</span><a name="line.78"></a>
+<span class="sourceLineNo">079</span> // set RIT stuck warning threshold to a small value<a name="line.79"></a>
+<span class="sourceLineNo">080</span> conf.setInt(HConstants.METRICS_RIT_STUCK_WARNING_THRESHOLD, 20);<a name="line.80"></a>
+<span class="sourceLineNo">081</span><a name="line.81"></a>
+<span class="sourceLineNo">082</span> // set msgInterval to 1 second<a name="line.82"></a>
+<span class="sourceLineNo">083</span> conf.setInt("hbase.regionserver.msginterval", MSG_INTERVAL);<a name="line.83"></a>
+<span class="sourceLineNo">084</span><a name="line.84"></a>
+<span class="sourceLineNo">085</span> // set tablesOnMaster to none<a name="line.85"></a>
+<span class="sourceLineNo">086</span> conf.set("hbase.balancer.tablesOnMaster", "none");<a name="line.86"></a>
+<span class="sourceLineNo">087</span><a name="line.87"></a>
+<span class="sourceLineNo">088</span> // set client sync wait timeout to 5sec<a name="line.88"></a>
+<span class="sourceLineNo">089</span> conf.setInt("hbase.client.sync.wait.timeout.msec", 5000);<a name="line.89"></a>
+<span class="sourceLineNo">090</span> conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 1);<a name="line.90"></a>
+<span class="sourceLineNo">091</span> conf.setInt(HConstants.HBASE_CLIENT_OPERATION_TIMEOUT, 2500);<a name="line.91"></a>
+<span class="sourceLineNo">092</span> // set a small interval for updating rit metrics<a name="line.92"></a>
+<span class="sourceLineNo">093</span> conf.setInt(AssignmentManager.RIT_CHORE_INTERVAL_MSEC_CONF_KEY, MSG_INTERVAL);<a name="line.93"></a>
+<span class="sourceLineNo">094</span><a name="line.94"></a>
+<span class="sourceLineNo">095</span> // keep rs online so it can report the failed opens.<a name="line.95"></a>
+<span class="sourceLineNo">096</span> conf.setBoolean(CoprocessorHost.ABORT_ON_ERROR_KEY, false);<a name="line.96"></a>
+<span class="sourceLineNo">097</span> TEST_UTIL.startMiniCluster(1);<a name="line.97"></a>
+<span class="sourceLineNo">098</span> CLUSTER = TEST_UTIL.getHBaseCluster();<a name="line.98"></a>
+<span class="sourceLineNo">099</span> MASTER = CLUSTER.getMaster();<a name="line.99"></a>
+<span class="sourceLineNo">100</span> }<a name="line.100"></a>
+<span class="sourceLineNo">101</span><a name="line.101"></a>
+<span class="sourceLineNo">102</span> @AfterClass<a name="line.102"></a>
+<span class="sourceLineNo">103</span> public static void after() throws Exception {<a name="line.103"></a>
+<span class="sourceLineNo">104</span> LOG.info("AFTER {} <= IS THIS NULL?", TEST_UTIL);<a name="line.104"></a>
+<span class="sourceLineNo">105</span> TEST_UTIL.shutdownMiniCluster();<a name="line.105"></a>
+<span class="sourceLineNo">106</span> }<a name="line.106"></a>
+<span class="sourceLineNo">107</span><a name="line.107"></a>
+<span class="sourceLineNo">108</span> @Test<a name="line.108"></a>
+<span class="sourceLineNo">109</span> public void testRITAssignmentManagerMetrics() throws Exception {<a name="line.109"></a>
+<span class="sourceLineNo">110</span> final TableName TABLENAME = TableName.valueOf(name.getMethodName());<a name="line.110"></a>
+<span class="sourceLineNo">111</span> final byte[] FAMILY = Bytes.toBytes("family");<a name="line.111"></a>
+<span class="sourceLineNo">112</span> try (Table table = TEST_UTIL.createTable(TABLENAME, FAMILY)){<a name="line.112"></a>
+<span class="sourceLineNo">113</span> final byte[] row = Bytes.toBytes("row");<a name="line.113"></a>
+<span class="sourceLineNo">114</span> final byte[] qualifier = Bytes.toBytes("qualifier");<a name="line.114"></a>
+<span class="sourceLineNo">115</span> final byte[] value = Bytes.toBytes("value");<a name="line.115"></a>
+<span class="sourceLineNo">116</span><a name="line.116"></a>
+<span class="sourceLineNo">117</span> Put put = new Put(row);<a name="line.117"></a>
+<span class="sourceLineNo">118</span> put.addColumn(FAMILY, qualifier, value);<a name="line.118"></a>
+<span class="sourceLineNo">119</span> table.put(put);<a name="line.119"></a>
<span class="sourceLineNo">120</span><a name="line.120"></a>
-<span class="sourceLineNo">121</span> // check the RIT is 0<a name="line.121"></a>
-<span class="sourceLineNo">122</span> MetricsAssignmentManagerSource amSource =<a name="line.122"></a>
-<span class="sourceLineNo">123</span> MASTER.getAssignmentManager().getAssignmentManagerMetrics().getMetricsProcSource();<a name="line.123"></a>
-<span class="sourceLineNo">124</span><a name="line.124"></a>
-<span class="sourceLineNo">125</span> METRICS_HELPER.assertGauge(MetricsAssignmentManagerSource.RIT_COUNT_NAME, 0, amSource);<a name="line.125"></a>
-<span class="sourceLineNo">126</span> METRICS_HELPER.assertGauge(MetricsAssignmentManagerSource.RIT_COUNT_OVER_THRESHOLD_NAME, 0,<a name="line.126"></a>
-<span class="sourceLineNo">127</span> amSource);<a name="line.127"></a>
-<span class="sourceLineNo">128</span><a name="line.128"></a>
-<span class="sourceLineNo">129</span> // alter table with a non-existing coprocessor<a name="line.129"></a>
-<span class="sourceLineNo">130</span><a name="line.130"></a>
-<span class="sourceLineNo">131</span> TableDescriptor htd = TableDescriptorBuilder.newBuilder(TABLENAME)<a name="line.131"></a>
-<span class="sourceLineNo">132</span> .setColumnFamily(ColumnFamilyDescriptorBuilder.of(FAMILY))<a name="line.132"></a>
-<span class="sourceLineNo">133</span> .setCoprocessor(CoprocessorDescriptorBuilder.newBuilder("com.foo.FooRegionObserver")<a name="line.133"></a>
-<span class="sourceLineNo">134</span> .setJarPath("hdfs:///foo.jar")<a name="line.134"></a>
-<span class="sourceLineNo">135</span> .setPriority(1001)<a name="line.135"></a>
-<span class="sourceLineNo">136</span> .setProperty("arg1", "1")<a name="line.136"></a>
-<span class="sourceLineNo">137</span> .setProperty("arg2", "2")<a name="line.137"></a>
-<span class="sourceLineNo">138</span> .build())<a name="line.138"></a>
-<span class="sourceLineNo">139</span> .build();<a name="line.139"></a>
-<span class="sourceLineNo">140</span> try {<a name="line.140"></a>
-<span class="sourceLineNo">141</span> TEST_UTIL.getAdmin().modifyTable(htd);<a name="line.141"></a>
-<span class="sourceLineNo">142</span> fail("Expected region failed to open");<a name="line.142"></a>
-<span class="sourceLineNo">143</span> } catch (IOException e) {<a name="line.143"></a>
-<span class="sourceLineNo">144</span> // expected, the RS will crash and the assignment will spin forever waiting for a RS<a name="line.144"></a>
-<span class="sourceLineNo">145</span> // to assign the region. the region will not go to FAILED_OPEN because in this case<a name="line.145"></a>
-<span class="sourceLineNo">146</span> // we have just one RS and it will do one retry.<a name="line.146"></a>
-<span class="sourceLineNo">147</span> LOG.info("Expected error", e);<a name="line.147"></a>
-<span class="sourceLineNo">148</span> }<a name="line.148"></a>
-<span class="sourceLineNo">149</span><a name="line.149"></a>
-<span class="sourceLineNo">150</span> // Sleep 3 seconds, wait for doMetrics chore catching up<a name="line.150"></a>
-<span class="sourceLineNo">151</span> Thread.sleep(MSG_INTERVAL * 3);<a name="line.151"></a>
-<span class="sourceLineNo">152</span> METRICS_HELPER.assertGauge(MetricsAssignmentManagerSource.RIT_COUNT_NAME, 1, amSource);<a name="line.152"></a>
-<span class="sourceLineNo">153</span> METRICS_HELPER.assertGauge(MetricsAssignmentManagerSource.RIT_COUNT_OVER_THRESHOLD_NAME, 1,<a name="line.153"></a>
-<span class="sourceLineNo">154</span> amSource);<a name="line.154"></a>
-<span class="sourceLineNo">155</span> }<a name="line.155"></a>
-<span class="sourceLineNo">156</span> }<a name="line.156"></a>
-<span class="sourceLineNo">157</span>}<a name="line.157"></a>
+<span class="sourceLineNo">121</span> // Sleep 3 seconds, wait for doMetrics chore catching up<a name="line.121"></a>
+<span class="sourceLineNo">122</span> Thread.sleep(MSG_INTERVAL * 3);<a name="line.122"></a>
+<span class="sourceLineNo">123</span><a name="line.123"></a>
+<span class="sourceLineNo">124</span> // check the RIT is 0<a name="line.124"></a>
+<span class="sourceLineNo">125</span> MetricsAssignmentManagerSource amSource =<a name="line.125"></a>
+<span class="sourceLineNo">126</span> MASTER.getAssignmentManager().getAssignmentManagerMetrics().getMetricsProcSource();<a name="line.126"></a>
+<span class="sourceLineNo">127</span><a name="line.127"></a>
+<span class="sourceLineNo">128</span> METRICS_HELPER.assertGauge(MetricsAssignmentManagerSource.RIT_COUNT_NAME, 0, amSource);<a name="line.128"></a>
+<span class="sourceLineNo">129</span> METRICS_HELPER.assertGauge(MetricsAssignmentManagerSource.RIT_COUNT_OVER_THRESHOLD_NAME, 0,<a name="line.129"></a>
+<span class="sourceLineNo">130</span> amSource);<a name="line.130"></a>
+<span class="sourceLineNo">131</span><a name="line.131"></a>
+<span class="sourceLineNo">132</span> // alter table with a non-existing coprocessor<a name="line.132"></a>
+<span class="sourceLineNo">133</span><a name="line.133"></a>
+<span class="sourceLineNo">134</span> TableDescriptor htd = TableDescriptorBuilder.newBuilder(TABLENAME)<a name="line.134"></a>
+<span class="sourceLineNo">135</span> .setColumnFamily(ColumnFamilyDescriptorBuilder.of(FAMILY))<a name="line.135"></a>
+<span class="sourceLineNo">136</span> .setCoprocessor(CoprocessorDescriptorBuilder.newBuilder("com.foo.FooRegionObserver")<a name="line.136"></a>
+<span class="sourceLineNo">137</span> .setJarPath("hdfs:///foo.jar")<a name="line.137"></a>
+<span class="sourceLineNo">138</span> .setPriority(1001)<a name="line.138"></a>
+<span class="sourceLineNo">139</span> .setProperty("arg1", "1")<a name="line.139"></a>
+<span class="sourceLineNo">140</span> .setProperty("arg2", "2")<a name="line.140"></a>
+<span class="sourceLineNo">141</span> .build())<a name="line.141"></a>
+<span class="sourceLineNo">142</span> .build();<a name="line.142"></a>
+<span class="sourceLineNo">143</span> try {<a name="line.143"></a>
+<span class="sourceLineNo">144</span> TEST_UTIL.getAdmin().modifyTable(htd);<a name="line.144"></a>
+<span class="sourceLineNo">145</span> fail("Expected region failed to open");<a name="line.145"></a>
+<span class="sourceLineNo">146</span> } catch (IOException e) {<a name="line.146"></a>
+<span class="sourceLineNo">147</span> // expected, the RS will crash and the assignment will spin forever waiting for a RS<a name="line.147"></a>
+<span class="sourceLineNo">148</span> // to assign the region. the region will not go to FAILED_OPEN because in this case<a name="line.148"></a>
+<span class="sourceLineNo">149</span> // we have just one RS and it will do one retry.<a name="line.149"></a>
+<span class="sourceLineNo">150</span> LOG.info("Expected error", e);<a name="line.150"></a>
+<span class="sourceLineNo">151</span> }<a name="line.151"></a>
+<span class="sourceLineNo">152</span><a name="line.152"></a>
+<span class="sourceLineNo">153</span> // Sleep 3 seconds, wait for doMetrics chore catching up<a name="line.153"></a>
+<span class="sourceLineNo">154</span> // the rit count consists of rit and failed opens. see RegionInTransitionStat#update<a name="line.154"></a>
+<span class="sourceLineNo">155</span> // Waiting for the completion of rit makes the assert stable.<a name="line.155"></a>
+<span class="sourceLineNo">156</span> TEST_UTIL.waitUntilNoRegionsInTransition();<a name="line.156"></a>
+<span class="sourceLineNo">157</span> Thread.sleep(MSG_INTERVAL * 3);<a name="line.157"></a>
+<span class="sourceLineNo">158</span> METRICS_HELPER.assertGauge(MetricsAssignmentManagerSource.RIT_COUNT_NAME, 1, amSource);<a name="line.158"></a>
+<span class="sourceLineNo">159</span> METRICS_HELPER.assertGauge(MetricsAssignmentManagerSource.RIT_COUNT_OVER_THRESHOLD_NAME, 1,<a name="line.159"></a>
+<span class="sourceLineNo">160</span> amSource);<a name="line.160"></a>
+<span class="sourceLineNo">161</span> }<a name="line.161"></a>
+<span class="sourceLineNo">162</span> }<a name="line.162"></a>
+<span class="sourceLineNo">163</span>}<a name="line.163"></a>
[13/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionChore.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionChore.html b/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionChore.html
index 078e4b1..fe1e077 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionChore.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionChore.html
@@ -1265,700 +1265,701 @@
<span class="sourceLineNo">1257</span> } else if (localState == State.OFFLINE || regionInfo.isOffline()) {<a name="line.1257"></a>
<span class="sourceLineNo">1258</span> regionStates.addToOfflineRegions(regionNode);<a name="line.1258"></a>
<span class="sourceLineNo">1259</span> } else if (localState == State.CLOSED && getTableStateManager().<a name="line.1259"></a>
-<span class="sourceLineNo">1260</span> isTableState(regionNode.getTable(), TableState.State.DISABLED)) {<a name="line.1260"></a>
-<span class="sourceLineNo">1261</span> // The region is CLOSED and the table is DISABLED, there is nothing to schedule;<a name="line.1261"></a>
-<span class="sourceLineNo">1262</span> // the region is inert.<a name="line.1262"></a>
-<span class="sourceLineNo">1263</span> } else {<a name="line.1263"></a>
-<span class="sourceLineNo">1264</span> // These regions should have a procedure in replay<a name="line.1264"></a>
-<span class="sourceLineNo">1265</span> regionStates.addRegionInTransition(regionNode, null);<a name="line.1265"></a>
-<span class="sourceLineNo">1266</span> }<a name="line.1266"></a>
-<span class="sourceLineNo">1267</span> }<a name="line.1267"></a>
-<span class="sourceLineNo">1268</span> }<a name="line.1268"></a>
-<span class="sourceLineNo">1269</span> }<a name="line.1269"></a>
-<span class="sourceLineNo">1270</span> });<a name="line.1270"></a>
-<span class="sourceLineNo">1271</span><a name="line.1271"></a>
-<span class="sourceLineNo">1272</span> // every assignment is blocked until meta is loaded.<a name="line.1272"></a>
-<span class="sourceLineNo">1273</span> wakeMetaLoadedEvent();<a name="line.1273"></a>
-<span class="sourceLineNo">1274</span> }<a name="line.1274"></a>
-<span class="sourceLineNo">1275</span><a name="line.1275"></a>
-<span class="sourceLineNo">1276</span> /**<a name="line.1276"></a>
-<span class="sourceLineNo">1277</span> * Look at what is in meta and the list of servers that have checked in and make reconciliation.<a name="line.1277"></a>
-<span class="sourceLineNo">1278</span> * We cannot tell definitively the difference between a clean shutdown and a cluster that has<a name="line.1278"></a>
-<span class="sourceLineNo">1279</span> * been crashed down. At this stage of a Master startup, they look the same: they have the<a name="line.1279"></a>
-<span class="sourceLineNo">1280</span> * same state in hbase:meta. We could do detective work probing ZK and the FS for old WALs to<a name="line.1280"></a>
-<span class="sourceLineNo">1281</span> * split but SCP does this already so just let it do its job.<a name="line.1281"></a>
-<span class="sourceLineNo">1282</span> * <p>>The profiles of clean shutdown and cluster crash-down are the same because on clean<a name="line.1282"></a>
-<span class="sourceLineNo">1283</span> * shutdown currently, we do not update hbase:meta with region close state (In AMv2, region<a name="line.1283"></a>
-<span class="sourceLineNo">1284</span> * state is kept in hbse:meta). Usually the master runs all region transitions as of AMv2 but on<a name="line.1284"></a>
-<span class="sourceLineNo">1285</span> * cluster controlled shutdown, the RegionServers close all their regions only reporting the<a name="line.1285"></a>
-<span class="sourceLineNo">1286</span> * final change to the Master. Currently this report is ignored. Later we could take it and<a name="line.1286"></a>
-<span class="sourceLineNo">1287</span> * update as many regions as we can before hbase:meta goes down or have the master run the<a name="line.1287"></a>
-<span class="sourceLineNo">1288</span> * close of all regions out on the cluster but we may never be able to achieve the proper state on<a name="line.1288"></a>
-<span class="sourceLineNo">1289</span> * all regions (at least not w/o lots of painful manipulations and waiting) so clean shutdown<a name="line.1289"></a>
-<span class="sourceLineNo">1290</span> * might not be possible especially on big clusters.... And clean shutdown will take time. Given<a name="line.1290"></a>
-<span class="sourceLineNo">1291</span> * this current state of affairs, we just run ServerCrashProcedure in both cases. It will always<a name="line.1291"></a>
-<span class="sourceLineNo">1292</span> * do the right thing.<a name="line.1292"></a>
-<span class="sourceLineNo">1293</span> * @return True if for sure this is a failover where a Master is starting up into an already<a name="line.1293"></a>
-<span class="sourceLineNo">1294</span> * running cluster.<a name="line.1294"></a>
-<span class="sourceLineNo">1295</span> */<a name="line.1295"></a>
-<span class="sourceLineNo">1296</span> // The assumption here is that if RSs are crashing while we are executing this<a name="line.1296"></a>
-<span class="sourceLineNo">1297</span> // they will be handled by the SSH that are put in the ServerManager deadservers "queue".<a name="line.1297"></a>
-<span class="sourceLineNo">1298</span> private boolean processofflineServersWithOnlineRegions() {<a name="line.1298"></a>
-<span class="sourceLineNo">1299</span> boolean deadServers = !master.getServerManager().getDeadServers().isEmpty();<a name="line.1299"></a>
-<span class="sourceLineNo">1300</span> final Set<ServerName> offlineServersWithOnlineRegions = new HashSet<>();<a name="line.1300"></a>
-<span class="sourceLineNo">1301</span> int size = regionStates.getRegionStateNodes().size();<a name="line.1301"></a>
-<span class="sourceLineNo">1302</span> final List<RegionInfo> offlineRegionsToAssign = new ArrayList<>(size);<a name="line.1302"></a>
-<span class="sourceLineNo">1303</span> // If deadservers then its a failover, else, we are not sure yet.<a name="line.1303"></a>
-<span class="sourceLineNo">1304</span> boolean failover = deadServers;<a name="line.1304"></a>
-<span class="sourceLineNo">1305</span> for (RegionStateNode regionNode: regionStates.getRegionStateNodes()) {<a name="line.1305"></a>
-<span class="sourceLineNo">1306</span> // Region State can be OPEN even if we did controlled cluster shutdown; Master does not close<a name="line.1306"></a>
-<span class="sourceLineNo">1307</span> // the regions in this case. The RegionServer does the close so hbase:meta is state in<a name="line.1307"></a>
-<span class="sourceLineNo">1308</span> // hbase:meta is not updated -- Master does all updates -- and is left with OPEN as region<a name="line.1308"></a>
-<span class="sourceLineNo">1309</span> // state in meta. How to tell difference between ordered shutdown and crashed-down cluster<a name="line.1309"></a>
-<span class="sourceLineNo">1310</span> // then? We can't. Not currently. Perhaps if we updated hbase:meta with CLOSED on ordered<a name="line.1310"></a>
-<span class="sourceLineNo">1311</span> // shutdown. This would slow shutdown though and not all edits would make it in anyways.<a name="line.1311"></a>
-<span class="sourceLineNo">1312</span> // TODO: Examine.<a name="line.1312"></a>
-<span class="sourceLineNo">1313</span> // Because we can't be sure it an ordered shutdown, we run ServerCrashProcedure always.<a name="line.1313"></a>
-<span class="sourceLineNo">1314</span> // ServerCrashProcedure will try to retain old deploy when it goes to assign.<a name="line.1314"></a>
-<span class="sourceLineNo">1315</span> if (regionNode.getState() == State.OPEN) {<a name="line.1315"></a>
-<span class="sourceLineNo">1316</span> final ServerName serverName = regionNode.getRegionLocation();<a name="line.1316"></a>
-<span class="sourceLineNo">1317</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1317"></a>
-<span class="sourceLineNo">1318</span> offlineServersWithOnlineRegions.add(serverName);<a name="line.1318"></a>
-<span class="sourceLineNo">1319</span> } else {<a name="line.1319"></a>
-<span class="sourceLineNo">1320</span> // Server is online. This a failover. Master is starting into already-running cluster.<a name="line.1320"></a>
-<span class="sourceLineNo">1321</span> failover = true;<a name="line.1321"></a>
-<span class="sourceLineNo">1322</span> }<a name="line.1322"></a>
-<span class="sourceLineNo">1323</span> } else if (regionNode.getState() == State.OFFLINE) {<a name="line.1323"></a>
-<span class="sourceLineNo">1324</span> if (isTableEnabled(regionNode.getTable())) {<a name="line.1324"></a>
-<span class="sourceLineNo">1325</span> offlineRegionsToAssign.add(regionNode.getRegionInfo());<a name="line.1325"></a>
-<span class="sourceLineNo">1326</span> }<a name="line.1326"></a>
-<span class="sourceLineNo">1327</span> }<a name="line.1327"></a>
-<span class="sourceLineNo">1328</span> }<a name="line.1328"></a>
-<span class="sourceLineNo">1329</span> // Kill servers with online regions just-in-case. Runs ServerCrashProcedure.<a name="line.1329"></a>
-<span class="sourceLineNo">1330</span> for (ServerName serverName: offlineServersWithOnlineRegions) {<a name="line.1330"></a>
-<span class="sourceLineNo">1331</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1331"></a>
-<span class="sourceLineNo">1332</span> LOG.info("KILL RegionServer=" + serverName + " hosting regions but not online.");<a name="line.1332"></a>
-<span class="sourceLineNo">1333</span> killRegionServer(serverName);<a name="line.1333"></a>
-<span class="sourceLineNo">1334</span> }<a name="line.1334"></a>
-<span class="sourceLineNo">1335</span> }<a name="line.1335"></a>
-<span class="sourceLineNo">1336</span> setFailoverCleanupDone(true);<a name="line.1336"></a>
-<span class="sourceLineNo">1337</span><a name="line.1337"></a>
-<span class="sourceLineNo">1338</span> // Assign offline regions. Uses round-robin.<a name="line.1338"></a>
-<span class="sourceLineNo">1339</span> if (offlineRegionsToAssign.size() > 0) {<a name="line.1339"></a>
-<span class="sourceLineNo">1340</span> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().<a name="line.1340"></a>
-<span class="sourceLineNo">1341</span> createRoundRobinAssignProcedures(offlineRegionsToAssign));<a name="line.1341"></a>
-<span class="sourceLineNo">1342</span> }<a name="line.1342"></a>
-<span class="sourceLineNo">1343</span><a name="line.1343"></a>
-<span class="sourceLineNo">1344</span> return failover;<a name="line.1344"></a>
-<span class="sourceLineNo">1345</span> }<a name="line.1345"></a>
-<span class="sourceLineNo">1346</span><a name="line.1346"></a>
-<span class="sourceLineNo">1347</span> /**<a name="line.1347"></a>
-<span class="sourceLineNo">1348</span> * Used by ServerCrashProcedure to make sure AssignmentManager has completed<a name="line.1348"></a>
-<span class="sourceLineNo">1349</span> * the failover cleanup before re-assigning regions of dead servers. So that<a name="line.1349"></a>
-<span class="sourceLineNo">1350</span> * when re-assignment happens, AssignmentManager has proper region states.<a name="line.1350"></a>
-<span class="sourceLineNo">1351</span> */<a name="line.1351"></a>
-<span class="sourceLineNo">1352</span> public boolean isFailoverCleanupDone() {<a name="line.1352"></a>
-<span class="sourceLineNo">1353</span> return failoverCleanupDone.isReady();<a name="line.1353"></a>
-<span class="sourceLineNo">1354</span> }<a name="line.1354"></a>
-<span class="sourceLineNo">1355</span><a name="line.1355"></a>
-<span class="sourceLineNo">1356</span> /**<a name="line.1356"></a>
-<span class="sourceLineNo">1357</span> * Used by ServerCrashProcedure tests verify the ability to suspend the<a name="line.1357"></a>
-<span class="sourceLineNo">1358</span> * execution of the ServerCrashProcedure.<a name="line.1358"></a>
-<span class="sourceLineNo">1359</span> */<a name="line.1359"></a>
-<span class="sourceLineNo">1360</span> @VisibleForTesting<a name="line.1360"></a>
-<span class="sourceLineNo">1361</span> public void setFailoverCleanupDone(final boolean b) {<a name="line.1361"></a>
-<span class="sourceLineNo">1362</span> master.getMasterProcedureExecutor().getEnvironment()<a name="line.1362"></a>
-<span class="sourceLineNo">1363</span> .setEventReady(failoverCleanupDone, b);<a name="line.1363"></a>
-<span class="sourceLineNo">1364</span> }<a name="line.1364"></a>
-<span class="sourceLineNo">1365</span><a name="line.1365"></a>
-<span class="sourceLineNo">1366</span> public ProcedureEvent getFailoverCleanupEvent() {<a name="line.1366"></a>
-<span class="sourceLineNo">1367</span> return failoverCleanupDone;<a name="line.1367"></a>
-<span class="sourceLineNo">1368</span> }<a name="line.1368"></a>
-<span class="sourceLineNo">1369</span><a name="line.1369"></a>
-<span class="sourceLineNo">1370</span> /**<a name="line.1370"></a>
-<span class="sourceLineNo">1371</span> * Used to check if the failover cleanup is done.<a name="line.1371"></a>
-<span class="sourceLineNo">1372</span> * if not we throw PleaseHoldException since we are rebuilding the RegionStates<a name="line.1372"></a>
-<span class="sourceLineNo">1373</span> * @param hri region to check if it is already rebuild<a name="line.1373"></a>
-<span class="sourceLineNo">1374</span> * @throws PleaseHoldException if the failover cleanup is not completed<a name="line.1374"></a>
-<span class="sourceLineNo">1375</span> */<a name="line.1375"></a>
-<span class="sourceLineNo">1376</span> private void checkFailoverCleanupCompleted(final RegionInfo hri) throws PleaseHoldException {<a name="line.1376"></a>
-<span class="sourceLineNo">1377</span> if (!isRunning()) {<a name="line.1377"></a>
-<span class="sourceLineNo">1378</span> throw new PleaseHoldException("AssignmentManager not running");<a name="line.1378"></a>
-<span class="sourceLineNo">1379</span> }<a name="line.1379"></a>
-<span class="sourceLineNo">1380</span><a name="line.1380"></a>
-<span class="sourceLineNo">1381</span> // TODO: can we avoid throwing an exception if hri is already loaded?<a name="line.1381"></a>
-<span class="sourceLineNo">1382</span> // at the moment we bypass only meta<a name="line.1382"></a>
-<span class="sourceLineNo">1383</span> boolean meta = isMetaRegion(hri);<a name="line.1383"></a>
-<span class="sourceLineNo">1384</span> boolean cleanup = isFailoverCleanupDone();<a name="line.1384"></a>
-<span class="sourceLineNo">1385</span> if (!isMetaRegion(hri) && !isFailoverCleanupDone()) {<a name="line.1385"></a>
-<span class="sourceLineNo">1386</span> String msg = "Master not fully online; hbase:meta=" + meta + ", failoverCleanup=" + cleanup;<a name="line.1386"></a>
-<span class="sourceLineNo">1387</span> throw new PleaseHoldException(msg);<a name="line.1387"></a>
-<span class="sourceLineNo">1388</span> }<a name="line.1388"></a>
-<span class="sourceLineNo">1389</span> }<a name="line.1389"></a>
-<span class="sourceLineNo">1390</span><a name="line.1390"></a>
-<span class="sourceLineNo">1391</span> // ============================================================================================<a name="line.1391"></a>
-<span class="sourceLineNo">1392</span> // TODO: Metrics<a name="line.1392"></a>
-<span class="sourceLineNo">1393</span> // ============================================================================================<a name="line.1393"></a>
-<span class="sourceLineNo">1394</span> public int getNumRegionsOpened() {<a name="line.1394"></a>
-<span class="sourceLineNo">1395</span> // TODO: Used by TestRegionPlacement.java and assume monotonically increasing value<a name="line.1395"></a>
-<span class="sourceLineNo">1396</span> return 0;<a name="line.1396"></a>
-<span class="sourceLineNo">1397</span> }<a name="line.1397"></a>
-<span class="sourceLineNo">1398</span><a name="line.1398"></a>
-<span class="sourceLineNo">1399</span> public void submitServerCrash(final ServerName serverName, final boolean shouldSplitWal) {<a name="line.1399"></a>
-<span class="sourceLineNo">1400</span> boolean carryingMeta = isCarryingMeta(serverName);<a name="line.1400"></a>
-<span class="sourceLineNo">1401</span> ProcedureExecutor<MasterProcedureEnv> procExec = this.master.getMasterProcedureExecutor();<a name="line.1401"></a>
-<span class="sourceLineNo">1402</span> procExec.submitProcedure(new ServerCrashProcedure(procExec.getEnvironment(), serverName,<a name="line.1402"></a>
-<span class="sourceLineNo">1403</span> shouldSplitWal, carryingMeta));<a name="line.1403"></a>
-<span class="sourceLineNo">1404</span> LOG.debug("Added=" + serverName +<a name="line.1404"></a>
-<span class="sourceLineNo">1405</span> " to dead servers, submitted shutdown handler to be executed meta=" + carryingMeta);<a name="line.1405"></a>
-<span class="sourceLineNo">1406</span> }<a name="line.1406"></a>
-<span class="sourceLineNo">1407</span><a name="line.1407"></a>
-<span class="sourceLineNo">1408</span> public void offlineRegion(final RegionInfo regionInfo) {<a name="line.1408"></a>
-<span class="sourceLineNo">1409</span> // TODO used by MasterRpcServices ServerCrashProcedure<a name="line.1409"></a>
-<span class="sourceLineNo">1410</span> final RegionStateNode node = regionStates.getRegionStateNode(regionInfo);<a name="line.1410"></a>
-<span class="sourceLineNo">1411</span> if (node != null) node.offline();<a name="line.1411"></a>
-<span class="sourceLineNo">1412</span> }<a name="line.1412"></a>
-<span class="sourceLineNo">1413</span><a name="line.1413"></a>
-<span class="sourceLineNo">1414</span> public void onlineRegion(final RegionInfo regionInfo, final ServerName serverName) {<a name="line.1414"></a>
-<span class="sourceLineNo">1415</span> // TODO used by TestSplitTransactionOnCluster.java<a name="line.1415"></a>
-<span class="sourceLineNo">1416</span> }<a name="line.1416"></a>
-<span class="sourceLineNo">1417</span><a name="line.1417"></a>
-<span class="sourceLineNo">1418</span> public Map<ServerName, List<RegionInfo>> getSnapShotOfAssignment(<a name="line.1418"></a>
-<span class="sourceLineNo">1419</span> final Collection<RegionInfo> regions) {<a name="line.1419"></a>
-<span class="sourceLineNo">1420</span> return regionStates.getSnapShotOfAssignment(regions);<a name="line.1420"></a>
-<span class="sourceLineNo">1421</span> }<a name="line.1421"></a>
-<span class="sourceLineNo">1422</span><a name="line.1422"></a>
-<span class="sourceLineNo">1423</span> // ============================================================================================<a name="line.1423"></a>
-<span class="sourceLineNo">1424</span> // TODO: UTILS/HELPERS?<a name="line.1424"></a>
-<span class="sourceLineNo">1425</span> // ============================================================================================<a name="line.1425"></a>
-<span class="sourceLineNo">1426</span> /**<a name="line.1426"></a>
-<span class="sourceLineNo">1427</span> * Used by the client (via master) to identify if all regions have the schema updates<a name="line.1427"></a>
-<span class="sourceLineNo">1428</span> *<a name="line.1428"></a>
-<span class="sourceLineNo">1429</span> * @param tableName<a name="line.1429"></a>
-<span class="sourceLineNo">1430</span> * @return Pair indicating the status of the alter command (pending/total)<a name="line.1430"></a>
-<span class="sourceLineNo">1431</span> * @throws IOException<a name="line.1431"></a>
-<span class="sourceLineNo">1432</span> */<a name="line.1432"></a>
-<span class="sourceLineNo">1433</span> public Pair<Integer, Integer> getReopenStatus(TableName tableName) {<a name="line.1433"></a>
-<span class="sourceLineNo">1434</span> if (isTableDisabled(tableName)) return new Pair<Integer, Integer>(0, 0);<a name="line.1434"></a>
-<span class="sourceLineNo">1435</span><a name="line.1435"></a>
-<span class="sourceLineNo">1436</span> final List<RegionState> states = regionStates.getTableRegionStates(tableName);<a name="line.1436"></a>
-<span class="sourceLineNo">1437</span> int ritCount = 0;<a name="line.1437"></a>
-<span class="sourceLineNo">1438</span> for (RegionState regionState: states) {<a name="line.1438"></a>
-<span class="sourceLineNo">1439</span> if (!regionState.isOpened()) ritCount++;<a name="line.1439"></a>
-<span class="sourceLineNo">1440</span> }<a name="line.1440"></a>
-<span class="sourceLineNo">1441</span> return new Pair<Integer, Integer>(ritCount, states.size());<a name="line.1441"></a>
-<span class="sourceLineNo">1442</span> }<a name="line.1442"></a>
-<span class="sourceLineNo">1443</span><a name="line.1443"></a>
-<span class="sourceLineNo">1444</span> // ============================================================================================<a name="line.1444"></a>
-<span class="sourceLineNo">1445</span> // TODO: Region State In Transition<a name="line.1445"></a>
-<span class="sourceLineNo">1446</span> // ============================================================================================<a name="line.1446"></a>
-<span class="sourceLineNo">1447</span> protected boolean addRegionInTransition(final RegionStateNode regionNode,<a name="line.1447"></a>
-<span class="sourceLineNo">1448</span> final RegionTransitionProcedure procedure) {<a name="line.1448"></a>
-<span class="sourceLineNo">1449</span> return regionStates.addRegionInTransition(regionNode, procedure);<a name="line.1449"></a>
-<span class="sourceLineNo">1450</span> }<a name="line.1450"></a>
-<span class="sourceLineNo">1451</span><a name="line.1451"></a>
-<span class="sourceLineNo">1452</span> protected void removeRegionInTransition(final RegionStateNode regionNode,<a name="line.1452"></a>
-<span class="sourceLineNo">1453</span> final RegionTransitionProcedure procedure) {<a name="line.1453"></a>
-<span class="sourceLineNo">1454</span> regionStates.removeRegionInTransition(regionNode, procedure);<a name="line.1454"></a>
-<span class="sourceLineNo">1455</span> }<a name="line.1455"></a>
-<span class="sourceLineNo">1456</span><a name="line.1456"></a>
-<span class="sourceLineNo">1457</span> public boolean hasRegionsInTransition() {<a name="line.1457"></a>
-<span class="sourceLineNo">1458</span> return regionStates.hasRegionsInTransition();<a name="line.1458"></a>
-<span class="sourceLineNo">1459</span> }<a name="line.1459"></a>
-<span class="sourceLineNo">1460</span><a name="line.1460"></a>
-<span class="sourceLineNo">1461</span> public List<RegionStateNode> getRegionsInTransition() {<a name="line.1461"></a>
-<span class="sourceLineNo">1462</span> return regionStates.getRegionsInTransition();<a name="line.1462"></a>
-<span class="sourceLineNo">1463</span> }<a name="line.1463"></a>
-<span class="sourceLineNo">1464</span><a name="line.1464"></a>
-<span class="sourceLineNo">1465</span> public List<RegionInfo> getAssignedRegions() {<a name="line.1465"></a>
-<span class="sourceLineNo">1466</span> return regionStates.getAssignedRegions();<a name="line.1466"></a>
-<span class="sourceLineNo">1467</span> }<a name="line.1467"></a>
-<span class="sourceLineNo">1468</span><a name="line.1468"></a>
-<span class="sourceLineNo">1469</span> public RegionInfo getRegionInfo(final byte[] regionName) {<a name="line.1469"></a>
-<span class="sourceLineNo">1470</span> final RegionStateNode regionState = regionStates.getRegionStateNodeFromName(regionName);<a name="line.1470"></a>
-<span class="sourceLineNo">1471</span> return regionState != null ? regionState.getRegionInfo() : null;<a name="line.1471"></a>
-<span class="sourceLineNo">1472</span> }<a name="line.1472"></a>
-<span class="sourceLineNo">1473</span><a name="line.1473"></a>
-<span class="sourceLineNo">1474</span> // ============================================================================================<a name="line.1474"></a>
-<span class="sourceLineNo">1475</span> // TODO: Region Status update<a name="line.1475"></a>
-<span class="sourceLineNo">1476</span> // ============================================================================================<a name="line.1476"></a>
-<span class="sourceLineNo">1477</span> private void sendRegionOpenedNotification(final RegionInfo regionInfo,<a name="line.1477"></a>
-<span class="sourceLineNo">1478</span> final ServerName serverName) {<a name="line.1478"></a>
-<span class="sourceLineNo">1479</span> getBalancer().regionOnline(regionInfo, serverName);<a name="line.1479"></a>
-<span class="sourceLineNo">1480</span> if (!this.listeners.isEmpty()) {<a name="line.1480"></a>
-<span class="sourceLineNo">1481</span> for (AssignmentListener listener : this.listeners) {<a name="line.1481"></a>
-<span class="sourceLineNo">1482</span> listener.regionOpened(regionInfo, serverName);<a name="line.1482"></a>
-<span class="sourceLineNo">1483</span> }<a name="line.1483"></a>
-<span class="sourceLineNo">1484</span> }<a name="line.1484"></a>
-<span class="sourceLineNo">1485</span> }<a name="line.1485"></a>
-<span class="sourceLineNo">1486</span><a name="line.1486"></a>
-<span class="sourceLineNo">1487</span> private void sendRegionClosedNotification(final RegionInfo regionInfo) {<a name="line.1487"></a>
-<span class="sourceLineNo">1488</span> getBalancer().regionOffline(regionInfo);<a name="line.1488"></a>
-<span class="sourceLineNo">1489</span> if (!this.listeners.isEmpty()) {<a name="line.1489"></a>
-<span class="sourceLineNo">1490</span> for (AssignmentListener listener : this.listeners) {<a name="line.1490"></a>
-<span class="sourceLineNo">1491</span> listener.regionClosed(regionInfo);<a name="line.1491"></a>
-<span class="sourceLineNo">1492</span> }<a name="line.1492"></a>
-<span class="sourceLineNo">1493</span> }<a name="line.1493"></a>
-<span class="sourceLineNo">1494</span> }<a name="line.1494"></a>
-<span class="sourceLineNo">1495</span><a name="line.1495"></a>
-<span class="sourceLineNo">1496</span> public void markRegionAsOpening(final RegionStateNode regionNode) throws IOException {<a name="line.1496"></a>
-<span class="sourceLineNo">1497</span> synchronized (regionNode) {<a name="line.1497"></a>
-<span class="sourceLineNo">1498</span> regionNode.transitionState(State.OPENING, RegionStates.STATES_EXPECTED_ON_OPEN);<a name="line.1498"></a>
-<span class="sourceLineNo">1499</span> regionStates.addRegionToServer(regionNode);<a name="line.1499"></a>
-<span class="sourceLineNo">1500</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1500"></a>
-<span class="sourceLineNo">1501</span> }<a name="line.1501"></a>
-<span class="sourceLineNo">1502</span><a name="line.1502"></a>
-<span class="sourceLineNo">1503</span> // update the operation count metrics<a name="line.1503"></a>
-<span class="sourceLineNo">1504</span> metrics.incrementOperationCounter();<a name="line.1504"></a>
-<span class="sourceLineNo">1505</span> }<a name="line.1505"></a>
-<span class="sourceLineNo">1506</span><a name="line.1506"></a>
-<span class="sourceLineNo">1507</span> public void undoRegionAsOpening(final RegionStateNode regionNode) {<a name="line.1507"></a>
-<span class="sourceLineNo">1508</span> boolean opening = false;<a name="line.1508"></a>
-<span class="sourceLineNo">1509</span> synchronized (regionNode) {<a name="line.1509"></a>
-<span class="sourceLineNo">1510</span> if (regionNode.isInState(State.OPENING)) {<a name="line.1510"></a>
-<span class="sourceLineNo">1511</span> opening = true;<a name="line.1511"></a>
-<span class="sourceLineNo">1512</span> regionStates.removeRegionFromServer(regionNode.getRegionLocation(), regionNode);<a name="line.1512"></a>
-<span class="sourceLineNo">1513</span> }<a name="line.1513"></a>
-<span class="sourceLineNo">1514</span> // Should we update hbase:meta?<a name="line.1514"></a>
-<span class="sourceLineNo">1515</span> }<a name="line.1515"></a>
-<span class="sourceLineNo">1516</span> if (opening) {<a name="line.1516"></a>
-<span class="sourceLineNo">1517</span> // TODO: Metrics. Do opposite of metrics.incrementOperationCounter();<a name="line.1517"></a>
-<span class="sourceLineNo">1518</span> }<a name="line.1518"></a>
-<span class="sourceLineNo">1519</span> }<a name="line.1519"></a>
-<span class="sourceLineNo">1520</span><a name="line.1520"></a>
-<span class="sourceLineNo">1521</span> public void markRegionAsOpened(final RegionStateNode regionNode) throws IOException {<a name="line.1521"></a>
-<span class="sourceLineNo">1522</span> final RegionInfo hri = regionNode.getRegionInfo();<a name="line.1522"></a>
-<span class="sourceLineNo">1523</span> synchronized (regionNode) {<a name="line.1523"></a>
-<span class="sourceLineNo">1524</span> regionNode.transitionState(State.OPEN, RegionStates.STATES_EXPECTED_ON_OPEN);<a name="line.1524"></a>
-<span class="sourceLineNo">1525</span> if (isMetaRegion(hri)) {<a name="line.1525"></a>
-<span class="sourceLineNo">1526</span> // Usually we'd set a table ENABLED at this stage but hbase:meta is ALWAYs enabled, it<a name="line.1526"></a>
-<span class="sourceLineNo">1527</span> // can't be disabled -- so skip the RPC (besides... enabled is managed by TableStateManager<a name="line.1527"></a>
-<span class="sourceLineNo">1528</span> // which is backed by hbase:meta... Avoid setting ENABLED to avoid having to update state<a name="line.1528"></a>
-<span class="sourceLineNo">1529</span> // on table that contains state.<a name="line.1529"></a>
-<span class="sourceLineNo">1530</span> setMetaInitialized(hri, true);<a name="line.1530"></a>
-<span class="sourceLineNo">1531</span> }<a name="line.1531"></a>
-<span class="sourceLineNo">1532</span> regionStates.addRegionToServer(regionNode);<a name="line.1532"></a>
-<span class="sourceLineNo">1533</span> // TODO: OPENING Updates hbase:meta too... we need to do both here and there?<a name="line.1533"></a>
-<span class="sourceLineNo">1534</span> // That is a lot of hbase:meta writing.<a name="line.1534"></a>
-<span class="sourceLineNo">1535</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1535"></a>
-<span class="sourceLineNo">1536</span> sendRegionOpenedNotification(hri, regionNode.getRegionLocation());<a name="line.1536"></a>
-<span class="sourceLineNo">1537</span> }<a name="line.1537"></a>
-<span class="sourceLineNo">1538</span> }<a name="line.1538"></a>
-<span class="sourceLineNo">1539</span><a name="line.1539"></a>
-<span class="sourceLineNo">1540</span> public void markRegionAsClosing(final RegionStateNode regionNode) throws IOException {<a name="line.1540"></a>
-<span class="sourceLineNo">1541</span> final RegionInfo hri = regionNode.getRegionInfo();<a name="line.1541"></a>
-<span class="sourceLineNo">1542</span> synchronized (regionNode) {<a name="line.1542"></a>
-<span class="sourceLineNo">1543</span> regionNode.transitionState(State.CLOSING, RegionStates.STATES_EXPECTED_ON_CLOSE);<a name="line.1543"></a>
-<span class="sourceLineNo">1544</span> // Set meta has not initialized early. so people trying to create/edit tables will wait<a name="line.1544"></a>
-<span class="sourceLineNo">1545</span> if (isMetaRegion(hri)) {<a name="line.1545"></a>
-<span class="sourceLineNo">1546</span> setMetaInitialized(hri, false);<a name="line.1546"></a>
-<span class="sourceLineNo">1547</span> }<a name="line.1547"></a>
-<span class="sourceLineNo">1548</span> regionStates.addRegionToServer(regionNode);<a name="line.1548"></a>
-<span class="sourceLineNo">1549</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1549"></a>
-<span class="sourceLineNo">1550</span> }<a name="line.1550"></a>
-<span class="sourceLineNo">1551</span><a name="line.1551"></a>
-<span class="sourceLineNo">1552</span> // update the operation count metrics<a name="line.1552"></a>
-<span class="sourceLineNo">1553</span> metrics.incrementOperationCounter();<a name="line.1553"></a>
-<span class="sourceLineNo">1554</span> }<a name="line.1554"></a>
-<span class="sourceLineNo">1555</span><a name="line.1555"></a>
-<span class="sourceLineNo">1556</span> public void undoRegionAsClosing(final RegionStateNode regionNode) {<a name="line.1556"></a>
-<span class="sourceLineNo">1557</span> // TODO: Metrics. Do opposite of metrics.incrementOperationCounter();<a name="line.1557"></a>
-<span class="sourceLineNo">1558</span> // There is nothing to undo?<a name="line.1558"></a>
-<span class="sourceLineNo">1559</span> }<a name="line.1559"></a>
-<span class="sourceLineNo">1560</span><a name="line.1560"></a>
-<span class="sourceLineNo">1561</span> public void markRegionAsClosed(final RegionStateNode regionNode) throws IOException {<a name="line.1561"></a>
-<span class="sourceLineNo">1562</span> final RegionInfo hri = regionNode.getRegionInfo();<a name="line.1562"></a>
-<span class="sourceLineNo">1563</span> synchronized (regionNode) {<a name="line.1563"></a>
-<span class="sourceLineNo">1564</span> regionNode.transitionState(State.CLOSED, RegionStates.STATES_EXPECTED_ON_CLOSE);<a name="line.1564"></a>
-<span class="sourceLineNo">1565</span> regionStates.removeRegionFromServer(regionNode.getRegionLocation(), regionNode);<a name="line.1565"></a>
-<span class="sourceLineNo">1566</span> regionNode.setLastHost(regionNode.getRegionLocation());<a name="line.1566"></a>
-<span class="sourceLineNo">1567</span> regionNode.setRegionLocation(null);<a name="line.1567"></a>
-<span class="sourceLineNo">1568</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1568"></a>
-<span class="sourceLineNo">1569</span> sendRegionClosedNotification(hri);<a name="line.1569"></a>
-<span class="sourceLineNo">1570</span> }<a name="line.1570"></a>
-<span class="sourceLineNo">1571</span> }<a name="line.1571"></a>
-<span class="sourceLineNo">1572</span><a name="line.1572"></a>
-<span class="sourceLineNo">1573</span> public void markRegionAsSplit(final RegionInfo parent, final ServerName serverName,<a name="line.1573"></a>
-<span class="sourceLineNo">1574</span> final RegionInfo daughterA, final RegionInfo daughterB) throws IOException {<a name="line.1574"></a>
-<span class="sourceLineNo">1575</span> // Update hbase:meta. Parent will be marked offline and split up in hbase:meta.<a name="line.1575"></a>
-<span class="sourceLineNo">1576</span> // The parent stays in regionStates until cleared when removed by CatalogJanitor.<a name="line.1576"></a>
-<span class="sourceLineNo">1577</span> // Update its state in regionStates to it shows as offline and split when read<a name="line.1577"></a>
-<span class="sourceLineNo">1578</span> // later figuring what regions are in a table and what are not: see<a name="line.1578"></a>
-<span class="sourceLineNo">1579</span> // regionStates#getRegionsOfTable<a name="line.1579"></a>
-<span class="sourceLineNo">1580</span> final RegionStateNode node = regionStates.getOrCreateRegionStateNode(parent);<a name="line.1580"></a>
-<span class="sourceLineNo">1581</span> node.setState(State.SPLIT);<a name="line.1581"></a>
-<span class="sourceLineNo">1582</span> final RegionStateNode nodeA = regionStates.getOrCreateRegionStateNode(daughterA);<a name="line.1582"></a>
-<span class="sourceLineNo">1583</span> nodeA.setState(State.SPLITTING_NEW);<a name="line.1583"></a>
-<span class="sourceLineNo">1584</span> final RegionStateNode nodeB = regionStates.getOrCreateRegionStateNode(daughterB);<a name="line.1584"></a>
-<span class="sourceLineNo">1585</span> nodeB.setState(State.SPLITTING_NEW);<a name="line.1585"></a>
-<span class="sourceLineNo">1586</span><a name="line.1586"></a>
-<span class="sourceLineNo">1587</span> regionStateStore.splitRegion(parent, daughterA, daughterB, serverName);<a name="line.1587"></a>
-<span class="sourceLineNo">1588</span> if (shouldAssignFavoredNodes(parent)) {<a name="line.1588"></a>
-<span class="sourceLineNo">1589</span> List<ServerName> onlineServers = this.master.getServerManager().getOnlineServersList();<a name="line.1589"></a>
-<span class="sourceLineNo">1590</span> ((FavoredNodesPromoter)getBalancer()).<a name="line.1590"></a>
-<span class="sourceLineNo">1591</span> generateFavoredNodesForDaughter(onlineServers, parent, daughterA, daughterB);<a name="line.1591"></a>
-<span class="sourceLineNo">1592</span> }<a name="line.1592"></a>
-<span class="sourceLineNo">1593</span> }<a name="line.1593"></a>
-<span class="sourceLineNo">1594</span><a name="line.1594"></a>
-<span class="sourceLineNo">1595</span> /**<a name="line.1595"></a>
-<span class="sourceLineNo">1596</span> * When called here, the merge has happened. The two merged regions have been<a name="line.1596"></a>
-<span class="sourceLineNo">1597</span> * unassigned and the above markRegionClosed has been called on each so they have been<a name="line.1597"></a>
-<span class="sourceLineNo">1598</span> * disassociated from a hosting Server. The merged region will be open after this call. The<a name="line.1598"></a>
-<span class="sourceLineNo">1599</span> * merged regions are removed from hbase:meta below> Later they are deleted from the filesystem<a name="line.1599"></a>
-<span class="sourceLineNo">1600</span> * by the catalog janitor running against hbase:meta. It notices when the merged region no<a name="line.1600"></a>
-<span class="sourceLineNo">1601</span> * longer holds references to the old regions.<a name="line.1601"></a>
-<span class="sourceLineNo">1602</span> */<a name="line.1602"></a>
-<span class="sourceLineNo">1603</span> public void markRegionAsMerged(final RegionInfo child, final ServerName serverName,<a name="line.1603"></a>
-<span class="sourceLineNo">1604</span> final RegionInfo mother, final RegionInfo father) throws IOException {<a name="line.1604"></a>
-<span class="sourceLineNo">1605</span> final RegionStateNode node = regionStates.getOrCreateRegionStateNode(child);<a name="line.1605"></a>
-<span class="sourceLineNo">1606</span> node.setState(State.MERGED);<a name="line.1606"></a>
-<span class="sourceLineNo">1607</span> regionStates.deleteRegion(mother);<a name="line.1607"></a>
-<span class="sourceLineNo">1608</span> regionStates.deleteRegion(father);<a name="line.1608"></a>
-<span class="sourceLineNo">1609</span> regionStateStore.mergeRegions(child, mother, father, serverName);<a name="line.1609"></a>
-<span class="sourceLineNo">1610</span> if (shouldAssignFavoredNodes(child)) {<a name="line.1610"></a>
-<span class="sourceLineNo">1611</span> ((FavoredNodesPromoter)getBalancer()).<a name="line.1611"></a>
-<span class="sourceLineNo">1612</span> generateFavoredNodesForMergedRegion(child, mother, father);<a name="line.1612"></a>
-<span class="sourceLineNo">1613</span> }<a name="line.1613"></a>
-<span class="sourceLineNo">1614</span> }<a name="line.1614"></a>
-<span class="sourceLineNo">1615</span><a name="line.1615"></a>
-<span class="sourceLineNo">1616</span> /*<a name="line.1616"></a>
-<span class="sourceLineNo">1617</span> * Favored nodes should be applied only when FavoredNodes balancer is configured and the region<a name="line.1617"></a>
-<span class="sourceLineNo">1618</span> * belongs to a non-system table.<a name="line.1618"></a>
-<span class="sourceLineNo">1619</span> */<a name="line.1619"></a>
-<span class="sourceLineNo">1620</span> private boolean shouldAssignFavoredNodes(RegionInfo region) {<a name="line.1620"></a>
-<span class="sourceLineNo">1621</span> return this.shouldAssignRegionsWithFavoredNodes &&<a name="line.1621"></a>
-<span class="sourceLineNo">1622</span> FavoredNodesManager.isFavoredNodeApplicable(region);<a name="line.1622"></a>
-<span class="sourceLineNo">1623</span> }<a name="line.1623"></a>
-<span class="sourceLineNo">1624</span><a name="line.1624"></a>
-<span class="sourceLineNo">1625</span> // ============================================================================================<a name="line.1625"></a>
-<span class="sourceLineNo">1626</span> // Assign Queue (Assign/Balance)<a name="line.1626"></a>
-<span class="sourceLineNo">1627</span> // ============================================================================================<a name="line.1627"></a>
-<span class="sourceLineNo">1628</span> private final ArrayList<RegionStateNode> pendingAssignQueue = new ArrayList<RegionStateNode>();<a name="line.1628"></a>
-<span class="sourceLineNo">1629</span> private final ReentrantLock assignQueueLock = new ReentrantLock();<a name="line.1629"></a>
-<span class="sourceLineNo">1630</span> private final Condition assignQueueFullCond = assignQueueLock.newCondition();<a name="line.1630"></a>
-<span class="sourceLineNo">1631</span><a name="line.1631"></a>
-<span class="sourceLineNo">1632</span> /**<a name="line.1632"></a>
-<span class="sourceLineNo">1633</span> * Add the assign operation to the assignment queue.<a name="line.1633"></a>
-<span class="sourceLineNo">1634</span> * The pending assignment operation will be processed,<a name="line.1634"></a>
-<span class="sourceLineNo">1635</span> * and each region will be assigned by a server using the balancer.<a name="line.1635"></a>
-<span class="sourceLineNo">1636</span> */<a name="line.1636"></a>
-<span class="sourceLineNo">1637</span> protected void queueAssign(final RegionStateNode regionNode) {<a name="line.1637"></a>
-<span class="sourceLineNo">1638</span> regionNode.getProcedureEvent().suspend();<a name="line.1638"></a>
-<span class="sourceLineNo">1639</span><a name="line.1639"></a>
-<span class="sourceLineNo">1640</span> // TODO: quick-start for meta and the other sys-tables?<a name="line.1640"></a>
-<span class="sourceLineNo">1641</span> assignQueueLock.lock();<a name="line.1641"></a>
-<span class="sourceLineNo">1642</span> try {<a name="line.1642"></a>
-<span class="sourceLineNo">1643</span> pendingAssignQueue.add(regionNode);<a name="line.1643"></a>
-<span class="sourceLineNo">1644</span> if (regionNode.isSystemTable() ||<a name="line.1644"></a>
-<span class="sourceLineNo">1645</span> pendingAssignQueue.size() == 1 ||<a name="line.1645"></a>
-<span class="sourceLineNo">1646</span> pendingAssignQueue.size() >= assignDispatchWaitQueueMaxSize) {<a name="line.1646"></a>
-<span class="sourceLineNo">1647</span> assignQueueFullCond.signal();<a name="line.1647"></a>
-<span class="sourceLineNo">1648</span> }<a name="line.1648"></a>
-<span class="sourceLineNo">1649</span> } finally {<a name="line.1649"></a>
-<span class="sourceLineNo">1650</span> assignQueueLock.unlock();<a name="line.1650"></a>
-<span class="sourceLineNo">1651</span> }<a name="line.1651"></a>
-<span class="sourceLineNo">1652</span> }<a name="line.1652"></a>
-<span class="sourceLineNo">1653</span><a name="line.1653"></a>
-<span class="sourceLineNo">1654</span> private void startAssignmentThread() {<a name="line.1654"></a>
-<span class="sourceLineNo">1655</span> // Get Server Thread name. Sometimes the Server is mocked so may not implement HasThread.<a name="line.1655"></a>
-<span class="sourceLineNo">1656</span> // For example, in tests.<a name="line.1656"></a>
-<span class="sourceLineNo">1657</span> String name = master instanceof HasThread? ((HasThread)master).getName():<a name="line.1657"></a>
-<span class="sourceLineNo">1658</span> master.getServerName().toShortString();<a name="line.1658"></a>
-<span class="sourceLineNo">1659</span> assignThread = new Thread(name) {<a name="line.1659"></a>
-<span class="sourceLineNo">1660</span> @Override<a name="line.1660"></a>
-<span class="sourceLineNo">1661</span> public void run() {<a name="line.1661"></a>
-<span class="sourceLineNo">1662</span> while (isRunning()) {<a name="line.1662"></a>
-<span class="sourceLineNo">1663</span> processAssignQueue();<a name="line.1663"></a>
-<span class="sourceLineNo">1664</span> }<a name="line.1664"></a>
-<span class="sourceLineNo">1665</span> pendingAssignQueue.clear();<a name="line.1665"></a>
-<span class="sourceLineNo">1666</span> }<a name="line.1666"></a>
-<span class="sourceLineNo">1667</span> };<a name="line.1667"></a>
-<span class="sourceLineNo">1668</span> assignThread.setDaemon(true);<a name="line.1668"></a>
-<span class="sourceLineNo">1669</span> assignThread.start();<a name="line.1669"></a>
-<span class="sourceLineNo">1670</span> }<a name="line.1670"></a>
-<span class="sourceLineNo">1671</span><a name="line.1671"></a>
-<span class="sourceLineNo">1672</span> private void stopAssignmentThread() {<a name="line.1672"></a>
-<span class="sourceLineNo">1673</span> assignQueueSignal();<a name="line.1673"></a>
-<span class="sourceLineNo">1674</span> try {<a name="line.1674"></a>
-<span class="sourceLineNo">1675</span> while (assignThread.isAlive()) {<a name="line.1675"></a>
-<span class="sourceLineNo">1676</span> assignQueueSignal();<a name="line.1676"></a>
-<span class="sourceLineNo">1677</span> assignThread.join(250);<a name="line.1677"></a>
-<span class="sourceLineNo">1678</span> }<a name="line.1678"></a>
-<span class="sourceLineNo">1679</span> } catch (InterruptedException e) {<a name="line.1679"></a>
-<span class="sourceLineNo">1680</span> LOG.warn("join interrupted", e);<a name="line.1680"></a>
-<span class="sourceLineNo">1681</span> Thread.currentThread().interrupt();<a name="line.1681"></a>
-<span class="sourceLineNo">1682</span> }<a name="line.1682"></a>
-<span class="sourceLineNo">1683</span> }<a name="line.1683"></a>
-<span class="sourceLineNo">1684</span><a name="line.1684"></a>
-<span class="sourceLineNo">1685</span> private void assignQueueSignal() {<a name="line.1685"></a>
-<span class="sourceLineNo">1686</span> assignQueueLock.lock();<a name="line.1686"></a>
-<span class="sourceLineNo">1687</span> try {<a name="line.1687"></a>
-<span class="sourceLineNo">1688</span> assignQueueFullCond.signal();<a name="line.1688"></a>
-<span class="sourceLineNo">1689</span> } finally {<a name="line.1689"></a>
-<span class="sourceLineNo">1690</span> assignQueueLock.unlock();<a name="line.1690"></a>
-<span class="sourceLineNo">1691</span> }<a name="line.1691"></a>
-<span class="sourceLineNo">1692</span> }<a name="line.1692"></a>
-<span class="sourceLineNo">1693</span><a name="line.1693"></a>
-<span class="sourceLineNo">1694</span> @edu.umd.cs.findbugs.annotations.SuppressWarnings("WA_AWAIT_NOT_IN_LOOP")<a name="line.1694"></a>
-<span class="sourceLineNo">1695</span> private HashMap<RegionInfo, RegionStateNode> waitOnAssignQueue() {<a name="line.1695"></a>
-<span class="sourceLineNo">1696</span> HashMap<RegionInfo, RegionStateNode> regions = null;<a name="line.1696"></a>
-<span class="sourceLineNo">1697</span><a name="line.1697"></a>
-<span class="sourceLineNo">1698</span> assignQueueLock.lock();<a name="line.1698"></a>
-<span class="sourceLineNo">1699</span> try {<a name="line.1699"></a>
-<span class="sourceLineNo">1700</span> if (pendingAssignQueue.isEmpty() && isRunning()) {<a name="line.1700"></a>
-<span class="sourceLineNo">1701</span> assignQueueFullCond.await();<a name="line.1701"></a>
-<span class="sourceLineNo">1702</span> }<a name="line.1702"></a>
-<span class="sourceLineNo">1703</span><a name="line.1703"></a>
-<span class="sourceLineNo">1704</span> if (!isRunning()) return null;<a name="line.1704"></a>
-<span class="sourceLineNo">1705</span> assignQueueFullCond.await(assignDispatchWaitMillis, TimeUnit.MILLISECONDS);<a name="line.1705"></a>
-<span class="sourceLineNo">1706</span> regions = new HashMap<RegionInfo, RegionStateNode>(pendingAssignQueue.size());<a name="line.1706"></a>
-<span class="sourceLineNo">1707</span> for (RegionStateNode regionNode: pendingAssignQueue) {<a name="line.1707"></a>
-<span class="sourceLineNo">1708</span> regions.put(regionNode.getRegionInfo(), regionNode);<a name="line.1708"></a>
-<span class="sourceLineNo">1709</span> }<a name="line.1709"></a>
-<span class="sourceLineNo">1710</span> pendingAssignQueue.clear();<a name="line.1710"></a>
-<span class="sourceLineNo">1711</span> } catch (InterruptedException e) {<a name="line.1711"></a>
-<span class="sourceLineNo">1712</span> LOG.warn("got interrupted ", e);<a name="line.1712"></a>
-<span class="sourceLineNo">1713</span> Thread.currentThread().interrupt();<a name="line.1713"></a>
-<span class="sourceLineNo">1714</span> } finally {<a name="line.1714"></a>
-<span class="sourceLineNo">1715</span> assignQueueLock.unlock();<a name="line.1715"></a>
-<span class="sourceLineNo">1716</span> }<a name="line.1716"></a>
-<span class="sourceLineNo">1717</span> return regions;<a name="line.1717"></a>
-<span class="sourceLineNo">1718</span> }<a name="line.1718"></a>
-<span class="sourceLineNo">1719</span><a name="line.1719"></a>
-<span class="sourceLineNo">1720</span> private void processAssignQueue() {<a name="line.1720"></a>
-<span class="sourceLineNo">1721</span> final HashMap<RegionInfo, RegionStateNode> regions = waitOnAssignQueue();<a name="line.1721"></a>
-<span class="sourceLineNo">1722</span> if (regions == null || regions.size() == 0 || !isRunning()) {<a name="line.1722"></a>
-<span class="sourceLineNo">1723</span> return;<a name="line.1723"></a>
-<span class="sourceLineNo">1724</span> }<a name="line.1724"></a>
-<span class="sourceLineNo">1725</span><a name="line.1725"></a>
-<span class="sourceLineNo">1726</span> if (LOG.isTraceEnabled()) {<a name="line.1726"></a>
-<span class="sourceLineNo">1727</span> LOG.trace("PROCESS ASSIGN QUEUE regionCount=" + regions.size());<a name="line.1727"></a>
-<span class="sourceLineNo">1728</span> }<a name="line.1728"></a>
-<span class="sourceLineNo">1729</span><a name="line.1729"></a>
-<span class="sourceLineNo">1730</span> // TODO: Optimize balancer. pass a RegionPlan?<a name="line.1730"></a>
-<span class="sourceLineNo">1731</span> final HashMap<RegionInfo, ServerName> retainMap = new HashMap<>();<a name="line.1731"></a>
-<span class="sourceLineNo">1732</span> final List<RegionInfo> userHRIs = new ArrayList<>(regions.size());<a name="line.1732"></a>
-<span class="sourceLineNo">1733</span> // Regions for system tables requiring reassignment<a name="line.1733"></a>
-<span class="sourceLineNo">1734</span> final List<RegionInfo> systemHRIs = new ArrayList<>();<a name="line.1734"></a>
-<span class="sourceLineNo">1735</span> for (RegionStateNode regionStateNode: regions.values()) {<a name="line.1735"></a>
-<span class="sourceLineNo">1736</span> boolean sysTable = regionStateNode.isSystemTable();<a name="line.1736"></a>
-<span class="sourceLineNo">1737</span> final List<RegionInfo> hris = sysTable? systemHRIs: userHRIs;<a name="line.1737"></a>
-<span class="sourceLineNo">1738</span> if (regionStateNode.getRegionLocation() != null) {<a name="line.1738"></a>
-<span class="sourceLineNo">1739</span> retainMap.put(regionStateNode.getRegionInfo(), regionStateNode.getRegionLocation());<a name="line.1739"></a>
-<span class="sourceLineNo">1740</span> } else {<a name="line.1740"></a>
-<span class="sourceLineNo">1741</span> hris.add(regionStateNode.getRegionInfo());<a name="line.1741"></a>
-<span class="sourceLineNo">1742</span> }<a name="line.1742"></a>
-<span class="sourceLineNo">1743</span> }<a name="line.1743"></a>
-<span class="sourceLineNo">1744</span><a name="line.1744"></a>
-<span class="sourceLineNo">1745</span> // TODO: connect with the listener to invalidate the cache<a name="line.1745"></a>
-<span class="sourceLineNo">1746</span><a name="line.1746"></a>
-<span class="sourceLineNo">1747</span> // TODO use events<a name="line.1747"></a>
-<span class="sourceLineNo">1748</span> List<ServerName> servers = master.getServerManager().createDestinationServersList();<a name="line.1748"></a>
-<span class="sourceLineNo">1749</span> for (int i = 0; servers.size() < 1; ++i) {<a name="line.1749"></a>
-<span class="sourceLineNo">1750</span> // Report every fourth time around this loop; try not to flood log.<a name="line.1750"></a>
-<span class="sourceLineNo">1751</span> if (i % 4 == 0) {<a name="line.1751"></a>
-<span class="sourceLineNo">1752</span> LOG.warn("No servers available; cannot place " + regions.size() + " unassigned regions.");<a name="line.1752"></a>
-<span class="sourceLineNo">1753</span> }<a name="line.1753"></a>
-<span class="sourceLineNo">1754</span><a name="line.1754"></a>
-<span class="sourceLineNo">1755</span> if (!isRunning()) {<a name="line.1755"></a>
-<span class="sourceLineNo">1756</span> LOG.debug("Stopped! Dropping assign of " + regions.size() + " queued regions.");<a name="line.1756"></a>
-<span class="sourceLineNo">1757</span> return;<a name="line.1757"></a>
-<span class="sourceLineNo">1758</span> }<a name="line.1758"></a>
-<span class="sourceLineNo">1759</span> Threads.sleep(250);<a name="line.1759"></a>
-<span class="sourceLineNo">1760</span> servers = master.getServerManager().createDestinationServersList();<a name="line.1760"></a>
-<span class="sourceLineNo">1761</span> }<a name="line.1761"></a>
-<span class="sourceLineNo">1762</span><a name="line.1762"></a>
-<span class="sourceLineNo">1763</span> if (!systemHRIs.isEmpty()) {<a name="line.1763"></a>
-<span class="sourceLineNo">1764</span> // System table regions requiring reassignment are present, get region servers<a name="line.1764"></a>
-<span class="sourceLineNo">1765</span> // not available for system table regions<a name="line.1765"></a>
-<span class="sourceLineNo">1766</span> final List<ServerName> excludeServers = getExcludedServersForSystemTable();<a name="line.1766"></a>
-<span class="sourceLineNo">1767</span> List<ServerName> serversForSysTables = servers.stream()<a name="line.1767"></a>
-<span class="sourceLineNo">1768</span> .filter(s -> !excludeServers.contains(s)).collect(Collectors.toList());<a name="line.1768"></a>
-<span class="sourceLineNo">1769</span> if (serversForSysTables.isEmpty()) {<a name="line.1769"></a>
-<span class="sourceLineNo">1770</span> LOG.warn("Filtering old server versions and the excluded produced an empty set; " +<a name="line.1770"></a>
-<span class="sourceLineNo">1771</span> "instead considering all candidate servers!");<a name="line.1771"></a>
-<span class="sourceLineNo">1772</span> }<a name="line.1772"></a>
-<span class="sourceLineNo">1773</span> LOG.debug("Processing assignQueue; systemServersCount=" + serversForSysTables.size() +<a name="line.1773"></a>
-<span class="sourceLineNo">1774</span> ", allServersCount=" + servers.size());<a name="line.1774"></a>
-<span class="sourceLineNo">1775</span> processAssignmentPlans(regions, null, systemHRIs,<a name="line.1775"></a>
-<span class="sourceLineNo">1776</span> serversForSysTables.isEmpty()? servers: serversForSysTables);<a name="line.1776"></a>
-<span class="sourceLineNo">1777</span> }<a name="line.1777"></a>
-<span class="sourceLineNo">1778</span><a name="line.1778"></a>
-<span class="sourceLineNo">1779</span> processAssignmentPlans(regions, retainMap, userHRIs, servers);<a name="line.1779"></a>
-<span class="sourceLineNo">1780</span> }<a name="line.1780"></a>
-<span class="sourceLineNo">1781</span><a name="line.1781"></a>
-<span class="sourceLineNo">1782</span> private void processAssignmentPlans(final HashMap<RegionInfo, RegionStateNode> regions,<a name="line.1782"></a>
-<span class="sourceLineNo">1783</span> final HashMap<RegionInfo, ServerName> retainMap, final List<RegionInfo> hris,<a name="line.1783"></a>
-<span class="sourceLineNo">1784</span> final List<ServerName> servers) {<a name="line.1784"></a>
-<span class="sourceLineNo">1785</span> boolean isTraceEnabled = LOG.isTraceEnabled();<a name="line.1785"></a>
-<span class="sourceLineNo">1786</span> if (isTraceEnabled) {<a name="line.1786"></a>
-<span class="sourceLineNo">1787</span> LOG.trace("Available servers count=" + servers.size() + ": " + servers);<a name="line.1787"></a>
-<span class="sourceLineNo">1788</span> }<a name="line.1788"></a>
-<span class="sourceLineNo">1789</span><a name="line.1789"></a>
-<span class="sourceLineNo">1790</span> final LoadBalancer balancer = getBalancer();<a name="line.1790"></a>
-<span class="sourceLineNo">1791</span> // ask the balancer where to place regions<a name="line.1791"></a>
-<span class="sourceLineNo">1792</span> if (retainMap != null && !retainMap.isEmpty()) {<a name="line.1792"></a>
-<span class="sourceLineNo">1793</span> if (isTraceEnabled) {<a name="line.1793"></a>
-<span class="sourceLineNo">1794</span> LOG.trace("retain assign regions=" + retainMap);<a name="line.1794"></a>
-<span class="sourceLineNo">1795</span> }<a name="line.1795"></a>
-<span class="sourceLineNo">1796</span> try {<a name="line.1796"></a>
-<span class="sourceLineNo">1797</span> acceptPlan(regions, balancer.retainAssignment(retainMap, servers));<a name="line.1797"></a>
-<span class="sourceLineNo">1798</span> } catch (HBaseIOException e) {<a name="line.1798"></a>
-<span class="sourceLineNo">1799</span> LOG.warn("unable to retain assignment", e);<a name="line.1799"></a>
-<span class="sourceLineNo">1800</span> addToPendingAssignment(regions, retainMap.keySet());<a name="line.1800"></a>
-<span class="sourceLineNo">1801</span> }<a name="line.1801"></a>
-<span class="sourceLineNo">1802</span> }<a name="line.1802"></a>
-<span class="sourceLineNo">1803</span><a name="line.1803"></a>
-<span class="sourceLineNo">1804</span> // TODO: Do we need to split retain and round-robin?<a name="line.1804"></a>
-<span class="sourceLineNo">1805</span> // the retain seems to fallback to round-robin/random if the region is not in the map.<a name="line.1805"></a>
-<span class="sourceLineNo">1806</span> if (!hris.isEmpty()) {<a name="line.1806"></a>
-<span class="sourceLineNo">1807</span> Collections.sort(hris, RegionInfo.COMPARATOR);<a name="line.1807"></a>
-<span class="sourceLineNo">1808</span> if (isTraceEnabled) {<a name="line.1808"></a>
-<span class="sourceLineNo">1809</span> LOG.trace("round robin regions=" + hris);<a name="line.1809"></a>
-<span class="sourceLineNo">1810</span> }<a name="line.1810"></a>
-<span class="sourceLineNo">1811</span> try {<a name="line.1811"></a>
-<span class="sourceLineNo">1812</span> acceptPlan(regions, balancer.roundRobinAssignment(hris, servers));<a name="line.1812"></a>
-<span class="sourceLineNo">1813</span> } catch (HBaseIOException e) {<a name="line.1813"></a>
-<span class="sourceLineNo">1814</span> LOG.warn("unable to round-robin assignment", e);<a name="line.1814"></a>
-<span class="sourceLineNo">1815</span> addToPendingAssignment(regions, hris);<a name="line.1815"></a>
-<span class="sourceLineNo">1816</span> }<a name="line.1816"></a>
-<span class="sourceLineNo">1817</span> }<a name="line.1817"></a>
-<span class="sourceLineNo">1818</span> }<a name="line.1818"></a>
-<span class="sourceLineNo">1819</span><a name="line.1819"></a>
-<span class="sourceLineNo">1820</span> private void acceptPlan(final HashMap<RegionInfo, RegionStateNode> regions,<a name="line.1820"></a>
-<span class="sourceLineNo">1821</span> final Map<ServerName, List<RegionInfo>> plan) throws HBaseIOException {<a name="line.1821"></a>
-<span class="sourceLineNo">1822</span> final ProcedureEvent[] events = new ProcedureEvent[regions.size()];<a name="line.1822"></a>
-<span class="sourceLineNo">1823</span> final long st = System.currentTimeMillis();<a name="line.1823"></a>
-<span class="sourceLineNo">1824</span><a name="line.1824"></a>
-<span class="sourceLineNo">1825</span> if (plan == null) {<a name="line.1825"></a>
-<span class="sourceLineNo">1826</span> throw new HBaseIOException("unable to compute plans for regions=" + regions.size());<a name="line.1826"></a>
-<span class="sourceLineNo">1827</span> }<a name="line.1827"></a>
-<span class="sourceLineNo">1828</span><a name="line.1828"></a>
-<span class="sourceLineNo">1829</span> if (plan.isEmpty()) return;<a name="line.1829"></a>
-<span class="sourceLineNo">1830</span><a name="line.1830"></a>
-<span class="sourceLineNo">1831</span> int evcount = 0;<a name="line.1831"></a>
-<span class="sourceLineNo">1832</span> for (Map.Entry<ServerName, List<RegionInfo>> entry: plan.entrySet()) {<a name="line.1832"></a>
-<span class="sourceLineNo">1833</span> final ServerName server = entry.getKey();<a name="line.1833"></a>
-<span class="sourceLineNo">1834</span> for (RegionInfo hri: entry.getValue()) {<a name="line.1834"></a>
-<span class="sourceLineNo">1835</span> final RegionStateNode regionNode = regions.get(hri);<a name="line.1835"></a>
-<span class="sourceLineNo">1836</span> regionNode.setRegionLocation(server);<a name="line.1836"></a>
-<span class="sourceLineNo">1837</span> events[evcount++] = regionNode.getProcedureEvent();<a name="line.1837"></a>
-<span class="sourceLineNo">1838</span> }<a name="line.1838"></a>
-<span class="sourceLineNo">1839</span> }<a name="line.1839"></a>
-<span class="sourceLineNo">1840</span> ProcedureEvent.wakeEvents(getProcedureScheduler(), events);<a name="line.1840"></a>
-<span class="sourceLineNo">1841</span><a name="line.1841"></a>
-<span class="sourceLineNo">1842</span> final long et = System.currentTimeMillis();<a name="line.1842"></a>
-<span class="sourceLineNo">1843</span> if (LOG.isTraceEnabled()) {<a name="line.1843"></a>
-<span class="sourceLineNo">1844</span> LOG.trace("ASSIGN ACCEPT " + events.length + " -> " +<a name="line.1844"></a>
-<span class="sourceLineNo">1845</span> StringUtils.humanTimeDiff(et - st));<a name="line.1845"></a>
-<span class="sourceLineNo">1846</span> }<a name="line.1846"></a>
-<span class="sourceLineNo">1847</span> }<a name="line.1847"></a>
-<span class="sourceLineNo">1848</span><a name="line.1848"></a>
-<span class="sourceLineNo">1849</span> private void addToPendingAssignment(final HashMap<RegionInfo, RegionStateNode> regions,<a name="line.1849"></a>
-<span class="sourceLineNo">1850</span> final Collection<RegionInfo> pendingRegions) {<a name="line.1850"></a>
-<span class="sourceLineNo">1851</span> assignQueueLock.lock();<a name="line.1851"></a>
-<span class="sourceLineNo">1852</span> try {<a name="line.1852"></a>
-<span class="sourceLineNo">1853</span> for (RegionInfo hri: pendingRegions) {<a name="line.1853"></a>
-<span class="sourceLineNo">1854</span> pendingAssignQueue.add(regions.get(hri));<a name="line.1854"></a>
-<span class="sourceLineNo">1855</span> }<a name="line.1855"></a>
-<span class="sourceLineNo">1856</span> } finally {<a name="line.1856"></a>
-<span class="sourceLineNo">1857</span> assignQueueLock.unlock();<a name="line.1857"></a>
-<span class="sourceLineNo">1858</span> }<a name="line.1858"></a>
-<span class="sourceLineNo">1859</span> }<a name="line.1859"></a>
-<span class="sourceLineNo">1860</span><a name="line.1860"></a>
-<span class="sourceLineNo">1861</span> /**<a name="line.1861"></a>
-<span class="sourceLineNo">1862</span> * Get a list of servers that this region cannot be assigned to.<a name="line.1862"></a>
-<span class="sourceLineNo">1863</span> * For system tables, we must assign them to a server with highest version.<a name="line.1863"></a>
-<span class="sourceLineNo">1864</span> */<a name="line.1864"></a>
-<span class="sourceLineNo">1865</span> public List<ServerName> getExcludedServersForSystemTable() {<a name="line.1865"></a>
-<span class="sourceLineNo">1866</span> // TODO: This should be a cached list kept by the ServerManager rather than calculated on each<a name="line.1866"></a>
-<span class="sourceLineNo">1867</span> // move or system region assign. The RegionServerTracker keeps list of online Servers with<a name="line.1867"></a>
-<span class="sourceLineNo">1868</span> // RegionServerInfo that includes Version.<a name="line.1868"></a>
-<span class="sourceLineNo">1869</span> List<Pair<ServerName, String>> serverList = master.getServerManager().getOnlineServersList()<a name="line.1869"></a>
-<span class="sourceLineNo">1870</span> .stream()<a name="line.1870"></a>
-<span class="sourceLineNo">1871</span> .map((s)->new Pair<>(s, master.getRegionServerVersion(s)))<a name="line.1871"></a>
-<span class="sourceLineNo">1872</span> .collect(Collectors.toList());<a name="line.1872"></a>
-<span class="sourceLineNo">1873</span> if (serverList.isEmpty()) {<a name="line.1873"></a>
-<span class="sourceLineNo">1874</span> return Collections.EMPTY_LIST;<a name="line.1874"></a>
-<span class="sourceLineNo">1875</span> }<a name="line.1875"></a>
-<span class="sourceLineNo">1876</span> String highestVersion = Collections.max(serverList,<a name="line.1876"></a>
-<span class="sourceLineNo">1877</span> (o1, o2) -> VersionInfo.compareVersion(o1.getSecond(), o2.getSecond())).getSecond();<a name="line.1877"></a>
-<span class="sourceLineNo">1878</span> return serverList.stream()<a name="line.1878"></a>
-<span class="sourceLineNo">1879</span> .filter((p)->!p.getSecond().equals(highestVersion))<a name="line.1879"></a>
-<span class="sourceLineNo">1880</span> .map(Pair::getFirst)<a name="line.1880"></a>
-<span class="sourceLineNo">1881</span> .collect(Collectors.toList());<a name="line.1881"></a>
-<span class="sourceLineNo">1882</span> }<a name="line.1882"></a>
-<span class="sourceLineNo">1883</span><a name="line.1883"></a>
-<span class="sourceLineNo">1884</span> // ============================================================================================<a name="line.1884"></a>
-<span class="sourceLineNo">1885</span> // Server Helpers<a name="line.1885"></a>
-<span class="sourceLineNo">1886</span> // ============================================================================================<a name="line.1886"></a>
-<span class="sourceLineNo">1887</span> @Override<a name="line.1887"></a>
-<span class="sourceLineNo">1888</span> public void serverAdded(final ServerName serverName) {<a name="line.1888"></a>
-<span class="sourceLineNo">1889</span> }<a name="line.1889"></a>
-<span class="sourceLineNo">1890</span><a name="line.1890"></a>
-<span class="sourceLineNo">1891</span> @Override<a name="line.1891"></a>
-<span class="sourceLineNo">1892</span> public void serverRemoved(final ServerName serverName) {<a name="line.1892"></a>
-<span class="sourceLineNo">1893</span> final ServerStateNode serverNode = regionStates.getServerNode(serverName);<a name="line.1893"></a>
-<span class="sourceLineNo">1894</span> if (serverNode == null) return;<a name="line.1894"></a>
-<span class="sourceLineNo">1895</span><a name="line.1895"></a>
-<span class="sourceLineNo">1896</span> // just in case, wake procedures waiting for this server report<a name="line.1896"></a>
-<span class="sourceLineNo">1897</span> wakeServerReportEvent(serverNode);<a name="line.1897"></a>
-<span class="sourceLineNo">1898</span> }<a name="line.1898"></a>
-<span class="sourceLineNo">1899</span><a name="line.1899"></a>
-<span class="sourceLineNo">1900</span> public int getServerVersion(final ServerName serverName) {<a name="line.1900"></a>
-<span class="sourceLineNo">1901</span> final ServerStateNode node = regionStates.getServerNode(serverName);<a name="line.1901"></a>
-<span class="sourceLineNo">1902</span> return node != null ? node.getVersionNumber() : 0;<a name="line.1902"></a>
-<span class="sourceLineNo">1903</span> }<a name="line.1903"></a>
-<span class="sourceLineNo">1904</span><a name="line.1904"></a>
-<span class="sourceLineNo">1905</span> public void killRegionServer(final ServerName serverName) {<a name="line.1905"></a>
-<span class="sourceLineNo">1906</span> final ServerStateNode serverNode = regionStates.getServerNode(serverName);<a name="line.1906"></a>
-<span class="sourceLineNo">1907</span> killRegionServer(serverNode);<a name="line.1907"></a>
-<span class="sourceLineNo">1908</span> }<a name="line.1908"></a>
-<span class="sourceLineNo">1909</span><a name="line.1909"></a>
-<span class="sourceLineNo">1910</span> public void killRegionServer(final ServerStateNode serverNode) {<a name="line.1910"></a>
-<span class="sourceLineNo">1911</span> /** Don't do this. Messes up accounting. Let ServerCrashProcedure do this.<a name="line.1911"></a>
-<span class="sourceLineNo">1912</span> for (RegionStateNode regionNode: serverNode.getRegions()) {<a name="line.1912"></a>
-<span class="sourceLineNo">1913</span> regionNode.offline();<a name="line.1913"></a>
-<span class="sourceLineNo">1914</span> }*/<a name="line.1914"></a>
-<span class="sourceLineNo">1915</span> master.getServerManager().expireServer(serverNode.getServerName());<a name="line.1915"></a>
-<span class="sourceLineNo">1916</span> }<a name="line.1916"></a>
-<span class="sourceLineNo">1917</span><a name="line.1917"></a>
-<span class="sourceLineNo">1918</span> /**<a name="line.1918"></a>
-<span class="sourceLineNo">1919</span> * Handle RIT of meta region against crashed server.<a name="line.1919"></a>
-<span class="sourceLineNo">1920</span> * Only used when ServerCrashProcedure is not enabled.<a name="line.1920"></a>
-<span class="sourceLineNo">1921</span> * See handleRIT in ServerCrashProcedure for similar function.<a name="line.1921"></a>
-<span class="sourceLineNo">1922</span> *<a name="line.1922"></a>
-<span class="sourceLineNo">1923</span> * @param serverName Server that has already crashed<a name="line.1923"></a>
-<span class="sourceLineNo">1924</span> */<a name="line.1924"></a>
-<span class="sourceLineNo">1925</span> public void handleMetaRITOnCrashedServer(ServerName serverName) {<a name="line.1925"></a>
-<span class="sourceLineNo">1926</span> RegionInfo hri = RegionReplicaUtil<a name="line.1926"></a>
-<span class="sourceLineNo">1927</span> .getRegionInfoForReplica(RegionInfoBuilder.FIRST_META_REGIONINFO,<a name="line.1927"></a>
-<span class="sourceLineNo">1928</span> RegionInfo.DEFAULT_REPLICA_ID);<a name="line.1928"></a>
-<span class="sourceLineNo">1929</span> RegionState regionStateNode = getRegionStates().getRegionState(hri);<a name="line.1929"></a>
-<span class="sourceLineNo">1930</span> if (regionStateNode == null) {<a name="line.1930"></a>
-<span class="sourceLineNo">1931</span> LOG.warn("RegionStateNode is null for " + hri);<a name="line.1931"></a>
-<span class="sourceLineNo">1932</span> return;<a name="line.1932"></a>
-<span class="sourceLineNo">1933</span> }<a name="line.1933"></a>
-<span class="sourceLineNo">1934</span> ServerName rsnServerName = regionStateNode.getServerName();<a name="line.1934"></a>
-<span class="sourceLineNo">1935</span> if (rsnServerName != null && !rsnServerName.equals(serverName)) {<a name="line.1935"></a>
-<span class="sourceLineNo">1936</span> return;<a name="line.1936"></a>
-<span class="sourceLineNo">1937</span> } else if (rsnServerName == null) {<a name="line.1937"></a>
-<span class="sourceLineNo">1938</span> LOG.warn("Empty ServerName in RegionStateNode; proceeding anyways in case latched " +<a name="line.1938"></a>
-<span class="sourceLineNo">1939</span> "RecoverMetaProcedure so meta latch gets cleaned up.");<a name="line.1939"></a>
-<span class="sourceLineNo">1940</span> }<a name="line.1940"></a>
-<span class="sourceLineNo">1941</span> // meta has been assigned to crashed server.<a name="line.1941"></a>
-<span class="sourceLineNo">1942</span> LOG.info("Meta assigned to crashed " + serverName + "; reassigning...");<a name="line.1942"></a>
-<span class="sourceLineNo">1943</span> // Handle failure and wake event<a name="line.1943"></a>
-<span class="sourceLineNo">1944</span> RegionTransitionProcedure rtp = getRegionStates().getRegionTransitionProcedure(hri);<a name="line.1944"></a>
-<span class="sourceLineNo">1945</span> // Do not need to consider for REGION_TRANSITION_QUEUE step<a name="line.1945"></a>
-<span class="sourceLineNo">1946</span> if (rtp != null && rtp.isMeta() &&<a name="line.1946"></a>
-<span class="sourceLineNo">1947</span> rtp.getTransitionState() == RegionTransitionState.REGION_TRANSITION_DISPATCH) {<a name="line.1947"></a>
-<span class="sourceLineNo">1948</span> LOG.debug("Failing " + rtp.toString());<a name="line.1948"></a>
-<span class="sourceLineNo">1949</span> rtp.remoteCallFailed(master.getMasterProcedureExecutor().getEnvironment(), serverName,<a name="line.1949"></a>
-<span class="sourceLineNo">1950</span> new ServerCrashException(rtp.getProcId(), serverName));<a name="line.1950"></a>
-<span class="sourceLineNo">1951</span> }<a name="line.1951"></a>
-<span class="sourceLineNo">1952</span> }<a name="line.1952"></a>
-<span class="sourceLineNo">1953</span>}<a name="line.1953"></a>
+<span class="sourceLineNo">1260</span> isTableState(regionNode.getTable(), TableState.State.DISABLED,<a name="line.1260"></a>
+<span class="sourceLineNo">1261</span> TableState.State.DISABLING)) {<a name="line.1261"></a>
+<span class="sourceLineNo">1262</span> // The region is CLOSED and the table is DISABLED/ DISABLING, there is nothing to<a name="line.1262"></a>
+<span class="sourceLineNo">1263</span> // schedule; the region is inert.<a name="line.1263"></a>
+<span class="sourceLineNo">1264</span> } else {<a name="line.1264"></a>
+<span class="sourceLineNo">1265</span> // These regions should have a procedure in replay<a name="line.1265"></a>
+<span class="sourceLineNo">1266</span> regionStates.addRegionInTransition(regionNode, null);<a name="line.1266"></a>
+<span class="sourceLineNo">1267</span> }<a name="line.1267"></a>
+<span class="sourceLineNo">1268</span> }<a name="line.1268"></a>
+<span class="sourceLineNo">1269</span> }<a name="line.1269"></a>
+<span class="sourceLineNo">1270</span> }<a name="line.1270"></a>
+<span class="sourceLineNo">1271</span> });<a name="line.1271"></a>
+<span class="sourceLineNo">1272</span><a name="line.1272"></a>
+<span class="sourceLineNo">1273</span> // every assignment is blocked until meta is loaded.<a name="line.1273"></a>
+<span class="sourceLineNo">1274</span> wakeMetaLoadedEvent();<a name="line.1274"></a>
+<span class="sourceLineNo">1275</span> }<a name="line.1275"></a>
+<span class="sourceLineNo">1276</span><a name="line.1276"></a>
+<span class="sourceLineNo">1277</span> /**<a name="line.1277"></a>
+<span class="sourceLineNo">1278</span> * Look at what is in meta and the list of servers that have checked in and make reconciliation.<a name="line.1278"></a>
+<span class="sourceLineNo">1279</span> * We cannot tell definitively the difference between a clean shutdown and a cluster that has<a name="line.1279"></a>
+<span class="sourceLineNo">1280</span> * been crashed down. At this stage of a Master startup, they look the same: they have the<a name="line.1280"></a>
+<span class="sourceLineNo">1281</span> * same state in hbase:meta. We could do detective work probing ZK and the FS for old WALs to<a name="line.1281"></a>
+<span class="sourceLineNo">1282</span> * split but SCP does this already so just let it do its job.<a name="line.1282"></a>
+<span class="sourceLineNo">1283</span> * <p>>The profiles of clean shutdown and cluster crash-down are the same because on clean<a name="line.1283"></a>
+<span class="sourceLineNo">1284</span> * shutdown currently, we do not update hbase:meta with region close state (In AMv2, region<a name="line.1284"></a>
+<span class="sourceLineNo">1285</span> * state is kept in hbse:meta). Usually the master runs all region transitions as of AMv2 but on<a name="line.1285"></a>
+<span class="sourceLineNo">1286</span> * cluster controlled shutdown, the RegionServers close all their regions only reporting the<a name="line.1286"></a>
+<span class="sourceLineNo">1287</span> * final change to the Master. Currently this report is ignored. Later we could take it and<a name="line.1287"></a>
+<span class="sourceLineNo">1288</span> * update as many regions as we can before hbase:meta goes down or have the master run the<a name="line.1288"></a>
+<span class="sourceLineNo">1289</span> * close of all regions out on the cluster but we may never be able to achieve the proper state on<a name="line.1289"></a>
+<span class="sourceLineNo">1290</span> * all regions (at least not w/o lots of painful manipulations and waiting) so clean shutdown<a name="line.1290"></a>
+<span class="sourceLineNo">1291</span> * might not be possible especially on big clusters.... And clean shutdown will take time. Given<a name="line.1291"></a>
+<span class="sourceLineNo">1292</span> * this current state of affairs, we just run ServerCrashProcedure in both cases. It will always<a name="line.1292"></a>
+<span class="sourceLineNo">1293</span> * do the right thing.<a name="line.1293"></a>
+<span class="sourceLineNo">1294</span> * @return True if for sure this is a failover where a Master is starting up into an already<a name="line.1294"></a>
+<span class="sourceLineNo">1295</span> * running cluster.<a name="line.1295"></a>
+<span class="sourceLineNo">1296</span> */<a name="line.1296"></a>
+<span class="sourceLineNo">1297</span> // The assumption here is that if RSs are crashing while we are executing this<a name="line.1297"></a>
+<span class="sourceLineNo">1298</span> // they will be handled by the SSH that are put in the ServerManager deadservers "queue".<a name="line.1298"></a>
+<span class="sourceLineNo">1299</span> private boolean processofflineServersWithOnlineRegions() {<a name="line.1299"></a>
+<span class="sourceLineNo">1300</span> boolean deadServers = !master.getServerManager().getDeadServers().isEmpty();<a name="line.1300"></a>
+<span class="sourceLineNo">1301</span> final Set<ServerName> offlineServersWithOnlineRegions = new HashSet<>();<a name="line.1301"></a>
+<span class="sourceLineNo">1302</span> int size = regionStates.getRegionStateNodes().size();<a name="line.1302"></a>
+<span class="sourceLineNo">1303</span> final List<RegionInfo> offlineRegionsToAssign = new ArrayList<>(size);<a name="line.1303"></a>
+<span class="sourceLineNo">1304</span> // If deadservers then its a failover, else, we are not sure yet.<a name="line.1304"></a>
+<span class="sourceLineNo">1305</span> boolean failover = deadServers;<a name="line.1305"></a>
+<span class="sourceLineNo">1306</span> for (RegionStateNode regionNode: regionStates.getRegionStateNodes()) {<a name="line.1306"></a>
+<span class="sourceLineNo">1307</span> // Region State can be OPEN even if we did controlled cluster shutdown; Master does not close<a name="line.1307"></a>
+<span class="sourceLineNo">1308</span> // the regions in this case. The RegionServer does the close so hbase:meta is state in<a name="line.1308"></a>
+<span class="sourceLineNo">1309</span> // hbase:meta is not updated -- Master does all updates -- and is left with OPEN as region<a name="line.1309"></a>
+<span class="sourceLineNo">1310</span> // state in meta. How to tell difference between ordered shutdown and crashed-down cluster<a name="line.1310"></a>
+<span class="sourceLineNo">1311</span> // then? We can't. Not currently. Perhaps if we updated hbase:meta with CLOSED on ordered<a name="line.1311"></a>
+<span class="sourceLineNo">1312</span> // shutdown. This would slow shutdown though and not all edits would make it in anyways.<a name="line.1312"></a>
+<span class="sourceLineNo">1313</span> // TODO: Examine.<a name="line.1313"></a>
+<span class="sourceLineNo">1314</span> // Because we can't be sure it an ordered shutdown, we run ServerCrashProcedure always.<a name="line.1314"></a>
+<span class="sourceLineNo">1315</span> // ServerCrashProcedure will try to retain old deploy when it goes to assign.<a name="line.1315"></a>
+<span class="sourceLineNo">1316</span> if (regionNode.getState() == State.OPEN) {<a name="line.1316"></a>
+<span class="sourceLineNo">1317</span> final ServerName serverName = regionNode.getRegionLocation();<a name="line.1317"></a>
+<span class="sourceLineNo">1318</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1318"></a>
+<span class="sourceLineNo">1319</span> offlineServersWithOnlineRegions.add(serverName);<a name="line.1319"></a>
+<span class="sourceLineNo">1320</span> } else {<a name="line.1320"></a>
+<span class="sourceLineNo">1321</span> // Server is online. This a failover. Master is starting into already-running cluster.<a name="line.1321"></a>
+<span class="sourceLineNo">1322</span> failover = true;<a name="line.1322"></a>
+<span class="sourceLineNo">1323</span> }<a name="line.1323"></a>
+<span class="sourceLineNo">1324</span> } else if (regionNode.getState() == State.OFFLINE) {<a name="line.1324"></a>
+<span class="sourceLineNo">1325</span> if (isTableEnabled(regionNode.getTable())) {<a name="line.1325"></a>
+<span class="sourceLineNo">1326</span> offlineRegionsToAssign.add(regionNode.getRegionInfo());<a name="line.1326"></a>
+<span class="sourceLineNo">1327</span> }<a name="line.1327"></a>
+<span class="sourceLineNo">1328</span> }<a name="line.1328"></a>
+<span class="sourceLineNo">1329</span> }<a name="line.1329"></a>
+<span class="sourceLineNo">1330</span> // Kill servers with online regions just-in-case. Runs ServerCrashProcedure.<a name="line.1330"></a>
+<span class="sourceLineNo">1331</span> for (ServerName serverName: offlineServersWithOnlineRegions) {<a name="line.1331"></a>
+<span class="sourceLineNo">1332</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1332"></a>
+<span class="sourceLineNo">1333</span> LOG.info("KILL RegionServer=" + serverName + " hosting regions but not online.");<a name="line.1333"></a>
+<span class="sourceLineNo">1334</span> killRegionServer(serverName);<a name="line.1334"></a>
+<span class="sourceLineNo">1335</span> }<a name="line.1335"></a>
+<span class="sourceLineNo">1336</span> }<a name="line.1336"></a>
+<span class="sourceLineNo">1337</span> setFailoverCleanupDone(true);<a name="line.1337"></a>
+<span class="sourceLineNo">1338</span><a name="line.1338"></a>
+<span class="sourceLineNo">1339</span> // Assign offline regions. Uses round-robin.<a name="line.1339"></a>
+<span class="sourceLineNo">1340</span> if (offlineRegionsToAssign.size() > 0) {<a name="line.1340"></a>
+<span class="sourceLineNo">1341</span> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().<a name="line.1341"></a>
+<span class="sourceLineNo">1342</span> createRoundRobinAssignProcedures(offlineRegionsToAssign));<a name="line.1342"></a>
+<span class="sourceLineNo">1343</span> }<a name="line.1343"></a>
+<span class="sourceLineNo">1344</span><a name="line.1344"></a>
+<span class="sourceLineNo">1345</span> return failover;<a name="line.1345"></a>
+<span class="sourceLineNo">1346</span> }<a name="line.1346"></a>
+<span class="sourceLineNo">1347</span><a name="line.1347"></a>
+<span class="sourceLineNo">1348</span> /**<a name="line.1348"></a>
+<span class="sourceLineNo">1349</span> * Used by ServerCrashProcedure to make sure AssignmentManager has completed<a name="line.1349"></a>
+<span class="sourceLineNo">1350</span> * the failover cleanup before re-assigning regions of dead servers. So that<a name="line.1350"></a>
+<span class="sourceLineNo">1351</span> * when re-assignment happens, AssignmentManager has proper region states.<a name="line.1351"></a>
+<span class="sourceLineNo">1352</span> */<a name="line.1352"></a>
+<span class="sourceLineNo">1353</span> public boolean isFailoverCleanupDone() {<a name="line.1353"></a>
+<span class="sourceLineNo">1354</span> return failoverCleanupDone.isReady();<a name="line.1354"></a>
+<span class="sourceLineNo">1355</span> }<a name="line.1355"></a>
+<span class="sourceLineNo">1356</span><a name="line.1356"></a>
+<span class="sourceLineNo">1357</span> /**<a name="line.1357"></a>
+<span class="sourceLineNo">1358</span> * Used by ServerCrashProcedure tests verify the ability to suspend the<a name="line.1358"></a>
+<span class="sourceLineNo">1359</span> * execution of the ServerCrashProcedure.<a name="line.1359"></a>
+<span class="sourceLineNo">1360</span> */<a name="line.1360"></a>
+<span class="sourceLineNo">1361</span> @VisibleForTesting<a name="line.1361"></a>
+<span class="sourceLineNo">1362</span> public void setFailoverCleanupDone(final boolean b) {<a name="line.1362"></a>
+<span class="sourceLineNo">1363</span> master.getMasterProcedureExecutor().getEnvironment()<a name="line.1363"></a>
+<span class="sourceLineNo">1364</span> .setEventReady(failoverCleanupDone, b);<a name="line.1364"></a>
+<span class="sourceLineNo">1365</span> }<a name="line.1365"></a>
+<span class="sourceLineNo">1366</span><a name="line.1366"></a>
+<span class="sourceLineNo">1367</span> public ProcedureEvent getFailoverCleanupEvent() {<a name="line.1367"></a>
+<span class="sourceLineNo">1368</span> return failoverCleanupDone;<a name="line.1368"></a>
+<span class="sourceLineNo">1369</span> }<a name="line.1369"></a>
+<span class="sourceLineNo">1370</span><a name="line.1370"></a>
+<span class="sourceLineNo">1371</span> /**<a name="line.1371"></a>
+<span class="sourceLineNo">1372</span> * Used to check if the failover cleanup is done.<a name="line.1372"></a>
+<span class="sourceLineNo">1373</span> * if not we throw PleaseHoldException since we are rebuilding the RegionStates<a name="line.1373"></a>
+<span class="sourceLineNo">1374</span> * @param hri region to check if it is already rebuild<a name="line.1374"></a>
+<span class="sourceLineNo">1375</span> * @throws PleaseHoldException if the failover cleanup is not completed<a name="line.1375"></a>
+<span class="sourceLineNo">1376</span> */<a name="line.1376"></a>
+<span class="sourceLineNo">1377</span> private void checkFailoverCleanupCompleted(final RegionInfo hri) throws PleaseHoldException {<a name="line.1377"></a>
+<span class="sourceLineNo">1378</span> if (!isRunning()) {<a name="line.1378"></a>
+<span class="sourceLineNo">1379</span> throw new PleaseHoldException("AssignmentManager not running");<a name="line.1379"></a>
+<span class="sourceLineNo">1380</span> }<a name="line.1380"></a>
+<span class="sourceLineNo">1381</span><a name="line.1381"></a>
+<span class="sourceLineNo">1382</span> // TODO: can we avoid throwing an exception if hri is already loaded?<a name="line.1382"></a>
+<span class="sourceLineNo">1383</span> // at the moment we bypass only meta<a name="line.1383"></a>
+<span class="sourceLineNo">1384</span> boolean meta = isMetaRegion(hri);<a name="line.1384"></a>
+<span class="sourceLineNo">1385</span> boolean cleanup = isFailoverCleanupDone();<a name="line.1385"></a>
+<span class="sourceLineNo">1386</span> if (!isMetaRegion(hri) && !isFailoverCleanupDone()) {<a name="line.1386"></a>
+<span class="sourceLineNo">1387</span> String msg = "Master not fully online; hbase:meta=" + meta + ", failoverCleanup=" + cleanup;<a name="line.1387"></a>
+<span class="sourceLineNo">1388</span> throw new PleaseHoldException(msg);<a name="line.1388"></a>
+<span class="sourceLineNo">1389</span> }<a name="line.1389"></a>
+<span class="sourceLineNo">1390</span> }<a name="line.1390"></a>
+<span
<TRUNCATED>
[08/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html
index 97aa79c..cc787e3 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html
@@ -272,1623 +272,1642 @@
<span class="sourceLineNo">264</span> private final CopyOnWriteArrayList<ProcedureExecutorListener> listeners = new CopyOnWriteArrayList<>();<a name="line.264"></a>
<span class="sourceLineNo">265</span><a name="line.265"></a>
<span class="sourceLineNo">266</span> private Configuration conf;<a name="line.266"></a>
-<span class="sourceLineNo">267</span> private ThreadGroup threadGroup;<a name="line.267"></a>
-<span class="sourceLineNo">268</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.268"></a>
-<span class="sourceLineNo">269</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.269"></a>
-<span class="sourceLineNo">270</span> private int corePoolSize;<a name="line.270"></a>
-<span class="sourceLineNo">271</span> private int maxPoolSize;<a name="line.271"></a>
-<span class="sourceLineNo">272</span><a name="line.272"></a>
-<span class="sourceLineNo">273</span> private volatile long keepAliveTime;<a name="line.273"></a>
-<span class="sourceLineNo">274</span><a name="line.274"></a>
-<span class="sourceLineNo">275</span> /**<a name="line.275"></a>
-<span class="sourceLineNo">276</span> * Scheduler/Queue that contains runnable procedures.<a name="line.276"></a>
-<span class="sourceLineNo">277</span> */<a name="line.277"></a>
-<span class="sourceLineNo">278</span> private final ProcedureScheduler scheduler;<a name="line.278"></a>
-<span class="sourceLineNo">279</span><a name="line.279"></a>
-<span class="sourceLineNo">280</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.280"></a>
-<span class="sourceLineNo">281</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.281"></a>
-<span class="sourceLineNo">282</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.282"></a>
-<span class="sourceLineNo">283</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.283"></a>
-<span class="sourceLineNo">284</span> private final TEnvironment environment;<a name="line.284"></a>
-<span class="sourceLineNo">285</span> private final ProcedureStore store;<a name="line.285"></a>
-<span class="sourceLineNo">286</span><a name="line.286"></a>
-<span class="sourceLineNo">287</span> private final boolean checkOwnerSet;<a name="line.287"></a>
-<span class="sourceLineNo">288</span><a name="line.288"></a>
-<span class="sourceLineNo">289</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.289"></a>
-<span class="sourceLineNo">290</span> final ProcedureStore store) {<a name="line.290"></a>
-<span class="sourceLineNo">291</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.291"></a>
-<span class="sourceLineNo">292</span> }<a name="line.292"></a>
-<span class="sourceLineNo">293</span><a name="line.293"></a>
-<span class="sourceLineNo">294</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.294"></a>
-<span class="sourceLineNo">295</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.295"></a>
-<span class="sourceLineNo">296</span> this.environment = environment;<a name="line.296"></a>
-<span class="sourceLineNo">297</span> this.scheduler = scheduler;<a name="line.297"></a>
-<span class="sourceLineNo">298</span> this.store = store;<a name="line.298"></a>
-<span class="sourceLineNo">299</span> this.conf = conf;<a name="line.299"></a>
-<span class="sourceLineNo">300</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.300"></a>
-<span class="sourceLineNo">301</span> refreshConfiguration(conf);<a name="line.301"></a>
-<span class="sourceLineNo">302</span> }<a name="line.302"></a>
-<span class="sourceLineNo">303</span><a name="line.303"></a>
-<span class="sourceLineNo">304</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.304"></a>
-<span class="sourceLineNo">305</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.305"></a>
-<span class="sourceLineNo">306</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.306"></a>
-<span class="sourceLineNo">307</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.307"></a>
-<span class="sourceLineNo">308</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.308"></a>
-<span class="sourceLineNo">309</span><a name="line.309"></a>
-<span class="sourceLineNo">310</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.310"></a>
-<span class="sourceLineNo">311</span> @Override<a name="line.311"></a>
-<span class="sourceLineNo">312</span> public void setMaxProcId(long maxProcId) {<a name="line.312"></a>
-<span class="sourceLineNo">313</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.313"></a>
-<span class="sourceLineNo">314</span> lastProcId.set(maxProcId);<a name="line.314"></a>
-<span class="sourceLineNo">315</span> }<a name="line.315"></a>
-<span class="sourceLineNo">316</span><a name="line.316"></a>
-<span class="sourceLineNo">317</span> @Override<a name="line.317"></a>
-<span class="sourceLineNo">318</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.318"></a>
-<span class="sourceLineNo">319</span> loadProcedures(procIter, abortOnCorruption);<a name="line.319"></a>
-<span class="sourceLineNo">320</span> }<a name="line.320"></a>
-<span class="sourceLineNo">321</span><a name="line.321"></a>
-<span class="sourceLineNo">322</span> @Override<a name="line.322"></a>
-<span class="sourceLineNo">323</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.323"></a>
-<span class="sourceLineNo">324</span> int corruptedCount = 0;<a name="line.324"></a>
-<span class="sourceLineNo">325</span> while (procIter.hasNext()) {<a name="line.325"></a>
-<span class="sourceLineNo">326</span> Procedure<?> proc = procIter.next();<a name="line.326"></a>
-<span class="sourceLineNo">327</span> LOG.error("Corrupt " + proc);<a name="line.327"></a>
-<span class="sourceLineNo">328</span> corruptedCount++;<a name="line.328"></a>
-<span class="sourceLineNo">329</span> }<a name="line.329"></a>
-<span class="sourceLineNo">330</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.330"></a>
-<span class="sourceLineNo">331</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.331"></a>
-<span class="sourceLineNo">332</span> }<a name="line.332"></a>
-<span class="sourceLineNo">333</span> }<a name="line.333"></a>
-<span class="sourceLineNo">334</span> });<a name="line.334"></a>
-<span class="sourceLineNo">335</span> }<a name="line.335"></a>
-<span class="sourceLineNo">336</span><a name="line.336"></a>
-<span class="sourceLineNo">337</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.337"></a>
-<span class="sourceLineNo">338</span> final boolean abortOnCorruption) throws IOException {<a name="line.338"></a>
-<span class="sourceLineNo">339</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.339"></a>
-<span class="sourceLineNo">340</span><a name="line.340"></a>
-<span class="sourceLineNo">341</span> // 1. Build the rollback stack<a name="line.341"></a>
-<span class="sourceLineNo">342</span> int runnablesCount = 0;<a name="line.342"></a>
-<span class="sourceLineNo">343</span> while (procIter.hasNext()) {<a name="line.343"></a>
-<span class="sourceLineNo">344</span> boolean finished = procIter.isNextFinished();<a name="line.344"></a>
-<span class="sourceLineNo">345</span> Procedure proc = procIter.next();<a name="line.345"></a>
-<span class="sourceLineNo">346</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.346"></a>
-<span class="sourceLineNo">347</span> long procId = proc.getProcId();<a name="line.347"></a>
-<span class="sourceLineNo">348</span><a name="line.348"></a>
-<span class="sourceLineNo">349</span> if (finished) {<a name="line.349"></a>
-<span class="sourceLineNo">350</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.350"></a>
-<span class="sourceLineNo">351</span> if (debugEnabled) {<a name="line.351"></a>
-<span class="sourceLineNo">352</span> LOG.debug("Completed " + proc);<a name="line.352"></a>
-<span class="sourceLineNo">353</span> }<a name="line.353"></a>
-<span class="sourceLineNo">354</span> } else {<a name="line.354"></a>
-<span class="sourceLineNo">355</span> if (!proc.hasParent()) {<a name="line.355"></a>
-<span class="sourceLineNo">356</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.356"></a>
-<span class="sourceLineNo">357</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.357"></a>
-<span class="sourceLineNo">358</span> }<a name="line.358"></a>
+<span class="sourceLineNo">267</span><a name="line.267"></a>
+<span class="sourceLineNo">268</span> /**<a name="line.268"></a>
+<span class="sourceLineNo">269</span> * Created in the {@link #start(int, boolean)} method. Destroyed in {@link #join()} (FIX! Doing<a name="line.269"></a>
+<span class="sourceLineNo">270</span> * resource handling rather than observing in a #join is unexpected).<a name="line.270"></a>
+<span class="sourceLineNo">271</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.271"></a>
+<span class="sourceLineNo">272</span> * (Should be ok).<a name="line.272"></a>
+<span class="sourceLineNo">273</span> */<a name="line.273"></a>
+<span class="sourceLineNo">274</span> private ThreadGroup threadGroup;<a name="line.274"></a>
+<span class="sourceLineNo">275</span><a name="line.275"></a>
+<span class="sourceLineNo">276</span> /**<a name="line.276"></a>
+<span class="sourceLineNo">277</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.277"></a>
+<span class="sourceLineNo">278</span> * resource handling rather than observing in a #join is unexpected).<a name="line.278"></a>
+<span class="sourceLineNo">279</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.279"></a>
+<span class="sourceLineNo">280</span> * (Should be ok).<a name="line.280"></a>
+<span class="sourceLineNo">281</span> */<a name="line.281"></a>
+<span class="sourceLineNo">282</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.282"></a>
+<span class="sourceLineNo">283</span><a name="line.283"></a>
+<span class="sourceLineNo">284</span> /**<a name="line.284"></a>
+<span class="sourceLineNo">285</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.285"></a>
+<span class="sourceLineNo">286</span> * resource handling rather than observing in a #join is unexpected).<a name="line.286"></a>
+<span class="sourceLineNo">287</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.287"></a>
+<span class="sourceLineNo">288</span> * (Should be ok).<a name="line.288"></a>
+<span class="sourceLineNo">289</span> */<a name="line.289"></a>
+<span class="sourceLineNo">290</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.290"></a>
+<span class="sourceLineNo">291</span><a name="line.291"></a>
+<span class="sourceLineNo">292</span> private int corePoolSize;<a name="line.292"></a>
+<span class="sourceLineNo">293</span> private int maxPoolSize;<a name="line.293"></a>
+<span class="sourceLineNo">294</span><a name="line.294"></a>
+<span class="sourceLineNo">295</span> private volatile long keepAliveTime;<a name="line.295"></a>
+<span class="sourceLineNo">296</span><a name="line.296"></a>
+<span class="sourceLineNo">297</span> /**<a name="line.297"></a>
+<span class="sourceLineNo">298</span> * Scheduler/Queue that contains runnable procedures.<a name="line.298"></a>
+<span class="sourceLineNo">299</span> */<a name="line.299"></a>
+<span class="sourceLineNo">300</span> private final ProcedureScheduler scheduler;<a name="line.300"></a>
+<span class="sourceLineNo">301</span><a name="line.301"></a>
+<span class="sourceLineNo">302</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.302"></a>
+<span class="sourceLineNo">303</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.303"></a>
+<span class="sourceLineNo">304</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.304"></a>
+<span class="sourceLineNo">305</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.305"></a>
+<span class="sourceLineNo">306</span> private final TEnvironment environment;<a name="line.306"></a>
+<span class="sourceLineNo">307</span> private final ProcedureStore store;<a name="line.307"></a>
+<span class="sourceLineNo">308</span><a name="line.308"></a>
+<span class="sourceLineNo">309</span> private final boolean checkOwnerSet;<a name="line.309"></a>
+<span class="sourceLineNo">310</span><a name="line.310"></a>
+<span class="sourceLineNo">311</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.311"></a>
+<span class="sourceLineNo">312</span> final ProcedureStore store) {<a name="line.312"></a>
+<span class="sourceLineNo">313</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.313"></a>
+<span class="sourceLineNo">314</span> }<a name="line.314"></a>
+<span class="sourceLineNo">315</span><a name="line.315"></a>
+<span class="sourceLineNo">316</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.316"></a>
+<span class="sourceLineNo">317</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.317"></a>
+<span class="sourceLineNo">318</span> this.environment = environment;<a name="line.318"></a>
+<span class="sourceLineNo">319</span> this.scheduler = scheduler;<a name="line.319"></a>
+<span class="sourceLineNo">320</span> this.store = store;<a name="line.320"></a>
+<span class="sourceLineNo">321</span> this.conf = conf;<a name="line.321"></a>
+<span class="sourceLineNo">322</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.322"></a>
+<span class="sourceLineNo">323</span> refreshConfiguration(conf);<a name="line.323"></a>
+<span class="sourceLineNo">324</span><a name="line.324"></a>
+<span class="sourceLineNo">325</span> }<a name="line.325"></a>
+<span class="sourceLineNo">326</span><a name="line.326"></a>
+<span class="sourceLineNo">327</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.327"></a>
+<span class="sourceLineNo">328</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.328"></a>
+<span class="sourceLineNo">329</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.329"></a>
+<span class="sourceLineNo">330</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.330"></a>
+<span class="sourceLineNo">331</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.331"></a>
+<span class="sourceLineNo">332</span><a name="line.332"></a>
+<span class="sourceLineNo">333</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.333"></a>
+<span class="sourceLineNo">334</span> @Override<a name="line.334"></a>
+<span class="sourceLineNo">335</span> public void setMaxProcId(long maxProcId) {<a name="line.335"></a>
+<span class="sourceLineNo">336</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.336"></a>
+<span class="sourceLineNo">337</span> lastProcId.set(maxProcId);<a name="line.337"></a>
+<span class="sourceLineNo">338</span> }<a name="line.338"></a>
+<span class="sourceLineNo">339</span><a name="line.339"></a>
+<span class="sourceLineNo">340</span> @Override<a name="line.340"></a>
+<span class="sourceLineNo">341</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.341"></a>
+<span class="sourceLineNo">342</span> loadProcedures(procIter, abortOnCorruption);<a name="line.342"></a>
+<span class="sourceLineNo">343</span> }<a name="line.343"></a>
+<span class="sourceLineNo">344</span><a name="line.344"></a>
+<span class="sourceLineNo">345</span> @Override<a name="line.345"></a>
+<span class="sourceLineNo">346</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.346"></a>
+<span class="sourceLineNo">347</span> int corruptedCount = 0;<a name="line.347"></a>
+<span class="sourceLineNo">348</span> while (procIter.hasNext()) {<a name="line.348"></a>
+<span class="sourceLineNo">349</span> Procedure<?> proc = procIter.next();<a name="line.349"></a>
+<span class="sourceLineNo">350</span> LOG.error("Corrupt " + proc);<a name="line.350"></a>
+<span class="sourceLineNo">351</span> corruptedCount++;<a name="line.351"></a>
+<span class="sourceLineNo">352</span> }<a name="line.352"></a>
+<span class="sourceLineNo">353</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.353"></a>
+<span class="sourceLineNo">354</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.354"></a>
+<span class="sourceLineNo">355</span> }<a name="line.355"></a>
+<span class="sourceLineNo">356</span> }<a name="line.356"></a>
+<span class="sourceLineNo">357</span> });<a name="line.357"></a>
+<span class="sourceLineNo">358</span> }<a name="line.358"></a>
<span class="sourceLineNo">359</span><a name="line.359"></a>
-<span class="sourceLineNo">360</span> // add the procedure to the map<a name="line.360"></a>
-<span class="sourceLineNo">361</span> proc.beforeReplay(getEnvironment());<a name="line.361"></a>
-<span class="sourceLineNo">362</span> procedures.put(proc.getProcId(), proc);<a name="line.362"></a>
+<span class="sourceLineNo">360</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.360"></a>
+<span class="sourceLineNo">361</span> final boolean abortOnCorruption) throws IOException {<a name="line.361"></a>
+<span class="sourceLineNo">362</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.362"></a>
<span class="sourceLineNo">363</span><a name="line.363"></a>
-<span class="sourceLineNo">364</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.364"></a>
-<span class="sourceLineNo">365</span> runnablesCount++;<a name="line.365"></a>
-<span class="sourceLineNo">366</span> }<a name="line.366"></a>
-<span class="sourceLineNo">367</span> }<a name="line.367"></a>
-<span class="sourceLineNo">368</span><a name="line.368"></a>
-<span class="sourceLineNo">369</span> // add the nonce to the map<a name="line.369"></a>
-<span class="sourceLineNo">370</span> if (nonceKey != null) {<a name="line.370"></a>
-<span class="sourceLineNo">371</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.371"></a>
-<span class="sourceLineNo">372</span> }<a name="line.372"></a>
-<span class="sourceLineNo">373</span> }<a name="line.373"></a>
-<span class="sourceLineNo">374</span><a name="line.374"></a>
-<span class="sourceLineNo">375</span> // 2. Initialize the stacks<a name="line.375"></a>
-<span class="sourceLineNo">376</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.376"></a>
-<span class="sourceLineNo">377</span> HashSet<Procedure> waitingSet = null;<a name="line.377"></a>
-<span class="sourceLineNo">378</span> procIter.reset();<a name="line.378"></a>
-<span class="sourceLineNo">379</span> while (procIter.hasNext()) {<a name="line.379"></a>
-<span class="sourceLineNo">380</span> if (procIter.isNextFinished()) {<a name="line.380"></a>
-<span class="sourceLineNo">381</span> procIter.skipNext();<a name="line.381"></a>
-<span class="sourceLineNo">382</span> continue;<a name="line.382"></a>
-<span class="sourceLineNo">383</span> }<a name="line.383"></a>
-<span class="sourceLineNo">384</span><a name="line.384"></a>
-<span class="sourceLineNo">385</span> Procedure proc = procIter.next();<a name="line.385"></a>
-<span class="sourceLineNo">386</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.386"></a>
-<span class="sourceLineNo">387</span><a name="line.387"></a>
-<span class="sourceLineNo">388</span> if (debugEnabled) {<a name="line.388"></a>
-<span class="sourceLineNo">389</span> LOG.debug(String.format("Loading %s", proc));<a name="line.389"></a>
+<span class="sourceLineNo">364</span> // 1. Build the rollback stack<a name="line.364"></a>
+<span class="sourceLineNo">365</span> int runnablesCount = 0;<a name="line.365"></a>
+<span class="sourceLineNo">366</span> while (procIter.hasNext()) {<a name="line.366"></a>
+<span class="sourceLineNo">367</span> boolean finished = procIter.isNextFinished();<a name="line.367"></a>
+<span class="sourceLineNo">368</span> Procedure proc = procIter.next();<a name="line.368"></a>
+<span class="sourceLineNo">369</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.369"></a>
+<span class="sourceLineNo">370</span> long procId = proc.getProcId();<a name="line.370"></a>
+<span class="sourceLineNo">371</span><a name="line.371"></a>
+<span class="sourceLineNo">372</span> if (finished) {<a name="line.372"></a>
+<span class="sourceLineNo">373</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.373"></a>
+<span class="sourceLineNo">374</span> if (debugEnabled) {<a name="line.374"></a>
+<span class="sourceLineNo">375</span> LOG.debug("Completed " + proc);<a name="line.375"></a>
+<span class="sourceLineNo">376</span> }<a name="line.376"></a>
+<span class="sourceLineNo">377</span> } else {<a name="line.377"></a>
+<span class="sourceLineNo">378</span> if (!proc.hasParent()) {<a name="line.378"></a>
+<span class="sourceLineNo">379</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.379"></a>
+<span class="sourceLineNo">380</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.380"></a>
+<span class="sourceLineNo">381</span> }<a name="line.381"></a>
+<span class="sourceLineNo">382</span><a name="line.382"></a>
+<span class="sourceLineNo">383</span> // add the procedure to the map<a name="line.383"></a>
+<span class="sourceLineNo">384</span> proc.beforeReplay(getEnvironment());<a name="line.384"></a>
+<span class="sourceLineNo">385</span> procedures.put(proc.getProcId(), proc);<a name="line.385"></a>
+<span class="sourceLineNo">386</span><a name="line.386"></a>
+<span class="sourceLineNo">387</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.387"></a>
+<span class="sourceLineNo">388</span> runnablesCount++;<a name="line.388"></a>
+<span class="sourceLineNo">389</span> }<a name="line.389"></a>
<span class="sourceLineNo">390</span> }<a name="line.390"></a>
<span class="sourceLineNo">391</span><a name="line.391"></a>
-<span class="sourceLineNo">392</span> Long rootProcId = getRootProcedureId(proc);<a name="line.392"></a>
-<span class="sourceLineNo">393</span> if (rootProcId == null) {<a name="line.393"></a>
-<span class="sourceLineNo">394</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.394"></a>
-<span class="sourceLineNo">395</span> scheduler.addBack(proc);<a name="line.395"></a>
-<span class="sourceLineNo">396</span> continue;<a name="line.396"></a>
-<span class="sourceLineNo">397</span> }<a name="line.397"></a>
-<span class="sourceLineNo">398</span><a name="line.398"></a>
-<span class="sourceLineNo">399</span> if (proc.hasParent()) {<a name="line.399"></a>
-<span class="sourceLineNo">400</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.400"></a>
-<span class="sourceLineNo">401</span> // corrupted procedures are handled later at step 3<a name="line.401"></a>
-<span class="sourceLineNo">402</span> if (parent != null && !proc.isFinished()) {<a name="line.402"></a>
-<span class="sourceLineNo">403</span> parent.incChildrenLatch();<a name="line.403"></a>
-<span class="sourceLineNo">404</span> }<a name="line.404"></a>
-<span class="sourceLineNo">405</span> }<a name="line.405"></a>
-<span class="sourceLineNo">406</span><a name="line.406"></a>
-<span class="sourceLineNo">407</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.407"></a>
-<span class="sourceLineNo">408</span> procStack.loadStack(proc);<a name="line.408"></a>
-<span class="sourceLineNo">409</span><a name="line.409"></a>
-<span class="sourceLineNo">410</span> proc.setRootProcId(rootProcId);<a name="line.410"></a>
-<span class="sourceLineNo">411</span> switch (proc.getState()) {<a name="line.411"></a>
-<span class="sourceLineNo">412</span> case RUNNABLE:<a name="line.412"></a>
-<span class="sourceLineNo">413</span> runnableList.add(proc);<a name="line.413"></a>
-<span class="sourceLineNo">414</span> break;<a name="line.414"></a>
-<span class="sourceLineNo">415</span> case WAITING:<a name="line.415"></a>
-<span class="sourceLineNo">416</span> if (!proc.hasChildren()) {<a name="line.416"></a>
-<span class="sourceLineNo">417</span> runnableList.add(proc);<a name="line.417"></a>
-<span class="sourceLineNo">418</span> }<a name="line.418"></a>
-<span class="sourceLineNo">419</span> break;<a name="line.419"></a>
-<span class="sourceLineNo">420</span> case WAITING_TIMEOUT:<a name="line.420"></a>
-<span class="sourceLineNo">421</span> if (waitingSet == null) {<a name="line.421"></a>
-<span class="sourceLineNo">422</span> waitingSet = new HashSet<>();<a name="line.422"></a>
-<span class="sourceLineNo">423</span> }<a name="line.423"></a>
-<span class="sourceLineNo">424</span> waitingSet.add(proc);<a name="line.424"></a>
-<span class="sourceLineNo">425</span> break;<a name="line.425"></a>
-<span class="sourceLineNo">426</span> case FAILED:<a name="line.426"></a>
-<span class="sourceLineNo">427</span> // add the proc to the scheduler to perform the rollback<a name="line.427"></a>
-<span class="sourceLineNo">428</span> scheduler.addBack(proc);<a name="line.428"></a>
-<span class="sourceLineNo">429</span> break;<a name="line.429"></a>
-<span class="sourceLineNo">430</span> case ROLLEDBACK:<a name="line.430"></a>
-<span class="sourceLineNo">431</span> case INITIALIZING:<a name="line.431"></a>
-<span class="sourceLineNo">432</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.432"></a>
-<span class="sourceLineNo">433</span> LOG.error(msg);<a name="line.433"></a>
-<span class="sourceLineNo">434</span> throw new UnsupportedOperationException(msg);<a name="line.434"></a>
-<span class="sourceLineNo">435</span> default:<a name="line.435"></a>
-<span class="sourceLineNo">436</span> break;<a name="line.436"></a>
-<span class="sourceLineNo">437</span> }<a name="line.437"></a>
-<span class="sourceLineNo">438</span> }<a name="line.438"></a>
-<span class="sourceLineNo">439</span><a name="line.439"></a>
-<span class="sourceLineNo">440</span> // 3. Validate the stacks<a name="line.440"></a>
-<span class="sourceLineNo">441</span> int corruptedCount = 0;<a name="line.441"></a>
-<span class="sourceLineNo">442</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.442"></a>
-<span class="sourceLineNo">443</span> while (itStack.hasNext()) {<a name="line.443"></a>
-<span class="sourceLineNo">444</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.444"></a>
-<span class="sourceLineNo">445</span> RootProcedureState procStack = entry.getValue();<a name="line.445"></a>
-<span class="sourceLineNo">446</span> if (procStack.isValid()) continue;<a name="line.446"></a>
-<span class="sourceLineNo">447</span><a name="line.447"></a>
-<span class="sourceLineNo">448</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.448"></a>
-<span class="sourceLineNo">449</span> LOG.error("Corrupted " + proc);<a name="line.449"></a>
-<span class="sourceLineNo">450</span> procedures.remove(proc.getProcId());<a name="line.450"></a>
-<span class="sourceLineNo">451</span> runnableList.remove(proc);<a name="line.451"></a>
-<span class="sourceLineNo">452</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.452"></a>
-<span class="sourceLineNo">453</span> corruptedCount++;<a name="line.453"></a>
-<span class="sourceLineNo">454</span> }<a name="line.454"></a>
-<span class="sourceLineNo">455</span> itStack.remove();<a name="line.455"></a>
-<span class="sourceLineNo">456</span> }<a name="line.456"></a>
-<span class="sourceLineNo">457</span><a name="line.457"></a>
-<span class="sourceLineNo">458</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.458"></a>
-<span class="sourceLineNo">459</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.459"></a>
-<span class="sourceLineNo">460</span> }<a name="line.460"></a>
-<span class="sourceLineNo">461</span><a name="line.461"></a>
-<span class="sourceLineNo">462</span> // 4. Push the procedures to the timeout executor<a name="line.462"></a>
-<span class="sourceLineNo">463</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.463"></a>
-<span class="sourceLineNo">464</span> for (Procedure proc: waitingSet) {<a name="line.464"></a>
-<span class="sourceLineNo">465</span> proc.afterReplay(getEnvironment());<a name="line.465"></a>
-<span class="sourceLineNo">466</span> timeoutExecutor.add(proc);<a name="line.466"></a>
-<span class="sourceLineNo">467</span> }<a name="line.467"></a>
-<span class="sourceLineNo">468</span> }<a name="line.468"></a>
-<span class="sourceLineNo">469</span><a name="line.469"></a>
-<span class="sourceLineNo">470</span> // 5. Push the procedure to the scheduler<a name="line.470"></a>
-<span class="sourceLineNo">471</span> if (!runnableList.isEmpty()) {<a name="line.471"></a>
-<span class="sourceLineNo">472</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.472"></a>
-<span class="sourceLineNo">473</span> // some procedure may be started way before this stuff.<a name="line.473"></a>
-<span class="sourceLineNo">474</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.474"></a>
-<span class="sourceLineNo">475</span> Procedure proc = runnableList.get(i);<a name="line.475"></a>
-<span class="sourceLineNo">476</span> proc.afterReplay(getEnvironment());<a name="line.476"></a>
-<span class="sourceLineNo">477</span> if (!proc.hasParent()) {<a name="line.477"></a>
-<span class="sourceLineNo">478</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.478"></a>
-<span class="sourceLineNo">479</span> }<a name="line.479"></a>
-<span class="sourceLineNo">480</span> if (proc.wasExecuted()) {<a name="line.480"></a>
-<span class="sourceLineNo">481</span> scheduler.addFront(proc);<a name="line.481"></a>
-<span class="sourceLineNo">482</span> } else {<a name="line.482"></a>
-<span class="sourceLineNo">483</span> // if it was not in execution, it can wait.<a name="line.483"></a>
-<span class="sourceLineNo">484</span> scheduler.addBack(proc);<a name="line.484"></a>
-<span class="sourceLineNo">485</span> }<a name="line.485"></a>
-<span class="sourceLineNo">486</span> }<a name="line.486"></a>
-<span class="sourceLineNo">487</span> }<a name="line.487"></a>
-<span class="sourceLineNo">488</span> }<a name="line.488"></a>
-<span class="sourceLineNo">489</span><a name="line.489"></a>
-<span class="sourceLineNo">490</span> /**<a name="line.490"></a>
-<span class="sourceLineNo">491</span> * Start the procedure executor.<a name="line.491"></a>
-<span class="sourceLineNo">492</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.492"></a>
-<span class="sourceLineNo">493</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.493"></a>
-<span class="sourceLineNo">494</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.494"></a>
-<span class="sourceLineNo">495</span> *<a name="line.495"></a>
-<span class="sourceLineNo">496</span> * @param numThreads number of threads available for procedure execution.<a name="line.496"></a>
-<span class="sourceLineNo">497</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.497"></a>
-<span class="sourceLineNo">498</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.498"></a>
-<span class="sourceLineNo">499</span> */<a name="line.499"></a>
-<span class="sourceLineNo">500</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.500"></a>
-<span class="sourceLineNo">501</span> if (!running.compareAndSet(false, true)) {<a name="line.501"></a>
-<span class="sourceLineNo">502</span> LOG.warn("Already running");<a name="line.502"></a>
-<span class="sourceLineNo">503</span> return;<a name="line.503"></a>
-<span class="sourceLineNo">504</span> }<a name="line.504"></a>
-<span class="sourceLineNo">505</span><a name="line.505"></a>
-<span class="sourceLineNo">506</span> // We have numThreads executor + one timer thread used for timing out<a name="line.506"></a>
-<span class="sourceLineNo">507</span> // procedures and triggering periodic procedures.<a name="line.507"></a>
-<span class="sourceLineNo">508</span> this.corePoolSize = numThreads;<a name="line.508"></a>
-<span class="sourceLineNo">509</span> this.maxPoolSize = 10 * numThreads;<a name="line.509"></a>
-<span class="sourceLineNo">510</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.510"></a>
-<span class="sourceLineNo">511</span> corePoolSize, maxPoolSize);<a name="line.511"></a>
+<span class="sourceLineNo">392</span> // add the nonce to the map<a name="line.392"></a>
+<span class="sourceLineNo">393</span> if (nonceKey != null) {<a name="line.393"></a>
+<span class="sourceLineNo">394</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.394"></a>
+<span class="sourceLineNo">395</span> }<a name="line.395"></a>
+<span class="sourceLineNo">396</span> }<a name="line.396"></a>
+<span class="sourceLineNo">397</span><a name="line.397"></a>
+<span class="sourceLineNo">398</span> // 2. Initialize the stacks<a name="line.398"></a>
+<span class="sourceLineNo">399</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.399"></a>
+<span class="sourceLineNo">400</span> HashSet<Procedure> waitingSet = null;<a name="line.400"></a>
+<span class="sourceLineNo">401</span> procIter.reset();<a name="line.401"></a>
+<span class="sourceLineNo">402</span> while (procIter.hasNext()) {<a name="line.402"></a>
+<span class="sourceLineNo">403</span> if (procIter.isNextFinished()) {<a name="line.403"></a>
+<span class="sourceLineNo">404</span> procIter.skipNext();<a name="line.404"></a>
+<span class="sourceLineNo">405</span> continue;<a name="line.405"></a>
+<span class="sourceLineNo">406</span> }<a name="line.406"></a>
+<span class="sourceLineNo">407</span><a name="line.407"></a>
+<span class="sourceLineNo">408</span> Procedure proc = procIter.next();<a name="line.408"></a>
+<span class="sourceLineNo">409</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.409"></a>
+<span class="sourceLineNo">410</span><a name="line.410"></a>
+<span class="sourceLineNo">411</span> if (debugEnabled) {<a name="line.411"></a>
+<span class="sourceLineNo">412</span> LOG.debug(String.format("Loading %s", proc));<a name="line.412"></a>
+<span class="sourceLineNo">413</span> }<a name="line.413"></a>
+<span class="sourceLineNo">414</span><a name="line.414"></a>
+<span class="sourceLineNo">415</span> Long rootProcId = getRootProcedureId(proc);<a name="line.415"></a>
+<span class="sourceLineNo">416</span> if (rootProcId == null) {<a name="line.416"></a>
+<span class="sourceLineNo">417</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.417"></a>
+<span class="sourceLineNo">418</span> scheduler.addBack(proc);<a name="line.418"></a>
+<span class="sourceLineNo">419</span> continue;<a name="line.419"></a>
+<span class="sourceLineNo">420</span> }<a name="line.420"></a>
+<span class="sourceLineNo">421</span><a name="line.421"></a>
+<span class="sourceLineNo">422</span> if (proc.hasParent()) {<a name="line.422"></a>
+<span class="sourceLineNo">423</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.423"></a>
+<span class="sourceLineNo">424</span> // corrupted procedures are handled later at step 3<a name="line.424"></a>
+<span class="sourceLineNo">425</span> if (parent != null && !proc.isFinished()) {<a name="line.425"></a>
+<span class="sourceLineNo">426</span> parent.incChildrenLatch();<a name="line.426"></a>
+<span class="sourceLineNo">427</span> }<a name="line.427"></a>
+<span class="sourceLineNo">428</span> }<a name="line.428"></a>
+<span class="sourceLineNo">429</span><a name="line.429"></a>
+<span class="sourceLineNo">430</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.430"></a>
+<span class="sourceLineNo">431</span> procStack.loadStack(proc);<a name="line.431"></a>
+<span class="sourceLineNo">432</span><a name="line.432"></a>
+<span class="sourceLineNo">433</span> proc.setRootProcId(rootProcId);<a name="line.433"></a>
+<span class="sourceLineNo">434</span> switch (proc.getState()) {<a name="line.434"></a>
+<span class="sourceLineNo">435</span> case RUNNABLE:<a name="line.435"></a>
+<span class="sourceLineNo">436</span> runnableList.add(proc);<a name="line.436"></a>
+<span class="sourceLineNo">437</span> break;<a name="line.437"></a>
+<span class="sourceLineNo">438</span> case WAITING:<a name="line.438"></a>
+<span class="sourceLineNo">439</span> if (!proc.hasChildren()) {<a name="line.439"></a>
+<span class="sourceLineNo">440</span> runnableList.add(proc);<a name="line.440"></a>
+<span class="sourceLineNo">441</span> }<a name="line.441"></a>
+<span class="sourceLineNo">442</span> break;<a name="line.442"></a>
+<span class="sourceLineNo">443</span> case WAITING_TIMEOUT:<a name="line.443"></a>
+<span class="sourceLineNo">444</span> if (waitingSet == null) {<a name="line.444"></a>
+<span class="sourceLineNo">445</span> waitingSet = new HashSet<>();<a name="line.445"></a>
+<span class="sourceLineNo">446</span> }<a name="line.446"></a>
+<span class="sourceLineNo">447</span> waitingSet.add(proc);<a name="line.447"></a>
+<span class="sourceLineNo">448</span> break;<a name="line.448"></a>
+<span class="sourceLineNo">449</span> case FAILED:<a name="line.449"></a>
+<span class="sourceLineNo">450</span> // add the proc to the scheduler to perform the rollback<a name="line.450"></a>
+<span class="sourceLineNo">451</span> scheduler.addBack(proc);<a name="line.451"></a>
+<span class="sourceLineNo">452</span> break;<a name="line.452"></a>
+<span class="sourceLineNo">453</span> case ROLLEDBACK:<a name="line.453"></a>
+<span class="sourceLineNo">454</span> case INITIALIZING:<a name="line.454"></a>
+<span class="sourceLineNo">455</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.455"></a>
+<span class="sourceLineNo">456</span> LOG.error(msg);<a name="line.456"></a>
+<span class="sourceLineNo">457</span> throw new UnsupportedOperationException(msg);<a name="line.457"></a>
+<span class="sourceLineNo">458</span> default:<a name="line.458"></a>
+<span class="sourceLineNo">459</span> break;<a name="line.459"></a>
+<span class="sourceLineNo">460</span> }<a name="line.460"></a>
+<span class="sourceLineNo">461</span> }<a name="line.461"></a>
+<span class="sourceLineNo">462</span><a name="line.462"></a>
+<span class="sourceLineNo">463</span> // 3. Validate the stacks<a name="line.463"></a>
+<span class="sourceLineNo">464</span> int corruptedCount = 0;<a name="line.464"></a>
+<span class="sourceLineNo">465</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.465"></a>
+<span class="sourceLineNo">466</span> while (itStack.hasNext()) {<a name="line.466"></a>
+<span class="sourceLineNo">467</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.467"></a>
+<span class="sourceLineNo">468</span> RootProcedureState procStack = entry.getValue();<a name="line.468"></a>
+<span class="sourceLineNo">469</span> if (procStack.isValid()) continue;<a name="line.469"></a>
+<span class="sourceLineNo">470</span><a name="line.470"></a>
+<span class="sourceLineNo">471</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.471"></a>
+<span class="sourceLineNo">472</span> LOG.error("Corrupted " + proc);<a name="line.472"></a>
+<span class="sourceLineNo">473</span> procedures.remove(proc.getProcId());<a name="line.473"></a>
+<span class="sourceLineNo">474</span> runnableList.remove(proc);<a name="line.474"></a>
+<span class="sourceLineNo">475</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.475"></a>
+<span class="sourceLineNo">476</span> corruptedCount++;<a name="line.476"></a>
+<span class="sourceLineNo">477</span> }<a name="line.477"></a>
+<span class="sourceLineNo">478</span> itStack.remove();<a name="line.478"></a>
+<span class="sourceLineNo">479</span> }<a name="line.479"></a>
+<span class="sourceLineNo">480</span><a name="line.480"></a>
+<span class="sourceLineNo">481</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.481"></a>
+<span class="sourceLineNo">482</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.482"></a>
+<span class="sourceLineNo">483</span> }<a name="line.483"></a>
+<span class="sourceLineNo">484</span><a name="line.484"></a>
+<span class="sourceLineNo">485</span> // 4. Push the procedures to the timeout executor<a name="line.485"></a>
+<span class="sourceLineNo">486</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.486"></a>
+<span class="sourceLineNo">487</span> for (Procedure proc: waitingSet) {<a name="line.487"></a>
+<span class="sourceLineNo">488</span> proc.afterReplay(getEnvironment());<a name="line.488"></a>
+<span class="sourceLineNo">489</span> timeoutExecutor.add(proc);<a name="line.489"></a>
+<span class="sourceLineNo">490</span> }<a name="line.490"></a>
+<span class="sourceLineNo">491</span> }<a name="line.491"></a>
+<span class="sourceLineNo">492</span><a name="line.492"></a>
+<span class="sourceLineNo">493</span> // 5. Push the procedure to the scheduler<a name="line.493"></a>
+<span class="sourceLineNo">494</span> if (!runnableList.isEmpty()) {<a name="line.494"></a>
+<span class="sourceLineNo">495</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.495"></a>
+<span class="sourceLineNo">496</span> // some procedure may be started way before this stuff.<a name="line.496"></a>
+<span class="sourceLineNo">497</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.497"></a>
+<span class="sourceLineNo">498</span> Procedure proc = runnableList.get(i);<a name="line.498"></a>
+<span class="sourceLineNo">499</span> proc.afterReplay(getEnvironment());<a name="line.499"></a>
+<span class="sourceLineNo">500</span> if (!proc.hasParent()) {<a name="line.500"></a>
+<span class="sourceLineNo">501</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.501"></a>
+<span class="sourceLineNo">502</span> }<a name="line.502"></a>
+<span class="sourceLineNo">503</span> if (proc.wasExecuted()) {<a name="line.503"></a>
+<span class="sourceLineNo">504</span> scheduler.addFront(proc);<a name="line.504"></a>
+<span class="sourceLineNo">505</span> } else {<a name="line.505"></a>
+<span class="sourceLineNo">506</span> // if it was not in execution, it can wait.<a name="line.506"></a>
+<span class="sourceLineNo">507</span> scheduler.addBack(proc);<a name="line.507"></a>
+<span class="sourceLineNo">508</span> }<a name="line.508"></a>
+<span class="sourceLineNo">509</span> }<a name="line.509"></a>
+<span class="sourceLineNo">510</span> }<a name="line.510"></a>
+<span class="sourceLineNo">511</span> }<a name="line.511"></a>
<span class="sourceLineNo">512</span><a name="line.512"></a>
-<span class="sourceLineNo">513</span> // Create the Thread Group for the executors<a name="line.513"></a>
-<span class="sourceLineNo">514</span> threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.514"></a>
-<span class="sourceLineNo">515</span><a name="line.515"></a>
-<span class="sourceLineNo">516</span> // Create the timeout executor<a name="line.516"></a>
-<span class="sourceLineNo">517</span> timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.517"></a>
-<span class="sourceLineNo">518</span><a name="line.518"></a>
-<span class="sourceLineNo">519</span> // Create the workers<a name="line.519"></a>
-<span class="sourceLineNo">520</span> workerId.set(0);<a name="line.520"></a>
-<span class="sourceLineNo">521</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.521"></a>
-<span class="sourceLineNo">522</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.522"></a>
-<span class="sourceLineNo">523</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.523"></a>
-<span class="sourceLineNo">524</span> }<a name="line.524"></a>
-<span class="sourceLineNo">525</span><a name="line.525"></a>
-<span class="sourceLineNo">526</span> long st, et;<a name="line.526"></a>
-<span class="sourceLineNo">527</span><a name="line.527"></a>
-<span class="sourceLineNo">528</span> // Acquire the store lease.<a name="line.528"></a>
-<span class="sourceLineNo">529</span> st = EnvironmentEdgeManager.currentTime();<a name="line.529"></a>
-<span class="sourceLineNo">530</span> store.recoverLease();<a name="line.530"></a>
-<span class="sourceLineNo">531</span> et = EnvironmentEdgeManager.currentTime();<a name="line.531"></a>
-<span class="sourceLineNo">532</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.532"></a>
-<span class="sourceLineNo">533</span> StringUtils.humanTimeDiff(et - st));<a name="line.533"></a>
-<span class="sourceLineNo">534</span><a name="line.534"></a>
-<span class="sourceLineNo">535</span> // start the procedure scheduler<a name="line.535"></a>
-<span class="sourceLineNo">536</span> scheduler.start();<a name="line.536"></a>
-<span class="sourceLineNo">537</span><a name="line.537"></a>
-<span class="sourceLineNo">538</span> // TODO: Split in two steps.<a name="line.538"></a>
-<span class="sourceLineNo">539</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.539"></a>
-<span class="sourceLineNo">540</span> // The first one will make sure that we have the latest id,<a name="line.540"></a>
-<span class="sourceLineNo">541</span> // so we can start the threads and accept new procedures.<a name="line.541"></a>
-<span class="sourceLineNo">542</span> // The second step will do the actual load of old procedures.<a name="line.542"></a>
-<span class="sourceLineNo">543</span> st = EnvironmentEdgeManager.currentTime();<a name="line.543"></a>
-<span class="sourceLineNo">544</span> load(abortOnCorruption);<a name="line.544"></a>
-<span class="sourceLineNo">545</span> et = EnvironmentEdgeManager.currentTime();<a name="line.545"></a>
-<span class="sourceLineNo">546</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.546"></a>
-<span class="sourceLineNo">547</span> StringUtils.humanTimeDiff(et - st));<a name="line.547"></a>
-<span class="sourceLineNo">548</span><a name="line.548"></a>
-<span class="sourceLineNo">549</span> // Start the executors. Here we must have the lastProcId set.<a name="line.549"></a>
-<span class="sourceLineNo">550</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.550"></a>
-<span class="sourceLineNo">551</span> timeoutExecutor.start();<a name="line.551"></a>
-<span class="sourceLineNo">552</span> for (WorkerThread worker: workerThreads) {<a name="line.552"></a>
-<span class="sourceLineNo">553</span> worker.start();<a name="line.553"></a>
-<span class="sourceLineNo">554</span> }<a name="line.554"></a>
-<span class="sourceLineNo">555</span><a name="line.555"></a>
-<span class="sourceLineNo">556</span> // Internal chores<a name="line.556"></a>
-<span class="sourceLineNo">557</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.557"></a>
-<span class="sourceLineNo">558</span><a name="line.558"></a>
-<span class="sourceLineNo">559</span> // Add completed cleaner chore<a name="line.559"></a>
-<span class="sourceLineNo">560</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.560"></a>
-<span class="sourceLineNo">561</span> }<a name="line.561"></a>
-<span class="sourceLineNo">562</span><a name="line.562"></a>
-<span class="sourceLineNo">563</span> public void stop() {<a name="line.563"></a>
-<span class="sourceLineNo">564</span> if (!running.getAndSet(false)) {<a name="line.564"></a>
-<span class="sourceLineNo">565</span> return;<a name="line.565"></a>
-<span class="sourceLineNo">566</span> }<a name="line.566"></a>
-<span class="sourceLineNo">567</span><a name="line.567"></a>
-<span class="sourceLineNo">568</span> LOG.info("Stopping");<a name="line.568"></a>
-<span class="sourceLineNo">569</span> scheduler.stop();<a name="line.569"></a>
-<span class="sourceLineNo">570</span> timeoutExecutor.sendStopSignal();<a name="line.570"></a>
-<span class="sourceLineNo">571</span> }<a name="line.571"></a>
-<span class="sourceLineNo">572</span><a name="line.572"></a>
-<span class="sourceLineNo">573</span> @VisibleForTesting<a name="line.573"></a>
-<span class="sourceLineNo">574</span> public void join() {<a name="line.574"></a>
-<span class="sourceLineNo">575</span> assert !isRunning() : "expected not running";<a name="line.575"></a>
-<span class="sourceLineNo">576</span><a name="line.576"></a>
-<span class="sourceLineNo">577</span> // stop the timeout executor<a name="line.577"></a>
-<span class="sourceLineNo">578</span> timeoutExecutor.awaitTermination();<a name="line.578"></a>
-<span class="sourceLineNo">579</span> timeoutExecutor = null;<a name="line.579"></a>
-<span class="sourceLineNo">580</span><a name="line.580"></a>
-<span class="sourceLineNo">581</span> // stop the worker threads<a name="line.581"></a>
-<span class="sourceLineNo">582</span> for (WorkerThread worker: workerThreads) {<a name="line.582"></a>
-<span class="sourceLineNo">583</span> worker.awaitTermination();<a name="line.583"></a>
-<span class="sourceLineNo">584</span> }<a name="line.584"></a>
-<span class="sourceLineNo">585</span> workerThreads = null;<a name="line.585"></a>
-<span class="sourceLineNo">586</span><a name="line.586"></a>
-<span class="sourceLineNo">587</span> // Destroy the Thread Group for the executors<a name="line.587"></a>
-<span class="sourceLineNo">588</span> try {<a name="line.588"></a>
-<span class="sourceLineNo">589</span> threadGroup.destroy();<a name="line.589"></a>
-<span class="sourceLineNo">590</span> } catch (IllegalThreadStateException e) {<a name="line.590"></a>
-<span class="sourceLineNo">591</span> LOG.error("ThreadGroup " + threadGroup + " contains running threads; " + e.getMessage());<a name="line.591"></a>
-<span class="sourceLineNo">592</span> threadGroup.list();<a name="line.592"></a>
-<span class="sourceLineNo">593</span> } finally {<a name="line.593"></a>
-<span class="sourceLineNo">594</span> threadGroup = null;<a name="line.594"></a>
-<span class="sourceLineNo">595</span> }<a name="line.595"></a>
+<span class="sourceLineNo">513</span> /**<a name="line.513"></a>
+<span class="sourceLineNo">514</span> * Start the procedure executor.<a name="line.514"></a>
+<span class="sourceLineNo">515</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.515"></a>
+<span class="sourceLineNo">516</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.516"></a>
+<span class="sourceLineNo">517</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.517"></a>
+<span class="sourceLineNo">518</span> *<a name="line.518"></a>
+<span class="sourceLineNo">519</span> * @param numThreads number of threads available for procedure execution.<a name="line.519"></a>
+<span class="sourceLineNo">520</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.520"></a>
+<span class="sourceLineNo">521</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.521"></a>
+<span class="sourceLineNo">522</span> */<a name="line.522"></a>
+<span class="sourceLineNo">523</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.523"></a>
+<span class="sourceLineNo">524</span> if (!running.compareAndSet(false, true)) {<a name="line.524"></a>
+<span class="sourceLineNo">525</span> LOG.warn("Already running");<a name="line.525"></a>
+<span class="sourceLineNo">526</span> return;<a name="line.526"></a>
+<span class="sourceLineNo">527</span> }<a name="line.527"></a>
+<span class="sourceLineNo">528</span><a name="line.528"></a>
+<span class="sourceLineNo">529</span> // We have numThreads executor + one timer thread used for timing out<a name="line.529"></a>
+<span class="sourceLineNo">530</span> // procedures and triggering periodic procedures.<a name="line.530"></a>
+<span class="sourceLineNo">531</span> this.corePoolSize = numThreads;<a name="line.531"></a>
+<span class="sourceLineNo">532</span> this.maxPoolSize = 10 * numThreads;<a name="line.532"></a>
+<span class="sourceLineNo">533</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.533"></a>
+<span class="sourceLineNo">534</span> corePoolSize, maxPoolSize);<a name="line.534"></a>
+<span class="sourceLineNo">535</span><a name="line.535"></a>
+<span class="sourceLineNo">536</span> this.threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.536"></a>
+<span class="sourceLineNo">537</span> this.timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.537"></a>
+<span class="sourceLineNo">538</span><a name="line.538"></a>
+<span class="sourceLineNo">539</span> // Create the workers<a name="line.539"></a>
+<span class="sourceLineNo">540</span> workerId.set(0);<a name="line.540"></a>
+<span class="sourceLineNo">541</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.541"></a>
+<span class="sourceLineNo">542</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.542"></a>
+<span class="sourceLineNo">543</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.543"></a>
+<span class="sourceLineNo">544</span> }<a name="line.544"></a>
+<span class="sourceLineNo">545</span><a name="line.545"></a>
+<span class="sourceLineNo">546</span> long st, et;<a name="line.546"></a>
+<span class="sourceLineNo">547</span><a name="line.547"></a>
+<span class="sourceLineNo">548</span> // Acquire the store lease.<a name="line.548"></a>
+<span class="sourceLineNo">549</span> st = EnvironmentEdgeManager.currentTime();<a name="line.549"></a>
+<span class="sourceLineNo">550</span> store.recoverLease();<a name="line.550"></a>
+<span class="sourceLineNo">551</span> et = EnvironmentEdgeManager.currentTime();<a name="line.551"></a>
+<span class="sourceLineNo">552</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.552"></a>
+<span class="sourceLineNo">553</span> StringUtils.humanTimeDiff(et - st));<a name="line.553"></a>
+<span class="sourceLineNo">554</span><a name="line.554"></a>
+<span class="sourceLineNo">555</span> // start the procedure scheduler<a name="line.555"></a>
+<span class="sourceLineNo">556</span> scheduler.start();<a name="line.556"></a>
+<span class="sourceLineNo">557</span><a name="line.557"></a>
+<span class="sourceLineNo">558</span> // TODO: Split in two steps.<a name="line.558"></a>
+<span class="sourceLineNo">559</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.559"></a>
+<span class="sourceLineNo">560</span> // The first one will make sure that we have the latest id,<a name="line.560"></a>
+<span class="sourceLineNo">561</span> // so we can start the threads and accept new procedures.<a name="line.561"></a>
+<span class="sourceLineNo">562</span> // The second step will do the actual load of old procedures.<a name="line.562"></a>
+<span class="sourceLineNo">563</span> st = EnvironmentEdgeManager.currentTime();<a name="line.563"></a>
+<span class="sourceLineNo">564</span> load(abortOnCorruption);<a name="line.564"></a>
+<span class="sourceLineNo">565</span> et = EnvironmentEdgeManager.currentTime();<a name="line.565"></a>
+<span class="sourceLineNo">566</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.566"></a>
+<span class="sourceLineNo">567</span> StringUtils.humanTimeDiff(et - st));<a name="line.567"></a>
+<span class="sourceLineNo">568</span><a name="line.568"></a>
+<span class="sourceLineNo">569</span> // Start the executors. Here we must have the lastProcId set.<a name="line.569"></a>
+<span class="sourceLineNo">570</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.570"></a>
+<span class="sourceLineNo">571</span> timeoutExecutor.start();<a name="line.571"></a>
+<span class="sourceLineNo">572</span> for (WorkerThread worker: workerThreads) {<a name="line.572"></a>
+<span class="sourceLineNo">573</span> worker.start();<a name="line.573"></a>
+<span class="sourceLineNo">574</span> }<a name="line.574"></a>
+<span class="sourceLineNo">575</span><a name="line.575"></a>
+<span class="sourceLineNo">576</span> // Internal chores<a name="line.576"></a>
+<span class="sourceLineNo">577</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.577"></a>
+<span class="sourceLineNo">578</span><a name="line.578"></a>
+<span class="sourceLineNo">579</span> // Add completed cleaner chore<a name="line.579"></a>
+<span class="sourceLineNo">580</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.580"></a>
+<span class="sourceLineNo">581</span> }<a name="line.581"></a>
+<span class="sourceLineNo">582</span><a name="line.582"></a>
+<span class="sourceLineNo">583</span> public void stop() {<a name="line.583"></a>
+<span class="sourceLineNo">584</span> if (!running.getAndSet(false)) {<a name="line.584"></a>
+<span class="sourceLineNo">585</span> return;<a name="line.585"></a>
+<span class="sourceLineNo">586</span> }<a name="line.586"></a>
+<span class="sourceLineNo">587</span><a name="line.587"></a>
+<span class="sourceLineNo">588</span> LOG.info("Stopping");<a name="line.588"></a>
+<span class="sourceLineNo">589</span> scheduler.stop();<a name="line.589"></a>
+<span class="sourceLineNo">590</span> timeoutExecutor.sendStopSignal();<a name="line.590"></a>
+<span class="sourceLineNo">591</span> }<a name="line.591"></a>
+<span class="sourceLineNo">592</span><a name="line.592"></a>
+<span class="sourceLineNo">593</span> @VisibleForTesting<a name="line.593"></a>
+<span class="sourceLineNo">594</span> public void join() {<a name="line.594"></a>
+<span class="sourceLineNo">595</span> assert !isRunning() : "expected not running";<a name="line.595"></a>
<span class="sourceLineNo">596</span><a name="line.596"></a>
-<span class="sourceLineNo">597</span> // reset the in-memory state for testing<a name="line.597"></a>
-<span class="sourceLineNo">598</span> completed.clear();<a name="line.598"></a>
-<span class="sourceLineNo">599</span> rollbackStack.clear();<a name="line.599"></a>
-<span class="sourceLineNo">600</span> procedures.clear();<a name="line.600"></a>
-<span class="sourceLineNo">601</span> nonceKeysToProcIdsMap.clear();<a name="line.601"></a>
-<span class="sourceLineNo">602</span> scheduler.clear();<a name="line.602"></a>
-<span class="sourceLineNo">603</span> lastProcId.set(-1);<a name="line.603"></a>
-<span class="sourceLineNo">604</span> }<a name="line.604"></a>
-<span class="sourceLineNo">605</span><a name="line.605"></a>
-<span class="sourceLineNo">606</span> public void refreshConfiguration(final Configuration conf) {<a name="line.606"></a>
-<span class="sourceLineNo">607</span> this.conf = conf;<a name="line.607"></a>
-<span class="sourceLineNo">608</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.608"></a>
-<span class="sourceLineNo">609</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.609"></a>
-<span class="sourceLineNo">610</span> }<a name="line.610"></a>
-<span class="sourceLineNo">611</span><a name="line.611"></a>
-<span class="sourceLineNo">612</span> // ==========================================================================<a name="line.612"></a>
-<span class="sourceLineNo">613</span> // Accessors<a name="line.613"></a>
-<span class="sourceLineNo">614</span> // ==========================================================================<a name="line.614"></a>
-<span class="sourceLineNo">615</span> public boolean isRunning() {<a name="line.615"></a>
-<span class="sourceLineNo">616</span> return running.get();<a name="line.616"></a>
-<span class="sourceLineNo">617</span> }<a name="line.617"></a>
-<span class="sourceLineNo">618</span><a name="line.618"></a>
-<span class="sourceLineNo">619</span> /**<a name="line.619"></a>
-<span class="sourceLineNo">620</span> * @return the current number of worker threads.<a name="line.620"></a>
-<span class="sourceLineNo">621</span> */<a name="line.621"></a>
-<span class="sourceLineNo">622</span> public int getWorkerThreadCount() {<a name="line.622"></a>
-<span class="sourceLineNo">623</span> return workerThreads.size();<a name="line.623"></a>
-<span class="sourceLineNo">624</span> }<a name="line.624"></a>
-<span class="sourceLineNo">625</span><a name="line.625"></a>
-<span class="sourceLineNo">626</span> /**<a name="line.626"></a>
-<span class="sourceLineNo">627</span> * @return the core pool size settings.<a name="line.627"></a>
-<span class="sourceLineNo">628</span> */<a name="line.628"></a>
-<span class="sourceLineNo">629</span> public int getCorePoolSize() {<a name="line.629"></a>
-<span class="sourceLineNo">630</span> return corePoolSize;<a name="line.630"></a>
-<span class="sourceLineNo">631</span> }<a name="line.631"></a>
-<span class="sourceLineNo">632</span><a name="line.632"></a>
-<span class="sourceLineNo">633</span> public int getActiveExecutorCount() {<a name="line.633"></a>
-<span class="sourceLineNo">634</span> return activeExecutorCount.get();<a name="line.634"></a>
-<span class="sourceLineNo">635</span> }<a name="line.635"></a>
-<span class="sourceLineNo">636</span><a name="line.636"></a>
-<span class="sourceLineNo">637</span> public TEnvironment getEnvironment() {<a name="line.637"></a>
-<span class="sourceLineNo">638</span> return this.environment;<a name="line.638"></a>
-<span class="sourceLineNo">639</span> }<a name="line.639"></a>
-<span class="sourceLineNo">640</span><a name="line.640"></a>
-<span class="sourceLineNo">641</span> public ProcedureStore getStore() {<a name="line.641"></a>
-<span class="sourceLineNo">642</span> return this.store;<a name="line.642"></a>
+<span class="sourceLineNo">597</span> // stop the timeout executor<a name="line.597"></a>
+<span class="sourceLineNo">598</span> timeoutExecutor.awaitTermination();<a name="line.598"></a>
+<span class="sourceLineNo">599</span><a name="line.599"></a>
+<span class="sourceLineNo">600</span> // stop the worker threads<a name="line.600"></a>
+<span class="sourceLineNo">601</span> for (WorkerThread worker: workerThreads) {<a name="line.601"></a>
+<span class="sourceLineNo">602</span> worker.awaitTermination();<a name="line.602"></a>
+<span class="sourceLineNo">603</span> }<a name="line.603"></a>
+<span class="sourceLineNo">604</span><a name="line.604"></a>
+<span class="sourceLineNo">605</span> // Destroy the Thread Group for the executors<a name="line.605"></a>
+<span class="sourceLineNo">606</span> // TODO: Fix. #join is not place to destroy resources.<a name="line.606"></a>
+<span class="sourceLineNo">607</span> try {<a name="line.607"></a>
+<span class="sourceLineNo">608</span> threadGroup.destroy();<a name="line.608"></a>
+<span class="sourceLineNo">609</span> } catch (IllegalThreadStateException e) {<a name="line.609"></a>
+<span class="sourceLineNo">610</span> LOG.error("ThreadGroup {} contains running threads; {}: See STDOUT",<a name="line.610"></a>
+<span class="sourceLineNo">611</span> this.threadGroup, e.getMessage());<a name="line.611"></a>
+<span class="sourceLineNo">612</span> // This dumps list of threads on STDOUT.<a name="line.612"></a>
+<span class="sourceLineNo">613</span> this.threadGroup.list();<a name="line.613"></a>
+<span class="sourceLineNo">614</span> }<a name="line.614"></a>
+<span class="sourceLineNo">615</span><a name="line.615"></a>
+<span class="sourceLineNo">616</span> // reset the in-memory state for testing<a name="line.616"></a>
+<span class="sourceLineNo">617</span> completed.clear();<a name="line.617"></a>
+<span class="sourceLineNo">618</span> rollbackStack.clear();<a name="line.618"></a>
+<span class="sourceLineNo">619</span> procedures.clear();<a name="line.619"></a>
+<span class="sourceLineNo">620</span> nonceKeysToProcIdsMap.clear();<a name="line.620"></a>
+<span class="sourceLineNo">621</span> scheduler.clear();<a name="line.621"></a>
+<span class="sourceLineNo">622</span> lastProcId.set(-1);<a name="line.622"></a>
+<span class="sourceLineNo">623</span> }<a name="line.623"></a>
+<span class="sourceLineNo">624</span><a name="line.624"></a>
+<span class="sourceLineNo">625</span> public void refreshConfiguration(final Configuration conf) {<a name="line.625"></a>
+<span class="sourceLineNo">626</span> this.conf = conf;<a name="line.626"></a>
+<span class="sourceLineNo">627</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.627"></a>
+<span class="sourceLineNo">628</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.628"></a>
+<span class="sourceLineNo">629</span> }<a name="line.629"></a>
+<span class="sourceLineNo">630</span><a name="line.630"></a>
+<span class="sourceLineNo">631</span> // ==========================================================================<a name="line.631"></a>
+<span class="sourceLineNo">632</span> // Accessors<a name="line.632"></a>
+<span class="sourceLineNo">633</span> // ==========================================================================<a name="line.633"></a>
+<span class="sourceLineNo">634</span> public boolean isRunning() {<a name="line.634"></a>
+<span class="sourceLineNo">635</span> return running.get();<a name="line.635"></a>
+<span class="sourceLineNo">636</span> }<a name="line.636"></a>
+<span class="sourceLineNo">637</span><a name="line.637"></a>
+<span class="sourceLineNo">638</span> /**<a name="line.638"></a>
+<span class="sourceLineNo">639</span> * @return the current number of worker threads.<a name="line.639"></a>
+<span class="sourceLineNo">640</span> */<a name="line.640"></a>
+<span class="sourceLineNo">641</span> public int getWorkerThreadCount() {<a name="line.641"></a>
+<span class="sourceLineNo">642</span> return workerThreads.size();<a name="line.642"></a>
<span class="sourceLineNo">643</span> }<a name="line.643"></a>
<span class="sourceLineNo">644</span><a name="line.644"></a>
-<span class="sourceLineNo">645</span> ProcedureScheduler getScheduler() {<a name="line.645"></a>
-<span class="sourceLineNo">646</span> return scheduler;<a name="line.646"></a>
-<span class="sourceLineNo">647</span> }<a name="line.647"></a>
-<span class="sourceLineNo">648</span><a name="line.648"></a>
-<span class="sourceLineNo">649</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.649"></a>
-<span class="sourceLineNo">650</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.650"></a>
-<span class="sourceLineNo">651</span> this.scheduler.signalAll();<a name="line.651"></a>
-<span class="sourceLineNo">652</span> }<a name="line.652"></a>
-<span class="sourceLineNo">653</span><a name="line.653"></a>
-<span class="sourceLineNo">654</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.654"></a>
-<span class="sourceLineNo">655</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.655"></a>
-<span class="sourceLineNo">656</span> }<a name="line.656"></a>
-<span class="sourceLineNo">657</span><a name="line.657"></a>
-<span class="sourceLineNo">658</span> // ==========================================================================<a name="line.658"></a>
-<span class="sourceLineNo">659</span> // Submit/Remove Chores<a name="line.659"></a>
-<span class="sourceLineNo">660</span> // ==========================================================================<a name="line.660"></a>
-<span class="sourceLineNo">661</span><a name="line.661"></a>
-<span class="sourceLineNo">662</span> /**<a name="line.662"></a>
-<span class="sourceLineNo">663</span> * Add a chore procedure to the executor<a name="line.663"></a>
-<span class="sourceLineNo">664</span> * @param chore the chore to add<a name="line.664"></a>
-<span class="sourceLineNo">665</span> */<a name="line.665"></a>
-<span class="sourceLineNo">666</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.666"></a>
-<span class="sourceLineNo">667</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.667"></a>
-<span class="sourceLineNo">668</span> timeoutExecutor.add(chore);<a name="line.668"></a>
-<span class="sourceLineNo">669</span> }<a name="line.669"></a>
-<span class="sourceLineNo">670</span><a name="line.670"></a>
-<span class="sourceLineNo">671</span> /**<a name="line.671"></a>
-<span class="sourceLineNo">672</span> * Remove a chore procedure from the executor<a name="line.672"></a>
-<span class="sourceLineNo">673</span> * @param chore the chore to remove<a name="line.673"></a>
-<span class="sourceLineNo">674</span> * @return whether the chore is removed, or it will be removed later<a name="line.674"></a>
-<span class="sourceLineNo">675</span> */<a name="line.675"></a>
-<span class="sourceLineNo">676</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.676"></a>
-<span class="sourceLineNo">677</span> chore.setState(ProcedureState.SUCCESS);<a name="line.677"></a>
-<span class="sourceLineNo">678</span> return timeoutExecutor.remove(chore);<a name="line.678"></a>
-<span class="sourceLineNo">679</span> }<a name="line.679"></a>
+<span class="sourceLineNo">645</span> /**<a name="line.645"></a>
+<span class="sourceLineNo">646</span> * @return the core pool size settings.<a name="line.646"></a>
+<span class="sourceLineNo">647</span> */<a name="line.647"></a>
+<span class="sourceLineNo">648</span> public int getCorePoolSize() {<a name="line.648"></a>
+<span class="sourceLineNo">649</span> return corePoolSize;<a name="line.649"></a>
+<span class="sourceLineNo">650</span> }<a name="line.650"></a>
+<span class="sourceLineNo">651</span><a name="line.651"></a>
+<span class="sourceLineNo">652</span> public int getActiveExecutorCount() {<a name="line.652"></a>
+<span class="sourceLineNo">653</span> return activeExecutorCount.get();<a name="line.653"></a>
+<span class="sourceLineNo">654</span> }<a name="line.654"></a>
+<span class="sourceLineNo">655</span><a name="line.655"></a>
+<span class="sourceLineNo">656</span> public TEnvironment getEnvironment() {<a name="line.656"></a>
+<span class="sourceLineNo">657</span> return this.environment;<a name="line.657"></a>
+<span class="sourceLineNo">658</span> }<a name="line.658"></a>
+<span class="sourceLineNo">659</span><a name="line.659"></a>
+<span class="sourceLineNo">660</span> public ProcedureStore getStore() {<a name="line.660"></a>
+<span class="sourceLineNo">661</span> return this.store;<a name="line.661"></a>
+<span class="sourceLineNo">662</span> }<a name="line.662"></a>
+<span class="sourceLineNo">663</span><a name="line.663"></a>
+<span class="sourceLineNo">664</span> ProcedureScheduler getScheduler() {<a name="line.664"></a>
+<span class="sourceLineNo">665</span> return scheduler;<a name="line.665"></a>
+<span class="sourceLineNo">666</span> }<a name="line.666"></a>
+<span class="sourceLineNo">667</span><a name="line.667"></a>
+<span class="sourceLineNo">668</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.668"></a>
+<span class="sourceLineNo">669</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.669"></a>
+<span class="sourceLineNo">670</span> this.scheduler.signalAll();<a name="line.670"></a>
+<span class="sourceLineNo">671</span> }<a name="line.671"></a>
+<span class="sourceLineNo">672</span><a name="line.672"></a>
+<span class="sourceLineNo">673</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.673"></a>
+<span class="sourceLineNo">674</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.674"></a>
+<span class="sourceLineNo">675</span> }<a name="line.675"></a>
+<span class="sourceLineNo">676</span><a name="line.676"></a>
+<span class="sourceLineNo">677</span> // ==========================================================================<a name="line.677"></a>
+<span class="sourceLineNo">678</span> // Submit/Remove Chores<a name="line.678"></a>
+<span class="sourceLineNo">679</span> // ==========================================================================<a name="line.679"></a>
<span class="sourceLineNo">680</span><a name="line.680"></a>
-<span class="sourceLineNo">681</span> // ==========================================================================<a name="line.681"></a>
-<span class="sourceLineNo">682</span> // Nonce Procedure helpers<a name="line.682"></a>
-<span class="sourceLineNo">683</span> // ==========================================================================<a name="line.683"></a>
-<span class="sourceLineNo">684</span> /**<a name="line.684"></a>
-<span class="sourceLineNo">685</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.685"></a>
-<span class="sourceLineNo">686</span> * @param nonceGroup<a name="line.686"></a>
-<span class="sourceLineNo">687</span> * @param nonce<a name="line.687"></a>
-<span class="sourceLineNo">688</span> * @return the generated NonceKey<a name="line.688"></a>
-<span class="sourceLineNo">689</span> */<a name="line.689"></a>
-<span class="sourceLineNo">690</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.690"></a>
-<span class="sourceLineNo">691</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.691"></a>
-<span class="sourceLineNo">692</span> }<a name="line.692"></a>
-<span class="sourceLineNo">693</span><a name="line.693"></a>
-<span class="sourceLineNo">694</span> /**<a name="line.694"></a>
-<span class="sourceLineNo">695</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.695"></a>
-<span class="sourceLineNo">696</span> * A procId will be reserved and on submitProcedure(),<a name="line.696"></a>
-<span class="sourceLineNo">697</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.697"></a>
-<span class="sourceLineNo">698</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.698"></a>
-<span class="sourceLineNo">699</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.699"></a>
-<span class="sourceLineNo">700</span> * and submit the procedure.<a name="line.700"></a>
-<span class="sourceLineNo">701</span> *<a name="line.701"></a>
-<span class="sourceLineNo">702</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.702"></a>
-<span class="sourceLineNo">703</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.703"></a>
-<span class="sourceLineNo">704</span> */<a name="line.704"></a>
-<span class="sourceLineNo">705</span> public long registerNonce(final NonceKey nonceKey) {<a name="line.705"></a>
-<span class="sourceLineNo">706</span> if (nonceKey == null) return -1;<a name="line.706"></a>
-<span class="sourceLineNo">707</span><a name="line.707"></a>
-<span class="sourceLineNo">708</span> // check if we have already a Reserved ID for the nonce<a name="line.708"></a>
-<span class="sourceLineNo">709</span> Long oldProcId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.709"></a>
-<span class="sourceLineNo">710</span> if (oldProcId == null) {<a name="line.710"></a>
-<span class="sourceLineNo">711</span> // reserve a new Procedure ID, this will be associated with the nonce<a name="line.711"></a>
-<span class="sourceLineNo">712</span> // and the procedure submitted with the specified nonce will use this ID.<a name="line.712"></a>
-<span class="sourceLineNo">713</span> final long newProcId = nextProcId();<a name="line.713"></a>
-<span class="sourceLineNo">714</span> oldProcId = nonceKeysToProcIdsMap.putIfAbsent(nonceKey, newProcId);<a name="line.714"></a>
-<span class="sourceLineNo">715</span> if (oldProcId == null) return -1;<a name="line.715"></a>
-<span class="sourceLineNo">716</span> }<a name="line.716"></a>
-<span class="sourceLineNo">717</span><a name="line.717"></a>
-<span class="sourceLineNo">718</span> // we found a registered nonce, but the procedure may not have been submitted yet.<a name="line.718"></a>
-<span class="sourceLineNo">719</span> // since the client expect the procedure to be submitted, spin here until it is.<a name="line.719"></a>
-<span class="sourceLineNo">720</span> final boolean traceEnabled = LOG.isTraceEnabled();<a name="line.720"></a>
-<span class="sourceLineNo">721</span> while (isRunning() &&<a name="line.721"></a>
-<span class="sourceLineNo">722</span> !(procedures.containsKey(oldProcId) || completed.containsKey(oldProcId)) &&<a name="line.722"></a>
-<span class="sourceLineNo">723</span> nonceKeysToProcIdsMap.containsKey(nonceKey)) {<a name="line.723"></a>
-<span class="sourceLineNo">724</span> if (traceEnabled) {<a name="line.724"></a>
-<span class="sourceLineNo">725</span> LOG.trace("Waiting for pid=" + oldProcId.longValue() + " to be submitted");<a name="line.725"></a>
-<span class="sourceLineNo">726</span> }<a name="line.726"></a>
-<span class="sourceLineNo">727</span> Threads.sleep(100);<a name="line.727"></a>
-<span class="sourceLineNo">728</span> }<a name="line.728"></a>
-<span class="sourceLineNo">729</span> return oldProcId.longValue();<a name="line.729"></a>
-<span class="sourceLineNo">730</span> }<a name="line.730"></a>
-<span class="sourceLineNo">731</span><a name="line.731"></a>
-<span class="sourceLineNo">732</span> /**<a name="line.732"></a>
-<span class="sourceLineNo">733</span> * Remove the NonceKey if the procedure was not submitted to the executor.<a name="line.733"></a>
-<span class="sourceLineNo">734</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.734"></a>
-<span class="sourceLineNo">735</span> */<a name="line.735"></a>
-<span class="sourceLineNo">736</span> public void unregisterNonceIfProcedureWasNotSubmitted(final NonceKey nonceKey) {<a name="line.736"></a>
-<span class="sourceLineNo">737</span> if (nonceKey == null) return;<a name="line.737"></a>
-<span class="sourceLineNo">738</span><a name="line.738"></a>
-<span class="sourceLineNo">739</span> final Long procId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.739"></a>
-<span class="sourceLineNo">740</span> if (procId == null) return;<a name="line.740"></a>
-<span class="sourceLineNo">741</span><a name="line.741"></a>
-<span class="sourceLineNo">742</span> // if the procedure was not submitted, remove the nonce<a name="line.742"></a>
-<span class="sourceLineNo">743</span> if (!(procedures.containsKey(procId) || completed.containsKey(procId))) {<a name="line.743"></a>
-<span class="sourceLineNo">744</span> nonceKeysToProcIdsMap.remove(nonceKey);<a name="line.744"></a>
-<span class="sourceLineNo">745</span> }<a name="line.745"></a>
-<span class="sourceLineNo">746</span> }<a name="line.746"></a>
-<span class="sourceLineNo">747</span><a name="line.747"></a>
-<span class="sourceLineNo">748</span> public static class FailedProcedure<TEnvironment> extends Procedure<TEnvironment> {<a name="line.748"></a>
-<span class="sourceLineNo">749</span> private String procName;<a name="line.749"></a>
+<span class="sourceLineNo">681</span> /**<a name="line.681"></a>
+<span class="sourceLineNo">682</span> * Add a chore procedure to the executor<a name="line.682"></a>
+<span class="sourceLineNo">683</span> * @param chore the chore to add<a name="line.683"></a>
+<span class="sourceLineNo">684</span> */<a name="line.684"></a>
+<span class="sourceLineNo">685</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.685"></a>
+<span class="sourceLineNo">686</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.686"></a>
+<span class="sourceLineNo">687</span> timeoutExecutor.add(chore);<a name="line.687"></a>
+<span class="sourceLineNo">688</span> }<a name="line.688"></a>
+<span class="sourceLineNo">689</span><a name="line.689"></a>
+<span class="sourceLineNo">690</span> /**<a name="line.690"></a>
+<span class="sourceLineNo">691</span> * Remove a chore procedure from the executor<a name="line.691"></a>
+<span class="sourceLineNo">692</span> * @param chore the chore to remove<a name="line.692"></a>
+<span class="sourceLineNo">693</span> * @return whether the chore is removed, or it will be removed later<a name="line.693"></a>
+<span class="sourceLineNo">694</span> */<a name="line.694"></a>
+<span class="sourceLineNo">695</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.695"></a>
+<span class="sourceLineNo">696</span> chore.setState(ProcedureState.SUCCESS);<a name="line.696"></a>
+<span class="sourceLineNo">697</span> return timeoutExecutor.remove(chore);<a name="line.697"></a>
+<span class="sourceLineNo">698</span> }<a name="line.698"></a>
+<span class="sourceLineNo">699</span><a name="line.699"></a>
+<span class="sourceLineNo">700</span> // ==========================================================================<a name="line.700"></a>
+<span class="sourceLineNo">701</span> // Nonce Procedure helpers<a name="line.701"></a>
+<span class="sourceLineNo">702</span> // ==========================================================================<a name="line.702"></a>
+<span class="sourceLineNo">703</span> /**<a name="line.703"></a>
+<span class="sourceLineNo">704</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.704"></a>
+<span class="sourceLineNo">705</span> * @param nonceGroup<a name="line.705"></a>
+<span class="sourceLineNo">706</span> * @param nonce<a name="line.706"></a>
+<span class="sourceLineNo">707</span> * @return the generated NonceKey<a name="line.707"></a>
+<span class="sourceLineNo">708</span> */<a name="line.708"></a>
+<span class="sourceLineNo">709</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.709"></a>
+<span class="sourceLineNo">710</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.710"></a>
+<span class="sourceLineNo">711</span> }<a name="line.711"></a>
+<span class="sourceLineNo">712</span><a name="line.712"></a>
+<span class="sourceLineNo">713</span> /**<a name="line.713"></a>
+<span class="sourceLineNo">714</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.714"></a>
+<span class="sourceLineNo">715</span> * A procId will be reserved and on submitProcedure(),<a name="line.715"></a>
+<span class="sourceLineNo">716</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.716"></a>
+<span class="sourceLineNo">717</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.717"></a>
+<span class="sourceLineNo">718</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.718"></a>
+<span class="sourceLineNo">719</span> * and submit the procedure.<a name="line.719"></a>
+<span class="sourceLineNo">720</span> *<a name="line.720"></a>
+<span class="sourceLineNo">721</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.721"></a>
+<span class="sourceLineNo">722</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a n
<TRUNCATED>
[15/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html
----------------------------------------------------------------------
diff --git a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html
index cad3f57..a650fb6 100644
--- a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html
+++ b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html
@@ -122,7 +122,7 @@ var activeTableTab = "activeTableTab";
</dl>
<hr>
<br>
-<pre>public static class <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.748">ProcedureExecutor.FailedProcedure</a><TEnvironment>
+<pre>public static class <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.767">ProcedureExecutor.FailedProcedure</a><TEnvironment>
extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a><TEnvironment></pre>
</li>
</ul>
@@ -279,7 +279,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.htm
<ul class="blockListLast">
<li class="blockList">
<h4>procName</h4>
-<pre>private <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.749">procName</a></pre>
+<pre>private <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.768">procName</a></pre>
</li>
</ul>
</li>
@@ -296,7 +296,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.htm
<ul class="blockList">
<li class="blockList">
<h4>FailedProcedure</h4>
-<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.751">FailedProcedure</a>()</pre>
+<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.770">FailedProcedure</a>()</pre>
</li>
</ul>
<a name="FailedProcedure-long-java.lang.String-org.apache.hadoop.hbase.security.User-org.apache.hadoop.hbase.util.NonceKey-java.io.IOException-">
@@ -305,7 +305,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.htm
<ul class="blockListLast">
<li class="blockList">
<h4>FailedProcedure</h4>
-<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.754">FailedProcedure</a>(long procId,
+<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.773">FailedProcedure</a>(long procId,
<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> procName,
<a href="../../../../../org/apache/hadoop/hbase/security/User.html" title="class in org.apache.hadoop.hbase.security">User</a> owner,
<a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> nonceKey,
@@ -326,7 +326,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.htm
<ul class="blockList">
<li class="blockList">
<h4>getProcName</h4>
-<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.768">getProcName</a>()</pre>
+<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.787">getProcName</a>()</pre>
<dl>
<dt><span class="overrideSpecifyLabel">Overrides:</span></dt>
<dd><code><a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html#getProcName--">getProcName</a></code> in class <code><a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a><<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html" title="type parameter in ProcedureExecutor.FailedProcedure">TEnvironment</a>></code></dd>
@@ -341,7 +341,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.htm
<ul class="blockList">
<li class="blockList">
<h4>execute</h4>
-<pre>protected <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a><<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html" title="type parameter in ProcedureExecutor.FailedProcedure">TEnvironment</a>>[] <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.773">execute</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html" title="type parameter in ProcedureExecutor.FailedProcedure">TEnvironment</a> env)
+<pre>protected <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a><<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html" title="type parameter in ProcedureExecutor.FailedProcedure">TEnvironment</a>>[] <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.792">execute</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html" title="type parameter in ProcedureExecutor.FailedProcedure">TEnvironment</a> env)
throws <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureYieldException.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureYieldException</a>,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureSuspendedException.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureSuspendedException</a>,
<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/InterruptedException.html?is-external=true" title="class or interface in java.lang">InterruptedException</a></pre>
@@ -373,7 +373,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.htm
<ul class="blockList">
<li class="blockList">
<h4>rollback</h4>
-<pre>protected void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.780">rollback</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html" title="type parameter in ProcedureExecutor.FailedProcedure">TEnvironment</a> env)
+<pre>protected void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.799">rollback</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html" title="type parameter in ProcedureExecutor.FailedProcedure">TEnvironment</a> env)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a>,
<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/InterruptedException.html?is-external=true" title="class or interface in java.lang">InterruptedException</a></pre>
<div class="block"><span class="descfrmTypeLabel">Description copied from class: <code><a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html#rollback-TEnvironment-">Procedure</a></code></span></div>
@@ -402,7 +402,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.htm
<ul class="blockList">
<li class="blockList">
<h4>abort</h4>
-<pre>protected boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.786">abort</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html" title="type parameter in ProcedureExecutor.FailedProcedure">TEnvironment</a> env)</pre>
+<pre>protected boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.805">abort</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html" title="type parameter in ProcedureExecutor.FailedProcedure">TEnvironment</a> env)</pre>
<div class="block"><span class="descfrmTypeLabel">Description copied from class: <code><a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html#abort-TEnvironment-">Procedure</a></code></span></div>
<div class="block">The abort() call is asynchronous and each procedure must decide how to deal
with it, if they want to be abortable. The simplest implementation
@@ -425,7 +425,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.htm
<ul class="blockList">
<li class="blockList">
<h4>serializeStateData</h4>
-<pre>protected void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.791">serializeStateData</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureStateSerializer.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureStateSerializer</a> serializer)
+<pre>protected void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.810">serializeStateData</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureStateSerializer.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureStateSerializer</a> serializer)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a></pre>
<div class="block"><span class="descfrmTypeLabel">Description copied from class: <code><a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html#serializeStateData-org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer-">Procedure</a></code></span></div>
<div class="block">The user-level code of the procedure may have some state to
@@ -447,7 +447,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.htm
<ul class="blockListLast">
<li class="blockList">
<h4>deserializeStateData</h4>
-<pre>protected void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.796">deserializeStateData</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureStateSerializer.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureStateSerializer</a> serializer)
+<pre>protected void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.FailedProcedure.html#line.815">deserializeStateData</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureStateSerializer.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureStateSerializer</a> serializer)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a></pre>
<div class="block"><span class="descfrmTypeLabel">Description copied from class: <code><a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html#deserializeStateData-org.apache.hadoop.hbase.procedure2.ProcedureStateSerializer-">Procedure</a></code></span></div>
<div class="block">Called on store load to allow the user to decode the previously serialized
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html
----------------------------------------------------------------------
diff --git a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html
index b432f6e..986b859 100644
--- a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html
+++ b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html
@@ -132,7 +132,7 @@ var activeTableTab = "activeTableTab";
</dl>
<hr>
<br>
-<pre>private final class <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1784">ProcedureExecutor.KeepAliveWorkerThread</a>
+<pre>private final class <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1803">ProcedureExecutor.KeepAliveWorkerThread</a>
extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.WorkerThread</a></pre>
</li>
</ul>
@@ -252,7 +252,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExec
<ul class="blockListLast">
<li class="blockList">
<h4>KeepAliveWorkerThread</h4>
-<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html#line.1786">KeepAliveWorkerThread</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadGroup.html?is-external=true" title="class or interface in java.lang">ThreadGroup</a> group)</pre>
+<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html#line.1805">KeepAliveWorkerThread</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadGroup.html?is-external=true" title="class or interface in java.lang">ThreadGroup</a> group)</pre>
</li>
</ul>
</li>
@@ -269,7 +269,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExec
<ul class="blockListLast">
<li class="blockList">
<h4>keepAlive</h4>
-<pre>protected boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html#line.1791">keepAlive</a>(long lastUpdate)</pre>
+<pre>protected boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html#line.1810">keepAlive</a>(long lastUpdate)</pre>
<dl>
<dt><span class="overrideSpecifyLabel">Overrides:</span></dt>
<dd><code><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#keepAlive-long-">keepAlive</a></code> in class <code><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.WorkerThread</a></code></dd>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html
----------------------------------------------------------------------
diff --git a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html
index 17427fd..48bf552 100644
--- a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html
+++ b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html
@@ -127,7 +127,7 @@ var activeTableTab = "activeTableTab";
</dl>
<hr>
<br>
-<pre>private final class <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1804">ProcedureExecutor.WorkerMonitor</a>
+<pre>private final class <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1823">ProcedureExecutor.WorkerMonitor</a>
extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.html" title="class in org.apache.hadoop.hbase.procedure2">InlineChore</a></pre>
</li>
</ul>
@@ -277,7 +277,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>WORKER_MONITOR_INTERVAL_CONF_KEY</h4>
-<pre>public static final <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1805">WORKER_MONITOR_INTERVAL_CONF_KEY</a></pre>
+<pre>public static final <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1824">WORKER_MONITOR_INTERVAL_CONF_KEY</a></pre>
<dl>
<dt><span class="seeLabel">See Also:</span></dt>
<dd><a href="../../../../../constant-values.html#org.apache.hadoop.hbase.procedure2.ProcedureExecutor.WorkerMonitor.WORKER_MONITOR_INTERVAL_CONF_KEY">Constant Field Values</a></dd>
@@ -290,7 +290,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>DEFAULT_WORKER_MONITOR_INTERVAL</h4>
-<pre>private static final int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1807">DEFAULT_WORKER_MONITOR_INTERVAL</a></pre>
+<pre>private static final int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1826">DEFAULT_WORKER_MONITOR_INTERVAL</a></pre>
<dl>
<dt><span class="seeLabel">See Also:</span></dt>
<dd><a href="../../../../../constant-values.html#org.apache.hadoop.hbase.procedure2.ProcedureExecutor.WorkerMonitor.DEFAULT_WORKER_MONITOR_INTERVAL">Constant Field Values</a></dd>
@@ -303,7 +303,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>WORKER_STUCK_THRESHOLD_CONF_KEY</h4>
-<pre>public static final <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1809">WORKER_STUCK_THRESHOLD_CONF_KEY</a></pre>
+<pre>public static final <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1828">WORKER_STUCK_THRESHOLD_CONF_KEY</a></pre>
<dl>
<dt><span class="seeLabel">See Also:</span></dt>
<dd><a href="../../../../../constant-values.html#org.apache.hadoop.hbase.procedure2.ProcedureExecutor.WorkerMonitor.WORKER_STUCK_THRESHOLD_CONF_KEY">Constant Field Values</a></dd>
@@ -316,7 +316,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>DEFAULT_WORKER_STUCK_THRESHOLD</h4>
-<pre>private static final int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1811">DEFAULT_WORKER_STUCK_THRESHOLD</a></pre>
+<pre>private static final int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1830">DEFAULT_WORKER_STUCK_THRESHOLD</a></pre>
<dl>
<dt><span class="seeLabel">See Also:</span></dt>
<dd><a href="../../../../../constant-values.html#org.apache.hadoop.hbase.procedure2.ProcedureExecutor.WorkerMonitor.DEFAULT_WORKER_STUCK_THRESHOLD">Constant Field Values</a></dd>
@@ -329,7 +329,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>WORKER_ADD_STUCK_PERCENTAGE_CONF_KEY</h4>
-<pre>public static final <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1813">WORKER_ADD_STUCK_PERCENTAGE_CONF_KEY</a></pre>
+<pre>public static final <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1832">WORKER_ADD_STUCK_PERCENTAGE_CONF_KEY</a></pre>
<dl>
<dt><span class="seeLabel">See Also:</span></dt>
<dd><a href="../../../../../constant-values.html#org.apache.hadoop.hbase.procedure2.ProcedureExecutor.WorkerMonitor.WORKER_ADD_STUCK_PERCENTAGE_CONF_KEY">Constant Field Values</a></dd>
@@ -342,7 +342,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>DEFAULT_WORKER_ADD_STUCK_PERCENTAGE</h4>
-<pre>private static final float <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1815">DEFAULT_WORKER_ADD_STUCK_PERCENTAGE</a></pre>
+<pre>private static final float <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1834">DEFAULT_WORKER_ADD_STUCK_PERCENTAGE</a></pre>
<dl>
<dt><span class="seeLabel">See Also:</span></dt>
<dd><a href="../../../../../constant-values.html#org.apache.hadoop.hbase.procedure2.ProcedureExecutor.WorkerMonitor.DEFAULT_WORKER_ADD_STUCK_PERCENTAGE">Constant Field Values</a></dd>
@@ -355,7 +355,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>addWorkerStuckPercentage</h4>
-<pre>private float <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1817">addWorkerStuckPercentage</a></pre>
+<pre>private float <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1836">addWorkerStuckPercentage</a></pre>
</li>
</ul>
<a name="timeoutInterval">
@@ -364,7 +364,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>timeoutInterval</h4>
-<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1818">timeoutInterval</a></pre>
+<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1837">timeoutInterval</a></pre>
</li>
</ul>
<a name="stuckThreshold">
@@ -373,7 +373,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockListLast">
<li class="blockList">
<h4>stuckThreshold</h4>
-<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1819">stuckThreshold</a></pre>
+<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1838">stuckThreshold</a></pre>
</li>
</ul>
</li>
@@ -390,7 +390,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockListLast">
<li class="blockList">
<h4>WorkerMonitor</h4>
-<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1821">WorkerMonitor</a>()</pre>
+<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1840">WorkerMonitor</a>()</pre>
</li>
</ul>
</li>
@@ -407,7 +407,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>run</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1826">run</a>()</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1845">run</a>()</pre>
</li>
</ul>
<a name="checkForStuckWorkers--">
@@ -416,7 +416,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>checkForStuckWorkers</h4>
-<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1834">checkForStuckWorkers</a>()</pre>
+<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1853">checkForStuckWorkers</a>()</pre>
</li>
</ul>
<a name="checkThreadCount-int-">
@@ -425,7 +425,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>checkThreadCount</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1850">checkThreadCount</a>(int stuckCount)</pre>
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1869">checkThreadCount</a>(int stuckCount)</pre>
</li>
</ul>
<a name="refreshConfig--">
@@ -434,7 +434,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockList">
<li class="blockList">
<h4>refreshConfig</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1869">refreshConfig</a>()</pre>
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1888">refreshConfig</a>()</pre>
</li>
</ul>
<a name="getTimeoutInterval--">
@@ -443,7 +443,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.h
<ul class="blockListLast">
<li class="blockList">
<h4>getTimeoutInterval</h4>
-<pre>public int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1879">getTimeoutInterval</a>()</pre>
+<pre>public int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html#line.1898">getTimeoutInterval</a>()</pre>
<dl>
<dt><span class="overrideSpecifyLabel">Specified by:</span></dt>
<dd><code><a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.html#getTimeoutInterval--">getTimeoutInterval</a></code> in class <code><a href="../../../../../org/apache/hadoop/hbase/procedure2/InlineChore.html" title="class in org.apache.hadoop.hbase.procedure2">InlineChore</a></code></dd>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html
----------------------------------------------------------------------
diff --git a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html
index 017962e..39523e3 100644
--- a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html
+++ b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html
@@ -131,7 +131,7 @@ var activeTableTab = "activeTableTab";
</dl>
<hr>
<br>
-<pre>private class <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1707">ProcedureExecutor.WorkerThread</a>
+<pre>private class <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1726">ProcedureExecutor.WorkerThread</a>
extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThread.html" title="class in org.apache.hadoop.hbase.procedure2">StoppableThread</a></pre>
</li>
</ul>
@@ -282,7 +282,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThre
<ul class="blockList">
<li class="blockList">
<h4>executionStartTime</h4>
-<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicLong.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicLong</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1708">executionStartTime</a></pre>
+<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicLong.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicLong</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1727">executionStartTime</a></pre>
</li>
</ul>
<a name="activeProcedure">
@@ -291,7 +291,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThre
<ul class="blockListLast">
<li class="blockList">
<h4>activeProcedure</h4>
-<pre>private volatile <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a><?> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1709">activeProcedure</a></pre>
+<pre>private volatile <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a><?> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1728">activeProcedure</a></pre>
</li>
</ul>
</li>
@@ -308,7 +308,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThre
<ul class="blockList">
<li class="blockList">
<h4>WorkerThread</h4>
-<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1711">WorkerThread</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadGroup.html?is-external=true" title="class or interface in java.lang">ThreadGroup</a> group)</pre>
+<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1730">WorkerThread</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadGroup.html?is-external=true" title="class or interface in java.lang">ThreadGroup</a> group)</pre>
</li>
</ul>
<a name="WorkerThread-java.lang.ThreadGroup-java.lang.String-">
@@ -317,7 +317,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThre
<ul class="blockListLast">
<li class="blockList">
<h4>WorkerThread</h4>
-<pre>protected <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1715">WorkerThread</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadGroup.html?is-external=true" title="class or interface in java.lang">ThreadGroup</a> group,
+<pre>protected <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1734">WorkerThread</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadGroup.html?is-external=true" title="class or interface in java.lang">ThreadGroup</a> group,
<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> prefix)</pre>
</li>
</ul>
@@ -335,7 +335,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThre
<ul class="blockList">
<li class="blockList">
<h4>sendStopSignal</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1721">sendStopSignal</a>()</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1740">sendStopSignal</a>()</pre>
<dl>
<dt><span class="overrideSpecifyLabel">Specified by:</span></dt>
<dd><code><a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThread.html#sendStopSignal--">sendStopSignal</a></code> in class <code><a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThread.html" title="class in org.apache.hadoop.hbase.procedure2">StoppableThread</a></code></dd>
@@ -348,7 +348,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThre
<ul class="blockList">
<li class="blockList">
<h4>run</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1726">run</a>()</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1745">run</a>()</pre>
<dl>
<dt><span class="overrideSpecifyLabel">Specified by:</span></dt>
<dd><code><a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Runnable.html?is-external=true#run--" title="class or interface in java.lang">run</a></code> in interface <code><a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Runnable.html?is-external=true" title="class or interface in java.lang">Runnable</a></code></dd>
@@ -363,7 +363,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThre
<ul class="blockList">
<li class="blockList">
<h4>toString</h4>
-<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1764">toString</a>()</pre>
+<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1783">toString</a>()</pre>
<dl>
<dt><span class="overrideSpecifyLabel">Overrides:</span></dt>
<dd><code><a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html?is-external=true#toString--" title="class or interface in java.lang">toString</a></code> in class <code><a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html?is-external=true" title="class or interface in java.lang">Thread</a></code></dd>
@@ -376,7 +376,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThre
<ul class="blockList">
<li class="blockList">
<h4>getCurrentRunTime</h4>
-<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1772">getCurrentRunTime</a>()</pre>
+<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1791">getCurrentRunTime</a>()</pre>
<dl>
<dt><span class="returnLabel">Returns:</span></dt>
<dd>the time since the current procedure is running</dd>
@@ -389,7 +389,7 @@ extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/StoppableThre
<ul class="blockListLast">
<li class="blockList">
<h4>keepAlive</h4>
-<pre>protected boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1777">keepAlive</a>(long lastUpdate)</pre>
+<pre>protected boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#line.1796">keepAlive</a>(long lastUpdate)</pre>
</li>
</ul>
</li>
[02/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html
index 97aa79c..cc787e3 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html
@@ -272,1623 +272,1642 @@
<span class="sourceLineNo">264</span> private final CopyOnWriteArrayList<ProcedureExecutorListener> listeners = new CopyOnWriteArrayList<>();<a name="line.264"></a>
<span class="sourceLineNo">265</span><a name="line.265"></a>
<span class="sourceLineNo">266</span> private Configuration conf;<a name="line.266"></a>
-<span class="sourceLineNo">267</span> private ThreadGroup threadGroup;<a name="line.267"></a>
-<span class="sourceLineNo">268</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.268"></a>
-<span class="sourceLineNo">269</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.269"></a>
-<span class="sourceLineNo">270</span> private int corePoolSize;<a name="line.270"></a>
-<span class="sourceLineNo">271</span> private int maxPoolSize;<a name="line.271"></a>
-<span class="sourceLineNo">272</span><a name="line.272"></a>
-<span class="sourceLineNo">273</span> private volatile long keepAliveTime;<a name="line.273"></a>
-<span class="sourceLineNo">274</span><a name="line.274"></a>
-<span class="sourceLineNo">275</span> /**<a name="line.275"></a>
-<span class="sourceLineNo">276</span> * Scheduler/Queue that contains runnable procedures.<a name="line.276"></a>
-<span class="sourceLineNo">277</span> */<a name="line.277"></a>
-<span class="sourceLineNo">278</span> private final ProcedureScheduler scheduler;<a name="line.278"></a>
-<span class="sourceLineNo">279</span><a name="line.279"></a>
-<span class="sourceLineNo">280</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.280"></a>
-<span class="sourceLineNo">281</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.281"></a>
-<span class="sourceLineNo">282</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.282"></a>
-<span class="sourceLineNo">283</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.283"></a>
-<span class="sourceLineNo">284</span> private final TEnvironment environment;<a name="line.284"></a>
-<span class="sourceLineNo">285</span> private final ProcedureStore store;<a name="line.285"></a>
-<span class="sourceLineNo">286</span><a name="line.286"></a>
-<span class="sourceLineNo">287</span> private final boolean checkOwnerSet;<a name="line.287"></a>
-<span class="sourceLineNo">288</span><a name="line.288"></a>
-<span class="sourceLineNo">289</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.289"></a>
-<span class="sourceLineNo">290</span> final ProcedureStore store) {<a name="line.290"></a>
-<span class="sourceLineNo">291</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.291"></a>
-<span class="sourceLineNo">292</span> }<a name="line.292"></a>
-<span class="sourceLineNo">293</span><a name="line.293"></a>
-<span class="sourceLineNo">294</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.294"></a>
-<span class="sourceLineNo">295</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.295"></a>
-<span class="sourceLineNo">296</span> this.environment = environment;<a name="line.296"></a>
-<span class="sourceLineNo">297</span> this.scheduler = scheduler;<a name="line.297"></a>
-<span class="sourceLineNo">298</span> this.store = store;<a name="line.298"></a>
-<span class="sourceLineNo">299</span> this.conf = conf;<a name="line.299"></a>
-<span class="sourceLineNo">300</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.300"></a>
-<span class="sourceLineNo">301</span> refreshConfiguration(conf);<a name="line.301"></a>
-<span class="sourceLineNo">302</span> }<a name="line.302"></a>
-<span class="sourceLineNo">303</span><a name="line.303"></a>
-<span class="sourceLineNo">304</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.304"></a>
-<span class="sourceLineNo">305</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.305"></a>
-<span class="sourceLineNo">306</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.306"></a>
-<span class="sourceLineNo">307</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.307"></a>
-<span class="sourceLineNo">308</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.308"></a>
-<span class="sourceLineNo">309</span><a name="line.309"></a>
-<span class="sourceLineNo">310</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.310"></a>
-<span class="sourceLineNo">311</span> @Override<a name="line.311"></a>
-<span class="sourceLineNo">312</span> public void setMaxProcId(long maxProcId) {<a name="line.312"></a>
-<span class="sourceLineNo">313</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.313"></a>
-<span class="sourceLineNo">314</span> lastProcId.set(maxProcId);<a name="line.314"></a>
-<span class="sourceLineNo">315</span> }<a name="line.315"></a>
-<span class="sourceLineNo">316</span><a name="line.316"></a>
-<span class="sourceLineNo">317</span> @Override<a name="line.317"></a>
-<span class="sourceLineNo">318</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.318"></a>
-<span class="sourceLineNo">319</span> loadProcedures(procIter, abortOnCorruption);<a name="line.319"></a>
-<span class="sourceLineNo">320</span> }<a name="line.320"></a>
-<span class="sourceLineNo">321</span><a name="line.321"></a>
-<span class="sourceLineNo">322</span> @Override<a name="line.322"></a>
-<span class="sourceLineNo">323</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.323"></a>
-<span class="sourceLineNo">324</span> int corruptedCount = 0;<a name="line.324"></a>
-<span class="sourceLineNo">325</span> while (procIter.hasNext()) {<a name="line.325"></a>
-<span class="sourceLineNo">326</span> Procedure<?> proc = procIter.next();<a name="line.326"></a>
-<span class="sourceLineNo">327</span> LOG.error("Corrupt " + proc);<a name="line.327"></a>
-<span class="sourceLineNo">328</span> corruptedCount++;<a name="line.328"></a>
-<span class="sourceLineNo">329</span> }<a name="line.329"></a>
-<span class="sourceLineNo">330</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.330"></a>
-<span class="sourceLineNo">331</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.331"></a>
-<span class="sourceLineNo">332</span> }<a name="line.332"></a>
-<span class="sourceLineNo">333</span> }<a name="line.333"></a>
-<span class="sourceLineNo">334</span> });<a name="line.334"></a>
-<span class="sourceLineNo">335</span> }<a name="line.335"></a>
-<span class="sourceLineNo">336</span><a name="line.336"></a>
-<span class="sourceLineNo">337</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.337"></a>
-<span class="sourceLineNo">338</span> final boolean abortOnCorruption) throws IOException {<a name="line.338"></a>
-<span class="sourceLineNo">339</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.339"></a>
-<span class="sourceLineNo">340</span><a name="line.340"></a>
-<span class="sourceLineNo">341</span> // 1. Build the rollback stack<a name="line.341"></a>
-<span class="sourceLineNo">342</span> int runnablesCount = 0;<a name="line.342"></a>
-<span class="sourceLineNo">343</span> while (procIter.hasNext()) {<a name="line.343"></a>
-<span class="sourceLineNo">344</span> boolean finished = procIter.isNextFinished();<a name="line.344"></a>
-<span class="sourceLineNo">345</span> Procedure proc = procIter.next();<a name="line.345"></a>
-<span class="sourceLineNo">346</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.346"></a>
-<span class="sourceLineNo">347</span> long procId = proc.getProcId();<a name="line.347"></a>
-<span class="sourceLineNo">348</span><a name="line.348"></a>
-<span class="sourceLineNo">349</span> if (finished) {<a name="line.349"></a>
-<span class="sourceLineNo">350</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.350"></a>
-<span class="sourceLineNo">351</span> if (debugEnabled) {<a name="line.351"></a>
-<span class="sourceLineNo">352</span> LOG.debug("Completed " + proc);<a name="line.352"></a>
-<span class="sourceLineNo">353</span> }<a name="line.353"></a>
-<span class="sourceLineNo">354</span> } else {<a name="line.354"></a>
-<span class="sourceLineNo">355</span> if (!proc.hasParent()) {<a name="line.355"></a>
-<span class="sourceLineNo">356</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.356"></a>
-<span class="sourceLineNo">357</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.357"></a>
-<span class="sourceLineNo">358</span> }<a name="line.358"></a>
+<span class="sourceLineNo">267</span><a name="line.267"></a>
+<span class="sourceLineNo">268</span> /**<a name="line.268"></a>
+<span class="sourceLineNo">269</span> * Created in the {@link #start(int, boolean)} method. Destroyed in {@link #join()} (FIX! Doing<a name="line.269"></a>
+<span class="sourceLineNo">270</span> * resource handling rather than observing in a #join is unexpected).<a name="line.270"></a>
+<span class="sourceLineNo">271</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.271"></a>
+<span class="sourceLineNo">272</span> * (Should be ok).<a name="line.272"></a>
+<span class="sourceLineNo">273</span> */<a name="line.273"></a>
+<span class="sourceLineNo">274</span> private ThreadGroup threadGroup;<a name="line.274"></a>
+<span class="sourceLineNo">275</span><a name="line.275"></a>
+<span class="sourceLineNo">276</span> /**<a name="line.276"></a>
+<span class="sourceLineNo">277</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.277"></a>
+<span class="sourceLineNo">278</span> * resource handling rather than observing in a #join is unexpected).<a name="line.278"></a>
+<span class="sourceLineNo">279</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.279"></a>
+<span class="sourceLineNo">280</span> * (Should be ok).<a name="line.280"></a>
+<span class="sourceLineNo">281</span> */<a name="line.281"></a>
+<span class="sourceLineNo">282</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.282"></a>
+<span class="sourceLineNo">283</span><a name="line.283"></a>
+<span class="sourceLineNo">284</span> /**<a name="line.284"></a>
+<span class="sourceLineNo">285</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.285"></a>
+<span class="sourceLineNo">286</span> * resource handling rather than observing in a #join is unexpected).<a name="line.286"></a>
+<span class="sourceLineNo">287</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.287"></a>
+<span class="sourceLineNo">288</span> * (Should be ok).<a name="line.288"></a>
+<span class="sourceLineNo">289</span> */<a name="line.289"></a>
+<span class="sourceLineNo">290</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.290"></a>
+<span class="sourceLineNo">291</span><a name="line.291"></a>
+<span class="sourceLineNo">292</span> private int corePoolSize;<a name="line.292"></a>
+<span class="sourceLineNo">293</span> private int maxPoolSize;<a name="line.293"></a>
+<span class="sourceLineNo">294</span><a name="line.294"></a>
+<span class="sourceLineNo">295</span> private volatile long keepAliveTime;<a name="line.295"></a>
+<span class="sourceLineNo">296</span><a name="line.296"></a>
+<span class="sourceLineNo">297</span> /**<a name="line.297"></a>
+<span class="sourceLineNo">298</span> * Scheduler/Queue that contains runnable procedures.<a name="line.298"></a>
+<span class="sourceLineNo">299</span> */<a name="line.299"></a>
+<span class="sourceLineNo">300</span> private final ProcedureScheduler scheduler;<a name="line.300"></a>
+<span class="sourceLineNo">301</span><a name="line.301"></a>
+<span class="sourceLineNo">302</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.302"></a>
+<span class="sourceLineNo">303</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.303"></a>
+<span class="sourceLineNo">304</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.304"></a>
+<span class="sourceLineNo">305</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.305"></a>
+<span class="sourceLineNo">306</span> private final TEnvironment environment;<a name="line.306"></a>
+<span class="sourceLineNo">307</span> private final ProcedureStore store;<a name="line.307"></a>
+<span class="sourceLineNo">308</span><a name="line.308"></a>
+<span class="sourceLineNo">309</span> private final boolean checkOwnerSet;<a name="line.309"></a>
+<span class="sourceLineNo">310</span><a name="line.310"></a>
+<span class="sourceLineNo">311</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.311"></a>
+<span class="sourceLineNo">312</span> final ProcedureStore store) {<a name="line.312"></a>
+<span class="sourceLineNo">313</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.313"></a>
+<span class="sourceLineNo">314</span> }<a name="line.314"></a>
+<span class="sourceLineNo">315</span><a name="line.315"></a>
+<span class="sourceLineNo">316</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.316"></a>
+<span class="sourceLineNo">317</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.317"></a>
+<span class="sourceLineNo">318</span> this.environment = environment;<a name="line.318"></a>
+<span class="sourceLineNo">319</span> this.scheduler = scheduler;<a name="line.319"></a>
+<span class="sourceLineNo">320</span> this.store = store;<a name="line.320"></a>
+<span class="sourceLineNo">321</span> this.conf = conf;<a name="line.321"></a>
+<span class="sourceLineNo">322</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.322"></a>
+<span class="sourceLineNo">323</span> refreshConfiguration(conf);<a name="line.323"></a>
+<span class="sourceLineNo">324</span><a name="line.324"></a>
+<span class="sourceLineNo">325</span> }<a name="line.325"></a>
+<span class="sourceLineNo">326</span><a name="line.326"></a>
+<span class="sourceLineNo">327</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.327"></a>
+<span class="sourceLineNo">328</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.328"></a>
+<span class="sourceLineNo">329</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.329"></a>
+<span class="sourceLineNo">330</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.330"></a>
+<span class="sourceLineNo">331</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.331"></a>
+<span class="sourceLineNo">332</span><a name="line.332"></a>
+<span class="sourceLineNo">333</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.333"></a>
+<span class="sourceLineNo">334</span> @Override<a name="line.334"></a>
+<span class="sourceLineNo">335</span> public void setMaxProcId(long maxProcId) {<a name="line.335"></a>
+<span class="sourceLineNo">336</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.336"></a>
+<span class="sourceLineNo">337</span> lastProcId.set(maxProcId);<a name="line.337"></a>
+<span class="sourceLineNo">338</span> }<a name="line.338"></a>
+<span class="sourceLineNo">339</span><a name="line.339"></a>
+<span class="sourceLineNo">340</span> @Override<a name="line.340"></a>
+<span class="sourceLineNo">341</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.341"></a>
+<span class="sourceLineNo">342</span> loadProcedures(procIter, abortOnCorruption);<a name="line.342"></a>
+<span class="sourceLineNo">343</span> }<a name="line.343"></a>
+<span class="sourceLineNo">344</span><a name="line.344"></a>
+<span class="sourceLineNo">345</span> @Override<a name="line.345"></a>
+<span class="sourceLineNo">346</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.346"></a>
+<span class="sourceLineNo">347</span> int corruptedCount = 0;<a name="line.347"></a>
+<span class="sourceLineNo">348</span> while (procIter.hasNext()) {<a name="line.348"></a>
+<span class="sourceLineNo">349</span> Procedure<?> proc = procIter.next();<a name="line.349"></a>
+<span class="sourceLineNo">350</span> LOG.error("Corrupt " + proc);<a name="line.350"></a>
+<span class="sourceLineNo">351</span> corruptedCount++;<a name="line.351"></a>
+<span class="sourceLineNo">352</span> }<a name="line.352"></a>
+<span class="sourceLineNo">353</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.353"></a>
+<span class="sourceLineNo">354</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.354"></a>
+<span class="sourceLineNo">355</span> }<a name="line.355"></a>
+<span class="sourceLineNo">356</span> }<a name="line.356"></a>
+<span class="sourceLineNo">357</span> });<a name="line.357"></a>
+<span class="sourceLineNo">358</span> }<a name="line.358"></a>
<span class="sourceLineNo">359</span><a name="line.359"></a>
-<span class="sourceLineNo">360</span> // add the procedure to the map<a name="line.360"></a>
-<span class="sourceLineNo">361</span> proc.beforeReplay(getEnvironment());<a name="line.361"></a>
-<span class="sourceLineNo">362</span> procedures.put(proc.getProcId(), proc);<a name="line.362"></a>
+<span class="sourceLineNo">360</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.360"></a>
+<span class="sourceLineNo">361</span> final boolean abortOnCorruption) throws IOException {<a name="line.361"></a>
+<span class="sourceLineNo">362</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.362"></a>
<span class="sourceLineNo">363</span><a name="line.363"></a>
-<span class="sourceLineNo">364</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.364"></a>
-<span class="sourceLineNo">365</span> runnablesCount++;<a name="line.365"></a>
-<span class="sourceLineNo">366</span> }<a name="line.366"></a>
-<span class="sourceLineNo">367</span> }<a name="line.367"></a>
-<span class="sourceLineNo">368</span><a name="line.368"></a>
-<span class="sourceLineNo">369</span> // add the nonce to the map<a name="line.369"></a>
-<span class="sourceLineNo">370</span> if (nonceKey != null) {<a name="line.370"></a>
-<span class="sourceLineNo">371</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.371"></a>
-<span class="sourceLineNo">372</span> }<a name="line.372"></a>
-<span class="sourceLineNo">373</span> }<a name="line.373"></a>
-<span class="sourceLineNo">374</span><a name="line.374"></a>
-<span class="sourceLineNo">375</span> // 2. Initialize the stacks<a name="line.375"></a>
-<span class="sourceLineNo">376</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.376"></a>
-<span class="sourceLineNo">377</span> HashSet<Procedure> waitingSet = null;<a name="line.377"></a>
-<span class="sourceLineNo">378</span> procIter.reset();<a name="line.378"></a>
-<span class="sourceLineNo">379</span> while (procIter.hasNext()) {<a name="line.379"></a>
-<span class="sourceLineNo">380</span> if (procIter.isNextFinished()) {<a name="line.380"></a>
-<span class="sourceLineNo">381</span> procIter.skipNext();<a name="line.381"></a>
-<span class="sourceLineNo">382</span> continue;<a name="line.382"></a>
-<span class="sourceLineNo">383</span> }<a name="line.383"></a>
-<span class="sourceLineNo">384</span><a name="line.384"></a>
-<span class="sourceLineNo">385</span> Procedure proc = procIter.next();<a name="line.385"></a>
-<span class="sourceLineNo">386</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.386"></a>
-<span class="sourceLineNo">387</span><a name="line.387"></a>
-<span class="sourceLineNo">388</span> if (debugEnabled) {<a name="line.388"></a>
-<span class="sourceLineNo">389</span> LOG.debug(String.format("Loading %s", proc));<a name="line.389"></a>
+<span class="sourceLineNo">364</span> // 1. Build the rollback stack<a name="line.364"></a>
+<span class="sourceLineNo">365</span> int runnablesCount = 0;<a name="line.365"></a>
+<span class="sourceLineNo">366</span> while (procIter.hasNext()) {<a name="line.366"></a>
+<span class="sourceLineNo">367</span> boolean finished = procIter.isNextFinished();<a name="line.367"></a>
+<span class="sourceLineNo">368</span> Procedure proc = procIter.next();<a name="line.368"></a>
+<span class="sourceLineNo">369</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.369"></a>
+<span class="sourceLineNo">370</span> long procId = proc.getProcId();<a name="line.370"></a>
+<span class="sourceLineNo">371</span><a name="line.371"></a>
+<span class="sourceLineNo">372</span> if (finished) {<a name="line.372"></a>
+<span class="sourceLineNo">373</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.373"></a>
+<span class="sourceLineNo">374</span> if (debugEnabled) {<a name="line.374"></a>
+<span class="sourceLineNo">375</span> LOG.debug("Completed " + proc);<a name="line.375"></a>
+<span class="sourceLineNo">376</span> }<a name="line.376"></a>
+<span class="sourceLineNo">377</span> } else {<a name="line.377"></a>
+<span class="sourceLineNo">378</span> if (!proc.hasParent()) {<a name="line.378"></a>
+<span class="sourceLineNo">379</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.379"></a>
+<span class="sourceLineNo">380</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.380"></a>
+<span class="sourceLineNo">381</span> }<a name="line.381"></a>
+<span class="sourceLineNo">382</span><a name="line.382"></a>
+<span class="sourceLineNo">383</span> // add the procedure to the map<a name="line.383"></a>
+<span class="sourceLineNo">384</span> proc.beforeReplay(getEnvironment());<a name="line.384"></a>
+<span class="sourceLineNo">385</span> procedures.put(proc.getProcId(), proc);<a name="line.385"></a>
+<span class="sourceLineNo">386</span><a name="line.386"></a>
+<span class="sourceLineNo">387</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.387"></a>
+<span class="sourceLineNo">388</span> runnablesCount++;<a name="line.388"></a>
+<span class="sourceLineNo">389</span> }<a name="line.389"></a>
<span class="sourceLineNo">390</span> }<a name="line.390"></a>
<span class="sourceLineNo">391</span><a name="line.391"></a>
-<span class="sourceLineNo">392</span> Long rootProcId = getRootProcedureId(proc);<a name="line.392"></a>
-<span class="sourceLineNo">393</span> if (rootProcId == null) {<a name="line.393"></a>
-<span class="sourceLineNo">394</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.394"></a>
-<span class="sourceLineNo">395</span> scheduler.addBack(proc);<a name="line.395"></a>
-<span class="sourceLineNo">396</span> continue;<a name="line.396"></a>
-<span class="sourceLineNo">397</span> }<a name="line.397"></a>
-<span class="sourceLineNo">398</span><a name="line.398"></a>
-<span class="sourceLineNo">399</span> if (proc.hasParent()) {<a name="line.399"></a>
-<span class="sourceLineNo">400</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.400"></a>
-<span class="sourceLineNo">401</span> // corrupted procedures are handled later at step 3<a name="line.401"></a>
-<span class="sourceLineNo">402</span> if (parent != null && !proc.isFinished()) {<a name="line.402"></a>
-<span class="sourceLineNo">403</span> parent.incChildrenLatch();<a name="line.403"></a>
-<span class="sourceLineNo">404</span> }<a name="line.404"></a>
-<span class="sourceLineNo">405</span> }<a name="line.405"></a>
-<span class="sourceLineNo">406</span><a name="line.406"></a>
-<span class="sourceLineNo">407</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.407"></a>
-<span class="sourceLineNo">408</span> procStack.loadStack(proc);<a name="line.408"></a>
-<span class="sourceLineNo">409</span><a name="line.409"></a>
-<span class="sourceLineNo">410</span> proc.setRootProcId(rootProcId);<a name="line.410"></a>
-<span class="sourceLineNo">411</span> switch (proc.getState()) {<a name="line.411"></a>
-<span class="sourceLineNo">412</span> case RUNNABLE:<a name="line.412"></a>
-<span class="sourceLineNo">413</span> runnableList.add(proc);<a name="line.413"></a>
-<span class="sourceLineNo">414</span> break;<a name="line.414"></a>
-<span class="sourceLineNo">415</span> case WAITING:<a name="line.415"></a>
-<span class="sourceLineNo">416</span> if (!proc.hasChildren()) {<a name="line.416"></a>
-<span class="sourceLineNo">417</span> runnableList.add(proc);<a name="line.417"></a>
-<span class="sourceLineNo">418</span> }<a name="line.418"></a>
-<span class="sourceLineNo">419</span> break;<a name="line.419"></a>
-<span class="sourceLineNo">420</span> case WAITING_TIMEOUT:<a name="line.420"></a>
-<span class="sourceLineNo">421</span> if (waitingSet == null) {<a name="line.421"></a>
-<span class="sourceLineNo">422</span> waitingSet = new HashSet<>();<a name="line.422"></a>
-<span class="sourceLineNo">423</span> }<a name="line.423"></a>
-<span class="sourceLineNo">424</span> waitingSet.add(proc);<a name="line.424"></a>
-<span class="sourceLineNo">425</span> break;<a name="line.425"></a>
-<span class="sourceLineNo">426</span> case FAILED:<a name="line.426"></a>
-<span class="sourceLineNo">427</span> // add the proc to the scheduler to perform the rollback<a name="line.427"></a>
-<span class="sourceLineNo">428</span> scheduler.addBack(proc);<a name="line.428"></a>
-<span class="sourceLineNo">429</span> break;<a name="line.429"></a>
-<span class="sourceLineNo">430</span> case ROLLEDBACK:<a name="line.430"></a>
-<span class="sourceLineNo">431</span> case INITIALIZING:<a name="line.431"></a>
-<span class="sourceLineNo">432</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.432"></a>
-<span class="sourceLineNo">433</span> LOG.error(msg);<a name="line.433"></a>
-<span class="sourceLineNo">434</span> throw new UnsupportedOperationException(msg);<a name="line.434"></a>
-<span class="sourceLineNo">435</span> default:<a name="line.435"></a>
-<span class="sourceLineNo">436</span> break;<a name="line.436"></a>
-<span class="sourceLineNo">437</span> }<a name="line.437"></a>
-<span class="sourceLineNo">438</span> }<a name="line.438"></a>
-<span class="sourceLineNo">439</span><a name="line.439"></a>
-<span class="sourceLineNo">440</span> // 3. Validate the stacks<a name="line.440"></a>
-<span class="sourceLineNo">441</span> int corruptedCount = 0;<a name="line.441"></a>
-<span class="sourceLineNo">442</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.442"></a>
-<span class="sourceLineNo">443</span> while (itStack.hasNext()) {<a name="line.443"></a>
-<span class="sourceLineNo">444</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.444"></a>
-<span class="sourceLineNo">445</span> RootProcedureState procStack = entry.getValue();<a name="line.445"></a>
-<span class="sourceLineNo">446</span> if (procStack.isValid()) continue;<a name="line.446"></a>
-<span class="sourceLineNo">447</span><a name="line.447"></a>
-<span class="sourceLineNo">448</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.448"></a>
-<span class="sourceLineNo">449</span> LOG.error("Corrupted " + proc);<a name="line.449"></a>
-<span class="sourceLineNo">450</span> procedures.remove(proc.getProcId());<a name="line.450"></a>
-<span class="sourceLineNo">451</span> runnableList.remove(proc);<a name="line.451"></a>
-<span class="sourceLineNo">452</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.452"></a>
-<span class="sourceLineNo">453</span> corruptedCount++;<a name="line.453"></a>
-<span class="sourceLineNo">454</span> }<a name="line.454"></a>
-<span class="sourceLineNo">455</span> itStack.remove();<a name="line.455"></a>
-<span class="sourceLineNo">456</span> }<a name="line.456"></a>
-<span class="sourceLineNo">457</span><a name="line.457"></a>
-<span class="sourceLineNo">458</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.458"></a>
-<span class="sourceLineNo">459</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.459"></a>
-<span class="sourceLineNo">460</span> }<a name="line.460"></a>
-<span class="sourceLineNo">461</span><a name="line.461"></a>
-<span class="sourceLineNo">462</span> // 4. Push the procedures to the timeout executor<a name="line.462"></a>
-<span class="sourceLineNo">463</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.463"></a>
-<span class="sourceLineNo">464</span> for (Procedure proc: waitingSet) {<a name="line.464"></a>
-<span class="sourceLineNo">465</span> proc.afterReplay(getEnvironment());<a name="line.465"></a>
-<span class="sourceLineNo">466</span> timeoutExecutor.add(proc);<a name="line.466"></a>
-<span class="sourceLineNo">467</span> }<a name="line.467"></a>
-<span class="sourceLineNo">468</span> }<a name="line.468"></a>
-<span class="sourceLineNo">469</span><a name="line.469"></a>
-<span class="sourceLineNo">470</span> // 5. Push the procedure to the scheduler<a name="line.470"></a>
-<span class="sourceLineNo">471</span> if (!runnableList.isEmpty()) {<a name="line.471"></a>
-<span class="sourceLineNo">472</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.472"></a>
-<span class="sourceLineNo">473</span> // some procedure may be started way before this stuff.<a name="line.473"></a>
-<span class="sourceLineNo">474</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.474"></a>
-<span class="sourceLineNo">475</span> Procedure proc = runnableList.get(i);<a name="line.475"></a>
-<span class="sourceLineNo">476</span> proc.afterReplay(getEnvironment());<a name="line.476"></a>
-<span class="sourceLineNo">477</span> if (!proc.hasParent()) {<a name="line.477"></a>
-<span class="sourceLineNo">478</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.478"></a>
-<span class="sourceLineNo">479</span> }<a name="line.479"></a>
-<span class="sourceLineNo">480</span> if (proc.wasExecuted()) {<a name="line.480"></a>
-<span class="sourceLineNo">481</span> scheduler.addFront(proc);<a name="line.481"></a>
-<span class="sourceLineNo">482</span> } else {<a name="line.482"></a>
-<span class="sourceLineNo">483</span> // if it was not in execution, it can wait.<a name="line.483"></a>
-<span class="sourceLineNo">484</span> scheduler.addBack(proc);<a name="line.484"></a>
-<span class="sourceLineNo">485</span> }<a name="line.485"></a>
-<span class="sourceLineNo">486</span> }<a name="line.486"></a>
-<span class="sourceLineNo">487</span> }<a name="line.487"></a>
-<span class="sourceLineNo">488</span> }<a name="line.488"></a>
-<span class="sourceLineNo">489</span><a name="line.489"></a>
-<span class="sourceLineNo">490</span> /**<a name="line.490"></a>
-<span class="sourceLineNo">491</span> * Start the procedure executor.<a name="line.491"></a>
-<span class="sourceLineNo">492</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.492"></a>
-<span class="sourceLineNo">493</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.493"></a>
-<span class="sourceLineNo">494</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.494"></a>
-<span class="sourceLineNo">495</span> *<a name="line.495"></a>
-<span class="sourceLineNo">496</span> * @param numThreads number of threads available for procedure execution.<a name="line.496"></a>
-<span class="sourceLineNo">497</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.497"></a>
-<span class="sourceLineNo">498</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.498"></a>
-<span class="sourceLineNo">499</span> */<a name="line.499"></a>
-<span class="sourceLineNo">500</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.500"></a>
-<span class="sourceLineNo">501</span> if (!running.compareAndSet(false, true)) {<a name="line.501"></a>
-<span class="sourceLineNo">502</span> LOG.warn("Already running");<a name="line.502"></a>
-<span class="sourceLineNo">503</span> return;<a name="line.503"></a>
-<span class="sourceLineNo">504</span> }<a name="line.504"></a>
-<span class="sourceLineNo">505</span><a name="line.505"></a>
-<span class="sourceLineNo">506</span> // We have numThreads executor + one timer thread used for timing out<a name="line.506"></a>
-<span class="sourceLineNo">507</span> // procedures and triggering periodic procedures.<a name="line.507"></a>
-<span class="sourceLineNo">508</span> this.corePoolSize = numThreads;<a name="line.508"></a>
-<span class="sourceLineNo">509</span> this.maxPoolSize = 10 * numThreads;<a name="line.509"></a>
-<span class="sourceLineNo">510</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.510"></a>
-<span class="sourceLineNo">511</span> corePoolSize, maxPoolSize);<a name="line.511"></a>
+<span class="sourceLineNo">392</span> // add the nonce to the map<a name="line.392"></a>
+<span class="sourceLineNo">393</span> if (nonceKey != null) {<a name="line.393"></a>
+<span class="sourceLineNo">394</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.394"></a>
+<span class="sourceLineNo">395</span> }<a name="line.395"></a>
+<span class="sourceLineNo">396</span> }<a name="line.396"></a>
+<span class="sourceLineNo">397</span><a name="line.397"></a>
+<span class="sourceLineNo">398</span> // 2. Initialize the stacks<a name="line.398"></a>
+<span class="sourceLineNo">399</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.399"></a>
+<span class="sourceLineNo">400</span> HashSet<Procedure> waitingSet = null;<a name="line.400"></a>
+<span class="sourceLineNo">401</span> procIter.reset();<a name="line.401"></a>
+<span class="sourceLineNo">402</span> while (procIter.hasNext()) {<a name="line.402"></a>
+<span class="sourceLineNo">403</span> if (procIter.isNextFinished()) {<a name="line.403"></a>
+<span class="sourceLineNo">404</span> procIter.skipNext();<a name="line.404"></a>
+<span class="sourceLineNo">405</span> continue;<a name="line.405"></a>
+<span class="sourceLineNo">406</span> }<a name="line.406"></a>
+<span class="sourceLineNo">407</span><a name="line.407"></a>
+<span class="sourceLineNo">408</span> Procedure proc = procIter.next();<a name="line.408"></a>
+<span class="sourceLineNo">409</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.409"></a>
+<span class="sourceLineNo">410</span><a name="line.410"></a>
+<span class="sourceLineNo">411</span> if (debugEnabled) {<a name="line.411"></a>
+<span class="sourceLineNo">412</span> LOG.debug(String.format("Loading %s", proc));<a name="line.412"></a>
+<span class="sourceLineNo">413</span> }<a name="line.413"></a>
+<span class="sourceLineNo">414</span><a name="line.414"></a>
+<span class="sourceLineNo">415</span> Long rootProcId = getRootProcedureId(proc);<a name="line.415"></a>
+<span class="sourceLineNo">416</span> if (rootProcId == null) {<a name="line.416"></a>
+<span class="sourceLineNo">417</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.417"></a>
+<span class="sourceLineNo">418</span> scheduler.addBack(proc);<a name="line.418"></a>
+<span class="sourceLineNo">419</span> continue;<a name="line.419"></a>
+<span class="sourceLineNo">420</span> }<a name="line.420"></a>
+<span class="sourceLineNo">421</span><a name="line.421"></a>
+<span class="sourceLineNo">422</span> if (proc.hasParent()) {<a name="line.422"></a>
+<span class="sourceLineNo">423</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.423"></a>
+<span class="sourceLineNo">424</span> // corrupted procedures are handled later at step 3<a name="line.424"></a>
+<span class="sourceLineNo">425</span> if (parent != null && !proc.isFinished()) {<a name="line.425"></a>
+<span class="sourceLineNo">426</span> parent.incChildrenLatch();<a name="line.426"></a>
+<span class="sourceLineNo">427</span> }<a name="line.427"></a>
+<span class="sourceLineNo">428</span> }<a name="line.428"></a>
+<span class="sourceLineNo">429</span><a name="line.429"></a>
+<span class="sourceLineNo">430</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.430"></a>
+<span class="sourceLineNo">431</span> procStack.loadStack(proc);<a name="line.431"></a>
+<span class="sourceLineNo">432</span><a name="line.432"></a>
+<span class="sourceLineNo">433</span> proc.setRootProcId(rootProcId);<a name="line.433"></a>
+<span class="sourceLineNo">434</span> switch (proc.getState()) {<a name="line.434"></a>
+<span class="sourceLineNo">435</span> case RUNNABLE:<a name="line.435"></a>
+<span class="sourceLineNo">436</span> runnableList.add(proc);<a name="line.436"></a>
+<span class="sourceLineNo">437</span> break;<a name="line.437"></a>
+<span class="sourceLineNo">438</span> case WAITING:<a name="line.438"></a>
+<span class="sourceLineNo">439</span> if (!proc.hasChildren()) {<a name="line.439"></a>
+<span class="sourceLineNo">440</span> runnableList.add(proc);<a name="line.440"></a>
+<span class="sourceLineNo">441</span> }<a name="line.441"></a>
+<span class="sourceLineNo">442</span> break;<a name="line.442"></a>
+<span class="sourceLineNo">443</span> case WAITING_TIMEOUT:<a name="line.443"></a>
+<span class="sourceLineNo">444</span> if (waitingSet == null) {<a name="line.444"></a>
+<span class="sourceLineNo">445</span> waitingSet = new HashSet<>();<a name="line.445"></a>
+<span class="sourceLineNo">446</span> }<a name="line.446"></a>
+<span class="sourceLineNo">447</span> waitingSet.add(proc);<a name="line.447"></a>
+<span class="sourceLineNo">448</span> break;<a name="line.448"></a>
+<span class="sourceLineNo">449</span> case FAILED:<a name="line.449"></a>
+<span class="sourceLineNo">450</span> // add the proc to the scheduler to perform the rollback<a name="line.450"></a>
+<span class="sourceLineNo">451</span> scheduler.addBack(proc);<a name="line.451"></a>
+<span class="sourceLineNo">452</span> break;<a name="line.452"></a>
+<span class="sourceLineNo">453</span> case ROLLEDBACK:<a name="line.453"></a>
+<span class="sourceLineNo">454</span> case INITIALIZING:<a name="line.454"></a>
+<span class="sourceLineNo">455</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.455"></a>
+<span class="sourceLineNo">456</span> LOG.error(msg);<a name="line.456"></a>
+<span class="sourceLineNo">457</span> throw new UnsupportedOperationException(msg);<a name="line.457"></a>
+<span class="sourceLineNo">458</span> default:<a name="line.458"></a>
+<span class="sourceLineNo">459</span> break;<a name="line.459"></a>
+<span class="sourceLineNo">460</span> }<a name="line.460"></a>
+<span class="sourceLineNo">461</span> }<a name="line.461"></a>
+<span class="sourceLineNo">462</span><a name="line.462"></a>
+<span class="sourceLineNo">463</span> // 3. Validate the stacks<a name="line.463"></a>
+<span class="sourceLineNo">464</span> int corruptedCount = 0;<a name="line.464"></a>
+<span class="sourceLineNo">465</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.465"></a>
+<span class="sourceLineNo">466</span> while (itStack.hasNext()) {<a name="line.466"></a>
+<span class="sourceLineNo">467</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.467"></a>
+<span class="sourceLineNo">468</span> RootProcedureState procStack = entry.getValue();<a name="line.468"></a>
+<span class="sourceLineNo">469</span> if (procStack.isValid()) continue;<a name="line.469"></a>
+<span class="sourceLineNo">470</span><a name="line.470"></a>
+<span class="sourceLineNo">471</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.471"></a>
+<span class="sourceLineNo">472</span> LOG.error("Corrupted " + proc);<a name="line.472"></a>
+<span class="sourceLineNo">473</span> procedures.remove(proc.getProcId());<a name="line.473"></a>
+<span class="sourceLineNo">474</span> runnableList.remove(proc);<a name="line.474"></a>
+<span class="sourceLineNo">475</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.475"></a>
+<span class="sourceLineNo">476</span> corruptedCount++;<a name="line.476"></a>
+<span class="sourceLineNo">477</span> }<a name="line.477"></a>
+<span class="sourceLineNo">478</span> itStack.remove();<a name="line.478"></a>
+<span class="sourceLineNo">479</span> }<a name="line.479"></a>
+<span class="sourceLineNo">480</span><a name="line.480"></a>
+<span class="sourceLineNo">481</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.481"></a>
+<span class="sourceLineNo">482</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.482"></a>
+<span class="sourceLineNo">483</span> }<a name="line.483"></a>
+<span class="sourceLineNo">484</span><a name="line.484"></a>
+<span class="sourceLineNo">485</span> // 4. Push the procedures to the timeout executor<a name="line.485"></a>
+<span class="sourceLineNo">486</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.486"></a>
+<span class="sourceLineNo">487</span> for (Procedure proc: waitingSet) {<a name="line.487"></a>
+<span class="sourceLineNo">488</span> proc.afterReplay(getEnvironment());<a name="line.488"></a>
+<span class="sourceLineNo">489</span> timeoutExecutor.add(proc);<a name="line.489"></a>
+<span class="sourceLineNo">490</span> }<a name="line.490"></a>
+<span class="sourceLineNo">491</span> }<a name="line.491"></a>
+<span class="sourceLineNo">492</span><a name="line.492"></a>
+<span class="sourceLineNo">493</span> // 5. Push the procedure to the scheduler<a name="line.493"></a>
+<span class="sourceLineNo">494</span> if (!runnableList.isEmpty()) {<a name="line.494"></a>
+<span class="sourceLineNo">495</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.495"></a>
+<span class="sourceLineNo">496</span> // some procedure may be started way before this stuff.<a name="line.496"></a>
+<span class="sourceLineNo">497</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.497"></a>
+<span class="sourceLineNo">498</span> Procedure proc = runnableList.get(i);<a name="line.498"></a>
+<span class="sourceLineNo">499</span> proc.afterReplay(getEnvironment());<a name="line.499"></a>
+<span class="sourceLineNo">500</span> if (!proc.hasParent()) {<a name="line.500"></a>
+<span class="sourceLineNo">501</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.501"></a>
+<span class="sourceLineNo">502</span> }<a name="line.502"></a>
+<span class="sourceLineNo">503</span> if (proc.wasExecuted()) {<a name="line.503"></a>
+<span class="sourceLineNo">504</span> scheduler.addFront(proc);<a name="line.504"></a>
+<span class="sourceLineNo">505</span> } else {<a name="line.505"></a>
+<span class="sourceLineNo">506</span> // if it was not in execution, it can wait.<a name="line.506"></a>
+<span class="sourceLineNo">507</span> scheduler.addBack(proc);<a name="line.507"></a>
+<span class="sourceLineNo">508</span> }<a name="line.508"></a>
+<span class="sourceLineNo">509</span> }<a name="line.509"></a>
+<span class="sourceLineNo">510</span> }<a name="line.510"></a>
+<span class="sourceLineNo">511</span> }<a name="line.511"></a>
<span class="sourceLineNo">512</span><a name="line.512"></a>
-<span class="sourceLineNo">513</span> // Create the Thread Group for the executors<a name="line.513"></a>
-<span class="sourceLineNo">514</span> threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.514"></a>
-<span class="sourceLineNo">515</span><a name="line.515"></a>
-<span class="sourceLineNo">516</span> // Create the timeout executor<a name="line.516"></a>
-<span class="sourceLineNo">517</span> timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.517"></a>
-<span class="sourceLineNo">518</span><a name="line.518"></a>
-<span class="sourceLineNo">519</span> // Create the workers<a name="line.519"></a>
-<span class="sourceLineNo">520</span> workerId.set(0);<a name="line.520"></a>
-<span class="sourceLineNo">521</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.521"></a>
-<span class="sourceLineNo">522</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.522"></a>
-<span class="sourceLineNo">523</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.523"></a>
-<span class="sourceLineNo">524</span> }<a name="line.524"></a>
-<span class="sourceLineNo">525</span><a name="line.525"></a>
-<span class="sourceLineNo">526</span> long st, et;<a name="line.526"></a>
-<span class="sourceLineNo">527</span><a name="line.527"></a>
-<span class="sourceLineNo">528</span> // Acquire the store lease.<a name="line.528"></a>
-<span class="sourceLineNo">529</span> st = EnvironmentEdgeManager.currentTime();<a name="line.529"></a>
-<span class="sourceLineNo">530</span> store.recoverLease();<a name="line.530"></a>
-<span class="sourceLineNo">531</span> et = EnvironmentEdgeManager.currentTime();<a name="line.531"></a>
-<span class="sourceLineNo">532</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.532"></a>
-<span class="sourceLineNo">533</span> StringUtils.humanTimeDiff(et - st));<a name="line.533"></a>
-<span class="sourceLineNo">534</span><a name="line.534"></a>
-<span class="sourceLineNo">535</span> // start the procedure scheduler<a name="line.535"></a>
-<span class="sourceLineNo">536</span> scheduler.start();<a name="line.536"></a>
-<span class="sourceLineNo">537</span><a name="line.537"></a>
-<span class="sourceLineNo">538</span> // TODO: Split in two steps.<a name="line.538"></a>
-<span class="sourceLineNo">539</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.539"></a>
-<span class="sourceLineNo">540</span> // The first one will make sure that we have the latest id,<a name="line.540"></a>
-<span class="sourceLineNo">541</span> // so we can start the threads and accept new procedures.<a name="line.541"></a>
-<span class="sourceLineNo">542</span> // The second step will do the actual load of old procedures.<a name="line.542"></a>
-<span class="sourceLineNo">543</span> st = EnvironmentEdgeManager.currentTime();<a name="line.543"></a>
-<span class="sourceLineNo">544</span> load(abortOnCorruption);<a name="line.544"></a>
-<span class="sourceLineNo">545</span> et = EnvironmentEdgeManager.currentTime();<a name="line.545"></a>
-<span class="sourceLineNo">546</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.546"></a>
-<span class="sourceLineNo">547</span> StringUtils.humanTimeDiff(et - st));<a name="line.547"></a>
-<span class="sourceLineNo">548</span><a name="line.548"></a>
-<span class="sourceLineNo">549</span> // Start the executors. Here we must have the lastProcId set.<a name="line.549"></a>
-<span class="sourceLineNo">550</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.550"></a>
-<span class="sourceLineNo">551</span> timeoutExecutor.start();<a name="line.551"></a>
-<span class="sourceLineNo">552</span> for (WorkerThread worker: workerThreads) {<a name="line.552"></a>
-<span class="sourceLineNo">553</span> worker.start();<a name="line.553"></a>
-<span class="sourceLineNo">554</span> }<a name="line.554"></a>
-<span class="sourceLineNo">555</span><a name="line.555"></a>
-<span class="sourceLineNo">556</span> // Internal chores<a name="line.556"></a>
-<span class="sourceLineNo">557</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.557"></a>
-<span class="sourceLineNo">558</span><a name="line.558"></a>
-<span class="sourceLineNo">559</span> // Add completed cleaner chore<a name="line.559"></a>
-<span class="sourceLineNo">560</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.560"></a>
-<span class="sourceLineNo">561</span> }<a name="line.561"></a>
-<span class="sourceLineNo">562</span><a name="line.562"></a>
-<span class="sourceLineNo">563</span> public void stop() {<a name="line.563"></a>
-<span class="sourceLineNo">564</span> if (!running.getAndSet(false)) {<a name="line.564"></a>
-<span class="sourceLineNo">565</span> return;<a name="line.565"></a>
-<span class="sourceLineNo">566</span> }<a name="line.566"></a>
-<span class="sourceLineNo">567</span><a name="line.567"></a>
-<span class="sourceLineNo">568</span> LOG.info("Stopping");<a name="line.568"></a>
-<span class="sourceLineNo">569</span> scheduler.stop();<a name="line.569"></a>
-<span class="sourceLineNo">570</span> timeoutExecutor.sendStopSignal();<a name="line.570"></a>
-<span class="sourceLineNo">571</span> }<a name="line.571"></a>
-<span class="sourceLineNo">572</span><a name="line.572"></a>
-<span class="sourceLineNo">573</span> @VisibleForTesting<a name="line.573"></a>
-<span class="sourceLineNo">574</span> public void join() {<a name="line.574"></a>
-<span class="sourceLineNo">575</span> assert !isRunning() : "expected not running";<a name="line.575"></a>
-<span class="sourceLineNo">576</span><a name="line.576"></a>
-<span class="sourceLineNo">577</span> // stop the timeout executor<a name="line.577"></a>
-<span class="sourceLineNo">578</span> timeoutExecutor.awaitTermination();<a name="line.578"></a>
-<span class="sourceLineNo">579</span> timeoutExecutor = null;<a name="line.579"></a>
-<span class="sourceLineNo">580</span><a name="line.580"></a>
-<span class="sourceLineNo">581</span> // stop the worker threads<a name="line.581"></a>
-<span class="sourceLineNo">582</span> for (WorkerThread worker: workerThreads) {<a name="line.582"></a>
-<span class="sourceLineNo">583</span> worker.awaitTermination();<a name="line.583"></a>
-<span class="sourceLineNo">584</span> }<a name="line.584"></a>
-<span class="sourceLineNo">585</span> workerThreads = null;<a name="line.585"></a>
-<span class="sourceLineNo">586</span><a name="line.586"></a>
-<span class="sourceLineNo">587</span> // Destroy the Thread Group for the executors<a name="line.587"></a>
-<span class="sourceLineNo">588</span> try {<a name="line.588"></a>
-<span class="sourceLineNo">589</span> threadGroup.destroy();<a name="line.589"></a>
-<span class="sourceLineNo">590</span> } catch (IllegalThreadStateException e) {<a name="line.590"></a>
-<span class="sourceLineNo">591</span> LOG.error("ThreadGroup " + threadGroup + " contains running threads; " + e.getMessage());<a name="line.591"></a>
-<span class="sourceLineNo">592</span> threadGroup.list();<a name="line.592"></a>
-<span class="sourceLineNo">593</span> } finally {<a name="line.593"></a>
-<span class="sourceLineNo">594</span> threadGroup = null;<a name="line.594"></a>
-<span class="sourceLineNo">595</span> }<a name="line.595"></a>
+<span class="sourceLineNo">513</span> /**<a name="line.513"></a>
+<span class="sourceLineNo">514</span> * Start the procedure executor.<a name="line.514"></a>
+<span class="sourceLineNo">515</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.515"></a>
+<span class="sourceLineNo">516</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.516"></a>
+<span class="sourceLineNo">517</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.517"></a>
+<span class="sourceLineNo">518</span> *<a name="line.518"></a>
+<span class="sourceLineNo">519</span> * @param numThreads number of threads available for procedure execution.<a name="line.519"></a>
+<span class="sourceLineNo">520</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.520"></a>
+<span class="sourceLineNo">521</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.521"></a>
+<span class="sourceLineNo">522</span> */<a name="line.522"></a>
+<span class="sourceLineNo">523</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.523"></a>
+<span class="sourceLineNo">524</span> if (!running.compareAndSet(false, true)) {<a name="line.524"></a>
+<span class="sourceLineNo">525</span> LOG.warn("Already running");<a name="line.525"></a>
+<span class="sourceLineNo">526</span> return;<a name="line.526"></a>
+<span class="sourceLineNo">527</span> }<a name="line.527"></a>
+<span class="sourceLineNo">528</span><a name="line.528"></a>
+<span class="sourceLineNo">529</span> // We have numThreads executor + one timer thread used for timing out<a name="line.529"></a>
+<span class="sourceLineNo">530</span> // procedures and triggering periodic procedures.<a name="line.530"></a>
+<span class="sourceLineNo">531</span> this.corePoolSize = numThreads;<a name="line.531"></a>
+<span class="sourceLineNo">532</span> this.maxPoolSize = 10 * numThreads;<a name="line.532"></a>
+<span class="sourceLineNo">533</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.533"></a>
+<span class="sourceLineNo">534</span> corePoolSize, maxPoolSize);<a name="line.534"></a>
+<span class="sourceLineNo">535</span><a name="line.535"></a>
+<span class="sourceLineNo">536</span> this.threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.536"></a>
+<span class="sourceLineNo">537</span> this.timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.537"></a>
+<span class="sourceLineNo">538</span><a name="line.538"></a>
+<span class="sourceLineNo">539</span> // Create the workers<a name="line.539"></a>
+<span class="sourceLineNo">540</span> workerId.set(0);<a name="line.540"></a>
+<span class="sourceLineNo">541</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.541"></a>
+<span class="sourceLineNo">542</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.542"></a>
+<span class="sourceLineNo">543</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.543"></a>
+<span class="sourceLineNo">544</span> }<a name="line.544"></a>
+<span class="sourceLineNo">545</span><a name="line.545"></a>
+<span class="sourceLineNo">546</span> long st, et;<a name="line.546"></a>
+<span class="sourceLineNo">547</span><a name="line.547"></a>
+<span class="sourceLineNo">548</span> // Acquire the store lease.<a name="line.548"></a>
+<span class="sourceLineNo">549</span> st = EnvironmentEdgeManager.currentTime();<a name="line.549"></a>
+<span class="sourceLineNo">550</span> store.recoverLease();<a name="line.550"></a>
+<span class="sourceLineNo">551</span> et = EnvironmentEdgeManager.currentTime();<a name="line.551"></a>
+<span class="sourceLineNo">552</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.552"></a>
+<span class="sourceLineNo">553</span> StringUtils.humanTimeDiff(et - st));<a name="line.553"></a>
+<span class="sourceLineNo">554</span><a name="line.554"></a>
+<span class="sourceLineNo">555</span> // start the procedure scheduler<a name="line.555"></a>
+<span class="sourceLineNo">556</span> scheduler.start();<a name="line.556"></a>
+<span class="sourceLineNo">557</span><a name="line.557"></a>
+<span class="sourceLineNo">558</span> // TODO: Split in two steps.<a name="line.558"></a>
+<span class="sourceLineNo">559</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.559"></a>
+<span class="sourceLineNo">560</span> // The first one will make sure that we have the latest id,<a name="line.560"></a>
+<span class="sourceLineNo">561</span> // so we can start the threads and accept new procedures.<a name="line.561"></a>
+<span class="sourceLineNo">562</span> // The second step will do the actual load of old procedures.<a name="line.562"></a>
+<span class="sourceLineNo">563</span> st = EnvironmentEdgeManager.currentTime();<a name="line.563"></a>
+<span class="sourceLineNo">564</span> load(abortOnCorruption);<a name="line.564"></a>
+<span class="sourceLineNo">565</span> et = EnvironmentEdgeManager.currentTime();<a name="line.565"></a>
+<span class="sourceLineNo">566</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.566"></a>
+<span class="sourceLineNo">567</span> StringUtils.humanTimeDiff(et - st));<a name="line.567"></a>
+<span class="sourceLineNo">568</span><a name="line.568"></a>
+<span class="sourceLineNo">569</span> // Start the executors. Here we must have the lastProcId set.<a name="line.569"></a>
+<span class="sourceLineNo">570</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.570"></a>
+<span class="sourceLineNo">571</span> timeoutExecutor.start();<a name="line.571"></a>
+<span class="sourceLineNo">572</span> for (WorkerThread worker: workerThreads) {<a name="line.572"></a>
+<span class="sourceLineNo">573</span> worker.start();<a name="line.573"></a>
+<span class="sourceLineNo">574</span> }<a name="line.574"></a>
+<span class="sourceLineNo">575</span><a name="line.575"></a>
+<span class="sourceLineNo">576</span> // Internal chores<a name="line.576"></a>
+<span class="sourceLineNo">577</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.577"></a>
+<span class="sourceLineNo">578</span><a name="line.578"></a>
+<span class="sourceLineNo">579</span> // Add completed cleaner chore<a name="line.579"></a>
+<span class="sourceLineNo">580</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.580"></a>
+<span class="sourceLineNo">581</span> }<a name="line.581"></a>
+<span class="sourceLineNo">582</span><a name="line.582"></a>
+<span class="sourceLineNo">583</span> public void stop() {<a name="line.583"></a>
+<span class="sourceLineNo">584</span> if (!running.getAndSet(false)) {<a name="line.584"></a>
+<span class="sourceLineNo">585</span> return;<a name="line.585"></a>
+<span class="sourceLineNo">586</span> }<a name="line.586"></a>
+<span class="sourceLineNo">587</span><a name="line.587"></a>
+<span class="sourceLineNo">588</span> LOG.info("Stopping");<a name="line.588"></a>
+<span class="sourceLineNo">589</span> scheduler.stop();<a name="line.589"></a>
+<span class="sourceLineNo">590</span> timeoutExecutor.sendStopSignal();<a name="line.590"></a>
+<span class="sourceLineNo">591</span> }<a name="line.591"></a>
+<span class="sourceLineNo">592</span><a name="line.592"></a>
+<span class="sourceLineNo">593</span> @VisibleForTesting<a name="line.593"></a>
+<span class="sourceLineNo">594</span> public void join() {<a name="line.594"></a>
+<span class="sourceLineNo">595</span> assert !isRunning() : "expected not running";<a name="line.595"></a>
<span class="sourceLineNo">596</span><a name="line.596"></a>
-<span class="sourceLineNo">597</span> // reset the in-memory state for testing<a name="line.597"></a>
-<span class="sourceLineNo">598</span> completed.clear();<a name="line.598"></a>
-<span class="sourceLineNo">599</span> rollbackStack.clear();<a name="line.599"></a>
-<span class="sourceLineNo">600</span> procedures.clear();<a name="line.600"></a>
-<span class="sourceLineNo">601</span> nonceKeysToProcIdsMap.clear();<a name="line.601"></a>
-<span class="sourceLineNo">602</span> scheduler.clear();<a name="line.602"></a>
-<span class="sourceLineNo">603</span> lastProcId.set(-1);<a name="line.603"></a>
-<span class="sourceLineNo">604</span> }<a name="line.604"></a>
-<span class="sourceLineNo">605</span><a name="line.605"></a>
-<span class="sourceLineNo">606</span> public void refreshConfiguration(final Configuration conf) {<a name="line.606"></a>
-<span class="sourceLineNo">607</span> this.conf = conf;<a name="line.607"></a>
-<span class="sourceLineNo">608</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.608"></a>
-<span class="sourceLineNo">609</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.609"></a>
-<span class="sourceLineNo">610</span> }<a name="line.610"></a>
-<span class="sourceLineNo">611</span><a name="line.611"></a>
-<span class="sourceLineNo">612</span> // ==========================================================================<a name="line.612"></a>
-<span class="sourceLineNo">613</span> // Accessors<a name="line.613"></a>
-<span class="sourceLineNo">614</span> // ==========================================================================<a name="line.614"></a>
-<span class="sourceLineNo">615</span> public boolean isRunning() {<a name="line.615"></a>
-<span class="sourceLineNo">616</span> return running.get();<a name="line.616"></a>
-<span class="sourceLineNo">617</span> }<a name="line.617"></a>
-<span class="sourceLineNo">618</span><a name="line.618"></a>
-<span class="sourceLineNo">619</span> /**<a name="line.619"></a>
-<span class="sourceLineNo">620</span> * @return the current number of worker threads.<a name="line.620"></a>
-<span class="sourceLineNo">621</span> */<a name="line.621"></a>
-<span class="sourceLineNo">622</span> public int getWorkerThreadCount() {<a name="line.622"></a>
-<span class="sourceLineNo">623</span> return workerThreads.size();<a name="line.623"></a>
-<span class="sourceLineNo">624</span> }<a name="line.624"></a>
-<span class="sourceLineNo">625</span><a name="line.625"></a>
-<span class="sourceLineNo">626</span> /**<a name="line.626"></a>
-<span class="sourceLineNo">627</span> * @return the core pool size settings.<a name="line.627"></a>
-<span class="sourceLineNo">628</span> */<a name="line.628"></a>
-<span class="sourceLineNo">629</span> public int getCorePoolSize() {<a name="line.629"></a>
-<span class="sourceLineNo">630</span> return corePoolSize;<a name="line.630"></a>
-<span class="sourceLineNo">631</span> }<a name="line.631"></a>
-<span class="sourceLineNo">632</span><a name="line.632"></a>
-<span class="sourceLineNo">633</span> public int getActiveExecutorCount() {<a name="line.633"></a>
-<span class="sourceLineNo">634</span> return activeExecutorCount.get();<a name="line.634"></a>
-<span class="sourceLineNo">635</span> }<a name="line.635"></a>
-<span class="sourceLineNo">636</span><a name="line.636"></a>
-<span class="sourceLineNo">637</span> public TEnvironment getEnvironment() {<a name="line.637"></a>
-<span class="sourceLineNo">638</span> return this.environment;<a name="line.638"></a>
-<span class="sourceLineNo">639</span> }<a name="line.639"></a>
-<span class="sourceLineNo">640</span><a name="line.640"></a>
-<span class="sourceLineNo">641</span> public ProcedureStore getStore() {<a name="line.641"></a>
-<span class="sourceLineNo">642</span> return this.store;<a name="line.642"></a>
+<span class="sourceLineNo">597</span> // stop the timeout executor<a name="line.597"></a>
+<span class="sourceLineNo">598</span> timeoutExecutor.awaitTermination();<a name="line.598"></a>
+<span class="sourceLineNo">599</span><a name="line.599"></a>
+<span class="sourceLineNo">600</span> // stop the worker threads<a name="line.600"></a>
+<span class="sourceLineNo">601</span> for (WorkerThread worker: workerThreads) {<a name="line.601"></a>
+<span class="sourceLineNo">602</span> worker.awaitTermination();<a name="line.602"></a>
+<span class="sourceLineNo">603</span> }<a name="line.603"></a>
+<span class="sourceLineNo">604</span><a name="line.604"></a>
+<span class="sourceLineNo">605</span> // Destroy the Thread Group for the executors<a name="line.605"></a>
+<span class="sourceLineNo">606</span> // TODO: Fix. #join is not place to destroy resources.<a name="line.606"></a>
+<span class="sourceLineNo">607</span> try {<a name="line.607"></a>
+<span class="sourceLineNo">608</span> threadGroup.destroy();<a name="line.608"></a>
+<span class="sourceLineNo">609</span> } catch (IllegalThreadStateException e) {<a name="line.609"></a>
+<span class="sourceLineNo">610</span> LOG.error("ThreadGroup {} contains running threads; {}: See STDOUT",<a name="line.610"></a>
+<span class="sourceLineNo">611</span> this.threadGroup, e.getMessage());<a name="line.611"></a>
+<span class="sourceLineNo">612</span> // This dumps list of threads on STDOUT.<a name="line.612"></a>
+<span class="sourceLineNo">613</span> this.threadGroup.list();<a name="line.613"></a>
+<span class="sourceLineNo">614</span> }<a name="line.614"></a>
+<span class="sourceLineNo">615</span><a name="line.615"></a>
+<span class="sourceLineNo">616</span> // reset the in-memory state for testing<a name="line.616"></a>
+<span class="sourceLineNo">617</span> completed.clear();<a name="line.617"></a>
+<span class="sourceLineNo">618</span> rollbackStack.clear();<a name="line.618"></a>
+<span class="sourceLineNo">619</span> procedures.clear();<a name="line.619"></a>
+<span class="sourceLineNo">620</span> nonceKeysToProcIdsMap.clear();<a name="line.620"></a>
+<span class="sourceLineNo">621</span> scheduler.clear();<a name="line.621"></a>
+<span class="sourceLineNo">622</span> lastProcId.set(-1);<a name="line.622"></a>
+<span class="sourceLineNo">623</span> }<a name="line.623"></a>
+<span class="sourceLineNo">624</span><a name="line.624"></a>
+<span class="sourceLineNo">625</span> public void refreshConfiguration(final Configuration conf) {<a name="line.625"></a>
+<span class="sourceLineNo">626</span> this.conf = conf;<a name="line.626"></a>
+<span class="sourceLineNo">627</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.627"></a>
+<span class="sourceLineNo">628</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.628"></a>
+<span class="sourceLineNo">629</span> }<a name="line.629"></a>
+<span class="sourceLineNo">630</span><a name="line.630"></a>
+<span class="sourceLineNo">631</span> // ==========================================================================<a name="line.631"></a>
+<span class="sourceLineNo">632</span> // Accessors<a name="line.632"></a>
+<span class="sourceLineNo">633</span> // ==========================================================================<a name="line.633"></a>
+<span class="sourceLineNo">634</span> public boolean isRunning() {<a name="line.634"></a>
+<span class="sourceLineNo">635</span> return running.get();<a name="line.635"></a>
+<span class="sourceLineNo">636</span> }<a name="line.636"></a>
+<span class="sourceLineNo">637</span><a name="line.637"></a>
+<span class="sourceLineNo">638</span> /**<a name="line.638"></a>
+<span class="sourceLineNo">639</span> * @return the current number of worker threads.<a name="line.639"></a>
+<span class="sourceLineNo">640</span> */<a name="line.640"></a>
+<span class="sourceLineNo">641</span> public int getWorkerThreadCount() {<a name="line.641"></a>
+<span class="sourceLineNo">642</span> return workerThreads.size();<a name="line.642"></a>
<span class="sourceLineNo">643</span> }<a name="line.643"></a>
<span class="sourceLineNo">644</span><a name="line.644"></a>
-<span class="sourceLineNo">645</span> ProcedureScheduler getScheduler() {<a name="line.645"></a>
-<span class="sourceLineNo">646</span> return scheduler;<a name="line.646"></a>
-<span class="sourceLineNo">647</span> }<a name="line.647"></a>
-<span class="sourceLineNo">648</span><a name="line.648"></a>
-<span class="sourceLineNo">649</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.649"></a>
-<span class="sourceLineNo">650</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.650"></a>
-<span class="sourceLineNo">651</span> this.scheduler.signalAll();<a name="line.651"></a>
-<span class="sourceLineNo">652</span> }<a name="line.652"></a>
-<span class="sourceLineNo">653</span><a name="line.653"></a>
-<span class="sourceLineNo">654</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.654"></a>
-<span class="sourceLineNo">655</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.655"></a>
-<span class="sourceLineNo">656</span> }<a name="line.656"></a>
-<span class="sourceLineNo">657</span><a name="line.657"></a>
-<span class="sourceLineNo">658</span> // ==========================================================================<a name="line.658"></a>
-<span class="sourceLineNo">659</span> // Submit/Remove Chores<a name="line.659"></a>
-<span class="sourceLineNo">660</span> // ==========================================================================<a name="line.660"></a>
-<span class="sourceLineNo">661</span><a name="line.661"></a>
-<span class="sourceLineNo">662</span> /**<a name="line.662"></a>
-<span class="sourceLineNo">663</span> * Add a chore procedure to the executor<a name="line.663"></a>
-<span class="sourceLineNo">664</span> * @param chore the chore to add<a name="line.664"></a>
-<span class="sourceLineNo">665</span> */<a name="line.665"></a>
-<span class="sourceLineNo">666</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.666"></a>
-<span class="sourceLineNo">667</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.667"></a>
-<span class="sourceLineNo">668</span> timeoutExecutor.add(chore);<a name="line.668"></a>
-<span class="sourceLineNo">669</span> }<a name="line.669"></a>
-<span class="sourceLineNo">670</span><a name="line.670"></a>
-<span class="sourceLineNo">671</span> /**<a name="line.671"></a>
-<span class="sourceLineNo">672</span> * Remove a chore procedure from the executor<a name="line.672"></a>
-<span class="sourceLineNo">673</span> * @param chore the chore to remove<a name="line.673"></a>
-<span class="sourceLineNo">674</span> * @return whether the chore is removed, or it will be removed later<a name="line.674"></a>
-<span class="sourceLineNo">675</span> */<a name="line.675"></a>
-<span class="sourceLineNo">676</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.676"></a>
-<span class="sourceLineNo">677</span> chore.setState(ProcedureState.SUCCESS);<a name="line.677"></a>
-<span class="sourceLineNo">678</span> return timeoutExecutor.remove(chore);<a name="line.678"></a>
-<span class="sourceLineNo">679</span> }<a name="line.679"></a>
+<span class="sourceLineNo">645</span> /**<a name="line.645"></a>
+<span class="sourceLineNo">646</span> * @return the core pool size settings.<a name="line.646"></a>
+<span class="sourceLineNo">647</span> */<a name="line.647"></a>
+<span class="sourceLineNo">648</span> public int getCorePoolSize() {<a name="line.648"></a>
+<span class="sourceLineNo">649</span> return corePoolSize;<a name="line.649"></a>
+<span class="sourceLineNo">650</span> }<a name="line.650"></a>
+<span class="sourceLineNo">651</span><a name="line.651"></a>
+<span class="sourceLineNo">652</span> public int getActiveExecutorCount() {<a name="line.652"></a>
+<span class="sourceLineNo">653</span> return activeExecutorCount.get();<a name="line.653"></a>
+<span class="sourceLineNo">654</span> }<a name="line.654"></a>
+<span class="sourceLineNo">655</span><a name="line.655"></a>
+<span class="sourceLineNo">656</span> public TEnvironment getEnvironment() {<a name="line.656"></a>
+<span class="sourceLineNo">657</span> return this.environment;<a name="line.657"></a>
+<span class="sourceLineNo">658</span> }<a name="line.658"></a>
+<span class="sourceLineNo">659</span><a name="line.659"></a>
+<span class="sourceLineNo">660</span> public ProcedureStore getStore() {<a name="line.660"></a>
+<span class="sourceLineNo">661</span> return this.store;<a name="line.661"></a>
+<span class="sourceLineNo">662</span> }<a name="line.662"></a>
+<span class="sourceLineNo">663</span><a name="line.663"></a>
+<span class="sourceLineNo">664</span> ProcedureScheduler getScheduler() {<a name="line.664"></a>
+<span class="sourceLineNo">665</span> return scheduler;<a name="line.665"></a>
+<span class="sourceLineNo">666</span> }<a name="line.666"></a>
+<span class="sourceLineNo">667</span><a name="line.667"></a>
+<span class="sourceLineNo">668</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.668"></a>
+<span class="sourceLineNo">669</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.669"></a>
+<span class="sourceLineNo">670</span> this.scheduler.signalAll();<a name="line.670"></a>
+<span class="sourceLineNo">671</span> }<a name="line.671"></a>
+<span class="sourceLineNo">672</span><a name="line.672"></a>
+<span class="sourceLineNo">673</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.673"></a>
+<span class="sourceLineNo">674</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.674"></a>
+<span class="sourceLineNo">675</span> }<a name="line.675"></a>
+<span class="sourceLineNo">676</span><a name="line.676"></a>
+<span class="sourceLineNo">677</span> // ==========================================================================<a name="line.677"></a>
+<span class="sourceLineNo">678</span> // Submit/Remove Chores<a name="line.678"></a>
+<span class="sourceLineNo">679</span> // ==========================================================================<a name="line.679"></a>
<span class="sourceLineNo">680</span><a name="line.680"></a>
-<span class="sourceLineNo">681</span> // ==========================================================================<a name="line.681"></a>
-<span class="sourceLineNo">682</span> // Nonce Procedure helpers<a name="line.682"></a>
-<span class="sourceLineNo">683</span> // ==========================================================================<a name="line.683"></a>
-<span class="sourceLineNo">684</span> /**<a name="line.684"></a>
-<span class="sourceLineNo">685</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.685"></a>
-<span class="sourceLineNo">686</span> * @param nonceGroup<a name="line.686"></a>
-<span class="sourceLineNo">687</span> * @param nonce<a name="line.687"></a>
-<span class="sourceLineNo">688</span> * @return the generated NonceKey<a name="line.688"></a>
-<span class="sourceLineNo">689</span> */<a name="line.689"></a>
-<span class="sourceLineNo">690</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.690"></a>
-<span class="sourceLineNo">691</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.691"></a>
-<span class="sourceLineNo">692</span> }<a name="line.692"></a>
-<span class="sourceLineNo">693</span><a name="line.693"></a>
-<span class="sourceLineNo">694</span> /**<a name="line.694"></a>
-<span class="sourceLineNo">695</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.695"></a>
-<span class="sourceLineNo">696</span> * A procId will be reserved and on submitProcedure(),<a name="line.696"></a>
-<span class="sourceLineNo">697</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.697"></a>
-<span class="sourceLineNo">698</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.698"></a>
-<span class="sourceLineNo">699</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.699"></a>
-<span class="sourceLineNo">700</span> * and submit the procedure.<a name="line.700"></a>
-<span class="sourceLineNo">701</span> *<a name="line.701"></a>
-<span class="sourceLineNo">702</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.702"></a>
-<span class="sourceLineNo">703</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.703"></a>
-<span class="sourceLineNo">704</span> */<a name="line.704"></a>
-<span class="sourceLineNo">705</span> public long registerNonce(final NonceKey nonceKey) {<a name="line.705"></a>
-<span class="sourceLineNo">706</span> if (nonceKey == null) return -1;<a name="line.706"></a>
-<span class="sourceLineNo">707</span><a name="line.707"></a>
-<span class="sourceLineNo">708</span> // check if we have already a Reserved ID for the nonce<a name="line.708"></a>
-<span class="sourceLineNo">709</span> Long oldProcId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.709"></a>
-<span class="sourceLineNo">710</span> if (oldProcId == null) {<a name="line.710"></a>
-<span class="sourceLineNo">711</span> // reserve a new Procedure ID, this will be associated with the nonce<a name="line.711"></a>
-<span class="sourceLineNo">712</span> // and the procedure submitted with the specified nonce will use this ID.<a name="line.712"></a>
-<span class="sourceLineNo">713</span> final long newProcId = nextProcId();<a name="line.713"></a>
-<span class="sourceLineNo">714</span> oldProcId = nonceKeysToProcIdsMap.putIfAbsent(nonceKey, newProcId);<a name="line.714"></a>
-<span class="sourceLineNo">715</span> if (oldProcId == null) return -1;<a name="line.715"></a>
-<span class="sourceLineNo">716</span> }<a name="line.716"></a>
-<span class="sourceLineNo">717</span><a name="line.717"></a>
-<span class="sourceLineNo">718</span> // we found a registered nonce, but the procedure may not have been submitted yet.<a name="line.718"></a>
-<span class="sourceLineNo">719</span> // since the client expect the procedure to be submitted, spin here until it is.<a name="line.719"></a>
-<span class="sourceLineNo">720</span> final boolean traceEnabled = LOG.isTraceEnabled();<a name="line.720"></a>
-<span class="sourceLineNo">721</span> while (isRunning() &&<a name="line.721"></a>
-<span class="sourceLineNo">722</span> !(procedures.containsKey(oldProcId) || completed.containsKey(oldProcId)) &&<a name="line.722"></a>
-<span class="sourceLineNo">723</span> nonceKeysToProcIdsMap.containsKey(nonceKey)) {<a name="line.723"></a>
-<span class="sourceLineNo">724</span> if (traceEnabled) {<a name="line.724"></a>
-<span class="sourceLineNo">725</span> LOG.trace("Waiting for pid=" + oldProcId.longValue() + " to be submitted");<a name="line.725"></a>
-<span class="sourceLineNo">726</span> }<a name="line.726"></a>
-<span class="sourceLineNo">727</span> Threads.sleep(100);<a name="line.727"></a>
-<span class="sourceLineNo">728</span> }<a name="line.728"></a>
-<span class="sourceLineNo">729</span> return oldProcId.longValue();<a name="line.729"></a>
-<span class="sourceLineNo">730</span> }<a name="line.730"></a>
-<span class="sourceLineNo">731</span><a name="line.731"></a>
-<span class="sourceLineNo">732</span> /**<a name="line.732"></a>
-<span class="sourceLineNo">733</span> * Remove the NonceKey if the procedure was not submitted to the executor.<a name="line.733"></a>
-<span class="sourceLineNo">734</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.734"></a>
-<span class="sourceLineNo">735</span> */<a name="line.735"></a>
-<span class="sourceLineNo">736</span> public void unregisterNonceIfProcedureWasNotSubmitted(final NonceKey nonceKey) {<a name="line.736"></a>
-<span class="sourceLineNo">737</span> if (nonceKey == null) return;<a name="line.737"></a>
-<span class="sourceLineNo">738</span><a name="line.738"></a>
-<span class="sourceLineNo">739</span> final Long procId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.739"></a>
-<span class="sourceLineNo">740</span> if (procId == null) return;<a name="line.740"></a>
-<span class="sourceLineNo">741</span><a name="line.741"></a>
-<span class="sourceLineNo">742</span> // if the procedure was not submitted, remove the nonce<a name="line.742"></a>
-<span class="sourceLineNo">743</span> if (!(procedures.containsKey(procId) || completed.containsKey(procId))) {<a name="line.743"></a>
-<span class="sourceLineNo">744</span> nonceKeysToProcIdsMap.remove(nonceKey);<a name="line.744"></a>
-<span class="sourceLineNo">745</span> }<a name="line.745"></a>
-<span class="sourceLineNo">746</span> }<a name="line.746"></a>
-<span class="sourceLineNo">747</span><a name="line.747"></a>
-<span class="sourceLineNo">748</span> public static class FailedProcedure<TEnvironment> extends Procedure<TEnvironment> {<a name="line.748"></a>
-<span class="sourceLineNo">749</span> private String procName;<a name="line.749"></a>
+<span class="sourceLineNo">681</span> /**<a name="line.681"></a>
+<span class="sourceLineNo">682</span> * Add a chore procedure to the executor<a name="line.682"></a>
+<span class="sourceLineNo">683</span> * @param chore the chore to add<a name="line.683"></a>
+<span class="sourceLineNo">684</span> */<a name="line.684"></a>
+<span class="sourceLineNo">685</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.685"></a>
+<span class="sourceLineNo">686</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.686"></a>
+<span class="sourceLineNo">687</span> timeoutExecutor.add(chore);<a name="line.687"></a>
+<span class="sourceLineNo">688</span> }<a name="line.688"></a>
+<span class="sourceLineNo">689</span><a name="line.689"></a>
+<span class="sourceLineNo">690</span> /**<a name="line.690"></a>
+<span class="sourceLineNo">691</span> * Remove a chore procedure from the executor<a name="line.691"></a>
+<span class="sourceLineNo">692</span> * @param chore the chore to remove<a name="line.692"></a>
+<span class="sourceLineNo">693</span> * @return whether the chore is removed, or it will be removed later<a name="line.693"></a>
+<span class="sourceLineNo">694</span> */<a name="line.694"></a>
+<span class="sourceLineNo">695</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.695"></a>
+<span class="sourceLineNo">696</span> chore.setState(ProcedureState.SUCCESS);<a name="line.696"></a>
+<span class="sourceLineNo">697</span> return timeoutExecutor.remove(chore);<a name="line.697"></a>
+<span class="sourceLineNo">698</span> }<a name="line.698"></a>
+<span class="sourceLineNo">699</span><a name="line.699"></a>
+<span class="sourceLineNo">700</span> // ==========================================================================<a name="line.700"></a>
+<span class="sourceLineNo">701</span> // Nonce Procedure helpers<a name="line.701"></a>
+<span class="sourceLineNo">702</span> // ==========================================================================<a name="line.702"></a>
+<span class="sourceLineNo">703</span> /**<a name="line.703"></a>
+<span class="sourceLineNo">704</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.704"></a>
+<span class="sourceLineNo">705</span> * @param nonceGroup<a name="line.705"></a>
+<span class="sourceLineNo">706</span> * @param nonce<a name="line.706"></a>
+<span class="sourceLineNo">707</span> * @return the generated NonceKey<a name="line.707"></a>
+<span class="sourceLineNo">708</span> */<a name="line.708"></a>
+<span class="sourceLineNo">709</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.709"></a>
+<span class="sourceLineNo">710</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.710"></a>
+<span class="sourceLineNo">711</span> }<a name="line.711"></a>
+<span class="sourceLineNo">712</span><a name="line.712"></a>
+<span class="sourceLineNo">713</span> /**<a name="line.713"></a>
+<span class="sourceLineNo">714</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.714"></a>
+<span class="sourceLineNo">715</span> * A procId will be reserved and on submitProcedure(),<a name="line.715"></a>
+<span class="sourceLineNo">716</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.716"></a>
+<span class="sourceLineNo">717</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.717"></a>
+<span class="sourceLineNo">718</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.718"></a>
+<span class="sourceLineNo">719</span> * and submit the procedure.<a name="line.719"></a>
+<span class="sourceLineNo">720</span> *<a name="line.720"></a>
+<span class="sourceLineNo">721</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.721"></a>
+<span class="sourceLineNo">722</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.722"></a>
+<span class="sourceLineNo">723</span> */<a name="line.723
<TRUNCATED>
[16/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
Published site at 4cb444e77b41cdb733544770a471068256d65bbe.
Project: http://git-wip-us.apache.org/repos/asf/hbase-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase-site/commit/2d03f412
Tree: http://git-wip-us.apache.org/repos/asf/hbase-site/tree/2d03f412
Diff: http://git-wip-us.apache.org/repos/asf/hbase-site/diff/2d03f412
Branch: refs/heads/asf-site
Commit: 2d03f412fad3ab74ddbf8889c9c5eb1e0005a2ce
Parents: 18642a4
Author: jenkins <bu...@apache.org>
Authored: Thu May 3 14:47:16 2018 +0000
Committer: jenkins <bu...@apache.org>
Committed: Thu May 3 14:47:16 2018 +0000
----------------------------------------------------------------------
acid-semantics.html | 4 +-
apache_hbase_reference_guide.pdf | 4 +-
book.html | 2 +-
bulk-loads.html | 4 +-
checkstyle-aggregate.html | 92 +-
coc.html | 4 +-
dependencies.html | 4 +-
dependency-convergence.html | 4 +-
dependency-info.html | 4 +-
dependency-management.html | 4 +-
devapidocs/constant-values.html | 6 +-
devapidocs/index-all.html | 12 +-
.../master/assignment/AssignmentManager.html | 94 +-
.../ProcedureExecutor.FailedProcedure.html | 20 +-
...ProcedureExecutor.KeepAliveWorkerThread.html | 6 +-
.../ProcedureExecutor.WorkerMonitor.html | 32 +-
.../ProcedureExecutor.WorkerThread.html | 20 +-
.../hbase/procedure2/ProcedureExecutor.html | 176 +-
.../ProcedureExecutor.WorkerThread.html | 4 +-
.../class-use/TimeoutExecutorThread.html | 4 +-
.../org/apache/hadoop/hbase/Version.html | 6 +-
...signmentManager.RegionInTransitionChore.html | 1389 ++++----
...ssignmentManager.RegionInTransitionStat.html | 1389 ++++----
.../master/assignment/AssignmentManager.html | 1389 ++++----
...edureExecutor.CompletedProcedureCleaner.html | 3171 +++++++++---------
...dureExecutor.CompletedProcedureRetainer.html | 3171 +++++++++---------
.../ProcedureExecutor.FailedProcedure.html | 3171 +++++++++---------
...ProcedureExecutor.KeepAliveWorkerThread.html | 3171 +++++++++---------
...edureExecutor.ProcedureExecutorListener.html | 3171 +++++++++---------
.../procedure2/ProcedureExecutor.Testing.html | 3171 +++++++++---------
.../ProcedureExecutor.WorkerMonitor.html | 3171 +++++++++---------
.../ProcedureExecutor.WorkerThread.html | 3171 +++++++++---------
.../hbase/procedure2/ProcedureExecutor.html | 3171 +++++++++---------
downloads.html | 4 +-
export_control.html | 4 +-
index.html | 4 +-
integration.html | 4 +-
issue-tracking.html | 4 +-
license.html | 4 +-
mail-lists.html | 4 +-
metrics.html | 4 +-
old_news.html | 4 +-
plugin-management.html | 4 +-
plugins.html | 4 +-
poweredbyhbase.html | 4 +-
project-info.html | 4 +-
project-reports.html | 4 +-
project-summary.html | 4 +-
pseudo-distributed.html | 4 +-
replication.html | 4 +-
resources.html | 4 +-
source-repository.html | 4 +-
sponsors.html | 4 +-
supportingprojects.html | 4 +-
team-list.html | 4 +-
.../master/TestAssignmentManagerMetrics.html | 26 +-
.../master/TestAssignmentManagerMetrics.html | 248 +-
57 files changed, 16891 insertions(+), 16683 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/acid-semantics.html
----------------------------------------------------------------------
diff --git a/acid-semantics.html b/acid-semantics.html
index 0bfe44a..80ab395 100644
--- a/acid-semantics.html
+++ b/acid-semantics.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase –
Apache HBase (TM) ACID Properties
@@ -601,7 +601,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/apache_hbase_reference_guide.pdf
----------------------------------------------------------------------
diff --git a/apache_hbase_reference_guide.pdf b/apache_hbase_reference_guide.pdf
index ec7e132..22dac6d 100644
--- a/apache_hbase_reference_guide.pdf
+++ b/apache_hbase_reference_guide.pdf
@@ -5,8 +5,8 @@
/Author (Apache HBase Team)
/Creator (Asciidoctor PDF 1.5.0.alpha.15, based on Prawn 2.2.2)
/Producer (Apache HBase Team)
-/ModDate (D:20180502142951+00'00')
-/CreationDate (D:20180502144424+00'00')
+/ModDate (D:20180503142953+00'00')
+/CreationDate (D:20180503144436+00'00')
>>
endobj
2 0 obj
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/book.html
----------------------------------------------------------------------
diff --git a/book.html b/book.html
index 3dd79a8..16d7ef2 100644
--- a/book.html
+++ b/book.html
@@ -37873,7 +37873,7 @@ The server will return cellblocks compressed using this same compressor as long
<div id="footer">
<div id="footer-text">
Version 3.0.0-SNAPSHOT<br>
-Last updated 2018-05-02 14:29:51 UTC
+Last updated 2018-05-03 14:29:53 UTC
</div>
</div>
</body>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/bulk-loads.html
----------------------------------------------------------------------
diff --git a/bulk-loads.html b/bulk-loads.html
index 66eadba..a3ebc81 100644
--- a/bulk-loads.html
+++ b/bulk-loads.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase –
Bulk Loads in Apache HBase (TM)
@@ -306,7 +306,7 @@ under the License. -->
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/checkstyle-aggregate.html
----------------------------------------------------------------------
diff --git a/checkstyle-aggregate.html b/checkstyle-aggregate.html
index 0186c79..eab9d19 100644
--- a/checkstyle-aggregate.html
+++ b/checkstyle-aggregate.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Checkstyle Results</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -10327,12 +10327,12 @@
<td><a class="externalLink" href="http://checkstyle.sourceforge.net/config_javadoc.html#JavadocTagContinuationIndentation">JavadocTagContinuationIndentation</a>
<ul>
<li>offset: <tt>"2"</tt></li></ul></td>
-<td>783</td>
+<td>797</td>
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td></tr>
<tr class="a">
<td></td>
<td><a class="externalLink" href="http://checkstyle.sourceforge.net/config_javadoc.html#NonEmptyAtclauseDescription">NonEmptyAtclauseDescription</a></td>
-<td>3837</td>
+<td>3823</td>
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td></tr>
<tr class="b">
<td>misc</td>
@@ -14541,7 +14541,7 @@
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 26 has parse error. Missed HTML close tag 'arg'. Sometimes it means that close tag missed for one of previous tags.</td>
<td>44</td></tr></table></div>
<div class="section">
@@ -15177,7 +15177,7 @@
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 4 has parse error. Missed HTML close tag 'pre'. Sometimes it means that close tag missed for one of previous tags.</td>
<td>59</td></tr></table></div>
<div class="section">
@@ -16932,7 +16932,7 @@
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 19 has parse error. Details: no viable alternative at input '<code>\n * List<Future<' while parsing HTML_TAG</td>
<td>167</td></tr></table></div>
<div class="section">
@@ -19854,7 +19854,7 @@
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 2 has parse error. Unrecognized error from ANTLR parser: null</td>
<td>83</td></tr>
<tr class="b">
@@ -33801,7 +33801,7 @@
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 37 has parse error. Details: no viable alternative at input '<ColumnFamily,' while parsing HTML_ELEMENT</td>
<td>29</td></tr></table></div>
<div class="section">
@@ -50097,7 +50097,7 @@
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 24 has parse error. Details: no viable alternative at input '<key,' while parsing HTML_ELEMENT</td>
<td>25</td></tr>
<tr class="a">
@@ -53685,7 +53685,7 @@
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 2 has parse error. Unrecognized error from ANTLR parser: null</td>
<td>587</td></tr>
<tr class="b">
@@ -65855,73 +65855,73 @@
<td>blocks</td>
<td>EmptyBlock</td>
<td>Must have at least one statement.</td>
-<td>1260</td></tr>
+<td>1261</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
<td>JavadocTagContinuationIndentation</td>
<td>Line continuation have incorrect indentation level, expected level should be 2.</td>
-<td>1294</td></tr>
+<td>1295</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>1411</td></tr>
+<td>1412</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
<td>NonEmptyAtclauseDescription</td>
<td>At-clause should have a non-empty description.</td>
-<td>1429</td></tr>
+<td>1430</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
<td>NonEmptyAtclauseDescription</td>
<td>At-clause should have a non-empty description.</td>
-<td>1431</td></tr>
+<td>1432</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>1434</td></tr>
+<td>1435</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>1439</td></tr>
+<td>1440</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>EmptyBlock</td>
<td>Must have at least one statement.</td>
-<td>1516</td></tr>
+<td>1517</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>1704</td></tr>
+<td>1705</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>1829</td></tr>
+<td>1830</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>indentation</td>
<td>Indentation</td>
<td>'lambda arguments' have incorrect indentation level 8, expected level should be 6.</td>
-<td>1877</td></tr>
+<td>1878</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>1894</td></tr></table></div>
+<td>1895</td></tr></table></div>
<div class="section">
<h3 id="org.apache.hadoop.hbase.master.assignment.GCMergedRegionsProcedure.java">org/apache/hadoop/hbase/master/assignment/GCMergedRegionsProcedure.java</h3>
<table border="0" class="table table-striped">
@@ -73296,7 +73296,7 @@
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 60 has parse error. Missed HTML close tag 'Comparable'. Sometimes it means that close tag missed for one of previous tags.</td>
<td>29</td></tr>
<tr class="a">
@@ -75038,97 +75038,97 @@
<td>sizes</td>
<td>MethodLength</td>
<td>Method length is 151 lines (max allowed is 150).</td>
-<td>337</td></tr>
+<td>360</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>446</td></tr>
+<td>469</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>452</td></tr>
+<td>475</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
<td>NonEmptyAtclauseDescription</td>
<td>At-clause should have a non-empty description.</td>
-<td>686</td></tr>
+<td>705</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
<td>NonEmptyAtclauseDescription</td>
<td>At-clause should have a non-empty description.</td>
-<td>687</td></tr>
+<td>706</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>706</td></tr>
+<td>725</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>715</td></tr>
+<td>734</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>737</td></tr>
+<td>756</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>740</td></tr>
+<td>759</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>812</td></tr>
+<td>831</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>815</td></tr>
+<td>834</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>1040</td></tr>
+<td>1059</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>1131</td></tr>
+<td>1150</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>indentation</td>
<td>Indentation</td>
<td>'block' child have incorrect indentation level 16, expected level should be 14.</td>
-<td>1180</td></tr>
+<td>1199</td></tr>
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>coding</td>
<td>InnerAssignment</td>
<td>Inner assignments should be avoided.</td>
-<td>1303</td></tr>
+<td>1322</td></tr>
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>blocks</td>
<td>NeedBraces</td>
<td>'if' construct must use '{}'s.</td>
-<td>1534</td></tr></table></div>
+<td>1553</td></tr></table></div>
<div class="section">
<h3 id="org.apache.hadoop.hbase.procedure2.ProcedureScheduler.java">org/apache/hadoop/hbase/procedure2/ProcedureScheduler.java</h3>
<table border="0" class="table table-striped">
@@ -77478,7 +77478,7 @@
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 0 has parse error. Unrecognized error from ANTLR parser: null</td>
<td>74</td></tr>
<tr class="a">
@@ -87408,7 +87408,7 @@
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 33 has parse error. Missed HTML close tag 'number'. Sometimes it means that close tag missed for one of previous tags.</td>
<td>127</td></tr>
<tr class="b">
@@ -115977,7 +115977,7 @@
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 23 has parse error. Details: no viable alternative at input '<expected,' while parsing HTML_ELEMENT</td>
<td>143</td></tr>
<tr class="b">
@@ -116181,7 +116181,7 @@
<tr class="a">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 78 has parse error. Details: no viable alternative at input '<Object>, Map<String,' while parsing HTML_TAG</td>
<td>33</td></tr>
<tr class="b">
@@ -116781,7 +116781,7 @@
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 35 has parse error. Missed HTML close tag 'index'. Sometimes it means that close tag missed for one of previous tags.</td>
<td>823</td></tr>
<tr class="a">
@@ -120594,7 +120594,7 @@
<tr class="b">
<td><img src="images/icon_error_sml.gif" alt="" /> Error</td>
<td>javadoc</td>
-<td>NonEmptyAtclauseDescription</td>
+<td>JavadocTagContinuationIndentation</td>
<td>Javadoc comment at column 58 has parse error. Details: no viable alternative at input '<byte[' while parsing HTML_ELEMENT</td>
<td>426</td></tr>
<tr class="a">
@@ -123435,7 +123435,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/coc.html
----------------------------------------------------------------------
diff --git a/coc.html b/coc.html
index 973d28a..741545b 100644
--- a/coc.html
+++ b/coc.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase –
Code of Conduct Policy
@@ -375,7 +375,7 @@ email to <a class="externalLink" href="mailto:private@hbase.apache.org">the priv
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/dependencies.html
----------------------------------------------------------------------
diff --git a/dependencies.html b/dependencies.html
index e3731d3..2d6b9c2 100644
--- a/dependencies.html
+++ b/dependencies.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Project Dependencies</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -440,7 +440,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/dependency-convergence.html
----------------------------------------------------------------------
diff --git a/dependency-convergence.html b/dependency-convergence.html
index 4bdbfd3..86fe4cf 100644
--- a/dependency-convergence.html
+++ b/dependency-convergence.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Reactor Dependency Convergence</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -1105,7 +1105,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/dependency-info.html
----------------------------------------------------------------------
diff --git a/dependency-info.html b/dependency-info.html
index ea7bd4d..12bcabc 100644
--- a/dependency-info.html
+++ b/dependency-info.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Dependency Information</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -313,7 +313,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/dependency-management.html
----------------------------------------------------------------------
diff --git a/dependency-management.html b/dependency-management.html
index 7fb9f61..202f8c9 100644
--- a/dependency-management.html
+++ b/dependency-management.html
@@ -7,7 +7,7 @@
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20180502" />
+ <meta name="Date-Revision-yyyymmdd" content="20180503" />
<meta http-equiv="Content-Language" content="en" />
<title>Apache HBase – Project Dependency Management</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.5-HBASE.min.css" />
@@ -969,7 +969,7 @@
<a href="https://www.apache.org/">The Apache Software Foundation</a>.
All rights reserved.
- <li id="publishDate" class="pull-right">Last Published: 2018-05-02</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-05-03</li>
</p>
</div>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/constant-values.html
----------------------------------------------------------------------
diff --git a/devapidocs/constant-values.html b/devapidocs/constant-values.html
index 852c5ce..fda2ae4 100644
--- a/devapidocs/constant-values.html
+++ b/devapidocs/constant-values.html
@@ -3768,21 +3768,21 @@
<!-- -->
</a><code>public static final <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a></code></td>
<td><code><a href="org/apache/hadoop/hbase/Version.html#date">date</a></code></td>
-<td class="colLast"><code>"Wed May 2 14:38:54 UTC 2018"</code></td>
+<td class="colLast"><code>"Thu May 3 14:39:11 UTC 2018"</code></td>
</tr>
<tr class="rowColor">
<td class="colFirst"><a name="org.apache.hadoop.hbase.Version.revision">
<!-- -->
</a><code>public static final <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a></code></td>
<td><code><a href="org/apache/hadoop/hbase/Version.html#revision">revision</a></code></td>
-<td class="colLast"><code>"2e9b96e4f14c2802b59c6037cbfe4a427ec93bea"</code></td>
+<td class="colLast"><code>"4cb444e77b41cdb733544770a471068256d65bbe"</code></td>
</tr>
<tr class="altColor">
<td class="colFirst"><a name="org.apache.hadoop.hbase.Version.srcChecksum">
<!-- -->
</a><code>public static final <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a></code></td>
<td><code><a href="org/apache/hadoop/hbase/Version.html#srcChecksum">srcChecksum</a></code></td>
-<td class="colLast"><code>"300425a7629c0718634c89407e4d606a"</code></td>
+<td class="colLast"><code>"b65d69cf3ec7b334d9360592429adf77"</code></td>
</tr>
<tr class="rowColor">
<td class="colFirst"><a name="org.apache.hadoop.hbase.Version.url">
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/index-all.html
----------------------------------------------------------------------
diff --git a/devapidocs/index-all.html b/devapidocs/index-all.html
index deffc39..6f0a532 100644
--- a/devapidocs/index-all.html
+++ b/devapidocs/index-all.html
@@ -110889,7 +110889,9 @@ service.</div>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/util/Threads.html#threadDumpingIsAlive-java.lang.Thread-">threadDumpingIsAlive(Thread)</a></span> - Static method in class org.apache.hadoop.hbase.util.<a href="org/apache/hadoop/hbase/util/Threads.html" title="class in org.apache.hadoop.hbase.util">Threads</a></dt>
<dd> </dd>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#threadGroup">threadGroup</a></span> - Variable in class org.apache.hadoop.hbase.procedure2.<a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor</a></dt>
-<dd> </dd>
+<dd>
+<div class="block">Created in the <a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>ProcedureExecutor.start(int, boolean)</code></a> method.</div>
+</dd>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/executor/ExecutorService.RunningEventStatus.html#threadInfo">threadInfo</a></span> - Variable in class org.apache.hadoop.hbase.executor.<a href="org/apache/hadoop/hbase/executor/ExecutorService.RunningEventStatus.html" title="class in org.apache.hadoop.hbase.executor">ExecutorService.RunningEventStatus</a></dt>
<dd> </dd>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.Args.html#threadKeepAliveTimeSec">threadKeepAliveTimeSec</a></span> - Variable in class org.apache.hadoop.hbase.thrift.<a href="org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.Args.html" title="class in org.apache.hadoop.hbase.thrift">TBoundedThreadPoolServer.Args</a></dt>
@@ -111418,7 +111420,9 @@ service.</div>
<div class="block">Create a generic timer for a task/process.</div>
</dd>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#timeoutExecutor">timeoutExecutor</a></span> - Variable in class org.apache.hadoop.hbase.procedure2.<a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor</a></dt>
-<dd> </dd>
+<dd>
+<div class="block">Created in the <a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>ProcedureExecutor.start(int, boolean)</code></a> method.</div>
+</dd>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.html#timeoutExecutor">timeoutExecutor</a></span> - Variable in class org.apache.hadoop.hbase.procedure2.<a href="org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.html" title="class in org.apache.hadoop.hbase.procedure2">RemoteProcedureDispatcher</a></dt>
<dd> </dd>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.TimeoutExecutorThread.html#TimeoutExecutorThread--">TimeoutExecutorThread()</a></span> - Constructor for class org.apache.hadoop.hbase.procedure2.<a href="org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.TimeoutExecutorThread.html" title="class in org.apache.hadoop.hbase.procedure2">RemoteProcedureDispatcher.TimeoutExecutorThread</a></dt>
@@ -120164,7 +120168,9 @@ the order they are declared.</div>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html#WorkerThread-java.lang.ThreadGroup-java.lang.String-">WorkerThread(ThreadGroup, String)</a></span> - Constructor for class org.apache.hadoop.hbase.procedure2.<a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.WorkerThread</a></dt>
<dd> </dd>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#workerThreads">workerThreads</a></span> - Variable in class org.apache.hadoop.hbase.procedure2.<a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor</a></dt>
-<dd> </dd>
+<dd>
+<div class="block">Created in the <a href="org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>ProcedureExecutor.start(int, boolean)</code></a> method.</div>
+</dd>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.html#workerThreads">workerThreads</a></span> - Variable in class org.apache.hadoop.hbase.replication.regionserver.<a href="org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.html" title="class in org.apache.hadoop.hbase.replication.regionserver">ReplicationSource</a></dt>
<dd> </dd>
<dt><span class="memberNameLink"><a href="org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.html#workingDir">workingDir</a></span> - Variable in class org.apache.hadoop.hbase.master.snapshot.<a href="org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.html" title="class in org.apache.hadoop.hbase.master.snapshot">TakeSnapshotHandler</a></dt>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html
----------------------------------------------------------------------
diff --git a/devapidocs/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html b/devapidocs/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html
index fa0c083..2e64b09 100644
--- a/devapidocs/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html
+++ b/devapidocs/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html
@@ -1304,7 +1304,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>pendingAssignQueue</h4>
-<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/ArrayList.html?is-external=true" title="class or interface in java.util">ArrayList</a><<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1628">pendingAssignQueue</a></pre>
+<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/ArrayList.html?is-external=true" title="class or interface in java.util">ArrayList</a><<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1629">pendingAssignQueue</a></pre>
</li>
</ul>
<a name="assignQueueLock">
@@ -1313,7 +1313,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>assignQueueLock</h4>
-<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantLock.html?is-external=true" title="class or interface in java.util.concurrent.locks">ReentrantLock</a> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1629">assignQueueLock</a></pre>
+<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/ReentrantLock.html?is-external=true" title="class or interface in java.util.concurrent.locks">ReentrantLock</a> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1630">assignQueueLock</a></pre>
</li>
</ul>
<a name="assignQueueFullCond">
@@ -1322,7 +1322,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockListLast">
<li class="blockList">
<h4>assignQueueFullCond</h4>
-<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/Condition.html?is-external=true" title="class or interface in java.util.concurrent.locks">Condition</a> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1630">assignQueueFullCond</a></pre>
+<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/locks/Condition.html?is-external=true" title="class or interface in java.util.concurrent.locks">Condition</a> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1631">assignQueueFullCond</a></pre>
</li>
</ul>
</li>
@@ -2236,7 +2236,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>processofflineServersWithOnlineRegions</h4>
-<pre>private boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1298">processofflineServersWithOnlineRegions</a>()</pre>
+<pre>private boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1299">processofflineServersWithOnlineRegions</a>()</pre>
<div class="block">Look at what is in meta and the list of servers that have checked in and make reconciliation.
We cannot tell definitively the difference between a clean shutdown and a cluster that has
been crashed down. At this stage of a Master startup, they look the same: they have the
@@ -2266,7 +2266,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>isFailoverCleanupDone</h4>
-<pre>public boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1352">isFailoverCleanupDone</a>()</pre>
+<pre>public boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1353">isFailoverCleanupDone</a>()</pre>
<div class="block">Used by ServerCrashProcedure to make sure AssignmentManager has completed
the failover cleanup before re-assigning regions of dead servers. So that
when re-assignment happens, AssignmentManager has proper region states.</div>
@@ -2278,7 +2278,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>setFailoverCleanupDone</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1361">setFailoverCleanupDone</a>(boolean b)</pre>
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1362">setFailoverCleanupDone</a>(boolean b)</pre>
<div class="block">Used by ServerCrashProcedure tests verify the ability to suspend the
execution of the ServerCrashProcedure.</div>
</li>
@@ -2289,7 +2289,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>getFailoverCleanupEvent</h4>
-<pre>public <a href="../../../../../../org/apache/hadoop/hbase/procedure2/ProcedureEvent.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureEvent</a> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1366">getFailoverCleanupEvent</a>()</pre>
+<pre>public <a href="../../../../../../org/apache/hadoop/hbase/procedure2/ProcedureEvent.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureEvent</a> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1367">getFailoverCleanupEvent</a>()</pre>
</li>
</ul>
<a name="checkFailoverCleanupCompleted-org.apache.hadoop.hbase.client.RegionInfo-">
@@ -2298,7 +2298,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>checkFailoverCleanupCompleted</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1376">checkFailoverCleanupCompleted</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> hri)
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1377">checkFailoverCleanupCompleted</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> hri)
throws <a href="../../../../../../org/apache/hadoop/hbase/PleaseHoldException.html" title="class in org.apache.hadoop.hbase">PleaseHoldException</a></pre>
<div class="block">Used to check if the failover cleanup is done.
if not we throw PleaseHoldException since we are rebuilding the RegionStates</div>
@@ -2316,7 +2316,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>getNumRegionsOpened</h4>
-<pre>public int <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1394">getNumRegionsOpened</a>()</pre>
+<pre>public int <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1395">getNumRegionsOpened</a>()</pre>
</li>
</ul>
<a name="submitServerCrash-org.apache.hadoop.hbase.ServerName-boolean-">
@@ -2325,7 +2325,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>submitServerCrash</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1399">submitServerCrash</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName,
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1400">submitServerCrash</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName,
boolean shouldSplitWal)</pre>
</li>
</ul>
@@ -2335,7 +2335,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>offlineRegion</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1408">offlineRegion</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> regionInfo)</pre>
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1409">offlineRegion</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> regionInfo)</pre>
</li>
</ul>
<a name="onlineRegion-org.apache.hadoop.hbase.client.RegionInfo-org.apache.hadoop.hbase.ServerName-">
@@ -2344,7 +2344,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>onlineRegion</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1414">onlineRegion</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> regionInfo,
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1415">onlineRegion</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> regionInfo,
<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
</li>
</ul>
@@ -2354,7 +2354,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>getSnapShotOfAssignment</h4>
-<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Map.html?is-external=true" title="class or interface in java.util">Map</a><<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a>,<a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1418">getSnapShotOfAssignment</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Collection.html?is-external=true" title="class or interface in java.util">Collection</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">R
egionInfo</a>> regions)</pre>
+<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Map.html?is-external=true" title="class or interface in java.util">Map</a><<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a>,<a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1419">getSnapShotOfAssignment</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Collection.html?is-external=true" title="class or interface in java.util">Collection</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">R
egionInfo</a>> regions)</pre>
</li>
</ul>
<a name="getReopenStatus-org.apache.hadoop.hbase.TableName-">
@@ -2363,7 +2363,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>getReopenStatus</h4>
-<pre>public <a href="../../../../../../org/apache/hadoop/hbase/util/Pair.html" title="class in org.apache.hadoop.hbase.util">Pair</a><<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Integer.html?is-external=true" title="class or interface in java.lang">Integer</a>,<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Integer.html?is-external=true" title="class or interface in java.lang">Integer</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1433">getReopenStatus</a>(<a href="../../../../../../org/apache/hadoop/hbase/TableName.html" title="class in org.apache.hadoop.hbase">TableName</a> tableName)</pre>
+<pre>public <a href="../../../../../../org/apache/hadoop/hbase/util/Pair.html" title="class in org.apache.hadoop.hbase.util">Pair</a><<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Integer.html?is-external=true" title="class or interface in java.lang">Integer</a>,<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Integer.html?is-external=true" title="class or interface in java.lang">Integer</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1434">getReopenStatus</a>(<a href="../../../../../../org/apache/hadoop/hbase/TableName.html" title="class in org.apache.hadoop.hbase">TableName</a> tableName)</pre>
<div class="block">Used by the client (via master) to identify if all regions have the schema updates</div>
<dl>
<dt><span class="paramLabel">Parameters:</span></dt>
@@ -2381,7 +2381,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>addRegionInTransition</h4>
-<pre>protected boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1447">addRegionInTransition</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode,
+<pre>protected boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1448">addRegionInTransition</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode,
<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionTransitionProcedure</a> procedure)</pre>
</li>
</ul>
@@ -2391,7 +2391,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>removeRegionInTransition</h4>
-<pre>protected void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1452">removeRegionInTransition</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode,
+<pre>protected void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1453">removeRegionInTransition</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode,
<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionTransitionProcedure</a> procedure)</pre>
</li>
</ul>
@@ -2401,7 +2401,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>hasRegionsInTransition</h4>
-<pre>public boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1457">hasRegionsInTransition</a>()</pre>
+<pre>public boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1458">hasRegionsInTransition</a>()</pre>
</li>
</ul>
<a name="getRegionsInTransition--">
@@ -2410,7 +2410,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>getRegionsInTransition</h4>
-<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1461">getRegionsInTransition</a>()</pre>
+<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1462">getRegionsInTransition</a>()</pre>
</li>
</ul>
<a name="getAssignedRegions--">
@@ -2419,7 +2419,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>getAssignedRegions</h4>
-<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1465">getAssignedRegions</a>()</pre>
+<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1466">getAssignedRegions</a>()</pre>
</li>
</ul>
<a name="getRegionInfo-byte:A-">
@@ -2428,7 +2428,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>getRegionInfo</h4>
-<pre>public <a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1469">getRegionInfo</a>(byte[] regionName)</pre>
+<pre>public <a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1470">getRegionInfo</a>(byte[] regionName)</pre>
</li>
</ul>
<a name="sendRegionOpenedNotification-org.apache.hadoop.hbase.client.RegionInfo-org.apache.hadoop.hbase.ServerName-">
@@ -2437,7 +2437,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>sendRegionOpenedNotification</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1477">sendRegionOpenedNotification</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> regionInfo,
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1478">sendRegionOpenedNotification</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> regionInfo,
<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
</li>
</ul>
@@ -2447,7 +2447,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>sendRegionClosedNotification</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1487">sendRegionClosedNotification</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> regionInfo)</pre>
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1488">sendRegionClosedNotification</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> regionInfo)</pre>
</li>
</ul>
<a name="markRegionAsOpening-org.apache.hadoop.hbase.master.assignment.RegionStates.RegionStateNode-">
@@ -2456,7 +2456,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>markRegionAsOpening</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1496">markRegionAsOpening</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1497">markRegionAsOpening</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a></pre>
<dl>
<dt><span class="throwsLabel">Throws:</span></dt>
@@ -2470,7 +2470,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>undoRegionAsOpening</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1507">undoRegionAsOpening</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)</pre>
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1508">undoRegionAsOpening</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)</pre>
</li>
</ul>
<a name="markRegionAsOpened-org.apache.hadoop.hbase.master.assignment.RegionStates.RegionStateNode-">
@@ -2479,7 +2479,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>markRegionAsOpened</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1521">markRegionAsOpened</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1522">markRegionAsOpened</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a></pre>
<dl>
<dt><span class="throwsLabel">Throws:</span></dt>
@@ -2493,7 +2493,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>markRegionAsClosing</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1540">markRegionAsClosing</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1541">markRegionAsClosing</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a></pre>
<dl>
<dt><span class="throwsLabel">Throws:</span></dt>
@@ -2507,7 +2507,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>undoRegionAsClosing</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1556">undoRegionAsClosing</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)</pre>
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1557">undoRegionAsClosing</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)</pre>
</li>
</ul>
<a name="markRegionAsClosed-org.apache.hadoop.hbase.master.assignment.RegionStates.RegionStateNode-">
@@ -2516,7 +2516,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>markRegionAsClosed</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1561">markRegionAsClosed</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1562">markRegionAsClosed</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a></pre>
<dl>
<dt><span class="throwsLabel">Throws:</span></dt>
@@ -2530,7 +2530,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>markRegionAsSplit</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1573">markRegionAsSplit</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> parent,
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1574">markRegionAsSplit</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> parent,
<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName,
<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> daughterA,
<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> daughterB)
@@ -2547,7 +2547,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>markRegionAsMerged</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1603">markRegionAsMerged</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> child,
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1604">markRegionAsMerged</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> child,
<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName,
<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> mother,
<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> father)
@@ -2570,7 +2570,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>shouldAssignFavoredNodes</h4>
-<pre>private boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1620">shouldAssignFavoredNodes</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> region)</pre>
+<pre>private boolean <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1621">shouldAssignFavoredNodes</a>(<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a> region)</pre>
</li>
</ul>
<a name="queueAssign-org.apache.hadoop.hbase.master.assignment.RegionStates.RegionStateNode-">
@@ -2579,7 +2579,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>queueAssign</h4>
-<pre>protected void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1637">queueAssign</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)</pre>
+<pre>protected void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1638">queueAssign</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a> regionNode)</pre>
<div class="block">Add the assign operation to the assignment queue.
The pending assignment operation will be processed,
and each region will be assigned by a server using the balancer.</div>
@@ -2591,7 +2591,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>startAssignmentThread</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1654">startAssignmentThread</a>()</pre>
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1655">startAssignmentThread</a>()</pre>
</li>
</ul>
<a name="stopAssignmentThread--">
@@ -2600,7 +2600,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>stopAssignmentThread</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1672">stopAssignmentThread</a>()</pre>
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1673">stopAssignmentThread</a>()</pre>
</li>
</ul>
<a name="assignQueueSignal--">
@@ -2609,7 +2609,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>assignQueueSignal</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1685">assignQueueSignal</a>()</pre>
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1686">assignQueueSignal</a>()</pre>
</li>
</ul>
<a name="waitOnAssignQueue--">
@@ -2618,7 +2618,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>waitOnAssignQueue</h4>
-<pre>private <a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html?is-external=true" title="class or interface in java.util">HashMap</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>,<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1695">waitOnAssignQueue</a>()</pre>
+<pre>private <a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html?is-external=true" title="class or interface in java.util">HashMap</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>,<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1696">waitOnAssignQueue</a>()</pre>
</li>
</ul>
<a name="processAssignQueue--">
@@ -2627,7 +2627,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>processAssignQueue</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1720">processAssignQueue</a>()</pre>
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1721">processAssignQueue</a>()</pre>
</li>
</ul>
<a name="processAssignmentPlans-java.util.HashMap-java.util.HashMap-java.util.List-java.util.List-">
@@ -2636,7 +2636,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>processAssignmentPlans</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1782">processAssignmentPlans</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html?is-external=true" title="class or interface in java.util">HashMap</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>,<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> regions,
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1783">processAssignmentPlans</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html?is-external=true" title="class or interface in java.util">HashMap</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>,<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> regions,
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html?is-external=true" title="class or interface in java.util">HashMap</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>,<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a>> retainMap,
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>> hris,
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a>> servers)</pre>
@@ -2648,7 +2648,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>acceptPlan</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1820">acceptPlan</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html?is-external=true" title="class or interface in java.util">HashMap</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>,<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> regions,
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1821">acceptPlan</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html?is-external=true" title="class or interface in java.util">HashMap</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>,<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> regions,
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Map.html?is-external=true" title="class or interface in java.util">Map</a><<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a>,<a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>>> plan)
throws <a href="../../../../../../org/apache/hadoop/hbase/HBaseIOException.html" title="class in org.apache.hadoop.hbase">HBaseIOException</a></pre>
<dl>
@@ -2663,7 +2663,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>addToPendingAssignment</h4>
-<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1849">addToPendingAssignment</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html?is-external=true" title="class or interface in java.util">HashMap</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>,<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> regions,
+<pre>private void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1850">addToPendingAssignment</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html?is-external=true" title="class or interface in java.util">HashMap</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>,<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.RegionStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.RegionStateNode</a>> regions,
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Collection.html?is-external=true" title="class or interface in java.util">Collection</a><<a href="../../../../../../org/apache/hadoop/hbase/client/RegionInfo.html" title="interface in org.apache.hadoop.hbase.client">RegionInfo</a>> pendingRegions)</pre>
</li>
</ul>
@@ -2673,7 +2673,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>getExcludedServersForSystemTable</h4>
-<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1865">getExcludedServersForSystemTable</a>()</pre>
+<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a>> <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1866">getExcludedServersForSystemTable</a>()</pre>
<div class="block">Get a list of servers that this region cannot be assigned to.
For system tables, we must assign them to a server with highest version.</div>
</li>
@@ -2684,7 +2684,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>serverAdded</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1888">serverAdded</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1889">serverAdded</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
<div class="block"><span class="descfrmTypeLabel">Description copied from interface: <code><a href="../../../../../../org/apache/hadoop/hbase/master/ServerListener.html#serverAdded-org.apache.hadoop.hbase.ServerName-">ServerListener</a></code></span></div>
<div class="block">The server has joined the cluster.</div>
<dl>
@@ -2701,7 +2701,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>serverRemoved</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1892">serverRemoved</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1893">serverRemoved</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
<div class="block"><span class="descfrmTypeLabel">Description copied from interface: <code><a href="../../../../../../org/apache/hadoop/hbase/master/ServerListener.html#serverRemoved-org.apache.hadoop.hbase.ServerName-">ServerListener</a></code></span></div>
<div class="block">The server was removed from the cluster.</div>
<dl>
@@ -2718,7 +2718,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>getServerVersion</h4>
-<pre>public int <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1900">getServerVersion</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
+<pre>public int <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1901">getServerVersion</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
</li>
</ul>
<a name="killRegionServer-org.apache.hadoop.hbase.ServerName-">
@@ -2727,7 +2727,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>killRegionServer</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1905">killRegionServer</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1906">killRegionServer</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
</li>
</ul>
<a name="killRegionServer-org.apache.hadoop.hbase.master.assignment.RegionStates.ServerStateNode-">
@@ -2736,7 +2736,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockList">
<li class="blockList">
<h4>killRegionServer</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1910">killRegionServer</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.ServerStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.ServerStateNode</a> serverNode)</pre>
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1911">killRegionServer</a>(<a href="../../../../../../org/apache/hadoop/hbase/master/assignment/RegionStates.ServerStateNode.html" title="class in org.apache.hadoop.hbase.master.assignment">RegionStates.ServerStateNode</a> serverNode)</pre>
</li>
</ul>
<a name="handleMetaRITOnCrashedServer-org.apache.hadoop.hbase.ServerName-">
@@ -2745,7 +2745,7 @@ implements <a href="../../../../../../org/apache/hadoop/hbase/master/ServerListe
<ul class="blockListLast">
<li class="blockList">
<h4>handleMetaRITOnCrashedServer</h4>
-<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1925">handleMetaRITOnCrashedServer</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
+<pre>public void <a href="../../../../../../src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html#line.1926">handleMetaRITOnCrashedServer</a>(<a href="../../../../../../org/apache/hadoop/hbase/ServerName.html" title="class in org.apache.hadoop.hbase">ServerName</a> serverName)</pre>
<div class="block">Handle RIT of meta region against crashed server.
Only used when ServerCrashProcedure is not enabled.
See handleRIT in ServerCrashProcedure for similar function.</div>
[03/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html
index 97aa79c..cc787e3 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html
@@ -272,1623 +272,1642 @@
<span class="sourceLineNo">264</span> private final CopyOnWriteArrayList<ProcedureExecutorListener> listeners = new CopyOnWriteArrayList<>();<a name="line.264"></a>
<span class="sourceLineNo">265</span><a name="line.265"></a>
<span class="sourceLineNo">266</span> private Configuration conf;<a name="line.266"></a>
-<span class="sourceLineNo">267</span> private ThreadGroup threadGroup;<a name="line.267"></a>
-<span class="sourceLineNo">268</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.268"></a>
-<span class="sourceLineNo">269</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.269"></a>
-<span class="sourceLineNo">270</span> private int corePoolSize;<a name="line.270"></a>
-<span class="sourceLineNo">271</span> private int maxPoolSize;<a name="line.271"></a>
-<span class="sourceLineNo">272</span><a name="line.272"></a>
-<span class="sourceLineNo">273</span> private volatile long keepAliveTime;<a name="line.273"></a>
-<span class="sourceLineNo">274</span><a name="line.274"></a>
-<span class="sourceLineNo">275</span> /**<a name="line.275"></a>
-<span class="sourceLineNo">276</span> * Scheduler/Queue that contains runnable procedures.<a name="line.276"></a>
-<span class="sourceLineNo">277</span> */<a name="line.277"></a>
-<span class="sourceLineNo">278</span> private final ProcedureScheduler scheduler;<a name="line.278"></a>
-<span class="sourceLineNo">279</span><a name="line.279"></a>
-<span class="sourceLineNo">280</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.280"></a>
-<span class="sourceLineNo">281</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.281"></a>
-<span class="sourceLineNo">282</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.282"></a>
-<span class="sourceLineNo">283</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.283"></a>
-<span class="sourceLineNo">284</span> private final TEnvironment environment;<a name="line.284"></a>
-<span class="sourceLineNo">285</span> private final ProcedureStore store;<a name="line.285"></a>
-<span class="sourceLineNo">286</span><a name="line.286"></a>
-<span class="sourceLineNo">287</span> private final boolean checkOwnerSet;<a name="line.287"></a>
-<span class="sourceLineNo">288</span><a name="line.288"></a>
-<span class="sourceLineNo">289</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.289"></a>
-<span class="sourceLineNo">290</span> final ProcedureStore store) {<a name="line.290"></a>
-<span class="sourceLineNo">291</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.291"></a>
-<span class="sourceLineNo">292</span> }<a name="line.292"></a>
-<span class="sourceLineNo">293</span><a name="line.293"></a>
-<span class="sourceLineNo">294</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.294"></a>
-<span class="sourceLineNo">295</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.295"></a>
-<span class="sourceLineNo">296</span> this.environment = environment;<a name="line.296"></a>
-<span class="sourceLineNo">297</span> this.scheduler = scheduler;<a name="line.297"></a>
-<span class="sourceLineNo">298</span> this.store = store;<a name="line.298"></a>
-<span class="sourceLineNo">299</span> this.conf = conf;<a name="line.299"></a>
-<span class="sourceLineNo">300</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.300"></a>
-<span class="sourceLineNo">301</span> refreshConfiguration(conf);<a name="line.301"></a>
-<span class="sourceLineNo">302</span> }<a name="line.302"></a>
-<span class="sourceLineNo">303</span><a name="line.303"></a>
-<span class="sourceLineNo">304</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.304"></a>
-<span class="sourceLineNo">305</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.305"></a>
-<span class="sourceLineNo">306</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.306"></a>
-<span class="sourceLineNo">307</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.307"></a>
-<span class="sourceLineNo">308</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.308"></a>
-<span class="sourceLineNo">309</span><a name="line.309"></a>
-<span class="sourceLineNo">310</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.310"></a>
-<span class="sourceLineNo">311</span> @Override<a name="line.311"></a>
-<span class="sourceLineNo">312</span> public void setMaxProcId(long maxProcId) {<a name="line.312"></a>
-<span class="sourceLineNo">313</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.313"></a>
-<span class="sourceLineNo">314</span> lastProcId.set(maxProcId);<a name="line.314"></a>
-<span class="sourceLineNo">315</span> }<a name="line.315"></a>
-<span class="sourceLineNo">316</span><a name="line.316"></a>
-<span class="sourceLineNo">317</span> @Override<a name="line.317"></a>
-<span class="sourceLineNo">318</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.318"></a>
-<span class="sourceLineNo">319</span> loadProcedures(procIter, abortOnCorruption);<a name="line.319"></a>
-<span class="sourceLineNo">320</span> }<a name="line.320"></a>
-<span class="sourceLineNo">321</span><a name="line.321"></a>
-<span class="sourceLineNo">322</span> @Override<a name="line.322"></a>
-<span class="sourceLineNo">323</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.323"></a>
-<span class="sourceLineNo">324</span> int corruptedCount = 0;<a name="line.324"></a>
-<span class="sourceLineNo">325</span> while (procIter.hasNext()) {<a name="line.325"></a>
-<span class="sourceLineNo">326</span> Procedure<?> proc = procIter.next();<a name="line.326"></a>
-<span class="sourceLineNo">327</span> LOG.error("Corrupt " + proc);<a name="line.327"></a>
-<span class="sourceLineNo">328</span> corruptedCount++;<a name="line.328"></a>
-<span class="sourceLineNo">329</span> }<a name="line.329"></a>
-<span class="sourceLineNo">330</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.330"></a>
-<span class="sourceLineNo">331</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.331"></a>
-<span class="sourceLineNo">332</span> }<a name="line.332"></a>
-<span class="sourceLineNo">333</span> }<a name="line.333"></a>
-<span class="sourceLineNo">334</span> });<a name="line.334"></a>
-<span class="sourceLineNo">335</span> }<a name="line.335"></a>
-<span class="sourceLineNo">336</span><a name="line.336"></a>
-<span class="sourceLineNo">337</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.337"></a>
-<span class="sourceLineNo">338</span> final boolean abortOnCorruption) throws IOException {<a name="line.338"></a>
-<span class="sourceLineNo">339</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.339"></a>
-<span class="sourceLineNo">340</span><a name="line.340"></a>
-<span class="sourceLineNo">341</span> // 1. Build the rollback stack<a name="line.341"></a>
-<span class="sourceLineNo">342</span> int runnablesCount = 0;<a name="line.342"></a>
-<span class="sourceLineNo">343</span> while (procIter.hasNext()) {<a name="line.343"></a>
-<span class="sourceLineNo">344</span> boolean finished = procIter.isNextFinished();<a name="line.344"></a>
-<span class="sourceLineNo">345</span> Procedure proc = procIter.next();<a name="line.345"></a>
-<span class="sourceLineNo">346</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.346"></a>
-<span class="sourceLineNo">347</span> long procId = proc.getProcId();<a name="line.347"></a>
-<span class="sourceLineNo">348</span><a name="line.348"></a>
-<span class="sourceLineNo">349</span> if (finished) {<a name="line.349"></a>
-<span class="sourceLineNo">350</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.350"></a>
-<span class="sourceLineNo">351</span> if (debugEnabled) {<a name="line.351"></a>
-<span class="sourceLineNo">352</span> LOG.debug("Completed " + proc);<a name="line.352"></a>
-<span class="sourceLineNo">353</span> }<a name="line.353"></a>
-<span class="sourceLineNo">354</span> } else {<a name="line.354"></a>
-<span class="sourceLineNo">355</span> if (!proc.hasParent()) {<a name="line.355"></a>
-<span class="sourceLineNo">356</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.356"></a>
-<span class="sourceLineNo">357</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.357"></a>
-<span class="sourceLineNo">358</span> }<a name="line.358"></a>
+<span class="sourceLineNo">267</span><a name="line.267"></a>
+<span class="sourceLineNo">268</span> /**<a name="line.268"></a>
+<span class="sourceLineNo">269</span> * Created in the {@link #start(int, boolean)} method. Destroyed in {@link #join()} (FIX! Doing<a name="line.269"></a>
+<span class="sourceLineNo">270</span> * resource handling rather than observing in a #join is unexpected).<a name="line.270"></a>
+<span class="sourceLineNo">271</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.271"></a>
+<span class="sourceLineNo">272</span> * (Should be ok).<a name="line.272"></a>
+<span class="sourceLineNo">273</span> */<a name="line.273"></a>
+<span class="sourceLineNo">274</span> private ThreadGroup threadGroup;<a name="line.274"></a>
+<span class="sourceLineNo">275</span><a name="line.275"></a>
+<span class="sourceLineNo">276</span> /**<a name="line.276"></a>
+<span class="sourceLineNo">277</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.277"></a>
+<span class="sourceLineNo">278</span> * resource handling rather than observing in a #join is unexpected).<a name="line.278"></a>
+<span class="sourceLineNo">279</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.279"></a>
+<span class="sourceLineNo">280</span> * (Should be ok).<a name="line.280"></a>
+<span class="sourceLineNo">281</span> */<a name="line.281"></a>
+<span class="sourceLineNo">282</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.282"></a>
+<span class="sourceLineNo">283</span><a name="line.283"></a>
+<span class="sourceLineNo">284</span> /**<a name="line.284"></a>
+<span class="sourceLineNo">285</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.285"></a>
+<span class="sourceLineNo">286</span> * resource handling rather than observing in a #join is unexpected).<a name="line.286"></a>
+<span class="sourceLineNo">287</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.287"></a>
+<span class="sourceLineNo">288</span> * (Should be ok).<a name="line.288"></a>
+<span class="sourceLineNo">289</span> */<a name="line.289"></a>
+<span class="sourceLineNo">290</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.290"></a>
+<span class="sourceLineNo">291</span><a name="line.291"></a>
+<span class="sourceLineNo">292</span> private int corePoolSize;<a name="line.292"></a>
+<span class="sourceLineNo">293</span> private int maxPoolSize;<a name="line.293"></a>
+<span class="sourceLineNo">294</span><a name="line.294"></a>
+<span class="sourceLineNo">295</span> private volatile long keepAliveTime;<a name="line.295"></a>
+<span class="sourceLineNo">296</span><a name="line.296"></a>
+<span class="sourceLineNo">297</span> /**<a name="line.297"></a>
+<span class="sourceLineNo">298</span> * Scheduler/Queue that contains runnable procedures.<a name="line.298"></a>
+<span class="sourceLineNo">299</span> */<a name="line.299"></a>
+<span class="sourceLineNo">300</span> private final ProcedureScheduler scheduler;<a name="line.300"></a>
+<span class="sourceLineNo">301</span><a name="line.301"></a>
+<span class="sourceLineNo">302</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.302"></a>
+<span class="sourceLineNo">303</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.303"></a>
+<span class="sourceLineNo">304</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.304"></a>
+<span class="sourceLineNo">305</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.305"></a>
+<span class="sourceLineNo">306</span> private final TEnvironment environment;<a name="line.306"></a>
+<span class="sourceLineNo">307</span> private final ProcedureStore store;<a name="line.307"></a>
+<span class="sourceLineNo">308</span><a name="line.308"></a>
+<span class="sourceLineNo">309</span> private final boolean checkOwnerSet;<a name="line.309"></a>
+<span class="sourceLineNo">310</span><a name="line.310"></a>
+<span class="sourceLineNo">311</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.311"></a>
+<span class="sourceLineNo">312</span> final ProcedureStore store) {<a name="line.312"></a>
+<span class="sourceLineNo">313</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.313"></a>
+<span class="sourceLineNo">314</span> }<a name="line.314"></a>
+<span class="sourceLineNo">315</span><a name="line.315"></a>
+<span class="sourceLineNo">316</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.316"></a>
+<span class="sourceLineNo">317</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.317"></a>
+<span class="sourceLineNo">318</span> this.environment = environment;<a name="line.318"></a>
+<span class="sourceLineNo">319</span> this.scheduler = scheduler;<a name="line.319"></a>
+<span class="sourceLineNo">320</span> this.store = store;<a name="line.320"></a>
+<span class="sourceLineNo">321</span> this.conf = conf;<a name="line.321"></a>
+<span class="sourceLineNo">322</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.322"></a>
+<span class="sourceLineNo">323</span> refreshConfiguration(conf);<a name="line.323"></a>
+<span class="sourceLineNo">324</span><a name="line.324"></a>
+<span class="sourceLineNo">325</span> }<a name="line.325"></a>
+<span class="sourceLineNo">326</span><a name="line.326"></a>
+<span class="sourceLineNo">327</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.327"></a>
+<span class="sourceLineNo">328</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.328"></a>
+<span class="sourceLineNo">329</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.329"></a>
+<span class="sourceLineNo">330</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.330"></a>
+<span class="sourceLineNo">331</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.331"></a>
+<span class="sourceLineNo">332</span><a name="line.332"></a>
+<span class="sourceLineNo">333</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.333"></a>
+<span class="sourceLineNo">334</span> @Override<a name="line.334"></a>
+<span class="sourceLineNo">335</span> public void setMaxProcId(long maxProcId) {<a name="line.335"></a>
+<span class="sourceLineNo">336</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.336"></a>
+<span class="sourceLineNo">337</span> lastProcId.set(maxProcId);<a name="line.337"></a>
+<span class="sourceLineNo">338</span> }<a name="line.338"></a>
+<span class="sourceLineNo">339</span><a name="line.339"></a>
+<span class="sourceLineNo">340</span> @Override<a name="line.340"></a>
+<span class="sourceLineNo">341</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.341"></a>
+<span class="sourceLineNo">342</span> loadProcedures(procIter, abortOnCorruption);<a name="line.342"></a>
+<span class="sourceLineNo">343</span> }<a name="line.343"></a>
+<span class="sourceLineNo">344</span><a name="line.344"></a>
+<span class="sourceLineNo">345</span> @Override<a name="line.345"></a>
+<span class="sourceLineNo">346</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.346"></a>
+<span class="sourceLineNo">347</span> int corruptedCount = 0;<a name="line.347"></a>
+<span class="sourceLineNo">348</span> while (procIter.hasNext()) {<a name="line.348"></a>
+<span class="sourceLineNo">349</span> Procedure<?> proc = procIter.next();<a name="line.349"></a>
+<span class="sourceLineNo">350</span> LOG.error("Corrupt " + proc);<a name="line.350"></a>
+<span class="sourceLineNo">351</span> corruptedCount++;<a name="line.351"></a>
+<span class="sourceLineNo">352</span> }<a name="line.352"></a>
+<span class="sourceLineNo">353</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.353"></a>
+<span class="sourceLineNo">354</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.354"></a>
+<span class="sourceLineNo">355</span> }<a name="line.355"></a>
+<span class="sourceLineNo">356</span> }<a name="line.356"></a>
+<span class="sourceLineNo">357</span> });<a name="line.357"></a>
+<span class="sourceLineNo">358</span> }<a name="line.358"></a>
<span class="sourceLineNo">359</span><a name="line.359"></a>
-<span class="sourceLineNo">360</span> // add the procedure to the map<a name="line.360"></a>
-<span class="sourceLineNo">361</span> proc.beforeReplay(getEnvironment());<a name="line.361"></a>
-<span class="sourceLineNo">362</span> procedures.put(proc.getProcId(), proc);<a name="line.362"></a>
+<span class="sourceLineNo">360</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.360"></a>
+<span class="sourceLineNo">361</span> final boolean abortOnCorruption) throws IOException {<a name="line.361"></a>
+<span class="sourceLineNo">362</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.362"></a>
<span class="sourceLineNo">363</span><a name="line.363"></a>
-<span class="sourceLineNo">364</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.364"></a>
-<span class="sourceLineNo">365</span> runnablesCount++;<a name="line.365"></a>
-<span class="sourceLineNo">366</span> }<a name="line.366"></a>
-<span class="sourceLineNo">367</span> }<a name="line.367"></a>
-<span class="sourceLineNo">368</span><a name="line.368"></a>
-<span class="sourceLineNo">369</span> // add the nonce to the map<a name="line.369"></a>
-<span class="sourceLineNo">370</span> if (nonceKey != null) {<a name="line.370"></a>
-<span class="sourceLineNo">371</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.371"></a>
-<span class="sourceLineNo">372</span> }<a name="line.372"></a>
-<span class="sourceLineNo">373</span> }<a name="line.373"></a>
-<span class="sourceLineNo">374</span><a name="line.374"></a>
-<span class="sourceLineNo">375</span> // 2. Initialize the stacks<a name="line.375"></a>
-<span class="sourceLineNo">376</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.376"></a>
-<span class="sourceLineNo">377</span> HashSet<Procedure> waitingSet = null;<a name="line.377"></a>
-<span class="sourceLineNo">378</span> procIter.reset();<a name="line.378"></a>
-<span class="sourceLineNo">379</span> while (procIter.hasNext()) {<a name="line.379"></a>
-<span class="sourceLineNo">380</span> if (procIter.isNextFinished()) {<a name="line.380"></a>
-<span class="sourceLineNo">381</span> procIter.skipNext();<a name="line.381"></a>
-<span class="sourceLineNo">382</span> continue;<a name="line.382"></a>
-<span class="sourceLineNo">383</span> }<a name="line.383"></a>
-<span class="sourceLineNo">384</span><a name="line.384"></a>
-<span class="sourceLineNo">385</span> Procedure proc = procIter.next();<a name="line.385"></a>
-<span class="sourceLineNo">386</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.386"></a>
-<span class="sourceLineNo">387</span><a name="line.387"></a>
-<span class="sourceLineNo">388</span> if (debugEnabled) {<a name="line.388"></a>
-<span class="sourceLineNo">389</span> LOG.debug(String.format("Loading %s", proc));<a name="line.389"></a>
+<span class="sourceLineNo">364</span> // 1. Build the rollback stack<a name="line.364"></a>
+<span class="sourceLineNo">365</span> int runnablesCount = 0;<a name="line.365"></a>
+<span class="sourceLineNo">366</span> while (procIter.hasNext()) {<a name="line.366"></a>
+<span class="sourceLineNo">367</span> boolean finished = procIter.isNextFinished();<a name="line.367"></a>
+<span class="sourceLineNo">368</span> Procedure proc = procIter.next();<a name="line.368"></a>
+<span class="sourceLineNo">369</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.369"></a>
+<span class="sourceLineNo">370</span> long procId = proc.getProcId();<a name="line.370"></a>
+<span class="sourceLineNo">371</span><a name="line.371"></a>
+<span class="sourceLineNo">372</span> if (finished) {<a name="line.372"></a>
+<span class="sourceLineNo">373</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.373"></a>
+<span class="sourceLineNo">374</span> if (debugEnabled) {<a name="line.374"></a>
+<span class="sourceLineNo">375</span> LOG.debug("Completed " + proc);<a name="line.375"></a>
+<span class="sourceLineNo">376</span> }<a name="line.376"></a>
+<span class="sourceLineNo">377</span> } else {<a name="line.377"></a>
+<span class="sourceLineNo">378</span> if (!proc.hasParent()) {<a name="line.378"></a>
+<span class="sourceLineNo">379</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.379"></a>
+<span class="sourceLineNo">380</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.380"></a>
+<span class="sourceLineNo">381</span> }<a name="line.381"></a>
+<span class="sourceLineNo">382</span><a name="line.382"></a>
+<span class="sourceLineNo">383</span> // add the procedure to the map<a name="line.383"></a>
+<span class="sourceLineNo">384</span> proc.beforeReplay(getEnvironment());<a name="line.384"></a>
+<span class="sourceLineNo">385</span> procedures.put(proc.getProcId(), proc);<a name="line.385"></a>
+<span class="sourceLineNo">386</span><a name="line.386"></a>
+<span class="sourceLineNo">387</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.387"></a>
+<span class="sourceLineNo">388</span> runnablesCount++;<a name="line.388"></a>
+<span class="sourceLineNo">389</span> }<a name="line.389"></a>
<span class="sourceLineNo">390</span> }<a name="line.390"></a>
<span class="sourceLineNo">391</span><a name="line.391"></a>
-<span class="sourceLineNo">392</span> Long rootProcId = getRootProcedureId(proc);<a name="line.392"></a>
-<span class="sourceLineNo">393</span> if (rootProcId == null) {<a name="line.393"></a>
-<span class="sourceLineNo">394</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.394"></a>
-<span class="sourceLineNo">395</span> scheduler.addBack(proc);<a name="line.395"></a>
-<span class="sourceLineNo">396</span> continue;<a name="line.396"></a>
-<span class="sourceLineNo">397</span> }<a name="line.397"></a>
-<span class="sourceLineNo">398</span><a name="line.398"></a>
-<span class="sourceLineNo">399</span> if (proc.hasParent()) {<a name="line.399"></a>
-<span class="sourceLineNo">400</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.400"></a>
-<span class="sourceLineNo">401</span> // corrupted procedures are handled later at step 3<a name="line.401"></a>
-<span class="sourceLineNo">402</span> if (parent != null && !proc.isFinished()) {<a name="line.402"></a>
-<span class="sourceLineNo">403</span> parent.incChildrenLatch();<a name="line.403"></a>
-<span class="sourceLineNo">404</span> }<a name="line.404"></a>
-<span class="sourceLineNo">405</span> }<a name="line.405"></a>
-<span class="sourceLineNo">406</span><a name="line.406"></a>
-<span class="sourceLineNo">407</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.407"></a>
-<span class="sourceLineNo">408</span> procStack.loadStack(proc);<a name="line.408"></a>
-<span class="sourceLineNo">409</span><a name="line.409"></a>
-<span class="sourceLineNo">410</span> proc.setRootProcId(rootProcId);<a name="line.410"></a>
-<span class="sourceLineNo">411</span> switch (proc.getState()) {<a name="line.411"></a>
-<span class="sourceLineNo">412</span> case RUNNABLE:<a name="line.412"></a>
-<span class="sourceLineNo">413</span> runnableList.add(proc);<a name="line.413"></a>
-<span class="sourceLineNo">414</span> break;<a name="line.414"></a>
-<span class="sourceLineNo">415</span> case WAITING:<a name="line.415"></a>
-<span class="sourceLineNo">416</span> if (!proc.hasChildren()) {<a name="line.416"></a>
-<span class="sourceLineNo">417</span> runnableList.add(proc);<a name="line.417"></a>
-<span class="sourceLineNo">418</span> }<a name="line.418"></a>
-<span class="sourceLineNo">419</span> break;<a name="line.419"></a>
-<span class="sourceLineNo">420</span> case WAITING_TIMEOUT:<a name="line.420"></a>
-<span class="sourceLineNo">421</span> if (waitingSet == null) {<a name="line.421"></a>
-<span class="sourceLineNo">422</span> waitingSet = new HashSet<>();<a name="line.422"></a>
-<span class="sourceLineNo">423</span> }<a name="line.423"></a>
-<span class="sourceLineNo">424</span> waitingSet.add(proc);<a name="line.424"></a>
-<span class="sourceLineNo">425</span> break;<a name="line.425"></a>
-<span class="sourceLineNo">426</span> case FAILED:<a name="line.426"></a>
-<span class="sourceLineNo">427</span> // add the proc to the scheduler to perform the rollback<a name="line.427"></a>
-<span class="sourceLineNo">428</span> scheduler.addBack(proc);<a name="line.428"></a>
-<span class="sourceLineNo">429</span> break;<a name="line.429"></a>
-<span class="sourceLineNo">430</span> case ROLLEDBACK:<a name="line.430"></a>
-<span class="sourceLineNo">431</span> case INITIALIZING:<a name="line.431"></a>
-<span class="sourceLineNo">432</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.432"></a>
-<span class="sourceLineNo">433</span> LOG.error(msg);<a name="line.433"></a>
-<span class="sourceLineNo">434</span> throw new UnsupportedOperationException(msg);<a name="line.434"></a>
-<span class="sourceLineNo">435</span> default:<a name="line.435"></a>
-<span class="sourceLineNo">436</span> break;<a name="line.436"></a>
-<span class="sourceLineNo">437</span> }<a name="line.437"></a>
-<span class="sourceLineNo">438</span> }<a name="line.438"></a>
-<span class="sourceLineNo">439</span><a name="line.439"></a>
-<span class="sourceLineNo">440</span> // 3. Validate the stacks<a name="line.440"></a>
-<span class="sourceLineNo">441</span> int corruptedCount = 0;<a name="line.441"></a>
-<span class="sourceLineNo">442</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.442"></a>
-<span class="sourceLineNo">443</span> while (itStack.hasNext()) {<a name="line.443"></a>
-<span class="sourceLineNo">444</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.444"></a>
-<span class="sourceLineNo">445</span> RootProcedureState procStack = entry.getValue();<a name="line.445"></a>
-<span class="sourceLineNo">446</span> if (procStack.isValid()) continue;<a name="line.446"></a>
-<span class="sourceLineNo">447</span><a name="line.447"></a>
-<span class="sourceLineNo">448</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.448"></a>
-<span class="sourceLineNo">449</span> LOG.error("Corrupted " + proc);<a name="line.449"></a>
-<span class="sourceLineNo">450</span> procedures.remove(proc.getProcId());<a name="line.450"></a>
-<span class="sourceLineNo">451</span> runnableList.remove(proc);<a name="line.451"></a>
-<span class="sourceLineNo">452</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.452"></a>
-<span class="sourceLineNo">453</span> corruptedCount++;<a name="line.453"></a>
-<span class="sourceLineNo">454</span> }<a name="line.454"></a>
-<span class="sourceLineNo">455</span> itStack.remove();<a name="line.455"></a>
-<span class="sourceLineNo">456</span> }<a name="line.456"></a>
-<span class="sourceLineNo">457</span><a name="line.457"></a>
-<span class="sourceLineNo">458</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.458"></a>
-<span class="sourceLineNo">459</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.459"></a>
-<span class="sourceLineNo">460</span> }<a name="line.460"></a>
-<span class="sourceLineNo">461</span><a name="line.461"></a>
-<span class="sourceLineNo">462</span> // 4. Push the procedures to the timeout executor<a name="line.462"></a>
-<span class="sourceLineNo">463</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.463"></a>
-<span class="sourceLineNo">464</span> for (Procedure proc: waitingSet) {<a name="line.464"></a>
-<span class="sourceLineNo">465</span> proc.afterReplay(getEnvironment());<a name="line.465"></a>
-<span class="sourceLineNo">466</span> timeoutExecutor.add(proc);<a name="line.466"></a>
-<span class="sourceLineNo">467</span> }<a name="line.467"></a>
-<span class="sourceLineNo">468</span> }<a name="line.468"></a>
-<span class="sourceLineNo">469</span><a name="line.469"></a>
-<span class="sourceLineNo">470</span> // 5. Push the procedure to the scheduler<a name="line.470"></a>
-<span class="sourceLineNo">471</span> if (!runnableList.isEmpty()) {<a name="line.471"></a>
-<span class="sourceLineNo">472</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.472"></a>
-<span class="sourceLineNo">473</span> // some procedure may be started way before this stuff.<a name="line.473"></a>
-<span class="sourceLineNo">474</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.474"></a>
-<span class="sourceLineNo">475</span> Procedure proc = runnableList.get(i);<a name="line.475"></a>
-<span class="sourceLineNo">476</span> proc.afterReplay(getEnvironment());<a name="line.476"></a>
-<span class="sourceLineNo">477</span> if (!proc.hasParent()) {<a name="line.477"></a>
-<span class="sourceLineNo">478</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.478"></a>
-<span class="sourceLineNo">479</span> }<a name="line.479"></a>
-<span class="sourceLineNo">480</span> if (proc.wasExecuted()) {<a name="line.480"></a>
-<span class="sourceLineNo">481</span> scheduler.addFront(proc);<a name="line.481"></a>
-<span class="sourceLineNo">482</span> } else {<a name="line.482"></a>
-<span class="sourceLineNo">483</span> // if it was not in execution, it can wait.<a name="line.483"></a>
-<span class="sourceLineNo">484</span> scheduler.addBack(proc);<a name="line.484"></a>
-<span class="sourceLineNo">485</span> }<a name="line.485"></a>
-<span class="sourceLineNo">486</span> }<a name="line.486"></a>
-<span class="sourceLineNo">487</span> }<a name="line.487"></a>
-<span class="sourceLineNo">488</span> }<a name="line.488"></a>
-<span class="sourceLineNo">489</span><a name="line.489"></a>
-<span class="sourceLineNo">490</span> /**<a name="line.490"></a>
-<span class="sourceLineNo">491</span> * Start the procedure executor.<a name="line.491"></a>
-<span class="sourceLineNo">492</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.492"></a>
-<span class="sourceLineNo">493</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.493"></a>
-<span class="sourceLineNo">494</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.494"></a>
-<span class="sourceLineNo">495</span> *<a name="line.495"></a>
-<span class="sourceLineNo">496</span> * @param numThreads number of threads available for procedure execution.<a name="line.496"></a>
-<span class="sourceLineNo">497</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.497"></a>
-<span class="sourceLineNo">498</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.498"></a>
-<span class="sourceLineNo">499</span> */<a name="line.499"></a>
-<span class="sourceLineNo">500</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.500"></a>
-<span class="sourceLineNo">501</span> if (!running.compareAndSet(false, true)) {<a name="line.501"></a>
-<span class="sourceLineNo">502</span> LOG.warn("Already running");<a name="line.502"></a>
-<span class="sourceLineNo">503</span> return;<a name="line.503"></a>
-<span class="sourceLineNo">504</span> }<a name="line.504"></a>
-<span class="sourceLineNo">505</span><a name="line.505"></a>
-<span class="sourceLineNo">506</span> // We have numThreads executor + one timer thread used for timing out<a name="line.506"></a>
-<span class="sourceLineNo">507</span> // procedures and triggering periodic procedures.<a name="line.507"></a>
-<span class="sourceLineNo">508</span> this.corePoolSize = numThreads;<a name="line.508"></a>
-<span class="sourceLineNo">509</span> this.maxPoolSize = 10 * numThreads;<a name="line.509"></a>
-<span class="sourceLineNo">510</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.510"></a>
-<span class="sourceLineNo">511</span> corePoolSize, maxPoolSize);<a name="line.511"></a>
+<span class="sourceLineNo">392</span> // add the nonce to the map<a name="line.392"></a>
+<span class="sourceLineNo">393</span> if (nonceKey != null) {<a name="line.393"></a>
+<span class="sourceLineNo">394</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.394"></a>
+<span class="sourceLineNo">395</span> }<a name="line.395"></a>
+<span class="sourceLineNo">396</span> }<a name="line.396"></a>
+<span class="sourceLineNo">397</span><a name="line.397"></a>
+<span class="sourceLineNo">398</span> // 2. Initialize the stacks<a name="line.398"></a>
+<span class="sourceLineNo">399</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.399"></a>
+<span class="sourceLineNo">400</span> HashSet<Procedure> waitingSet = null;<a name="line.400"></a>
+<span class="sourceLineNo">401</span> procIter.reset();<a name="line.401"></a>
+<span class="sourceLineNo">402</span> while (procIter.hasNext()) {<a name="line.402"></a>
+<span class="sourceLineNo">403</span> if (procIter.isNextFinished()) {<a name="line.403"></a>
+<span class="sourceLineNo">404</span> procIter.skipNext();<a name="line.404"></a>
+<span class="sourceLineNo">405</span> continue;<a name="line.405"></a>
+<span class="sourceLineNo">406</span> }<a name="line.406"></a>
+<span class="sourceLineNo">407</span><a name="line.407"></a>
+<span class="sourceLineNo">408</span> Procedure proc = procIter.next();<a name="line.408"></a>
+<span class="sourceLineNo">409</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.409"></a>
+<span class="sourceLineNo">410</span><a name="line.410"></a>
+<span class="sourceLineNo">411</span> if (debugEnabled) {<a name="line.411"></a>
+<span class="sourceLineNo">412</span> LOG.debug(String.format("Loading %s", proc));<a name="line.412"></a>
+<span class="sourceLineNo">413</span> }<a name="line.413"></a>
+<span class="sourceLineNo">414</span><a name="line.414"></a>
+<span class="sourceLineNo">415</span> Long rootProcId = getRootProcedureId(proc);<a name="line.415"></a>
+<span class="sourceLineNo">416</span> if (rootProcId == null) {<a name="line.416"></a>
+<span class="sourceLineNo">417</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.417"></a>
+<span class="sourceLineNo">418</span> scheduler.addBack(proc);<a name="line.418"></a>
+<span class="sourceLineNo">419</span> continue;<a name="line.419"></a>
+<span class="sourceLineNo">420</span> }<a name="line.420"></a>
+<span class="sourceLineNo">421</span><a name="line.421"></a>
+<span class="sourceLineNo">422</span> if (proc.hasParent()) {<a name="line.422"></a>
+<span class="sourceLineNo">423</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.423"></a>
+<span class="sourceLineNo">424</span> // corrupted procedures are handled later at step 3<a name="line.424"></a>
+<span class="sourceLineNo">425</span> if (parent != null && !proc.isFinished()) {<a name="line.425"></a>
+<span class="sourceLineNo">426</span> parent.incChildrenLatch();<a name="line.426"></a>
+<span class="sourceLineNo">427</span> }<a name="line.427"></a>
+<span class="sourceLineNo">428</span> }<a name="line.428"></a>
+<span class="sourceLineNo">429</span><a name="line.429"></a>
+<span class="sourceLineNo">430</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.430"></a>
+<span class="sourceLineNo">431</span> procStack.loadStack(proc);<a name="line.431"></a>
+<span class="sourceLineNo">432</span><a name="line.432"></a>
+<span class="sourceLineNo">433</span> proc.setRootProcId(rootProcId);<a name="line.433"></a>
+<span class="sourceLineNo">434</span> switch (proc.getState()) {<a name="line.434"></a>
+<span class="sourceLineNo">435</span> case RUNNABLE:<a name="line.435"></a>
+<span class="sourceLineNo">436</span> runnableList.add(proc);<a name="line.436"></a>
+<span class="sourceLineNo">437</span> break;<a name="line.437"></a>
+<span class="sourceLineNo">438</span> case WAITING:<a name="line.438"></a>
+<span class="sourceLineNo">439</span> if (!proc.hasChildren()) {<a name="line.439"></a>
+<span class="sourceLineNo">440</span> runnableList.add(proc);<a name="line.440"></a>
+<span class="sourceLineNo">441</span> }<a name="line.441"></a>
+<span class="sourceLineNo">442</span> break;<a name="line.442"></a>
+<span class="sourceLineNo">443</span> case WAITING_TIMEOUT:<a name="line.443"></a>
+<span class="sourceLineNo">444</span> if (waitingSet == null) {<a name="line.444"></a>
+<span class="sourceLineNo">445</span> waitingSet = new HashSet<>();<a name="line.445"></a>
+<span class="sourceLineNo">446</span> }<a name="line.446"></a>
+<span class="sourceLineNo">447</span> waitingSet.add(proc);<a name="line.447"></a>
+<span class="sourceLineNo">448</span> break;<a name="line.448"></a>
+<span class="sourceLineNo">449</span> case FAILED:<a name="line.449"></a>
+<span class="sourceLineNo">450</span> // add the proc to the scheduler to perform the rollback<a name="line.450"></a>
+<span class="sourceLineNo">451</span> scheduler.addBack(proc);<a name="line.451"></a>
+<span class="sourceLineNo">452</span> break;<a name="line.452"></a>
+<span class="sourceLineNo">453</span> case ROLLEDBACK:<a name="line.453"></a>
+<span class="sourceLineNo">454</span> case INITIALIZING:<a name="line.454"></a>
+<span class="sourceLineNo">455</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.455"></a>
+<span class="sourceLineNo">456</span> LOG.error(msg);<a name="line.456"></a>
+<span class="sourceLineNo">457</span> throw new UnsupportedOperationException(msg);<a name="line.457"></a>
+<span class="sourceLineNo">458</span> default:<a name="line.458"></a>
+<span class="sourceLineNo">459</span> break;<a name="line.459"></a>
+<span class="sourceLineNo">460</span> }<a name="line.460"></a>
+<span class="sourceLineNo">461</span> }<a name="line.461"></a>
+<span class="sourceLineNo">462</span><a name="line.462"></a>
+<span class="sourceLineNo">463</span> // 3. Validate the stacks<a name="line.463"></a>
+<span class="sourceLineNo">464</span> int corruptedCount = 0;<a name="line.464"></a>
+<span class="sourceLineNo">465</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.465"></a>
+<span class="sourceLineNo">466</span> while (itStack.hasNext()) {<a name="line.466"></a>
+<span class="sourceLineNo">467</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.467"></a>
+<span class="sourceLineNo">468</span> RootProcedureState procStack = entry.getValue();<a name="line.468"></a>
+<span class="sourceLineNo">469</span> if (procStack.isValid()) continue;<a name="line.469"></a>
+<span class="sourceLineNo">470</span><a name="line.470"></a>
+<span class="sourceLineNo">471</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.471"></a>
+<span class="sourceLineNo">472</span> LOG.error("Corrupted " + proc);<a name="line.472"></a>
+<span class="sourceLineNo">473</span> procedures.remove(proc.getProcId());<a name="line.473"></a>
+<span class="sourceLineNo">474</span> runnableList.remove(proc);<a name="line.474"></a>
+<span class="sourceLineNo">475</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.475"></a>
+<span class="sourceLineNo">476</span> corruptedCount++;<a name="line.476"></a>
+<span class="sourceLineNo">477</span> }<a name="line.477"></a>
+<span class="sourceLineNo">478</span> itStack.remove();<a name="line.478"></a>
+<span class="sourceLineNo">479</span> }<a name="line.479"></a>
+<span class="sourceLineNo">480</span><a name="line.480"></a>
+<span class="sourceLineNo">481</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.481"></a>
+<span class="sourceLineNo">482</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.482"></a>
+<span class="sourceLineNo">483</span> }<a name="line.483"></a>
+<span class="sourceLineNo">484</span><a name="line.484"></a>
+<span class="sourceLineNo">485</span> // 4. Push the procedures to the timeout executor<a name="line.485"></a>
+<span class="sourceLineNo">486</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.486"></a>
+<span class="sourceLineNo">487</span> for (Procedure proc: waitingSet) {<a name="line.487"></a>
+<span class="sourceLineNo">488</span> proc.afterReplay(getEnvironment());<a name="line.488"></a>
+<span class="sourceLineNo">489</span> timeoutExecutor.add(proc);<a name="line.489"></a>
+<span class="sourceLineNo">490</span> }<a name="line.490"></a>
+<span class="sourceLineNo">491</span> }<a name="line.491"></a>
+<span class="sourceLineNo">492</span><a name="line.492"></a>
+<span class="sourceLineNo">493</span> // 5. Push the procedure to the scheduler<a name="line.493"></a>
+<span class="sourceLineNo">494</span> if (!runnableList.isEmpty()) {<a name="line.494"></a>
+<span class="sourceLineNo">495</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.495"></a>
+<span class="sourceLineNo">496</span> // some procedure may be started way before this stuff.<a name="line.496"></a>
+<span class="sourceLineNo">497</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.497"></a>
+<span class="sourceLineNo">498</span> Procedure proc = runnableList.get(i);<a name="line.498"></a>
+<span class="sourceLineNo">499</span> proc.afterReplay(getEnvironment());<a name="line.499"></a>
+<span class="sourceLineNo">500</span> if (!proc.hasParent()) {<a name="line.500"></a>
+<span class="sourceLineNo">501</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.501"></a>
+<span class="sourceLineNo">502</span> }<a name="line.502"></a>
+<span class="sourceLineNo">503</span> if (proc.wasExecuted()) {<a name="line.503"></a>
+<span class="sourceLineNo">504</span> scheduler.addFront(proc);<a name="line.504"></a>
+<span class="sourceLineNo">505</span> } else {<a name="line.505"></a>
+<span class="sourceLineNo">506</span> // if it was not in execution, it can wait.<a name="line.506"></a>
+<span class="sourceLineNo">507</span> scheduler.addBack(proc);<a name="line.507"></a>
+<span class="sourceLineNo">508</span> }<a name="line.508"></a>
+<span class="sourceLineNo">509</span> }<a name="line.509"></a>
+<span class="sourceLineNo">510</span> }<a name="line.510"></a>
+<span class="sourceLineNo">511</span> }<a name="line.511"></a>
<span class="sourceLineNo">512</span><a name="line.512"></a>
-<span class="sourceLineNo">513</span> // Create the Thread Group for the executors<a name="line.513"></a>
-<span class="sourceLineNo">514</span> threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.514"></a>
-<span class="sourceLineNo">515</span><a name="line.515"></a>
-<span class="sourceLineNo">516</span> // Create the timeout executor<a name="line.516"></a>
-<span class="sourceLineNo">517</span> timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.517"></a>
-<span class="sourceLineNo">518</span><a name="line.518"></a>
-<span class="sourceLineNo">519</span> // Create the workers<a name="line.519"></a>
-<span class="sourceLineNo">520</span> workerId.set(0);<a name="line.520"></a>
-<span class="sourceLineNo">521</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.521"></a>
-<span class="sourceLineNo">522</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.522"></a>
-<span class="sourceLineNo">523</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.523"></a>
-<span class="sourceLineNo">524</span> }<a name="line.524"></a>
-<span class="sourceLineNo">525</span><a name="line.525"></a>
-<span class="sourceLineNo">526</span> long st, et;<a name="line.526"></a>
-<span class="sourceLineNo">527</span><a name="line.527"></a>
-<span class="sourceLineNo">528</span> // Acquire the store lease.<a name="line.528"></a>
-<span class="sourceLineNo">529</span> st = EnvironmentEdgeManager.currentTime();<a name="line.529"></a>
-<span class="sourceLineNo">530</span> store.recoverLease();<a name="line.530"></a>
-<span class="sourceLineNo">531</span> et = EnvironmentEdgeManager.currentTime();<a name="line.531"></a>
-<span class="sourceLineNo">532</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.532"></a>
-<span class="sourceLineNo">533</span> StringUtils.humanTimeDiff(et - st));<a name="line.533"></a>
-<span class="sourceLineNo">534</span><a name="line.534"></a>
-<span class="sourceLineNo">535</span> // start the procedure scheduler<a name="line.535"></a>
-<span class="sourceLineNo">536</span> scheduler.start();<a name="line.536"></a>
-<span class="sourceLineNo">537</span><a name="line.537"></a>
-<span class="sourceLineNo">538</span> // TODO: Split in two steps.<a name="line.538"></a>
-<span class="sourceLineNo">539</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.539"></a>
-<span class="sourceLineNo">540</span> // The first one will make sure that we have the latest id,<a name="line.540"></a>
-<span class="sourceLineNo">541</span> // so we can start the threads and accept new procedures.<a name="line.541"></a>
-<span class="sourceLineNo">542</span> // The second step will do the actual load of old procedures.<a name="line.542"></a>
-<span class="sourceLineNo">543</span> st = EnvironmentEdgeManager.currentTime();<a name="line.543"></a>
-<span class="sourceLineNo">544</span> load(abortOnCorruption);<a name="line.544"></a>
-<span class="sourceLineNo">545</span> et = EnvironmentEdgeManager.currentTime();<a name="line.545"></a>
-<span class="sourceLineNo">546</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.546"></a>
-<span class="sourceLineNo">547</span> StringUtils.humanTimeDiff(et - st));<a name="line.547"></a>
-<span class="sourceLineNo">548</span><a name="line.548"></a>
-<span class="sourceLineNo">549</span> // Start the executors. Here we must have the lastProcId set.<a name="line.549"></a>
-<span class="sourceLineNo">550</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.550"></a>
-<span class="sourceLineNo">551</span> timeoutExecutor.start();<a name="line.551"></a>
-<span class="sourceLineNo">552</span> for (WorkerThread worker: workerThreads) {<a name="line.552"></a>
-<span class="sourceLineNo">553</span> worker.start();<a name="line.553"></a>
-<span class="sourceLineNo">554</span> }<a name="line.554"></a>
-<span class="sourceLineNo">555</span><a name="line.555"></a>
-<span class="sourceLineNo">556</span> // Internal chores<a name="line.556"></a>
-<span class="sourceLineNo">557</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.557"></a>
-<span class="sourceLineNo">558</span><a name="line.558"></a>
-<span class="sourceLineNo">559</span> // Add completed cleaner chore<a name="line.559"></a>
-<span class="sourceLineNo">560</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.560"></a>
-<span class="sourceLineNo">561</span> }<a name="line.561"></a>
-<span class="sourceLineNo">562</span><a name="line.562"></a>
-<span class="sourceLineNo">563</span> public void stop() {<a name="line.563"></a>
-<span class="sourceLineNo">564</span> if (!running.getAndSet(false)) {<a name="line.564"></a>
-<span class="sourceLineNo">565</span> return;<a name="line.565"></a>
-<span class="sourceLineNo">566</span> }<a name="line.566"></a>
-<span class="sourceLineNo">567</span><a name="line.567"></a>
-<span class="sourceLineNo">568</span> LOG.info("Stopping");<a name="line.568"></a>
-<span class="sourceLineNo">569</span> scheduler.stop();<a name="line.569"></a>
-<span class="sourceLineNo">570</span> timeoutExecutor.sendStopSignal();<a name="line.570"></a>
-<span class="sourceLineNo">571</span> }<a name="line.571"></a>
-<span class="sourceLineNo">572</span><a name="line.572"></a>
-<span class="sourceLineNo">573</span> @VisibleForTesting<a name="line.573"></a>
-<span class="sourceLineNo">574</span> public void join() {<a name="line.574"></a>
-<span class="sourceLineNo">575</span> assert !isRunning() : "expected not running";<a name="line.575"></a>
-<span class="sourceLineNo">576</span><a name="line.576"></a>
-<span class="sourceLineNo">577</span> // stop the timeout executor<a name="line.577"></a>
-<span class="sourceLineNo">578</span> timeoutExecutor.awaitTermination();<a name="line.578"></a>
-<span class="sourceLineNo">579</span> timeoutExecutor = null;<a name="line.579"></a>
-<span class="sourceLineNo">580</span><a name="line.580"></a>
-<span class="sourceLineNo">581</span> // stop the worker threads<a name="line.581"></a>
-<span class="sourceLineNo">582</span> for (WorkerThread worker: workerThreads) {<a name="line.582"></a>
-<span class="sourceLineNo">583</span> worker.awaitTermination();<a name="line.583"></a>
-<span class="sourceLineNo">584</span> }<a name="line.584"></a>
-<span class="sourceLineNo">585</span> workerThreads = null;<a name="line.585"></a>
-<span class="sourceLineNo">586</span><a name="line.586"></a>
-<span class="sourceLineNo">587</span> // Destroy the Thread Group for the executors<a name="line.587"></a>
-<span class="sourceLineNo">588</span> try {<a name="line.588"></a>
-<span class="sourceLineNo">589</span> threadGroup.destroy();<a name="line.589"></a>
-<span class="sourceLineNo">590</span> } catch (IllegalThreadStateException e) {<a name="line.590"></a>
-<span class="sourceLineNo">591</span> LOG.error("ThreadGroup " + threadGroup + " contains running threads; " + e.getMessage());<a name="line.591"></a>
-<span class="sourceLineNo">592</span> threadGroup.list();<a name="line.592"></a>
-<span class="sourceLineNo">593</span> } finally {<a name="line.593"></a>
-<span class="sourceLineNo">594</span> threadGroup = null;<a name="line.594"></a>
-<span class="sourceLineNo">595</span> }<a name="line.595"></a>
+<span class="sourceLineNo">513</span> /**<a name="line.513"></a>
+<span class="sourceLineNo">514</span> * Start the procedure executor.<a name="line.514"></a>
+<span class="sourceLineNo">515</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.515"></a>
+<span class="sourceLineNo">516</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.516"></a>
+<span class="sourceLineNo">517</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.517"></a>
+<span class="sourceLineNo">518</span> *<a name="line.518"></a>
+<span class="sourceLineNo">519</span> * @param numThreads number of threads available for procedure execution.<a name="line.519"></a>
+<span class="sourceLineNo">520</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.520"></a>
+<span class="sourceLineNo">521</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.521"></a>
+<span class="sourceLineNo">522</span> */<a name="line.522"></a>
+<span class="sourceLineNo">523</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.523"></a>
+<span class="sourceLineNo">524</span> if (!running.compareAndSet(false, true)) {<a name="line.524"></a>
+<span class="sourceLineNo">525</span> LOG.warn("Already running");<a name="line.525"></a>
+<span class="sourceLineNo">526</span> return;<a name="line.526"></a>
+<span class="sourceLineNo">527</span> }<a name="line.527"></a>
+<span class="sourceLineNo">528</span><a name="line.528"></a>
+<span class="sourceLineNo">529</span> // We have numThreads executor + one timer thread used for timing out<a name="line.529"></a>
+<span class="sourceLineNo">530</span> // procedures and triggering periodic procedures.<a name="line.530"></a>
+<span class="sourceLineNo">531</span> this.corePoolSize = numThreads;<a name="line.531"></a>
+<span class="sourceLineNo">532</span> this.maxPoolSize = 10 * numThreads;<a name="line.532"></a>
+<span class="sourceLineNo">533</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.533"></a>
+<span class="sourceLineNo">534</span> corePoolSize, maxPoolSize);<a name="line.534"></a>
+<span class="sourceLineNo">535</span><a name="line.535"></a>
+<span class="sourceLineNo">536</span> this.threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.536"></a>
+<span class="sourceLineNo">537</span> this.timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.537"></a>
+<span class="sourceLineNo">538</span><a name="line.538"></a>
+<span class="sourceLineNo">539</span> // Create the workers<a name="line.539"></a>
+<span class="sourceLineNo">540</span> workerId.set(0);<a name="line.540"></a>
+<span class="sourceLineNo">541</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.541"></a>
+<span class="sourceLineNo">542</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.542"></a>
+<span class="sourceLineNo">543</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.543"></a>
+<span class="sourceLineNo">544</span> }<a name="line.544"></a>
+<span class="sourceLineNo">545</span><a name="line.545"></a>
+<span class="sourceLineNo">546</span> long st, et;<a name="line.546"></a>
+<span class="sourceLineNo">547</span><a name="line.547"></a>
+<span class="sourceLineNo">548</span> // Acquire the store lease.<a name="line.548"></a>
+<span class="sourceLineNo">549</span> st = EnvironmentEdgeManager.currentTime();<a name="line.549"></a>
+<span class="sourceLineNo">550</span> store.recoverLease();<a name="line.550"></a>
+<span class="sourceLineNo">551</span> et = EnvironmentEdgeManager.currentTime();<a name="line.551"></a>
+<span class="sourceLineNo">552</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.552"></a>
+<span class="sourceLineNo">553</span> StringUtils.humanTimeDiff(et - st));<a name="line.553"></a>
+<span class="sourceLineNo">554</span><a name="line.554"></a>
+<span class="sourceLineNo">555</span> // start the procedure scheduler<a name="line.555"></a>
+<span class="sourceLineNo">556</span> scheduler.start();<a name="line.556"></a>
+<span class="sourceLineNo">557</span><a name="line.557"></a>
+<span class="sourceLineNo">558</span> // TODO: Split in two steps.<a name="line.558"></a>
+<span class="sourceLineNo">559</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.559"></a>
+<span class="sourceLineNo">560</span> // The first one will make sure that we have the latest id,<a name="line.560"></a>
+<span class="sourceLineNo">561</span> // so we can start the threads and accept new procedures.<a name="line.561"></a>
+<span class="sourceLineNo">562</span> // The second step will do the actual load of old procedures.<a name="line.562"></a>
+<span class="sourceLineNo">563</span> st = EnvironmentEdgeManager.currentTime();<a name="line.563"></a>
+<span class="sourceLineNo">564</span> load(abortOnCorruption);<a name="line.564"></a>
+<span class="sourceLineNo">565</span> et = EnvironmentEdgeManager.currentTime();<a name="line.565"></a>
+<span class="sourceLineNo">566</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.566"></a>
+<span class="sourceLineNo">567</span> StringUtils.humanTimeDiff(et - st));<a name="line.567"></a>
+<span class="sourceLineNo">568</span><a name="line.568"></a>
+<span class="sourceLineNo">569</span> // Start the executors. Here we must have the lastProcId set.<a name="line.569"></a>
+<span class="sourceLineNo">570</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.570"></a>
+<span class="sourceLineNo">571</span> timeoutExecutor.start();<a name="line.571"></a>
+<span class="sourceLineNo">572</span> for (WorkerThread worker: workerThreads) {<a name="line.572"></a>
+<span class="sourceLineNo">573</span> worker.start();<a name="line.573"></a>
+<span class="sourceLineNo">574</span> }<a name="line.574"></a>
+<span class="sourceLineNo">575</span><a name="line.575"></a>
+<span class="sourceLineNo">576</span> // Internal chores<a name="line.576"></a>
+<span class="sourceLineNo">577</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.577"></a>
+<span class="sourceLineNo">578</span><a name="line.578"></a>
+<span class="sourceLineNo">579</span> // Add completed cleaner chore<a name="line.579"></a>
+<span class="sourceLineNo">580</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.580"></a>
+<span class="sourceLineNo">581</span> }<a name="line.581"></a>
+<span class="sourceLineNo">582</span><a name="line.582"></a>
+<span class="sourceLineNo">583</span> public void stop() {<a name="line.583"></a>
+<span class="sourceLineNo">584</span> if (!running.getAndSet(false)) {<a name="line.584"></a>
+<span class="sourceLineNo">585</span> return;<a name="line.585"></a>
+<span class="sourceLineNo">586</span> }<a name="line.586"></a>
+<span class="sourceLineNo">587</span><a name="line.587"></a>
+<span class="sourceLineNo">588</span> LOG.info("Stopping");<a name="line.588"></a>
+<span class="sourceLineNo">589</span> scheduler.stop();<a name="line.589"></a>
+<span class="sourceLineNo">590</span> timeoutExecutor.sendStopSignal();<a name="line.590"></a>
+<span class="sourceLineNo">591</span> }<a name="line.591"></a>
+<span class="sourceLineNo">592</span><a name="line.592"></a>
+<span class="sourceLineNo">593</span> @VisibleForTesting<a name="line.593"></a>
+<span class="sourceLineNo">594</span> public void join() {<a name="line.594"></a>
+<span class="sourceLineNo">595</span> assert !isRunning() : "expected not running";<a name="line.595"></a>
<span class="sourceLineNo">596</span><a name="line.596"></a>
-<span class="sourceLineNo">597</span> // reset the in-memory state for testing<a name="line.597"></a>
-<span class="sourceLineNo">598</span> completed.clear();<a name="line.598"></a>
-<span class="sourceLineNo">599</span> rollbackStack.clear();<a name="line.599"></a>
-<span class="sourceLineNo">600</span> procedures.clear();<a name="line.600"></a>
-<span class="sourceLineNo">601</span> nonceKeysToProcIdsMap.clear();<a name="line.601"></a>
-<span class="sourceLineNo">602</span> scheduler.clear();<a name="line.602"></a>
-<span class="sourceLineNo">603</span> lastProcId.set(-1);<a name="line.603"></a>
-<span class="sourceLineNo">604</span> }<a name="line.604"></a>
-<span class="sourceLineNo">605</span><a name="line.605"></a>
-<span class="sourceLineNo">606</span> public void refreshConfiguration(final Configuration conf) {<a name="line.606"></a>
-<span class="sourceLineNo">607</span> this.conf = conf;<a name="line.607"></a>
-<span class="sourceLineNo">608</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.608"></a>
-<span class="sourceLineNo">609</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.609"></a>
-<span class="sourceLineNo">610</span> }<a name="line.610"></a>
-<span class="sourceLineNo">611</span><a name="line.611"></a>
-<span class="sourceLineNo">612</span> // ==========================================================================<a name="line.612"></a>
-<span class="sourceLineNo">613</span> // Accessors<a name="line.613"></a>
-<span class="sourceLineNo">614</span> // ==========================================================================<a name="line.614"></a>
-<span class="sourceLineNo">615</span> public boolean isRunning() {<a name="line.615"></a>
-<span class="sourceLineNo">616</span> return running.get();<a name="line.616"></a>
-<span class="sourceLineNo">617</span> }<a name="line.617"></a>
-<span class="sourceLineNo">618</span><a name="line.618"></a>
-<span class="sourceLineNo">619</span> /**<a name="line.619"></a>
-<span class="sourceLineNo">620</span> * @return the current number of worker threads.<a name="line.620"></a>
-<span class="sourceLineNo">621</span> */<a name="line.621"></a>
-<span class="sourceLineNo">622</span> public int getWorkerThreadCount() {<a name="line.622"></a>
-<span class="sourceLineNo">623</span> return workerThreads.size();<a name="line.623"></a>
-<span class="sourceLineNo">624</span> }<a name="line.624"></a>
-<span class="sourceLineNo">625</span><a name="line.625"></a>
-<span class="sourceLineNo">626</span> /**<a name="line.626"></a>
-<span class="sourceLineNo">627</span> * @return the core pool size settings.<a name="line.627"></a>
-<span class="sourceLineNo">628</span> */<a name="line.628"></a>
-<span class="sourceLineNo">629</span> public int getCorePoolSize() {<a name="line.629"></a>
-<span class="sourceLineNo">630</span> return corePoolSize;<a name="line.630"></a>
-<span class="sourceLineNo">631</span> }<a name="line.631"></a>
-<span class="sourceLineNo">632</span><a name="line.632"></a>
-<span class="sourceLineNo">633</span> public int getActiveExecutorCount() {<a name="line.633"></a>
-<span class="sourceLineNo">634</span> return activeExecutorCount.get();<a name="line.634"></a>
-<span class="sourceLineNo">635</span> }<a name="line.635"></a>
-<span class="sourceLineNo">636</span><a name="line.636"></a>
-<span class="sourceLineNo">637</span> public TEnvironment getEnvironment() {<a name="line.637"></a>
-<span class="sourceLineNo">638</span> return this.environment;<a name="line.638"></a>
-<span class="sourceLineNo">639</span> }<a name="line.639"></a>
-<span class="sourceLineNo">640</span><a name="line.640"></a>
-<span class="sourceLineNo">641</span> public ProcedureStore getStore() {<a name="line.641"></a>
-<span class="sourceLineNo">642</span> return this.store;<a name="line.642"></a>
+<span class="sourceLineNo">597</span> // stop the timeout executor<a name="line.597"></a>
+<span class="sourceLineNo">598</span> timeoutExecutor.awaitTermination();<a name="line.598"></a>
+<span class="sourceLineNo">599</span><a name="line.599"></a>
+<span class="sourceLineNo">600</span> // stop the worker threads<a name="line.600"></a>
+<span class="sourceLineNo">601</span> for (WorkerThread worker: workerThreads) {<a name="line.601"></a>
+<span class="sourceLineNo">602</span> worker.awaitTermination();<a name="line.602"></a>
+<span class="sourceLineNo">603</span> }<a name="line.603"></a>
+<span class="sourceLineNo">604</span><a name="line.604"></a>
+<span class="sourceLineNo">605</span> // Destroy the Thread Group for the executors<a name="line.605"></a>
+<span class="sourceLineNo">606</span> // TODO: Fix. #join is not place to destroy resources.<a name="line.606"></a>
+<span class="sourceLineNo">607</span> try {<a name="line.607"></a>
+<span class="sourceLineNo">608</span> threadGroup.destroy();<a name="line.608"></a>
+<span class="sourceLineNo">609</span> } catch (IllegalThreadStateException e) {<a name="line.609"></a>
+<span class="sourceLineNo">610</span> LOG.error("ThreadGroup {} contains running threads; {}: See STDOUT",<a name="line.610"></a>
+<span class="sourceLineNo">611</span> this.threadGroup, e.getMessage());<a name="line.611"></a>
+<span class="sourceLineNo">612</span> // This dumps list of threads on STDOUT.<a name="line.612"></a>
+<span class="sourceLineNo">613</span> this.threadGroup.list();<a name="line.613"></a>
+<span class="sourceLineNo">614</span> }<a name="line.614"></a>
+<span class="sourceLineNo">615</span><a name="line.615"></a>
+<span class="sourceLineNo">616</span> // reset the in-memory state for testing<a name="line.616"></a>
+<span class="sourceLineNo">617</span> completed.clear();<a name="line.617"></a>
+<span class="sourceLineNo">618</span> rollbackStack.clear();<a name="line.618"></a>
+<span class="sourceLineNo">619</span> procedures.clear();<a name="line.619"></a>
+<span class="sourceLineNo">620</span> nonceKeysToProcIdsMap.clear();<a name="line.620"></a>
+<span class="sourceLineNo">621</span> scheduler.clear();<a name="line.621"></a>
+<span class="sourceLineNo">622</span> lastProcId.set(-1);<a name="line.622"></a>
+<span class="sourceLineNo">623</span> }<a name="line.623"></a>
+<span class="sourceLineNo">624</span><a name="line.624"></a>
+<span class="sourceLineNo">625</span> public void refreshConfiguration(final Configuration conf) {<a name="line.625"></a>
+<span class="sourceLineNo">626</span> this.conf = conf;<a name="line.626"></a>
+<span class="sourceLineNo">627</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.627"></a>
+<span class="sourceLineNo">628</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.628"></a>
+<span class="sourceLineNo">629</span> }<a name="line.629"></a>
+<span class="sourceLineNo">630</span><a name="line.630"></a>
+<span class="sourceLineNo">631</span> // ==========================================================================<a name="line.631"></a>
+<span class="sourceLineNo">632</span> // Accessors<a name="line.632"></a>
+<span class="sourceLineNo">633</span> // ==========================================================================<a name="line.633"></a>
+<span class="sourceLineNo">634</span> public boolean isRunning() {<a name="line.634"></a>
+<span class="sourceLineNo">635</span> return running.get();<a name="line.635"></a>
+<span class="sourceLineNo">636</span> }<a name="line.636"></a>
+<span class="sourceLineNo">637</span><a name="line.637"></a>
+<span class="sourceLineNo">638</span> /**<a name="line.638"></a>
+<span class="sourceLineNo">639</span> * @return the current number of worker threads.<a name="line.639"></a>
+<span class="sourceLineNo">640</span> */<a name="line.640"></a>
+<span class="sourceLineNo">641</span> public int getWorkerThreadCount() {<a name="line.641"></a>
+<span class="sourceLineNo">642</span> return workerThreads.size();<a name="line.642"></a>
<span class="sourceLineNo">643</span> }<a name="line.643"></a>
<span class="sourceLineNo">644</span><a name="line.644"></a>
-<span class="sourceLineNo">645</span> ProcedureScheduler getScheduler() {<a name="line.645"></a>
-<span class="sourceLineNo">646</span> return scheduler;<a name="line.646"></a>
-<span class="sourceLineNo">647</span> }<a name="line.647"></a>
-<span class="sourceLineNo">648</span><a name="line.648"></a>
-<span class="sourceLineNo">649</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.649"></a>
-<span class="sourceLineNo">650</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.650"></a>
-<span class="sourceLineNo">651</span> this.scheduler.signalAll();<a name="line.651"></a>
-<span class="sourceLineNo">652</span> }<a name="line.652"></a>
-<span class="sourceLineNo">653</span><a name="line.653"></a>
-<span class="sourceLineNo">654</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.654"></a>
-<span class="sourceLineNo">655</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.655"></a>
-<span class="sourceLineNo">656</span> }<a name="line.656"></a>
-<span class="sourceLineNo">657</span><a name="line.657"></a>
-<span class="sourceLineNo">658</span> // ==========================================================================<a name="line.658"></a>
-<span class="sourceLineNo">659</span> // Submit/Remove Chores<a name="line.659"></a>
-<span class="sourceLineNo">660</span> // ==========================================================================<a name="line.660"></a>
-<span class="sourceLineNo">661</span><a name="line.661"></a>
-<span class="sourceLineNo">662</span> /**<a name="line.662"></a>
-<span class="sourceLineNo">663</span> * Add a chore procedure to the executor<a name="line.663"></a>
-<span class="sourceLineNo">664</span> * @param chore the chore to add<a name="line.664"></a>
-<span class="sourceLineNo">665</span> */<a name="line.665"></a>
-<span class="sourceLineNo">666</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.666"></a>
-<span class="sourceLineNo">667</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.667"></a>
-<span class="sourceLineNo">668</span> timeoutExecutor.add(chore);<a name="line.668"></a>
-<span class="sourceLineNo">669</span> }<a name="line.669"></a>
-<span class="sourceLineNo">670</span><a name="line.670"></a>
-<span class="sourceLineNo">671</span> /**<a name="line.671"></a>
-<span class="sourceLineNo">672</span> * Remove a chore procedure from the executor<a name="line.672"></a>
-<span class="sourceLineNo">673</span> * @param chore the chore to remove<a name="line.673"></a>
-<span class="sourceLineNo">674</span> * @return whether the chore is removed, or it will be removed later<a name="line.674"></a>
-<span class="sourceLineNo">675</span> */<a name="line.675"></a>
-<span class="sourceLineNo">676</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.676"></a>
-<span class="sourceLineNo">677</span> chore.setState(ProcedureState.SUCCESS);<a name="line.677"></a>
-<span class="sourceLineNo">678</span> return timeoutExecutor.remove(chore);<a name="line.678"></a>
-<span class="sourceLineNo">679</span> }<a name="line.679"></a>
+<span class="sourceLineNo">645</span> /**<a name="line.645"></a>
+<span class="sourceLineNo">646</span> * @return the core pool size settings.<a name="line.646"></a>
+<span class="sourceLineNo">647</span> */<a name="line.647"></a>
+<span class="sourceLineNo">648</span> public int getCorePoolSize() {<a name="line.648"></a>
+<span class="sourceLineNo">649</span> return corePoolSize;<a name="line.649"></a>
+<span class="sourceLineNo">650</span> }<a name="line.650"></a>
+<span class="sourceLineNo">651</span><a name="line.651"></a>
+<span class="sourceLineNo">652</span> public int getActiveExecutorCount() {<a name="line.652"></a>
+<span class="sourceLineNo">653</span> return activeExecutorCount.get();<a name="line.653"></a>
+<span class="sourceLineNo">654</span> }<a name="line.654"></a>
+<span class="sourceLineNo">655</span><a name="line.655"></a>
+<span class="sourceLineNo">656</span> public TEnvironment getEnvironment() {<a name="line.656"></a>
+<span class="sourceLineNo">657</span> return this.environment;<a name="line.657"></a>
+<span class="sourceLineNo">658</span> }<a name="line.658"></a>
+<span class="sourceLineNo">659</span><a name="line.659"></a>
+<span class="sourceLineNo">660</span> public ProcedureStore getStore() {<a name="line.660"></a>
+<span class="sourceLineNo">661</span> return this.store;<a name="line.661"></a>
+<span class="sourceLineNo">662</span> }<a name="line.662"></a>
+<span class="sourceLineNo">663</span><a name="line.663"></a>
+<span class="sourceLineNo">664</span> ProcedureScheduler getScheduler() {<a name="line.664"></a>
+<span class="sourceLineNo">665</span> return scheduler;<a name="line.665"></a>
+<span class="sourceLineNo">666</span> }<a name="line.666"></a>
+<span class="sourceLineNo">667</span><a name="line.667"></a>
+<span class="sourceLineNo">668</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.668"></a>
+<span class="sourceLineNo">669</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.669"></a>
+<span class="sourceLineNo">670</span> this.scheduler.signalAll();<a name="line.670"></a>
+<span class="sourceLineNo">671</span> }<a name="line.671"></a>
+<span class="sourceLineNo">672</span><a name="line.672"></a>
+<span class="sourceLineNo">673</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.673"></a>
+<span class="sourceLineNo">674</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.674"></a>
+<span class="sourceLineNo">675</span> }<a name="line.675"></a>
+<span class="sourceLineNo">676</span><a name="line.676"></a>
+<span class="sourceLineNo">677</span> // ==========================================================================<a name="line.677"></a>
+<span class="sourceLineNo">678</span> // Submit/Remove Chores<a name="line.678"></a>
+<span class="sourceLineNo">679</span> // ==========================================================================<a name="line.679"></a>
<span class="sourceLineNo">680</span><a name="line.680"></a>
-<span class="sourceLineNo">681</span> // ==========================================================================<a name="line.681"></a>
-<span class="sourceLineNo">682</span> // Nonce Procedure helpers<a name="line.682"></a>
-<span class="sourceLineNo">683</span> // ==========================================================================<a name="line.683"></a>
-<span class="sourceLineNo">684</span> /**<a name="line.684"></a>
-<span class="sourceLineNo">685</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.685"></a>
-<span class="sourceLineNo">686</span> * @param nonceGroup<a name="line.686"></a>
-<span class="sourceLineNo">687</span> * @param nonce<a name="line.687"></a>
-<span class="sourceLineNo">688</span> * @return the generated NonceKey<a name="line.688"></a>
-<span class="sourceLineNo">689</span> */<a name="line.689"></a>
-<span class="sourceLineNo">690</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.690"></a>
-<span class="sourceLineNo">691</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.691"></a>
-<span class="sourceLineNo">692</span> }<a name="line.692"></a>
-<span class="sourceLineNo">693</span><a name="line.693"></a>
-<span class="sourceLineNo">694</span> /**<a name="line.694"></a>
-<span class="sourceLineNo">695</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.695"></a>
-<span class="sourceLineNo">696</span> * A procId will be reserved and on submitProcedure(),<a name="line.696"></a>
-<span class="sourceLineNo">697</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.697"></a>
-<span class="sourceLineNo">698</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.698"></a>
-<span class="sourceLineNo">699</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.699"></a>
-<span class="sourceLineNo">700</span> * and submit the procedure.<a name="line.700"></a>
-<span class="sourceLineNo">701</span> *<a name="line.701"></a>
-<span class="sourceLineNo">702</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.702"></a>
-<span class="sourceLineNo">703</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.703"></a>
-<span class="sourceLineNo">704</span> */<a name="line.704"></a>
-<span class="sourceLineNo">705</span> public long registerNonce(final NonceKey nonceKey) {<a name="line.705"></a>
-<span class="sourceLineNo">706</span> if (nonceKey == null) return -1;<a name="line.706"></a>
-<span class="sourceLineNo">707</span><a name="line.707"></a>
-<span class="sourceLineNo">708</span> // check if we have already a Reserved ID for the nonce<a name="line.708"></a>
-<span class="sourceLineNo">709</span> Long oldProcId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.709"></a>
-<span class="sourceLineNo">710</span> if (oldProcId == null) {<a name="line.710"></a>
-<span class="sourceLineNo">711</span> // reserve a new Procedure ID, this will be associated with the nonce<a name="line.711"></a>
-<span class="sourceLineNo">712</span> // and the procedure submitted with the specified nonce will use this ID.<a name="line.712"></a>
-<span class="sourceLineNo">713</span> final long newProcId = nextProcId();<a name="line.713"></a>
-<span class="sourceLineNo">714</span> oldProcId = nonceKeysToProcIdsMap.putIfAbsent(nonceKey, newProcId);<a name="line.714"></a>
-<span class="sourceLineNo">715</span> if (oldProcId == null) return -1;<a name="line.715"></a>
-<span class="sourceLineNo">716</span> }<a name="line.716"></a>
-<span class="sourceLineNo">717</span><a name="line.717"></a>
-<span class="sourceLineNo">718</span> // we found a registered nonce, but the procedure may not have been submitted yet.<a name="line.718"></a>
-<span class="sourceLineNo">719</span> // since the client expect the procedure to be submitted, spin here until it is.<a name="line.719"></a>
-<span class="sourceLineNo">720</span> final boolean traceEnabled = LOG.isTraceEnabled();<a name="line.720"></a>
-<span class="sourceLineNo">721</span> while (isRunning() &&<a name="line.721"></a>
-<span class="sourceLineNo">722</span> !(procedures.containsKey(oldProcId) || completed.containsKey(oldProcId)) &&<a name="line.722"></a>
-<span class="sourceLineNo">723</span> nonceKeysToProcIdsMap.containsKey(nonceKey)) {<a name="line.723"></a>
-<span class="sourceLineNo">724</span> if (traceEnabled) {<a name="line.724"></a>
-<span class="sourceLineNo">725</span> LOG.trace("Waiting for pid=" + oldProcId.longValue() + " to be submitted");<a name="line.725"></a>
-<span class="sourceLineNo">726</span> }<a name="line.726"></a>
-<span class="sourceLineNo">727</span> Threads.sleep(100);<a name="line.727"></a>
-<span class="sourceLineNo">728</span> }<a name="line.728"></a>
-<span class="sourceLineNo">729</span> return oldProcId.longValue();<a name="line.729"></a>
-<span class="sourceLineNo">730</span> }<a name="line.730"></a>
-<span class="sourceLineNo">731</span><a name="line.731"></a>
-<span class="sourceLineNo">732</span> /**<a name="line.732"></a>
-<span class="sourceLineNo">733</span> * Remove the NonceKey if the procedure was not submitted to the executor.<a name="line.733"></a>
-<span class="sourceLineNo">734</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.734"></a>
-<span class="sourceLineNo">735</span> */<a name="line.735"></a>
-<span class="sourceLineNo">736</span> public void unregisterNonceIfProcedureWasNotSubmitted(final NonceKey nonceKey) {<a name="line.736"></a>
-<span class="sourceLineNo">737</span> if (nonceKey == null) return;<a name="line.737"></a>
-<span class="sourceLineNo">738</span><a name="line.738"></a>
-<span class="sourceLineNo">739</span> final Long procId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.739"></a>
-<span class="sourceLineNo">740</span> if (procId == null) return;<a name="line.740"></a>
-<span class="sourceLineNo">741</span><a name="line.741"></a>
-<span class="sourceLineNo">742</span> // if the procedure was not submitted, remove the nonce<a name="line.742"></a>
-<span class="sourceLineNo">743</span> if (!(procedures.containsKey(procId) || completed.containsKey(procId))) {<a name="line.743"></a>
-<span class="sourceLineNo">744</span> nonceKeysToProcIdsMap.remove(nonceKey);<a name="line.744"></a>
-<span class="sourceLineNo">745</span> }<a name="line.745"></a>
-<span class="sourceLineNo">746</span> }<a name="line.746"></a>
-<span class="sourceLineNo">747</span><a name="line.747"></a>
-<span class="sourceLineNo">748</span> public static class FailedProcedure<TEnvironment> extends Procedure<TEnvironment> {<a name="line.748"></a>
-<span class="sourceLineNo">749</span> private String procName;<a name="line.749"></a>
+<span class="sourceLineNo">681</span> /**<a name="line.681"></a>
+<span class="sourceLineNo">682</span> * Add a chore procedure to the executor<a name="line.682"></a>
+<span class="sourceLineNo">683</span> * @param chore the chore to add<a name="line.683"></a>
+<span class="sourceLineNo">684</span> */<a name="line.684"></a>
+<span class="sourceLineNo">685</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.685"></a>
+<span class="sourceLineNo">686</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.686"></a>
+<span class="sourceLineNo">687</span> timeoutExecutor.add(chore);<a name="line.687"></a>
+<span class="sourceLineNo">688</span> }<a name="line.688"></a>
+<span class="sourceLineNo">689</span><a name="line.689"></a>
+<span class="sourceLineNo">690</span> /**<a name="line.690"></a>
+<span class="sourceLineNo">691</span> * Remove a chore procedure from the executor<a name="line.691"></a>
+<span class="sourceLineNo">692</span> * @param chore the chore to remove<a name="line.692"></a>
+<span class="sourceLineNo">693</span> * @return whether the chore is removed, or it will be removed later<a name="line.693"></a>
+<span class="sourceLineNo">694</span> */<a name="line.694"></a>
+<span class="sourceLineNo">695</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.695"></a>
+<span class="sourceLineNo">696</span> chore.setState(ProcedureState.SUCCESS);<a name="line.696"></a>
+<span class="sourceLineNo">697</span> return timeoutExecutor.remove(chore);<a name="line.697"></a>
+<span class="sourceLineNo">698</span> }<a name="line.698"></a>
+<span class="sourceLineNo">699</span><a name="line.699"></a>
+<span class="sourceLineNo">700</span> // ==========================================================================<a name="line.700"></a>
+<span class="sourceLineNo">701</span> // Nonce Procedure helpers<a name="line.701"></a>
+<span class="sourceLineNo">702</span> // ==========================================================================<a name="line.702"></a>
+<span class="sourceLineNo">703</span> /**<a name="line.703"></a>
+<span class="sourceLineNo">704</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.704"></a>
+<span class="sourceLineNo">705</span> * @param nonceGroup<a name="line.705"></a>
+<span class="sourceLineNo">706</span> * @param nonce<a name="line.706"></a>
+<span class="sourceLineNo">707</span> * @return the generated NonceKey<a name="line.707"></a>
+<span class="sourceLineNo">708</span> */<a name="line.708"></a>
+<span class="sourceLineNo">709</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.709"></a>
+<span class="sourceLineNo">710</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.710"></a>
+<span class="sourceLineNo">711</span> }<a name="line.711"></a>
+<span class="sourceLineNo">712</span><a name="line.712"></a>
+<span class="sourceLineNo">713</span> /**<a name="line.713"></a>
+<span class="sourceLineNo">714</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.714"></a>
+<span class="sourceLineNo">715</span> * A procId will be reserved and on submitProcedure(),<a name="line.715"></a>
+<span class="sourceLineNo">716</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.716"></a>
+<span class="sourceLineNo">717</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.717"></a>
+<span class="sourceLineNo">718</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.718"></a>
+<span class="sourceLineNo">719</span> * and submit the procedure.<a name="line.719"></a>
+<span class="sourceLineNo">720</span> *<a name="line.720"></a>
+<span class="sourceLineNo">721</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.721"></a>
+<span class="sourceLineNo">722</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.722">
<TRUNCATED>
[05/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.Testing.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.Testing.html b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.Testing.html
index 97aa79c..cc787e3 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.Testing.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.Testing.html
@@ -272,1623 +272,1642 @@
<span class="sourceLineNo">264</span> private final CopyOnWriteArrayList<ProcedureExecutorListener> listeners = new CopyOnWriteArrayList<>();<a name="line.264"></a>
<span class="sourceLineNo">265</span><a name="line.265"></a>
<span class="sourceLineNo">266</span> private Configuration conf;<a name="line.266"></a>
-<span class="sourceLineNo">267</span> private ThreadGroup threadGroup;<a name="line.267"></a>
-<span class="sourceLineNo">268</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.268"></a>
-<span class="sourceLineNo">269</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.269"></a>
-<span class="sourceLineNo">270</span> private int corePoolSize;<a name="line.270"></a>
-<span class="sourceLineNo">271</span> private int maxPoolSize;<a name="line.271"></a>
-<span class="sourceLineNo">272</span><a name="line.272"></a>
-<span class="sourceLineNo">273</span> private volatile long keepAliveTime;<a name="line.273"></a>
-<span class="sourceLineNo">274</span><a name="line.274"></a>
-<span class="sourceLineNo">275</span> /**<a name="line.275"></a>
-<span class="sourceLineNo">276</span> * Scheduler/Queue that contains runnable procedures.<a name="line.276"></a>
-<span class="sourceLineNo">277</span> */<a name="line.277"></a>
-<span class="sourceLineNo">278</span> private final ProcedureScheduler scheduler;<a name="line.278"></a>
-<span class="sourceLineNo">279</span><a name="line.279"></a>
-<span class="sourceLineNo">280</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.280"></a>
-<span class="sourceLineNo">281</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.281"></a>
-<span class="sourceLineNo">282</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.282"></a>
-<span class="sourceLineNo">283</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.283"></a>
-<span class="sourceLineNo">284</span> private final TEnvironment environment;<a name="line.284"></a>
-<span class="sourceLineNo">285</span> private final ProcedureStore store;<a name="line.285"></a>
-<span class="sourceLineNo">286</span><a name="line.286"></a>
-<span class="sourceLineNo">287</span> private final boolean checkOwnerSet;<a name="line.287"></a>
-<span class="sourceLineNo">288</span><a name="line.288"></a>
-<span class="sourceLineNo">289</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.289"></a>
-<span class="sourceLineNo">290</span> final ProcedureStore store) {<a name="line.290"></a>
-<span class="sourceLineNo">291</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.291"></a>
-<span class="sourceLineNo">292</span> }<a name="line.292"></a>
-<span class="sourceLineNo">293</span><a name="line.293"></a>
-<span class="sourceLineNo">294</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.294"></a>
-<span class="sourceLineNo">295</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.295"></a>
-<span class="sourceLineNo">296</span> this.environment = environment;<a name="line.296"></a>
-<span class="sourceLineNo">297</span> this.scheduler = scheduler;<a name="line.297"></a>
-<span class="sourceLineNo">298</span> this.store = store;<a name="line.298"></a>
-<span class="sourceLineNo">299</span> this.conf = conf;<a name="line.299"></a>
-<span class="sourceLineNo">300</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.300"></a>
-<span class="sourceLineNo">301</span> refreshConfiguration(conf);<a name="line.301"></a>
-<span class="sourceLineNo">302</span> }<a name="line.302"></a>
-<span class="sourceLineNo">303</span><a name="line.303"></a>
-<span class="sourceLineNo">304</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.304"></a>
-<span class="sourceLineNo">305</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.305"></a>
-<span class="sourceLineNo">306</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.306"></a>
-<span class="sourceLineNo">307</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.307"></a>
-<span class="sourceLineNo">308</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.308"></a>
-<span class="sourceLineNo">309</span><a name="line.309"></a>
-<span class="sourceLineNo">310</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.310"></a>
-<span class="sourceLineNo">311</span> @Override<a name="line.311"></a>
-<span class="sourceLineNo">312</span> public void setMaxProcId(long maxProcId) {<a name="line.312"></a>
-<span class="sourceLineNo">313</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.313"></a>
-<span class="sourceLineNo">314</span> lastProcId.set(maxProcId);<a name="line.314"></a>
-<span class="sourceLineNo">315</span> }<a name="line.315"></a>
-<span class="sourceLineNo">316</span><a name="line.316"></a>
-<span class="sourceLineNo">317</span> @Override<a name="line.317"></a>
-<span class="sourceLineNo">318</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.318"></a>
-<span class="sourceLineNo">319</span> loadProcedures(procIter, abortOnCorruption);<a name="line.319"></a>
-<span class="sourceLineNo">320</span> }<a name="line.320"></a>
-<span class="sourceLineNo">321</span><a name="line.321"></a>
-<span class="sourceLineNo">322</span> @Override<a name="line.322"></a>
-<span class="sourceLineNo">323</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.323"></a>
-<span class="sourceLineNo">324</span> int corruptedCount = 0;<a name="line.324"></a>
-<span class="sourceLineNo">325</span> while (procIter.hasNext()) {<a name="line.325"></a>
-<span class="sourceLineNo">326</span> Procedure<?> proc = procIter.next();<a name="line.326"></a>
-<span class="sourceLineNo">327</span> LOG.error("Corrupt " + proc);<a name="line.327"></a>
-<span class="sourceLineNo">328</span> corruptedCount++;<a name="line.328"></a>
-<span class="sourceLineNo">329</span> }<a name="line.329"></a>
-<span class="sourceLineNo">330</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.330"></a>
-<span class="sourceLineNo">331</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.331"></a>
-<span class="sourceLineNo">332</span> }<a name="line.332"></a>
-<span class="sourceLineNo">333</span> }<a name="line.333"></a>
-<span class="sourceLineNo">334</span> });<a name="line.334"></a>
-<span class="sourceLineNo">335</span> }<a name="line.335"></a>
-<span class="sourceLineNo">336</span><a name="line.336"></a>
-<span class="sourceLineNo">337</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.337"></a>
-<span class="sourceLineNo">338</span> final boolean abortOnCorruption) throws IOException {<a name="line.338"></a>
-<span class="sourceLineNo">339</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.339"></a>
-<span class="sourceLineNo">340</span><a name="line.340"></a>
-<span class="sourceLineNo">341</span> // 1. Build the rollback stack<a name="line.341"></a>
-<span class="sourceLineNo">342</span> int runnablesCount = 0;<a name="line.342"></a>
-<span class="sourceLineNo">343</span> while (procIter.hasNext()) {<a name="line.343"></a>
-<span class="sourceLineNo">344</span> boolean finished = procIter.isNextFinished();<a name="line.344"></a>
-<span class="sourceLineNo">345</span> Procedure proc = procIter.next();<a name="line.345"></a>
-<span class="sourceLineNo">346</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.346"></a>
-<span class="sourceLineNo">347</span> long procId = proc.getProcId();<a name="line.347"></a>
-<span class="sourceLineNo">348</span><a name="line.348"></a>
-<span class="sourceLineNo">349</span> if (finished) {<a name="line.349"></a>
-<span class="sourceLineNo">350</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.350"></a>
-<span class="sourceLineNo">351</span> if (debugEnabled) {<a name="line.351"></a>
-<span class="sourceLineNo">352</span> LOG.debug("Completed " + proc);<a name="line.352"></a>
-<span class="sourceLineNo">353</span> }<a name="line.353"></a>
-<span class="sourceLineNo">354</span> } else {<a name="line.354"></a>
-<span class="sourceLineNo">355</span> if (!proc.hasParent()) {<a name="line.355"></a>
-<span class="sourceLineNo">356</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.356"></a>
-<span class="sourceLineNo">357</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.357"></a>
-<span class="sourceLineNo">358</span> }<a name="line.358"></a>
+<span class="sourceLineNo">267</span><a name="line.267"></a>
+<span class="sourceLineNo">268</span> /**<a name="line.268"></a>
+<span class="sourceLineNo">269</span> * Created in the {@link #start(int, boolean)} method. Destroyed in {@link #join()} (FIX! Doing<a name="line.269"></a>
+<span class="sourceLineNo">270</span> * resource handling rather than observing in a #join is unexpected).<a name="line.270"></a>
+<span class="sourceLineNo">271</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.271"></a>
+<span class="sourceLineNo">272</span> * (Should be ok).<a name="line.272"></a>
+<span class="sourceLineNo">273</span> */<a name="line.273"></a>
+<span class="sourceLineNo">274</span> private ThreadGroup threadGroup;<a name="line.274"></a>
+<span class="sourceLineNo">275</span><a name="line.275"></a>
+<span class="sourceLineNo">276</span> /**<a name="line.276"></a>
+<span class="sourceLineNo">277</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.277"></a>
+<span class="sourceLineNo">278</span> * resource handling rather than observing in a #join is unexpected).<a name="line.278"></a>
+<span class="sourceLineNo">279</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.279"></a>
+<span class="sourceLineNo">280</span> * (Should be ok).<a name="line.280"></a>
+<span class="sourceLineNo">281</span> */<a name="line.281"></a>
+<span class="sourceLineNo">282</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.282"></a>
+<span class="sourceLineNo">283</span><a name="line.283"></a>
+<span class="sourceLineNo">284</span> /**<a name="line.284"></a>
+<span class="sourceLineNo">285</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.285"></a>
+<span class="sourceLineNo">286</span> * resource handling rather than observing in a #join is unexpected).<a name="line.286"></a>
+<span class="sourceLineNo">287</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.287"></a>
+<span class="sourceLineNo">288</span> * (Should be ok).<a name="line.288"></a>
+<span class="sourceLineNo">289</span> */<a name="line.289"></a>
+<span class="sourceLineNo">290</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.290"></a>
+<span class="sourceLineNo">291</span><a name="line.291"></a>
+<span class="sourceLineNo">292</span> private int corePoolSize;<a name="line.292"></a>
+<span class="sourceLineNo">293</span> private int maxPoolSize;<a name="line.293"></a>
+<span class="sourceLineNo">294</span><a name="line.294"></a>
+<span class="sourceLineNo">295</span> private volatile long keepAliveTime;<a name="line.295"></a>
+<span class="sourceLineNo">296</span><a name="line.296"></a>
+<span class="sourceLineNo">297</span> /**<a name="line.297"></a>
+<span class="sourceLineNo">298</span> * Scheduler/Queue that contains runnable procedures.<a name="line.298"></a>
+<span class="sourceLineNo">299</span> */<a name="line.299"></a>
+<span class="sourceLineNo">300</span> private final ProcedureScheduler scheduler;<a name="line.300"></a>
+<span class="sourceLineNo">301</span><a name="line.301"></a>
+<span class="sourceLineNo">302</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.302"></a>
+<span class="sourceLineNo">303</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.303"></a>
+<span class="sourceLineNo">304</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.304"></a>
+<span class="sourceLineNo">305</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.305"></a>
+<span class="sourceLineNo">306</span> private final TEnvironment environment;<a name="line.306"></a>
+<span class="sourceLineNo">307</span> private final ProcedureStore store;<a name="line.307"></a>
+<span class="sourceLineNo">308</span><a name="line.308"></a>
+<span class="sourceLineNo">309</span> private final boolean checkOwnerSet;<a name="line.309"></a>
+<span class="sourceLineNo">310</span><a name="line.310"></a>
+<span class="sourceLineNo">311</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.311"></a>
+<span class="sourceLineNo">312</span> final ProcedureStore store) {<a name="line.312"></a>
+<span class="sourceLineNo">313</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.313"></a>
+<span class="sourceLineNo">314</span> }<a name="line.314"></a>
+<span class="sourceLineNo">315</span><a name="line.315"></a>
+<span class="sourceLineNo">316</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.316"></a>
+<span class="sourceLineNo">317</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.317"></a>
+<span class="sourceLineNo">318</span> this.environment = environment;<a name="line.318"></a>
+<span class="sourceLineNo">319</span> this.scheduler = scheduler;<a name="line.319"></a>
+<span class="sourceLineNo">320</span> this.store = store;<a name="line.320"></a>
+<span class="sourceLineNo">321</span> this.conf = conf;<a name="line.321"></a>
+<span class="sourceLineNo">322</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.322"></a>
+<span class="sourceLineNo">323</span> refreshConfiguration(conf);<a name="line.323"></a>
+<span class="sourceLineNo">324</span><a name="line.324"></a>
+<span class="sourceLineNo">325</span> }<a name="line.325"></a>
+<span class="sourceLineNo">326</span><a name="line.326"></a>
+<span class="sourceLineNo">327</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.327"></a>
+<span class="sourceLineNo">328</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.328"></a>
+<span class="sourceLineNo">329</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.329"></a>
+<span class="sourceLineNo">330</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.330"></a>
+<span class="sourceLineNo">331</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.331"></a>
+<span class="sourceLineNo">332</span><a name="line.332"></a>
+<span class="sourceLineNo">333</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.333"></a>
+<span class="sourceLineNo">334</span> @Override<a name="line.334"></a>
+<span class="sourceLineNo">335</span> public void setMaxProcId(long maxProcId) {<a name="line.335"></a>
+<span class="sourceLineNo">336</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.336"></a>
+<span class="sourceLineNo">337</span> lastProcId.set(maxProcId);<a name="line.337"></a>
+<span class="sourceLineNo">338</span> }<a name="line.338"></a>
+<span class="sourceLineNo">339</span><a name="line.339"></a>
+<span class="sourceLineNo">340</span> @Override<a name="line.340"></a>
+<span class="sourceLineNo">341</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.341"></a>
+<span class="sourceLineNo">342</span> loadProcedures(procIter, abortOnCorruption);<a name="line.342"></a>
+<span class="sourceLineNo">343</span> }<a name="line.343"></a>
+<span class="sourceLineNo">344</span><a name="line.344"></a>
+<span class="sourceLineNo">345</span> @Override<a name="line.345"></a>
+<span class="sourceLineNo">346</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.346"></a>
+<span class="sourceLineNo">347</span> int corruptedCount = 0;<a name="line.347"></a>
+<span class="sourceLineNo">348</span> while (procIter.hasNext()) {<a name="line.348"></a>
+<span class="sourceLineNo">349</span> Procedure<?> proc = procIter.next();<a name="line.349"></a>
+<span class="sourceLineNo">350</span> LOG.error("Corrupt " + proc);<a name="line.350"></a>
+<span class="sourceLineNo">351</span> corruptedCount++;<a name="line.351"></a>
+<span class="sourceLineNo">352</span> }<a name="line.352"></a>
+<span class="sourceLineNo">353</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.353"></a>
+<span class="sourceLineNo">354</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.354"></a>
+<span class="sourceLineNo">355</span> }<a name="line.355"></a>
+<span class="sourceLineNo">356</span> }<a name="line.356"></a>
+<span class="sourceLineNo">357</span> });<a name="line.357"></a>
+<span class="sourceLineNo">358</span> }<a name="line.358"></a>
<span class="sourceLineNo">359</span><a name="line.359"></a>
-<span class="sourceLineNo">360</span> // add the procedure to the map<a name="line.360"></a>
-<span class="sourceLineNo">361</span> proc.beforeReplay(getEnvironment());<a name="line.361"></a>
-<span class="sourceLineNo">362</span> procedures.put(proc.getProcId(), proc);<a name="line.362"></a>
+<span class="sourceLineNo">360</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.360"></a>
+<span class="sourceLineNo">361</span> final boolean abortOnCorruption) throws IOException {<a name="line.361"></a>
+<span class="sourceLineNo">362</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.362"></a>
<span class="sourceLineNo">363</span><a name="line.363"></a>
-<span class="sourceLineNo">364</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.364"></a>
-<span class="sourceLineNo">365</span> runnablesCount++;<a name="line.365"></a>
-<span class="sourceLineNo">366</span> }<a name="line.366"></a>
-<span class="sourceLineNo">367</span> }<a name="line.367"></a>
-<span class="sourceLineNo">368</span><a name="line.368"></a>
-<span class="sourceLineNo">369</span> // add the nonce to the map<a name="line.369"></a>
-<span class="sourceLineNo">370</span> if (nonceKey != null) {<a name="line.370"></a>
-<span class="sourceLineNo">371</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.371"></a>
-<span class="sourceLineNo">372</span> }<a name="line.372"></a>
-<span class="sourceLineNo">373</span> }<a name="line.373"></a>
-<span class="sourceLineNo">374</span><a name="line.374"></a>
-<span class="sourceLineNo">375</span> // 2. Initialize the stacks<a name="line.375"></a>
-<span class="sourceLineNo">376</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.376"></a>
-<span class="sourceLineNo">377</span> HashSet<Procedure> waitingSet = null;<a name="line.377"></a>
-<span class="sourceLineNo">378</span> procIter.reset();<a name="line.378"></a>
-<span class="sourceLineNo">379</span> while (procIter.hasNext()) {<a name="line.379"></a>
-<span class="sourceLineNo">380</span> if (procIter.isNextFinished()) {<a name="line.380"></a>
-<span class="sourceLineNo">381</span> procIter.skipNext();<a name="line.381"></a>
-<span class="sourceLineNo">382</span> continue;<a name="line.382"></a>
-<span class="sourceLineNo">383</span> }<a name="line.383"></a>
-<span class="sourceLineNo">384</span><a name="line.384"></a>
-<span class="sourceLineNo">385</span> Procedure proc = procIter.next();<a name="line.385"></a>
-<span class="sourceLineNo">386</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.386"></a>
-<span class="sourceLineNo">387</span><a name="line.387"></a>
-<span class="sourceLineNo">388</span> if (debugEnabled) {<a name="line.388"></a>
-<span class="sourceLineNo">389</span> LOG.debug(String.format("Loading %s", proc));<a name="line.389"></a>
+<span class="sourceLineNo">364</span> // 1. Build the rollback stack<a name="line.364"></a>
+<span class="sourceLineNo">365</span> int runnablesCount = 0;<a name="line.365"></a>
+<span class="sourceLineNo">366</span> while (procIter.hasNext()) {<a name="line.366"></a>
+<span class="sourceLineNo">367</span> boolean finished = procIter.isNextFinished();<a name="line.367"></a>
+<span class="sourceLineNo">368</span> Procedure proc = procIter.next();<a name="line.368"></a>
+<span class="sourceLineNo">369</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.369"></a>
+<span class="sourceLineNo">370</span> long procId = proc.getProcId();<a name="line.370"></a>
+<span class="sourceLineNo">371</span><a name="line.371"></a>
+<span class="sourceLineNo">372</span> if (finished) {<a name="line.372"></a>
+<span class="sourceLineNo">373</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.373"></a>
+<span class="sourceLineNo">374</span> if (debugEnabled) {<a name="line.374"></a>
+<span class="sourceLineNo">375</span> LOG.debug("Completed " + proc);<a name="line.375"></a>
+<span class="sourceLineNo">376</span> }<a name="line.376"></a>
+<span class="sourceLineNo">377</span> } else {<a name="line.377"></a>
+<span class="sourceLineNo">378</span> if (!proc.hasParent()) {<a name="line.378"></a>
+<span class="sourceLineNo">379</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.379"></a>
+<span class="sourceLineNo">380</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.380"></a>
+<span class="sourceLineNo">381</span> }<a name="line.381"></a>
+<span class="sourceLineNo">382</span><a name="line.382"></a>
+<span class="sourceLineNo">383</span> // add the procedure to the map<a name="line.383"></a>
+<span class="sourceLineNo">384</span> proc.beforeReplay(getEnvironment());<a name="line.384"></a>
+<span class="sourceLineNo">385</span> procedures.put(proc.getProcId(), proc);<a name="line.385"></a>
+<span class="sourceLineNo">386</span><a name="line.386"></a>
+<span class="sourceLineNo">387</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.387"></a>
+<span class="sourceLineNo">388</span> runnablesCount++;<a name="line.388"></a>
+<span class="sourceLineNo">389</span> }<a name="line.389"></a>
<span class="sourceLineNo">390</span> }<a name="line.390"></a>
<span class="sourceLineNo">391</span><a name="line.391"></a>
-<span class="sourceLineNo">392</span> Long rootProcId = getRootProcedureId(proc);<a name="line.392"></a>
-<span class="sourceLineNo">393</span> if (rootProcId == null) {<a name="line.393"></a>
-<span class="sourceLineNo">394</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.394"></a>
-<span class="sourceLineNo">395</span> scheduler.addBack(proc);<a name="line.395"></a>
-<span class="sourceLineNo">396</span> continue;<a name="line.396"></a>
-<span class="sourceLineNo">397</span> }<a name="line.397"></a>
-<span class="sourceLineNo">398</span><a name="line.398"></a>
-<span class="sourceLineNo">399</span> if (proc.hasParent()) {<a name="line.399"></a>
-<span class="sourceLineNo">400</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.400"></a>
-<span class="sourceLineNo">401</span> // corrupted procedures are handled later at step 3<a name="line.401"></a>
-<span class="sourceLineNo">402</span> if (parent != null && !proc.isFinished()) {<a name="line.402"></a>
-<span class="sourceLineNo">403</span> parent.incChildrenLatch();<a name="line.403"></a>
-<span class="sourceLineNo">404</span> }<a name="line.404"></a>
-<span class="sourceLineNo">405</span> }<a name="line.405"></a>
-<span class="sourceLineNo">406</span><a name="line.406"></a>
-<span class="sourceLineNo">407</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.407"></a>
-<span class="sourceLineNo">408</span> procStack.loadStack(proc);<a name="line.408"></a>
-<span class="sourceLineNo">409</span><a name="line.409"></a>
-<span class="sourceLineNo">410</span> proc.setRootProcId(rootProcId);<a name="line.410"></a>
-<span class="sourceLineNo">411</span> switch (proc.getState()) {<a name="line.411"></a>
-<span class="sourceLineNo">412</span> case RUNNABLE:<a name="line.412"></a>
-<span class="sourceLineNo">413</span> runnableList.add(proc);<a name="line.413"></a>
-<span class="sourceLineNo">414</span> break;<a name="line.414"></a>
-<span class="sourceLineNo">415</span> case WAITING:<a name="line.415"></a>
-<span class="sourceLineNo">416</span> if (!proc.hasChildren()) {<a name="line.416"></a>
-<span class="sourceLineNo">417</span> runnableList.add(proc);<a name="line.417"></a>
-<span class="sourceLineNo">418</span> }<a name="line.418"></a>
-<span class="sourceLineNo">419</span> break;<a name="line.419"></a>
-<span class="sourceLineNo">420</span> case WAITING_TIMEOUT:<a name="line.420"></a>
-<span class="sourceLineNo">421</span> if (waitingSet == null) {<a name="line.421"></a>
-<span class="sourceLineNo">422</span> waitingSet = new HashSet<>();<a name="line.422"></a>
-<span class="sourceLineNo">423</span> }<a name="line.423"></a>
-<span class="sourceLineNo">424</span> waitingSet.add(proc);<a name="line.424"></a>
-<span class="sourceLineNo">425</span> break;<a name="line.425"></a>
-<span class="sourceLineNo">426</span> case FAILED:<a name="line.426"></a>
-<span class="sourceLineNo">427</span> // add the proc to the scheduler to perform the rollback<a name="line.427"></a>
-<span class="sourceLineNo">428</span> scheduler.addBack(proc);<a name="line.428"></a>
-<span class="sourceLineNo">429</span> break;<a name="line.429"></a>
-<span class="sourceLineNo">430</span> case ROLLEDBACK:<a name="line.430"></a>
-<span class="sourceLineNo">431</span> case INITIALIZING:<a name="line.431"></a>
-<span class="sourceLineNo">432</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.432"></a>
-<span class="sourceLineNo">433</span> LOG.error(msg);<a name="line.433"></a>
-<span class="sourceLineNo">434</span> throw new UnsupportedOperationException(msg);<a name="line.434"></a>
-<span class="sourceLineNo">435</span> default:<a name="line.435"></a>
-<span class="sourceLineNo">436</span> break;<a name="line.436"></a>
-<span class="sourceLineNo">437</span> }<a name="line.437"></a>
-<span class="sourceLineNo">438</span> }<a name="line.438"></a>
-<span class="sourceLineNo">439</span><a name="line.439"></a>
-<span class="sourceLineNo">440</span> // 3. Validate the stacks<a name="line.440"></a>
-<span class="sourceLineNo">441</span> int corruptedCount = 0;<a name="line.441"></a>
-<span class="sourceLineNo">442</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.442"></a>
-<span class="sourceLineNo">443</span> while (itStack.hasNext()) {<a name="line.443"></a>
-<span class="sourceLineNo">444</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.444"></a>
-<span class="sourceLineNo">445</span> RootProcedureState procStack = entry.getValue();<a name="line.445"></a>
-<span class="sourceLineNo">446</span> if (procStack.isValid()) continue;<a name="line.446"></a>
-<span class="sourceLineNo">447</span><a name="line.447"></a>
-<span class="sourceLineNo">448</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.448"></a>
-<span class="sourceLineNo">449</span> LOG.error("Corrupted " + proc);<a name="line.449"></a>
-<span class="sourceLineNo">450</span> procedures.remove(proc.getProcId());<a name="line.450"></a>
-<span class="sourceLineNo">451</span> runnableList.remove(proc);<a name="line.451"></a>
-<span class="sourceLineNo">452</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.452"></a>
-<span class="sourceLineNo">453</span> corruptedCount++;<a name="line.453"></a>
-<span class="sourceLineNo">454</span> }<a name="line.454"></a>
-<span class="sourceLineNo">455</span> itStack.remove();<a name="line.455"></a>
-<span class="sourceLineNo">456</span> }<a name="line.456"></a>
-<span class="sourceLineNo">457</span><a name="line.457"></a>
-<span class="sourceLineNo">458</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.458"></a>
-<span class="sourceLineNo">459</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.459"></a>
-<span class="sourceLineNo">460</span> }<a name="line.460"></a>
-<span class="sourceLineNo">461</span><a name="line.461"></a>
-<span class="sourceLineNo">462</span> // 4. Push the procedures to the timeout executor<a name="line.462"></a>
-<span class="sourceLineNo">463</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.463"></a>
-<span class="sourceLineNo">464</span> for (Procedure proc: waitingSet) {<a name="line.464"></a>
-<span class="sourceLineNo">465</span> proc.afterReplay(getEnvironment());<a name="line.465"></a>
-<span class="sourceLineNo">466</span> timeoutExecutor.add(proc);<a name="line.466"></a>
-<span class="sourceLineNo">467</span> }<a name="line.467"></a>
-<span class="sourceLineNo">468</span> }<a name="line.468"></a>
-<span class="sourceLineNo">469</span><a name="line.469"></a>
-<span class="sourceLineNo">470</span> // 5. Push the procedure to the scheduler<a name="line.470"></a>
-<span class="sourceLineNo">471</span> if (!runnableList.isEmpty()) {<a name="line.471"></a>
-<span class="sourceLineNo">472</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.472"></a>
-<span class="sourceLineNo">473</span> // some procedure may be started way before this stuff.<a name="line.473"></a>
-<span class="sourceLineNo">474</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.474"></a>
-<span class="sourceLineNo">475</span> Procedure proc = runnableList.get(i);<a name="line.475"></a>
-<span class="sourceLineNo">476</span> proc.afterReplay(getEnvironment());<a name="line.476"></a>
-<span class="sourceLineNo">477</span> if (!proc.hasParent()) {<a name="line.477"></a>
-<span class="sourceLineNo">478</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.478"></a>
-<span class="sourceLineNo">479</span> }<a name="line.479"></a>
-<span class="sourceLineNo">480</span> if (proc.wasExecuted()) {<a name="line.480"></a>
-<span class="sourceLineNo">481</span> scheduler.addFront(proc);<a name="line.481"></a>
-<span class="sourceLineNo">482</span> } else {<a name="line.482"></a>
-<span class="sourceLineNo">483</span> // if it was not in execution, it can wait.<a name="line.483"></a>
-<span class="sourceLineNo">484</span> scheduler.addBack(proc);<a name="line.484"></a>
-<span class="sourceLineNo">485</span> }<a name="line.485"></a>
-<span class="sourceLineNo">486</span> }<a name="line.486"></a>
-<span class="sourceLineNo">487</span> }<a name="line.487"></a>
-<span class="sourceLineNo">488</span> }<a name="line.488"></a>
-<span class="sourceLineNo">489</span><a name="line.489"></a>
-<span class="sourceLineNo">490</span> /**<a name="line.490"></a>
-<span class="sourceLineNo">491</span> * Start the procedure executor.<a name="line.491"></a>
-<span class="sourceLineNo">492</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.492"></a>
-<span class="sourceLineNo">493</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.493"></a>
-<span class="sourceLineNo">494</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.494"></a>
-<span class="sourceLineNo">495</span> *<a name="line.495"></a>
-<span class="sourceLineNo">496</span> * @param numThreads number of threads available for procedure execution.<a name="line.496"></a>
-<span class="sourceLineNo">497</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.497"></a>
-<span class="sourceLineNo">498</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.498"></a>
-<span class="sourceLineNo">499</span> */<a name="line.499"></a>
-<span class="sourceLineNo">500</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.500"></a>
-<span class="sourceLineNo">501</span> if (!running.compareAndSet(false, true)) {<a name="line.501"></a>
-<span class="sourceLineNo">502</span> LOG.warn("Already running");<a name="line.502"></a>
-<span class="sourceLineNo">503</span> return;<a name="line.503"></a>
-<span class="sourceLineNo">504</span> }<a name="line.504"></a>
-<span class="sourceLineNo">505</span><a name="line.505"></a>
-<span class="sourceLineNo">506</span> // We have numThreads executor + one timer thread used for timing out<a name="line.506"></a>
-<span class="sourceLineNo">507</span> // procedures and triggering periodic procedures.<a name="line.507"></a>
-<span class="sourceLineNo">508</span> this.corePoolSize = numThreads;<a name="line.508"></a>
-<span class="sourceLineNo">509</span> this.maxPoolSize = 10 * numThreads;<a name="line.509"></a>
-<span class="sourceLineNo">510</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.510"></a>
-<span class="sourceLineNo">511</span> corePoolSize, maxPoolSize);<a name="line.511"></a>
+<span class="sourceLineNo">392</span> // add the nonce to the map<a name="line.392"></a>
+<span class="sourceLineNo">393</span> if (nonceKey != null) {<a name="line.393"></a>
+<span class="sourceLineNo">394</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.394"></a>
+<span class="sourceLineNo">395</span> }<a name="line.395"></a>
+<span class="sourceLineNo">396</span> }<a name="line.396"></a>
+<span class="sourceLineNo">397</span><a name="line.397"></a>
+<span class="sourceLineNo">398</span> // 2. Initialize the stacks<a name="line.398"></a>
+<span class="sourceLineNo">399</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.399"></a>
+<span class="sourceLineNo">400</span> HashSet<Procedure> waitingSet = null;<a name="line.400"></a>
+<span class="sourceLineNo">401</span> procIter.reset();<a name="line.401"></a>
+<span class="sourceLineNo">402</span> while (procIter.hasNext()) {<a name="line.402"></a>
+<span class="sourceLineNo">403</span> if (procIter.isNextFinished()) {<a name="line.403"></a>
+<span class="sourceLineNo">404</span> procIter.skipNext();<a name="line.404"></a>
+<span class="sourceLineNo">405</span> continue;<a name="line.405"></a>
+<span class="sourceLineNo">406</span> }<a name="line.406"></a>
+<span class="sourceLineNo">407</span><a name="line.407"></a>
+<span class="sourceLineNo">408</span> Procedure proc = procIter.next();<a name="line.408"></a>
+<span class="sourceLineNo">409</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.409"></a>
+<span class="sourceLineNo">410</span><a name="line.410"></a>
+<span class="sourceLineNo">411</span> if (debugEnabled) {<a name="line.411"></a>
+<span class="sourceLineNo">412</span> LOG.debug(String.format("Loading %s", proc));<a name="line.412"></a>
+<span class="sourceLineNo">413</span> }<a name="line.413"></a>
+<span class="sourceLineNo">414</span><a name="line.414"></a>
+<span class="sourceLineNo">415</span> Long rootProcId = getRootProcedureId(proc);<a name="line.415"></a>
+<span class="sourceLineNo">416</span> if (rootProcId == null) {<a name="line.416"></a>
+<span class="sourceLineNo">417</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.417"></a>
+<span class="sourceLineNo">418</span> scheduler.addBack(proc);<a name="line.418"></a>
+<span class="sourceLineNo">419</span> continue;<a name="line.419"></a>
+<span class="sourceLineNo">420</span> }<a name="line.420"></a>
+<span class="sourceLineNo">421</span><a name="line.421"></a>
+<span class="sourceLineNo">422</span> if (proc.hasParent()) {<a name="line.422"></a>
+<span class="sourceLineNo">423</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.423"></a>
+<span class="sourceLineNo">424</span> // corrupted procedures are handled later at step 3<a name="line.424"></a>
+<span class="sourceLineNo">425</span> if (parent != null && !proc.isFinished()) {<a name="line.425"></a>
+<span class="sourceLineNo">426</span> parent.incChildrenLatch();<a name="line.426"></a>
+<span class="sourceLineNo">427</span> }<a name="line.427"></a>
+<span class="sourceLineNo">428</span> }<a name="line.428"></a>
+<span class="sourceLineNo">429</span><a name="line.429"></a>
+<span class="sourceLineNo">430</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.430"></a>
+<span class="sourceLineNo">431</span> procStack.loadStack(proc);<a name="line.431"></a>
+<span class="sourceLineNo">432</span><a name="line.432"></a>
+<span class="sourceLineNo">433</span> proc.setRootProcId(rootProcId);<a name="line.433"></a>
+<span class="sourceLineNo">434</span> switch (proc.getState()) {<a name="line.434"></a>
+<span class="sourceLineNo">435</span> case RUNNABLE:<a name="line.435"></a>
+<span class="sourceLineNo">436</span> runnableList.add(proc);<a name="line.436"></a>
+<span class="sourceLineNo">437</span> break;<a name="line.437"></a>
+<span class="sourceLineNo">438</span> case WAITING:<a name="line.438"></a>
+<span class="sourceLineNo">439</span> if (!proc.hasChildren()) {<a name="line.439"></a>
+<span class="sourceLineNo">440</span> runnableList.add(proc);<a name="line.440"></a>
+<span class="sourceLineNo">441</span> }<a name="line.441"></a>
+<span class="sourceLineNo">442</span> break;<a name="line.442"></a>
+<span class="sourceLineNo">443</span> case WAITING_TIMEOUT:<a name="line.443"></a>
+<span class="sourceLineNo">444</span> if (waitingSet == null) {<a name="line.444"></a>
+<span class="sourceLineNo">445</span> waitingSet = new HashSet<>();<a name="line.445"></a>
+<span class="sourceLineNo">446</span> }<a name="line.446"></a>
+<span class="sourceLineNo">447</span> waitingSet.add(proc);<a name="line.447"></a>
+<span class="sourceLineNo">448</span> break;<a name="line.448"></a>
+<span class="sourceLineNo">449</span> case FAILED:<a name="line.449"></a>
+<span class="sourceLineNo">450</span> // add the proc to the scheduler to perform the rollback<a name="line.450"></a>
+<span class="sourceLineNo">451</span> scheduler.addBack(proc);<a name="line.451"></a>
+<span class="sourceLineNo">452</span> break;<a name="line.452"></a>
+<span class="sourceLineNo">453</span> case ROLLEDBACK:<a name="line.453"></a>
+<span class="sourceLineNo">454</span> case INITIALIZING:<a name="line.454"></a>
+<span class="sourceLineNo">455</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.455"></a>
+<span class="sourceLineNo">456</span> LOG.error(msg);<a name="line.456"></a>
+<span class="sourceLineNo">457</span> throw new UnsupportedOperationException(msg);<a name="line.457"></a>
+<span class="sourceLineNo">458</span> default:<a name="line.458"></a>
+<span class="sourceLineNo">459</span> break;<a name="line.459"></a>
+<span class="sourceLineNo">460</span> }<a name="line.460"></a>
+<span class="sourceLineNo">461</span> }<a name="line.461"></a>
+<span class="sourceLineNo">462</span><a name="line.462"></a>
+<span class="sourceLineNo">463</span> // 3. Validate the stacks<a name="line.463"></a>
+<span class="sourceLineNo">464</span> int corruptedCount = 0;<a name="line.464"></a>
+<span class="sourceLineNo">465</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.465"></a>
+<span class="sourceLineNo">466</span> while (itStack.hasNext()) {<a name="line.466"></a>
+<span class="sourceLineNo">467</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.467"></a>
+<span class="sourceLineNo">468</span> RootProcedureState procStack = entry.getValue();<a name="line.468"></a>
+<span class="sourceLineNo">469</span> if (procStack.isValid()) continue;<a name="line.469"></a>
+<span class="sourceLineNo">470</span><a name="line.470"></a>
+<span class="sourceLineNo">471</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.471"></a>
+<span class="sourceLineNo">472</span> LOG.error("Corrupted " + proc);<a name="line.472"></a>
+<span class="sourceLineNo">473</span> procedures.remove(proc.getProcId());<a name="line.473"></a>
+<span class="sourceLineNo">474</span> runnableList.remove(proc);<a name="line.474"></a>
+<span class="sourceLineNo">475</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.475"></a>
+<span class="sourceLineNo">476</span> corruptedCount++;<a name="line.476"></a>
+<span class="sourceLineNo">477</span> }<a name="line.477"></a>
+<span class="sourceLineNo">478</span> itStack.remove();<a name="line.478"></a>
+<span class="sourceLineNo">479</span> }<a name="line.479"></a>
+<span class="sourceLineNo">480</span><a name="line.480"></a>
+<span class="sourceLineNo">481</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.481"></a>
+<span class="sourceLineNo">482</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.482"></a>
+<span class="sourceLineNo">483</span> }<a name="line.483"></a>
+<span class="sourceLineNo">484</span><a name="line.484"></a>
+<span class="sourceLineNo">485</span> // 4. Push the procedures to the timeout executor<a name="line.485"></a>
+<span class="sourceLineNo">486</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.486"></a>
+<span class="sourceLineNo">487</span> for (Procedure proc: waitingSet) {<a name="line.487"></a>
+<span class="sourceLineNo">488</span> proc.afterReplay(getEnvironment());<a name="line.488"></a>
+<span class="sourceLineNo">489</span> timeoutExecutor.add(proc);<a name="line.489"></a>
+<span class="sourceLineNo">490</span> }<a name="line.490"></a>
+<span class="sourceLineNo">491</span> }<a name="line.491"></a>
+<span class="sourceLineNo">492</span><a name="line.492"></a>
+<span class="sourceLineNo">493</span> // 5. Push the procedure to the scheduler<a name="line.493"></a>
+<span class="sourceLineNo">494</span> if (!runnableList.isEmpty()) {<a name="line.494"></a>
+<span class="sourceLineNo">495</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.495"></a>
+<span class="sourceLineNo">496</span> // some procedure may be started way before this stuff.<a name="line.496"></a>
+<span class="sourceLineNo">497</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.497"></a>
+<span class="sourceLineNo">498</span> Procedure proc = runnableList.get(i);<a name="line.498"></a>
+<span class="sourceLineNo">499</span> proc.afterReplay(getEnvironment());<a name="line.499"></a>
+<span class="sourceLineNo">500</span> if (!proc.hasParent()) {<a name="line.500"></a>
+<span class="sourceLineNo">501</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.501"></a>
+<span class="sourceLineNo">502</span> }<a name="line.502"></a>
+<span class="sourceLineNo">503</span> if (proc.wasExecuted()) {<a name="line.503"></a>
+<span class="sourceLineNo">504</span> scheduler.addFront(proc);<a name="line.504"></a>
+<span class="sourceLineNo">505</span> } else {<a name="line.505"></a>
+<span class="sourceLineNo">506</span> // if it was not in execution, it can wait.<a name="line.506"></a>
+<span class="sourceLineNo">507</span> scheduler.addBack(proc);<a name="line.507"></a>
+<span class="sourceLineNo">508</span> }<a name="line.508"></a>
+<span class="sourceLineNo">509</span> }<a name="line.509"></a>
+<span class="sourceLineNo">510</span> }<a name="line.510"></a>
+<span class="sourceLineNo">511</span> }<a name="line.511"></a>
<span class="sourceLineNo">512</span><a name="line.512"></a>
-<span class="sourceLineNo">513</span> // Create the Thread Group for the executors<a name="line.513"></a>
-<span class="sourceLineNo">514</span> threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.514"></a>
-<span class="sourceLineNo">515</span><a name="line.515"></a>
-<span class="sourceLineNo">516</span> // Create the timeout executor<a name="line.516"></a>
-<span class="sourceLineNo">517</span> timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.517"></a>
-<span class="sourceLineNo">518</span><a name="line.518"></a>
-<span class="sourceLineNo">519</span> // Create the workers<a name="line.519"></a>
-<span class="sourceLineNo">520</span> workerId.set(0);<a name="line.520"></a>
-<span class="sourceLineNo">521</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.521"></a>
-<span class="sourceLineNo">522</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.522"></a>
-<span class="sourceLineNo">523</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.523"></a>
-<span class="sourceLineNo">524</span> }<a name="line.524"></a>
-<span class="sourceLineNo">525</span><a name="line.525"></a>
-<span class="sourceLineNo">526</span> long st, et;<a name="line.526"></a>
-<span class="sourceLineNo">527</span><a name="line.527"></a>
-<span class="sourceLineNo">528</span> // Acquire the store lease.<a name="line.528"></a>
-<span class="sourceLineNo">529</span> st = EnvironmentEdgeManager.currentTime();<a name="line.529"></a>
-<span class="sourceLineNo">530</span> store.recoverLease();<a name="line.530"></a>
-<span class="sourceLineNo">531</span> et = EnvironmentEdgeManager.currentTime();<a name="line.531"></a>
-<span class="sourceLineNo">532</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.532"></a>
-<span class="sourceLineNo">533</span> StringUtils.humanTimeDiff(et - st));<a name="line.533"></a>
-<span class="sourceLineNo">534</span><a name="line.534"></a>
-<span class="sourceLineNo">535</span> // start the procedure scheduler<a name="line.535"></a>
-<span class="sourceLineNo">536</span> scheduler.start();<a name="line.536"></a>
-<span class="sourceLineNo">537</span><a name="line.537"></a>
-<span class="sourceLineNo">538</span> // TODO: Split in two steps.<a name="line.538"></a>
-<span class="sourceLineNo">539</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.539"></a>
-<span class="sourceLineNo">540</span> // The first one will make sure that we have the latest id,<a name="line.540"></a>
-<span class="sourceLineNo">541</span> // so we can start the threads and accept new procedures.<a name="line.541"></a>
-<span class="sourceLineNo">542</span> // The second step will do the actual load of old procedures.<a name="line.542"></a>
-<span class="sourceLineNo">543</span> st = EnvironmentEdgeManager.currentTime();<a name="line.543"></a>
-<span class="sourceLineNo">544</span> load(abortOnCorruption);<a name="line.544"></a>
-<span class="sourceLineNo">545</span> et = EnvironmentEdgeManager.currentTime();<a name="line.545"></a>
-<span class="sourceLineNo">546</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.546"></a>
-<span class="sourceLineNo">547</span> StringUtils.humanTimeDiff(et - st));<a name="line.547"></a>
-<span class="sourceLineNo">548</span><a name="line.548"></a>
-<span class="sourceLineNo">549</span> // Start the executors. Here we must have the lastProcId set.<a name="line.549"></a>
-<span class="sourceLineNo">550</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.550"></a>
-<span class="sourceLineNo">551</span> timeoutExecutor.start();<a name="line.551"></a>
-<span class="sourceLineNo">552</span> for (WorkerThread worker: workerThreads) {<a name="line.552"></a>
-<span class="sourceLineNo">553</span> worker.start();<a name="line.553"></a>
-<span class="sourceLineNo">554</span> }<a name="line.554"></a>
-<span class="sourceLineNo">555</span><a name="line.555"></a>
-<span class="sourceLineNo">556</span> // Internal chores<a name="line.556"></a>
-<span class="sourceLineNo">557</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.557"></a>
-<span class="sourceLineNo">558</span><a name="line.558"></a>
-<span class="sourceLineNo">559</span> // Add completed cleaner chore<a name="line.559"></a>
-<span class="sourceLineNo">560</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.560"></a>
-<span class="sourceLineNo">561</span> }<a name="line.561"></a>
-<span class="sourceLineNo">562</span><a name="line.562"></a>
-<span class="sourceLineNo">563</span> public void stop() {<a name="line.563"></a>
-<span class="sourceLineNo">564</span> if (!running.getAndSet(false)) {<a name="line.564"></a>
-<span class="sourceLineNo">565</span> return;<a name="line.565"></a>
-<span class="sourceLineNo">566</span> }<a name="line.566"></a>
-<span class="sourceLineNo">567</span><a name="line.567"></a>
-<span class="sourceLineNo">568</span> LOG.info("Stopping");<a name="line.568"></a>
-<span class="sourceLineNo">569</span> scheduler.stop();<a name="line.569"></a>
-<span class="sourceLineNo">570</span> timeoutExecutor.sendStopSignal();<a name="line.570"></a>
-<span class="sourceLineNo">571</span> }<a name="line.571"></a>
-<span class="sourceLineNo">572</span><a name="line.572"></a>
-<span class="sourceLineNo">573</span> @VisibleForTesting<a name="line.573"></a>
-<span class="sourceLineNo">574</span> public void join() {<a name="line.574"></a>
-<span class="sourceLineNo">575</span> assert !isRunning() : "expected not running";<a name="line.575"></a>
-<span class="sourceLineNo">576</span><a name="line.576"></a>
-<span class="sourceLineNo">577</span> // stop the timeout executor<a name="line.577"></a>
-<span class="sourceLineNo">578</span> timeoutExecutor.awaitTermination();<a name="line.578"></a>
-<span class="sourceLineNo">579</span> timeoutExecutor = null;<a name="line.579"></a>
-<span class="sourceLineNo">580</span><a name="line.580"></a>
-<span class="sourceLineNo">581</span> // stop the worker threads<a name="line.581"></a>
-<span class="sourceLineNo">582</span> for (WorkerThread worker: workerThreads) {<a name="line.582"></a>
-<span class="sourceLineNo">583</span> worker.awaitTermination();<a name="line.583"></a>
-<span class="sourceLineNo">584</span> }<a name="line.584"></a>
-<span class="sourceLineNo">585</span> workerThreads = null;<a name="line.585"></a>
-<span class="sourceLineNo">586</span><a name="line.586"></a>
-<span class="sourceLineNo">587</span> // Destroy the Thread Group for the executors<a name="line.587"></a>
-<span class="sourceLineNo">588</span> try {<a name="line.588"></a>
-<span class="sourceLineNo">589</span> threadGroup.destroy();<a name="line.589"></a>
-<span class="sourceLineNo">590</span> } catch (IllegalThreadStateException e) {<a name="line.590"></a>
-<span class="sourceLineNo">591</span> LOG.error("ThreadGroup " + threadGroup + " contains running threads; " + e.getMessage());<a name="line.591"></a>
-<span class="sourceLineNo">592</span> threadGroup.list();<a name="line.592"></a>
-<span class="sourceLineNo">593</span> } finally {<a name="line.593"></a>
-<span class="sourceLineNo">594</span> threadGroup = null;<a name="line.594"></a>
-<span class="sourceLineNo">595</span> }<a name="line.595"></a>
+<span class="sourceLineNo">513</span> /**<a name="line.513"></a>
+<span class="sourceLineNo">514</span> * Start the procedure executor.<a name="line.514"></a>
+<span class="sourceLineNo">515</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.515"></a>
+<span class="sourceLineNo">516</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.516"></a>
+<span class="sourceLineNo">517</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.517"></a>
+<span class="sourceLineNo">518</span> *<a name="line.518"></a>
+<span class="sourceLineNo">519</span> * @param numThreads number of threads available for procedure execution.<a name="line.519"></a>
+<span class="sourceLineNo">520</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.520"></a>
+<span class="sourceLineNo">521</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.521"></a>
+<span class="sourceLineNo">522</span> */<a name="line.522"></a>
+<span class="sourceLineNo">523</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.523"></a>
+<span class="sourceLineNo">524</span> if (!running.compareAndSet(false, true)) {<a name="line.524"></a>
+<span class="sourceLineNo">525</span> LOG.warn("Already running");<a name="line.525"></a>
+<span class="sourceLineNo">526</span> return;<a name="line.526"></a>
+<span class="sourceLineNo">527</span> }<a name="line.527"></a>
+<span class="sourceLineNo">528</span><a name="line.528"></a>
+<span class="sourceLineNo">529</span> // We have numThreads executor + one timer thread used for timing out<a name="line.529"></a>
+<span class="sourceLineNo">530</span> // procedures and triggering periodic procedures.<a name="line.530"></a>
+<span class="sourceLineNo">531</span> this.corePoolSize = numThreads;<a name="line.531"></a>
+<span class="sourceLineNo">532</span> this.maxPoolSize = 10 * numThreads;<a name="line.532"></a>
+<span class="sourceLineNo">533</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.533"></a>
+<span class="sourceLineNo">534</span> corePoolSize, maxPoolSize);<a name="line.534"></a>
+<span class="sourceLineNo">535</span><a name="line.535"></a>
+<span class="sourceLineNo">536</span> this.threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.536"></a>
+<span class="sourceLineNo">537</span> this.timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.537"></a>
+<span class="sourceLineNo">538</span><a name="line.538"></a>
+<span class="sourceLineNo">539</span> // Create the workers<a name="line.539"></a>
+<span class="sourceLineNo">540</span> workerId.set(0);<a name="line.540"></a>
+<span class="sourceLineNo">541</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.541"></a>
+<span class="sourceLineNo">542</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.542"></a>
+<span class="sourceLineNo">543</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.543"></a>
+<span class="sourceLineNo">544</span> }<a name="line.544"></a>
+<span class="sourceLineNo">545</span><a name="line.545"></a>
+<span class="sourceLineNo">546</span> long st, et;<a name="line.546"></a>
+<span class="sourceLineNo">547</span><a name="line.547"></a>
+<span class="sourceLineNo">548</span> // Acquire the store lease.<a name="line.548"></a>
+<span class="sourceLineNo">549</span> st = EnvironmentEdgeManager.currentTime();<a name="line.549"></a>
+<span class="sourceLineNo">550</span> store.recoverLease();<a name="line.550"></a>
+<span class="sourceLineNo">551</span> et = EnvironmentEdgeManager.currentTime();<a name="line.551"></a>
+<span class="sourceLineNo">552</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.552"></a>
+<span class="sourceLineNo">553</span> StringUtils.humanTimeDiff(et - st));<a name="line.553"></a>
+<span class="sourceLineNo">554</span><a name="line.554"></a>
+<span class="sourceLineNo">555</span> // start the procedure scheduler<a name="line.555"></a>
+<span class="sourceLineNo">556</span> scheduler.start();<a name="line.556"></a>
+<span class="sourceLineNo">557</span><a name="line.557"></a>
+<span class="sourceLineNo">558</span> // TODO: Split in two steps.<a name="line.558"></a>
+<span class="sourceLineNo">559</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.559"></a>
+<span class="sourceLineNo">560</span> // The first one will make sure that we have the latest id,<a name="line.560"></a>
+<span class="sourceLineNo">561</span> // so we can start the threads and accept new procedures.<a name="line.561"></a>
+<span class="sourceLineNo">562</span> // The second step will do the actual load of old procedures.<a name="line.562"></a>
+<span class="sourceLineNo">563</span> st = EnvironmentEdgeManager.currentTime();<a name="line.563"></a>
+<span class="sourceLineNo">564</span> load(abortOnCorruption);<a name="line.564"></a>
+<span class="sourceLineNo">565</span> et = EnvironmentEdgeManager.currentTime();<a name="line.565"></a>
+<span class="sourceLineNo">566</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.566"></a>
+<span class="sourceLineNo">567</span> StringUtils.humanTimeDiff(et - st));<a name="line.567"></a>
+<span class="sourceLineNo">568</span><a name="line.568"></a>
+<span class="sourceLineNo">569</span> // Start the executors. Here we must have the lastProcId set.<a name="line.569"></a>
+<span class="sourceLineNo">570</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.570"></a>
+<span class="sourceLineNo">571</span> timeoutExecutor.start();<a name="line.571"></a>
+<span class="sourceLineNo">572</span> for (WorkerThread worker: workerThreads) {<a name="line.572"></a>
+<span class="sourceLineNo">573</span> worker.start();<a name="line.573"></a>
+<span class="sourceLineNo">574</span> }<a name="line.574"></a>
+<span class="sourceLineNo">575</span><a name="line.575"></a>
+<span class="sourceLineNo">576</span> // Internal chores<a name="line.576"></a>
+<span class="sourceLineNo">577</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.577"></a>
+<span class="sourceLineNo">578</span><a name="line.578"></a>
+<span class="sourceLineNo">579</span> // Add completed cleaner chore<a name="line.579"></a>
+<span class="sourceLineNo">580</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.580"></a>
+<span class="sourceLineNo">581</span> }<a name="line.581"></a>
+<span class="sourceLineNo">582</span><a name="line.582"></a>
+<span class="sourceLineNo">583</span> public void stop() {<a name="line.583"></a>
+<span class="sourceLineNo">584</span> if (!running.getAndSet(false)) {<a name="line.584"></a>
+<span class="sourceLineNo">585</span> return;<a name="line.585"></a>
+<span class="sourceLineNo">586</span> }<a name="line.586"></a>
+<span class="sourceLineNo">587</span><a name="line.587"></a>
+<span class="sourceLineNo">588</span> LOG.info("Stopping");<a name="line.588"></a>
+<span class="sourceLineNo">589</span> scheduler.stop();<a name="line.589"></a>
+<span class="sourceLineNo">590</span> timeoutExecutor.sendStopSignal();<a name="line.590"></a>
+<span class="sourceLineNo">591</span> }<a name="line.591"></a>
+<span class="sourceLineNo">592</span><a name="line.592"></a>
+<span class="sourceLineNo">593</span> @VisibleForTesting<a name="line.593"></a>
+<span class="sourceLineNo">594</span> public void join() {<a name="line.594"></a>
+<span class="sourceLineNo">595</span> assert !isRunning() : "expected not running";<a name="line.595"></a>
<span class="sourceLineNo">596</span><a name="line.596"></a>
-<span class="sourceLineNo">597</span> // reset the in-memory state for testing<a name="line.597"></a>
-<span class="sourceLineNo">598</span> completed.clear();<a name="line.598"></a>
-<span class="sourceLineNo">599</span> rollbackStack.clear();<a name="line.599"></a>
-<span class="sourceLineNo">600</span> procedures.clear();<a name="line.600"></a>
-<span class="sourceLineNo">601</span> nonceKeysToProcIdsMap.clear();<a name="line.601"></a>
-<span class="sourceLineNo">602</span> scheduler.clear();<a name="line.602"></a>
-<span class="sourceLineNo">603</span> lastProcId.set(-1);<a name="line.603"></a>
-<span class="sourceLineNo">604</span> }<a name="line.604"></a>
-<span class="sourceLineNo">605</span><a name="line.605"></a>
-<span class="sourceLineNo">606</span> public void refreshConfiguration(final Configuration conf) {<a name="line.606"></a>
-<span class="sourceLineNo">607</span> this.conf = conf;<a name="line.607"></a>
-<span class="sourceLineNo">608</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.608"></a>
-<span class="sourceLineNo">609</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.609"></a>
-<span class="sourceLineNo">610</span> }<a name="line.610"></a>
-<span class="sourceLineNo">611</span><a name="line.611"></a>
-<span class="sourceLineNo">612</span> // ==========================================================================<a name="line.612"></a>
-<span class="sourceLineNo">613</span> // Accessors<a name="line.613"></a>
-<span class="sourceLineNo">614</span> // ==========================================================================<a name="line.614"></a>
-<span class="sourceLineNo">615</span> public boolean isRunning() {<a name="line.615"></a>
-<span class="sourceLineNo">616</span> return running.get();<a name="line.616"></a>
-<span class="sourceLineNo">617</span> }<a name="line.617"></a>
-<span class="sourceLineNo">618</span><a name="line.618"></a>
-<span class="sourceLineNo">619</span> /**<a name="line.619"></a>
-<span class="sourceLineNo">620</span> * @return the current number of worker threads.<a name="line.620"></a>
-<span class="sourceLineNo">621</span> */<a name="line.621"></a>
-<span class="sourceLineNo">622</span> public int getWorkerThreadCount() {<a name="line.622"></a>
-<span class="sourceLineNo">623</span> return workerThreads.size();<a name="line.623"></a>
-<span class="sourceLineNo">624</span> }<a name="line.624"></a>
-<span class="sourceLineNo">625</span><a name="line.625"></a>
-<span class="sourceLineNo">626</span> /**<a name="line.626"></a>
-<span class="sourceLineNo">627</span> * @return the core pool size settings.<a name="line.627"></a>
-<span class="sourceLineNo">628</span> */<a name="line.628"></a>
-<span class="sourceLineNo">629</span> public int getCorePoolSize() {<a name="line.629"></a>
-<span class="sourceLineNo">630</span> return corePoolSize;<a name="line.630"></a>
-<span class="sourceLineNo">631</span> }<a name="line.631"></a>
-<span class="sourceLineNo">632</span><a name="line.632"></a>
-<span class="sourceLineNo">633</span> public int getActiveExecutorCount() {<a name="line.633"></a>
-<span class="sourceLineNo">634</span> return activeExecutorCount.get();<a name="line.634"></a>
-<span class="sourceLineNo">635</span> }<a name="line.635"></a>
-<span class="sourceLineNo">636</span><a name="line.636"></a>
-<span class="sourceLineNo">637</span> public TEnvironment getEnvironment() {<a name="line.637"></a>
-<span class="sourceLineNo">638</span> return this.environment;<a name="line.638"></a>
-<span class="sourceLineNo">639</span> }<a name="line.639"></a>
-<span class="sourceLineNo">640</span><a name="line.640"></a>
-<span class="sourceLineNo">641</span> public ProcedureStore getStore() {<a name="line.641"></a>
-<span class="sourceLineNo">642</span> return this.store;<a name="line.642"></a>
+<span class="sourceLineNo">597</span> // stop the timeout executor<a name="line.597"></a>
+<span class="sourceLineNo">598</span> timeoutExecutor.awaitTermination();<a name="line.598"></a>
+<span class="sourceLineNo">599</span><a name="line.599"></a>
+<span class="sourceLineNo">600</span> // stop the worker threads<a name="line.600"></a>
+<span class="sourceLineNo">601</span> for (WorkerThread worker: workerThreads) {<a name="line.601"></a>
+<span class="sourceLineNo">602</span> worker.awaitTermination();<a name="line.602"></a>
+<span class="sourceLineNo">603</span> }<a name="line.603"></a>
+<span class="sourceLineNo">604</span><a name="line.604"></a>
+<span class="sourceLineNo">605</span> // Destroy the Thread Group for the executors<a name="line.605"></a>
+<span class="sourceLineNo">606</span> // TODO: Fix. #join is not place to destroy resources.<a name="line.606"></a>
+<span class="sourceLineNo">607</span> try {<a name="line.607"></a>
+<span class="sourceLineNo">608</span> threadGroup.destroy();<a name="line.608"></a>
+<span class="sourceLineNo">609</span> } catch (IllegalThreadStateException e) {<a name="line.609"></a>
+<span class="sourceLineNo">610</span> LOG.error("ThreadGroup {} contains running threads; {}: See STDOUT",<a name="line.610"></a>
+<span class="sourceLineNo">611</span> this.threadGroup, e.getMessage());<a name="line.611"></a>
+<span class="sourceLineNo">612</span> // This dumps list of threads on STDOUT.<a name="line.612"></a>
+<span class="sourceLineNo">613</span> this.threadGroup.list();<a name="line.613"></a>
+<span class="sourceLineNo">614</span> }<a name="line.614"></a>
+<span class="sourceLineNo">615</span><a name="line.615"></a>
+<span class="sourceLineNo">616</span> // reset the in-memory state for testing<a name="line.616"></a>
+<span class="sourceLineNo">617</span> completed.clear();<a name="line.617"></a>
+<span class="sourceLineNo">618</span> rollbackStack.clear();<a name="line.618"></a>
+<span class="sourceLineNo">619</span> procedures.clear();<a name="line.619"></a>
+<span class="sourceLineNo">620</span> nonceKeysToProcIdsMap.clear();<a name="line.620"></a>
+<span class="sourceLineNo">621</span> scheduler.clear();<a name="line.621"></a>
+<span class="sourceLineNo">622</span> lastProcId.set(-1);<a name="line.622"></a>
+<span class="sourceLineNo">623</span> }<a name="line.623"></a>
+<span class="sourceLineNo">624</span><a name="line.624"></a>
+<span class="sourceLineNo">625</span> public void refreshConfiguration(final Configuration conf) {<a name="line.625"></a>
+<span class="sourceLineNo">626</span> this.conf = conf;<a name="line.626"></a>
+<span class="sourceLineNo">627</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.627"></a>
+<span class="sourceLineNo">628</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.628"></a>
+<span class="sourceLineNo">629</span> }<a name="line.629"></a>
+<span class="sourceLineNo">630</span><a name="line.630"></a>
+<span class="sourceLineNo">631</span> // ==========================================================================<a name="line.631"></a>
+<span class="sourceLineNo">632</span> // Accessors<a name="line.632"></a>
+<span class="sourceLineNo">633</span> // ==========================================================================<a name="line.633"></a>
+<span class="sourceLineNo">634</span> public boolean isRunning() {<a name="line.634"></a>
+<span class="sourceLineNo">635</span> return running.get();<a name="line.635"></a>
+<span class="sourceLineNo">636</span> }<a name="line.636"></a>
+<span class="sourceLineNo">637</span><a name="line.637"></a>
+<span class="sourceLineNo">638</span> /**<a name="line.638"></a>
+<span class="sourceLineNo">639</span> * @return the current number of worker threads.<a name="line.639"></a>
+<span class="sourceLineNo">640</span> */<a name="line.640"></a>
+<span class="sourceLineNo">641</span> public int getWorkerThreadCount() {<a name="line.641"></a>
+<span class="sourceLineNo">642</span> return workerThreads.size();<a name="line.642"></a>
<span class="sourceLineNo">643</span> }<a name="line.643"></a>
<span class="sourceLineNo">644</span><a name="line.644"></a>
-<span class="sourceLineNo">645</span> ProcedureScheduler getScheduler() {<a name="line.645"></a>
-<span class="sourceLineNo">646</span> return scheduler;<a name="line.646"></a>
-<span class="sourceLineNo">647</span> }<a name="line.647"></a>
-<span class="sourceLineNo">648</span><a name="line.648"></a>
-<span class="sourceLineNo">649</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.649"></a>
-<span class="sourceLineNo">650</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.650"></a>
-<span class="sourceLineNo">651</span> this.scheduler.signalAll();<a name="line.651"></a>
-<span class="sourceLineNo">652</span> }<a name="line.652"></a>
-<span class="sourceLineNo">653</span><a name="line.653"></a>
-<span class="sourceLineNo">654</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.654"></a>
-<span class="sourceLineNo">655</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.655"></a>
-<span class="sourceLineNo">656</span> }<a name="line.656"></a>
-<span class="sourceLineNo">657</span><a name="line.657"></a>
-<span class="sourceLineNo">658</span> // ==========================================================================<a name="line.658"></a>
-<span class="sourceLineNo">659</span> // Submit/Remove Chores<a name="line.659"></a>
-<span class="sourceLineNo">660</span> // ==========================================================================<a name="line.660"></a>
-<span class="sourceLineNo">661</span><a name="line.661"></a>
-<span class="sourceLineNo">662</span> /**<a name="line.662"></a>
-<span class="sourceLineNo">663</span> * Add a chore procedure to the executor<a name="line.663"></a>
-<span class="sourceLineNo">664</span> * @param chore the chore to add<a name="line.664"></a>
-<span class="sourceLineNo">665</span> */<a name="line.665"></a>
-<span class="sourceLineNo">666</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.666"></a>
-<span class="sourceLineNo">667</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.667"></a>
-<span class="sourceLineNo">668</span> timeoutExecutor.add(chore);<a name="line.668"></a>
-<span class="sourceLineNo">669</span> }<a name="line.669"></a>
-<span class="sourceLineNo">670</span><a name="line.670"></a>
-<span class="sourceLineNo">671</span> /**<a name="line.671"></a>
-<span class="sourceLineNo">672</span> * Remove a chore procedure from the executor<a name="line.672"></a>
-<span class="sourceLineNo">673</span> * @param chore the chore to remove<a name="line.673"></a>
-<span class="sourceLineNo">674</span> * @return whether the chore is removed, or it will be removed later<a name="line.674"></a>
-<span class="sourceLineNo">675</span> */<a name="line.675"></a>
-<span class="sourceLineNo">676</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.676"></a>
-<span class="sourceLineNo">677</span> chore.setState(ProcedureState.SUCCESS);<a name="line.677"></a>
-<span class="sourceLineNo">678</span> return timeoutExecutor.remove(chore);<a name="line.678"></a>
-<span class="sourceLineNo">679</span> }<a name="line.679"></a>
+<span class="sourceLineNo">645</span> /**<a name="line.645"></a>
+<span class="sourceLineNo">646</span> * @return the core pool size settings.<a name="line.646"></a>
+<span class="sourceLineNo">647</span> */<a name="line.647"></a>
+<span class="sourceLineNo">648</span> public int getCorePoolSize() {<a name="line.648"></a>
+<span class="sourceLineNo">649</span> return corePoolSize;<a name="line.649"></a>
+<span class="sourceLineNo">650</span> }<a name="line.650"></a>
+<span class="sourceLineNo">651</span><a name="line.651"></a>
+<span class="sourceLineNo">652</span> public int getActiveExecutorCount() {<a name="line.652"></a>
+<span class="sourceLineNo">653</span> return activeExecutorCount.get();<a name="line.653"></a>
+<span class="sourceLineNo">654</span> }<a name="line.654"></a>
+<span class="sourceLineNo">655</span><a name="line.655"></a>
+<span class="sourceLineNo">656</span> public TEnvironment getEnvironment() {<a name="line.656"></a>
+<span class="sourceLineNo">657</span> return this.environment;<a name="line.657"></a>
+<span class="sourceLineNo">658</span> }<a name="line.658"></a>
+<span class="sourceLineNo">659</span><a name="line.659"></a>
+<span class="sourceLineNo">660</span> public ProcedureStore getStore() {<a name="line.660"></a>
+<span class="sourceLineNo">661</span> return this.store;<a name="line.661"></a>
+<span class="sourceLineNo">662</span> }<a name="line.662"></a>
+<span class="sourceLineNo">663</span><a name="line.663"></a>
+<span class="sourceLineNo">664</span> ProcedureScheduler getScheduler() {<a name="line.664"></a>
+<span class="sourceLineNo">665</span> return scheduler;<a name="line.665"></a>
+<span class="sourceLineNo">666</span> }<a name="line.666"></a>
+<span class="sourceLineNo">667</span><a name="line.667"></a>
+<span class="sourceLineNo">668</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.668"></a>
+<span class="sourceLineNo">669</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.669"></a>
+<span class="sourceLineNo">670</span> this.scheduler.signalAll();<a name="line.670"></a>
+<span class="sourceLineNo">671</span> }<a name="line.671"></a>
+<span class="sourceLineNo">672</span><a name="line.672"></a>
+<span class="sourceLineNo">673</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.673"></a>
+<span class="sourceLineNo">674</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.674"></a>
+<span class="sourceLineNo">675</span> }<a name="line.675"></a>
+<span class="sourceLineNo">676</span><a name="line.676"></a>
+<span class="sourceLineNo">677</span> // ==========================================================================<a name="line.677"></a>
+<span class="sourceLineNo">678</span> // Submit/Remove Chores<a name="line.678"></a>
+<span class="sourceLineNo">679</span> // ==========================================================================<a name="line.679"></a>
<span class="sourceLineNo">680</span><a name="line.680"></a>
-<span class="sourceLineNo">681</span> // ==========================================================================<a name="line.681"></a>
-<span class="sourceLineNo">682</span> // Nonce Procedure helpers<a name="line.682"></a>
-<span class="sourceLineNo">683</span> // ==========================================================================<a name="line.683"></a>
-<span class="sourceLineNo">684</span> /**<a name="line.684"></a>
-<span class="sourceLineNo">685</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.685"></a>
-<span class="sourceLineNo">686</span> * @param nonceGroup<a name="line.686"></a>
-<span class="sourceLineNo">687</span> * @param nonce<a name="line.687"></a>
-<span class="sourceLineNo">688</span> * @return the generated NonceKey<a name="line.688"></a>
-<span class="sourceLineNo">689</span> */<a name="line.689"></a>
-<span class="sourceLineNo">690</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.690"></a>
-<span class="sourceLineNo">691</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.691"></a>
-<span class="sourceLineNo">692</span> }<a name="line.692"></a>
-<span class="sourceLineNo">693</span><a name="line.693"></a>
-<span class="sourceLineNo">694</span> /**<a name="line.694"></a>
-<span class="sourceLineNo">695</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.695"></a>
-<span class="sourceLineNo">696</span> * A procId will be reserved and on submitProcedure(),<a name="line.696"></a>
-<span class="sourceLineNo">697</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.697"></a>
-<span class="sourceLineNo">698</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.698"></a>
-<span class="sourceLineNo">699</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.699"></a>
-<span class="sourceLineNo">700</span> * and submit the procedure.<a name="line.700"></a>
-<span class="sourceLineNo">701</span> *<a name="line.701"></a>
-<span class="sourceLineNo">702</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.702"></a>
-<span class="sourceLineNo">703</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.703"></a>
-<span class="sourceLineNo">704</span> */<a name="line.704"></a>
-<span class="sourceLineNo">705</span> public long registerNonce(final NonceKey nonceKey) {<a name="line.705"></a>
-<span class="sourceLineNo">706</span> if (nonceKey == null) return -1;<a name="line.706"></a>
-<span class="sourceLineNo">707</span><a name="line.707"></a>
-<span class="sourceLineNo">708</span> // check if we have already a Reserved ID for the nonce<a name="line.708"></a>
-<span class="sourceLineNo">709</span> Long oldProcId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.709"></a>
-<span class="sourceLineNo">710</span> if (oldProcId == null) {<a name="line.710"></a>
-<span class="sourceLineNo">711</span> // reserve a new Procedure ID, this will be associated with the nonce<a name="line.711"></a>
-<span class="sourceLineNo">712</span> // and the procedure submitted with the specified nonce will use this ID.<a name="line.712"></a>
-<span class="sourceLineNo">713</span> final long newProcId = nextProcId();<a name="line.713"></a>
-<span class="sourceLineNo">714</span> oldProcId = nonceKeysToProcIdsMap.putIfAbsent(nonceKey, newProcId);<a name="line.714"></a>
-<span class="sourceLineNo">715</span> if (oldProcId == null) return -1;<a name="line.715"></a>
-<span class="sourceLineNo">716</span> }<a name="line.716"></a>
-<span class="sourceLineNo">717</span><a name="line.717"></a>
-<span class="sourceLineNo">718</span> // we found a registered nonce, but the procedure may not have been submitted yet.<a name="line.718"></a>
-<span class="sourceLineNo">719</span> // since the client expect the procedure to be submitted, spin here until it is.<a name="line.719"></a>
-<span class="sourceLineNo">720</span> final boolean traceEnabled = LOG.isTraceEnabled();<a name="line.720"></a>
-<span class="sourceLineNo">721</span> while (isRunning() &&<a name="line.721"></a>
-<span class="sourceLineNo">722</span> !(procedures.containsKey(oldProcId) || completed.containsKey(oldProcId)) &&<a name="line.722"></a>
-<span class="sourceLineNo">723</span> nonceKeysToProcIdsMap.containsKey(nonceKey)) {<a name="line.723"></a>
-<span class="sourceLineNo">724</span> if (traceEnabled) {<a name="line.724"></a>
-<span class="sourceLineNo">725</span> LOG.trace("Waiting for pid=" + oldProcId.longValue() + " to be submitted");<a name="line.725"></a>
-<span class="sourceLineNo">726</span> }<a name="line.726"></a>
-<span class="sourceLineNo">727</span> Threads.sleep(100);<a name="line.727"></a>
-<span class="sourceLineNo">728</span> }<a name="line.728"></a>
-<span class="sourceLineNo">729</span> return oldProcId.longValue();<a name="line.729"></a>
-<span class="sourceLineNo">730</span> }<a name="line.730"></a>
-<span class="sourceLineNo">731</span><a name="line.731"></a>
-<span class="sourceLineNo">732</span> /**<a name="line.732"></a>
-<span class="sourceLineNo">733</span> * Remove the NonceKey if the procedure was not submitted to the executor.<a name="line.733"></a>
-<span class="sourceLineNo">734</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.734"></a>
-<span class="sourceLineNo">735</span> */<a name="line.735"></a>
-<span class="sourceLineNo">736</span> public void unregisterNonceIfProcedureWasNotSubmitted(final NonceKey nonceKey) {<a name="line.736"></a>
-<span class="sourceLineNo">737</span> if (nonceKey == null) return;<a name="line.737"></a>
-<span class="sourceLineNo">738</span><a name="line.738"></a>
-<span class="sourceLineNo">739</span> final Long procId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.739"></a>
-<span class="sourceLineNo">740</span> if (procId == null) return;<a name="line.740"></a>
-<span class="sourceLineNo">741</span><a name="line.741"></a>
-<span class="sourceLineNo">742</span> // if the procedure was not submitted, remove the nonce<a name="line.742"></a>
-<span class="sourceLineNo">743</span> if (!(procedures.containsKey(procId) || completed.containsKey(procId))) {<a name="line.743"></a>
-<span class="sourceLineNo">744</span> nonceKeysToProcIdsMap.remove(nonceKey);<a name="line.744"></a>
-<span class="sourceLineNo">745</span> }<a name="line.745"></a>
-<span class="sourceLineNo">746</span> }<a name="line.746"></a>
-<span class="sourceLineNo">747</span><a name="line.747"></a>
-<span class="sourceLineNo">748</span> public static class FailedProcedure<TEnvironment> extends Procedure<TEnvironment> {<a name="line.748"></a>
-<span class="sourceLineNo">749</span> private String procName;<a name="line.749"></a>
+<span class="sourceLineNo">681</span> /**<a name="line.681"></a>
+<span class="sourceLineNo">682</span> * Add a chore procedure to the executor<a name="line.682"></a>
+<span class="sourceLineNo">683</span> * @param chore the chore to add<a name="line.683"></a>
+<span class="sourceLineNo">684</span> */<a name="line.684"></a>
+<span class="sourceLineNo">685</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.685"></a>
+<span class="sourceLineNo">686</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.686"></a>
+<span class="sourceLineNo">687</span> timeoutExecutor.add(chore);<a name="line.687"></a>
+<span class="sourceLineNo">688</span> }<a name="line.688"></a>
+<span class="sourceLineNo">689</span><a name="line.689"></a>
+<span class="sourceLineNo">690</span> /**<a name="line.690"></a>
+<span class="sourceLineNo">691</span> * Remove a chore procedure from the executor<a name="line.691"></a>
+<span class="sourceLineNo">692</span> * @param chore the chore to remove<a name="line.692"></a>
+<span class="sourceLineNo">693</span> * @return whether the chore is removed, or it will be removed later<a name="line.693"></a>
+<span class="sourceLineNo">694</span> */<a name="line.694"></a>
+<span class="sourceLineNo">695</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.695"></a>
+<span class="sourceLineNo">696</span> chore.setState(ProcedureState.SUCCESS);<a name="line.696"></a>
+<span class="sourceLineNo">697</span> return timeoutExecutor.remove(chore);<a name="line.697"></a>
+<span class="sourceLineNo">698</span> }<a name="line.698"></a>
+<span class="sourceLineNo">699</span><a name="line.699"></a>
+<span class="sourceLineNo">700</span> // ==========================================================================<a name="line.700"></a>
+<span class="sourceLineNo">701</span> // Nonce Procedure helpers<a name="line.701"></a>
+<span class="sourceLineNo">702</span> // ==========================================================================<a name="line.702"></a>
+<span class="sourceLineNo">703</span> /**<a name="line.703"></a>
+<span class="sourceLineNo">704</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.704"></a>
+<span class="sourceLineNo">705</span> * @param nonceGroup<a name="line.705"></a>
+<span class="sourceLineNo">706</span> * @param nonce<a name="line.706"></a>
+<span class="sourceLineNo">707</span> * @return the generated NonceKey<a name="line.707"></a>
+<span class="sourceLineNo">708</span> */<a name="line.708"></a>
+<span class="sourceLineNo">709</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.709"></a>
+<span class="sourceLineNo">710</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.710"></a>
+<span class="sourceLineNo">711</span> }<a name="line.711"></a>
+<span class="sourceLineNo">712</span><a name="line.712"></a>
+<span class="sourceLineNo">713</span> /**<a name="line.713"></a>
+<span class="sourceLineNo">714</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.714"></a>
+<span class="sourceLineNo">715</span> * A procId will be reserved and on submitProcedure(),<a name="line.715"></a>
+<span class="sourceLineNo">716</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.716"></a>
+<span class="sourceLineNo">717</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.717"></a>
+<span class="sourceLineNo">718</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.718"></a>
+<span class="sourceLineNo">719</span> * and submit the procedure.<a name="line.719"></a>
+<span class="sourceLineNo">720</span> *<a name="line.720"></a>
+<span class="sourceLineNo">721</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.721"></a>
+<span class="sourceLineNo">722</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.722"></a>
+<span class="source
<TRUNCATED>
[11/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html b/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html
index 078e4b1..fe1e077 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.html
@@ -1265,700 +1265,701 @@
<span class="sourceLineNo">1257</span> } else if (localState == State.OFFLINE || regionInfo.isOffline()) {<a name="line.1257"></a>
<span class="sourceLineNo">1258</span> regionStates.addToOfflineRegions(regionNode);<a name="line.1258"></a>
<span class="sourceLineNo">1259</span> } else if (localState == State.CLOSED && getTableStateManager().<a name="line.1259"></a>
-<span class="sourceLineNo">1260</span> isTableState(regionNode.getTable(), TableState.State.DISABLED)) {<a name="line.1260"></a>
-<span class="sourceLineNo">1261</span> // The region is CLOSED and the table is DISABLED, there is nothing to schedule;<a name="line.1261"></a>
-<span class="sourceLineNo">1262</span> // the region is inert.<a name="line.1262"></a>
-<span class="sourceLineNo">1263</span> } else {<a name="line.1263"></a>
-<span class="sourceLineNo">1264</span> // These regions should have a procedure in replay<a name="line.1264"></a>
-<span class="sourceLineNo">1265</span> regionStates.addRegionInTransition(regionNode, null);<a name="line.1265"></a>
-<span class="sourceLineNo">1266</span> }<a name="line.1266"></a>
-<span class="sourceLineNo">1267</span> }<a name="line.1267"></a>
-<span class="sourceLineNo">1268</span> }<a name="line.1268"></a>
-<span class="sourceLineNo">1269</span> }<a name="line.1269"></a>
-<span class="sourceLineNo">1270</span> });<a name="line.1270"></a>
-<span class="sourceLineNo">1271</span><a name="line.1271"></a>
-<span class="sourceLineNo">1272</span> // every assignment is blocked until meta is loaded.<a name="line.1272"></a>
-<span class="sourceLineNo">1273</span> wakeMetaLoadedEvent();<a name="line.1273"></a>
-<span class="sourceLineNo">1274</span> }<a name="line.1274"></a>
-<span class="sourceLineNo">1275</span><a name="line.1275"></a>
-<span class="sourceLineNo">1276</span> /**<a name="line.1276"></a>
-<span class="sourceLineNo">1277</span> * Look at what is in meta and the list of servers that have checked in and make reconciliation.<a name="line.1277"></a>
-<span class="sourceLineNo">1278</span> * We cannot tell definitively the difference between a clean shutdown and a cluster that has<a name="line.1278"></a>
-<span class="sourceLineNo">1279</span> * been crashed down. At this stage of a Master startup, they look the same: they have the<a name="line.1279"></a>
-<span class="sourceLineNo">1280</span> * same state in hbase:meta. We could do detective work probing ZK and the FS for old WALs to<a name="line.1280"></a>
-<span class="sourceLineNo">1281</span> * split but SCP does this already so just let it do its job.<a name="line.1281"></a>
-<span class="sourceLineNo">1282</span> * <p>>The profiles of clean shutdown and cluster crash-down are the same because on clean<a name="line.1282"></a>
-<span class="sourceLineNo">1283</span> * shutdown currently, we do not update hbase:meta with region close state (In AMv2, region<a name="line.1283"></a>
-<span class="sourceLineNo">1284</span> * state is kept in hbse:meta). Usually the master runs all region transitions as of AMv2 but on<a name="line.1284"></a>
-<span class="sourceLineNo">1285</span> * cluster controlled shutdown, the RegionServers close all their regions only reporting the<a name="line.1285"></a>
-<span class="sourceLineNo">1286</span> * final change to the Master. Currently this report is ignored. Later we could take it and<a name="line.1286"></a>
-<span class="sourceLineNo">1287</span> * update as many regions as we can before hbase:meta goes down or have the master run the<a name="line.1287"></a>
-<span class="sourceLineNo">1288</span> * close of all regions out on the cluster but we may never be able to achieve the proper state on<a name="line.1288"></a>
-<span class="sourceLineNo">1289</span> * all regions (at least not w/o lots of painful manipulations and waiting) so clean shutdown<a name="line.1289"></a>
-<span class="sourceLineNo">1290</span> * might not be possible especially on big clusters.... And clean shutdown will take time. Given<a name="line.1290"></a>
-<span class="sourceLineNo">1291</span> * this current state of affairs, we just run ServerCrashProcedure in both cases. It will always<a name="line.1291"></a>
-<span class="sourceLineNo">1292</span> * do the right thing.<a name="line.1292"></a>
-<span class="sourceLineNo">1293</span> * @return True if for sure this is a failover where a Master is starting up into an already<a name="line.1293"></a>
-<span class="sourceLineNo">1294</span> * running cluster.<a name="line.1294"></a>
-<span class="sourceLineNo">1295</span> */<a name="line.1295"></a>
-<span class="sourceLineNo">1296</span> // The assumption here is that if RSs are crashing while we are executing this<a name="line.1296"></a>
-<span class="sourceLineNo">1297</span> // they will be handled by the SSH that are put in the ServerManager deadservers "queue".<a name="line.1297"></a>
-<span class="sourceLineNo">1298</span> private boolean processofflineServersWithOnlineRegions() {<a name="line.1298"></a>
-<span class="sourceLineNo">1299</span> boolean deadServers = !master.getServerManager().getDeadServers().isEmpty();<a name="line.1299"></a>
-<span class="sourceLineNo">1300</span> final Set<ServerName> offlineServersWithOnlineRegions = new HashSet<>();<a name="line.1300"></a>
-<span class="sourceLineNo">1301</span> int size = regionStates.getRegionStateNodes().size();<a name="line.1301"></a>
-<span class="sourceLineNo">1302</span> final List<RegionInfo> offlineRegionsToAssign = new ArrayList<>(size);<a name="line.1302"></a>
-<span class="sourceLineNo">1303</span> // If deadservers then its a failover, else, we are not sure yet.<a name="line.1303"></a>
-<span class="sourceLineNo">1304</span> boolean failover = deadServers;<a name="line.1304"></a>
-<span class="sourceLineNo">1305</span> for (RegionStateNode regionNode: regionStates.getRegionStateNodes()) {<a name="line.1305"></a>
-<span class="sourceLineNo">1306</span> // Region State can be OPEN even if we did controlled cluster shutdown; Master does not close<a name="line.1306"></a>
-<span class="sourceLineNo">1307</span> // the regions in this case. The RegionServer does the close so hbase:meta is state in<a name="line.1307"></a>
-<span class="sourceLineNo">1308</span> // hbase:meta is not updated -- Master does all updates -- and is left with OPEN as region<a name="line.1308"></a>
-<span class="sourceLineNo">1309</span> // state in meta. How to tell difference between ordered shutdown and crashed-down cluster<a name="line.1309"></a>
-<span class="sourceLineNo">1310</span> // then? We can't. Not currently. Perhaps if we updated hbase:meta with CLOSED on ordered<a name="line.1310"></a>
-<span class="sourceLineNo">1311</span> // shutdown. This would slow shutdown though and not all edits would make it in anyways.<a name="line.1311"></a>
-<span class="sourceLineNo">1312</span> // TODO: Examine.<a name="line.1312"></a>
-<span class="sourceLineNo">1313</span> // Because we can't be sure it an ordered shutdown, we run ServerCrashProcedure always.<a name="line.1313"></a>
-<span class="sourceLineNo">1314</span> // ServerCrashProcedure will try to retain old deploy when it goes to assign.<a name="line.1314"></a>
-<span class="sourceLineNo">1315</span> if (regionNode.getState() == State.OPEN) {<a name="line.1315"></a>
-<span class="sourceLineNo">1316</span> final ServerName serverName = regionNode.getRegionLocation();<a name="line.1316"></a>
-<span class="sourceLineNo">1317</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1317"></a>
-<span class="sourceLineNo">1318</span> offlineServersWithOnlineRegions.add(serverName);<a name="line.1318"></a>
-<span class="sourceLineNo">1319</span> } else {<a name="line.1319"></a>
-<span class="sourceLineNo">1320</span> // Server is online. This a failover. Master is starting into already-running cluster.<a name="line.1320"></a>
-<span class="sourceLineNo">1321</span> failover = true;<a name="line.1321"></a>
-<span class="sourceLineNo">1322</span> }<a name="line.1322"></a>
-<span class="sourceLineNo">1323</span> } else if (regionNode.getState() == State.OFFLINE) {<a name="line.1323"></a>
-<span class="sourceLineNo">1324</span> if (isTableEnabled(regionNode.getTable())) {<a name="line.1324"></a>
-<span class="sourceLineNo">1325</span> offlineRegionsToAssign.add(regionNode.getRegionInfo());<a name="line.1325"></a>
-<span class="sourceLineNo">1326</span> }<a name="line.1326"></a>
-<span class="sourceLineNo">1327</span> }<a name="line.1327"></a>
-<span class="sourceLineNo">1328</span> }<a name="line.1328"></a>
-<span class="sourceLineNo">1329</span> // Kill servers with online regions just-in-case. Runs ServerCrashProcedure.<a name="line.1329"></a>
-<span class="sourceLineNo">1330</span> for (ServerName serverName: offlineServersWithOnlineRegions) {<a name="line.1330"></a>
-<span class="sourceLineNo">1331</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1331"></a>
-<span class="sourceLineNo">1332</span> LOG.info("KILL RegionServer=" + serverName + " hosting regions but not online.");<a name="line.1332"></a>
-<span class="sourceLineNo">1333</span> killRegionServer(serverName);<a name="line.1333"></a>
-<span class="sourceLineNo">1334</span> }<a name="line.1334"></a>
-<span class="sourceLineNo">1335</span> }<a name="line.1335"></a>
-<span class="sourceLineNo">1336</span> setFailoverCleanupDone(true);<a name="line.1336"></a>
-<span class="sourceLineNo">1337</span><a name="line.1337"></a>
-<span class="sourceLineNo">1338</span> // Assign offline regions. Uses round-robin.<a name="line.1338"></a>
-<span class="sourceLineNo">1339</span> if (offlineRegionsToAssign.size() > 0) {<a name="line.1339"></a>
-<span class="sourceLineNo">1340</span> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().<a name="line.1340"></a>
-<span class="sourceLineNo">1341</span> createRoundRobinAssignProcedures(offlineRegionsToAssign));<a name="line.1341"></a>
-<span class="sourceLineNo">1342</span> }<a name="line.1342"></a>
-<span class="sourceLineNo">1343</span><a name="line.1343"></a>
-<span class="sourceLineNo">1344</span> return failover;<a name="line.1344"></a>
-<span class="sourceLineNo">1345</span> }<a name="line.1345"></a>
-<span class="sourceLineNo">1346</span><a name="line.1346"></a>
-<span class="sourceLineNo">1347</span> /**<a name="line.1347"></a>
-<span class="sourceLineNo">1348</span> * Used by ServerCrashProcedure to make sure AssignmentManager has completed<a name="line.1348"></a>
-<span class="sourceLineNo">1349</span> * the failover cleanup before re-assigning regions of dead servers. So that<a name="line.1349"></a>
-<span class="sourceLineNo">1350</span> * when re-assignment happens, AssignmentManager has proper region states.<a name="line.1350"></a>
-<span class="sourceLineNo">1351</span> */<a name="line.1351"></a>
-<span class="sourceLineNo">1352</span> public boolean isFailoverCleanupDone() {<a name="line.1352"></a>
-<span class="sourceLineNo">1353</span> return failoverCleanupDone.isReady();<a name="line.1353"></a>
-<span class="sourceLineNo">1354</span> }<a name="line.1354"></a>
-<span class="sourceLineNo">1355</span><a name="line.1355"></a>
-<span class="sourceLineNo">1356</span> /**<a name="line.1356"></a>
-<span class="sourceLineNo">1357</span> * Used by ServerCrashProcedure tests verify the ability to suspend the<a name="line.1357"></a>
-<span class="sourceLineNo">1358</span> * execution of the ServerCrashProcedure.<a name="line.1358"></a>
-<span class="sourceLineNo">1359</span> */<a name="line.1359"></a>
-<span class="sourceLineNo">1360</span> @VisibleForTesting<a name="line.1360"></a>
-<span class="sourceLineNo">1361</span> public void setFailoverCleanupDone(final boolean b) {<a name="line.1361"></a>
-<span class="sourceLineNo">1362</span> master.getMasterProcedureExecutor().getEnvironment()<a name="line.1362"></a>
-<span class="sourceLineNo">1363</span> .setEventReady(failoverCleanupDone, b);<a name="line.1363"></a>
-<span class="sourceLineNo">1364</span> }<a name="line.1364"></a>
-<span class="sourceLineNo">1365</span><a name="line.1365"></a>
-<span class="sourceLineNo">1366</span> public ProcedureEvent getFailoverCleanupEvent() {<a name="line.1366"></a>
-<span class="sourceLineNo">1367</span> return failoverCleanupDone;<a name="line.1367"></a>
-<span class="sourceLineNo">1368</span> }<a name="line.1368"></a>
-<span class="sourceLineNo">1369</span><a name="line.1369"></a>
-<span class="sourceLineNo">1370</span> /**<a name="line.1370"></a>
-<span class="sourceLineNo">1371</span> * Used to check if the failover cleanup is done.<a name="line.1371"></a>
-<span class="sourceLineNo">1372</span> * if not we throw PleaseHoldException since we are rebuilding the RegionStates<a name="line.1372"></a>
-<span class="sourceLineNo">1373</span> * @param hri region to check if it is already rebuild<a name="line.1373"></a>
-<span class="sourceLineNo">1374</span> * @throws PleaseHoldException if the failover cleanup is not completed<a name="line.1374"></a>
-<span class="sourceLineNo">1375</span> */<a name="line.1375"></a>
-<span class="sourceLineNo">1376</span> private void checkFailoverCleanupCompleted(final RegionInfo hri) throws PleaseHoldException {<a name="line.1376"></a>
-<span class="sourceLineNo">1377</span> if (!isRunning()) {<a name="line.1377"></a>
-<span class="sourceLineNo">1378</span> throw new PleaseHoldException("AssignmentManager not running");<a name="line.1378"></a>
-<span class="sourceLineNo">1379</span> }<a name="line.1379"></a>
-<span class="sourceLineNo">1380</span><a name="line.1380"></a>
-<span class="sourceLineNo">1381</span> // TODO: can we avoid throwing an exception if hri is already loaded?<a name="line.1381"></a>
-<span class="sourceLineNo">1382</span> // at the moment we bypass only meta<a name="line.1382"></a>
-<span class="sourceLineNo">1383</span> boolean meta = isMetaRegion(hri);<a name="line.1383"></a>
-<span class="sourceLineNo">1384</span> boolean cleanup = isFailoverCleanupDone();<a name="line.1384"></a>
-<span class="sourceLineNo">1385</span> if (!isMetaRegion(hri) && !isFailoverCleanupDone()) {<a name="line.1385"></a>
-<span class="sourceLineNo">1386</span> String msg = "Master not fully online; hbase:meta=" + meta + ", failoverCleanup=" + cleanup;<a name="line.1386"></a>
-<span class="sourceLineNo">1387</span> throw new PleaseHoldException(msg);<a name="line.1387"></a>
-<span class="sourceLineNo">1388</span> }<a name="line.1388"></a>
-<span class="sourceLineNo">1389</span> }<a name="line.1389"></a>
-<span class="sourceLineNo">1390</span><a name="line.1390"></a>
-<span class="sourceLineNo">1391</span> // ============================================================================================<a name="line.1391"></a>
-<span class="sourceLineNo">1392</span> // TODO: Metrics<a name="line.1392"></a>
-<span class="sourceLineNo">1393</span> // ============================================================================================<a name="line.1393"></a>
-<span class="sourceLineNo">1394</span> public int getNumRegionsOpened() {<a name="line.1394"></a>
-<span class="sourceLineNo">1395</span> // TODO: Used by TestRegionPlacement.java and assume monotonically increasing value<a name="line.1395"></a>
-<span class="sourceLineNo">1396</span> return 0;<a name="line.1396"></a>
-<span class="sourceLineNo">1397</span> }<a name="line.1397"></a>
-<span class="sourceLineNo">1398</span><a name="line.1398"></a>
-<span class="sourceLineNo">1399</span> public void submitServerCrash(final ServerName serverName, final boolean shouldSplitWal) {<a name="line.1399"></a>
-<span class="sourceLineNo">1400</span> boolean carryingMeta = isCarryingMeta(serverName);<a name="line.1400"></a>
-<span class="sourceLineNo">1401</span> ProcedureExecutor<MasterProcedureEnv> procExec = this.master.getMasterProcedureExecutor();<a name="line.1401"></a>
-<span class="sourceLineNo">1402</span> procExec.submitProcedure(new ServerCrashProcedure(procExec.getEnvironment(), serverName,<a name="line.1402"></a>
-<span class="sourceLineNo">1403</span> shouldSplitWal, carryingMeta));<a name="line.1403"></a>
-<span class="sourceLineNo">1404</span> LOG.debug("Added=" + serverName +<a name="line.1404"></a>
-<span class="sourceLineNo">1405</span> " to dead servers, submitted shutdown handler to be executed meta=" + carryingMeta);<a name="line.1405"></a>
-<span class="sourceLineNo">1406</span> }<a name="line.1406"></a>
-<span class="sourceLineNo">1407</span><a name="line.1407"></a>
-<span class="sourceLineNo">1408</span> public void offlineRegion(final RegionInfo regionInfo) {<a name="line.1408"></a>
-<span class="sourceLineNo">1409</span> // TODO used by MasterRpcServices ServerCrashProcedure<a name="line.1409"></a>
-<span class="sourceLineNo">1410</span> final RegionStateNode node = regionStates.getRegionStateNode(regionInfo);<a name="line.1410"></a>
-<span class="sourceLineNo">1411</span> if (node != null) node.offline();<a name="line.1411"></a>
-<span class="sourceLineNo">1412</span> }<a name="line.1412"></a>
-<span class="sourceLineNo">1413</span><a name="line.1413"></a>
-<span class="sourceLineNo">1414</span> public void onlineRegion(final RegionInfo regionInfo, final ServerName serverName) {<a name="line.1414"></a>
-<span class="sourceLineNo">1415</span> // TODO used by TestSplitTransactionOnCluster.java<a name="line.1415"></a>
-<span class="sourceLineNo">1416</span> }<a name="line.1416"></a>
-<span class="sourceLineNo">1417</span><a name="line.1417"></a>
-<span class="sourceLineNo">1418</span> public Map<ServerName, List<RegionInfo>> getSnapShotOfAssignment(<a name="line.1418"></a>
-<span class="sourceLineNo">1419</span> final Collection<RegionInfo> regions) {<a name="line.1419"></a>
-<span class="sourceLineNo">1420</span> return regionStates.getSnapShotOfAssignment(regions);<a name="line.1420"></a>
-<span class="sourceLineNo">1421</span> }<a name="line.1421"></a>
-<span class="sourceLineNo">1422</span><a name="line.1422"></a>
-<span class="sourceLineNo">1423</span> // ============================================================================================<a name="line.1423"></a>
-<span class="sourceLineNo">1424</span> // TODO: UTILS/HELPERS?<a name="line.1424"></a>
-<span class="sourceLineNo">1425</span> // ============================================================================================<a name="line.1425"></a>
-<span class="sourceLineNo">1426</span> /**<a name="line.1426"></a>
-<span class="sourceLineNo">1427</span> * Used by the client (via master) to identify if all regions have the schema updates<a name="line.1427"></a>
-<span class="sourceLineNo">1428</span> *<a name="line.1428"></a>
-<span class="sourceLineNo">1429</span> * @param tableName<a name="line.1429"></a>
-<span class="sourceLineNo">1430</span> * @return Pair indicating the status of the alter command (pending/total)<a name="line.1430"></a>
-<span class="sourceLineNo">1431</span> * @throws IOException<a name="line.1431"></a>
-<span class="sourceLineNo">1432</span> */<a name="line.1432"></a>
-<span class="sourceLineNo">1433</span> public Pair<Integer, Integer> getReopenStatus(TableName tableName) {<a name="line.1433"></a>
-<span class="sourceLineNo">1434</span> if (isTableDisabled(tableName)) return new Pair<Integer, Integer>(0, 0);<a name="line.1434"></a>
-<span class="sourceLineNo">1435</span><a name="line.1435"></a>
-<span class="sourceLineNo">1436</span> final List<RegionState> states = regionStates.getTableRegionStates(tableName);<a name="line.1436"></a>
-<span class="sourceLineNo">1437</span> int ritCount = 0;<a name="line.1437"></a>
-<span class="sourceLineNo">1438</span> for (RegionState regionState: states) {<a name="line.1438"></a>
-<span class="sourceLineNo">1439</span> if (!regionState.isOpened()) ritCount++;<a name="line.1439"></a>
-<span class="sourceLineNo">1440</span> }<a name="line.1440"></a>
-<span class="sourceLineNo">1441</span> return new Pair<Integer, Integer>(ritCount, states.size());<a name="line.1441"></a>
-<span class="sourceLineNo">1442</span> }<a name="line.1442"></a>
-<span class="sourceLineNo">1443</span><a name="line.1443"></a>
-<span class="sourceLineNo">1444</span> // ============================================================================================<a name="line.1444"></a>
-<span class="sourceLineNo">1445</span> // TODO: Region State In Transition<a name="line.1445"></a>
-<span class="sourceLineNo">1446</span> // ============================================================================================<a name="line.1446"></a>
-<span class="sourceLineNo">1447</span> protected boolean addRegionInTransition(final RegionStateNode regionNode,<a name="line.1447"></a>
-<span class="sourceLineNo">1448</span> final RegionTransitionProcedure procedure) {<a name="line.1448"></a>
-<span class="sourceLineNo">1449</span> return regionStates.addRegionInTransition(regionNode, procedure);<a name="line.1449"></a>
-<span class="sourceLineNo">1450</span> }<a name="line.1450"></a>
-<span class="sourceLineNo">1451</span><a name="line.1451"></a>
-<span class="sourceLineNo">1452</span> protected void removeRegionInTransition(final RegionStateNode regionNode,<a name="line.1452"></a>
-<span class="sourceLineNo">1453</span> final RegionTransitionProcedure procedure) {<a name="line.1453"></a>
-<span class="sourceLineNo">1454</span> regionStates.removeRegionInTransition(regionNode, procedure);<a name="line.1454"></a>
-<span class="sourceLineNo">1455</span> }<a name="line.1455"></a>
-<span class="sourceLineNo">1456</span><a name="line.1456"></a>
-<span class="sourceLineNo">1457</span> public boolean hasRegionsInTransition() {<a name="line.1457"></a>
-<span class="sourceLineNo">1458</span> return regionStates.hasRegionsInTransition();<a name="line.1458"></a>
-<span class="sourceLineNo">1459</span> }<a name="line.1459"></a>
-<span class="sourceLineNo">1460</span><a name="line.1460"></a>
-<span class="sourceLineNo">1461</span> public List<RegionStateNode> getRegionsInTransition() {<a name="line.1461"></a>
-<span class="sourceLineNo">1462</span> return regionStates.getRegionsInTransition();<a name="line.1462"></a>
-<span class="sourceLineNo">1463</span> }<a name="line.1463"></a>
-<span class="sourceLineNo">1464</span><a name="line.1464"></a>
-<span class="sourceLineNo">1465</span> public List<RegionInfo> getAssignedRegions() {<a name="line.1465"></a>
-<span class="sourceLineNo">1466</span> return regionStates.getAssignedRegions();<a name="line.1466"></a>
-<span class="sourceLineNo">1467</span> }<a name="line.1467"></a>
-<span class="sourceLineNo">1468</span><a name="line.1468"></a>
-<span class="sourceLineNo">1469</span> public RegionInfo getRegionInfo(final byte[] regionName) {<a name="line.1469"></a>
-<span class="sourceLineNo">1470</span> final RegionStateNode regionState = regionStates.getRegionStateNodeFromName(regionName);<a name="line.1470"></a>
-<span class="sourceLineNo">1471</span> return regionState != null ? regionState.getRegionInfo() : null;<a name="line.1471"></a>
-<span class="sourceLineNo">1472</span> }<a name="line.1472"></a>
-<span class="sourceLineNo">1473</span><a name="line.1473"></a>
-<span class="sourceLineNo">1474</span> // ============================================================================================<a name="line.1474"></a>
-<span class="sourceLineNo">1475</span> // TODO: Region Status update<a name="line.1475"></a>
-<span class="sourceLineNo">1476</span> // ============================================================================================<a name="line.1476"></a>
-<span class="sourceLineNo">1477</span> private void sendRegionOpenedNotification(final RegionInfo regionInfo,<a name="line.1477"></a>
-<span class="sourceLineNo">1478</span> final ServerName serverName) {<a name="line.1478"></a>
-<span class="sourceLineNo">1479</span> getBalancer().regionOnline(regionInfo, serverName);<a name="line.1479"></a>
-<span class="sourceLineNo">1480</span> if (!this.listeners.isEmpty()) {<a name="line.1480"></a>
-<span class="sourceLineNo">1481</span> for (AssignmentListener listener : this.listeners) {<a name="line.1481"></a>
-<span class="sourceLineNo">1482</span> listener.regionOpened(regionInfo, serverName);<a name="line.1482"></a>
-<span class="sourceLineNo">1483</span> }<a name="line.1483"></a>
-<span class="sourceLineNo">1484</span> }<a name="line.1484"></a>
-<span class="sourceLineNo">1485</span> }<a name="line.1485"></a>
-<span class="sourceLineNo">1486</span><a name="line.1486"></a>
-<span class="sourceLineNo">1487</span> private void sendRegionClosedNotification(final RegionInfo regionInfo) {<a name="line.1487"></a>
-<span class="sourceLineNo">1488</span> getBalancer().regionOffline(regionInfo);<a name="line.1488"></a>
-<span class="sourceLineNo">1489</span> if (!this.listeners.isEmpty()) {<a name="line.1489"></a>
-<span class="sourceLineNo">1490</span> for (AssignmentListener listener : this.listeners) {<a name="line.1490"></a>
-<span class="sourceLineNo">1491</span> listener.regionClosed(regionInfo);<a name="line.1491"></a>
-<span class="sourceLineNo">1492</span> }<a name="line.1492"></a>
-<span class="sourceLineNo">1493</span> }<a name="line.1493"></a>
-<span class="sourceLineNo">1494</span> }<a name="line.1494"></a>
-<span class="sourceLineNo">1495</span><a name="line.1495"></a>
-<span class="sourceLineNo">1496</span> public void markRegionAsOpening(final RegionStateNode regionNode) throws IOException {<a name="line.1496"></a>
-<span class="sourceLineNo">1497</span> synchronized (regionNode) {<a name="line.1497"></a>
-<span class="sourceLineNo">1498</span> regionNode.transitionState(State.OPENING, RegionStates.STATES_EXPECTED_ON_OPEN);<a name="line.1498"></a>
-<span class="sourceLineNo">1499</span> regionStates.addRegionToServer(regionNode);<a name="line.1499"></a>
-<span class="sourceLineNo">1500</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1500"></a>
-<span class="sourceLineNo">1501</span> }<a name="line.1501"></a>
-<span class="sourceLineNo">1502</span><a name="line.1502"></a>
-<span class="sourceLineNo">1503</span> // update the operation count metrics<a name="line.1503"></a>
-<span class="sourceLineNo">1504</span> metrics.incrementOperationCounter();<a name="line.1504"></a>
-<span class="sourceLineNo">1505</span> }<a name="line.1505"></a>
-<span class="sourceLineNo">1506</span><a name="line.1506"></a>
-<span class="sourceLineNo">1507</span> public void undoRegionAsOpening(final RegionStateNode regionNode) {<a name="line.1507"></a>
-<span class="sourceLineNo">1508</span> boolean opening = false;<a name="line.1508"></a>
-<span class="sourceLineNo">1509</span> synchronized (regionNode) {<a name="line.1509"></a>
-<span class="sourceLineNo">1510</span> if (regionNode.isInState(State.OPENING)) {<a name="line.1510"></a>
-<span class="sourceLineNo">1511</span> opening = true;<a name="line.1511"></a>
-<span class="sourceLineNo">1512</span> regionStates.removeRegionFromServer(regionNode.getRegionLocation(), regionNode);<a name="line.1512"></a>
-<span class="sourceLineNo">1513</span> }<a name="line.1513"></a>
-<span class="sourceLineNo">1514</span> // Should we update hbase:meta?<a name="line.1514"></a>
-<span class="sourceLineNo">1515</span> }<a name="line.1515"></a>
-<span class="sourceLineNo">1516</span> if (opening) {<a name="line.1516"></a>
-<span class="sourceLineNo">1517</span> // TODO: Metrics. Do opposite of metrics.incrementOperationCounter();<a name="line.1517"></a>
-<span class="sourceLineNo">1518</span> }<a name="line.1518"></a>
-<span class="sourceLineNo">1519</span> }<a name="line.1519"></a>
-<span class="sourceLineNo">1520</span><a name="line.1520"></a>
-<span class="sourceLineNo">1521</span> public void markRegionAsOpened(final RegionStateNode regionNode) throws IOException {<a name="line.1521"></a>
-<span class="sourceLineNo">1522</span> final RegionInfo hri = regionNode.getRegionInfo();<a name="line.1522"></a>
-<span class="sourceLineNo">1523</span> synchronized (regionNode) {<a name="line.1523"></a>
-<span class="sourceLineNo">1524</span> regionNode.transitionState(State.OPEN, RegionStates.STATES_EXPECTED_ON_OPEN);<a name="line.1524"></a>
-<span class="sourceLineNo">1525</span> if (isMetaRegion(hri)) {<a name="line.1525"></a>
-<span class="sourceLineNo">1526</span> // Usually we'd set a table ENABLED at this stage but hbase:meta is ALWAYs enabled, it<a name="line.1526"></a>
-<span class="sourceLineNo">1527</span> // can't be disabled -- so skip the RPC (besides... enabled is managed by TableStateManager<a name="line.1527"></a>
-<span class="sourceLineNo">1528</span> // which is backed by hbase:meta... Avoid setting ENABLED to avoid having to update state<a name="line.1528"></a>
-<span class="sourceLineNo">1529</span> // on table that contains state.<a name="line.1529"></a>
-<span class="sourceLineNo">1530</span> setMetaInitialized(hri, true);<a name="line.1530"></a>
-<span class="sourceLineNo">1531</span> }<a name="line.1531"></a>
-<span class="sourceLineNo">1532</span> regionStates.addRegionToServer(regionNode);<a name="line.1532"></a>
-<span class="sourceLineNo">1533</span> // TODO: OPENING Updates hbase:meta too... we need to do both here and there?<a name="line.1533"></a>
-<span class="sourceLineNo">1534</span> // That is a lot of hbase:meta writing.<a name="line.1534"></a>
-<span class="sourceLineNo">1535</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1535"></a>
-<span class="sourceLineNo">1536</span> sendRegionOpenedNotification(hri, regionNode.getRegionLocation());<a name="line.1536"></a>
-<span class="sourceLineNo">1537</span> }<a name="line.1537"></a>
-<span class="sourceLineNo">1538</span> }<a name="line.1538"></a>
-<span class="sourceLineNo">1539</span><a name="line.1539"></a>
-<span class="sourceLineNo">1540</span> public void markRegionAsClosing(final RegionStateNode regionNode) throws IOException {<a name="line.1540"></a>
-<span class="sourceLineNo">1541</span> final RegionInfo hri = regionNode.getRegionInfo();<a name="line.1541"></a>
-<span class="sourceLineNo">1542</span> synchronized (regionNode) {<a name="line.1542"></a>
-<span class="sourceLineNo">1543</span> regionNode.transitionState(State.CLOSING, RegionStates.STATES_EXPECTED_ON_CLOSE);<a name="line.1543"></a>
-<span class="sourceLineNo">1544</span> // Set meta has not initialized early. so people trying to create/edit tables will wait<a name="line.1544"></a>
-<span class="sourceLineNo">1545</span> if (isMetaRegion(hri)) {<a name="line.1545"></a>
-<span class="sourceLineNo">1546</span> setMetaInitialized(hri, false);<a name="line.1546"></a>
-<span class="sourceLineNo">1547</span> }<a name="line.1547"></a>
-<span class="sourceLineNo">1548</span> regionStates.addRegionToServer(regionNode);<a name="line.1548"></a>
-<span class="sourceLineNo">1549</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1549"></a>
-<span class="sourceLineNo">1550</span> }<a name="line.1550"></a>
-<span class="sourceLineNo">1551</span><a name="line.1551"></a>
-<span class="sourceLineNo">1552</span> // update the operation count metrics<a name="line.1552"></a>
-<span class="sourceLineNo">1553</span> metrics.incrementOperationCounter();<a name="line.1553"></a>
-<span class="sourceLineNo">1554</span> }<a name="line.1554"></a>
-<span class="sourceLineNo">1555</span><a name="line.1555"></a>
-<span class="sourceLineNo">1556</span> public void undoRegionAsClosing(final RegionStateNode regionNode) {<a name="line.1556"></a>
-<span class="sourceLineNo">1557</span> // TODO: Metrics. Do opposite of metrics.incrementOperationCounter();<a name="line.1557"></a>
-<span class="sourceLineNo">1558</span> // There is nothing to undo?<a name="line.1558"></a>
-<span class="sourceLineNo">1559</span> }<a name="line.1559"></a>
-<span class="sourceLineNo">1560</span><a name="line.1560"></a>
-<span class="sourceLineNo">1561</span> public void markRegionAsClosed(final RegionStateNode regionNode) throws IOException {<a name="line.1561"></a>
-<span class="sourceLineNo">1562</span> final RegionInfo hri = regionNode.getRegionInfo();<a name="line.1562"></a>
-<span class="sourceLineNo">1563</span> synchronized (regionNode) {<a name="line.1563"></a>
-<span class="sourceLineNo">1564</span> regionNode.transitionState(State.CLOSED, RegionStates.STATES_EXPECTED_ON_CLOSE);<a name="line.1564"></a>
-<span class="sourceLineNo">1565</span> regionStates.removeRegionFromServer(regionNode.getRegionLocation(), regionNode);<a name="line.1565"></a>
-<span class="sourceLineNo">1566</span> regionNode.setLastHost(regionNode.getRegionLocation());<a name="line.1566"></a>
-<span class="sourceLineNo">1567</span> regionNode.setRegionLocation(null);<a name="line.1567"></a>
-<span class="sourceLineNo">1568</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1568"></a>
-<span class="sourceLineNo">1569</span> sendRegionClosedNotification(hri);<a name="line.1569"></a>
-<span class="sourceLineNo">1570</span> }<a name="line.1570"></a>
-<span class="sourceLineNo">1571</span> }<a name="line.1571"></a>
-<span class="sourceLineNo">1572</span><a name="line.1572"></a>
-<span class="sourceLineNo">1573</span> public void markRegionAsSplit(final RegionInfo parent, final ServerName serverName,<a name="line.1573"></a>
-<span class="sourceLineNo">1574</span> final RegionInfo daughterA, final RegionInfo daughterB) throws IOException {<a name="line.1574"></a>
-<span class="sourceLineNo">1575</span> // Update hbase:meta. Parent will be marked offline and split up in hbase:meta.<a name="line.1575"></a>
-<span class="sourceLineNo">1576</span> // The parent stays in regionStates until cleared when removed by CatalogJanitor.<a name="line.1576"></a>
-<span class="sourceLineNo">1577</span> // Update its state in regionStates to it shows as offline and split when read<a name="line.1577"></a>
-<span class="sourceLineNo">1578</span> // later figuring what regions are in a table and what are not: see<a name="line.1578"></a>
-<span class="sourceLineNo">1579</span> // regionStates#getRegionsOfTable<a name="line.1579"></a>
-<span class="sourceLineNo">1580</span> final RegionStateNode node = regionStates.getOrCreateRegionStateNode(parent);<a name="line.1580"></a>
-<span class="sourceLineNo">1581</span> node.setState(State.SPLIT);<a name="line.1581"></a>
-<span class="sourceLineNo">1582</span> final RegionStateNode nodeA = regionStates.getOrCreateRegionStateNode(daughterA);<a name="line.1582"></a>
-<span class="sourceLineNo">1583</span> nodeA.setState(State.SPLITTING_NEW);<a name="line.1583"></a>
-<span class="sourceLineNo">1584</span> final RegionStateNode nodeB = regionStates.getOrCreateRegionStateNode(daughterB);<a name="line.1584"></a>
-<span class="sourceLineNo">1585</span> nodeB.setState(State.SPLITTING_NEW);<a name="line.1585"></a>
-<span class="sourceLineNo">1586</span><a name="line.1586"></a>
-<span class="sourceLineNo">1587</span> regionStateStore.splitRegion(parent, daughterA, daughterB, serverName);<a name="line.1587"></a>
-<span class="sourceLineNo">1588</span> if (shouldAssignFavoredNodes(parent)) {<a name="line.1588"></a>
-<span class="sourceLineNo">1589</span> List<ServerName> onlineServers = this.master.getServerManager().getOnlineServersList();<a name="line.1589"></a>
-<span class="sourceLineNo">1590</span> ((FavoredNodesPromoter)getBalancer()).<a name="line.1590"></a>
-<span class="sourceLineNo">1591</span> generateFavoredNodesForDaughter(onlineServers, parent, daughterA, daughterB);<a name="line.1591"></a>
-<span class="sourceLineNo">1592</span> }<a name="line.1592"></a>
-<span class="sourceLineNo">1593</span> }<a name="line.1593"></a>
-<span class="sourceLineNo">1594</span><a name="line.1594"></a>
-<span class="sourceLineNo">1595</span> /**<a name="line.1595"></a>
-<span class="sourceLineNo">1596</span> * When called here, the merge has happened. The two merged regions have been<a name="line.1596"></a>
-<span class="sourceLineNo">1597</span> * unassigned and the above markRegionClosed has been called on each so they have been<a name="line.1597"></a>
-<span class="sourceLineNo">1598</span> * disassociated from a hosting Server. The merged region will be open after this call. The<a name="line.1598"></a>
-<span class="sourceLineNo">1599</span> * merged regions are removed from hbase:meta below> Later they are deleted from the filesystem<a name="line.1599"></a>
-<span class="sourceLineNo">1600</span> * by the catalog janitor running against hbase:meta. It notices when the merged region no<a name="line.1600"></a>
-<span class="sourceLineNo">1601</span> * longer holds references to the old regions.<a name="line.1601"></a>
-<span class="sourceLineNo">1602</span> */<a name="line.1602"></a>
-<span class="sourceLineNo">1603</span> public void markRegionAsMerged(final RegionInfo child, final ServerName serverName,<a name="line.1603"></a>
-<span class="sourceLineNo">1604</span> final RegionInfo mother, final RegionInfo father) throws IOException {<a name="line.1604"></a>
-<span class="sourceLineNo">1605</span> final RegionStateNode node = regionStates.getOrCreateRegionStateNode(child);<a name="line.1605"></a>
-<span class="sourceLineNo">1606</span> node.setState(State.MERGED);<a name="line.1606"></a>
-<span class="sourceLineNo">1607</span> regionStates.deleteRegion(mother);<a name="line.1607"></a>
-<span class="sourceLineNo">1608</span> regionStates.deleteRegion(father);<a name="line.1608"></a>
-<span class="sourceLineNo">1609</span> regionStateStore.mergeRegions(child, mother, father, serverName);<a name="line.1609"></a>
-<span class="sourceLineNo">1610</span> if (shouldAssignFavoredNodes(child)) {<a name="line.1610"></a>
-<span class="sourceLineNo">1611</span> ((FavoredNodesPromoter)getBalancer()).<a name="line.1611"></a>
-<span class="sourceLineNo">1612</span> generateFavoredNodesForMergedRegion(child, mother, father);<a name="line.1612"></a>
-<span class="sourceLineNo">1613</span> }<a name="line.1613"></a>
-<span class="sourceLineNo">1614</span> }<a name="line.1614"></a>
-<span class="sourceLineNo">1615</span><a name="line.1615"></a>
-<span class="sourceLineNo">1616</span> /*<a name="line.1616"></a>
-<span class="sourceLineNo">1617</span> * Favored nodes should be applied only when FavoredNodes balancer is configured and the region<a name="line.1617"></a>
-<span class="sourceLineNo">1618</span> * belongs to a non-system table.<a name="line.1618"></a>
-<span class="sourceLineNo">1619</span> */<a name="line.1619"></a>
-<span class="sourceLineNo">1620</span> private boolean shouldAssignFavoredNodes(RegionInfo region) {<a name="line.1620"></a>
-<span class="sourceLineNo">1621</span> return this.shouldAssignRegionsWithFavoredNodes &&<a name="line.1621"></a>
-<span class="sourceLineNo">1622</span> FavoredNodesManager.isFavoredNodeApplicable(region);<a name="line.1622"></a>
-<span class="sourceLineNo">1623</span> }<a name="line.1623"></a>
-<span class="sourceLineNo">1624</span><a name="line.1624"></a>
-<span class="sourceLineNo">1625</span> // ============================================================================================<a name="line.1625"></a>
-<span class="sourceLineNo">1626</span> // Assign Queue (Assign/Balance)<a name="line.1626"></a>
-<span class="sourceLineNo">1627</span> // ============================================================================================<a name="line.1627"></a>
-<span class="sourceLineNo">1628</span> private final ArrayList<RegionStateNode> pendingAssignQueue = new ArrayList<RegionStateNode>();<a name="line.1628"></a>
-<span class="sourceLineNo">1629</span> private final ReentrantLock assignQueueLock = new ReentrantLock();<a name="line.1629"></a>
-<span class="sourceLineNo">1630</span> private final Condition assignQueueFullCond = assignQueueLock.newCondition();<a name="line.1630"></a>
-<span class="sourceLineNo">1631</span><a name="line.1631"></a>
-<span class="sourceLineNo">1632</span> /**<a name="line.1632"></a>
-<span class="sourceLineNo">1633</span> * Add the assign operation to the assignment queue.<a name="line.1633"></a>
-<span class="sourceLineNo">1634</span> * The pending assignment operation will be processed,<a name="line.1634"></a>
-<span class="sourceLineNo">1635</span> * and each region will be assigned by a server using the balancer.<a name="line.1635"></a>
-<span class="sourceLineNo">1636</span> */<a name="line.1636"></a>
-<span class="sourceLineNo">1637</span> protected void queueAssign(final RegionStateNode regionNode) {<a name="line.1637"></a>
-<span class="sourceLineNo">1638</span> regionNode.getProcedureEvent().suspend();<a name="line.1638"></a>
-<span class="sourceLineNo">1639</span><a name="line.1639"></a>
-<span class="sourceLineNo">1640</span> // TODO: quick-start for meta and the other sys-tables?<a name="line.1640"></a>
-<span class="sourceLineNo">1641</span> assignQueueLock.lock();<a name="line.1641"></a>
-<span class="sourceLineNo">1642</span> try {<a name="line.1642"></a>
-<span class="sourceLineNo">1643</span> pendingAssignQueue.add(regionNode);<a name="line.1643"></a>
-<span class="sourceLineNo">1644</span> if (regionNode.isSystemTable() ||<a name="line.1644"></a>
-<span class="sourceLineNo">1645</span> pendingAssignQueue.size() == 1 ||<a name="line.1645"></a>
-<span class="sourceLineNo">1646</span> pendingAssignQueue.size() >= assignDispatchWaitQueueMaxSize) {<a name="line.1646"></a>
-<span class="sourceLineNo">1647</span> assignQueueFullCond.signal();<a name="line.1647"></a>
-<span class="sourceLineNo">1648</span> }<a name="line.1648"></a>
-<span class="sourceLineNo">1649</span> } finally {<a name="line.1649"></a>
-<span class="sourceLineNo">1650</span> assignQueueLock.unlock();<a name="line.1650"></a>
-<span class="sourceLineNo">1651</span> }<a name="line.1651"></a>
-<span class="sourceLineNo">1652</span> }<a name="line.1652"></a>
-<span class="sourceLineNo">1653</span><a name="line.1653"></a>
-<span class="sourceLineNo">1654</span> private void startAssignmentThread() {<a name="line.1654"></a>
-<span class="sourceLineNo">1655</span> // Get Server Thread name. Sometimes the Server is mocked so may not implement HasThread.<a name="line.1655"></a>
-<span class="sourceLineNo">1656</span> // For example, in tests.<a name="line.1656"></a>
-<span class="sourceLineNo">1657</span> String name = master instanceof HasThread? ((HasThread)master).getName():<a name="line.1657"></a>
-<span class="sourceLineNo">1658</span> master.getServerName().toShortString();<a name="line.1658"></a>
-<span class="sourceLineNo">1659</span> assignThread = new Thread(name) {<a name="line.1659"></a>
-<span class="sourceLineNo">1660</span> @Override<a name="line.1660"></a>
-<span class="sourceLineNo">1661</span> public void run() {<a name="line.1661"></a>
-<span class="sourceLineNo">1662</span> while (isRunning()) {<a name="line.1662"></a>
-<span class="sourceLineNo">1663</span> processAssignQueue();<a name="line.1663"></a>
-<span class="sourceLineNo">1664</span> }<a name="line.1664"></a>
-<span class="sourceLineNo">1665</span> pendingAssignQueue.clear();<a name="line.1665"></a>
-<span class="sourceLineNo">1666</span> }<a name="line.1666"></a>
-<span class="sourceLineNo">1667</span> };<a name="line.1667"></a>
-<span class="sourceLineNo">1668</span> assignThread.setDaemon(true);<a name="line.1668"></a>
-<span class="sourceLineNo">1669</span> assignThread.start();<a name="line.1669"></a>
-<span class="sourceLineNo">1670</span> }<a name="line.1670"></a>
-<span class="sourceLineNo">1671</span><a name="line.1671"></a>
-<span class="sourceLineNo">1672</span> private void stopAssignmentThread() {<a name="line.1672"></a>
-<span class="sourceLineNo">1673</span> assignQueueSignal();<a name="line.1673"></a>
-<span class="sourceLineNo">1674</span> try {<a name="line.1674"></a>
-<span class="sourceLineNo">1675</span> while (assignThread.isAlive()) {<a name="line.1675"></a>
-<span class="sourceLineNo">1676</span> assignQueueSignal();<a name="line.1676"></a>
-<span class="sourceLineNo">1677</span> assignThread.join(250);<a name="line.1677"></a>
-<span class="sourceLineNo">1678</span> }<a name="line.1678"></a>
-<span class="sourceLineNo">1679</span> } catch (InterruptedException e) {<a name="line.1679"></a>
-<span class="sourceLineNo">1680</span> LOG.warn("join interrupted", e);<a name="line.1680"></a>
-<span class="sourceLineNo">1681</span> Thread.currentThread().interrupt();<a name="line.1681"></a>
-<span class="sourceLineNo">1682</span> }<a name="line.1682"></a>
-<span class="sourceLineNo">1683</span> }<a name="line.1683"></a>
-<span class="sourceLineNo">1684</span><a name="line.1684"></a>
-<span class="sourceLineNo">1685</span> private void assignQueueSignal() {<a name="line.1685"></a>
-<span class="sourceLineNo">1686</span> assignQueueLock.lock();<a name="line.1686"></a>
-<span class="sourceLineNo">1687</span> try {<a name="line.1687"></a>
-<span class="sourceLineNo">1688</span> assignQueueFullCond.signal();<a name="line.1688"></a>
-<span class="sourceLineNo">1689</span> } finally {<a name="line.1689"></a>
-<span class="sourceLineNo">1690</span> assignQueueLock.unlock();<a name="line.1690"></a>
-<span class="sourceLineNo">1691</span> }<a name="line.1691"></a>
-<span class="sourceLineNo">1692</span> }<a name="line.1692"></a>
-<span class="sourceLineNo">1693</span><a name="line.1693"></a>
-<span class="sourceLineNo">1694</span> @edu.umd.cs.findbugs.annotations.SuppressWarnings("WA_AWAIT_NOT_IN_LOOP")<a name="line.1694"></a>
-<span class="sourceLineNo">1695</span> private HashMap<RegionInfo, RegionStateNode> waitOnAssignQueue() {<a name="line.1695"></a>
-<span class="sourceLineNo">1696</span> HashMap<RegionInfo, RegionStateNode> regions = null;<a name="line.1696"></a>
-<span class="sourceLineNo">1697</span><a name="line.1697"></a>
-<span class="sourceLineNo">1698</span> assignQueueLock.lock();<a name="line.1698"></a>
-<span class="sourceLineNo">1699</span> try {<a name="line.1699"></a>
-<span class="sourceLineNo">1700</span> if (pendingAssignQueue.isEmpty() && isRunning()) {<a name="line.1700"></a>
-<span class="sourceLineNo">1701</span> assignQueueFullCond.await();<a name="line.1701"></a>
-<span class="sourceLineNo">1702</span> }<a name="line.1702"></a>
-<span class="sourceLineNo">1703</span><a name="line.1703"></a>
-<span class="sourceLineNo">1704</span> if (!isRunning()) return null;<a name="line.1704"></a>
-<span class="sourceLineNo">1705</span> assignQueueFullCond.await(assignDispatchWaitMillis, TimeUnit.MILLISECONDS);<a name="line.1705"></a>
-<span class="sourceLineNo">1706</span> regions = new HashMap<RegionInfo, RegionStateNode>(pendingAssignQueue.size());<a name="line.1706"></a>
-<span class="sourceLineNo">1707</span> for (RegionStateNode regionNode: pendingAssignQueue) {<a name="line.1707"></a>
-<span class="sourceLineNo">1708</span> regions.put(regionNode.getRegionInfo(), regionNode);<a name="line.1708"></a>
-<span class="sourceLineNo">1709</span> }<a name="line.1709"></a>
-<span class="sourceLineNo">1710</span> pendingAssignQueue.clear();<a name="line.1710"></a>
-<span class="sourceLineNo">1711</span> } catch (InterruptedException e) {<a name="line.1711"></a>
-<span class="sourceLineNo">1712</span> LOG.warn("got interrupted ", e);<a name="line.1712"></a>
-<span class="sourceLineNo">1713</span> Thread.currentThread().interrupt();<a name="line.1713"></a>
-<span class="sourceLineNo">1714</span> } finally {<a name="line.1714"></a>
-<span class="sourceLineNo">1715</span> assignQueueLock.unlock();<a name="line.1715"></a>
-<span class="sourceLineNo">1716</span> }<a name="line.1716"></a>
-<span class="sourceLineNo">1717</span> return regions;<a name="line.1717"></a>
-<span class="sourceLineNo">1718</span> }<a name="line.1718"></a>
-<span class="sourceLineNo">1719</span><a name="line.1719"></a>
-<span class="sourceLineNo">1720</span> private void processAssignQueue() {<a name="line.1720"></a>
-<span class="sourceLineNo">1721</span> final HashMap<RegionInfo, RegionStateNode> regions = waitOnAssignQueue();<a name="line.1721"></a>
-<span class="sourceLineNo">1722</span> if (regions == null || regions.size() == 0 || !isRunning()) {<a name="line.1722"></a>
-<span class="sourceLineNo">1723</span> return;<a name="line.1723"></a>
-<span class="sourceLineNo">1724</span> }<a name="line.1724"></a>
-<span class="sourceLineNo">1725</span><a name="line.1725"></a>
-<span class="sourceLineNo">1726</span> if (LOG.isTraceEnabled()) {<a name="line.1726"></a>
-<span class="sourceLineNo">1727</span> LOG.trace("PROCESS ASSIGN QUEUE regionCount=" + regions.size());<a name="line.1727"></a>
-<span class="sourceLineNo">1728</span> }<a name="line.1728"></a>
-<span class="sourceLineNo">1729</span><a name="line.1729"></a>
-<span class="sourceLineNo">1730</span> // TODO: Optimize balancer. pass a RegionPlan?<a name="line.1730"></a>
-<span class="sourceLineNo">1731</span> final HashMap<RegionInfo, ServerName> retainMap = new HashMap<>();<a name="line.1731"></a>
-<span class="sourceLineNo">1732</span> final List<RegionInfo> userHRIs = new ArrayList<>(regions.size());<a name="line.1732"></a>
-<span class="sourceLineNo">1733</span> // Regions for system tables requiring reassignment<a name="line.1733"></a>
-<span class="sourceLineNo">1734</span> final List<RegionInfo> systemHRIs = new ArrayList<>();<a name="line.1734"></a>
-<span class="sourceLineNo">1735</span> for (RegionStateNode regionStateNode: regions.values()) {<a name="line.1735"></a>
-<span class="sourceLineNo">1736</span> boolean sysTable = regionStateNode.isSystemTable();<a name="line.1736"></a>
-<span class="sourceLineNo">1737</span> final List<RegionInfo> hris = sysTable? systemHRIs: userHRIs;<a name="line.1737"></a>
-<span class="sourceLineNo">1738</span> if (regionStateNode.getRegionLocation() != null) {<a name="line.1738"></a>
-<span class="sourceLineNo">1739</span> retainMap.put(regionStateNode.getRegionInfo(), regionStateNode.getRegionLocation());<a name="line.1739"></a>
-<span class="sourceLineNo">1740</span> } else {<a name="line.1740"></a>
-<span class="sourceLineNo">1741</span> hris.add(regionStateNode.getRegionInfo());<a name="line.1741"></a>
-<span class="sourceLineNo">1742</span> }<a name="line.1742"></a>
-<span class="sourceLineNo">1743</span> }<a name="line.1743"></a>
-<span class="sourceLineNo">1744</span><a name="line.1744"></a>
-<span class="sourceLineNo">1745</span> // TODO: connect with the listener to invalidate the cache<a name="line.1745"></a>
-<span class="sourceLineNo">1746</span><a name="line.1746"></a>
-<span class="sourceLineNo">1747</span> // TODO use events<a name="line.1747"></a>
-<span class="sourceLineNo">1748</span> List<ServerName> servers = master.getServerManager().createDestinationServersList();<a name="line.1748"></a>
-<span class="sourceLineNo">1749</span> for (int i = 0; servers.size() < 1; ++i) {<a name="line.1749"></a>
-<span class="sourceLineNo">1750</span> // Report every fourth time around this loop; try not to flood log.<a name="line.1750"></a>
-<span class="sourceLineNo">1751</span> if (i % 4 == 0) {<a name="line.1751"></a>
-<span class="sourceLineNo">1752</span> LOG.warn("No servers available; cannot place " + regions.size() + " unassigned regions.");<a name="line.1752"></a>
-<span class="sourceLineNo">1753</span> }<a name="line.1753"></a>
-<span class="sourceLineNo">1754</span><a name="line.1754"></a>
-<span class="sourceLineNo">1755</span> if (!isRunning()) {<a name="line.1755"></a>
-<span class="sourceLineNo">1756</span> LOG.debug("Stopped! Dropping assign of " + regions.size() + " queued regions.");<a name="line.1756"></a>
-<span class="sourceLineNo">1757</span> return;<a name="line.1757"></a>
-<span class="sourceLineNo">1758</span> }<a name="line.1758"></a>
-<span class="sourceLineNo">1759</span> Threads.sleep(250);<a name="line.1759"></a>
-<span class="sourceLineNo">1760</span> servers = master.getServerManager().createDestinationServersList();<a name="line.1760"></a>
-<span class="sourceLineNo">1761</span> }<a name="line.1761"></a>
-<span class="sourceLineNo">1762</span><a name="line.1762"></a>
-<span class="sourceLineNo">1763</span> if (!systemHRIs.isEmpty()) {<a name="line.1763"></a>
-<span class="sourceLineNo">1764</span> // System table regions requiring reassignment are present, get region servers<a name="line.1764"></a>
-<span class="sourceLineNo">1765</span> // not available for system table regions<a name="line.1765"></a>
-<span class="sourceLineNo">1766</span> final List<ServerName> excludeServers = getExcludedServersForSystemTable();<a name="line.1766"></a>
-<span class="sourceLineNo">1767</span> List<ServerName> serversForSysTables = servers.stream()<a name="line.1767"></a>
-<span class="sourceLineNo">1768</span> .filter(s -> !excludeServers.contains(s)).collect(Collectors.toList());<a name="line.1768"></a>
-<span class="sourceLineNo">1769</span> if (serversForSysTables.isEmpty()) {<a name="line.1769"></a>
-<span class="sourceLineNo">1770</span> LOG.warn("Filtering old server versions and the excluded produced an empty set; " +<a name="line.1770"></a>
-<span class="sourceLineNo">1771</span> "instead considering all candidate servers!");<a name="line.1771"></a>
-<span class="sourceLineNo">1772</span> }<a name="line.1772"></a>
-<span class="sourceLineNo">1773</span> LOG.debug("Processing assignQueue; systemServersCount=" + serversForSysTables.size() +<a name="line.1773"></a>
-<span class="sourceLineNo">1774</span> ", allServersCount=" + servers.size());<a name="line.1774"></a>
-<span class="sourceLineNo">1775</span> processAssignmentPlans(regions, null, systemHRIs,<a name="line.1775"></a>
-<span class="sourceLineNo">1776</span> serversForSysTables.isEmpty()? servers: serversForSysTables);<a name="line.1776"></a>
-<span class="sourceLineNo">1777</span> }<a name="line.1777"></a>
-<span class="sourceLineNo">1778</span><a name="line.1778"></a>
-<span class="sourceLineNo">1779</span> processAssignmentPlans(regions, retainMap, userHRIs, servers);<a name="line.1779"></a>
-<span class="sourceLineNo">1780</span> }<a name="line.1780"></a>
-<span class="sourceLineNo">1781</span><a name="line.1781"></a>
-<span class="sourceLineNo">1782</span> private void processAssignmentPlans(final HashMap<RegionInfo, RegionStateNode> regions,<a name="line.1782"></a>
-<span class="sourceLineNo">1783</span> final HashMap<RegionInfo, ServerName> retainMap, final List<RegionInfo> hris,<a name="line.1783"></a>
-<span class="sourceLineNo">1784</span> final List<ServerName> servers) {<a name="line.1784"></a>
-<span class="sourceLineNo">1785</span> boolean isTraceEnabled = LOG.isTraceEnabled();<a name="line.1785"></a>
-<span class="sourceLineNo">1786</span> if (isTraceEnabled) {<a name="line.1786"></a>
-<span class="sourceLineNo">1787</span> LOG.trace("Available servers count=" + servers.size() + ": " + servers);<a name="line.1787"></a>
-<span class="sourceLineNo">1788</span> }<a name="line.1788"></a>
-<span class="sourceLineNo">1789</span><a name="line.1789"></a>
-<span class="sourceLineNo">1790</span> final LoadBalancer balancer = getBalancer();<a name="line.1790"></a>
-<span class="sourceLineNo">1791</span> // ask the balancer where to place regions<a name="line.1791"></a>
-<span class="sourceLineNo">1792</span> if (retainMap != null && !retainMap.isEmpty()) {<a name="line.1792"></a>
-<span class="sourceLineNo">1793</span> if (isTraceEnabled) {<a name="line.1793"></a>
-<span class="sourceLineNo">1794</span> LOG.trace("retain assign regions=" + retainMap);<a name="line.1794"></a>
-<span class="sourceLineNo">1795</span> }<a name="line.1795"></a>
-<span class="sourceLineNo">1796</span> try {<a name="line.1796"></a>
-<span class="sourceLineNo">1797</span> acceptPlan(regions, balancer.retainAssignment(retainMap, servers));<a name="line.1797"></a>
-<span class="sourceLineNo">1798</span> } catch (HBaseIOException e) {<a name="line.1798"></a>
-<span class="sourceLineNo">1799</span> LOG.warn("unable to retain assignment", e);<a name="line.1799"></a>
-<span class="sourceLineNo">1800</span> addToPendingAssignment(regions, retainMap.keySet());<a name="line.1800"></a>
-<span class="sourceLineNo">1801</span> }<a name="line.1801"></a>
-<span class="sourceLineNo">1802</span> }<a name="line.1802"></a>
-<span class="sourceLineNo">1803</span><a name="line.1803"></a>
-<span class="sourceLineNo">1804</span> // TODO: Do we need to split retain and round-robin?<a name="line.1804"></a>
-<span class="sourceLineNo">1805</span> // the retain seems to fallback to round-robin/random if the region is not in the map.<a name="line.1805"></a>
-<span class="sourceLineNo">1806</span> if (!hris.isEmpty()) {<a name="line.1806"></a>
-<span class="sourceLineNo">1807</span> Collections.sort(hris, RegionInfo.COMPARATOR);<a name="line.1807"></a>
-<span class="sourceLineNo">1808</span> if (isTraceEnabled) {<a name="line.1808"></a>
-<span class="sourceLineNo">1809</span> LOG.trace("round robin regions=" + hris);<a name="line.1809"></a>
-<span class="sourceLineNo">1810</span> }<a name="line.1810"></a>
-<span class="sourceLineNo">1811</span> try {<a name="line.1811"></a>
-<span class="sourceLineNo">1812</span> acceptPlan(regions, balancer.roundRobinAssignment(hris, servers));<a name="line.1812"></a>
-<span class="sourceLineNo">1813</span> } catch (HBaseIOException e) {<a name="line.1813"></a>
-<span class="sourceLineNo">1814</span> LOG.warn("unable to round-robin assignment", e);<a name="line.1814"></a>
-<span class="sourceLineNo">1815</span> addToPendingAssignment(regions, hris);<a name="line.1815"></a>
-<span class="sourceLineNo">1816</span> }<a name="line.1816"></a>
-<span class="sourceLineNo">1817</span> }<a name="line.1817"></a>
-<span class="sourceLineNo">1818</span> }<a name="line.1818"></a>
-<span class="sourceLineNo">1819</span><a name="line.1819"></a>
-<span class="sourceLineNo">1820</span> private void acceptPlan(final HashMap<RegionInfo, RegionStateNode> regions,<a name="line.1820"></a>
-<span class="sourceLineNo">1821</span> final Map<ServerName, List<RegionInfo>> plan) throws HBaseIOException {<a name="line.1821"></a>
-<span class="sourceLineNo">1822</span> final ProcedureEvent[] events = new ProcedureEvent[regions.size()];<a name="line.1822"></a>
-<span class="sourceLineNo">1823</span> final long st = System.currentTimeMillis();<a name="line.1823"></a>
-<span class="sourceLineNo">1824</span><a name="line.1824"></a>
-<span class="sourceLineNo">1825</span> if (plan == null) {<a name="line.1825"></a>
-<span class="sourceLineNo">1826</span> throw new HBaseIOException("unable to compute plans for regions=" + regions.size());<a name="line.1826"></a>
-<span class="sourceLineNo">1827</span> }<a name="line.1827"></a>
-<span class="sourceLineNo">1828</span><a name="line.1828"></a>
-<span class="sourceLineNo">1829</span> if (plan.isEmpty()) return;<a name="line.1829"></a>
-<span class="sourceLineNo">1830</span><a name="line.1830"></a>
-<span class="sourceLineNo">1831</span> int evcount = 0;<a name="line.1831"></a>
-<span class="sourceLineNo">1832</span> for (Map.Entry<ServerName, List<RegionInfo>> entry: plan.entrySet()) {<a name="line.1832"></a>
-<span class="sourceLineNo">1833</span> final ServerName server = entry.getKey();<a name="line.1833"></a>
-<span class="sourceLineNo">1834</span> for (RegionInfo hri: entry.getValue()) {<a name="line.1834"></a>
-<span class="sourceLineNo">1835</span> final RegionStateNode regionNode = regions.get(hri);<a name="line.1835"></a>
-<span class="sourceLineNo">1836</span> regionNode.setRegionLocation(server);<a name="line.1836"></a>
-<span class="sourceLineNo">1837</span> events[evcount++] = regionNode.getProcedureEvent();<a name="line.1837"></a>
-<span class="sourceLineNo">1838</span> }<a name="line.1838"></a>
-<span class="sourceLineNo">1839</span> }<a name="line.1839"></a>
-<span class="sourceLineNo">1840</span> ProcedureEvent.wakeEvents(getProcedureScheduler(), events);<a name="line.1840"></a>
-<span class="sourceLineNo">1841</span><a name="line.1841"></a>
-<span class="sourceLineNo">1842</span> final long et = System.currentTimeMillis();<a name="line.1842"></a>
-<span class="sourceLineNo">1843</span> if (LOG.isTraceEnabled()) {<a name="line.1843"></a>
-<span class="sourceLineNo">1844</span> LOG.trace("ASSIGN ACCEPT " + events.length + " -> " +<a name="line.1844"></a>
-<span class="sourceLineNo">1845</span> StringUtils.humanTimeDiff(et - st));<a name="line.1845"></a>
-<span class="sourceLineNo">1846</span> }<a name="line.1846"></a>
-<span class="sourceLineNo">1847</span> }<a name="line.1847"></a>
-<span class="sourceLineNo">1848</span><a name="line.1848"></a>
-<span class="sourceLineNo">1849</span> private void addToPendingAssignment(final HashMap<RegionInfo, RegionStateNode> regions,<a name="line.1849"></a>
-<span class="sourceLineNo">1850</span> final Collection<RegionInfo> pendingRegions) {<a name="line.1850"></a>
-<span class="sourceLineNo">1851</span> assignQueueLock.lock();<a name="line.1851"></a>
-<span class="sourceLineNo">1852</span> try {<a name="line.1852"></a>
-<span class="sourceLineNo">1853</span> for (RegionInfo hri: pendingRegions) {<a name="line.1853"></a>
-<span class="sourceLineNo">1854</span> pendingAssignQueue.add(regions.get(hri));<a name="line.1854"></a>
-<span class="sourceLineNo">1855</span> }<a name="line.1855"></a>
-<span class="sourceLineNo">1856</span> } finally {<a name="line.1856"></a>
-<span class="sourceLineNo">1857</span> assignQueueLock.unlock();<a name="line.1857"></a>
-<span class="sourceLineNo">1858</span> }<a name="line.1858"></a>
-<span class="sourceLineNo">1859</span> }<a name="line.1859"></a>
-<span class="sourceLineNo">1860</span><a name="line.1860"></a>
-<span class="sourceLineNo">1861</span> /**<a name="line.1861"></a>
-<span class="sourceLineNo">1862</span> * Get a list of servers that this region cannot be assigned to.<a name="line.1862"></a>
-<span class="sourceLineNo">1863</span> * For system tables, we must assign them to a server with highest version.<a name="line.1863"></a>
-<span class="sourceLineNo">1864</span> */<a name="line.1864"></a>
-<span class="sourceLineNo">1865</span> public List<ServerName> getExcludedServersForSystemTable() {<a name="line.1865"></a>
-<span class="sourceLineNo">1866</span> // TODO: This should be a cached list kept by the ServerManager rather than calculated on each<a name="line.1866"></a>
-<span class="sourceLineNo">1867</span> // move or system region assign. The RegionServerTracker keeps list of online Servers with<a name="line.1867"></a>
-<span class="sourceLineNo">1868</span> // RegionServerInfo that includes Version.<a name="line.1868"></a>
-<span class="sourceLineNo">1869</span> List<Pair<ServerName, String>> serverList = master.getServerManager().getOnlineServersList()<a name="line.1869"></a>
-<span class="sourceLineNo">1870</span> .stream()<a name="line.1870"></a>
-<span class="sourceLineNo">1871</span> .map((s)->new Pair<>(s, master.getRegionServerVersion(s)))<a name="line.1871"></a>
-<span class="sourceLineNo">1872</span> .collect(Collectors.toList());<a name="line.1872"></a>
-<span class="sourceLineNo">1873</span> if (serverList.isEmpty()) {<a name="line.1873"></a>
-<span class="sourceLineNo">1874</span> return Collections.EMPTY_LIST;<a name="line.1874"></a>
-<span class="sourceLineNo">1875</span> }<a name="line.1875"></a>
-<span class="sourceLineNo">1876</span> String highestVersion = Collections.max(serverList,<a name="line.1876"></a>
-<span class="sourceLineNo">1877</span> (o1, o2) -> VersionInfo.compareVersion(o1.getSecond(), o2.getSecond())).getSecond();<a name="line.1877"></a>
-<span class="sourceLineNo">1878</span> return serverList.stream()<a name="line.1878"></a>
-<span class="sourceLineNo">1879</span> .filter((p)->!p.getSecond().equals(highestVersion))<a name="line.1879"></a>
-<span class="sourceLineNo">1880</span> .map(Pair::getFirst)<a name="line.1880"></a>
-<span class="sourceLineNo">1881</span> .collect(Collectors.toList());<a name="line.1881"></a>
-<span class="sourceLineNo">1882</span> }<a name="line.1882"></a>
-<span class="sourceLineNo">1883</span><a name="line.1883"></a>
-<span class="sourceLineNo">1884</span> // ============================================================================================<a name="line.1884"></a>
-<span class="sourceLineNo">1885</span> // Server Helpers<a name="line.1885"></a>
-<span class="sourceLineNo">1886</span> // ============================================================================================<a name="line.1886"></a>
-<span class="sourceLineNo">1887</span> @Override<a name="line.1887"></a>
-<span class="sourceLineNo">1888</span> public void serverAdded(final ServerName serverName) {<a name="line.1888"></a>
-<span class="sourceLineNo">1889</span> }<a name="line.1889"></a>
-<span class="sourceLineNo">1890</span><a name="line.1890"></a>
-<span class="sourceLineNo">1891</span> @Override<a name="line.1891"></a>
-<span class="sourceLineNo">1892</span> public void serverRemoved(final ServerName serverName) {<a name="line.1892"></a>
-<span class="sourceLineNo">1893</span> final ServerStateNode serverNode = regionStates.getServerNode(serverName);<a name="line.1893"></a>
-<span class="sourceLineNo">1894</span> if (serverNode == null) return;<a name="line.1894"></a>
-<span class="sourceLineNo">1895</span><a name="line.1895"></a>
-<span class="sourceLineNo">1896</span> // just in case, wake procedures waiting for this server report<a name="line.1896"></a>
-<span class="sourceLineNo">1897</span> wakeServerReportEvent(serverNode);<a name="line.1897"></a>
-<span class="sourceLineNo">1898</span> }<a name="line.1898"></a>
-<span class="sourceLineNo">1899</span><a name="line.1899"></a>
-<span class="sourceLineNo">1900</span> public int getServerVersion(final ServerName serverName) {<a name="line.1900"></a>
-<span class="sourceLineNo">1901</span> final ServerStateNode node = regionStates.getServerNode(serverName);<a name="line.1901"></a>
-<span class="sourceLineNo">1902</span> return node != null ? node.getVersionNumber() : 0;<a name="line.1902"></a>
-<span class="sourceLineNo">1903</span> }<a name="line.1903"></a>
-<span class="sourceLineNo">1904</span><a name="line.1904"></a>
-<span class="sourceLineNo">1905</span> public void killRegionServer(final ServerName serverName) {<a name="line.1905"></a>
-<span class="sourceLineNo">1906</span> final ServerStateNode serverNode = regionStates.getServerNode(serverName);<a name="line.1906"></a>
-<span class="sourceLineNo">1907</span> killRegionServer(serverNode);<a name="line.1907"></a>
-<span class="sourceLineNo">1908</span> }<a name="line.1908"></a>
-<span class="sourceLineNo">1909</span><a name="line.1909"></a>
-<span class="sourceLineNo">1910</span> public void killRegionServer(final ServerStateNode serverNode) {<a name="line.1910"></a>
-<span class="sourceLineNo">1911</span> /** Don't do this. Messes up accounting. Let ServerCrashProcedure do this.<a name="line.1911"></a>
-<span class="sourceLineNo">1912</span> for (RegionStateNode regionNode: serverNode.getRegions()) {<a name="line.1912"></a>
-<span class="sourceLineNo">1913</span> regionNode.offline();<a name="line.1913"></a>
-<span class="sourceLineNo">1914</span> }*/<a name="line.1914"></a>
-<span class="sourceLineNo">1915</span> master.getServerManager().expireServer(serverNode.getServerName());<a name="line.1915"></a>
-<span class="sourceLineNo">1916</span> }<a name="line.1916"></a>
-<span class="sourceLineNo">1917</span><a name="line.1917"></a>
-<span class="sourceLineNo">1918</span> /**<a name="line.1918"></a>
-<span class="sourceLineNo">1919</span> * Handle RIT of meta region against crashed server.<a name="line.1919"></a>
-<span class="sourceLineNo">1920</span> * Only used when ServerCrashProcedure is not enabled.<a name="line.1920"></a>
-<span class="sourceLineNo">1921</span> * See handleRIT in ServerCrashProcedure for similar function.<a name="line.1921"></a>
-<span class="sourceLineNo">1922</span> *<a name="line.1922"></a>
-<span class="sourceLineNo">1923</span> * @param serverName Server that has already crashed<a name="line.1923"></a>
-<span class="sourceLineNo">1924</span> */<a name="line.1924"></a>
-<span class="sourceLineNo">1925</span> public void handleMetaRITOnCrashedServer(ServerName serverName) {<a name="line.1925"></a>
-<span class="sourceLineNo">1926</span> RegionInfo hri = RegionReplicaUtil<a name="line.1926"></a>
-<span class="sourceLineNo">1927</span> .getRegionInfoForReplica(RegionInfoBuilder.FIRST_META_REGIONINFO,<a name="line.1927"></a>
-<span class="sourceLineNo">1928</span> RegionInfo.DEFAULT_REPLICA_ID);<a name="line.1928"></a>
-<span class="sourceLineNo">1929</span> RegionState regionStateNode = getRegionStates().getRegionState(hri);<a name="line.1929"></a>
-<span class="sourceLineNo">1930</span> if (regionStateNode == null) {<a name="line.1930"></a>
-<span class="sourceLineNo">1931</span> LOG.warn("RegionStateNode is null for " + hri);<a name="line.1931"></a>
-<span class="sourceLineNo">1932</span> return;<a name="line.1932"></a>
-<span class="sourceLineNo">1933</span> }<a name="line.1933"></a>
-<span class="sourceLineNo">1934</span> ServerName rsnServerName = regionStateNode.getServerName();<a name="line.1934"></a>
-<span class="sourceLineNo">1935</span> if (rsnServerName != null && !rsnServerName.equals(serverName)) {<a name="line.1935"></a>
-<span class="sourceLineNo">1936</span> return;<a name="line.1936"></a>
-<span class="sourceLineNo">1937</span> } else if (rsnServerName == null) {<a name="line.1937"></a>
-<span class="sourceLineNo">1938</span> LOG.warn("Empty ServerName in RegionStateNode; proceeding anyways in case latched " +<a name="line.1938"></a>
-<span class="sourceLineNo">1939</span> "RecoverMetaProcedure so meta latch gets cleaned up.");<a name="line.1939"></a>
-<span class="sourceLineNo">1940</span> }<a name="line.1940"></a>
-<span class="sourceLineNo">1941</span> // meta has been assigned to crashed server.<a name="line.1941"></a>
-<span class="sourceLineNo">1942</span> LOG.info("Meta assigned to crashed " + serverName + "; reassigning...");<a name="line.1942"></a>
-<span class="sourceLineNo">1943</span> // Handle failure and wake event<a name="line.1943"></a>
-<span class="sourceLineNo">1944</span> RegionTransitionProcedure rtp = getRegionStates().getRegionTransitionProcedure(hri);<a name="line.1944"></a>
-<span class="sourceLineNo">1945</span> // Do not need to consider for REGION_TRANSITION_QUEUE step<a name="line.1945"></a>
-<span class="sourceLineNo">1946</span> if (rtp != null && rtp.isMeta() &&<a name="line.1946"></a>
-<span class="sourceLineNo">1947</span> rtp.getTransitionState() == RegionTransitionState.REGION_TRANSITION_DISPATCH) {<a name="line.1947"></a>
-<span class="sourceLineNo">1948</span> LOG.debug("Failing " + rtp.toString());<a name="line.1948"></a>
-<span class="sourceLineNo">1949</span> rtp.remoteCallFailed(master.getMasterProcedureExecutor().getEnvironment(), serverName,<a name="line.1949"></a>
-<span class="sourceLineNo">1950</span> new ServerCrashException(rtp.getProcId(), serverName));<a name="line.1950"></a>
-<span class="sourceLineNo">1951</span> }<a name="line.1951"></a>
-<span class="sourceLineNo">1952</span> }<a name="line.1952"></a>
-<span class="sourceLineNo">1953</span>}<a name="line.1953"></a>
+<span class="sourceLineNo">1260</span> isTableState(regionNode.getTable(), TableState.State.DISABLED,<a name="line.1260"></a>
+<span class="sourceLineNo">1261</span> TableState.State.DISABLING)) {<a name="line.1261"></a>
+<span class="sourceLineNo">1262</span> // The region is CLOSED and the table is DISABLED/ DISABLING, there is nothing to<a name="line.1262"></a>
+<span class="sourceLineNo">1263</span> // schedule; the region is inert.<a name="line.1263"></a>
+<span class="sourceLineNo">1264</span> } else {<a name="line.1264"></a>
+<span class="sourceLineNo">1265</span> // These regions should have a procedure in replay<a name="line.1265"></a>
+<span class="sourceLineNo">1266</span> regionStates.addRegionInTransition(regionNode, null);<a name="line.1266"></a>
+<span class="sourceLineNo">1267</span> }<a name="line.1267"></a>
+<span class="sourceLineNo">1268</span> }<a name="line.1268"></a>
+<span class="sourceLineNo">1269</span> }<a name="line.1269"></a>
+<span class="sourceLineNo">1270</span> }<a name="line.1270"></a>
+<span class="sourceLineNo">1271</span> });<a name="line.1271"></a>
+<span class="sourceLineNo">1272</span><a name="line.1272"></a>
+<span class="sourceLineNo">1273</span> // every assignment is blocked until meta is loaded.<a name="line.1273"></a>
+<span class="sourceLineNo">1274</span> wakeMetaLoadedEvent();<a name="line.1274"></a>
+<span class="sourceLineNo">1275</span> }<a name="line.1275"></a>
+<span class="sourceLineNo">1276</span><a name="line.1276"></a>
+<span class="sourceLineNo">1277</span> /**<a name="line.1277"></a>
+<span class="sourceLineNo">1278</span> * Look at what is in meta and the list of servers that have checked in and make reconciliation.<a name="line.1278"></a>
+<span class="sourceLineNo">1279</span> * We cannot tell definitively the difference between a clean shutdown and a cluster that has<a name="line.1279"></a>
+<span class="sourceLineNo">1280</span> * been crashed down. At this stage of a Master startup, they look the same: they have the<a name="line.1280"></a>
+<span class="sourceLineNo">1281</span> * same state in hbase:meta. We could do detective work probing ZK and the FS for old WALs to<a name="line.1281"></a>
+<span class="sourceLineNo">1282</span> * split but SCP does this already so just let it do its job.<a name="line.1282"></a>
+<span class="sourceLineNo">1283</span> * <p>>The profiles of clean shutdown and cluster crash-down are the same because on clean<a name="line.1283"></a>
+<span class="sourceLineNo">1284</span> * shutdown currently, we do not update hbase:meta with region close state (In AMv2, region<a name="line.1284"></a>
+<span class="sourceLineNo">1285</span> * state is kept in hbse:meta). Usually the master runs all region transitions as of AMv2 but on<a name="line.1285"></a>
+<span class="sourceLineNo">1286</span> * cluster controlled shutdown, the RegionServers close all their regions only reporting the<a name="line.1286"></a>
+<span class="sourceLineNo">1287</span> * final change to the Master. Currently this report is ignored. Later we could take it and<a name="line.1287"></a>
+<span class="sourceLineNo">1288</span> * update as many regions as we can before hbase:meta goes down or have the master run the<a name="line.1288"></a>
+<span class="sourceLineNo">1289</span> * close of all regions out on the cluster but we may never be able to achieve the proper state on<a name="line.1289"></a>
+<span class="sourceLineNo">1290</span> * all regions (at least not w/o lots of painful manipulations and waiting) so clean shutdown<a name="line.1290"></a>
+<span class="sourceLineNo">1291</span> * might not be possible especially on big clusters.... And clean shutdown will take time. Given<a name="line.1291"></a>
+<span class="sourceLineNo">1292</span> * this current state of affairs, we just run ServerCrashProcedure in both cases. It will always<a name="line.1292"></a>
+<span class="sourceLineNo">1293</span> * do the right thing.<a name="line.1293"></a>
+<span class="sourceLineNo">1294</span> * @return True if for sure this is a failover where a Master is starting up into an already<a name="line.1294"></a>
+<span class="sourceLineNo">1295</span> * running cluster.<a name="line.1295"></a>
+<span class="sourceLineNo">1296</span> */<a name="line.1296"></a>
+<span class="sourceLineNo">1297</span> // The assumption here is that if RSs are crashing while we are executing this<a name="line.1297"></a>
+<span class="sourceLineNo">1298</span> // they will be handled by the SSH that are put in the ServerManager deadservers "queue".<a name="line.1298"></a>
+<span class="sourceLineNo">1299</span> private boolean processofflineServersWithOnlineRegions() {<a name="line.1299"></a>
+<span class="sourceLineNo">1300</span> boolean deadServers = !master.getServerManager().getDeadServers().isEmpty();<a name="line.1300"></a>
+<span class="sourceLineNo">1301</span> final Set<ServerName> offlineServersWithOnlineRegions = new HashSet<>();<a name="line.1301"></a>
+<span class="sourceLineNo">1302</span> int size = regionStates.getRegionStateNodes().size();<a name="line.1302"></a>
+<span class="sourceLineNo">1303</span> final List<RegionInfo> offlineRegionsToAssign = new ArrayList<>(size);<a name="line.1303"></a>
+<span class="sourceLineNo">1304</span> // If deadservers then its a failover, else, we are not sure yet.<a name="line.1304"></a>
+<span class="sourceLineNo">1305</span> boolean failover = deadServers;<a name="line.1305"></a>
+<span class="sourceLineNo">1306</span> for (RegionStateNode regionNode: regionStates.getRegionStateNodes()) {<a name="line.1306"></a>
+<span class="sourceLineNo">1307</span> // Region State can be OPEN even if we did controlled cluster shutdown; Master does not close<a name="line.1307"></a>
+<span class="sourceLineNo">1308</span> // the regions in this case. The RegionServer does the close so hbase:meta is state in<a name="line.1308"></a>
+<span class="sourceLineNo">1309</span> // hbase:meta is not updated -- Master does all updates -- and is left with OPEN as region<a name="line.1309"></a>
+<span class="sourceLineNo">1310</span> // state in meta. How to tell difference between ordered shutdown and crashed-down cluster<a name="line.1310"></a>
+<span class="sourceLineNo">1311</span> // then? We can't. Not currently. Perhaps if we updated hbase:meta with CLOSED on ordered<a name="line.1311"></a>
+<span class="sourceLineNo">1312</span> // shutdown. This would slow shutdown though and not all edits would make it in anyways.<a name="line.1312"></a>
+<span class="sourceLineNo">1313</span> // TODO: Examine.<a name="line.1313"></a>
+<span class="sourceLineNo">1314</span> // Because we can't be sure it an ordered shutdown, we run ServerCrashProcedure always.<a name="line.1314"></a>
+<span class="sourceLineNo">1315</span> // ServerCrashProcedure will try to retain old deploy when it goes to assign.<a name="line.1315"></a>
+<span class="sourceLineNo">1316</span> if (regionNode.getState() == State.OPEN) {<a name="line.1316"></a>
+<span class="sourceLineNo">1317</span> final ServerName serverName = regionNode.getRegionLocation();<a name="line.1317"></a>
+<span class="sourceLineNo">1318</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1318"></a>
+<span class="sourceLineNo">1319</span> offlineServersWithOnlineRegions.add(serverName);<a name="line.1319"></a>
+<span class="sourceLineNo">1320</span> } else {<a name="line.1320"></a>
+<span class="sourceLineNo">1321</span> // Server is online. This a failover. Master is starting into already-running cluster.<a name="line.1321"></a>
+<span class="sourceLineNo">1322</span> failover = true;<a name="line.1322"></a>
+<span class="sourceLineNo">1323</span> }<a name="line.1323"></a>
+<span class="sourceLineNo">1324</span> } else if (regionNode.getState() == State.OFFLINE) {<a name="line.1324"></a>
+<span class="sourceLineNo">1325</span> if (isTableEnabled(regionNode.getTable())) {<a name="line.1325"></a>
+<span class="sourceLineNo">1326</span> offlineRegionsToAssign.add(regionNode.getRegionInfo());<a name="line.1326"></a>
+<span class="sourceLineNo">1327</span> }<a name="line.1327"></a>
+<span class="sourceLineNo">1328</span> }<a name="line.1328"></a>
+<span class="sourceLineNo">1329</span> }<a name="line.1329"></a>
+<span class="sourceLineNo">1330</span> // Kill servers with online regions just-in-case. Runs ServerCrashProcedure.<a name="line.1330"></a>
+<span class="sourceLineNo">1331</span> for (ServerName serverName: offlineServersWithOnlineRegions) {<a name="line.1331"></a>
+<span class="sourceLineNo">1332</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1332"></a>
+<span class="sourceLineNo">1333</span> LOG.info("KILL RegionServer=" + serverName + " hosting regions but not online.");<a name="line.1333"></a>
+<span class="sourceLineNo">1334</span> killRegionServer(serverName);<a name="line.1334"></a>
+<span class="sourceLineNo">1335</span> }<a name="line.1335"></a>
+<span class="sourceLineNo">1336</span> }<a name="line.1336"></a>
+<span class="sourceLineNo">1337</span> setFailoverCleanupDone(true);<a name="line.1337"></a>
+<span class="sourceLineNo">1338</span><a name="line.1338"></a>
+<span class="sourceLineNo">1339</span> // Assign offline regions. Uses round-robin.<a name="line.1339"></a>
+<span class="sourceLineNo">1340</span> if (offlineRegionsToAssign.size() > 0) {<a name="line.1340"></a>
+<span class="sourceLineNo">1341</span> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().<a name="line.1341"></a>
+<span class="sourceLineNo">1342</span> createRoundRobinAssignProcedures(offlineRegionsToAssign));<a name="line.1342"></a>
+<span class="sourceLineNo">1343</span> }<a name="line.1343"></a>
+<span class="sourceLineNo">1344</span><a name="line.1344"></a>
+<span class="sourceLineNo">1345</span> return failover;<a name="line.1345"></a>
+<span class="sourceLineNo">1346</span> }<a name="line.1346"></a>
+<span class="sourceLineNo">1347</span><a name="line.1347"></a>
+<span class="sourceLineNo">1348</span> /**<a name="line.1348"></a>
+<span class="sourceLineNo">1349</span> * Used by ServerCrashProcedure to make sure AssignmentManager has completed<a name="line.1349"></a>
+<span class="sourceLineNo">1350</span> * the failover cleanup before re-assigning regions of dead servers. So that<a name="line.1350"></a>
+<span class="sourceLineNo">1351</span> * when re-assignment happens, AssignmentManager has proper region states.<a name="line.1351"></a>
+<span class="sourceLineNo">1352</span> */<a name="line.1352"></a>
+<span class="sourceLineNo">1353</span> public boolean isFailoverCleanupDone() {<a name="line.1353"></a>
+<span class="sourceLineNo">1354</span> return failoverCleanupDone.isReady();<a name="line.1354"></a>
+<span class="sourceLineNo">1355</span> }<a name="line.1355"></a>
+<span class="sourceLineNo">1356</span><a name="line.1356"></a>
+<span class="sourceLineNo">1357</span> /**<a name="line.1357"></a>
+<span class="sourceLineNo">1358</span> * Used by ServerCrashProcedure tests verify the ability to suspend the<a name="line.1358"></a>
+<span class="sourceLineNo">1359</span> * execution of the ServerCrashProcedure.<a name="line.1359"></a>
+<span class="sourceLineNo">1360</span> */<a name="line.1360"></a>
+<span class="sourceLineNo">1361</span> @VisibleForTesting<a name="line.1361"></a>
+<span class="sourceLineNo">1362</span> public void setFailoverCleanupDone(final boolean b) {<a name="line.1362"></a>
+<span class="sourceLineNo">1363</span> master.getMasterProcedureExecutor().getEnvironment()<a name="line.1363"></a>
+<span class="sourceLineNo">1364</span> .setEventReady(failoverCleanupDone, b);<a name="line.1364"></a>
+<span class="sourceLineNo">1365</span> }<a name="line.1365"></a>
+<span class="sourceLineNo">1366</span><a name="line.1366"></a>
+<span class="sourceLineNo">1367</span> public ProcedureEvent getFailoverCleanupEvent() {<a name="line.1367"></a>
+<span class="sourceLineNo">1368</span> return failoverCleanupDone;<a name="line.1368"></a>
+<span class="sourceLineNo">1369</span> }<a name="line.1369"></a>
+<span class="sourceLineNo">1370</span><a name="line.1370"></a>
+<span class="sourceLineNo">1371</span> /**<a name="line.1371"></a>
+<span class="sourceLineNo">1372</span> * Used to check if the failover cleanup is done.<a name="line.1372"></a>
+<span class="sourceLineNo">1373</span> * if not we throw PleaseHoldException since we are rebuilding the RegionStates<a name="line.1373"></a>
+<span class="sourceLineNo">1374</span> * @param hri region to check if it is already rebuild<a name="line.1374"></a>
+<span class="sourceLineNo">1375</span> * @throws PleaseHoldException if the failover cleanup is not completed<a name="line.1375"></a>
+<span class="sourceLineNo">1376</span> */<a name="line.1376"></a>
+<span class="sourceLineNo">1377</span> private void checkFailoverCleanupCompleted(final RegionInfo hri) throws PleaseHoldException {<a name="line.1377"></a>
+<span class="sourceLineNo">1378</span> if (!isRunning()) {<a name="line.1378"></a>
+<span class="sourceLineNo">1379</span> throw new PleaseHoldException("AssignmentManager not running");<a name="line.1379"></a>
+<span class="sourceLineNo">1380</span> }<a name="line.1380"></a>
+<span class="sourceLineNo">1381</span><a name="line.1381"></a>
+<span class="sourceLineNo">1382</span> // TODO: can we avoid throwing an exception if hri is already loaded?<a name="line.1382"></a>
+<span class="sourceLineNo">1383</span> // at the moment we bypass only meta<a name="line.1383"></a>
+<span class="sourceLineNo">1384</span> boolean meta = isMetaRegion(hri);<a name="line.1384"></a>
+<span class="sourceLineNo">1385</span> boolean cleanup = isFailoverCleanupDone();<a name="line.1385"></a>
+<span class="sourceLineNo">1386</span> if (!isMetaRegion(hri) && !isFailoverCleanupDone()) {<a name="line.1386"></a>
+<span class="sourceLineNo">1387</span> String msg = "Master not fully online; hbase:meta=" + meta + ", failoverCleanup=" + cleanup;<a name="line.1387"></a>
+<span class="sourceLineNo">1388</span> throw new PleaseHoldException(msg);<a name="line.1388"></a>
+<span class="sourceLineNo">1389</span> }<a name="line.1389"></a>
+<span class="sourceLineNo">1390</span> }<a name="line.1390"></a>
+<span class="sourceLineNo">1391</span><a name="line.1391"></a>
+<span class="sourceLineNo">1392</span> // ==================
<TRUNCATED>
[10/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureCleaner.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureCleaner.html b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureCleaner.html
index 97aa79c..cc787e3 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureCleaner.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureCleaner.html
@@ -272,1623 +272,1642 @@
<span class="sourceLineNo">264</span> private final CopyOnWriteArrayList<ProcedureExecutorListener> listeners = new CopyOnWriteArrayList<>();<a name="line.264"></a>
<span class="sourceLineNo">265</span><a name="line.265"></a>
<span class="sourceLineNo">266</span> private Configuration conf;<a name="line.266"></a>
-<span class="sourceLineNo">267</span> private ThreadGroup threadGroup;<a name="line.267"></a>
-<span class="sourceLineNo">268</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.268"></a>
-<span class="sourceLineNo">269</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.269"></a>
-<span class="sourceLineNo">270</span> private int corePoolSize;<a name="line.270"></a>
-<span class="sourceLineNo">271</span> private int maxPoolSize;<a name="line.271"></a>
-<span class="sourceLineNo">272</span><a name="line.272"></a>
-<span class="sourceLineNo">273</span> private volatile long keepAliveTime;<a name="line.273"></a>
-<span class="sourceLineNo">274</span><a name="line.274"></a>
-<span class="sourceLineNo">275</span> /**<a name="line.275"></a>
-<span class="sourceLineNo">276</span> * Scheduler/Queue that contains runnable procedures.<a name="line.276"></a>
-<span class="sourceLineNo">277</span> */<a name="line.277"></a>
-<span class="sourceLineNo">278</span> private final ProcedureScheduler scheduler;<a name="line.278"></a>
-<span class="sourceLineNo">279</span><a name="line.279"></a>
-<span class="sourceLineNo">280</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.280"></a>
-<span class="sourceLineNo">281</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.281"></a>
-<span class="sourceLineNo">282</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.282"></a>
-<span class="sourceLineNo">283</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.283"></a>
-<span class="sourceLineNo">284</span> private final TEnvironment environment;<a name="line.284"></a>
-<span class="sourceLineNo">285</span> private final ProcedureStore store;<a name="line.285"></a>
-<span class="sourceLineNo">286</span><a name="line.286"></a>
-<span class="sourceLineNo">287</span> private final boolean checkOwnerSet;<a name="line.287"></a>
-<span class="sourceLineNo">288</span><a name="line.288"></a>
-<span class="sourceLineNo">289</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.289"></a>
-<span class="sourceLineNo">290</span> final ProcedureStore store) {<a name="line.290"></a>
-<span class="sourceLineNo">291</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.291"></a>
-<span class="sourceLineNo">292</span> }<a name="line.292"></a>
-<span class="sourceLineNo">293</span><a name="line.293"></a>
-<span class="sourceLineNo">294</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.294"></a>
-<span class="sourceLineNo">295</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.295"></a>
-<span class="sourceLineNo">296</span> this.environment = environment;<a name="line.296"></a>
-<span class="sourceLineNo">297</span> this.scheduler = scheduler;<a name="line.297"></a>
-<span class="sourceLineNo">298</span> this.store = store;<a name="line.298"></a>
-<span class="sourceLineNo">299</span> this.conf = conf;<a name="line.299"></a>
-<span class="sourceLineNo">300</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.300"></a>
-<span class="sourceLineNo">301</span> refreshConfiguration(conf);<a name="line.301"></a>
-<span class="sourceLineNo">302</span> }<a name="line.302"></a>
-<span class="sourceLineNo">303</span><a name="line.303"></a>
-<span class="sourceLineNo">304</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.304"></a>
-<span class="sourceLineNo">305</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.305"></a>
-<span class="sourceLineNo">306</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.306"></a>
-<span class="sourceLineNo">307</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.307"></a>
-<span class="sourceLineNo">308</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.308"></a>
-<span class="sourceLineNo">309</span><a name="line.309"></a>
-<span class="sourceLineNo">310</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.310"></a>
-<span class="sourceLineNo">311</span> @Override<a name="line.311"></a>
-<span class="sourceLineNo">312</span> public void setMaxProcId(long maxProcId) {<a name="line.312"></a>
-<span class="sourceLineNo">313</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.313"></a>
-<span class="sourceLineNo">314</span> lastProcId.set(maxProcId);<a name="line.314"></a>
-<span class="sourceLineNo">315</span> }<a name="line.315"></a>
-<span class="sourceLineNo">316</span><a name="line.316"></a>
-<span class="sourceLineNo">317</span> @Override<a name="line.317"></a>
-<span class="sourceLineNo">318</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.318"></a>
-<span class="sourceLineNo">319</span> loadProcedures(procIter, abortOnCorruption);<a name="line.319"></a>
-<span class="sourceLineNo">320</span> }<a name="line.320"></a>
-<span class="sourceLineNo">321</span><a name="line.321"></a>
-<span class="sourceLineNo">322</span> @Override<a name="line.322"></a>
-<span class="sourceLineNo">323</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.323"></a>
-<span class="sourceLineNo">324</span> int corruptedCount = 0;<a name="line.324"></a>
-<span class="sourceLineNo">325</span> while (procIter.hasNext()) {<a name="line.325"></a>
-<span class="sourceLineNo">326</span> Procedure<?> proc = procIter.next();<a name="line.326"></a>
-<span class="sourceLineNo">327</span> LOG.error("Corrupt " + proc);<a name="line.327"></a>
-<span class="sourceLineNo">328</span> corruptedCount++;<a name="line.328"></a>
-<span class="sourceLineNo">329</span> }<a name="line.329"></a>
-<span class="sourceLineNo">330</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.330"></a>
-<span class="sourceLineNo">331</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.331"></a>
-<span class="sourceLineNo">332</span> }<a name="line.332"></a>
-<span class="sourceLineNo">333</span> }<a name="line.333"></a>
-<span class="sourceLineNo">334</span> });<a name="line.334"></a>
-<span class="sourceLineNo">335</span> }<a name="line.335"></a>
-<span class="sourceLineNo">336</span><a name="line.336"></a>
-<span class="sourceLineNo">337</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.337"></a>
-<span class="sourceLineNo">338</span> final boolean abortOnCorruption) throws IOException {<a name="line.338"></a>
-<span class="sourceLineNo">339</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.339"></a>
-<span class="sourceLineNo">340</span><a name="line.340"></a>
-<span class="sourceLineNo">341</span> // 1. Build the rollback stack<a name="line.341"></a>
-<span class="sourceLineNo">342</span> int runnablesCount = 0;<a name="line.342"></a>
-<span class="sourceLineNo">343</span> while (procIter.hasNext()) {<a name="line.343"></a>
-<span class="sourceLineNo">344</span> boolean finished = procIter.isNextFinished();<a name="line.344"></a>
-<span class="sourceLineNo">345</span> Procedure proc = procIter.next();<a name="line.345"></a>
-<span class="sourceLineNo">346</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.346"></a>
-<span class="sourceLineNo">347</span> long procId = proc.getProcId();<a name="line.347"></a>
-<span class="sourceLineNo">348</span><a name="line.348"></a>
-<span class="sourceLineNo">349</span> if (finished) {<a name="line.349"></a>
-<span class="sourceLineNo">350</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.350"></a>
-<span class="sourceLineNo">351</span> if (debugEnabled) {<a name="line.351"></a>
-<span class="sourceLineNo">352</span> LOG.debug("Completed " + proc);<a name="line.352"></a>
-<span class="sourceLineNo">353</span> }<a name="line.353"></a>
-<span class="sourceLineNo">354</span> } else {<a name="line.354"></a>
-<span class="sourceLineNo">355</span> if (!proc.hasParent()) {<a name="line.355"></a>
-<span class="sourceLineNo">356</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.356"></a>
-<span class="sourceLineNo">357</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.357"></a>
-<span class="sourceLineNo">358</span> }<a name="line.358"></a>
+<span class="sourceLineNo">267</span><a name="line.267"></a>
+<span class="sourceLineNo">268</span> /**<a name="line.268"></a>
+<span class="sourceLineNo">269</span> * Created in the {@link #start(int, boolean)} method. Destroyed in {@link #join()} (FIX! Doing<a name="line.269"></a>
+<span class="sourceLineNo">270</span> * resource handling rather than observing in a #join is unexpected).<a name="line.270"></a>
+<span class="sourceLineNo">271</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.271"></a>
+<span class="sourceLineNo">272</span> * (Should be ok).<a name="line.272"></a>
+<span class="sourceLineNo">273</span> */<a name="line.273"></a>
+<span class="sourceLineNo">274</span> private ThreadGroup threadGroup;<a name="line.274"></a>
+<span class="sourceLineNo">275</span><a name="line.275"></a>
+<span class="sourceLineNo">276</span> /**<a name="line.276"></a>
+<span class="sourceLineNo">277</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.277"></a>
+<span class="sourceLineNo">278</span> * resource handling rather than observing in a #join is unexpected).<a name="line.278"></a>
+<span class="sourceLineNo">279</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.279"></a>
+<span class="sourceLineNo">280</span> * (Should be ok).<a name="line.280"></a>
+<span class="sourceLineNo">281</span> */<a name="line.281"></a>
+<span class="sourceLineNo">282</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.282"></a>
+<span class="sourceLineNo">283</span><a name="line.283"></a>
+<span class="sourceLineNo">284</span> /**<a name="line.284"></a>
+<span class="sourceLineNo">285</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.285"></a>
+<span class="sourceLineNo">286</span> * resource handling rather than observing in a #join is unexpected).<a name="line.286"></a>
+<span class="sourceLineNo">287</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.287"></a>
+<span class="sourceLineNo">288</span> * (Should be ok).<a name="line.288"></a>
+<span class="sourceLineNo">289</span> */<a name="line.289"></a>
+<span class="sourceLineNo">290</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.290"></a>
+<span class="sourceLineNo">291</span><a name="line.291"></a>
+<span class="sourceLineNo">292</span> private int corePoolSize;<a name="line.292"></a>
+<span class="sourceLineNo">293</span> private int maxPoolSize;<a name="line.293"></a>
+<span class="sourceLineNo">294</span><a name="line.294"></a>
+<span class="sourceLineNo">295</span> private volatile long keepAliveTime;<a name="line.295"></a>
+<span class="sourceLineNo">296</span><a name="line.296"></a>
+<span class="sourceLineNo">297</span> /**<a name="line.297"></a>
+<span class="sourceLineNo">298</span> * Scheduler/Queue that contains runnable procedures.<a name="line.298"></a>
+<span class="sourceLineNo">299</span> */<a name="line.299"></a>
+<span class="sourceLineNo">300</span> private final ProcedureScheduler scheduler;<a name="line.300"></a>
+<span class="sourceLineNo">301</span><a name="line.301"></a>
+<span class="sourceLineNo">302</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.302"></a>
+<span class="sourceLineNo">303</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.303"></a>
+<span class="sourceLineNo">304</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.304"></a>
+<span class="sourceLineNo">305</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.305"></a>
+<span class="sourceLineNo">306</span> private final TEnvironment environment;<a name="line.306"></a>
+<span class="sourceLineNo">307</span> private final ProcedureStore store;<a name="line.307"></a>
+<span class="sourceLineNo">308</span><a name="line.308"></a>
+<span class="sourceLineNo">309</span> private final boolean checkOwnerSet;<a name="line.309"></a>
+<span class="sourceLineNo">310</span><a name="line.310"></a>
+<span class="sourceLineNo">311</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.311"></a>
+<span class="sourceLineNo">312</span> final ProcedureStore store) {<a name="line.312"></a>
+<span class="sourceLineNo">313</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.313"></a>
+<span class="sourceLineNo">314</span> }<a name="line.314"></a>
+<span class="sourceLineNo">315</span><a name="line.315"></a>
+<span class="sourceLineNo">316</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.316"></a>
+<span class="sourceLineNo">317</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.317"></a>
+<span class="sourceLineNo">318</span> this.environment = environment;<a name="line.318"></a>
+<span class="sourceLineNo">319</span> this.scheduler = scheduler;<a name="line.319"></a>
+<span class="sourceLineNo">320</span> this.store = store;<a name="line.320"></a>
+<span class="sourceLineNo">321</span> this.conf = conf;<a name="line.321"></a>
+<span class="sourceLineNo">322</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.322"></a>
+<span class="sourceLineNo">323</span> refreshConfiguration(conf);<a name="line.323"></a>
+<span class="sourceLineNo">324</span><a name="line.324"></a>
+<span class="sourceLineNo">325</span> }<a name="line.325"></a>
+<span class="sourceLineNo">326</span><a name="line.326"></a>
+<span class="sourceLineNo">327</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.327"></a>
+<span class="sourceLineNo">328</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.328"></a>
+<span class="sourceLineNo">329</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.329"></a>
+<span class="sourceLineNo">330</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.330"></a>
+<span class="sourceLineNo">331</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.331"></a>
+<span class="sourceLineNo">332</span><a name="line.332"></a>
+<span class="sourceLineNo">333</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.333"></a>
+<span class="sourceLineNo">334</span> @Override<a name="line.334"></a>
+<span class="sourceLineNo">335</span> public void setMaxProcId(long maxProcId) {<a name="line.335"></a>
+<span class="sourceLineNo">336</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.336"></a>
+<span class="sourceLineNo">337</span> lastProcId.set(maxProcId);<a name="line.337"></a>
+<span class="sourceLineNo">338</span> }<a name="line.338"></a>
+<span class="sourceLineNo">339</span><a name="line.339"></a>
+<span class="sourceLineNo">340</span> @Override<a name="line.340"></a>
+<span class="sourceLineNo">341</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.341"></a>
+<span class="sourceLineNo">342</span> loadProcedures(procIter, abortOnCorruption);<a name="line.342"></a>
+<span class="sourceLineNo">343</span> }<a name="line.343"></a>
+<span class="sourceLineNo">344</span><a name="line.344"></a>
+<span class="sourceLineNo">345</span> @Override<a name="line.345"></a>
+<span class="sourceLineNo">346</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.346"></a>
+<span class="sourceLineNo">347</span> int corruptedCount = 0;<a name="line.347"></a>
+<span class="sourceLineNo">348</span> while (procIter.hasNext()) {<a name="line.348"></a>
+<span class="sourceLineNo">349</span> Procedure<?> proc = procIter.next();<a name="line.349"></a>
+<span class="sourceLineNo">350</span> LOG.error("Corrupt " + proc);<a name="line.350"></a>
+<span class="sourceLineNo">351</span> corruptedCount++;<a name="line.351"></a>
+<span class="sourceLineNo">352</span> }<a name="line.352"></a>
+<span class="sourceLineNo">353</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.353"></a>
+<span class="sourceLineNo">354</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.354"></a>
+<span class="sourceLineNo">355</span> }<a name="line.355"></a>
+<span class="sourceLineNo">356</span> }<a name="line.356"></a>
+<span class="sourceLineNo">357</span> });<a name="line.357"></a>
+<span class="sourceLineNo">358</span> }<a name="line.358"></a>
<span class="sourceLineNo">359</span><a name="line.359"></a>
-<span class="sourceLineNo">360</span> // add the procedure to the map<a name="line.360"></a>
-<span class="sourceLineNo">361</span> proc.beforeReplay(getEnvironment());<a name="line.361"></a>
-<span class="sourceLineNo">362</span> procedures.put(proc.getProcId(), proc);<a name="line.362"></a>
+<span class="sourceLineNo">360</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.360"></a>
+<span class="sourceLineNo">361</span> final boolean abortOnCorruption) throws IOException {<a name="line.361"></a>
+<span class="sourceLineNo">362</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.362"></a>
<span class="sourceLineNo">363</span><a name="line.363"></a>
-<span class="sourceLineNo">364</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.364"></a>
-<span class="sourceLineNo">365</span> runnablesCount++;<a name="line.365"></a>
-<span class="sourceLineNo">366</span> }<a name="line.366"></a>
-<span class="sourceLineNo">367</span> }<a name="line.367"></a>
-<span class="sourceLineNo">368</span><a name="line.368"></a>
-<span class="sourceLineNo">369</span> // add the nonce to the map<a name="line.369"></a>
-<span class="sourceLineNo">370</span> if (nonceKey != null) {<a name="line.370"></a>
-<span class="sourceLineNo">371</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.371"></a>
-<span class="sourceLineNo">372</span> }<a name="line.372"></a>
-<span class="sourceLineNo">373</span> }<a name="line.373"></a>
-<span class="sourceLineNo">374</span><a name="line.374"></a>
-<span class="sourceLineNo">375</span> // 2. Initialize the stacks<a name="line.375"></a>
-<span class="sourceLineNo">376</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.376"></a>
-<span class="sourceLineNo">377</span> HashSet<Procedure> waitingSet = null;<a name="line.377"></a>
-<span class="sourceLineNo">378</span> procIter.reset();<a name="line.378"></a>
-<span class="sourceLineNo">379</span> while (procIter.hasNext()) {<a name="line.379"></a>
-<span class="sourceLineNo">380</span> if (procIter.isNextFinished()) {<a name="line.380"></a>
-<span class="sourceLineNo">381</span> procIter.skipNext();<a name="line.381"></a>
-<span class="sourceLineNo">382</span> continue;<a name="line.382"></a>
-<span class="sourceLineNo">383</span> }<a name="line.383"></a>
-<span class="sourceLineNo">384</span><a name="line.384"></a>
-<span class="sourceLineNo">385</span> Procedure proc = procIter.next();<a name="line.385"></a>
-<span class="sourceLineNo">386</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.386"></a>
-<span class="sourceLineNo">387</span><a name="line.387"></a>
-<span class="sourceLineNo">388</span> if (debugEnabled) {<a name="line.388"></a>
-<span class="sourceLineNo">389</span> LOG.debug(String.format("Loading %s", proc));<a name="line.389"></a>
+<span class="sourceLineNo">364</span> // 1. Build the rollback stack<a name="line.364"></a>
+<span class="sourceLineNo">365</span> int runnablesCount = 0;<a name="line.365"></a>
+<span class="sourceLineNo">366</span> while (procIter.hasNext()) {<a name="line.366"></a>
+<span class="sourceLineNo">367</span> boolean finished = procIter.isNextFinished();<a name="line.367"></a>
+<span class="sourceLineNo">368</span> Procedure proc = procIter.next();<a name="line.368"></a>
+<span class="sourceLineNo">369</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.369"></a>
+<span class="sourceLineNo">370</span> long procId = proc.getProcId();<a name="line.370"></a>
+<span class="sourceLineNo">371</span><a name="line.371"></a>
+<span class="sourceLineNo">372</span> if (finished) {<a name="line.372"></a>
+<span class="sourceLineNo">373</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.373"></a>
+<span class="sourceLineNo">374</span> if (debugEnabled) {<a name="line.374"></a>
+<span class="sourceLineNo">375</span> LOG.debug("Completed " + proc);<a name="line.375"></a>
+<span class="sourceLineNo">376</span> }<a name="line.376"></a>
+<span class="sourceLineNo">377</span> } else {<a name="line.377"></a>
+<span class="sourceLineNo">378</span> if (!proc.hasParent()) {<a name="line.378"></a>
+<span class="sourceLineNo">379</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.379"></a>
+<span class="sourceLineNo">380</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.380"></a>
+<span class="sourceLineNo">381</span> }<a name="line.381"></a>
+<span class="sourceLineNo">382</span><a name="line.382"></a>
+<span class="sourceLineNo">383</span> // add the procedure to the map<a name="line.383"></a>
+<span class="sourceLineNo">384</span> proc.beforeReplay(getEnvironment());<a name="line.384"></a>
+<span class="sourceLineNo">385</span> procedures.put(proc.getProcId(), proc);<a name="line.385"></a>
+<span class="sourceLineNo">386</span><a name="line.386"></a>
+<span class="sourceLineNo">387</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.387"></a>
+<span class="sourceLineNo">388</span> runnablesCount++;<a name="line.388"></a>
+<span class="sourceLineNo">389</span> }<a name="line.389"></a>
<span class="sourceLineNo">390</span> }<a name="line.390"></a>
<span class="sourceLineNo">391</span><a name="line.391"></a>
-<span class="sourceLineNo">392</span> Long rootProcId = getRootProcedureId(proc);<a name="line.392"></a>
-<span class="sourceLineNo">393</span> if (rootProcId == null) {<a name="line.393"></a>
-<span class="sourceLineNo">394</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.394"></a>
-<span class="sourceLineNo">395</span> scheduler.addBack(proc);<a name="line.395"></a>
-<span class="sourceLineNo">396</span> continue;<a name="line.396"></a>
-<span class="sourceLineNo">397</span> }<a name="line.397"></a>
-<span class="sourceLineNo">398</span><a name="line.398"></a>
-<span class="sourceLineNo">399</span> if (proc.hasParent()) {<a name="line.399"></a>
-<span class="sourceLineNo">400</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.400"></a>
-<span class="sourceLineNo">401</span> // corrupted procedures are handled later at step 3<a name="line.401"></a>
-<span class="sourceLineNo">402</span> if (parent != null && !proc.isFinished()) {<a name="line.402"></a>
-<span class="sourceLineNo">403</span> parent.incChildrenLatch();<a name="line.403"></a>
-<span class="sourceLineNo">404</span> }<a name="line.404"></a>
-<span class="sourceLineNo">405</span> }<a name="line.405"></a>
-<span class="sourceLineNo">406</span><a name="line.406"></a>
-<span class="sourceLineNo">407</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.407"></a>
-<span class="sourceLineNo">408</span> procStack.loadStack(proc);<a name="line.408"></a>
-<span class="sourceLineNo">409</span><a name="line.409"></a>
-<span class="sourceLineNo">410</span> proc.setRootProcId(rootProcId);<a name="line.410"></a>
-<span class="sourceLineNo">411</span> switch (proc.getState()) {<a name="line.411"></a>
-<span class="sourceLineNo">412</span> case RUNNABLE:<a name="line.412"></a>
-<span class="sourceLineNo">413</span> runnableList.add(proc);<a name="line.413"></a>
-<span class="sourceLineNo">414</span> break;<a name="line.414"></a>
-<span class="sourceLineNo">415</span> case WAITING:<a name="line.415"></a>
-<span class="sourceLineNo">416</span> if (!proc.hasChildren()) {<a name="line.416"></a>
-<span class="sourceLineNo">417</span> runnableList.add(proc);<a name="line.417"></a>
-<span class="sourceLineNo">418</span> }<a name="line.418"></a>
-<span class="sourceLineNo">419</span> break;<a name="line.419"></a>
-<span class="sourceLineNo">420</span> case WAITING_TIMEOUT:<a name="line.420"></a>
-<span class="sourceLineNo">421</span> if (waitingSet == null) {<a name="line.421"></a>
-<span class="sourceLineNo">422</span> waitingSet = new HashSet<>();<a name="line.422"></a>
-<span class="sourceLineNo">423</span> }<a name="line.423"></a>
-<span class="sourceLineNo">424</span> waitingSet.add(proc);<a name="line.424"></a>
-<span class="sourceLineNo">425</span> break;<a name="line.425"></a>
-<span class="sourceLineNo">426</span> case FAILED:<a name="line.426"></a>
-<span class="sourceLineNo">427</span> // add the proc to the scheduler to perform the rollback<a name="line.427"></a>
-<span class="sourceLineNo">428</span> scheduler.addBack(proc);<a name="line.428"></a>
-<span class="sourceLineNo">429</span> break;<a name="line.429"></a>
-<span class="sourceLineNo">430</span> case ROLLEDBACK:<a name="line.430"></a>
-<span class="sourceLineNo">431</span> case INITIALIZING:<a name="line.431"></a>
-<span class="sourceLineNo">432</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.432"></a>
-<span class="sourceLineNo">433</span> LOG.error(msg);<a name="line.433"></a>
-<span class="sourceLineNo">434</span> throw new UnsupportedOperationException(msg);<a name="line.434"></a>
-<span class="sourceLineNo">435</span> default:<a name="line.435"></a>
-<span class="sourceLineNo">436</span> break;<a name="line.436"></a>
-<span class="sourceLineNo">437</span> }<a name="line.437"></a>
-<span class="sourceLineNo">438</span> }<a name="line.438"></a>
-<span class="sourceLineNo">439</span><a name="line.439"></a>
-<span class="sourceLineNo">440</span> // 3. Validate the stacks<a name="line.440"></a>
-<span class="sourceLineNo">441</span> int corruptedCount = 0;<a name="line.441"></a>
-<span class="sourceLineNo">442</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.442"></a>
-<span class="sourceLineNo">443</span> while (itStack.hasNext()) {<a name="line.443"></a>
-<span class="sourceLineNo">444</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.444"></a>
-<span class="sourceLineNo">445</span> RootProcedureState procStack = entry.getValue();<a name="line.445"></a>
-<span class="sourceLineNo">446</span> if (procStack.isValid()) continue;<a name="line.446"></a>
-<span class="sourceLineNo">447</span><a name="line.447"></a>
-<span class="sourceLineNo">448</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.448"></a>
-<span class="sourceLineNo">449</span> LOG.error("Corrupted " + proc);<a name="line.449"></a>
-<span class="sourceLineNo">450</span> procedures.remove(proc.getProcId());<a name="line.450"></a>
-<span class="sourceLineNo">451</span> runnableList.remove(proc);<a name="line.451"></a>
-<span class="sourceLineNo">452</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.452"></a>
-<span class="sourceLineNo">453</span> corruptedCount++;<a name="line.453"></a>
-<span class="sourceLineNo">454</span> }<a name="line.454"></a>
-<span class="sourceLineNo">455</span> itStack.remove();<a name="line.455"></a>
-<span class="sourceLineNo">456</span> }<a name="line.456"></a>
-<span class="sourceLineNo">457</span><a name="line.457"></a>
-<span class="sourceLineNo">458</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.458"></a>
-<span class="sourceLineNo">459</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.459"></a>
-<span class="sourceLineNo">460</span> }<a name="line.460"></a>
-<span class="sourceLineNo">461</span><a name="line.461"></a>
-<span class="sourceLineNo">462</span> // 4. Push the procedures to the timeout executor<a name="line.462"></a>
-<span class="sourceLineNo">463</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.463"></a>
-<span class="sourceLineNo">464</span> for (Procedure proc: waitingSet) {<a name="line.464"></a>
-<span class="sourceLineNo">465</span> proc.afterReplay(getEnvironment());<a name="line.465"></a>
-<span class="sourceLineNo">466</span> timeoutExecutor.add(proc);<a name="line.466"></a>
-<span class="sourceLineNo">467</span> }<a name="line.467"></a>
-<span class="sourceLineNo">468</span> }<a name="line.468"></a>
-<span class="sourceLineNo">469</span><a name="line.469"></a>
-<span class="sourceLineNo">470</span> // 5. Push the procedure to the scheduler<a name="line.470"></a>
-<span class="sourceLineNo">471</span> if (!runnableList.isEmpty()) {<a name="line.471"></a>
-<span class="sourceLineNo">472</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.472"></a>
-<span class="sourceLineNo">473</span> // some procedure may be started way before this stuff.<a name="line.473"></a>
-<span class="sourceLineNo">474</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.474"></a>
-<span class="sourceLineNo">475</span> Procedure proc = runnableList.get(i);<a name="line.475"></a>
-<span class="sourceLineNo">476</span> proc.afterReplay(getEnvironment());<a name="line.476"></a>
-<span class="sourceLineNo">477</span> if (!proc.hasParent()) {<a name="line.477"></a>
-<span class="sourceLineNo">478</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.478"></a>
-<span class="sourceLineNo">479</span> }<a name="line.479"></a>
-<span class="sourceLineNo">480</span> if (proc.wasExecuted()) {<a name="line.480"></a>
-<span class="sourceLineNo">481</span> scheduler.addFront(proc);<a name="line.481"></a>
-<span class="sourceLineNo">482</span> } else {<a name="line.482"></a>
-<span class="sourceLineNo">483</span> // if it was not in execution, it can wait.<a name="line.483"></a>
-<span class="sourceLineNo">484</span> scheduler.addBack(proc);<a name="line.484"></a>
-<span class="sourceLineNo">485</span> }<a name="line.485"></a>
-<span class="sourceLineNo">486</span> }<a name="line.486"></a>
-<span class="sourceLineNo">487</span> }<a name="line.487"></a>
-<span class="sourceLineNo">488</span> }<a name="line.488"></a>
-<span class="sourceLineNo">489</span><a name="line.489"></a>
-<span class="sourceLineNo">490</span> /**<a name="line.490"></a>
-<span class="sourceLineNo">491</span> * Start the procedure executor.<a name="line.491"></a>
-<span class="sourceLineNo">492</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.492"></a>
-<span class="sourceLineNo">493</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.493"></a>
-<span class="sourceLineNo">494</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.494"></a>
-<span class="sourceLineNo">495</span> *<a name="line.495"></a>
-<span class="sourceLineNo">496</span> * @param numThreads number of threads available for procedure execution.<a name="line.496"></a>
-<span class="sourceLineNo">497</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.497"></a>
-<span class="sourceLineNo">498</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.498"></a>
-<span class="sourceLineNo">499</span> */<a name="line.499"></a>
-<span class="sourceLineNo">500</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.500"></a>
-<span class="sourceLineNo">501</span> if (!running.compareAndSet(false, true)) {<a name="line.501"></a>
-<span class="sourceLineNo">502</span> LOG.warn("Already running");<a name="line.502"></a>
-<span class="sourceLineNo">503</span> return;<a name="line.503"></a>
-<span class="sourceLineNo">504</span> }<a name="line.504"></a>
-<span class="sourceLineNo">505</span><a name="line.505"></a>
-<span class="sourceLineNo">506</span> // We have numThreads executor + one timer thread used for timing out<a name="line.506"></a>
-<span class="sourceLineNo">507</span> // procedures and triggering periodic procedures.<a name="line.507"></a>
-<span class="sourceLineNo">508</span> this.corePoolSize = numThreads;<a name="line.508"></a>
-<span class="sourceLineNo">509</span> this.maxPoolSize = 10 * numThreads;<a name="line.509"></a>
-<span class="sourceLineNo">510</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.510"></a>
-<span class="sourceLineNo">511</span> corePoolSize, maxPoolSize);<a name="line.511"></a>
+<span class="sourceLineNo">392</span> // add the nonce to the map<a name="line.392"></a>
+<span class="sourceLineNo">393</span> if (nonceKey != null) {<a name="line.393"></a>
+<span class="sourceLineNo">394</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.394"></a>
+<span class="sourceLineNo">395</span> }<a name="line.395"></a>
+<span class="sourceLineNo">396</span> }<a name="line.396"></a>
+<span class="sourceLineNo">397</span><a name="line.397"></a>
+<span class="sourceLineNo">398</span> // 2. Initialize the stacks<a name="line.398"></a>
+<span class="sourceLineNo">399</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.399"></a>
+<span class="sourceLineNo">400</span> HashSet<Procedure> waitingSet = null;<a name="line.400"></a>
+<span class="sourceLineNo">401</span> procIter.reset();<a name="line.401"></a>
+<span class="sourceLineNo">402</span> while (procIter.hasNext()) {<a name="line.402"></a>
+<span class="sourceLineNo">403</span> if (procIter.isNextFinished()) {<a name="line.403"></a>
+<span class="sourceLineNo">404</span> procIter.skipNext();<a name="line.404"></a>
+<span class="sourceLineNo">405</span> continue;<a name="line.405"></a>
+<span class="sourceLineNo">406</span> }<a name="line.406"></a>
+<span class="sourceLineNo">407</span><a name="line.407"></a>
+<span class="sourceLineNo">408</span> Procedure proc = procIter.next();<a name="line.408"></a>
+<span class="sourceLineNo">409</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.409"></a>
+<span class="sourceLineNo">410</span><a name="line.410"></a>
+<span class="sourceLineNo">411</span> if (debugEnabled) {<a name="line.411"></a>
+<span class="sourceLineNo">412</span> LOG.debug(String.format("Loading %s", proc));<a name="line.412"></a>
+<span class="sourceLineNo">413</span> }<a name="line.413"></a>
+<span class="sourceLineNo">414</span><a name="line.414"></a>
+<span class="sourceLineNo">415</span> Long rootProcId = getRootProcedureId(proc);<a name="line.415"></a>
+<span class="sourceLineNo">416</span> if (rootProcId == null) {<a name="line.416"></a>
+<span class="sourceLineNo">417</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.417"></a>
+<span class="sourceLineNo">418</span> scheduler.addBack(proc);<a name="line.418"></a>
+<span class="sourceLineNo">419</span> continue;<a name="line.419"></a>
+<span class="sourceLineNo">420</span> }<a name="line.420"></a>
+<span class="sourceLineNo">421</span><a name="line.421"></a>
+<span class="sourceLineNo">422</span> if (proc.hasParent()) {<a name="line.422"></a>
+<span class="sourceLineNo">423</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.423"></a>
+<span class="sourceLineNo">424</span> // corrupted procedures are handled later at step 3<a name="line.424"></a>
+<span class="sourceLineNo">425</span> if (parent != null && !proc.isFinished()) {<a name="line.425"></a>
+<span class="sourceLineNo">426</span> parent.incChildrenLatch();<a name="line.426"></a>
+<span class="sourceLineNo">427</span> }<a name="line.427"></a>
+<span class="sourceLineNo">428</span> }<a name="line.428"></a>
+<span class="sourceLineNo">429</span><a name="line.429"></a>
+<span class="sourceLineNo">430</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.430"></a>
+<span class="sourceLineNo">431</span> procStack.loadStack(proc);<a name="line.431"></a>
+<span class="sourceLineNo">432</span><a name="line.432"></a>
+<span class="sourceLineNo">433</span> proc.setRootProcId(rootProcId);<a name="line.433"></a>
+<span class="sourceLineNo">434</span> switch (proc.getState()) {<a name="line.434"></a>
+<span class="sourceLineNo">435</span> case RUNNABLE:<a name="line.435"></a>
+<span class="sourceLineNo">436</span> runnableList.add(proc);<a name="line.436"></a>
+<span class="sourceLineNo">437</span> break;<a name="line.437"></a>
+<span class="sourceLineNo">438</span> case WAITING:<a name="line.438"></a>
+<span class="sourceLineNo">439</span> if (!proc.hasChildren()) {<a name="line.439"></a>
+<span class="sourceLineNo">440</span> runnableList.add(proc);<a name="line.440"></a>
+<span class="sourceLineNo">441</span> }<a name="line.441"></a>
+<span class="sourceLineNo">442</span> break;<a name="line.442"></a>
+<span class="sourceLineNo">443</span> case WAITING_TIMEOUT:<a name="line.443"></a>
+<span class="sourceLineNo">444</span> if (waitingSet == null) {<a name="line.444"></a>
+<span class="sourceLineNo">445</span> waitingSet = new HashSet<>();<a name="line.445"></a>
+<span class="sourceLineNo">446</span> }<a name="line.446"></a>
+<span class="sourceLineNo">447</span> waitingSet.add(proc);<a name="line.447"></a>
+<span class="sourceLineNo">448</span> break;<a name="line.448"></a>
+<span class="sourceLineNo">449</span> case FAILED:<a name="line.449"></a>
+<span class="sourceLineNo">450</span> // add the proc to the scheduler to perform the rollback<a name="line.450"></a>
+<span class="sourceLineNo">451</span> scheduler.addBack(proc);<a name="line.451"></a>
+<span class="sourceLineNo">452</span> break;<a name="line.452"></a>
+<span class="sourceLineNo">453</span> case ROLLEDBACK:<a name="line.453"></a>
+<span class="sourceLineNo">454</span> case INITIALIZING:<a name="line.454"></a>
+<span class="sourceLineNo">455</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.455"></a>
+<span class="sourceLineNo">456</span> LOG.error(msg);<a name="line.456"></a>
+<span class="sourceLineNo">457</span> throw new UnsupportedOperationException(msg);<a name="line.457"></a>
+<span class="sourceLineNo">458</span> default:<a name="line.458"></a>
+<span class="sourceLineNo">459</span> break;<a name="line.459"></a>
+<span class="sourceLineNo">460</span> }<a name="line.460"></a>
+<span class="sourceLineNo">461</span> }<a name="line.461"></a>
+<span class="sourceLineNo">462</span><a name="line.462"></a>
+<span class="sourceLineNo">463</span> // 3. Validate the stacks<a name="line.463"></a>
+<span class="sourceLineNo">464</span> int corruptedCount = 0;<a name="line.464"></a>
+<span class="sourceLineNo">465</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.465"></a>
+<span class="sourceLineNo">466</span> while (itStack.hasNext()) {<a name="line.466"></a>
+<span class="sourceLineNo">467</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.467"></a>
+<span class="sourceLineNo">468</span> RootProcedureState procStack = entry.getValue();<a name="line.468"></a>
+<span class="sourceLineNo">469</span> if (procStack.isValid()) continue;<a name="line.469"></a>
+<span class="sourceLineNo">470</span><a name="line.470"></a>
+<span class="sourceLineNo">471</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.471"></a>
+<span class="sourceLineNo">472</span> LOG.error("Corrupted " + proc);<a name="line.472"></a>
+<span class="sourceLineNo">473</span> procedures.remove(proc.getProcId());<a name="line.473"></a>
+<span class="sourceLineNo">474</span> runnableList.remove(proc);<a name="line.474"></a>
+<span class="sourceLineNo">475</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.475"></a>
+<span class="sourceLineNo">476</span> corruptedCount++;<a name="line.476"></a>
+<span class="sourceLineNo">477</span> }<a name="line.477"></a>
+<span class="sourceLineNo">478</span> itStack.remove();<a name="line.478"></a>
+<span class="sourceLineNo">479</span> }<a name="line.479"></a>
+<span class="sourceLineNo">480</span><a name="line.480"></a>
+<span class="sourceLineNo">481</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.481"></a>
+<span class="sourceLineNo">482</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.482"></a>
+<span class="sourceLineNo">483</span> }<a name="line.483"></a>
+<span class="sourceLineNo">484</span><a name="line.484"></a>
+<span class="sourceLineNo">485</span> // 4. Push the procedures to the timeout executor<a name="line.485"></a>
+<span class="sourceLineNo">486</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.486"></a>
+<span class="sourceLineNo">487</span> for (Procedure proc: waitingSet) {<a name="line.487"></a>
+<span class="sourceLineNo">488</span> proc.afterReplay(getEnvironment());<a name="line.488"></a>
+<span class="sourceLineNo">489</span> timeoutExecutor.add(proc);<a name="line.489"></a>
+<span class="sourceLineNo">490</span> }<a name="line.490"></a>
+<span class="sourceLineNo">491</span> }<a name="line.491"></a>
+<span class="sourceLineNo">492</span><a name="line.492"></a>
+<span class="sourceLineNo">493</span> // 5. Push the procedure to the scheduler<a name="line.493"></a>
+<span class="sourceLineNo">494</span> if (!runnableList.isEmpty()) {<a name="line.494"></a>
+<span class="sourceLineNo">495</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.495"></a>
+<span class="sourceLineNo">496</span> // some procedure may be started way before this stuff.<a name="line.496"></a>
+<span class="sourceLineNo">497</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.497"></a>
+<span class="sourceLineNo">498</span> Procedure proc = runnableList.get(i);<a name="line.498"></a>
+<span class="sourceLineNo">499</span> proc.afterReplay(getEnvironment());<a name="line.499"></a>
+<span class="sourceLineNo">500</span> if (!proc.hasParent()) {<a name="line.500"></a>
+<span class="sourceLineNo">501</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.501"></a>
+<span class="sourceLineNo">502</span> }<a name="line.502"></a>
+<span class="sourceLineNo">503</span> if (proc.wasExecuted()) {<a name="line.503"></a>
+<span class="sourceLineNo">504</span> scheduler.addFront(proc);<a name="line.504"></a>
+<span class="sourceLineNo">505</span> } else {<a name="line.505"></a>
+<span class="sourceLineNo">506</span> // if it was not in execution, it can wait.<a name="line.506"></a>
+<span class="sourceLineNo">507</span> scheduler.addBack(proc);<a name="line.507"></a>
+<span class="sourceLineNo">508</span> }<a name="line.508"></a>
+<span class="sourceLineNo">509</span> }<a name="line.509"></a>
+<span class="sourceLineNo">510</span> }<a name="line.510"></a>
+<span class="sourceLineNo">511</span> }<a name="line.511"></a>
<span class="sourceLineNo">512</span><a name="line.512"></a>
-<span class="sourceLineNo">513</span> // Create the Thread Group for the executors<a name="line.513"></a>
-<span class="sourceLineNo">514</span> threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.514"></a>
-<span class="sourceLineNo">515</span><a name="line.515"></a>
-<span class="sourceLineNo">516</span> // Create the timeout executor<a name="line.516"></a>
-<span class="sourceLineNo">517</span> timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.517"></a>
-<span class="sourceLineNo">518</span><a name="line.518"></a>
-<span class="sourceLineNo">519</span> // Create the workers<a name="line.519"></a>
-<span class="sourceLineNo">520</span> workerId.set(0);<a name="line.520"></a>
-<span class="sourceLineNo">521</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.521"></a>
-<span class="sourceLineNo">522</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.522"></a>
-<span class="sourceLineNo">523</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.523"></a>
-<span class="sourceLineNo">524</span> }<a name="line.524"></a>
-<span class="sourceLineNo">525</span><a name="line.525"></a>
-<span class="sourceLineNo">526</span> long st, et;<a name="line.526"></a>
-<span class="sourceLineNo">527</span><a name="line.527"></a>
-<span class="sourceLineNo">528</span> // Acquire the store lease.<a name="line.528"></a>
-<span class="sourceLineNo">529</span> st = EnvironmentEdgeManager.currentTime();<a name="line.529"></a>
-<span class="sourceLineNo">530</span> store.recoverLease();<a name="line.530"></a>
-<span class="sourceLineNo">531</span> et = EnvironmentEdgeManager.currentTime();<a name="line.531"></a>
-<span class="sourceLineNo">532</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.532"></a>
-<span class="sourceLineNo">533</span> StringUtils.humanTimeDiff(et - st));<a name="line.533"></a>
-<span class="sourceLineNo">534</span><a name="line.534"></a>
-<span class="sourceLineNo">535</span> // start the procedure scheduler<a name="line.535"></a>
-<span class="sourceLineNo">536</span> scheduler.start();<a name="line.536"></a>
-<span class="sourceLineNo">537</span><a name="line.537"></a>
-<span class="sourceLineNo">538</span> // TODO: Split in two steps.<a name="line.538"></a>
-<span class="sourceLineNo">539</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.539"></a>
-<span class="sourceLineNo">540</span> // The first one will make sure that we have the latest id,<a name="line.540"></a>
-<span class="sourceLineNo">541</span> // so we can start the threads and accept new procedures.<a name="line.541"></a>
-<span class="sourceLineNo">542</span> // The second step will do the actual load of old procedures.<a name="line.542"></a>
-<span class="sourceLineNo">543</span> st = EnvironmentEdgeManager.currentTime();<a name="line.543"></a>
-<span class="sourceLineNo">544</span> load(abortOnCorruption);<a name="line.544"></a>
-<span class="sourceLineNo">545</span> et = EnvironmentEdgeManager.currentTime();<a name="line.545"></a>
-<span class="sourceLineNo">546</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.546"></a>
-<span class="sourceLineNo">547</span> StringUtils.humanTimeDiff(et - st));<a name="line.547"></a>
-<span class="sourceLineNo">548</span><a name="line.548"></a>
-<span class="sourceLineNo">549</span> // Start the executors. Here we must have the lastProcId set.<a name="line.549"></a>
-<span class="sourceLineNo">550</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.550"></a>
-<span class="sourceLineNo">551</span> timeoutExecutor.start();<a name="line.551"></a>
-<span class="sourceLineNo">552</span> for (WorkerThread worker: workerThreads) {<a name="line.552"></a>
-<span class="sourceLineNo">553</span> worker.start();<a name="line.553"></a>
-<span class="sourceLineNo">554</span> }<a name="line.554"></a>
-<span class="sourceLineNo">555</span><a name="line.555"></a>
-<span class="sourceLineNo">556</span> // Internal chores<a name="line.556"></a>
-<span class="sourceLineNo">557</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.557"></a>
-<span class="sourceLineNo">558</span><a name="line.558"></a>
-<span class="sourceLineNo">559</span> // Add completed cleaner chore<a name="line.559"></a>
-<span class="sourceLineNo">560</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.560"></a>
-<span class="sourceLineNo">561</span> }<a name="line.561"></a>
-<span class="sourceLineNo">562</span><a name="line.562"></a>
-<span class="sourceLineNo">563</span> public void stop() {<a name="line.563"></a>
-<span class="sourceLineNo">564</span> if (!running.getAndSet(false)) {<a name="line.564"></a>
-<span class="sourceLineNo">565</span> return;<a name="line.565"></a>
-<span class="sourceLineNo">566</span> }<a name="line.566"></a>
-<span class="sourceLineNo">567</span><a name="line.567"></a>
-<span class="sourceLineNo">568</span> LOG.info("Stopping");<a name="line.568"></a>
-<span class="sourceLineNo">569</span> scheduler.stop();<a name="line.569"></a>
-<span class="sourceLineNo">570</span> timeoutExecutor.sendStopSignal();<a name="line.570"></a>
-<span class="sourceLineNo">571</span> }<a name="line.571"></a>
-<span class="sourceLineNo">572</span><a name="line.572"></a>
-<span class="sourceLineNo">573</span> @VisibleForTesting<a name="line.573"></a>
-<span class="sourceLineNo">574</span> public void join() {<a name="line.574"></a>
-<span class="sourceLineNo">575</span> assert !isRunning() : "expected not running";<a name="line.575"></a>
-<span class="sourceLineNo">576</span><a name="line.576"></a>
-<span class="sourceLineNo">577</span> // stop the timeout executor<a name="line.577"></a>
-<span class="sourceLineNo">578</span> timeoutExecutor.awaitTermination();<a name="line.578"></a>
-<span class="sourceLineNo">579</span> timeoutExecutor = null;<a name="line.579"></a>
-<span class="sourceLineNo">580</span><a name="line.580"></a>
-<span class="sourceLineNo">581</span> // stop the worker threads<a name="line.581"></a>
-<span class="sourceLineNo">582</span> for (WorkerThread worker: workerThreads) {<a name="line.582"></a>
-<span class="sourceLineNo">583</span> worker.awaitTermination();<a name="line.583"></a>
-<span class="sourceLineNo">584</span> }<a name="line.584"></a>
-<span class="sourceLineNo">585</span> workerThreads = null;<a name="line.585"></a>
-<span class="sourceLineNo">586</span><a name="line.586"></a>
-<span class="sourceLineNo">587</span> // Destroy the Thread Group for the executors<a name="line.587"></a>
-<span class="sourceLineNo">588</span> try {<a name="line.588"></a>
-<span class="sourceLineNo">589</span> threadGroup.destroy();<a name="line.589"></a>
-<span class="sourceLineNo">590</span> } catch (IllegalThreadStateException e) {<a name="line.590"></a>
-<span class="sourceLineNo">591</span> LOG.error("ThreadGroup " + threadGroup + " contains running threads; " + e.getMessage());<a name="line.591"></a>
-<span class="sourceLineNo">592</span> threadGroup.list();<a name="line.592"></a>
-<span class="sourceLineNo">593</span> } finally {<a name="line.593"></a>
-<span class="sourceLineNo">594</span> threadGroup = null;<a name="line.594"></a>
-<span class="sourceLineNo">595</span> }<a name="line.595"></a>
+<span class="sourceLineNo">513</span> /**<a name="line.513"></a>
+<span class="sourceLineNo">514</span> * Start the procedure executor.<a name="line.514"></a>
+<span class="sourceLineNo">515</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.515"></a>
+<span class="sourceLineNo">516</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.516"></a>
+<span class="sourceLineNo">517</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.517"></a>
+<span class="sourceLineNo">518</span> *<a name="line.518"></a>
+<span class="sourceLineNo">519</span> * @param numThreads number of threads available for procedure execution.<a name="line.519"></a>
+<span class="sourceLineNo">520</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.520"></a>
+<span class="sourceLineNo">521</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.521"></a>
+<span class="sourceLineNo">522</span> */<a name="line.522"></a>
+<span class="sourceLineNo">523</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.523"></a>
+<span class="sourceLineNo">524</span> if (!running.compareAndSet(false, true)) {<a name="line.524"></a>
+<span class="sourceLineNo">525</span> LOG.warn("Already running");<a name="line.525"></a>
+<span class="sourceLineNo">526</span> return;<a name="line.526"></a>
+<span class="sourceLineNo">527</span> }<a name="line.527"></a>
+<span class="sourceLineNo">528</span><a name="line.528"></a>
+<span class="sourceLineNo">529</span> // We have numThreads executor + one timer thread used for timing out<a name="line.529"></a>
+<span class="sourceLineNo">530</span> // procedures and triggering periodic procedures.<a name="line.530"></a>
+<span class="sourceLineNo">531</span> this.corePoolSize = numThreads;<a name="line.531"></a>
+<span class="sourceLineNo">532</span> this.maxPoolSize = 10 * numThreads;<a name="line.532"></a>
+<span class="sourceLineNo">533</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.533"></a>
+<span class="sourceLineNo">534</span> corePoolSize, maxPoolSize);<a name="line.534"></a>
+<span class="sourceLineNo">535</span><a name="line.535"></a>
+<span class="sourceLineNo">536</span> this.threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.536"></a>
+<span class="sourceLineNo">537</span> this.timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.537"></a>
+<span class="sourceLineNo">538</span><a name="line.538"></a>
+<span class="sourceLineNo">539</span> // Create the workers<a name="line.539"></a>
+<span class="sourceLineNo">540</span> workerId.set(0);<a name="line.540"></a>
+<span class="sourceLineNo">541</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.541"></a>
+<span class="sourceLineNo">542</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.542"></a>
+<span class="sourceLineNo">543</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.543"></a>
+<span class="sourceLineNo">544</span> }<a name="line.544"></a>
+<span class="sourceLineNo">545</span><a name="line.545"></a>
+<span class="sourceLineNo">546</span> long st, et;<a name="line.546"></a>
+<span class="sourceLineNo">547</span><a name="line.547"></a>
+<span class="sourceLineNo">548</span> // Acquire the store lease.<a name="line.548"></a>
+<span class="sourceLineNo">549</span> st = EnvironmentEdgeManager.currentTime();<a name="line.549"></a>
+<span class="sourceLineNo">550</span> store.recoverLease();<a name="line.550"></a>
+<span class="sourceLineNo">551</span> et = EnvironmentEdgeManager.currentTime();<a name="line.551"></a>
+<span class="sourceLineNo">552</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.552"></a>
+<span class="sourceLineNo">553</span> StringUtils.humanTimeDiff(et - st));<a name="line.553"></a>
+<span class="sourceLineNo">554</span><a name="line.554"></a>
+<span class="sourceLineNo">555</span> // start the procedure scheduler<a name="line.555"></a>
+<span class="sourceLineNo">556</span> scheduler.start();<a name="line.556"></a>
+<span class="sourceLineNo">557</span><a name="line.557"></a>
+<span class="sourceLineNo">558</span> // TODO: Split in two steps.<a name="line.558"></a>
+<span class="sourceLineNo">559</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.559"></a>
+<span class="sourceLineNo">560</span> // The first one will make sure that we have the latest id,<a name="line.560"></a>
+<span class="sourceLineNo">561</span> // so we can start the threads and accept new procedures.<a name="line.561"></a>
+<span class="sourceLineNo">562</span> // The second step will do the actual load of old procedures.<a name="line.562"></a>
+<span class="sourceLineNo">563</span> st = EnvironmentEdgeManager.currentTime();<a name="line.563"></a>
+<span class="sourceLineNo">564</span> load(abortOnCorruption);<a name="line.564"></a>
+<span class="sourceLineNo">565</span> et = EnvironmentEdgeManager.currentTime();<a name="line.565"></a>
+<span class="sourceLineNo">566</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.566"></a>
+<span class="sourceLineNo">567</span> StringUtils.humanTimeDiff(et - st));<a name="line.567"></a>
+<span class="sourceLineNo">568</span><a name="line.568"></a>
+<span class="sourceLineNo">569</span> // Start the executors. Here we must have the lastProcId set.<a name="line.569"></a>
+<span class="sourceLineNo">570</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.570"></a>
+<span class="sourceLineNo">571</span> timeoutExecutor.start();<a name="line.571"></a>
+<span class="sourceLineNo">572</span> for (WorkerThread worker: workerThreads) {<a name="line.572"></a>
+<span class="sourceLineNo">573</span> worker.start();<a name="line.573"></a>
+<span class="sourceLineNo">574</span> }<a name="line.574"></a>
+<span class="sourceLineNo">575</span><a name="line.575"></a>
+<span class="sourceLineNo">576</span> // Internal chores<a name="line.576"></a>
+<span class="sourceLineNo">577</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.577"></a>
+<span class="sourceLineNo">578</span><a name="line.578"></a>
+<span class="sourceLineNo">579</span> // Add completed cleaner chore<a name="line.579"></a>
+<span class="sourceLineNo">580</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.580"></a>
+<span class="sourceLineNo">581</span> }<a name="line.581"></a>
+<span class="sourceLineNo">582</span><a name="line.582"></a>
+<span class="sourceLineNo">583</span> public void stop() {<a name="line.583"></a>
+<span class="sourceLineNo">584</span> if (!running.getAndSet(false)) {<a name="line.584"></a>
+<span class="sourceLineNo">585</span> return;<a name="line.585"></a>
+<span class="sourceLineNo">586</span> }<a name="line.586"></a>
+<span class="sourceLineNo">587</span><a name="line.587"></a>
+<span class="sourceLineNo">588</span> LOG.info("Stopping");<a name="line.588"></a>
+<span class="sourceLineNo">589</span> scheduler.stop();<a name="line.589"></a>
+<span class="sourceLineNo">590</span> timeoutExecutor.sendStopSignal();<a name="line.590"></a>
+<span class="sourceLineNo">591</span> }<a name="line.591"></a>
+<span class="sourceLineNo">592</span><a name="line.592"></a>
+<span class="sourceLineNo">593</span> @VisibleForTesting<a name="line.593"></a>
+<span class="sourceLineNo">594</span> public void join() {<a name="line.594"></a>
+<span class="sourceLineNo">595</span> assert !isRunning() : "expected not running";<a name="line.595"></a>
<span class="sourceLineNo">596</span><a name="line.596"></a>
-<span class="sourceLineNo">597</span> // reset the in-memory state for testing<a name="line.597"></a>
-<span class="sourceLineNo">598</span> completed.clear();<a name="line.598"></a>
-<span class="sourceLineNo">599</span> rollbackStack.clear();<a name="line.599"></a>
-<span class="sourceLineNo">600</span> procedures.clear();<a name="line.600"></a>
-<span class="sourceLineNo">601</span> nonceKeysToProcIdsMap.clear();<a name="line.601"></a>
-<span class="sourceLineNo">602</span> scheduler.clear();<a name="line.602"></a>
-<span class="sourceLineNo">603</span> lastProcId.set(-1);<a name="line.603"></a>
-<span class="sourceLineNo">604</span> }<a name="line.604"></a>
-<span class="sourceLineNo">605</span><a name="line.605"></a>
-<span class="sourceLineNo">606</span> public void refreshConfiguration(final Configuration conf) {<a name="line.606"></a>
-<span class="sourceLineNo">607</span> this.conf = conf;<a name="line.607"></a>
-<span class="sourceLineNo">608</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.608"></a>
-<span class="sourceLineNo">609</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.609"></a>
-<span class="sourceLineNo">610</span> }<a name="line.610"></a>
-<span class="sourceLineNo">611</span><a name="line.611"></a>
-<span class="sourceLineNo">612</span> // ==========================================================================<a name="line.612"></a>
-<span class="sourceLineNo">613</span> // Accessors<a name="line.613"></a>
-<span class="sourceLineNo">614</span> // ==========================================================================<a name="line.614"></a>
-<span class="sourceLineNo">615</span> public boolean isRunning() {<a name="line.615"></a>
-<span class="sourceLineNo">616</span> return running.get();<a name="line.616"></a>
-<span class="sourceLineNo">617</span> }<a name="line.617"></a>
-<span class="sourceLineNo">618</span><a name="line.618"></a>
-<span class="sourceLineNo">619</span> /**<a name="line.619"></a>
-<span class="sourceLineNo">620</span> * @return the current number of worker threads.<a name="line.620"></a>
-<span class="sourceLineNo">621</span> */<a name="line.621"></a>
-<span class="sourceLineNo">622</span> public int getWorkerThreadCount() {<a name="line.622"></a>
-<span class="sourceLineNo">623</span> return workerThreads.size();<a name="line.623"></a>
-<span class="sourceLineNo">624</span> }<a name="line.624"></a>
-<span class="sourceLineNo">625</span><a name="line.625"></a>
-<span class="sourceLineNo">626</span> /**<a name="line.626"></a>
-<span class="sourceLineNo">627</span> * @return the core pool size settings.<a name="line.627"></a>
-<span class="sourceLineNo">628</span> */<a name="line.628"></a>
-<span class="sourceLineNo">629</span> public int getCorePoolSize() {<a name="line.629"></a>
-<span class="sourceLineNo">630</span> return corePoolSize;<a name="line.630"></a>
-<span class="sourceLineNo">631</span> }<a name="line.631"></a>
-<span class="sourceLineNo">632</span><a name="line.632"></a>
-<span class="sourceLineNo">633</span> public int getActiveExecutorCount() {<a name="line.633"></a>
-<span class="sourceLineNo">634</span> return activeExecutorCount.get();<a name="line.634"></a>
-<span class="sourceLineNo">635</span> }<a name="line.635"></a>
-<span class="sourceLineNo">636</span><a name="line.636"></a>
-<span class="sourceLineNo">637</span> public TEnvironment getEnvironment() {<a name="line.637"></a>
-<span class="sourceLineNo">638</span> return this.environment;<a name="line.638"></a>
-<span class="sourceLineNo">639</span> }<a name="line.639"></a>
-<span class="sourceLineNo">640</span><a name="line.640"></a>
-<span class="sourceLineNo">641</span> public ProcedureStore getStore() {<a name="line.641"></a>
-<span class="sourceLineNo">642</span> return this.store;<a name="line.642"></a>
+<span class="sourceLineNo">597</span> // stop the timeout executor<a name="line.597"></a>
+<span class="sourceLineNo">598</span> timeoutExecutor.awaitTermination();<a name="line.598"></a>
+<span class="sourceLineNo">599</span><a name="line.599"></a>
+<span class="sourceLineNo">600</span> // stop the worker threads<a name="line.600"></a>
+<span class="sourceLineNo">601</span> for (WorkerThread worker: workerThreads) {<a name="line.601"></a>
+<span class="sourceLineNo">602</span> worker.awaitTermination();<a name="line.602"></a>
+<span class="sourceLineNo">603</span> }<a name="line.603"></a>
+<span class="sourceLineNo">604</span><a name="line.604"></a>
+<span class="sourceLineNo">605</span> // Destroy the Thread Group for the executors<a name="line.605"></a>
+<span class="sourceLineNo">606</span> // TODO: Fix. #join is not place to destroy resources.<a name="line.606"></a>
+<span class="sourceLineNo">607</span> try {<a name="line.607"></a>
+<span class="sourceLineNo">608</span> threadGroup.destroy();<a name="line.608"></a>
+<span class="sourceLineNo">609</span> } catch (IllegalThreadStateException e) {<a name="line.609"></a>
+<span class="sourceLineNo">610</span> LOG.error("ThreadGroup {} contains running threads; {}: See STDOUT",<a name="line.610"></a>
+<span class="sourceLineNo">611</span> this.threadGroup, e.getMessage());<a name="line.611"></a>
+<span class="sourceLineNo">612</span> // This dumps list of threads on STDOUT.<a name="line.612"></a>
+<span class="sourceLineNo">613</span> this.threadGroup.list();<a name="line.613"></a>
+<span class="sourceLineNo">614</span> }<a name="line.614"></a>
+<span class="sourceLineNo">615</span><a name="line.615"></a>
+<span class="sourceLineNo">616</span> // reset the in-memory state for testing<a name="line.616"></a>
+<span class="sourceLineNo">617</span> completed.clear();<a name="line.617"></a>
+<span class="sourceLineNo">618</span> rollbackStack.clear();<a name="line.618"></a>
+<span class="sourceLineNo">619</span> procedures.clear();<a name="line.619"></a>
+<span class="sourceLineNo">620</span> nonceKeysToProcIdsMap.clear();<a name="line.620"></a>
+<span class="sourceLineNo">621</span> scheduler.clear();<a name="line.621"></a>
+<span class="sourceLineNo">622</span> lastProcId.set(-1);<a name="line.622"></a>
+<span class="sourceLineNo">623</span> }<a name="line.623"></a>
+<span class="sourceLineNo">624</span><a name="line.624"></a>
+<span class="sourceLineNo">625</span> public void refreshConfiguration(final Configuration conf) {<a name="line.625"></a>
+<span class="sourceLineNo">626</span> this.conf = conf;<a name="line.626"></a>
+<span class="sourceLineNo">627</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.627"></a>
+<span class="sourceLineNo">628</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.628"></a>
+<span class="sourceLineNo">629</span> }<a name="line.629"></a>
+<span class="sourceLineNo">630</span><a name="line.630"></a>
+<span class="sourceLineNo">631</span> // ==========================================================================<a name="line.631"></a>
+<span class="sourceLineNo">632</span> // Accessors<a name="line.632"></a>
+<span class="sourceLineNo">633</span> // ==========================================================================<a name="line.633"></a>
+<span class="sourceLineNo">634</span> public boolean isRunning() {<a name="line.634"></a>
+<span class="sourceLineNo">635</span> return running.get();<a name="line.635"></a>
+<span class="sourceLineNo">636</span> }<a name="line.636"></a>
+<span class="sourceLineNo">637</span><a name="line.637"></a>
+<span class="sourceLineNo">638</span> /**<a name="line.638"></a>
+<span class="sourceLineNo">639</span> * @return the current number of worker threads.<a name="line.639"></a>
+<span class="sourceLineNo">640</span> */<a name="line.640"></a>
+<span class="sourceLineNo">641</span> public int getWorkerThreadCount() {<a name="line.641"></a>
+<span class="sourceLineNo">642</span> return workerThreads.size();<a name="line.642"></a>
<span class="sourceLineNo">643</span> }<a name="line.643"></a>
<span class="sourceLineNo">644</span><a name="line.644"></a>
-<span class="sourceLineNo">645</span> ProcedureScheduler getScheduler() {<a name="line.645"></a>
-<span class="sourceLineNo">646</span> return scheduler;<a name="line.646"></a>
-<span class="sourceLineNo">647</span> }<a name="line.647"></a>
-<span class="sourceLineNo">648</span><a name="line.648"></a>
-<span class="sourceLineNo">649</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.649"></a>
-<span class="sourceLineNo">650</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.650"></a>
-<span class="sourceLineNo">651</span> this.scheduler.signalAll();<a name="line.651"></a>
-<span class="sourceLineNo">652</span> }<a name="line.652"></a>
-<span class="sourceLineNo">653</span><a name="line.653"></a>
-<span class="sourceLineNo">654</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.654"></a>
-<span class="sourceLineNo">655</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.655"></a>
-<span class="sourceLineNo">656</span> }<a name="line.656"></a>
-<span class="sourceLineNo">657</span><a name="line.657"></a>
-<span class="sourceLineNo">658</span> // ==========================================================================<a name="line.658"></a>
-<span class="sourceLineNo">659</span> // Submit/Remove Chores<a name="line.659"></a>
-<span class="sourceLineNo">660</span> // ==========================================================================<a name="line.660"></a>
-<span class="sourceLineNo">661</span><a name="line.661"></a>
-<span class="sourceLineNo">662</span> /**<a name="line.662"></a>
-<span class="sourceLineNo">663</span> * Add a chore procedure to the executor<a name="line.663"></a>
-<span class="sourceLineNo">664</span> * @param chore the chore to add<a name="line.664"></a>
-<span class="sourceLineNo">665</span> */<a name="line.665"></a>
-<span class="sourceLineNo">666</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.666"></a>
-<span class="sourceLineNo">667</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.667"></a>
-<span class="sourceLineNo">668</span> timeoutExecutor.add(chore);<a name="line.668"></a>
-<span class="sourceLineNo">669</span> }<a name="line.669"></a>
-<span class="sourceLineNo">670</span><a name="line.670"></a>
-<span class="sourceLineNo">671</span> /**<a name="line.671"></a>
-<span class="sourceLineNo">672</span> * Remove a chore procedure from the executor<a name="line.672"></a>
-<span class="sourceLineNo">673</span> * @param chore the chore to remove<a name="line.673"></a>
-<span class="sourceLineNo">674</span> * @return whether the chore is removed, or it will be removed later<a name="line.674"></a>
-<span class="sourceLineNo">675</span> */<a name="line.675"></a>
-<span class="sourceLineNo">676</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.676"></a>
-<span class="sourceLineNo">677</span> chore.setState(ProcedureState.SUCCESS);<a name="line.677"></a>
-<span class="sourceLineNo">678</span> return timeoutExecutor.remove(chore);<a name="line.678"></a>
-<span class="sourceLineNo">679</span> }<a name="line.679"></a>
+<span class="sourceLineNo">645</span> /**<a name="line.645"></a>
+<span class="sourceLineNo">646</span> * @return the core pool size settings.<a name="line.646"></a>
+<span class="sourceLineNo">647</span> */<a name="line.647"></a>
+<span class="sourceLineNo">648</span> public int getCorePoolSize() {<a name="line.648"></a>
+<span class="sourceLineNo">649</span> return corePoolSize;<a name="line.649"></a>
+<span class="sourceLineNo">650</span> }<a name="line.650"></a>
+<span class="sourceLineNo">651</span><a name="line.651"></a>
+<span class="sourceLineNo">652</span> public int getActiveExecutorCount() {<a name="line.652"></a>
+<span class="sourceLineNo">653</span> return activeExecutorCount.get();<a name="line.653"></a>
+<span class="sourceLineNo">654</span> }<a name="line.654"></a>
+<span class="sourceLineNo">655</span><a name="line.655"></a>
+<span class="sourceLineNo">656</span> public TEnvironment getEnvironment() {<a name="line.656"></a>
+<span class="sourceLineNo">657</span> return this.environment;<a name="line.657"></a>
+<span class="sourceLineNo">658</span> }<a name="line.658"></a>
+<span class="sourceLineNo">659</span><a name="line.659"></a>
+<span class="sourceLineNo">660</span> public ProcedureStore getStore() {<a name="line.660"></a>
+<span class="sourceLineNo">661</span> return this.store;<a name="line.661"></a>
+<span class="sourceLineNo">662</span> }<a name="line.662"></a>
+<span class="sourceLineNo">663</span><a name="line.663"></a>
+<span class="sourceLineNo">664</span> ProcedureScheduler getScheduler() {<a name="line.664"></a>
+<span class="sourceLineNo">665</span> return scheduler;<a name="line.665"></a>
+<span class="sourceLineNo">666</span> }<a name="line.666"></a>
+<span class="sourceLineNo">667</span><a name="line.667"></a>
+<span class="sourceLineNo">668</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.668"></a>
+<span class="sourceLineNo">669</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.669"></a>
+<span class="sourceLineNo">670</span> this.scheduler.signalAll();<a name="line.670"></a>
+<span class="sourceLineNo">671</span> }<a name="line.671"></a>
+<span class="sourceLineNo">672</span><a name="line.672"></a>
+<span class="sourceLineNo">673</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.673"></a>
+<span class="sourceLineNo">674</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.674"></a>
+<span class="sourceLineNo">675</span> }<a name="line.675"></a>
+<span class="sourceLineNo">676</span><a name="line.676"></a>
+<span class="sourceLineNo">677</span> // ==========================================================================<a name="line.677"></a>
+<span class="sourceLineNo">678</span> // Submit/Remove Chores<a name="line.678"></a>
+<span class="sourceLineNo">679</span> // ==========================================================================<a name="line.679"></a>
<span class="sourceLineNo">680</span><a name="line.680"></a>
-<span class="sourceLineNo">681</span> // ==========================================================================<a name="line.681"></a>
-<span class="sourceLineNo">682</span> // Nonce Procedure helpers<a name="line.682"></a>
-<span class="sourceLineNo">683</span> // ==========================================================================<a name="line.683"></a>
-<span class="sourceLineNo">684</span> /**<a name="line.684"></a>
-<span class="sourceLineNo">685</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.685"></a>
-<span class="sourceLineNo">686</span> * @param nonceGroup<a name="line.686"></a>
-<span class="sourceLineNo">687</span> * @param nonce<a name="line.687"></a>
-<span class="sourceLineNo">688</span> * @return the generated NonceKey<a name="line.688"></a>
-<span class="sourceLineNo">689</span> */<a name="line.689"></a>
-<span class="sourceLineNo">690</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.690"></a>
-<span class="sourceLineNo">691</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.691"></a>
-<span class="sourceLineNo">692</span> }<a name="line.692"></a>
-<span class="sourceLineNo">693</span><a name="line.693"></a>
-<span class="sourceLineNo">694</span> /**<a name="line.694"></a>
-<span class="sourceLineNo">695</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.695"></a>
-<span class="sourceLineNo">696</span> * A procId will be reserved and on submitProcedure(),<a name="line.696"></a>
-<span class="sourceLineNo">697</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.697"></a>
-<span class="sourceLineNo">698</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.698"></a>
-<span class="sourceLineNo">699</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.699"></a>
-<span class="sourceLineNo">700</span> * and submit the procedure.<a name="line.700"></a>
-<span class="sourceLineNo">701</span> *<a name="line.701"></a>
-<span class="sourceLineNo">702</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.702"></a>
-<span class="sourceLineNo">703</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.703"></a>
-<span class="sourceLineNo">704</span> */<a name="line.704"></a>
-<span class="sourceLineNo">705</span> public long registerNonce(final NonceKey nonceKey) {<a name="line.705"></a>
-<span class="sourceLineNo">706</span> if (nonceKey == null) return -1;<a name="line.706"></a>
-<span class="sourceLineNo">707</span><a name="line.707"></a>
-<span class="sourceLineNo">708</span> // check if we have already a Reserved ID for the nonce<a name="line.708"></a>
-<span class="sourceLineNo">709</span> Long oldProcId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.709"></a>
-<span class="sourceLineNo">710</span> if (oldProcId == null) {<a name="line.710"></a>
-<span class="sourceLineNo">711</span> // reserve a new Procedure ID, this will be associated with the nonce<a name="line.711"></a>
-<span class="sourceLineNo">712</span> // and the procedure submitted with the specified nonce will use this ID.<a name="line.712"></a>
-<span class="sourceLineNo">713</span> final long newProcId = nextProcId();<a name="line.713"></a>
-<span class="sourceLineNo">714</span> oldProcId = nonceKeysToProcIdsMap.putIfAbsent(nonceKey, newProcId);<a name="line.714"></a>
-<span class="sourceLineNo">715</span> if (oldProcId == null) return -1;<a name="line.715"></a>
-<span class="sourceLineNo">716</span> }<a name="line.716"></a>
-<span class="sourceLineNo">717</span><a name="line.717"></a>
-<span class="sourceLineNo">718</span> // we found a registered nonce, but the procedure may not have been submitted yet.<a name="line.718"></a>
-<span class="sourceLineNo">719</span> // since the client expect the procedure to be submitted, spin here until it is.<a name="line.719"></a>
-<span class="sourceLineNo">720</span> final boolean traceEnabled = LOG.isTraceEnabled();<a name="line.720"></a>
-<span class="sourceLineNo">721</span> while (isRunning() &&<a name="line.721"></a>
-<span class="sourceLineNo">722</span> !(procedures.containsKey(oldProcId) || completed.containsKey(oldProcId)) &&<a name="line.722"></a>
-<span class="sourceLineNo">723</span> nonceKeysToProcIdsMap.containsKey(nonceKey)) {<a name="line.723"></a>
-<span class="sourceLineNo">724</span> if (traceEnabled) {<a name="line.724"></a>
-<span class="sourceLineNo">725</span> LOG.trace("Waiting for pid=" + oldProcId.longValue() + " to be submitted");<a name="line.725"></a>
-<span class="sourceLineNo">726</span> }<a name="line.726"></a>
-<span class="sourceLineNo">727</span> Threads.sleep(100);<a name="line.727"></a>
-<span class="sourceLineNo">728</span> }<a name="line.728"></a>
-<span class="sourceLineNo">729</span> return oldProcId.longValue();<a name="line.729"></a>
-<span class="sourceLineNo">730</span> }<a name="line.730"></a>
-<span class="sourceLineNo">731</span><a name="line.731"></a>
-<span class="sourceLineNo">732</span> /**<a name="line.732"></a>
-<span class="sourceLineNo">733</span> * Remove the NonceKey if the procedure was not submitted to the executor.<a name="line.733"></a>
-<span class="sourceLineNo">734</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.734"></a>
-<span class="sourceLineNo">735</span> */<a name="line.735"></a>
-<span class="sourceLineNo">736</span> public void unregisterNonceIfProcedureWasNotSubmitted(final NonceKey nonceKey) {<a name="line.736"></a>
-<span class="sourceLineNo">737</span> if (nonceKey == null) return;<a name="line.737"></a>
-<span class="sourceLineNo">738</span><a name="line.738"></a>
-<span class="sourceLineNo">739</span> final Long procId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.739"></a>
-<span class="sourceLineNo">740</span> if (procId == null) return;<a name="line.740"></a>
-<span class="sourceLineNo">741</span><a name="line.741"></a>
-<span class="sourceLineNo">742</span> // if the procedure was not submitted, remove the nonce<a name="line.742"></a>
-<span class="sourceLineNo">743</span> if (!(procedures.containsKey(procId) || completed.containsKey(procId))) {<a name="line.743"></a>
-<span class="sourceLineNo">744</span> nonceKeysToProcIdsMap.remove(nonceKey);<a name="line.744"></a>
-<span class="sourceLineNo">745</span> }<a name="line.745"></a>
-<span class="sourceLineNo">746</span> }<a name="line.746"></a>
-<span class="sourceLineNo">747</span><a name="line.747"></a>
-<span class="sourceLineNo">748</span> public static class FailedProcedure<TEnvironment> extends Procedure<TEnvironment> {<a name="line.748"></a>
-<span class="sourceLineNo">749</span> private String procName;<a name="line.749"></a>
+<span class="sourceLineNo">681</span> /**<a name="line.681"></a>
+<span class="sourceLineNo">682</span> * Add a chore procedure to the executor<a name="line.682"></a>
+<span class="sourceLineNo">683</span> * @param chore the chore to add<a name="line.683"></a>
+<span class="sourceLineNo">684</span> */<a name="line.684"></a>
+<span class="sourceLineNo">685</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.685"></a>
+<span class="sourceLineNo">686</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.686"></a>
+<span class="sourceLineNo">687</span> timeoutExecutor.add(chore);<a name="line.687"></a>
+<span class="sourceLineNo">688</span> }<a name="line.688"></a>
+<span class="sourceLineNo">689</span><a name="line.689"></a>
+<span class="sourceLineNo">690</span> /**<a name="line.690"></a>
+<span class="sourceLineNo">691</span> * Remove a chore procedure from the executor<a name="line.691"></a>
+<span class="sourceLineNo">692</span> * @param chore the chore to remove<a name="line.692"></a>
+<span class="sourceLineNo">693</span> * @return whether the chore is removed, or it will be removed later<a name="line.693"></a>
+<span class="sourceLineNo">694</span> */<a name="line.694"></a>
+<span class="sourceLineNo">695</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.695"></a>
+<span class="sourceLineNo">696</span> chore.setState(ProcedureState.SUCCESS);<a name="line.696"></a>
+<span class="sourceLineNo">697</span> return timeoutExecutor.remove(chore);<a name="line.697"></a>
+<span class="sourceLineNo">698</span> }<a name="line.698"></a>
+<span class="sourceLineNo">699</span><a name="line.699"></a>
+<span class="sourceLineNo">700</span> // ==========================================================================<a name="line.700"></a>
+<span class="sourceLineNo">701</span> // Nonce Procedure helpers<a name="line.701"></a>
+<span class="sourceLineNo">702</span> // ==========================================================================<a name="line.702"></a>
+<span class="sourceLineNo">703</span> /**<a name="line.703"></a>
+<span class="sourceLineNo">704</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.704"></a>
+<span class="sourceLineNo">705</span> * @param nonceGroup<a name="line.705"></a>
+<span class="sourceLineNo">706</span> * @param nonce<a name="line.706"></a>
+<span class="sourceLineNo">707</span> * @return the generated NonceKey<a name="line.707"></a>
+<span class="sourceLineNo">708</span> */<a name="line.708"></a>
+<span class="sourceLineNo">709</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.709"></a>
+<span class="sourceLineNo">710</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.710"></a>
+<span class="sourceLineNo">711</span> }<a name="line.711"></a>
+<span class="sourceLineNo">712</span><a name="line.712"></a>
+<span class="sourceLineNo">713</span> /**<a name="line.713"></a>
+<span class="sourceLineNo">714</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.714"></a>
+<span class="sourceLineNo">715</span> * A procId will be reserved and on submitProcedure(),<a name="line.715"></a>
+<span class="sourceLineNo">716</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.716"></a>
+<span class="sourceLineNo">717</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.717"></a>
+<span class="sourceLineNo">718</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.718"></a>
+<span class="sourceLineNo">719</span> * and submit the procedure.<a name="line.719"></a>
+<span class="sourceLineNo">720</span> *<a name="line.720"></a>
+<span class="sourceLineNo">721</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.721"></a>
+<span class="sourceLineNo">722</span> * @return the procId associated with
<TRUNCATED>
[06/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.ProcedureExecutorListener.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.ProcedureExecutorListener.html b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.ProcedureExecutorListener.html
index 97aa79c..cc787e3 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.ProcedureExecutorListener.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.ProcedureExecutorListener.html
@@ -272,1623 +272,1642 @@
<span class="sourceLineNo">264</span> private final CopyOnWriteArrayList<ProcedureExecutorListener> listeners = new CopyOnWriteArrayList<>();<a name="line.264"></a>
<span class="sourceLineNo">265</span><a name="line.265"></a>
<span class="sourceLineNo">266</span> private Configuration conf;<a name="line.266"></a>
-<span class="sourceLineNo">267</span> private ThreadGroup threadGroup;<a name="line.267"></a>
-<span class="sourceLineNo">268</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.268"></a>
-<span class="sourceLineNo">269</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.269"></a>
-<span class="sourceLineNo">270</span> private int corePoolSize;<a name="line.270"></a>
-<span class="sourceLineNo">271</span> private int maxPoolSize;<a name="line.271"></a>
-<span class="sourceLineNo">272</span><a name="line.272"></a>
-<span class="sourceLineNo">273</span> private volatile long keepAliveTime;<a name="line.273"></a>
-<span class="sourceLineNo">274</span><a name="line.274"></a>
-<span class="sourceLineNo">275</span> /**<a name="line.275"></a>
-<span class="sourceLineNo">276</span> * Scheduler/Queue that contains runnable procedures.<a name="line.276"></a>
-<span class="sourceLineNo">277</span> */<a name="line.277"></a>
-<span class="sourceLineNo">278</span> private final ProcedureScheduler scheduler;<a name="line.278"></a>
-<span class="sourceLineNo">279</span><a name="line.279"></a>
-<span class="sourceLineNo">280</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.280"></a>
-<span class="sourceLineNo">281</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.281"></a>
-<span class="sourceLineNo">282</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.282"></a>
-<span class="sourceLineNo">283</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.283"></a>
-<span class="sourceLineNo">284</span> private final TEnvironment environment;<a name="line.284"></a>
-<span class="sourceLineNo">285</span> private final ProcedureStore store;<a name="line.285"></a>
-<span class="sourceLineNo">286</span><a name="line.286"></a>
-<span class="sourceLineNo">287</span> private final boolean checkOwnerSet;<a name="line.287"></a>
-<span class="sourceLineNo">288</span><a name="line.288"></a>
-<span class="sourceLineNo">289</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.289"></a>
-<span class="sourceLineNo">290</span> final ProcedureStore store) {<a name="line.290"></a>
-<span class="sourceLineNo">291</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.291"></a>
-<span class="sourceLineNo">292</span> }<a name="line.292"></a>
-<span class="sourceLineNo">293</span><a name="line.293"></a>
-<span class="sourceLineNo">294</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.294"></a>
-<span class="sourceLineNo">295</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.295"></a>
-<span class="sourceLineNo">296</span> this.environment = environment;<a name="line.296"></a>
-<span class="sourceLineNo">297</span> this.scheduler = scheduler;<a name="line.297"></a>
-<span class="sourceLineNo">298</span> this.store = store;<a name="line.298"></a>
-<span class="sourceLineNo">299</span> this.conf = conf;<a name="line.299"></a>
-<span class="sourceLineNo">300</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.300"></a>
-<span class="sourceLineNo">301</span> refreshConfiguration(conf);<a name="line.301"></a>
-<span class="sourceLineNo">302</span> }<a name="line.302"></a>
-<span class="sourceLineNo">303</span><a name="line.303"></a>
-<span class="sourceLineNo">304</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.304"></a>
-<span class="sourceLineNo">305</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.305"></a>
-<span class="sourceLineNo">306</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.306"></a>
-<span class="sourceLineNo">307</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.307"></a>
-<span class="sourceLineNo">308</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.308"></a>
-<span class="sourceLineNo">309</span><a name="line.309"></a>
-<span class="sourceLineNo">310</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.310"></a>
-<span class="sourceLineNo">311</span> @Override<a name="line.311"></a>
-<span class="sourceLineNo">312</span> public void setMaxProcId(long maxProcId) {<a name="line.312"></a>
-<span class="sourceLineNo">313</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.313"></a>
-<span class="sourceLineNo">314</span> lastProcId.set(maxProcId);<a name="line.314"></a>
-<span class="sourceLineNo">315</span> }<a name="line.315"></a>
-<span class="sourceLineNo">316</span><a name="line.316"></a>
-<span class="sourceLineNo">317</span> @Override<a name="line.317"></a>
-<span class="sourceLineNo">318</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.318"></a>
-<span class="sourceLineNo">319</span> loadProcedures(procIter, abortOnCorruption);<a name="line.319"></a>
-<span class="sourceLineNo">320</span> }<a name="line.320"></a>
-<span class="sourceLineNo">321</span><a name="line.321"></a>
-<span class="sourceLineNo">322</span> @Override<a name="line.322"></a>
-<span class="sourceLineNo">323</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.323"></a>
-<span class="sourceLineNo">324</span> int corruptedCount = 0;<a name="line.324"></a>
-<span class="sourceLineNo">325</span> while (procIter.hasNext()) {<a name="line.325"></a>
-<span class="sourceLineNo">326</span> Procedure<?> proc = procIter.next();<a name="line.326"></a>
-<span class="sourceLineNo">327</span> LOG.error("Corrupt " + proc);<a name="line.327"></a>
-<span class="sourceLineNo">328</span> corruptedCount++;<a name="line.328"></a>
-<span class="sourceLineNo">329</span> }<a name="line.329"></a>
-<span class="sourceLineNo">330</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.330"></a>
-<span class="sourceLineNo">331</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.331"></a>
-<span class="sourceLineNo">332</span> }<a name="line.332"></a>
-<span class="sourceLineNo">333</span> }<a name="line.333"></a>
-<span class="sourceLineNo">334</span> });<a name="line.334"></a>
-<span class="sourceLineNo">335</span> }<a name="line.335"></a>
-<span class="sourceLineNo">336</span><a name="line.336"></a>
-<span class="sourceLineNo">337</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.337"></a>
-<span class="sourceLineNo">338</span> final boolean abortOnCorruption) throws IOException {<a name="line.338"></a>
-<span class="sourceLineNo">339</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.339"></a>
-<span class="sourceLineNo">340</span><a name="line.340"></a>
-<span class="sourceLineNo">341</span> // 1. Build the rollback stack<a name="line.341"></a>
-<span class="sourceLineNo">342</span> int runnablesCount = 0;<a name="line.342"></a>
-<span class="sourceLineNo">343</span> while (procIter.hasNext()) {<a name="line.343"></a>
-<span class="sourceLineNo">344</span> boolean finished = procIter.isNextFinished();<a name="line.344"></a>
-<span class="sourceLineNo">345</span> Procedure proc = procIter.next();<a name="line.345"></a>
-<span class="sourceLineNo">346</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.346"></a>
-<span class="sourceLineNo">347</span> long procId = proc.getProcId();<a name="line.347"></a>
-<span class="sourceLineNo">348</span><a name="line.348"></a>
-<span class="sourceLineNo">349</span> if (finished) {<a name="line.349"></a>
-<span class="sourceLineNo">350</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.350"></a>
-<span class="sourceLineNo">351</span> if (debugEnabled) {<a name="line.351"></a>
-<span class="sourceLineNo">352</span> LOG.debug("Completed " + proc);<a name="line.352"></a>
-<span class="sourceLineNo">353</span> }<a name="line.353"></a>
-<span class="sourceLineNo">354</span> } else {<a name="line.354"></a>
-<span class="sourceLineNo">355</span> if (!proc.hasParent()) {<a name="line.355"></a>
-<span class="sourceLineNo">356</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.356"></a>
-<span class="sourceLineNo">357</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.357"></a>
-<span class="sourceLineNo">358</span> }<a name="line.358"></a>
+<span class="sourceLineNo">267</span><a name="line.267"></a>
+<span class="sourceLineNo">268</span> /**<a name="line.268"></a>
+<span class="sourceLineNo">269</span> * Created in the {@link #start(int, boolean)} method. Destroyed in {@link #join()} (FIX! Doing<a name="line.269"></a>
+<span class="sourceLineNo">270</span> * resource handling rather than observing in a #join is unexpected).<a name="line.270"></a>
+<span class="sourceLineNo">271</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.271"></a>
+<span class="sourceLineNo">272</span> * (Should be ok).<a name="line.272"></a>
+<span class="sourceLineNo">273</span> */<a name="line.273"></a>
+<span class="sourceLineNo">274</span> private ThreadGroup threadGroup;<a name="line.274"></a>
+<span class="sourceLineNo">275</span><a name="line.275"></a>
+<span class="sourceLineNo">276</span> /**<a name="line.276"></a>
+<span class="sourceLineNo">277</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.277"></a>
+<span class="sourceLineNo">278</span> * resource handling rather than observing in a #join is unexpected).<a name="line.278"></a>
+<span class="sourceLineNo">279</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.279"></a>
+<span class="sourceLineNo">280</span> * (Should be ok).<a name="line.280"></a>
+<span class="sourceLineNo">281</span> */<a name="line.281"></a>
+<span class="sourceLineNo">282</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.282"></a>
+<span class="sourceLineNo">283</span><a name="line.283"></a>
+<span class="sourceLineNo">284</span> /**<a name="line.284"></a>
+<span class="sourceLineNo">285</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.285"></a>
+<span class="sourceLineNo">286</span> * resource handling rather than observing in a #join is unexpected).<a name="line.286"></a>
+<span class="sourceLineNo">287</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.287"></a>
+<span class="sourceLineNo">288</span> * (Should be ok).<a name="line.288"></a>
+<span class="sourceLineNo">289</span> */<a name="line.289"></a>
+<span class="sourceLineNo">290</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.290"></a>
+<span class="sourceLineNo">291</span><a name="line.291"></a>
+<span class="sourceLineNo">292</span> private int corePoolSize;<a name="line.292"></a>
+<span class="sourceLineNo">293</span> private int maxPoolSize;<a name="line.293"></a>
+<span class="sourceLineNo">294</span><a name="line.294"></a>
+<span class="sourceLineNo">295</span> private volatile long keepAliveTime;<a name="line.295"></a>
+<span class="sourceLineNo">296</span><a name="line.296"></a>
+<span class="sourceLineNo">297</span> /**<a name="line.297"></a>
+<span class="sourceLineNo">298</span> * Scheduler/Queue that contains runnable procedures.<a name="line.298"></a>
+<span class="sourceLineNo">299</span> */<a name="line.299"></a>
+<span class="sourceLineNo">300</span> private final ProcedureScheduler scheduler;<a name="line.300"></a>
+<span class="sourceLineNo">301</span><a name="line.301"></a>
+<span class="sourceLineNo">302</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.302"></a>
+<span class="sourceLineNo">303</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.303"></a>
+<span class="sourceLineNo">304</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.304"></a>
+<span class="sourceLineNo">305</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.305"></a>
+<span class="sourceLineNo">306</span> private final TEnvironment environment;<a name="line.306"></a>
+<span class="sourceLineNo">307</span> private final ProcedureStore store;<a name="line.307"></a>
+<span class="sourceLineNo">308</span><a name="line.308"></a>
+<span class="sourceLineNo">309</span> private final boolean checkOwnerSet;<a name="line.309"></a>
+<span class="sourceLineNo">310</span><a name="line.310"></a>
+<span class="sourceLineNo">311</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.311"></a>
+<span class="sourceLineNo">312</span> final ProcedureStore store) {<a name="line.312"></a>
+<span class="sourceLineNo">313</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.313"></a>
+<span class="sourceLineNo">314</span> }<a name="line.314"></a>
+<span class="sourceLineNo">315</span><a name="line.315"></a>
+<span class="sourceLineNo">316</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.316"></a>
+<span class="sourceLineNo">317</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.317"></a>
+<span class="sourceLineNo">318</span> this.environment = environment;<a name="line.318"></a>
+<span class="sourceLineNo">319</span> this.scheduler = scheduler;<a name="line.319"></a>
+<span class="sourceLineNo">320</span> this.store = store;<a name="line.320"></a>
+<span class="sourceLineNo">321</span> this.conf = conf;<a name="line.321"></a>
+<span class="sourceLineNo">322</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.322"></a>
+<span class="sourceLineNo">323</span> refreshConfiguration(conf);<a name="line.323"></a>
+<span class="sourceLineNo">324</span><a name="line.324"></a>
+<span class="sourceLineNo">325</span> }<a name="line.325"></a>
+<span class="sourceLineNo">326</span><a name="line.326"></a>
+<span class="sourceLineNo">327</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.327"></a>
+<span class="sourceLineNo">328</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.328"></a>
+<span class="sourceLineNo">329</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.329"></a>
+<span class="sourceLineNo">330</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.330"></a>
+<span class="sourceLineNo">331</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.331"></a>
+<span class="sourceLineNo">332</span><a name="line.332"></a>
+<span class="sourceLineNo">333</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.333"></a>
+<span class="sourceLineNo">334</span> @Override<a name="line.334"></a>
+<span class="sourceLineNo">335</span> public void setMaxProcId(long maxProcId) {<a name="line.335"></a>
+<span class="sourceLineNo">336</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.336"></a>
+<span class="sourceLineNo">337</span> lastProcId.set(maxProcId);<a name="line.337"></a>
+<span class="sourceLineNo">338</span> }<a name="line.338"></a>
+<span class="sourceLineNo">339</span><a name="line.339"></a>
+<span class="sourceLineNo">340</span> @Override<a name="line.340"></a>
+<span class="sourceLineNo">341</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.341"></a>
+<span class="sourceLineNo">342</span> loadProcedures(procIter, abortOnCorruption);<a name="line.342"></a>
+<span class="sourceLineNo">343</span> }<a name="line.343"></a>
+<span class="sourceLineNo">344</span><a name="line.344"></a>
+<span class="sourceLineNo">345</span> @Override<a name="line.345"></a>
+<span class="sourceLineNo">346</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.346"></a>
+<span class="sourceLineNo">347</span> int corruptedCount = 0;<a name="line.347"></a>
+<span class="sourceLineNo">348</span> while (procIter.hasNext()) {<a name="line.348"></a>
+<span class="sourceLineNo">349</span> Procedure<?> proc = procIter.next();<a name="line.349"></a>
+<span class="sourceLineNo">350</span> LOG.error("Corrupt " + proc);<a name="line.350"></a>
+<span class="sourceLineNo">351</span> corruptedCount++;<a name="line.351"></a>
+<span class="sourceLineNo">352</span> }<a name="line.352"></a>
+<span class="sourceLineNo">353</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.353"></a>
+<span class="sourceLineNo">354</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.354"></a>
+<span class="sourceLineNo">355</span> }<a name="line.355"></a>
+<span class="sourceLineNo">356</span> }<a name="line.356"></a>
+<span class="sourceLineNo">357</span> });<a name="line.357"></a>
+<span class="sourceLineNo">358</span> }<a name="line.358"></a>
<span class="sourceLineNo">359</span><a name="line.359"></a>
-<span class="sourceLineNo">360</span> // add the procedure to the map<a name="line.360"></a>
-<span class="sourceLineNo">361</span> proc.beforeReplay(getEnvironment());<a name="line.361"></a>
-<span class="sourceLineNo">362</span> procedures.put(proc.getProcId(), proc);<a name="line.362"></a>
+<span class="sourceLineNo">360</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.360"></a>
+<span class="sourceLineNo">361</span> final boolean abortOnCorruption) throws IOException {<a name="line.361"></a>
+<span class="sourceLineNo">362</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.362"></a>
<span class="sourceLineNo">363</span><a name="line.363"></a>
-<span class="sourceLineNo">364</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.364"></a>
-<span class="sourceLineNo">365</span> runnablesCount++;<a name="line.365"></a>
-<span class="sourceLineNo">366</span> }<a name="line.366"></a>
-<span class="sourceLineNo">367</span> }<a name="line.367"></a>
-<span class="sourceLineNo">368</span><a name="line.368"></a>
-<span class="sourceLineNo">369</span> // add the nonce to the map<a name="line.369"></a>
-<span class="sourceLineNo">370</span> if (nonceKey != null) {<a name="line.370"></a>
-<span class="sourceLineNo">371</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.371"></a>
-<span class="sourceLineNo">372</span> }<a name="line.372"></a>
-<span class="sourceLineNo">373</span> }<a name="line.373"></a>
-<span class="sourceLineNo">374</span><a name="line.374"></a>
-<span class="sourceLineNo">375</span> // 2. Initialize the stacks<a name="line.375"></a>
-<span class="sourceLineNo">376</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.376"></a>
-<span class="sourceLineNo">377</span> HashSet<Procedure> waitingSet = null;<a name="line.377"></a>
-<span class="sourceLineNo">378</span> procIter.reset();<a name="line.378"></a>
-<span class="sourceLineNo">379</span> while (procIter.hasNext()) {<a name="line.379"></a>
-<span class="sourceLineNo">380</span> if (procIter.isNextFinished()) {<a name="line.380"></a>
-<span class="sourceLineNo">381</span> procIter.skipNext();<a name="line.381"></a>
-<span class="sourceLineNo">382</span> continue;<a name="line.382"></a>
-<span class="sourceLineNo">383</span> }<a name="line.383"></a>
-<span class="sourceLineNo">384</span><a name="line.384"></a>
-<span class="sourceLineNo">385</span> Procedure proc = procIter.next();<a name="line.385"></a>
-<span class="sourceLineNo">386</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.386"></a>
-<span class="sourceLineNo">387</span><a name="line.387"></a>
-<span class="sourceLineNo">388</span> if (debugEnabled) {<a name="line.388"></a>
-<span class="sourceLineNo">389</span> LOG.debug(String.format("Loading %s", proc));<a name="line.389"></a>
+<span class="sourceLineNo">364</span> // 1. Build the rollback stack<a name="line.364"></a>
+<span class="sourceLineNo">365</span> int runnablesCount = 0;<a name="line.365"></a>
+<span class="sourceLineNo">366</span> while (procIter.hasNext()) {<a name="line.366"></a>
+<span class="sourceLineNo">367</span> boolean finished = procIter.isNextFinished();<a name="line.367"></a>
+<span class="sourceLineNo">368</span> Procedure proc = procIter.next();<a name="line.368"></a>
+<span class="sourceLineNo">369</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.369"></a>
+<span class="sourceLineNo">370</span> long procId = proc.getProcId();<a name="line.370"></a>
+<span class="sourceLineNo">371</span><a name="line.371"></a>
+<span class="sourceLineNo">372</span> if (finished) {<a name="line.372"></a>
+<span class="sourceLineNo">373</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.373"></a>
+<span class="sourceLineNo">374</span> if (debugEnabled) {<a name="line.374"></a>
+<span class="sourceLineNo">375</span> LOG.debug("Completed " + proc);<a name="line.375"></a>
+<span class="sourceLineNo">376</span> }<a name="line.376"></a>
+<span class="sourceLineNo">377</span> } else {<a name="line.377"></a>
+<span class="sourceLineNo">378</span> if (!proc.hasParent()) {<a name="line.378"></a>
+<span class="sourceLineNo">379</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.379"></a>
+<span class="sourceLineNo">380</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.380"></a>
+<span class="sourceLineNo">381</span> }<a name="line.381"></a>
+<span class="sourceLineNo">382</span><a name="line.382"></a>
+<span class="sourceLineNo">383</span> // add the procedure to the map<a name="line.383"></a>
+<span class="sourceLineNo">384</span> proc.beforeReplay(getEnvironment());<a name="line.384"></a>
+<span class="sourceLineNo">385</span> procedures.put(proc.getProcId(), proc);<a name="line.385"></a>
+<span class="sourceLineNo">386</span><a name="line.386"></a>
+<span class="sourceLineNo">387</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.387"></a>
+<span class="sourceLineNo">388</span> runnablesCount++;<a name="line.388"></a>
+<span class="sourceLineNo">389</span> }<a name="line.389"></a>
<span class="sourceLineNo">390</span> }<a name="line.390"></a>
<span class="sourceLineNo">391</span><a name="line.391"></a>
-<span class="sourceLineNo">392</span> Long rootProcId = getRootProcedureId(proc);<a name="line.392"></a>
-<span class="sourceLineNo">393</span> if (rootProcId == null) {<a name="line.393"></a>
-<span class="sourceLineNo">394</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.394"></a>
-<span class="sourceLineNo">395</span> scheduler.addBack(proc);<a name="line.395"></a>
-<span class="sourceLineNo">396</span> continue;<a name="line.396"></a>
-<span class="sourceLineNo">397</span> }<a name="line.397"></a>
-<span class="sourceLineNo">398</span><a name="line.398"></a>
-<span class="sourceLineNo">399</span> if (proc.hasParent()) {<a name="line.399"></a>
-<span class="sourceLineNo">400</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.400"></a>
-<span class="sourceLineNo">401</span> // corrupted procedures are handled later at step 3<a name="line.401"></a>
-<span class="sourceLineNo">402</span> if (parent != null && !proc.isFinished()) {<a name="line.402"></a>
-<span class="sourceLineNo">403</span> parent.incChildrenLatch();<a name="line.403"></a>
-<span class="sourceLineNo">404</span> }<a name="line.404"></a>
-<span class="sourceLineNo">405</span> }<a name="line.405"></a>
-<span class="sourceLineNo">406</span><a name="line.406"></a>
-<span class="sourceLineNo">407</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.407"></a>
-<span class="sourceLineNo">408</span> procStack.loadStack(proc);<a name="line.408"></a>
-<span class="sourceLineNo">409</span><a name="line.409"></a>
-<span class="sourceLineNo">410</span> proc.setRootProcId(rootProcId);<a name="line.410"></a>
-<span class="sourceLineNo">411</span> switch (proc.getState()) {<a name="line.411"></a>
-<span class="sourceLineNo">412</span> case RUNNABLE:<a name="line.412"></a>
-<span class="sourceLineNo">413</span> runnableList.add(proc);<a name="line.413"></a>
-<span class="sourceLineNo">414</span> break;<a name="line.414"></a>
-<span class="sourceLineNo">415</span> case WAITING:<a name="line.415"></a>
-<span class="sourceLineNo">416</span> if (!proc.hasChildren()) {<a name="line.416"></a>
-<span class="sourceLineNo">417</span> runnableList.add(proc);<a name="line.417"></a>
-<span class="sourceLineNo">418</span> }<a name="line.418"></a>
-<span class="sourceLineNo">419</span> break;<a name="line.419"></a>
-<span class="sourceLineNo">420</span> case WAITING_TIMEOUT:<a name="line.420"></a>
-<span class="sourceLineNo">421</span> if (waitingSet == null) {<a name="line.421"></a>
-<span class="sourceLineNo">422</span> waitingSet = new HashSet<>();<a name="line.422"></a>
-<span class="sourceLineNo">423</span> }<a name="line.423"></a>
-<span class="sourceLineNo">424</span> waitingSet.add(proc);<a name="line.424"></a>
-<span class="sourceLineNo">425</span> break;<a name="line.425"></a>
-<span class="sourceLineNo">426</span> case FAILED:<a name="line.426"></a>
-<span class="sourceLineNo">427</span> // add the proc to the scheduler to perform the rollback<a name="line.427"></a>
-<span class="sourceLineNo">428</span> scheduler.addBack(proc);<a name="line.428"></a>
-<span class="sourceLineNo">429</span> break;<a name="line.429"></a>
-<span class="sourceLineNo">430</span> case ROLLEDBACK:<a name="line.430"></a>
-<span class="sourceLineNo">431</span> case INITIALIZING:<a name="line.431"></a>
-<span class="sourceLineNo">432</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.432"></a>
-<span class="sourceLineNo">433</span> LOG.error(msg);<a name="line.433"></a>
-<span class="sourceLineNo">434</span> throw new UnsupportedOperationException(msg);<a name="line.434"></a>
-<span class="sourceLineNo">435</span> default:<a name="line.435"></a>
-<span class="sourceLineNo">436</span> break;<a name="line.436"></a>
-<span class="sourceLineNo">437</span> }<a name="line.437"></a>
-<span class="sourceLineNo">438</span> }<a name="line.438"></a>
-<span class="sourceLineNo">439</span><a name="line.439"></a>
-<span class="sourceLineNo">440</span> // 3. Validate the stacks<a name="line.440"></a>
-<span class="sourceLineNo">441</span> int corruptedCount = 0;<a name="line.441"></a>
-<span class="sourceLineNo">442</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.442"></a>
-<span class="sourceLineNo">443</span> while (itStack.hasNext()) {<a name="line.443"></a>
-<span class="sourceLineNo">444</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.444"></a>
-<span class="sourceLineNo">445</span> RootProcedureState procStack = entry.getValue();<a name="line.445"></a>
-<span class="sourceLineNo">446</span> if (procStack.isValid()) continue;<a name="line.446"></a>
-<span class="sourceLineNo">447</span><a name="line.447"></a>
-<span class="sourceLineNo">448</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.448"></a>
-<span class="sourceLineNo">449</span> LOG.error("Corrupted " + proc);<a name="line.449"></a>
-<span class="sourceLineNo">450</span> procedures.remove(proc.getProcId());<a name="line.450"></a>
-<span class="sourceLineNo">451</span> runnableList.remove(proc);<a name="line.451"></a>
-<span class="sourceLineNo">452</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.452"></a>
-<span class="sourceLineNo">453</span> corruptedCount++;<a name="line.453"></a>
-<span class="sourceLineNo">454</span> }<a name="line.454"></a>
-<span class="sourceLineNo">455</span> itStack.remove();<a name="line.455"></a>
-<span class="sourceLineNo">456</span> }<a name="line.456"></a>
-<span class="sourceLineNo">457</span><a name="line.457"></a>
-<span class="sourceLineNo">458</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.458"></a>
-<span class="sourceLineNo">459</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.459"></a>
-<span class="sourceLineNo">460</span> }<a name="line.460"></a>
-<span class="sourceLineNo">461</span><a name="line.461"></a>
-<span class="sourceLineNo">462</span> // 4. Push the procedures to the timeout executor<a name="line.462"></a>
-<span class="sourceLineNo">463</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.463"></a>
-<span class="sourceLineNo">464</span> for (Procedure proc: waitingSet) {<a name="line.464"></a>
-<span class="sourceLineNo">465</span> proc.afterReplay(getEnvironment());<a name="line.465"></a>
-<span class="sourceLineNo">466</span> timeoutExecutor.add(proc);<a name="line.466"></a>
-<span class="sourceLineNo">467</span> }<a name="line.467"></a>
-<span class="sourceLineNo">468</span> }<a name="line.468"></a>
-<span class="sourceLineNo">469</span><a name="line.469"></a>
-<span class="sourceLineNo">470</span> // 5. Push the procedure to the scheduler<a name="line.470"></a>
-<span class="sourceLineNo">471</span> if (!runnableList.isEmpty()) {<a name="line.471"></a>
-<span class="sourceLineNo">472</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.472"></a>
-<span class="sourceLineNo">473</span> // some procedure may be started way before this stuff.<a name="line.473"></a>
-<span class="sourceLineNo">474</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.474"></a>
-<span class="sourceLineNo">475</span> Procedure proc = runnableList.get(i);<a name="line.475"></a>
-<span class="sourceLineNo">476</span> proc.afterReplay(getEnvironment());<a name="line.476"></a>
-<span class="sourceLineNo">477</span> if (!proc.hasParent()) {<a name="line.477"></a>
-<span class="sourceLineNo">478</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.478"></a>
-<span class="sourceLineNo">479</span> }<a name="line.479"></a>
-<span class="sourceLineNo">480</span> if (proc.wasExecuted()) {<a name="line.480"></a>
-<span class="sourceLineNo">481</span> scheduler.addFront(proc);<a name="line.481"></a>
-<span class="sourceLineNo">482</span> } else {<a name="line.482"></a>
-<span class="sourceLineNo">483</span> // if it was not in execution, it can wait.<a name="line.483"></a>
-<span class="sourceLineNo">484</span> scheduler.addBack(proc);<a name="line.484"></a>
-<span class="sourceLineNo">485</span> }<a name="line.485"></a>
-<span class="sourceLineNo">486</span> }<a name="line.486"></a>
-<span class="sourceLineNo">487</span> }<a name="line.487"></a>
-<span class="sourceLineNo">488</span> }<a name="line.488"></a>
-<span class="sourceLineNo">489</span><a name="line.489"></a>
-<span class="sourceLineNo">490</span> /**<a name="line.490"></a>
-<span class="sourceLineNo">491</span> * Start the procedure executor.<a name="line.491"></a>
-<span class="sourceLineNo">492</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.492"></a>
-<span class="sourceLineNo">493</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.493"></a>
-<span class="sourceLineNo">494</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.494"></a>
-<span class="sourceLineNo">495</span> *<a name="line.495"></a>
-<span class="sourceLineNo">496</span> * @param numThreads number of threads available for procedure execution.<a name="line.496"></a>
-<span class="sourceLineNo">497</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.497"></a>
-<span class="sourceLineNo">498</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.498"></a>
-<span class="sourceLineNo">499</span> */<a name="line.499"></a>
-<span class="sourceLineNo">500</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.500"></a>
-<span class="sourceLineNo">501</span> if (!running.compareAndSet(false, true)) {<a name="line.501"></a>
-<span class="sourceLineNo">502</span> LOG.warn("Already running");<a name="line.502"></a>
-<span class="sourceLineNo">503</span> return;<a name="line.503"></a>
-<span class="sourceLineNo">504</span> }<a name="line.504"></a>
-<span class="sourceLineNo">505</span><a name="line.505"></a>
-<span class="sourceLineNo">506</span> // We have numThreads executor + one timer thread used for timing out<a name="line.506"></a>
-<span class="sourceLineNo">507</span> // procedures and triggering periodic procedures.<a name="line.507"></a>
-<span class="sourceLineNo">508</span> this.corePoolSize = numThreads;<a name="line.508"></a>
-<span class="sourceLineNo">509</span> this.maxPoolSize = 10 * numThreads;<a name="line.509"></a>
-<span class="sourceLineNo">510</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.510"></a>
-<span class="sourceLineNo">511</span> corePoolSize, maxPoolSize);<a name="line.511"></a>
+<span class="sourceLineNo">392</span> // add the nonce to the map<a name="line.392"></a>
+<span class="sourceLineNo">393</span> if (nonceKey != null) {<a name="line.393"></a>
+<span class="sourceLineNo">394</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.394"></a>
+<span class="sourceLineNo">395</span> }<a name="line.395"></a>
+<span class="sourceLineNo">396</span> }<a name="line.396"></a>
+<span class="sourceLineNo">397</span><a name="line.397"></a>
+<span class="sourceLineNo">398</span> // 2. Initialize the stacks<a name="line.398"></a>
+<span class="sourceLineNo">399</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.399"></a>
+<span class="sourceLineNo">400</span> HashSet<Procedure> waitingSet = null;<a name="line.400"></a>
+<span class="sourceLineNo">401</span> procIter.reset();<a name="line.401"></a>
+<span class="sourceLineNo">402</span> while (procIter.hasNext()) {<a name="line.402"></a>
+<span class="sourceLineNo">403</span> if (procIter.isNextFinished()) {<a name="line.403"></a>
+<span class="sourceLineNo">404</span> procIter.skipNext();<a name="line.404"></a>
+<span class="sourceLineNo">405</span> continue;<a name="line.405"></a>
+<span class="sourceLineNo">406</span> }<a name="line.406"></a>
+<span class="sourceLineNo">407</span><a name="line.407"></a>
+<span class="sourceLineNo">408</span> Procedure proc = procIter.next();<a name="line.408"></a>
+<span class="sourceLineNo">409</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.409"></a>
+<span class="sourceLineNo">410</span><a name="line.410"></a>
+<span class="sourceLineNo">411</span> if (debugEnabled) {<a name="line.411"></a>
+<span class="sourceLineNo">412</span> LOG.debug(String.format("Loading %s", proc));<a name="line.412"></a>
+<span class="sourceLineNo">413</span> }<a name="line.413"></a>
+<span class="sourceLineNo">414</span><a name="line.414"></a>
+<span class="sourceLineNo">415</span> Long rootProcId = getRootProcedureId(proc);<a name="line.415"></a>
+<span class="sourceLineNo">416</span> if (rootProcId == null) {<a name="line.416"></a>
+<span class="sourceLineNo">417</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.417"></a>
+<span class="sourceLineNo">418</span> scheduler.addBack(proc);<a name="line.418"></a>
+<span class="sourceLineNo">419</span> continue;<a name="line.419"></a>
+<span class="sourceLineNo">420</span> }<a name="line.420"></a>
+<span class="sourceLineNo">421</span><a name="line.421"></a>
+<span class="sourceLineNo">422</span> if (proc.hasParent()) {<a name="line.422"></a>
+<span class="sourceLineNo">423</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.423"></a>
+<span class="sourceLineNo">424</span> // corrupted procedures are handled later at step 3<a name="line.424"></a>
+<span class="sourceLineNo">425</span> if (parent != null && !proc.isFinished()) {<a name="line.425"></a>
+<span class="sourceLineNo">426</span> parent.incChildrenLatch();<a name="line.426"></a>
+<span class="sourceLineNo">427</span> }<a name="line.427"></a>
+<span class="sourceLineNo">428</span> }<a name="line.428"></a>
+<span class="sourceLineNo">429</span><a name="line.429"></a>
+<span class="sourceLineNo">430</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.430"></a>
+<span class="sourceLineNo">431</span> procStack.loadStack(proc);<a name="line.431"></a>
+<span class="sourceLineNo">432</span><a name="line.432"></a>
+<span class="sourceLineNo">433</span> proc.setRootProcId(rootProcId);<a name="line.433"></a>
+<span class="sourceLineNo">434</span> switch (proc.getState()) {<a name="line.434"></a>
+<span class="sourceLineNo">435</span> case RUNNABLE:<a name="line.435"></a>
+<span class="sourceLineNo">436</span> runnableList.add(proc);<a name="line.436"></a>
+<span class="sourceLineNo">437</span> break;<a name="line.437"></a>
+<span class="sourceLineNo">438</span> case WAITING:<a name="line.438"></a>
+<span class="sourceLineNo">439</span> if (!proc.hasChildren()) {<a name="line.439"></a>
+<span class="sourceLineNo">440</span> runnableList.add(proc);<a name="line.440"></a>
+<span class="sourceLineNo">441</span> }<a name="line.441"></a>
+<span class="sourceLineNo">442</span> break;<a name="line.442"></a>
+<span class="sourceLineNo">443</span> case WAITING_TIMEOUT:<a name="line.443"></a>
+<span class="sourceLineNo">444</span> if (waitingSet == null) {<a name="line.444"></a>
+<span class="sourceLineNo">445</span> waitingSet = new HashSet<>();<a name="line.445"></a>
+<span class="sourceLineNo">446</span> }<a name="line.446"></a>
+<span class="sourceLineNo">447</span> waitingSet.add(proc);<a name="line.447"></a>
+<span class="sourceLineNo">448</span> break;<a name="line.448"></a>
+<span class="sourceLineNo">449</span> case FAILED:<a name="line.449"></a>
+<span class="sourceLineNo">450</span> // add the proc to the scheduler to perform the rollback<a name="line.450"></a>
+<span class="sourceLineNo">451</span> scheduler.addBack(proc);<a name="line.451"></a>
+<span class="sourceLineNo">452</span> break;<a name="line.452"></a>
+<span class="sourceLineNo">453</span> case ROLLEDBACK:<a name="line.453"></a>
+<span class="sourceLineNo">454</span> case INITIALIZING:<a name="line.454"></a>
+<span class="sourceLineNo">455</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.455"></a>
+<span class="sourceLineNo">456</span> LOG.error(msg);<a name="line.456"></a>
+<span class="sourceLineNo">457</span> throw new UnsupportedOperationException(msg);<a name="line.457"></a>
+<span class="sourceLineNo">458</span> default:<a name="line.458"></a>
+<span class="sourceLineNo">459</span> break;<a name="line.459"></a>
+<span class="sourceLineNo">460</span> }<a name="line.460"></a>
+<span class="sourceLineNo">461</span> }<a name="line.461"></a>
+<span class="sourceLineNo">462</span><a name="line.462"></a>
+<span class="sourceLineNo">463</span> // 3. Validate the stacks<a name="line.463"></a>
+<span class="sourceLineNo">464</span> int corruptedCount = 0;<a name="line.464"></a>
+<span class="sourceLineNo">465</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.465"></a>
+<span class="sourceLineNo">466</span> while (itStack.hasNext()) {<a name="line.466"></a>
+<span class="sourceLineNo">467</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.467"></a>
+<span class="sourceLineNo">468</span> RootProcedureState procStack = entry.getValue();<a name="line.468"></a>
+<span class="sourceLineNo">469</span> if (procStack.isValid()) continue;<a name="line.469"></a>
+<span class="sourceLineNo">470</span><a name="line.470"></a>
+<span class="sourceLineNo">471</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.471"></a>
+<span class="sourceLineNo">472</span> LOG.error("Corrupted " + proc);<a name="line.472"></a>
+<span class="sourceLineNo">473</span> procedures.remove(proc.getProcId());<a name="line.473"></a>
+<span class="sourceLineNo">474</span> runnableList.remove(proc);<a name="line.474"></a>
+<span class="sourceLineNo">475</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.475"></a>
+<span class="sourceLineNo">476</span> corruptedCount++;<a name="line.476"></a>
+<span class="sourceLineNo">477</span> }<a name="line.477"></a>
+<span class="sourceLineNo">478</span> itStack.remove();<a name="line.478"></a>
+<span class="sourceLineNo">479</span> }<a name="line.479"></a>
+<span class="sourceLineNo">480</span><a name="line.480"></a>
+<span class="sourceLineNo">481</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.481"></a>
+<span class="sourceLineNo">482</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.482"></a>
+<span class="sourceLineNo">483</span> }<a name="line.483"></a>
+<span class="sourceLineNo">484</span><a name="line.484"></a>
+<span class="sourceLineNo">485</span> // 4. Push the procedures to the timeout executor<a name="line.485"></a>
+<span class="sourceLineNo">486</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.486"></a>
+<span class="sourceLineNo">487</span> for (Procedure proc: waitingSet) {<a name="line.487"></a>
+<span class="sourceLineNo">488</span> proc.afterReplay(getEnvironment());<a name="line.488"></a>
+<span class="sourceLineNo">489</span> timeoutExecutor.add(proc);<a name="line.489"></a>
+<span class="sourceLineNo">490</span> }<a name="line.490"></a>
+<span class="sourceLineNo">491</span> }<a name="line.491"></a>
+<span class="sourceLineNo">492</span><a name="line.492"></a>
+<span class="sourceLineNo">493</span> // 5. Push the procedure to the scheduler<a name="line.493"></a>
+<span class="sourceLineNo">494</span> if (!runnableList.isEmpty()) {<a name="line.494"></a>
+<span class="sourceLineNo">495</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.495"></a>
+<span class="sourceLineNo">496</span> // some procedure may be started way before this stuff.<a name="line.496"></a>
+<span class="sourceLineNo">497</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.497"></a>
+<span class="sourceLineNo">498</span> Procedure proc = runnableList.get(i);<a name="line.498"></a>
+<span class="sourceLineNo">499</span> proc.afterReplay(getEnvironment());<a name="line.499"></a>
+<span class="sourceLineNo">500</span> if (!proc.hasParent()) {<a name="line.500"></a>
+<span class="sourceLineNo">501</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.501"></a>
+<span class="sourceLineNo">502</span> }<a name="line.502"></a>
+<span class="sourceLineNo">503</span> if (proc.wasExecuted()) {<a name="line.503"></a>
+<span class="sourceLineNo">504</span> scheduler.addFront(proc);<a name="line.504"></a>
+<span class="sourceLineNo">505</span> } else {<a name="line.505"></a>
+<span class="sourceLineNo">506</span> // if it was not in execution, it can wait.<a name="line.506"></a>
+<span class="sourceLineNo">507</span> scheduler.addBack(proc);<a name="line.507"></a>
+<span class="sourceLineNo">508</span> }<a name="line.508"></a>
+<span class="sourceLineNo">509</span> }<a name="line.509"></a>
+<span class="sourceLineNo">510</span> }<a name="line.510"></a>
+<span class="sourceLineNo">511</span> }<a name="line.511"></a>
<span class="sourceLineNo">512</span><a name="line.512"></a>
-<span class="sourceLineNo">513</span> // Create the Thread Group for the executors<a name="line.513"></a>
-<span class="sourceLineNo">514</span> threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.514"></a>
-<span class="sourceLineNo">515</span><a name="line.515"></a>
-<span class="sourceLineNo">516</span> // Create the timeout executor<a name="line.516"></a>
-<span class="sourceLineNo">517</span> timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.517"></a>
-<span class="sourceLineNo">518</span><a name="line.518"></a>
-<span class="sourceLineNo">519</span> // Create the workers<a name="line.519"></a>
-<span class="sourceLineNo">520</span> workerId.set(0);<a name="line.520"></a>
-<span class="sourceLineNo">521</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.521"></a>
-<span class="sourceLineNo">522</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.522"></a>
-<span class="sourceLineNo">523</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.523"></a>
-<span class="sourceLineNo">524</span> }<a name="line.524"></a>
-<span class="sourceLineNo">525</span><a name="line.525"></a>
-<span class="sourceLineNo">526</span> long st, et;<a name="line.526"></a>
-<span class="sourceLineNo">527</span><a name="line.527"></a>
-<span class="sourceLineNo">528</span> // Acquire the store lease.<a name="line.528"></a>
-<span class="sourceLineNo">529</span> st = EnvironmentEdgeManager.currentTime();<a name="line.529"></a>
-<span class="sourceLineNo">530</span> store.recoverLease();<a name="line.530"></a>
-<span class="sourceLineNo">531</span> et = EnvironmentEdgeManager.currentTime();<a name="line.531"></a>
-<span class="sourceLineNo">532</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.532"></a>
-<span class="sourceLineNo">533</span> StringUtils.humanTimeDiff(et - st));<a name="line.533"></a>
-<span class="sourceLineNo">534</span><a name="line.534"></a>
-<span class="sourceLineNo">535</span> // start the procedure scheduler<a name="line.535"></a>
-<span class="sourceLineNo">536</span> scheduler.start();<a name="line.536"></a>
-<span class="sourceLineNo">537</span><a name="line.537"></a>
-<span class="sourceLineNo">538</span> // TODO: Split in two steps.<a name="line.538"></a>
-<span class="sourceLineNo">539</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.539"></a>
-<span class="sourceLineNo">540</span> // The first one will make sure that we have the latest id,<a name="line.540"></a>
-<span class="sourceLineNo">541</span> // so we can start the threads and accept new procedures.<a name="line.541"></a>
-<span class="sourceLineNo">542</span> // The second step will do the actual load of old procedures.<a name="line.542"></a>
-<span class="sourceLineNo">543</span> st = EnvironmentEdgeManager.currentTime();<a name="line.543"></a>
-<span class="sourceLineNo">544</span> load(abortOnCorruption);<a name="line.544"></a>
-<span class="sourceLineNo">545</span> et = EnvironmentEdgeManager.currentTime();<a name="line.545"></a>
-<span class="sourceLineNo">546</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.546"></a>
-<span class="sourceLineNo">547</span> StringUtils.humanTimeDiff(et - st));<a name="line.547"></a>
-<span class="sourceLineNo">548</span><a name="line.548"></a>
-<span class="sourceLineNo">549</span> // Start the executors. Here we must have the lastProcId set.<a name="line.549"></a>
-<span class="sourceLineNo">550</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.550"></a>
-<span class="sourceLineNo">551</span> timeoutExecutor.start();<a name="line.551"></a>
-<span class="sourceLineNo">552</span> for (WorkerThread worker: workerThreads) {<a name="line.552"></a>
-<span class="sourceLineNo">553</span> worker.start();<a name="line.553"></a>
-<span class="sourceLineNo">554</span> }<a name="line.554"></a>
-<span class="sourceLineNo">555</span><a name="line.555"></a>
-<span class="sourceLineNo">556</span> // Internal chores<a name="line.556"></a>
-<span class="sourceLineNo">557</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.557"></a>
-<span class="sourceLineNo">558</span><a name="line.558"></a>
-<span class="sourceLineNo">559</span> // Add completed cleaner chore<a name="line.559"></a>
-<span class="sourceLineNo">560</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.560"></a>
-<span class="sourceLineNo">561</span> }<a name="line.561"></a>
-<span class="sourceLineNo">562</span><a name="line.562"></a>
-<span class="sourceLineNo">563</span> public void stop() {<a name="line.563"></a>
-<span class="sourceLineNo">564</span> if (!running.getAndSet(false)) {<a name="line.564"></a>
-<span class="sourceLineNo">565</span> return;<a name="line.565"></a>
-<span class="sourceLineNo">566</span> }<a name="line.566"></a>
-<span class="sourceLineNo">567</span><a name="line.567"></a>
-<span class="sourceLineNo">568</span> LOG.info("Stopping");<a name="line.568"></a>
-<span class="sourceLineNo">569</span> scheduler.stop();<a name="line.569"></a>
-<span class="sourceLineNo">570</span> timeoutExecutor.sendStopSignal();<a name="line.570"></a>
-<span class="sourceLineNo">571</span> }<a name="line.571"></a>
-<span class="sourceLineNo">572</span><a name="line.572"></a>
-<span class="sourceLineNo">573</span> @VisibleForTesting<a name="line.573"></a>
-<span class="sourceLineNo">574</span> public void join() {<a name="line.574"></a>
-<span class="sourceLineNo">575</span> assert !isRunning() : "expected not running";<a name="line.575"></a>
-<span class="sourceLineNo">576</span><a name="line.576"></a>
-<span class="sourceLineNo">577</span> // stop the timeout executor<a name="line.577"></a>
-<span class="sourceLineNo">578</span> timeoutExecutor.awaitTermination();<a name="line.578"></a>
-<span class="sourceLineNo">579</span> timeoutExecutor = null;<a name="line.579"></a>
-<span class="sourceLineNo">580</span><a name="line.580"></a>
-<span class="sourceLineNo">581</span> // stop the worker threads<a name="line.581"></a>
-<span class="sourceLineNo">582</span> for (WorkerThread worker: workerThreads) {<a name="line.582"></a>
-<span class="sourceLineNo">583</span> worker.awaitTermination();<a name="line.583"></a>
-<span class="sourceLineNo">584</span> }<a name="line.584"></a>
-<span class="sourceLineNo">585</span> workerThreads = null;<a name="line.585"></a>
-<span class="sourceLineNo">586</span><a name="line.586"></a>
-<span class="sourceLineNo">587</span> // Destroy the Thread Group for the executors<a name="line.587"></a>
-<span class="sourceLineNo">588</span> try {<a name="line.588"></a>
-<span class="sourceLineNo">589</span> threadGroup.destroy();<a name="line.589"></a>
-<span class="sourceLineNo">590</span> } catch (IllegalThreadStateException e) {<a name="line.590"></a>
-<span class="sourceLineNo">591</span> LOG.error("ThreadGroup " + threadGroup + " contains running threads; " + e.getMessage());<a name="line.591"></a>
-<span class="sourceLineNo">592</span> threadGroup.list();<a name="line.592"></a>
-<span class="sourceLineNo">593</span> } finally {<a name="line.593"></a>
-<span class="sourceLineNo">594</span> threadGroup = null;<a name="line.594"></a>
-<span class="sourceLineNo">595</span> }<a name="line.595"></a>
+<span class="sourceLineNo">513</span> /**<a name="line.513"></a>
+<span class="sourceLineNo">514</span> * Start the procedure executor.<a name="line.514"></a>
+<span class="sourceLineNo">515</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.515"></a>
+<span class="sourceLineNo">516</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.516"></a>
+<span class="sourceLineNo">517</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.517"></a>
+<span class="sourceLineNo">518</span> *<a name="line.518"></a>
+<span class="sourceLineNo">519</span> * @param numThreads number of threads available for procedure execution.<a name="line.519"></a>
+<span class="sourceLineNo">520</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.520"></a>
+<span class="sourceLineNo">521</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.521"></a>
+<span class="sourceLineNo">522</span> */<a name="line.522"></a>
+<span class="sourceLineNo">523</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.523"></a>
+<span class="sourceLineNo">524</span> if (!running.compareAndSet(false, true)) {<a name="line.524"></a>
+<span class="sourceLineNo">525</span> LOG.warn("Already running");<a name="line.525"></a>
+<span class="sourceLineNo">526</span> return;<a name="line.526"></a>
+<span class="sourceLineNo">527</span> }<a name="line.527"></a>
+<span class="sourceLineNo">528</span><a name="line.528"></a>
+<span class="sourceLineNo">529</span> // We have numThreads executor + one timer thread used for timing out<a name="line.529"></a>
+<span class="sourceLineNo">530</span> // procedures and triggering periodic procedures.<a name="line.530"></a>
+<span class="sourceLineNo">531</span> this.corePoolSize = numThreads;<a name="line.531"></a>
+<span class="sourceLineNo">532</span> this.maxPoolSize = 10 * numThreads;<a name="line.532"></a>
+<span class="sourceLineNo">533</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.533"></a>
+<span class="sourceLineNo">534</span> corePoolSize, maxPoolSize);<a name="line.534"></a>
+<span class="sourceLineNo">535</span><a name="line.535"></a>
+<span class="sourceLineNo">536</span> this.threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.536"></a>
+<span class="sourceLineNo">537</span> this.timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.537"></a>
+<span class="sourceLineNo">538</span><a name="line.538"></a>
+<span class="sourceLineNo">539</span> // Create the workers<a name="line.539"></a>
+<span class="sourceLineNo">540</span> workerId.set(0);<a name="line.540"></a>
+<span class="sourceLineNo">541</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.541"></a>
+<span class="sourceLineNo">542</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.542"></a>
+<span class="sourceLineNo">543</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.543"></a>
+<span class="sourceLineNo">544</span> }<a name="line.544"></a>
+<span class="sourceLineNo">545</span><a name="line.545"></a>
+<span class="sourceLineNo">546</span> long st, et;<a name="line.546"></a>
+<span class="sourceLineNo">547</span><a name="line.547"></a>
+<span class="sourceLineNo">548</span> // Acquire the store lease.<a name="line.548"></a>
+<span class="sourceLineNo">549</span> st = EnvironmentEdgeManager.currentTime();<a name="line.549"></a>
+<span class="sourceLineNo">550</span> store.recoverLease();<a name="line.550"></a>
+<span class="sourceLineNo">551</span> et = EnvironmentEdgeManager.currentTime();<a name="line.551"></a>
+<span class="sourceLineNo">552</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.552"></a>
+<span class="sourceLineNo">553</span> StringUtils.humanTimeDiff(et - st));<a name="line.553"></a>
+<span class="sourceLineNo">554</span><a name="line.554"></a>
+<span class="sourceLineNo">555</span> // start the procedure scheduler<a name="line.555"></a>
+<span class="sourceLineNo">556</span> scheduler.start();<a name="line.556"></a>
+<span class="sourceLineNo">557</span><a name="line.557"></a>
+<span class="sourceLineNo">558</span> // TODO: Split in two steps.<a name="line.558"></a>
+<span class="sourceLineNo">559</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.559"></a>
+<span class="sourceLineNo">560</span> // The first one will make sure that we have the latest id,<a name="line.560"></a>
+<span class="sourceLineNo">561</span> // so we can start the threads and accept new procedures.<a name="line.561"></a>
+<span class="sourceLineNo">562</span> // The second step will do the actual load of old procedures.<a name="line.562"></a>
+<span class="sourceLineNo">563</span> st = EnvironmentEdgeManager.currentTime();<a name="line.563"></a>
+<span class="sourceLineNo">564</span> load(abortOnCorruption);<a name="line.564"></a>
+<span class="sourceLineNo">565</span> et = EnvironmentEdgeManager.currentTime();<a name="line.565"></a>
+<span class="sourceLineNo">566</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.566"></a>
+<span class="sourceLineNo">567</span> StringUtils.humanTimeDiff(et - st));<a name="line.567"></a>
+<span class="sourceLineNo">568</span><a name="line.568"></a>
+<span class="sourceLineNo">569</span> // Start the executors. Here we must have the lastProcId set.<a name="line.569"></a>
+<span class="sourceLineNo">570</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.570"></a>
+<span class="sourceLineNo">571</span> timeoutExecutor.start();<a name="line.571"></a>
+<span class="sourceLineNo">572</span> for (WorkerThread worker: workerThreads) {<a name="line.572"></a>
+<span class="sourceLineNo">573</span> worker.start();<a name="line.573"></a>
+<span class="sourceLineNo">574</span> }<a name="line.574"></a>
+<span class="sourceLineNo">575</span><a name="line.575"></a>
+<span class="sourceLineNo">576</span> // Internal chores<a name="line.576"></a>
+<span class="sourceLineNo">577</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.577"></a>
+<span class="sourceLineNo">578</span><a name="line.578"></a>
+<span class="sourceLineNo">579</span> // Add completed cleaner chore<a name="line.579"></a>
+<span class="sourceLineNo">580</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.580"></a>
+<span class="sourceLineNo">581</span> }<a name="line.581"></a>
+<span class="sourceLineNo">582</span><a name="line.582"></a>
+<span class="sourceLineNo">583</span> public void stop() {<a name="line.583"></a>
+<span class="sourceLineNo">584</span> if (!running.getAndSet(false)) {<a name="line.584"></a>
+<span class="sourceLineNo">585</span> return;<a name="line.585"></a>
+<span class="sourceLineNo">586</span> }<a name="line.586"></a>
+<span class="sourceLineNo">587</span><a name="line.587"></a>
+<span class="sourceLineNo">588</span> LOG.info("Stopping");<a name="line.588"></a>
+<span class="sourceLineNo">589</span> scheduler.stop();<a name="line.589"></a>
+<span class="sourceLineNo">590</span> timeoutExecutor.sendStopSignal();<a name="line.590"></a>
+<span class="sourceLineNo">591</span> }<a name="line.591"></a>
+<span class="sourceLineNo">592</span><a name="line.592"></a>
+<span class="sourceLineNo">593</span> @VisibleForTesting<a name="line.593"></a>
+<span class="sourceLineNo">594</span> public void join() {<a name="line.594"></a>
+<span class="sourceLineNo">595</span> assert !isRunning() : "expected not running";<a name="line.595"></a>
<span class="sourceLineNo">596</span><a name="line.596"></a>
-<span class="sourceLineNo">597</span> // reset the in-memory state for testing<a name="line.597"></a>
-<span class="sourceLineNo">598</span> completed.clear();<a name="line.598"></a>
-<span class="sourceLineNo">599</span> rollbackStack.clear();<a name="line.599"></a>
-<span class="sourceLineNo">600</span> procedures.clear();<a name="line.600"></a>
-<span class="sourceLineNo">601</span> nonceKeysToProcIdsMap.clear();<a name="line.601"></a>
-<span class="sourceLineNo">602</span> scheduler.clear();<a name="line.602"></a>
-<span class="sourceLineNo">603</span> lastProcId.set(-1);<a name="line.603"></a>
-<span class="sourceLineNo">604</span> }<a name="line.604"></a>
-<span class="sourceLineNo">605</span><a name="line.605"></a>
-<span class="sourceLineNo">606</span> public void refreshConfiguration(final Configuration conf) {<a name="line.606"></a>
-<span class="sourceLineNo">607</span> this.conf = conf;<a name="line.607"></a>
-<span class="sourceLineNo">608</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.608"></a>
-<span class="sourceLineNo">609</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.609"></a>
-<span class="sourceLineNo">610</span> }<a name="line.610"></a>
-<span class="sourceLineNo">611</span><a name="line.611"></a>
-<span class="sourceLineNo">612</span> // ==========================================================================<a name="line.612"></a>
-<span class="sourceLineNo">613</span> // Accessors<a name="line.613"></a>
-<span class="sourceLineNo">614</span> // ==========================================================================<a name="line.614"></a>
-<span class="sourceLineNo">615</span> public boolean isRunning() {<a name="line.615"></a>
-<span class="sourceLineNo">616</span> return running.get();<a name="line.616"></a>
-<span class="sourceLineNo">617</span> }<a name="line.617"></a>
-<span class="sourceLineNo">618</span><a name="line.618"></a>
-<span class="sourceLineNo">619</span> /**<a name="line.619"></a>
-<span class="sourceLineNo">620</span> * @return the current number of worker threads.<a name="line.620"></a>
-<span class="sourceLineNo">621</span> */<a name="line.621"></a>
-<span class="sourceLineNo">622</span> public int getWorkerThreadCount() {<a name="line.622"></a>
-<span class="sourceLineNo">623</span> return workerThreads.size();<a name="line.623"></a>
-<span class="sourceLineNo">624</span> }<a name="line.624"></a>
-<span class="sourceLineNo">625</span><a name="line.625"></a>
-<span class="sourceLineNo">626</span> /**<a name="line.626"></a>
-<span class="sourceLineNo">627</span> * @return the core pool size settings.<a name="line.627"></a>
-<span class="sourceLineNo">628</span> */<a name="line.628"></a>
-<span class="sourceLineNo">629</span> public int getCorePoolSize() {<a name="line.629"></a>
-<span class="sourceLineNo">630</span> return corePoolSize;<a name="line.630"></a>
-<span class="sourceLineNo">631</span> }<a name="line.631"></a>
-<span class="sourceLineNo">632</span><a name="line.632"></a>
-<span class="sourceLineNo">633</span> public int getActiveExecutorCount() {<a name="line.633"></a>
-<span class="sourceLineNo">634</span> return activeExecutorCount.get();<a name="line.634"></a>
-<span class="sourceLineNo">635</span> }<a name="line.635"></a>
-<span class="sourceLineNo">636</span><a name="line.636"></a>
-<span class="sourceLineNo">637</span> public TEnvironment getEnvironment() {<a name="line.637"></a>
-<span class="sourceLineNo">638</span> return this.environment;<a name="line.638"></a>
-<span class="sourceLineNo">639</span> }<a name="line.639"></a>
-<span class="sourceLineNo">640</span><a name="line.640"></a>
-<span class="sourceLineNo">641</span> public ProcedureStore getStore() {<a name="line.641"></a>
-<span class="sourceLineNo">642</span> return this.store;<a name="line.642"></a>
+<span class="sourceLineNo">597</span> // stop the timeout executor<a name="line.597"></a>
+<span class="sourceLineNo">598</span> timeoutExecutor.awaitTermination();<a name="line.598"></a>
+<span class="sourceLineNo">599</span><a name="line.599"></a>
+<span class="sourceLineNo">600</span> // stop the worker threads<a name="line.600"></a>
+<span class="sourceLineNo">601</span> for (WorkerThread worker: workerThreads) {<a name="line.601"></a>
+<span class="sourceLineNo">602</span> worker.awaitTermination();<a name="line.602"></a>
+<span class="sourceLineNo">603</span> }<a name="line.603"></a>
+<span class="sourceLineNo">604</span><a name="line.604"></a>
+<span class="sourceLineNo">605</span> // Destroy the Thread Group for the executors<a name="line.605"></a>
+<span class="sourceLineNo">606</span> // TODO: Fix. #join is not place to destroy resources.<a name="line.606"></a>
+<span class="sourceLineNo">607</span> try {<a name="line.607"></a>
+<span class="sourceLineNo">608</span> threadGroup.destroy();<a name="line.608"></a>
+<span class="sourceLineNo">609</span> } catch (IllegalThreadStateException e) {<a name="line.609"></a>
+<span class="sourceLineNo">610</span> LOG.error("ThreadGroup {} contains running threads; {}: See STDOUT",<a name="line.610"></a>
+<span class="sourceLineNo">611</span> this.threadGroup, e.getMessage());<a name="line.611"></a>
+<span class="sourceLineNo">612</span> // This dumps list of threads on STDOUT.<a name="line.612"></a>
+<span class="sourceLineNo">613</span> this.threadGroup.list();<a name="line.613"></a>
+<span class="sourceLineNo">614</span> }<a name="line.614"></a>
+<span class="sourceLineNo">615</span><a name="line.615"></a>
+<span class="sourceLineNo">616</span> // reset the in-memory state for testing<a name="line.616"></a>
+<span class="sourceLineNo">617</span> completed.clear();<a name="line.617"></a>
+<span class="sourceLineNo">618</span> rollbackStack.clear();<a name="line.618"></a>
+<span class="sourceLineNo">619</span> procedures.clear();<a name="line.619"></a>
+<span class="sourceLineNo">620</span> nonceKeysToProcIdsMap.clear();<a name="line.620"></a>
+<span class="sourceLineNo">621</span> scheduler.clear();<a name="line.621"></a>
+<span class="sourceLineNo">622</span> lastProcId.set(-1);<a name="line.622"></a>
+<span class="sourceLineNo">623</span> }<a name="line.623"></a>
+<span class="sourceLineNo">624</span><a name="line.624"></a>
+<span class="sourceLineNo">625</span> public void refreshConfiguration(final Configuration conf) {<a name="line.625"></a>
+<span class="sourceLineNo">626</span> this.conf = conf;<a name="line.626"></a>
+<span class="sourceLineNo">627</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.627"></a>
+<span class="sourceLineNo">628</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.628"></a>
+<span class="sourceLineNo">629</span> }<a name="line.629"></a>
+<span class="sourceLineNo">630</span><a name="line.630"></a>
+<span class="sourceLineNo">631</span> // ==========================================================================<a name="line.631"></a>
+<span class="sourceLineNo">632</span> // Accessors<a name="line.632"></a>
+<span class="sourceLineNo">633</span> // ==========================================================================<a name="line.633"></a>
+<span class="sourceLineNo">634</span> public boolean isRunning() {<a name="line.634"></a>
+<span class="sourceLineNo">635</span> return running.get();<a name="line.635"></a>
+<span class="sourceLineNo">636</span> }<a name="line.636"></a>
+<span class="sourceLineNo">637</span><a name="line.637"></a>
+<span class="sourceLineNo">638</span> /**<a name="line.638"></a>
+<span class="sourceLineNo">639</span> * @return the current number of worker threads.<a name="line.639"></a>
+<span class="sourceLineNo">640</span> */<a name="line.640"></a>
+<span class="sourceLineNo">641</span> public int getWorkerThreadCount() {<a name="line.641"></a>
+<span class="sourceLineNo">642</span> return workerThreads.size();<a name="line.642"></a>
<span class="sourceLineNo">643</span> }<a name="line.643"></a>
<span class="sourceLineNo">644</span><a name="line.644"></a>
-<span class="sourceLineNo">645</span> ProcedureScheduler getScheduler() {<a name="line.645"></a>
-<span class="sourceLineNo">646</span> return scheduler;<a name="line.646"></a>
-<span class="sourceLineNo">647</span> }<a name="line.647"></a>
-<span class="sourceLineNo">648</span><a name="line.648"></a>
-<span class="sourceLineNo">649</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.649"></a>
-<span class="sourceLineNo">650</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.650"></a>
-<span class="sourceLineNo">651</span> this.scheduler.signalAll();<a name="line.651"></a>
-<span class="sourceLineNo">652</span> }<a name="line.652"></a>
-<span class="sourceLineNo">653</span><a name="line.653"></a>
-<span class="sourceLineNo">654</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.654"></a>
-<span class="sourceLineNo">655</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.655"></a>
-<span class="sourceLineNo">656</span> }<a name="line.656"></a>
-<span class="sourceLineNo">657</span><a name="line.657"></a>
-<span class="sourceLineNo">658</span> // ==========================================================================<a name="line.658"></a>
-<span class="sourceLineNo">659</span> // Submit/Remove Chores<a name="line.659"></a>
-<span class="sourceLineNo">660</span> // ==========================================================================<a name="line.660"></a>
-<span class="sourceLineNo">661</span><a name="line.661"></a>
-<span class="sourceLineNo">662</span> /**<a name="line.662"></a>
-<span class="sourceLineNo">663</span> * Add a chore procedure to the executor<a name="line.663"></a>
-<span class="sourceLineNo">664</span> * @param chore the chore to add<a name="line.664"></a>
-<span class="sourceLineNo">665</span> */<a name="line.665"></a>
-<span class="sourceLineNo">666</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.666"></a>
-<span class="sourceLineNo">667</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.667"></a>
-<span class="sourceLineNo">668</span> timeoutExecutor.add(chore);<a name="line.668"></a>
-<span class="sourceLineNo">669</span> }<a name="line.669"></a>
-<span class="sourceLineNo">670</span><a name="line.670"></a>
-<span class="sourceLineNo">671</span> /**<a name="line.671"></a>
-<span class="sourceLineNo">672</span> * Remove a chore procedure from the executor<a name="line.672"></a>
-<span class="sourceLineNo">673</span> * @param chore the chore to remove<a name="line.673"></a>
-<span class="sourceLineNo">674</span> * @return whether the chore is removed, or it will be removed later<a name="line.674"></a>
-<span class="sourceLineNo">675</span> */<a name="line.675"></a>
-<span class="sourceLineNo">676</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.676"></a>
-<span class="sourceLineNo">677</span> chore.setState(ProcedureState.SUCCESS);<a name="line.677"></a>
-<span class="sourceLineNo">678</span> return timeoutExecutor.remove(chore);<a name="line.678"></a>
-<span class="sourceLineNo">679</span> }<a name="line.679"></a>
+<span class="sourceLineNo">645</span> /**<a name="line.645"></a>
+<span class="sourceLineNo">646</span> * @return the core pool size settings.<a name="line.646"></a>
+<span class="sourceLineNo">647</span> */<a name="line.647"></a>
+<span class="sourceLineNo">648</span> public int getCorePoolSize() {<a name="line.648"></a>
+<span class="sourceLineNo">649</span> return corePoolSize;<a name="line.649"></a>
+<span class="sourceLineNo">650</span> }<a name="line.650"></a>
+<span class="sourceLineNo">651</span><a name="line.651"></a>
+<span class="sourceLineNo">652</span> public int getActiveExecutorCount() {<a name="line.652"></a>
+<span class="sourceLineNo">653</span> return activeExecutorCount.get();<a name="line.653"></a>
+<span class="sourceLineNo">654</span> }<a name="line.654"></a>
+<span class="sourceLineNo">655</span><a name="line.655"></a>
+<span class="sourceLineNo">656</span> public TEnvironment getEnvironment() {<a name="line.656"></a>
+<span class="sourceLineNo">657</span> return this.environment;<a name="line.657"></a>
+<span class="sourceLineNo">658</span> }<a name="line.658"></a>
+<span class="sourceLineNo">659</span><a name="line.659"></a>
+<span class="sourceLineNo">660</span> public ProcedureStore getStore() {<a name="line.660"></a>
+<span class="sourceLineNo">661</span> return this.store;<a name="line.661"></a>
+<span class="sourceLineNo">662</span> }<a name="line.662"></a>
+<span class="sourceLineNo">663</span><a name="line.663"></a>
+<span class="sourceLineNo">664</span> ProcedureScheduler getScheduler() {<a name="line.664"></a>
+<span class="sourceLineNo">665</span> return scheduler;<a name="line.665"></a>
+<span class="sourceLineNo">666</span> }<a name="line.666"></a>
+<span class="sourceLineNo">667</span><a name="line.667"></a>
+<span class="sourceLineNo">668</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.668"></a>
+<span class="sourceLineNo">669</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.669"></a>
+<span class="sourceLineNo">670</span> this.scheduler.signalAll();<a name="line.670"></a>
+<span class="sourceLineNo">671</span> }<a name="line.671"></a>
+<span class="sourceLineNo">672</span><a name="line.672"></a>
+<span class="sourceLineNo">673</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.673"></a>
+<span class="sourceLineNo">674</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.674"></a>
+<span class="sourceLineNo">675</span> }<a name="line.675"></a>
+<span class="sourceLineNo">676</span><a name="line.676"></a>
+<span class="sourceLineNo">677</span> // ==========================================================================<a name="line.677"></a>
+<span class="sourceLineNo">678</span> // Submit/Remove Chores<a name="line.678"></a>
+<span class="sourceLineNo">679</span> // ==========================================================================<a name="line.679"></a>
<span class="sourceLineNo">680</span><a name="line.680"></a>
-<span class="sourceLineNo">681</span> // ==========================================================================<a name="line.681"></a>
-<span class="sourceLineNo">682</span> // Nonce Procedure helpers<a name="line.682"></a>
-<span class="sourceLineNo">683</span> // ==========================================================================<a name="line.683"></a>
-<span class="sourceLineNo">684</span> /**<a name="line.684"></a>
-<span class="sourceLineNo">685</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.685"></a>
-<span class="sourceLineNo">686</span> * @param nonceGroup<a name="line.686"></a>
-<span class="sourceLineNo">687</span> * @param nonce<a name="line.687"></a>
-<span class="sourceLineNo">688</span> * @return the generated NonceKey<a name="line.688"></a>
-<span class="sourceLineNo">689</span> */<a name="line.689"></a>
-<span class="sourceLineNo">690</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.690"></a>
-<span class="sourceLineNo">691</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.691"></a>
-<span class="sourceLineNo">692</span> }<a name="line.692"></a>
-<span class="sourceLineNo">693</span><a name="line.693"></a>
-<span class="sourceLineNo">694</span> /**<a name="line.694"></a>
-<span class="sourceLineNo">695</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.695"></a>
-<span class="sourceLineNo">696</span> * A procId will be reserved and on submitProcedure(),<a name="line.696"></a>
-<span class="sourceLineNo">697</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.697"></a>
-<span class="sourceLineNo">698</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.698"></a>
-<span class="sourceLineNo">699</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.699"></a>
-<span class="sourceLineNo">700</span> * and submit the procedure.<a name="line.700"></a>
-<span class="sourceLineNo">701</span> *<a name="line.701"></a>
-<span class="sourceLineNo">702</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.702"></a>
-<span class="sourceLineNo">703</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.703"></a>
-<span class="sourceLineNo">704</span> */<a name="line.704"></a>
-<span class="sourceLineNo">705</span> public long registerNonce(final NonceKey nonceKey) {<a name="line.705"></a>
-<span class="sourceLineNo">706</span> if (nonceKey == null) return -1;<a name="line.706"></a>
-<span class="sourceLineNo">707</span><a name="line.707"></a>
-<span class="sourceLineNo">708</span> // check if we have already a Reserved ID for the nonce<a name="line.708"></a>
-<span class="sourceLineNo">709</span> Long oldProcId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.709"></a>
-<span class="sourceLineNo">710</span> if (oldProcId == null) {<a name="line.710"></a>
-<span class="sourceLineNo">711</span> // reserve a new Procedure ID, this will be associated with the nonce<a name="line.711"></a>
-<span class="sourceLineNo">712</span> // and the procedure submitted with the specified nonce will use this ID.<a name="line.712"></a>
-<span class="sourceLineNo">713</span> final long newProcId = nextProcId();<a name="line.713"></a>
-<span class="sourceLineNo">714</span> oldProcId = nonceKeysToProcIdsMap.putIfAbsent(nonceKey, newProcId);<a name="line.714"></a>
-<span class="sourceLineNo">715</span> if (oldProcId == null) return -1;<a name="line.715"></a>
-<span class="sourceLineNo">716</span> }<a name="line.716"></a>
-<span class="sourceLineNo">717</span><a name="line.717"></a>
-<span class="sourceLineNo">718</span> // we found a registered nonce, but the procedure may not have been submitted yet.<a name="line.718"></a>
-<span class="sourceLineNo">719</span> // since the client expect the procedure to be submitted, spin here until it is.<a name="line.719"></a>
-<span class="sourceLineNo">720</span> final boolean traceEnabled = LOG.isTraceEnabled();<a name="line.720"></a>
-<span class="sourceLineNo">721</span> while (isRunning() &&<a name="line.721"></a>
-<span class="sourceLineNo">722</span> !(procedures.containsKey(oldProcId) || completed.containsKey(oldProcId)) &&<a name="line.722"></a>
-<span class="sourceLineNo">723</span> nonceKeysToProcIdsMap.containsKey(nonceKey)) {<a name="line.723"></a>
-<span class="sourceLineNo">724</span> if (traceEnabled) {<a name="line.724"></a>
-<span class="sourceLineNo">725</span> LOG.trace("Waiting for pid=" + oldProcId.longValue() + " to be submitted");<a name="line.725"></a>
-<span class="sourceLineNo">726</span> }<a name="line.726"></a>
-<span class="sourceLineNo">727</span> Threads.sleep(100);<a name="line.727"></a>
-<span class="sourceLineNo">728</span> }<a name="line.728"></a>
-<span class="sourceLineNo">729</span> return oldProcId.longValue();<a name="line.729"></a>
-<span class="sourceLineNo">730</span> }<a name="line.730"></a>
-<span class="sourceLineNo">731</span><a name="line.731"></a>
-<span class="sourceLineNo">732</span> /**<a name="line.732"></a>
-<span class="sourceLineNo">733</span> * Remove the NonceKey if the procedure was not submitted to the executor.<a name="line.733"></a>
-<span class="sourceLineNo">734</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.734"></a>
-<span class="sourceLineNo">735</span> */<a name="line.735"></a>
-<span class="sourceLineNo">736</span> public void unregisterNonceIfProcedureWasNotSubmitted(final NonceKey nonceKey) {<a name="line.736"></a>
-<span class="sourceLineNo">737</span> if (nonceKey == null) return;<a name="line.737"></a>
-<span class="sourceLineNo">738</span><a name="line.738"></a>
-<span class="sourceLineNo">739</span> final Long procId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.739"></a>
-<span class="sourceLineNo">740</span> if (procId == null) return;<a name="line.740"></a>
-<span class="sourceLineNo">741</span><a name="line.741"></a>
-<span class="sourceLineNo">742</span> // if the procedure was not submitted, remove the nonce<a name="line.742"></a>
-<span class="sourceLineNo">743</span> if (!(procedures.containsKey(procId) || completed.containsKey(procId))) {<a name="line.743"></a>
-<span class="sourceLineNo">744</span> nonceKeysToProcIdsMap.remove(nonceKey);<a name="line.744"></a>
-<span class="sourceLineNo">745</span> }<a name="line.745"></a>
-<span class="sourceLineNo">746</span> }<a name="line.746"></a>
-<span class="sourceLineNo">747</span><a name="line.747"></a>
-<span class="sourceLineNo">748</span> public static class FailedProcedure<TEnvironment> extends Procedure<TEnvironment> {<a name="line.748"></a>
-<span class="sourceLineNo">749</span> private String procName;<a name="line.749"></a>
+<span class="sourceLineNo">681</span> /**<a name="line.681"></a>
+<span class="sourceLineNo">682</span> * Add a chore procedure to the executor<a name="line.682"></a>
+<span class="sourceLineNo">683</span> * @param chore the chore to add<a name="line.683"></a>
+<span class="sourceLineNo">684</span> */<a name="line.684"></a>
+<span class="sourceLineNo">685</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.685"></a>
+<span class="sourceLineNo">686</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.686"></a>
+<span class="sourceLineNo">687</span> timeoutExecutor.add(chore);<a name="line.687"></a>
+<span class="sourceLineNo">688</span> }<a name="line.688"></a>
+<span class="sourceLineNo">689</span><a name="line.689"></a>
+<span class="sourceLineNo">690</span> /**<a name="line.690"></a>
+<span class="sourceLineNo">691</span> * Remove a chore procedure from the executor<a name="line.691"></a>
+<span class="sourceLineNo">692</span> * @param chore the chore to remove<a name="line.692"></a>
+<span class="sourceLineNo">693</span> * @return whether the chore is removed, or it will be removed later<a name="line.693"></a>
+<span class="sourceLineNo">694</span> */<a name="line.694"></a>
+<span class="sourceLineNo">695</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.695"></a>
+<span class="sourceLineNo">696</span> chore.setState(ProcedureState.SUCCESS);<a name="line.696"></a>
+<span class="sourceLineNo">697</span> return timeoutExecutor.remove(chore);<a name="line.697"></a>
+<span class="sourceLineNo">698</span> }<a name="line.698"></a>
+<span class="sourceLineNo">699</span><a name="line.699"></a>
+<span class="sourceLineNo">700</span> // ==========================================================================<a name="line.700"></a>
+<span class="sourceLineNo">701</span> // Nonce Procedure helpers<a name="line.701"></a>
+<span class="sourceLineNo">702</span> // ==========================================================================<a name="line.702"></a>
+<span class="sourceLineNo">703</span> /**<a name="line.703"></a>
+<span class="sourceLineNo">704</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.704"></a>
+<span class="sourceLineNo">705</span> * @param nonceGroup<a name="line.705"></a>
+<span class="sourceLineNo">706</span> * @param nonce<a name="line.706"></a>
+<span class="sourceLineNo">707</span> * @return the generated NonceKey<a name="line.707"></a>
+<span class="sourceLineNo">708</span> */<a name="line.708"></a>
+<span class="sourceLineNo">709</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.709"></a>
+<span class="sourceLineNo">710</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.710"></a>
+<span class="sourceLineNo">711</span> }<a name="line.711"></a>
+<span class="sourceLineNo">712</span><a name="line.712"></a>
+<span class="sourceLineNo">713</span> /**<a name="line.713"></a>
+<span class="sourceLineNo">714</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.714"></a>
+<span class="sourceLineNo">715</span> * A procId will be reserved and on submitProcedure(),<a name="line.715"></a>
+<span class="sourceLineNo">716</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.716"></a>
+<span class="sourceLineNo">717</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.717"></a>
+<span class="sourceLineNo">718</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.718"></a>
+<span class="sourceLineNo">719</span> * and submit the procedure.<a name="line.719"></a>
+<span class="sourceLineNo">720</span> *<a name="line.720"></a>
+<span class="sourceLineNo">721</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.721"></a>
+<span class="sourceLineNo">722</span> * @return the procId associated with
<TRUNCATED>
[14/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html
----------------------------------------------------------------------
diff --git a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html
index e2b1e92..a9d814d 100644
--- a/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html
+++ b/devapidocs/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html
@@ -287,11 +287,15 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
</tr>
<tr class="rowColor">
<td class="colFirst"><code>private <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadGroup.html?is-external=true" title="class or interface in java.lang">ThreadGroup</a></code></td>
-<td class="colLast"><code><span class="memberNameLink"><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#threadGroup">threadGroup</a></span></code> </td>
+<td class="colLast"><code><span class="memberNameLink"><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#threadGroup">threadGroup</a></span></code>
+<div class="block">Created in the <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>start(int, boolean)</code></a> method.</div>
+</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/TimeoutExecutorThread.html" title="class in org.apache.hadoop.hbase.procedure2">TimeoutExecutorThread</a></code></td>
-<td class="colLast"><code><span class="memberNameLink"><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#timeoutExecutor">timeoutExecutor</a></span></code> </td>
+<td class="colLast"><code><span class="memberNameLink"><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#timeoutExecutor">timeoutExecutor</a></span></code>
+<div class="block">Created in the <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>start(int, boolean)</code></a> method.</div>
+</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>static <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a></code></td>
@@ -303,7 +307,9 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
</tr>
<tr class="rowColor">
<td class="colFirst"><code>private <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CopyOnWriteArrayList.html?is-external=true" title="class or interface in java.util.concurrent">CopyOnWriteArrayList</a><<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.WorkerThread</a>></code></td>
-<td class="colLast"><code><span class="memberNameLink"><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#workerThreads">workerThreads</a></span></code> </td>
+<td class="colLast"><code><span class="memberNameLink"><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#workerThreads">workerThreads</a></span></code>
+<div class="block">Created in the <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>start(int, boolean)</code></a> method.</div>
+</td>
</tr>
</table>
</li>
@@ -807,7 +813,11 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>threadGroup</h4>
-<pre>private <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadGroup.html?is-external=true" title="class or interface in java.lang">ThreadGroup</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.267">threadGroup</a></pre>
+<pre>private <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadGroup.html?is-external=true" title="class or interface in java.lang">ThreadGroup</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.274">threadGroup</a></pre>
+<div class="block">Created in the <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>start(int, boolean)</code></a> method. Destroyed in <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#join--"><code>join()</code></a> (FIX! Doing
+ resource handling rather than observing in a #join is unexpected).
+ Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery
+ (Should be ok).</div>
</li>
</ul>
<a name="workerThreads">
@@ -816,7 +826,11 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>workerThreads</h4>
-<pre>private <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CopyOnWriteArrayList.html?is-external=true" title="class or interface in java.util.concurrent">CopyOnWriteArrayList</a><<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.WorkerThread</a>> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.268">workerThreads</a></pre>
+<pre>private <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CopyOnWriteArrayList.html?is-external=true" title="class or interface in java.util.concurrent">CopyOnWriteArrayList</a><<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.WorkerThread</a>> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.282">workerThreads</a></pre>
+<div class="block">Created in the <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>start(int, boolean)</code></a> method. Terminated in <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#join--"><code>join()</code></a> (FIX! Doing
+ resource handling rather than observing in a #join is unexpected).
+ Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery
+ (Should be ok).</div>
</li>
</ul>
<a name="timeoutExecutor">
@@ -825,7 +839,11 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>timeoutExecutor</h4>
-<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/TimeoutExecutorThread.html" title="class in org.apache.hadoop.hbase.procedure2">TimeoutExecutorThread</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.269">timeoutExecutor</a></pre>
+<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/TimeoutExecutorThread.html" title="class in org.apache.hadoop.hbase.procedure2">TimeoutExecutorThread</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.290">timeoutExecutor</a></pre>
+<div class="block">Created in the <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>start(int, boolean)</code></a> method. Terminated in <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#join--"><code>join()</code></a> (FIX! Doing
+ resource handling rather than observing in a #join is unexpected).
+ Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery
+ (Should be ok).</div>
</li>
</ul>
<a name="corePoolSize">
@@ -834,7 +852,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>corePoolSize</h4>
-<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.270">corePoolSize</a></pre>
+<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.292">corePoolSize</a></pre>
</li>
</ul>
<a name="maxPoolSize">
@@ -843,7 +861,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>maxPoolSize</h4>
-<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.271">maxPoolSize</a></pre>
+<pre>private int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.293">maxPoolSize</a></pre>
</li>
</ul>
<a name="keepAliveTime">
@@ -852,7 +870,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>keepAliveTime</h4>
-<pre>private volatile long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.273">keepAliveTime</a></pre>
+<pre>private volatile long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.295">keepAliveTime</a></pre>
</li>
</ul>
<a name="scheduler">
@@ -861,7 +879,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>scheduler</h4>
-<pre>private final <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureScheduler.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureScheduler</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.278">scheduler</a></pre>
+<pre>private final <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureScheduler.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureScheduler</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.300">scheduler</a></pre>
<div class="block">Scheduler/Queue that contains runnable procedures.</div>
</li>
</ul>
@@ -871,7 +889,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>lastProcId</h4>
-<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicLong.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicLong</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.280">lastProcId</a></pre>
+<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicLong.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicLong</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.302">lastProcId</a></pre>
</li>
</ul>
<a name="workerId">
@@ -880,7 +898,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>workerId</h4>
-<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicLong.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicLong</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.281">workerId</a></pre>
+<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicLong.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicLong</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.303">workerId</a></pre>
</li>
</ul>
<a name="activeExecutorCount">
@@ -889,7 +907,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>activeExecutorCount</h4>
-<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicInteger.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicInteger</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.282">activeExecutorCount</a></pre>
+<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicInteger.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicInteger</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.304">activeExecutorCount</a></pre>
</li>
</ul>
<a name="running">
@@ -898,7 +916,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>running</h4>
-<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicBoolean.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicBoolean</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.283">running</a></pre>
+<pre>private final <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/AtomicBoolean.html?is-external=true" title="class or interface in java.util.concurrent.atomic">AtomicBoolean</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.305">running</a></pre>
</li>
</ul>
<a name="environment">
@@ -907,7 +925,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>environment</h4>
-<pre>private final <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="type parameter in ProcedureExecutor">TEnvironment</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.284">environment</a></pre>
+<pre>private final <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="type parameter in ProcedureExecutor">TEnvironment</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.306">environment</a></pre>
</li>
</ul>
<a name="store">
@@ -916,7 +934,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>store</h4>
-<pre>private final <a href="../../../../../org/apache/hadoop/hbase/procedure2/store/ProcedureStore.html" title="interface in org.apache.hadoop.hbase.procedure2.store">ProcedureStore</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.285">store</a></pre>
+<pre>private final <a href="../../../../../org/apache/hadoop/hbase/procedure2/store/ProcedureStore.html" title="interface in org.apache.hadoop.hbase.procedure2.store">ProcedureStore</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.307">store</a></pre>
</li>
</ul>
<a name="checkOwnerSet">
@@ -925,7 +943,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockListLast">
<li class="blockList">
<h4>checkOwnerSet</h4>
-<pre>private final boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.287">checkOwnerSet</a></pre>
+<pre>private final boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.309">checkOwnerSet</a></pre>
</li>
</ul>
</li>
@@ -944,7 +962,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>ProcedureExecutor</h4>
-<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.289">ProcedureExecutor</a>(org.apache.hadoop.conf.Configuration conf,
+<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.311">ProcedureExecutor</a>(org.apache.hadoop.conf.Configuration conf,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="type parameter in ProcedureExecutor">TEnvironment</a> environment,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/store/ProcedureStore.html" title="interface in org.apache.hadoop.hbase.procedure2.store">ProcedureStore</a> store)</pre>
</li>
@@ -957,7 +975,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockListLast">
<li class="blockList">
<h4>ProcedureExecutor</h4>
-<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.294">ProcedureExecutor</a>(org.apache.hadoop.conf.Configuration conf,
+<pre>public <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.316">ProcedureExecutor</a>(org.apache.hadoop.conf.Configuration conf,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="type parameter in ProcedureExecutor">TEnvironment</a> environment,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/store/ProcedureStore.html" title="interface in org.apache.hadoop.hbase.procedure2.store">ProcedureStore</a> store,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureScheduler.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureScheduler</a> scheduler)</pre>
@@ -977,7 +995,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>load</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.304">load</a>(boolean abortOnCorruption)
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.327">load</a>(boolean abortOnCorruption)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a></pre>
<dl>
<dt><span class="throwsLabel">Throws:</span></dt>
@@ -991,7 +1009,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>loadProcedures</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.337">loadProcedures</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/store/ProcedureStore.ProcedureIterator.html" title="interface in org.apache.hadoop.hbase.procedure2.store">ProcedureStore.ProcedureIterator</a> procIter,
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.360">loadProcedures</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/store/ProcedureStore.ProcedureIterator.html" title="interface in org.apache.hadoop.hbase.procedure2.store">ProcedureStore.ProcedureIterator</a> procIter,
boolean abortOnCorruption)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a></pre>
<dl>
@@ -1006,7 +1024,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>start</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.500">start</a>(int numThreads,
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.523">start</a>(int numThreads,
boolean abortOnCorruption)
throws <a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a></pre>
<div class="block">Start the procedure executor.
@@ -1029,7 +1047,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>stop</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.563">stop</a>()</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.583">stop</a>()</pre>
</li>
</ul>
<a name="join--">
@@ -1038,7 +1056,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>join</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.574">join</a>()</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.594">join</a>()</pre>
</li>
</ul>
<a name="refreshConfiguration-org.apache.hadoop.conf.Configuration-">
@@ -1047,7 +1065,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>refreshConfiguration</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.606">refreshConfiguration</a>(org.apache.hadoop.conf.Configuration conf)</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.625">refreshConfiguration</a>(org.apache.hadoop.conf.Configuration conf)</pre>
</li>
</ul>
<a name="isRunning--">
@@ -1056,7 +1074,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>isRunning</h4>
-<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.615">isRunning</a>()</pre>
+<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.634">isRunning</a>()</pre>
</li>
</ul>
<a name="getWorkerThreadCount--">
@@ -1065,7 +1083,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getWorkerThreadCount</h4>
-<pre>public int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.622">getWorkerThreadCount</a>()</pre>
+<pre>public int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.641">getWorkerThreadCount</a>()</pre>
<dl>
<dt><span class="returnLabel">Returns:</span></dt>
<dd>the current number of worker threads.</dd>
@@ -1078,7 +1096,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getCorePoolSize</h4>
-<pre>public int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.629">getCorePoolSize</a>()</pre>
+<pre>public int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.648">getCorePoolSize</a>()</pre>
<dl>
<dt><span class="returnLabel">Returns:</span></dt>
<dd>the core pool size settings.</dd>
@@ -1091,7 +1109,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getActiveExecutorCount</h4>
-<pre>public int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.633">getActiveExecutorCount</a>()</pre>
+<pre>public int <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.652">getActiveExecutorCount</a>()</pre>
</li>
</ul>
<a name="getEnvironment--">
@@ -1100,7 +1118,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getEnvironment</h4>
-<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="type parameter in ProcedureExecutor">TEnvironment</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.637">getEnvironment</a>()</pre>
+<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="type parameter in ProcedureExecutor">TEnvironment</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.656">getEnvironment</a>()</pre>
</li>
</ul>
<a name="getStore--">
@@ -1109,7 +1127,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getStore</h4>
-<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/store/ProcedureStore.html" title="interface in org.apache.hadoop.hbase.procedure2.store">ProcedureStore</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.641">getStore</a>()</pre>
+<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/store/ProcedureStore.html" title="interface in org.apache.hadoop.hbase.procedure2.store">ProcedureStore</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.660">getStore</a>()</pre>
</li>
</ul>
<a name="getScheduler--">
@@ -1118,7 +1136,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getScheduler</h4>
-<pre><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureScheduler.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureScheduler</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.645">getScheduler</a>()</pre>
+<pre><a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureScheduler.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureScheduler</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.664">getScheduler</a>()</pre>
</li>
</ul>
<a name="setKeepAliveTime-long-java.util.concurrent.TimeUnit-">
@@ -1127,7 +1145,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>setKeepAliveTime</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.649">setKeepAliveTime</a>(long keepAliveTime,
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.668">setKeepAliveTime</a>(long keepAliveTime,
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/TimeUnit.html?is-external=true" title="class or interface in java.util.concurrent">TimeUnit</a> timeUnit)</pre>
</li>
</ul>
@@ -1137,7 +1155,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getKeepAliveTime</h4>
-<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.654">getKeepAliveTime</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/TimeUnit.html?is-external=true" title="class or interface in java.util.concurrent">TimeUnit</a> timeUnit)</pre>
+<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.673">getKeepAliveTime</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/TimeUnit.html?is-external=true" title="class or interface in java.util.concurrent">TimeUnit</a> timeUnit)</pre>
</li>
</ul>
<a name="addChore-org.apache.hadoop.hbase.procedure2.ProcedureInMemoryChore-">
@@ -1146,7 +1164,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>addChore</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.666">addChore</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureInMemoryChore.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureInMemoryChore</a> chore)</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.685">addChore</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureInMemoryChore.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureInMemoryChore</a> chore)</pre>
<div class="block">Add a chore procedure to the executor</div>
<dl>
<dt><span class="paramLabel">Parameters:</span></dt>
@@ -1160,7 +1178,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>removeChore</h4>
-<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.676">removeChore</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureInMemoryChore.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureInMemoryChore</a> chore)</pre>
+<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.695">removeChore</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureInMemoryChore.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureInMemoryChore</a> chore)</pre>
<div class="block">Remove a chore procedure from the executor</div>
<dl>
<dt><span class="paramLabel">Parameters:</span></dt>
@@ -1176,7 +1194,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>createNonceKey</h4>
-<pre>public <a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.690">createNonceKey</a>(long nonceGroup,
+<pre>public <a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.709">createNonceKey</a>(long nonceGroup,
long nonce)</pre>
<div class="block">Create a NoneKey from the specified nonceGroup and nonce.</div>
<dl>
@@ -1194,7 +1212,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>registerNonce</h4>
-<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.705">registerNonce</a>(<a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> nonceKey)</pre>
+<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.724">registerNonce</a>(<a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> nonceKey)</pre>
<div class="block">Register a nonce for a procedure that is going to be submitted.
A procId will be reserved and on submitProcedure(),
the procedure with the specified nonce will take the reserved ProcId.
@@ -1215,7 +1233,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>unregisterNonceIfProcedureWasNotSubmitted</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.736">unregisterNonceIfProcedureWasNotSubmitted</a>(<a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> nonceKey)</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.755">unregisterNonceIfProcedureWasNotSubmitted</a>(<a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> nonceKey)</pre>
<div class="block">Remove the NonceKey if the procedure was not submitted to the executor.</div>
<dl>
<dt><span class="paramLabel">Parameters:</span></dt>
@@ -1229,7 +1247,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>setFailureResultForNonce</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.810">setFailureResultForNonce</a>(<a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> nonceKey,
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.829">setFailureResultForNonce</a>(<a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> nonceKey,
<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/String.html?is-external=true" title="class or interface in java.lang">String</a> procName,
<a href="../../../../../org/apache/hadoop/hbase/security/User.html" title="class in org.apache.hadoop.hbase.security">User</a> procOwner,
<a href="https://docs.oracle.com/javase/8/docs/api/java/io/IOException.html?is-external=true" title="class or interface in java.io">IOException</a> exception)</pre>
@@ -1250,7 +1268,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>submitProcedure</h4>
-<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.831">submitProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
+<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.850">submitProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
<div class="block">Add a new root-procedure to the executor.</div>
<dl>
<dt><span class="paramLabel">Parameters:</span></dt>
@@ -1266,7 +1284,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>submitProcedure</h4>
-<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.843">submitProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc,
+<pre>public long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.862">submitProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc,
<a href="../../../../../org/apache/hadoop/hbase/util/NonceKey.html" title="class in org.apache.hadoop.hbase.util">NonceKey</a> nonceKey)</pre>
<div class="block">Add a new root-procedure to the executor.</div>
<dl>
@@ -1284,7 +1302,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>submitProcedures</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.877">submitProcedures</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>[] procs)</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.896">submitProcedures</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>[] procs)</pre>
<div class="block">Add a set of new root-procedure to the executor.</div>
<dl>
<dt><span class="paramLabel">Parameters:</span></dt>
@@ -1298,7 +1316,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>prepareProcedure</h4>
-<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.901">prepareProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
+<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.920">prepareProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
</li>
</ul>
<a name="pushProcedure-org.apache.hadoop.hbase.procedure2.Procedure-">
@@ -1307,7 +1325,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>pushProcedure</h4>
-<pre>private long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.911">pushProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
+<pre>private long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.930">pushProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
</li>
</ul>
<a name="abort-long-">
@@ -1316,7 +1334,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>abort</h4>
-<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.935">abort</a>(long procId)</pre>
+<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.954">abort</a>(long procId)</pre>
<div class="block">Send an abort notification the specified procedure.
Depending on the procedure implementation the abort can be considered or ignored.</div>
<dl>
@@ -1333,7 +1351,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>abort</h4>
-<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.946">abort</a>(long procId,
+<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.965">abort</a>(long procId,
boolean mayInterruptIfRunning)</pre>
<div class="block">Send an abort notification to the specified procedure.
Depending on the procedure implementation, the abort can be considered or ignored.</div>
@@ -1352,7 +1370,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getProcedure</h4>
-<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.960">getProcedure</a>(long procId)</pre>
+<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.979">getProcedure</a>(long procId)</pre>
</li>
</ul>
<a name="getProcedure-java.lang.Class-long-">
@@ -1361,7 +1379,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getProcedure</h4>
-<pre>public <T extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>> T <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.964">getProcedure</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html?is-external=true" title="class or interface in java.lang">Class</a><T> clazz,
+<pre>public <T extends <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>> T <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.983">getProcedure</a>(<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html?is-external=true" title="class or interface in java.lang">Class</a><T> clazz,
long procId)</pre>
</li>
</ul>
@@ -1371,7 +1389,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getResult</h4>
-<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.972">getResult</a>(long procId)</pre>
+<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.991">getResult</a>(long procId)</pre>
</li>
</ul>
<a name="isFinished-long-">
@@ -1380,7 +1398,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>isFinished</h4>
-<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.988">isFinished</a>(long procId)</pre>
+<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1007">isFinished</a>(long procId)</pre>
<div class="block">Return true if the procedure is finished.
The state may be "completed successfully" or "failed and rolledback".
Use getResult() to check the state or get the result data.</div>
@@ -1398,7 +1416,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>isStarted</h4>
-<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.997">isStarted</a>(long procId)</pre>
+<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1016">isStarted</a>(long procId)</pre>
<div class="block">Return true if the procedure is started.</div>
<dl>
<dt><span class="paramLabel">Parameters:</span></dt>
@@ -1414,7 +1432,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>removeResult</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1009">removeResult</a>(long procId)</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1028">removeResult</a>(long procId)</pre>
<div class="block">Mark the specified completed procedure, as ready to remove.</div>
<dl>
<dt><span class="paramLabel">Parameters:</span></dt>
@@ -1428,7 +1446,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getResultOrProcedure</h4>
-<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1023">getResultOrProcedure</a>(long procId)</pre>
+<pre>public <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1042">getResultOrProcedure</a>(long procId)</pre>
</li>
</ul>
<a name="isProcedureOwner-long-org.apache.hadoop.hbase.security.User-">
@@ -1437,7 +1455,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>isProcedureOwner</h4>
-<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1039">isProcedureOwner</a>(long procId,
+<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1058">isProcedureOwner</a>(long procId,
<a href="../../../../../org/apache/hadoop/hbase/security/User.html" title="class in org.apache.hadoop.hbase.security">User</a> user)</pre>
<div class="block">Check if the user is this procedure's owner</div>
<dl>
@@ -1456,7 +1474,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getProcedures</h4>
-<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a><?>> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1061">getProcedures</a>()</pre>
+<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/List.html?is-external=true" title="class or interface in java.util">List</a><<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a><?>> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1080">getProcedures</a>()</pre>
<div class="block">Get procedures.</div>
<dl>
<dt><span class="returnLabel">Returns:</span></dt>
@@ -1470,7 +1488,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>registerListener</h4>
-<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1079">registerListener</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.ProcedureExecutorListener.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.ProcedureExecutorListener</a> listener)</pre>
+<pre>public void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1098">registerListener</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.ProcedureExecutorListener.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.ProcedureExecutorListener</a> listener)</pre>
</li>
</ul>
<a name="unregisterListener-org.apache.hadoop.hbase.procedure2.ProcedureExecutor.ProcedureExecutorListener-">
@@ -1479,7 +1497,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>unregisterListener</h4>
-<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1083">unregisterListener</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.ProcedureExecutorListener.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.ProcedureExecutorListener</a> listener)</pre>
+<pre>public boolean <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1102">unregisterListener</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.ProcedureExecutorListener.html" title="interface in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.ProcedureExecutorListener</a> listener)</pre>
</li>
</ul>
<a name="sendProcedureLoadedNotification-long-">
@@ -1488,7 +1506,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>sendProcedureLoadedNotification</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1087">sendProcedureLoadedNotification</a>(long procId)</pre>
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1106">sendProcedureLoadedNotification</a>(long procId)</pre>
</li>
</ul>
<a name="sendProcedureAddedNotification-long-">
@@ -1497,7 +1515,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>sendProcedureAddedNotification</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1099">sendProcedureAddedNotification</a>(long procId)</pre>
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1118">sendProcedureAddedNotification</a>(long procId)</pre>
</li>
</ul>
<a name="sendProcedureFinishedNotification-long-">
@@ -1506,7 +1524,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>sendProcedureFinishedNotification</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1111">sendProcedureFinishedNotification</a>(long procId)</pre>
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1130">sendProcedureFinishedNotification</a>(long procId)</pre>
</li>
</ul>
<a name="nextProcId--">
@@ -1515,7 +1533,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>nextProcId</h4>
-<pre>private long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1126">nextProcId</a>()</pre>
+<pre>private long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1145">nextProcId</a>()</pre>
</li>
</ul>
<a name="getLastProcId--">
@@ -1524,7 +1542,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getLastProcId</h4>
-<pre>protected long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1143">getLastProcId</a>()</pre>
+<pre>protected long <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1162">getLastProcId</a>()</pre>
</li>
</ul>
<a name="getActiveProcIds--">
@@ -1533,7 +1551,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getActiveProcIds</h4>
-<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Set.html?is-external=true" title="class or interface in java.util">Set</a><<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Long.html?is-external=true" title="class or interface in java.lang">Long</a>> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1148">getActiveProcIds</a>()</pre>
+<pre>public <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Set.html?is-external=true" title="class or interface in java.util">Set</a><<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Long.html?is-external=true" title="class or interface in java.lang">Long</a>> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1167">getActiveProcIds</a>()</pre>
</li>
</ul>
<a name="getRootProcedureId-org.apache.hadoop.hbase.procedure2.Procedure-">
@@ -1542,7 +1560,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>getRootProcedureId</h4>
-<pre><a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Long.html?is-external=true" title="class or interface in java.lang">Long</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1152">getRootProcedureId</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
+<pre><a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Long.html?is-external=true" title="class or interface in java.lang">Long</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1171">getRootProcedureId</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
</li>
</ul>
<a name="executeProcedure-org.apache.hadoop.hbase.procedure2.Procedure-">
@@ -1551,7 +1569,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>executeProcedure</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1159">executeProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1178">executeProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
</li>
</ul>
<a name="acquireLock-org.apache.hadoop.hbase.procedure2.Procedure-">
@@ -1560,7 +1578,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>acquireLock</h4>
-<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.LockState.html" title="enum in org.apache.hadoop.hbase.procedure2">Procedure.LockState</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1258">acquireLock</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
+<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.LockState.html" title="enum in org.apache.hadoop.hbase.procedure2">Procedure.LockState</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1277">acquireLock</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
</li>
</ul>
<a name="releaseLock-org.apache.hadoop.hbase.procedure2.Procedure-boolean-">
@@ -1569,7 +1587,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>releaseLock</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1269">releaseLock</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc,
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1288">releaseLock</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc,
boolean force)</pre>
</li>
</ul>
@@ -1579,7 +1597,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>executeRollback</h4>
-<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.LockState.html" title="enum in org.apache.hadoop.hbase.procedure2">Procedure.LockState</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1283">executeRollback</a>(long rootProcId,
+<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.LockState.html" title="enum in org.apache.hadoop.hbase.procedure2">Procedure.LockState</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1302">executeRollback</a>(long rootProcId,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> procStack)</pre>
<div class="block">Execute the rollback of the full procedure stack.
Once the procedure is rolledback, the root-procedure will be visible as
@@ -1592,7 +1610,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>executeRollback</h4>
-<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.LockState.html" title="enum in org.apache.hadoop.hbase.procedure2">Procedure.LockState</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1351">executeRollback</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
+<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.LockState.html" title="enum in org.apache.hadoop.hbase.procedure2">Procedure.LockState</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1370">executeRollback</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
<div class="block">Execute the rollback of the procedure step.
It updates the store with the new state (stack index)
or will remove completly the procedure in case it is a child.</div>
@@ -1604,7 +1622,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>execProcedure</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1428">execProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> procStack,
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1447">execProcedure</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> procStack,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a><<a href="../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html" title="type parameter in ProcedureExecutor">TEnvironment</a>> procedure)</pre>
<div class="block">Executes <code>procedure</code>
<ul>
@@ -1641,7 +1659,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>initializeChildren</h4>
-<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>[] <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1556">initializeChildren</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> procStack,
+<pre>private <a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>[] <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1575">initializeChildren</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> procStack,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> procedure,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>[] subprocs)</pre>
</li>
@@ -1652,7 +1670,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>submitChildrenProcedures</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1592">submitChildrenProcedures</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>[] subprocs)</pre>
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1611">submitChildrenProcedures</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>[] subprocs)</pre>
</li>
</ul>
<a name="countDownChildren-org.apache.hadoop.hbase.procedure2.RootProcedureState-org.apache.hadoop.hbase.procedure2.Procedure-">
@@ -1661,7 +1679,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>countDownChildren</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1602">countDownChildren</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> procStack,
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1621">countDownChildren</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> procStack,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> procedure)</pre>
</li>
</ul>
@@ -1671,7 +1689,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>updateStoreOnExec</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1620">updateStoreOnExec</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> procStack,
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1639">updateStoreOnExec</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> procStack,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> procedure,
<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a>[] subprocs)</pre>
</li>
@@ -1682,7 +1700,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>handleInterruptedException</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1648">handleInterruptedException</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc,
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1667">handleInterruptedException</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc,
<a href="https://docs.oracle.com/javase/8/docs/api/java/lang/InterruptedException.html?is-external=true" title="class or interface in java.lang">InterruptedException</a> e)</pre>
</li>
</ul>
@@ -1692,7 +1710,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>execCompletionCleanup</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1660">execCompletionCleanup</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1679">execCompletionCleanup</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
</li>
</ul>
<a name="procedureFinished-org.apache.hadoop.hbase.procedure2.Procedure-">
@@ -1701,7 +1719,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockList">
<li class="blockList">
<h4>procedureFinished</h4>
-<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1673">procedureFinished</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
+<pre>private void <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1692">procedureFinished</a>(<a href="../../../../../org/apache/hadoop/hbase/procedure2/Procedure.html" title="class in org.apache.hadoop.hbase.procedure2">Procedure</a> proc)</pre>
</li>
</ul>
<a name="getProcStack-long-">
@@ -1710,7 +1728,7 @@ extends <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Object.html
<ul class="blockListLast">
<li class="blockList">
<h4>getProcStack</h4>
-<pre><a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1700">getProcStack</a>(long rootProcId)</pre>
+<pre><a href="../../../../../org/apache/hadoop/hbase/procedure2/RootProcedureState.html" title="class in org.apache.hadoop.hbase.procedure2">RootProcedureState</a> <a href="../../../../../src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#line.1719">getProcStack</a>(long rootProcId)</pre>
</li>
</ul>
</li>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/ProcedureExecutor.WorkerThread.html
----------------------------------------------------------------------
diff --git a/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/ProcedureExecutor.WorkerThread.html b/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/ProcedureExecutor.WorkerThread.html
index 4bc42f4..502515c 100644
--- a/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/ProcedureExecutor.WorkerThread.html
+++ b/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/ProcedureExecutor.WorkerThread.html
@@ -117,7 +117,9 @@
<tbody>
<tr class="altColor">
<td class="colFirst"><code>private <a href="https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CopyOnWriteArrayList.html?is-external=true" title="class or interface in java.util.concurrent">CopyOnWriteArrayList</a><<a href="../../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerThread.html" title="class in org.apache.hadoop.hbase.procedure2">ProcedureExecutor.WorkerThread</a>></code></td>
-<td class="colLast"><span class="typeNameLabel">ProcedureExecutor.</span><code><span class="memberNameLink"><a href="../../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#workerThreads">workerThreads</a></span></code> </td>
+<td class="colLast"><span class="typeNameLabel">ProcedureExecutor.</span><code><span class="memberNameLink"><a href="../../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#workerThreads">workerThreads</a></span></code>
+<div class="block">Created in the <a href="../../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>ProcedureExecutor.start(int, boolean)</code></a> method.</div>
+</td>
</tr>
</tbody>
</table>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/TimeoutExecutorThread.html
----------------------------------------------------------------------
diff --git a/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/TimeoutExecutorThread.html b/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/TimeoutExecutorThread.html
index c13961f..c5358c9 100644
--- a/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/TimeoutExecutorThread.html
+++ b/devapidocs/org/apache/hadoop/hbase/procedure2/class-use/TimeoutExecutorThread.html
@@ -104,7 +104,9 @@
<tbody>
<tr class="altColor">
<td class="colFirst"><code>private <a href="../../../../../../org/apache/hadoop/hbase/procedure2/TimeoutExecutorThread.html" title="class in org.apache.hadoop.hbase.procedure2">TimeoutExecutorThread</a></code></td>
-<td class="colLast"><span class="typeNameLabel">ProcedureExecutor.</span><code><span class="memberNameLink"><a href="../../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#timeoutExecutor">timeoutExecutor</a></span></code> </td>
+<td class="colLast"><span class="typeNameLabel">ProcedureExecutor.</span><code><span class="memberNameLink"><a href="../../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#timeoutExecutor">timeoutExecutor</a></span></code>
+<div class="block">Created in the <a href="../../../../../../org/apache/hadoop/hbase/procedure2/ProcedureExecutor.html#start-int-boolean-"><code>ProcedureExecutor.start(int, boolean)</code></a> method.</div>
+</td>
</tr>
</tbody>
</table>
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/Version.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/Version.html b/devapidocs/src-html/org/apache/hadoop/hbase/Version.html
index be7b664..69ab5ed 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/Version.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/Version.html
@@ -16,11 +16,11 @@
<span class="sourceLineNo">008</span>@InterfaceAudience.Private<a name="line.8"></a>
<span class="sourceLineNo">009</span>public class Version {<a name="line.9"></a>
<span class="sourceLineNo">010</span> public static final String version = "3.0.0-SNAPSHOT";<a name="line.10"></a>
-<span class="sourceLineNo">011</span> public static final String revision = "2e9b96e4f14c2802b59c6037cbfe4a427ec93bea";<a name="line.11"></a>
+<span class="sourceLineNo">011</span> public static final String revision = "4cb444e77b41cdb733544770a471068256d65bbe";<a name="line.11"></a>
<span class="sourceLineNo">012</span> public static final String user = "jenkins";<a name="line.12"></a>
-<span class="sourceLineNo">013</span> public static final String date = "Wed May 2 14:38:54 UTC 2018";<a name="line.13"></a>
+<span class="sourceLineNo">013</span> public static final String date = "Thu May 3 14:39:11 UTC 2018";<a name="line.13"></a>
<span class="sourceLineNo">014</span> public static final String url = "git://jenkins-websites1.apache.org/home/jenkins/jenkins-slave/workspace/hbase_generate_website/hbase";<a name="line.14"></a>
-<span class="sourceLineNo">015</span> public static final String srcChecksum = "300425a7629c0718634c89407e4d606a";<a name="line.15"></a>
+<span class="sourceLineNo">015</span> public static final String srcChecksum = "b65d69cf3ec7b334d9360592429adf77";<a name="line.15"></a>
<span class="sourceLineNo">016</span>}<a name="line.16"></a>
[09/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureRetainer.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureRetainer.html b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureRetainer.html
index 97aa79c..cc787e3 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureRetainer.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.CompletedProcedureRetainer.html
@@ -272,1623 +272,1642 @@
<span class="sourceLineNo">264</span> private final CopyOnWriteArrayList<ProcedureExecutorListener> listeners = new CopyOnWriteArrayList<>();<a name="line.264"></a>
<span class="sourceLineNo">265</span><a name="line.265"></a>
<span class="sourceLineNo">266</span> private Configuration conf;<a name="line.266"></a>
-<span class="sourceLineNo">267</span> private ThreadGroup threadGroup;<a name="line.267"></a>
-<span class="sourceLineNo">268</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.268"></a>
-<span class="sourceLineNo">269</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.269"></a>
-<span class="sourceLineNo">270</span> private int corePoolSize;<a name="line.270"></a>
-<span class="sourceLineNo">271</span> private int maxPoolSize;<a name="line.271"></a>
-<span class="sourceLineNo">272</span><a name="line.272"></a>
-<span class="sourceLineNo">273</span> private volatile long keepAliveTime;<a name="line.273"></a>
-<span class="sourceLineNo">274</span><a name="line.274"></a>
-<span class="sourceLineNo">275</span> /**<a name="line.275"></a>
-<span class="sourceLineNo">276</span> * Scheduler/Queue that contains runnable procedures.<a name="line.276"></a>
-<span class="sourceLineNo">277</span> */<a name="line.277"></a>
-<span class="sourceLineNo">278</span> private final ProcedureScheduler scheduler;<a name="line.278"></a>
-<span class="sourceLineNo">279</span><a name="line.279"></a>
-<span class="sourceLineNo">280</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.280"></a>
-<span class="sourceLineNo">281</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.281"></a>
-<span class="sourceLineNo">282</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.282"></a>
-<span class="sourceLineNo">283</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.283"></a>
-<span class="sourceLineNo">284</span> private final TEnvironment environment;<a name="line.284"></a>
-<span class="sourceLineNo">285</span> private final ProcedureStore store;<a name="line.285"></a>
-<span class="sourceLineNo">286</span><a name="line.286"></a>
-<span class="sourceLineNo">287</span> private final boolean checkOwnerSet;<a name="line.287"></a>
-<span class="sourceLineNo">288</span><a name="line.288"></a>
-<span class="sourceLineNo">289</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.289"></a>
-<span class="sourceLineNo">290</span> final ProcedureStore store) {<a name="line.290"></a>
-<span class="sourceLineNo">291</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.291"></a>
-<span class="sourceLineNo">292</span> }<a name="line.292"></a>
-<span class="sourceLineNo">293</span><a name="line.293"></a>
-<span class="sourceLineNo">294</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.294"></a>
-<span class="sourceLineNo">295</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.295"></a>
-<span class="sourceLineNo">296</span> this.environment = environment;<a name="line.296"></a>
-<span class="sourceLineNo">297</span> this.scheduler = scheduler;<a name="line.297"></a>
-<span class="sourceLineNo">298</span> this.store = store;<a name="line.298"></a>
-<span class="sourceLineNo">299</span> this.conf = conf;<a name="line.299"></a>
-<span class="sourceLineNo">300</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.300"></a>
-<span class="sourceLineNo">301</span> refreshConfiguration(conf);<a name="line.301"></a>
-<span class="sourceLineNo">302</span> }<a name="line.302"></a>
-<span class="sourceLineNo">303</span><a name="line.303"></a>
-<span class="sourceLineNo">304</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.304"></a>
-<span class="sourceLineNo">305</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.305"></a>
-<span class="sourceLineNo">306</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.306"></a>
-<span class="sourceLineNo">307</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.307"></a>
-<span class="sourceLineNo">308</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.308"></a>
-<span class="sourceLineNo">309</span><a name="line.309"></a>
-<span class="sourceLineNo">310</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.310"></a>
-<span class="sourceLineNo">311</span> @Override<a name="line.311"></a>
-<span class="sourceLineNo">312</span> public void setMaxProcId(long maxProcId) {<a name="line.312"></a>
-<span class="sourceLineNo">313</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.313"></a>
-<span class="sourceLineNo">314</span> lastProcId.set(maxProcId);<a name="line.314"></a>
-<span class="sourceLineNo">315</span> }<a name="line.315"></a>
-<span class="sourceLineNo">316</span><a name="line.316"></a>
-<span class="sourceLineNo">317</span> @Override<a name="line.317"></a>
-<span class="sourceLineNo">318</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.318"></a>
-<span class="sourceLineNo">319</span> loadProcedures(procIter, abortOnCorruption);<a name="line.319"></a>
-<span class="sourceLineNo">320</span> }<a name="line.320"></a>
-<span class="sourceLineNo">321</span><a name="line.321"></a>
-<span class="sourceLineNo">322</span> @Override<a name="line.322"></a>
-<span class="sourceLineNo">323</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.323"></a>
-<span class="sourceLineNo">324</span> int corruptedCount = 0;<a name="line.324"></a>
-<span class="sourceLineNo">325</span> while (procIter.hasNext()) {<a name="line.325"></a>
-<span class="sourceLineNo">326</span> Procedure<?> proc = procIter.next();<a name="line.326"></a>
-<span class="sourceLineNo">327</span> LOG.error("Corrupt " + proc);<a name="line.327"></a>
-<span class="sourceLineNo">328</span> corruptedCount++;<a name="line.328"></a>
-<span class="sourceLineNo">329</span> }<a name="line.329"></a>
-<span class="sourceLineNo">330</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.330"></a>
-<span class="sourceLineNo">331</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.331"></a>
-<span class="sourceLineNo">332</span> }<a name="line.332"></a>
-<span class="sourceLineNo">333</span> }<a name="line.333"></a>
-<span class="sourceLineNo">334</span> });<a name="line.334"></a>
-<span class="sourceLineNo">335</span> }<a name="line.335"></a>
-<span class="sourceLineNo">336</span><a name="line.336"></a>
-<span class="sourceLineNo">337</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.337"></a>
-<span class="sourceLineNo">338</span> final boolean abortOnCorruption) throws IOException {<a name="line.338"></a>
-<span class="sourceLineNo">339</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.339"></a>
-<span class="sourceLineNo">340</span><a name="line.340"></a>
-<span class="sourceLineNo">341</span> // 1. Build the rollback stack<a name="line.341"></a>
-<span class="sourceLineNo">342</span> int runnablesCount = 0;<a name="line.342"></a>
-<span class="sourceLineNo">343</span> while (procIter.hasNext()) {<a name="line.343"></a>
-<span class="sourceLineNo">344</span> boolean finished = procIter.isNextFinished();<a name="line.344"></a>
-<span class="sourceLineNo">345</span> Procedure proc = procIter.next();<a name="line.345"></a>
-<span class="sourceLineNo">346</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.346"></a>
-<span class="sourceLineNo">347</span> long procId = proc.getProcId();<a name="line.347"></a>
-<span class="sourceLineNo">348</span><a name="line.348"></a>
-<span class="sourceLineNo">349</span> if (finished) {<a name="line.349"></a>
-<span class="sourceLineNo">350</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.350"></a>
-<span class="sourceLineNo">351</span> if (debugEnabled) {<a name="line.351"></a>
-<span class="sourceLineNo">352</span> LOG.debug("Completed " + proc);<a name="line.352"></a>
-<span class="sourceLineNo">353</span> }<a name="line.353"></a>
-<span class="sourceLineNo">354</span> } else {<a name="line.354"></a>
-<span class="sourceLineNo">355</span> if (!proc.hasParent()) {<a name="line.355"></a>
-<span class="sourceLineNo">356</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.356"></a>
-<span class="sourceLineNo">357</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.357"></a>
-<span class="sourceLineNo">358</span> }<a name="line.358"></a>
+<span class="sourceLineNo">267</span><a name="line.267"></a>
+<span class="sourceLineNo">268</span> /**<a name="line.268"></a>
+<span class="sourceLineNo">269</span> * Created in the {@link #start(int, boolean)} method. Destroyed in {@link #join()} (FIX! Doing<a name="line.269"></a>
+<span class="sourceLineNo">270</span> * resource handling rather than observing in a #join is unexpected).<a name="line.270"></a>
+<span class="sourceLineNo">271</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.271"></a>
+<span class="sourceLineNo">272</span> * (Should be ok).<a name="line.272"></a>
+<span class="sourceLineNo">273</span> */<a name="line.273"></a>
+<span class="sourceLineNo">274</span> private ThreadGroup threadGroup;<a name="line.274"></a>
+<span class="sourceLineNo">275</span><a name="line.275"></a>
+<span class="sourceLineNo">276</span> /**<a name="line.276"></a>
+<span class="sourceLineNo">277</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.277"></a>
+<span class="sourceLineNo">278</span> * resource handling rather than observing in a #join is unexpected).<a name="line.278"></a>
+<span class="sourceLineNo">279</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.279"></a>
+<span class="sourceLineNo">280</span> * (Should be ok).<a name="line.280"></a>
+<span class="sourceLineNo">281</span> */<a name="line.281"></a>
+<span class="sourceLineNo">282</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.282"></a>
+<span class="sourceLineNo">283</span><a name="line.283"></a>
+<span class="sourceLineNo">284</span> /**<a name="line.284"></a>
+<span class="sourceLineNo">285</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.285"></a>
+<span class="sourceLineNo">286</span> * resource handling rather than observing in a #join is unexpected).<a name="line.286"></a>
+<span class="sourceLineNo">287</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.287"></a>
+<span class="sourceLineNo">288</span> * (Should be ok).<a name="line.288"></a>
+<span class="sourceLineNo">289</span> */<a name="line.289"></a>
+<span class="sourceLineNo">290</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.290"></a>
+<span class="sourceLineNo">291</span><a name="line.291"></a>
+<span class="sourceLineNo">292</span> private int corePoolSize;<a name="line.292"></a>
+<span class="sourceLineNo">293</span> private int maxPoolSize;<a name="line.293"></a>
+<span class="sourceLineNo">294</span><a name="line.294"></a>
+<span class="sourceLineNo">295</span> private volatile long keepAliveTime;<a name="line.295"></a>
+<span class="sourceLineNo">296</span><a name="line.296"></a>
+<span class="sourceLineNo">297</span> /**<a name="line.297"></a>
+<span class="sourceLineNo">298</span> * Scheduler/Queue that contains runnable procedures.<a name="line.298"></a>
+<span class="sourceLineNo">299</span> */<a name="line.299"></a>
+<span class="sourceLineNo">300</span> private final ProcedureScheduler scheduler;<a name="line.300"></a>
+<span class="sourceLineNo">301</span><a name="line.301"></a>
+<span class="sourceLineNo">302</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.302"></a>
+<span class="sourceLineNo">303</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.303"></a>
+<span class="sourceLineNo">304</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.304"></a>
+<span class="sourceLineNo">305</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.305"></a>
+<span class="sourceLineNo">306</span> private final TEnvironment environment;<a name="line.306"></a>
+<span class="sourceLineNo">307</span> private final ProcedureStore store;<a name="line.307"></a>
+<span class="sourceLineNo">308</span><a name="line.308"></a>
+<span class="sourceLineNo">309</span> private final boolean checkOwnerSet;<a name="line.309"></a>
+<span class="sourceLineNo">310</span><a name="line.310"></a>
+<span class="sourceLineNo">311</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.311"></a>
+<span class="sourceLineNo">312</span> final ProcedureStore store) {<a name="line.312"></a>
+<span class="sourceLineNo">313</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.313"></a>
+<span class="sourceLineNo">314</span> }<a name="line.314"></a>
+<span class="sourceLineNo">315</span><a name="line.315"></a>
+<span class="sourceLineNo">316</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.316"></a>
+<span class="sourceLineNo">317</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.317"></a>
+<span class="sourceLineNo">318</span> this.environment = environment;<a name="line.318"></a>
+<span class="sourceLineNo">319</span> this.scheduler = scheduler;<a name="line.319"></a>
+<span class="sourceLineNo">320</span> this.store = store;<a name="line.320"></a>
+<span class="sourceLineNo">321</span> this.conf = conf;<a name="line.321"></a>
+<span class="sourceLineNo">322</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.322"></a>
+<span class="sourceLineNo">323</span> refreshConfiguration(conf);<a name="line.323"></a>
+<span class="sourceLineNo">324</span><a name="line.324"></a>
+<span class="sourceLineNo">325</span> }<a name="line.325"></a>
+<span class="sourceLineNo">326</span><a name="line.326"></a>
+<span class="sourceLineNo">327</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.327"></a>
+<span class="sourceLineNo">328</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.328"></a>
+<span class="sourceLineNo">329</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.329"></a>
+<span class="sourceLineNo">330</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.330"></a>
+<span class="sourceLineNo">331</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.331"></a>
+<span class="sourceLineNo">332</span><a name="line.332"></a>
+<span class="sourceLineNo">333</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.333"></a>
+<span class="sourceLineNo">334</span> @Override<a name="line.334"></a>
+<span class="sourceLineNo">335</span> public void setMaxProcId(long maxProcId) {<a name="line.335"></a>
+<span class="sourceLineNo">336</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.336"></a>
+<span class="sourceLineNo">337</span> lastProcId.set(maxProcId);<a name="line.337"></a>
+<span class="sourceLineNo">338</span> }<a name="line.338"></a>
+<span class="sourceLineNo">339</span><a name="line.339"></a>
+<span class="sourceLineNo">340</span> @Override<a name="line.340"></a>
+<span class="sourceLineNo">341</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.341"></a>
+<span class="sourceLineNo">342</span> loadProcedures(procIter, abortOnCorruption);<a name="line.342"></a>
+<span class="sourceLineNo">343</span> }<a name="line.343"></a>
+<span class="sourceLineNo">344</span><a name="line.344"></a>
+<span class="sourceLineNo">345</span> @Override<a name="line.345"></a>
+<span class="sourceLineNo">346</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.346"></a>
+<span class="sourceLineNo">347</span> int corruptedCount = 0;<a name="line.347"></a>
+<span class="sourceLineNo">348</span> while (procIter.hasNext()) {<a name="line.348"></a>
+<span class="sourceLineNo">349</span> Procedure<?> proc = procIter.next();<a name="line.349"></a>
+<span class="sourceLineNo">350</span> LOG.error("Corrupt " + proc);<a name="line.350"></a>
+<span class="sourceLineNo">351</span> corruptedCount++;<a name="line.351"></a>
+<span class="sourceLineNo">352</span> }<a name="line.352"></a>
+<span class="sourceLineNo">353</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.353"></a>
+<span class="sourceLineNo">354</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.354"></a>
+<span class="sourceLineNo">355</span> }<a name="line.355"></a>
+<span class="sourceLineNo">356</span> }<a name="line.356"></a>
+<span class="sourceLineNo">357</span> });<a name="line.357"></a>
+<span class="sourceLineNo">358</span> }<a name="line.358"></a>
<span class="sourceLineNo">359</span><a name="line.359"></a>
-<span class="sourceLineNo">360</span> // add the procedure to the map<a name="line.360"></a>
-<span class="sourceLineNo">361</span> proc.beforeReplay(getEnvironment());<a name="line.361"></a>
-<span class="sourceLineNo">362</span> procedures.put(proc.getProcId(), proc);<a name="line.362"></a>
+<span class="sourceLineNo">360</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.360"></a>
+<span class="sourceLineNo">361</span> final boolean abortOnCorruption) throws IOException {<a name="line.361"></a>
+<span class="sourceLineNo">362</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.362"></a>
<span class="sourceLineNo">363</span><a name="line.363"></a>
-<span class="sourceLineNo">364</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.364"></a>
-<span class="sourceLineNo">365</span> runnablesCount++;<a name="line.365"></a>
-<span class="sourceLineNo">366</span> }<a name="line.366"></a>
-<span class="sourceLineNo">367</span> }<a name="line.367"></a>
-<span class="sourceLineNo">368</span><a name="line.368"></a>
-<span class="sourceLineNo">369</span> // add the nonce to the map<a name="line.369"></a>
-<span class="sourceLineNo">370</span> if (nonceKey != null) {<a name="line.370"></a>
-<span class="sourceLineNo">371</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.371"></a>
-<span class="sourceLineNo">372</span> }<a name="line.372"></a>
-<span class="sourceLineNo">373</span> }<a name="line.373"></a>
-<span class="sourceLineNo">374</span><a name="line.374"></a>
-<span class="sourceLineNo">375</span> // 2. Initialize the stacks<a name="line.375"></a>
-<span class="sourceLineNo">376</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.376"></a>
-<span class="sourceLineNo">377</span> HashSet<Procedure> waitingSet = null;<a name="line.377"></a>
-<span class="sourceLineNo">378</span> procIter.reset();<a name="line.378"></a>
-<span class="sourceLineNo">379</span> while (procIter.hasNext()) {<a name="line.379"></a>
-<span class="sourceLineNo">380</span> if (procIter.isNextFinished()) {<a name="line.380"></a>
-<span class="sourceLineNo">381</span> procIter.skipNext();<a name="line.381"></a>
-<span class="sourceLineNo">382</span> continue;<a name="line.382"></a>
-<span class="sourceLineNo">383</span> }<a name="line.383"></a>
-<span class="sourceLineNo">384</span><a name="line.384"></a>
-<span class="sourceLineNo">385</span> Procedure proc = procIter.next();<a name="line.385"></a>
-<span class="sourceLineNo">386</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.386"></a>
-<span class="sourceLineNo">387</span><a name="line.387"></a>
-<span class="sourceLineNo">388</span> if (debugEnabled) {<a name="line.388"></a>
-<span class="sourceLineNo">389</span> LOG.debug(String.format("Loading %s", proc));<a name="line.389"></a>
+<span class="sourceLineNo">364</span> // 1. Build the rollback stack<a name="line.364"></a>
+<span class="sourceLineNo">365</span> int runnablesCount = 0;<a name="line.365"></a>
+<span class="sourceLineNo">366</span> while (procIter.hasNext()) {<a name="line.366"></a>
+<span class="sourceLineNo">367</span> boolean finished = procIter.isNextFinished();<a name="line.367"></a>
+<span class="sourceLineNo">368</span> Procedure proc = procIter.next();<a name="line.368"></a>
+<span class="sourceLineNo">369</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.369"></a>
+<span class="sourceLineNo">370</span> long procId = proc.getProcId();<a name="line.370"></a>
+<span class="sourceLineNo">371</span><a name="line.371"></a>
+<span class="sourceLineNo">372</span> if (finished) {<a name="line.372"></a>
+<span class="sourceLineNo">373</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.373"></a>
+<span class="sourceLineNo">374</span> if (debugEnabled) {<a name="line.374"></a>
+<span class="sourceLineNo">375</span> LOG.debug("Completed " + proc);<a name="line.375"></a>
+<span class="sourceLineNo">376</span> }<a name="line.376"></a>
+<span class="sourceLineNo">377</span> } else {<a name="line.377"></a>
+<span class="sourceLineNo">378</span> if (!proc.hasParent()) {<a name="line.378"></a>
+<span class="sourceLineNo">379</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.379"></a>
+<span class="sourceLineNo">380</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.380"></a>
+<span class="sourceLineNo">381</span> }<a name="line.381"></a>
+<span class="sourceLineNo">382</span><a name="line.382"></a>
+<span class="sourceLineNo">383</span> // add the procedure to the map<a name="line.383"></a>
+<span class="sourceLineNo">384</span> proc.beforeReplay(getEnvironment());<a name="line.384"></a>
+<span class="sourceLineNo">385</span> procedures.put(proc.getProcId(), proc);<a name="line.385"></a>
+<span class="sourceLineNo">386</span><a name="line.386"></a>
+<span class="sourceLineNo">387</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.387"></a>
+<span class="sourceLineNo">388</span> runnablesCount++;<a name="line.388"></a>
+<span class="sourceLineNo">389</span> }<a name="line.389"></a>
<span class="sourceLineNo">390</span> }<a name="line.390"></a>
<span class="sourceLineNo">391</span><a name="line.391"></a>
-<span class="sourceLineNo">392</span> Long rootProcId = getRootProcedureId(proc);<a name="line.392"></a>
-<span class="sourceLineNo">393</span> if (rootProcId == null) {<a name="line.393"></a>
-<span class="sourceLineNo">394</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.394"></a>
-<span class="sourceLineNo">395</span> scheduler.addBack(proc);<a name="line.395"></a>
-<span class="sourceLineNo">396</span> continue;<a name="line.396"></a>
-<span class="sourceLineNo">397</span> }<a name="line.397"></a>
-<span class="sourceLineNo">398</span><a name="line.398"></a>
-<span class="sourceLineNo">399</span> if (proc.hasParent()) {<a name="line.399"></a>
-<span class="sourceLineNo">400</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.400"></a>
-<span class="sourceLineNo">401</span> // corrupted procedures are handled later at step 3<a name="line.401"></a>
-<span class="sourceLineNo">402</span> if (parent != null && !proc.isFinished()) {<a name="line.402"></a>
-<span class="sourceLineNo">403</span> parent.incChildrenLatch();<a name="line.403"></a>
-<span class="sourceLineNo">404</span> }<a name="line.404"></a>
-<span class="sourceLineNo">405</span> }<a name="line.405"></a>
-<span class="sourceLineNo">406</span><a name="line.406"></a>
-<span class="sourceLineNo">407</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.407"></a>
-<span class="sourceLineNo">408</span> procStack.loadStack(proc);<a name="line.408"></a>
-<span class="sourceLineNo">409</span><a name="line.409"></a>
-<span class="sourceLineNo">410</span> proc.setRootProcId(rootProcId);<a name="line.410"></a>
-<span class="sourceLineNo">411</span> switch (proc.getState()) {<a name="line.411"></a>
-<span class="sourceLineNo">412</span> case RUNNABLE:<a name="line.412"></a>
-<span class="sourceLineNo">413</span> runnableList.add(proc);<a name="line.413"></a>
-<span class="sourceLineNo">414</span> break;<a name="line.414"></a>
-<span class="sourceLineNo">415</span> case WAITING:<a name="line.415"></a>
-<span class="sourceLineNo">416</span> if (!proc.hasChildren()) {<a name="line.416"></a>
-<span class="sourceLineNo">417</span> runnableList.add(proc);<a name="line.417"></a>
-<span class="sourceLineNo">418</span> }<a name="line.418"></a>
-<span class="sourceLineNo">419</span> break;<a name="line.419"></a>
-<span class="sourceLineNo">420</span> case WAITING_TIMEOUT:<a name="line.420"></a>
-<span class="sourceLineNo">421</span> if (waitingSet == null) {<a name="line.421"></a>
-<span class="sourceLineNo">422</span> waitingSet = new HashSet<>();<a name="line.422"></a>
-<span class="sourceLineNo">423</span> }<a name="line.423"></a>
-<span class="sourceLineNo">424</span> waitingSet.add(proc);<a name="line.424"></a>
-<span class="sourceLineNo">425</span> break;<a name="line.425"></a>
-<span class="sourceLineNo">426</span> case FAILED:<a name="line.426"></a>
-<span class="sourceLineNo">427</span> // add the proc to the scheduler to perform the rollback<a name="line.427"></a>
-<span class="sourceLineNo">428</span> scheduler.addBack(proc);<a name="line.428"></a>
-<span class="sourceLineNo">429</span> break;<a name="line.429"></a>
-<span class="sourceLineNo">430</span> case ROLLEDBACK:<a name="line.430"></a>
-<span class="sourceLineNo">431</span> case INITIALIZING:<a name="line.431"></a>
-<span class="sourceLineNo">432</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.432"></a>
-<span class="sourceLineNo">433</span> LOG.error(msg);<a name="line.433"></a>
-<span class="sourceLineNo">434</span> throw new UnsupportedOperationException(msg);<a name="line.434"></a>
-<span class="sourceLineNo">435</span> default:<a name="line.435"></a>
-<span class="sourceLineNo">436</span> break;<a name="line.436"></a>
-<span class="sourceLineNo">437</span> }<a name="line.437"></a>
-<span class="sourceLineNo">438</span> }<a name="line.438"></a>
-<span class="sourceLineNo">439</span><a name="line.439"></a>
-<span class="sourceLineNo">440</span> // 3. Validate the stacks<a name="line.440"></a>
-<span class="sourceLineNo">441</span> int corruptedCount = 0;<a name="line.441"></a>
-<span class="sourceLineNo">442</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.442"></a>
-<span class="sourceLineNo">443</span> while (itStack.hasNext()) {<a name="line.443"></a>
-<span class="sourceLineNo">444</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.444"></a>
-<span class="sourceLineNo">445</span> RootProcedureState procStack = entry.getValue();<a name="line.445"></a>
-<span class="sourceLineNo">446</span> if (procStack.isValid()) continue;<a name="line.446"></a>
-<span class="sourceLineNo">447</span><a name="line.447"></a>
-<span class="sourceLineNo">448</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.448"></a>
-<span class="sourceLineNo">449</span> LOG.error("Corrupted " + proc);<a name="line.449"></a>
-<span class="sourceLineNo">450</span> procedures.remove(proc.getProcId());<a name="line.450"></a>
-<span class="sourceLineNo">451</span> runnableList.remove(proc);<a name="line.451"></a>
-<span class="sourceLineNo">452</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.452"></a>
-<span class="sourceLineNo">453</span> corruptedCount++;<a name="line.453"></a>
-<span class="sourceLineNo">454</span> }<a name="line.454"></a>
-<span class="sourceLineNo">455</span> itStack.remove();<a name="line.455"></a>
-<span class="sourceLineNo">456</span> }<a name="line.456"></a>
-<span class="sourceLineNo">457</span><a name="line.457"></a>
-<span class="sourceLineNo">458</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.458"></a>
-<span class="sourceLineNo">459</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.459"></a>
-<span class="sourceLineNo">460</span> }<a name="line.460"></a>
-<span class="sourceLineNo">461</span><a name="line.461"></a>
-<span class="sourceLineNo">462</span> // 4. Push the procedures to the timeout executor<a name="line.462"></a>
-<span class="sourceLineNo">463</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.463"></a>
-<span class="sourceLineNo">464</span> for (Procedure proc: waitingSet) {<a name="line.464"></a>
-<span class="sourceLineNo">465</span> proc.afterReplay(getEnvironment());<a name="line.465"></a>
-<span class="sourceLineNo">466</span> timeoutExecutor.add(proc);<a name="line.466"></a>
-<span class="sourceLineNo">467</span> }<a name="line.467"></a>
-<span class="sourceLineNo">468</span> }<a name="line.468"></a>
-<span class="sourceLineNo">469</span><a name="line.469"></a>
-<span class="sourceLineNo">470</span> // 5. Push the procedure to the scheduler<a name="line.470"></a>
-<span class="sourceLineNo">471</span> if (!runnableList.isEmpty()) {<a name="line.471"></a>
-<span class="sourceLineNo">472</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.472"></a>
-<span class="sourceLineNo">473</span> // some procedure may be started way before this stuff.<a name="line.473"></a>
-<span class="sourceLineNo">474</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.474"></a>
-<span class="sourceLineNo">475</span> Procedure proc = runnableList.get(i);<a name="line.475"></a>
-<span class="sourceLineNo">476</span> proc.afterReplay(getEnvironment());<a name="line.476"></a>
-<span class="sourceLineNo">477</span> if (!proc.hasParent()) {<a name="line.477"></a>
-<span class="sourceLineNo">478</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.478"></a>
-<span class="sourceLineNo">479</span> }<a name="line.479"></a>
-<span class="sourceLineNo">480</span> if (proc.wasExecuted()) {<a name="line.480"></a>
-<span class="sourceLineNo">481</span> scheduler.addFront(proc);<a name="line.481"></a>
-<span class="sourceLineNo">482</span> } else {<a name="line.482"></a>
-<span class="sourceLineNo">483</span> // if it was not in execution, it can wait.<a name="line.483"></a>
-<span class="sourceLineNo">484</span> scheduler.addBack(proc);<a name="line.484"></a>
-<span class="sourceLineNo">485</span> }<a name="line.485"></a>
-<span class="sourceLineNo">486</span> }<a name="line.486"></a>
-<span class="sourceLineNo">487</span> }<a name="line.487"></a>
-<span class="sourceLineNo">488</span> }<a name="line.488"></a>
-<span class="sourceLineNo">489</span><a name="line.489"></a>
-<span class="sourceLineNo">490</span> /**<a name="line.490"></a>
-<span class="sourceLineNo">491</span> * Start the procedure executor.<a name="line.491"></a>
-<span class="sourceLineNo">492</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.492"></a>
-<span class="sourceLineNo">493</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.493"></a>
-<span class="sourceLineNo">494</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.494"></a>
-<span class="sourceLineNo">495</span> *<a name="line.495"></a>
-<span class="sourceLineNo">496</span> * @param numThreads number of threads available for procedure execution.<a name="line.496"></a>
-<span class="sourceLineNo">497</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.497"></a>
-<span class="sourceLineNo">498</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.498"></a>
-<span class="sourceLineNo">499</span> */<a name="line.499"></a>
-<span class="sourceLineNo">500</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.500"></a>
-<span class="sourceLineNo">501</span> if (!running.compareAndSet(false, true)) {<a name="line.501"></a>
-<span class="sourceLineNo">502</span> LOG.warn("Already running");<a name="line.502"></a>
-<span class="sourceLineNo">503</span> return;<a name="line.503"></a>
-<span class="sourceLineNo">504</span> }<a name="line.504"></a>
-<span class="sourceLineNo">505</span><a name="line.505"></a>
-<span class="sourceLineNo">506</span> // We have numThreads executor + one timer thread used for timing out<a name="line.506"></a>
-<span class="sourceLineNo">507</span> // procedures and triggering periodic procedures.<a name="line.507"></a>
-<span class="sourceLineNo">508</span> this.corePoolSize = numThreads;<a name="line.508"></a>
-<span class="sourceLineNo">509</span> this.maxPoolSize = 10 * numThreads;<a name="line.509"></a>
-<span class="sourceLineNo">510</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.510"></a>
-<span class="sourceLineNo">511</span> corePoolSize, maxPoolSize);<a name="line.511"></a>
+<span class="sourceLineNo">392</span> // add the nonce to the map<a name="line.392"></a>
+<span class="sourceLineNo">393</span> if (nonceKey != null) {<a name="line.393"></a>
+<span class="sourceLineNo">394</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.394"></a>
+<span class="sourceLineNo">395</span> }<a name="line.395"></a>
+<span class="sourceLineNo">396</span> }<a name="line.396"></a>
+<span class="sourceLineNo">397</span><a name="line.397"></a>
+<span class="sourceLineNo">398</span> // 2. Initialize the stacks<a name="line.398"></a>
+<span class="sourceLineNo">399</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.399"></a>
+<span class="sourceLineNo">400</span> HashSet<Procedure> waitingSet = null;<a name="line.400"></a>
+<span class="sourceLineNo">401</span> procIter.reset();<a name="line.401"></a>
+<span class="sourceLineNo">402</span> while (procIter.hasNext()) {<a name="line.402"></a>
+<span class="sourceLineNo">403</span> if (procIter.isNextFinished()) {<a name="line.403"></a>
+<span class="sourceLineNo">404</span> procIter.skipNext();<a name="line.404"></a>
+<span class="sourceLineNo">405</span> continue;<a name="line.405"></a>
+<span class="sourceLineNo">406</span> }<a name="line.406"></a>
+<span class="sourceLineNo">407</span><a name="line.407"></a>
+<span class="sourceLineNo">408</span> Procedure proc = procIter.next();<a name="line.408"></a>
+<span class="sourceLineNo">409</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.409"></a>
+<span class="sourceLineNo">410</span><a name="line.410"></a>
+<span class="sourceLineNo">411</span> if (debugEnabled) {<a name="line.411"></a>
+<span class="sourceLineNo">412</span> LOG.debug(String.format("Loading %s", proc));<a name="line.412"></a>
+<span class="sourceLineNo">413</span> }<a name="line.413"></a>
+<span class="sourceLineNo">414</span><a name="line.414"></a>
+<span class="sourceLineNo">415</span> Long rootProcId = getRootProcedureId(proc);<a name="line.415"></a>
+<span class="sourceLineNo">416</span> if (rootProcId == null) {<a name="line.416"></a>
+<span class="sourceLineNo">417</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.417"></a>
+<span class="sourceLineNo">418</span> scheduler.addBack(proc);<a name="line.418"></a>
+<span class="sourceLineNo">419</span> continue;<a name="line.419"></a>
+<span class="sourceLineNo">420</span> }<a name="line.420"></a>
+<span class="sourceLineNo">421</span><a name="line.421"></a>
+<span class="sourceLineNo">422</span> if (proc.hasParent()) {<a name="line.422"></a>
+<span class="sourceLineNo">423</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.423"></a>
+<span class="sourceLineNo">424</span> // corrupted procedures are handled later at step 3<a name="line.424"></a>
+<span class="sourceLineNo">425</span> if (parent != null && !proc.isFinished()) {<a name="line.425"></a>
+<span class="sourceLineNo">426</span> parent.incChildrenLatch();<a name="line.426"></a>
+<span class="sourceLineNo">427</span> }<a name="line.427"></a>
+<span class="sourceLineNo">428</span> }<a name="line.428"></a>
+<span class="sourceLineNo">429</span><a name="line.429"></a>
+<span class="sourceLineNo">430</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.430"></a>
+<span class="sourceLineNo">431</span> procStack.loadStack(proc);<a name="line.431"></a>
+<span class="sourceLineNo">432</span><a name="line.432"></a>
+<span class="sourceLineNo">433</span> proc.setRootProcId(rootProcId);<a name="line.433"></a>
+<span class="sourceLineNo">434</span> switch (proc.getState()) {<a name="line.434"></a>
+<span class="sourceLineNo">435</span> case RUNNABLE:<a name="line.435"></a>
+<span class="sourceLineNo">436</span> runnableList.add(proc);<a name="line.436"></a>
+<span class="sourceLineNo">437</span> break;<a name="line.437"></a>
+<span class="sourceLineNo">438</span> case WAITING:<a name="line.438"></a>
+<span class="sourceLineNo">439</span> if (!proc.hasChildren()) {<a name="line.439"></a>
+<span class="sourceLineNo">440</span> runnableList.add(proc);<a name="line.440"></a>
+<span class="sourceLineNo">441</span> }<a name="line.441"></a>
+<span class="sourceLineNo">442</span> break;<a name="line.442"></a>
+<span class="sourceLineNo">443</span> case WAITING_TIMEOUT:<a name="line.443"></a>
+<span class="sourceLineNo">444</span> if (waitingSet == null) {<a name="line.444"></a>
+<span class="sourceLineNo">445</span> waitingSet = new HashSet<>();<a name="line.445"></a>
+<span class="sourceLineNo">446</span> }<a name="line.446"></a>
+<span class="sourceLineNo">447</span> waitingSet.add(proc);<a name="line.447"></a>
+<span class="sourceLineNo">448</span> break;<a name="line.448"></a>
+<span class="sourceLineNo">449</span> case FAILED:<a name="line.449"></a>
+<span class="sourceLineNo">450</span> // add the proc to the scheduler to perform the rollback<a name="line.450"></a>
+<span class="sourceLineNo">451</span> scheduler.addBack(proc);<a name="line.451"></a>
+<span class="sourceLineNo">452</span> break;<a name="line.452"></a>
+<span class="sourceLineNo">453</span> case ROLLEDBACK:<a name="line.453"></a>
+<span class="sourceLineNo">454</span> case INITIALIZING:<a name="line.454"></a>
+<span class="sourceLineNo">455</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.455"></a>
+<span class="sourceLineNo">456</span> LOG.error(msg);<a name="line.456"></a>
+<span class="sourceLineNo">457</span> throw new UnsupportedOperationException(msg);<a name="line.457"></a>
+<span class="sourceLineNo">458</span> default:<a name="line.458"></a>
+<span class="sourceLineNo">459</span> break;<a name="line.459"></a>
+<span class="sourceLineNo">460</span> }<a name="line.460"></a>
+<span class="sourceLineNo">461</span> }<a name="line.461"></a>
+<span class="sourceLineNo">462</span><a name="line.462"></a>
+<span class="sourceLineNo">463</span> // 3. Validate the stacks<a name="line.463"></a>
+<span class="sourceLineNo">464</span> int corruptedCount = 0;<a name="line.464"></a>
+<span class="sourceLineNo">465</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.465"></a>
+<span class="sourceLineNo">466</span> while (itStack.hasNext()) {<a name="line.466"></a>
+<span class="sourceLineNo">467</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.467"></a>
+<span class="sourceLineNo">468</span> RootProcedureState procStack = entry.getValue();<a name="line.468"></a>
+<span class="sourceLineNo">469</span> if (procStack.isValid()) continue;<a name="line.469"></a>
+<span class="sourceLineNo">470</span><a name="line.470"></a>
+<span class="sourceLineNo">471</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.471"></a>
+<span class="sourceLineNo">472</span> LOG.error("Corrupted " + proc);<a name="line.472"></a>
+<span class="sourceLineNo">473</span> procedures.remove(proc.getProcId());<a name="line.473"></a>
+<span class="sourceLineNo">474</span> runnableList.remove(proc);<a name="line.474"></a>
+<span class="sourceLineNo">475</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.475"></a>
+<span class="sourceLineNo">476</span> corruptedCount++;<a name="line.476"></a>
+<span class="sourceLineNo">477</span> }<a name="line.477"></a>
+<span class="sourceLineNo">478</span> itStack.remove();<a name="line.478"></a>
+<span class="sourceLineNo">479</span> }<a name="line.479"></a>
+<span class="sourceLineNo">480</span><a name="line.480"></a>
+<span class="sourceLineNo">481</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.481"></a>
+<span class="sourceLineNo">482</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.482"></a>
+<span class="sourceLineNo">483</span> }<a name="line.483"></a>
+<span class="sourceLineNo">484</span><a name="line.484"></a>
+<span class="sourceLineNo">485</span> // 4. Push the procedures to the timeout executor<a name="line.485"></a>
+<span class="sourceLineNo">486</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.486"></a>
+<span class="sourceLineNo">487</span> for (Procedure proc: waitingSet) {<a name="line.487"></a>
+<span class="sourceLineNo">488</span> proc.afterReplay(getEnvironment());<a name="line.488"></a>
+<span class="sourceLineNo">489</span> timeoutExecutor.add(proc);<a name="line.489"></a>
+<span class="sourceLineNo">490</span> }<a name="line.490"></a>
+<span class="sourceLineNo">491</span> }<a name="line.491"></a>
+<span class="sourceLineNo">492</span><a name="line.492"></a>
+<span class="sourceLineNo">493</span> // 5. Push the procedure to the scheduler<a name="line.493"></a>
+<span class="sourceLineNo">494</span> if (!runnableList.isEmpty()) {<a name="line.494"></a>
+<span class="sourceLineNo">495</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.495"></a>
+<span class="sourceLineNo">496</span> // some procedure may be started way before this stuff.<a name="line.496"></a>
+<span class="sourceLineNo">497</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.497"></a>
+<span class="sourceLineNo">498</span> Procedure proc = runnableList.get(i);<a name="line.498"></a>
+<span class="sourceLineNo">499</span> proc.afterReplay(getEnvironment());<a name="line.499"></a>
+<span class="sourceLineNo">500</span> if (!proc.hasParent()) {<a name="line.500"></a>
+<span class="sourceLineNo">501</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.501"></a>
+<span class="sourceLineNo">502</span> }<a name="line.502"></a>
+<span class="sourceLineNo">503</span> if (proc.wasExecuted()) {<a name="line.503"></a>
+<span class="sourceLineNo">504</span> scheduler.addFront(proc);<a name="line.504"></a>
+<span class="sourceLineNo">505</span> } else {<a name="line.505"></a>
+<span class="sourceLineNo">506</span> // if it was not in execution, it can wait.<a name="line.506"></a>
+<span class="sourceLineNo">507</span> scheduler.addBack(proc);<a name="line.507"></a>
+<span class="sourceLineNo">508</span> }<a name="line.508"></a>
+<span class="sourceLineNo">509</span> }<a name="line.509"></a>
+<span class="sourceLineNo">510</span> }<a name="line.510"></a>
+<span class="sourceLineNo">511</span> }<a name="line.511"></a>
<span class="sourceLineNo">512</span><a name="line.512"></a>
-<span class="sourceLineNo">513</span> // Create the Thread Group for the executors<a name="line.513"></a>
-<span class="sourceLineNo">514</span> threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.514"></a>
-<span class="sourceLineNo">515</span><a name="line.515"></a>
-<span class="sourceLineNo">516</span> // Create the timeout executor<a name="line.516"></a>
-<span class="sourceLineNo">517</span> timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.517"></a>
-<span class="sourceLineNo">518</span><a name="line.518"></a>
-<span class="sourceLineNo">519</span> // Create the workers<a name="line.519"></a>
-<span class="sourceLineNo">520</span> workerId.set(0);<a name="line.520"></a>
-<span class="sourceLineNo">521</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.521"></a>
-<span class="sourceLineNo">522</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.522"></a>
-<span class="sourceLineNo">523</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.523"></a>
-<span class="sourceLineNo">524</span> }<a name="line.524"></a>
-<span class="sourceLineNo">525</span><a name="line.525"></a>
-<span class="sourceLineNo">526</span> long st, et;<a name="line.526"></a>
-<span class="sourceLineNo">527</span><a name="line.527"></a>
-<span class="sourceLineNo">528</span> // Acquire the store lease.<a name="line.528"></a>
-<span class="sourceLineNo">529</span> st = EnvironmentEdgeManager.currentTime();<a name="line.529"></a>
-<span class="sourceLineNo">530</span> store.recoverLease();<a name="line.530"></a>
-<span class="sourceLineNo">531</span> et = EnvironmentEdgeManager.currentTime();<a name="line.531"></a>
-<span class="sourceLineNo">532</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.532"></a>
-<span class="sourceLineNo">533</span> StringUtils.humanTimeDiff(et - st));<a name="line.533"></a>
-<span class="sourceLineNo">534</span><a name="line.534"></a>
-<span class="sourceLineNo">535</span> // start the procedure scheduler<a name="line.535"></a>
-<span class="sourceLineNo">536</span> scheduler.start();<a name="line.536"></a>
-<span class="sourceLineNo">537</span><a name="line.537"></a>
-<span class="sourceLineNo">538</span> // TODO: Split in two steps.<a name="line.538"></a>
-<span class="sourceLineNo">539</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.539"></a>
-<span class="sourceLineNo">540</span> // The first one will make sure that we have the latest id,<a name="line.540"></a>
-<span class="sourceLineNo">541</span> // so we can start the threads and accept new procedures.<a name="line.541"></a>
-<span class="sourceLineNo">542</span> // The second step will do the actual load of old procedures.<a name="line.542"></a>
-<span class="sourceLineNo">543</span> st = EnvironmentEdgeManager.currentTime();<a name="line.543"></a>
-<span class="sourceLineNo">544</span> load(abortOnCorruption);<a name="line.544"></a>
-<span class="sourceLineNo">545</span> et = EnvironmentEdgeManager.currentTime();<a name="line.545"></a>
-<span class="sourceLineNo">546</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.546"></a>
-<span class="sourceLineNo">547</span> StringUtils.humanTimeDiff(et - st));<a name="line.547"></a>
-<span class="sourceLineNo">548</span><a name="line.548"></a>
-<span class="sourceLineNo">549</span> // Start the executors. Here we must have the lastProcId set.<a name="line.549"></a>
-<span class="sourceLineNo">550</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.550"></a>
-<span class="sourceLineNo">551</span> timeoutExecutor.start();<a name="line.551"></a>
-<span class="sourceLineNo">552</span> for (WorkerThread worker: workerThreads) {<a name="line.552"></a>
-<span class="sourceLineNo">553</span> worker.start();<a name="line.553"></a>
-<span class="sourceLineNo">554</span> }<a name="line.554"></a>
-<span class="sourceLineNo">555</span><a name="line.555"></a>
-<span class="sourceLineNo">556</span> // Internal chores<a name="line.556"></a>
-<span class="sourceLineNo">557</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.557"></a>
-<span class="sourceLineNo">558</span><a name="line.558"></a>
-<span class="sourceLineNo">559</span> // Add completed cleaner chore<a name="line.559"></a>
-<span class="sourceLineNo">560</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.560"></a>
-<span class="sourceLineNo">561</span> }<a name="line.561"></a>
-<span class="sourceLineNo">562</span><a name="line.562"></a>
-<span class="sourceLineNo">563</span> public void stop() {<a name="line.563"></a>
-<span class="sourceLineNo">564</span> if (!running.getAndSet(false)) {<a name="line.564"></a>
-<span class="sourceLineNo">565</span> return;<a name="line.565"></a>
-<span class="sourceLineNo">566</span> }<a name="line.566"></a>
-<span class="sourceLineNo">567</span><a name="line.567"></a>
-<span class="sourceLineNo">568</span> LOG.info("Stopping");<a name="line.568"></a>
-<span class="sourceLineNo">569</span> scheduler.stop();<a name="line.569"></a>
-<span class="sourceLineNo">570</span> timeoutExecutor.sendStopSignal();<a name="line.570"></a>
-<span class="sourceLineNo">571</span> }<a name="line.571"></a>
-<span class="sourceLineNo">572</span><a name="line.572"></a>
-<span class="sourceLineNo">573</span> @VisibleForTesting<a name="line.573"></a>
-<span class="sourceLineNo">574</span> public void join() {<a name="line.574"></a>
-<span class="sourceLineNo">575</span> assert !isRunning() : "expected not running";<a name="line.575"></a>
-<span class="sourceLineNo">576</span><a name="line.576"></a>
-<span class="sourceLineNo">577</span> // stop the timeout executor<a name="line.577"></a>
-<span class="sourceLineNo">578</span> timeoutExecutor.awaitTermination();<a name="line.578"></a>
-<span class="sourceLineNo">579</span> timeoutExecutor = null;<a name="line.579"></a>
-<span class="sourceLineNo">580</span><a name="line.580"></a>
-<span class="sourceLineNo">581</span> // stop the worker threads<a name="line.581"></a>
-<span class="sourceLineNo">582</span> for (WorkerThread worker: workerThreads) {<a name="line.582"></a>
-<span class="sourceLineNo">583</span> worker.awaitTermination();<a name="line.583"></a>
-<span class="sourceLineNo">584</span> }<a name="line.584"></a>
-<span class="sourceLineNo">585</span> workerThreads = null;<a name="line.585"></a>
-<span class="sourceLineNo">586</span><a name="line.586"></a>
-<span class="sourceLineNo">587</span> // Destroy the Thread Group for the executors<a name="line.587"></a>
-<span class="sourceLineNo">588</span> try {<a name="line.588"></a>
-<span class="sourceLineNo">589</span> threadGroup.destroy();<a name="line.589"></a>
-<span class="sourceLineNo">590</span> } catch (IllegalThreadStateException e) {<a name="line.590"></a>
-<span class="sourceLineNo">591</span> LOG.error("ThreadGroup " + threadGroup + " contains running threads; " + e.getMessage());<a name="line.591"></a>
-<span class="sourceLineNo">592</span> threadGroup.list();<a name="line.592"></a>
-<span class="sourceLineNo">593</span> } finally {<a name="line.593"></a>
-<span class="sourceLineNo">594</span> threadGroup = null;<a name="line.594"></a>
-<span class="sourceLineNo">595</span> }<a name="line.595"></a>
+<span class="sourceLineNo">513</span> /**<a name="line.513"></a>
+<span class="sourceLineNo">514</span> * Start the procedure executor.<a name="line.514"></a>
+<span class="sourceLineNo">515</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.515"></a>
+<span class="sourceLineNo">516</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.516"></a>
+<span class="sourceLineNo">517</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.517"></a>
+<span class="sourceLineNo">518</span> *<a name="line.518"></a>
+<span class="sourceLineNo">519</span> * @param numThreads number of threads available for procedure execution.<a name="line.519"></a>
+<span class="sourceLineNo">520</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.520"></a>
+<span class="sourceLineNo">521</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.521"></a>
+<span class="sourceLineNo">522</span> */<a name="line.522"></a>
+<span class="sourceLineNo">523</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.523"></a>
+<span class="sourceLineNo">524</span> if (!running.compareAndSet(false, true)) {<a name="line.524"></a>
+<span class="sourceLineNo">525</span> LOG.warn("Already running");<a name="line.525"></a>
+<span class="sourceLineNo">526</span> return;<a name="line.526"></a>
+<span class="sourceLineNo">527</span> }<a name="line.527"></a>
+<span class="sourceLineNo">528</span><a name="line.528"></a>
+<span class="sourceLineNo">529</span> // We have numThreads executor + one timer thread used for timing out<a name="line.529"></a>
+<span class="sourceLineNo">530</span> // procedures and triggering periodic procedures.<a name="line.530"></a>
+<span class="sourceLineNo">531</span> this.corePoolSize = numThreads;<a name="line.531"></a>
+<span class="sourceLineNo">532</span> this.maxPoolSize = 10 * numThreads;<a name="line.532"></a>
+<span class="sourceLineNo">533</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.533"></a>
+<span class="sourceLineNo">534</span> corePoolSize, maxPoolSize);<a name="line.534"></a>
+<span class="sourceLineNo">535</span><a name="line.535"></a>
+<span class="sourceLineNo">536</span> this.threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.536"></a>
+<span class="sourceLineNo">537</span> this.timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.537"></a>
+<span class="sourceLineNo">538</span><a name="line.538"></a>
+<span class="sourceLineNo">539</span> // Create the workers<a name="line.539"></a>
+<span class="sourceLineNo">540</span> workerId.set(0);<a name="line.540"></a>
+<span class="sourceLineNo">541</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.541"></a>
+<span class="sourceLineNo">542</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.542"></a>
+<span class="sourceLineNo">543</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.543"></a>
+<span class="sourceLineNo">544</span> }<a name="line.544"></a>
+<span class="sourceLineNo">545</span><a name="line.545"></a>
+<span class="sourceLineNo">546</span> long st, et;<a name="line.546"></a>
+<span class="sourceLineNo">547</span><a name="line.547"></a>
+<span class="sourceLineNo">548</span> // Acquire the store lease.<a name="line.548"></a>
+<span class="sourceLineNo">549</span> st = EnvironmentEdgeManager.currentTime();<a name="line.549"></a>
+<span class="sourceLineNo">550</span> store.recoverLease();<a name="line.550"></a>
+<span class="sourceLineNo">551</span> et = EnvironmentEdgeManager.currentTime();<a name="line.551"></a>
+<span class="sourceLineNo">552</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.552"></a>
+<span class="sourceLineNo">553</span> StringUtils.humanTimeDiff(et - st));<a name="line.553"></a>
+<span class="sourceLineNo">554</span><a name="line.554"></a>
+<span class="sourceLineNo">555</span> // start the procedure scheduler<a name="line.555"></a>
+<span class="sourceLineNo">556</span> scheduler.start();<a name="line.556"></a>
+<span class="sourceLineNo">557</span><a name="line.557"></a>
+<span class="sourceLineNo">558</span> // TODO: Split in two steps.<a name="line.558"></a>
+<span class="sourceLineNo">559</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.559"></a>
+<span class="sourceLineNo">560</span> // The first one will make sure that we have the latest id,<a name="line.560"></a>
+<span class="sourceLineNo">561</span> // so we can start the threads and accept new procedures.<a name="line.561"></a>
+<span class="sourceLineNo">562</span> // The second step will do the actual load of old procedures.<a name="line.562"></a>
+<span class="sourceLineNo">563</span> st = EnvironmentEdgeManager.currentTime();<a name="line.563"></a>
+<span class="sourceLineNo">564</span> load(abortOnCorruption);<a name="line.564"></a>
+<span class="sourceLineNo">565</span> et = EnvironmentEdgeManager.currentTime();<a name="line.565"></a>
+<span class="sourceLineNo">566</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.566"></a>
+<span class="sourceLineNo">567</span> StringUtils.humanTimeDiff(et - st));<a name="line.567"></a>
+<span class="sourceLineNo">568</span><a name="line.568"></a>
+<span class="sourceLineNo">569</span> // Start the executors. Here we must have the lastProcId set.<a name="line.569"></a>
+<span class="sourceLineNo">570</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.570"></a>
+<span class="sourceLineNo">571</span> timeoutExecutor.start();<a name="line.571"></a>
+<span class="sourceLineNo">572</span> for (WorkerThread worker: workerThreads) {<a name="line.572"></a>
+<span class="sourceLineNo">573</span> worker.start();<a name="line.573"></a>
+<span class="sourceLineNo">574</span> }<a name="line.574"></a>
+<span class="sourceLineNo">575</span><a name="line.575"></a>
+<span class="sourceLineNo">576</span> // Internal chores<a name="line.576"></a>
+<span class="sourceLineNo">577</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.577"></a>
+<span class="sourceLineNo">578</span><a name="line.578"></a>
+<span class="sourceLineNo">579</span> // Add completed cleaner chore<a name="line.579"></a>
+<span class="sourceLineNo">580</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.580"></a>
+<span class="sourceLineNo">581</span> }<a name="line.581"></a>
+<span class="sourceLineNo">582</span><a name="line.582"></a>
+<span class="sourceLineNo">583</span> public void stop() {<a name="line.583"></a>
+<span class="sourceLineNo">584</span> if (!running.getAndSet(false)) {<a name="line.584"></a>
+<span class="sourceLineNo">585</span> return;<a name="line.585"></a>
+<span class="sourceLineNo">586</span> }<a name="line.586"></a>
+<span class="sourceLineNo">587</span><a name="line.587"></a>
+<span class="sourceLineNo">588</span> LOG.info("Stopping");<a name="line.588"></a>
+<span class="sourceLineNo">589</span> scheduler.stop();<a name="line.589"></a>
+<span class="sourceLineNo">590</span> timeoutExecutor.sendStopSignal();<a name="line.590"></a>
+<span class="sourceLineNo">591</span> }<a name="line.591"></a>
+<span class="sourceLineNo">592</span><a name="line.592"></a>
+<span class="sourceLineNo">593</span> @VisibleForTesting<a name="line.593"></a>
+<span class="sourceLineNo">594</span> public void join() {<a name="line.594"></a>
+<span class="sourceLineNo">595</span> assert !isRunning() : "expected not running";<a name="line.595"></a>
<span class="sourceLineNo">596</span><a name="line.596"></a>
-<span class="sourceLineNo">597</span> // reset the in-memory state for testing<a name="line.597"></a>
-<span class="sourceLineNo">598</span> completed.clear();<a name="line.598"></a>
-<span class="sourceLineNo">599</span> rollbackStack.clear();<a name="line.599"></a>
-<span class="sourceLineNo">600</span> procedures.clear();<a name="line.600"></a>
-<span class="sourceLineNo">601</span> nonceKeysToProcIdsMap.clear();<a name="line.601"></a>
-<span class="sourceLineNo">602</span> scheduler.clear();<a name="line.602"></a>
-<span class="sourceLineNo">603</span> lastProcId.set(-1);<a name="line.603"></a>
-<span class="sourceLineNo">604</span> }<a name="line.604"></a>
-<span class="sourceLineNo">605</span><a name="line.605"></a>
-<span class="sourceLineNo">606</span> public void refreshConfiguration(final Configuration conf) {<a name="line.606"></a>
-<span class="sourceLineNo">607</span> this.conf = conf;<a name="line.607"></a>
-<span class="sourceLineNo">608</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.608"></a>
-<span class="sourceLineNo">609</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.609"></a>
-<span class="sourceLineNo">610</span> }<a name="line.610"></a>
-<span class="sourceLineNo">611</span><a name="line.611"></a>
-<span class="sourceLineNo">612</span> // ==========================================================================<a name="line.612"></a>
-<span class="sourceLineNo">613</span> // Accessors<a name="line.613"></a>
-<span class="sourceLineNo">614</span> // ==========================================================================<a name="line.614"></a>
-<span class="sourceLineNo">615</span> public boolean isRunning() {<a name="line.615"></a>
-<span class="sourceLineNo">616</span> return running.get();<a name="line.616"></a>
-<span class="sourceLineNo">617</span> }<a name="line.617"></a>
-<span class="sourceLineNo">618</span><a name="line.618"></a>
-<span class="sourceLineNo">619</span> /**<a name="line.619"></a>
-<span class="sourceLineNo">620</span> * @return the current number of worker threads.<a name="line.620"></a>
-<span class="sourceLineNo">621</span> */<a name="line.621"></a>
-<span class="sourceLineNo">622</span> public int getWorkerThreadCount() {<a name="line.622"></a>
-<span class="sourceLineNo">623</span> return workerThreads.size();<a name="line.623"></a>
-<span class="sourceLineNo">624</span> }<a name="line.624"></a>
-<span class="sourceLineNo">625</span><a name="line.625"></a>
-<span class="sourceLineNo">626</span> /**<a name="line.626"></a>
-<span class="sourceLineNo">627</span> * @return the core pool size settings.<a name="line.627"></a>
-<span class="sourceLineNo">628</span> */<a name="line.628"></a>
-<span class="sourceLineNo">629</span> public int getCorePoolSize() {<a name="line.629"></a>
-<span class="sourceLineNo">630</span> return corePoolSize;<a name="line.630"></a>
-<span class="sourceLineNo">631</span> }<a name="line.631"></a>
-<span class="sourceLineNo">632</span><a name="line.632"></a>
-<span class="sourceLineNo">633</span> public int getActiveExecutorCount() {<a name="line.633"></a>
-<span class="sourceLineNo">634</span> return activeExecutorCount.get();<a name="line.634"></a>
-<span class="sourceLineNo">635</span> }<a name="line.635"></a>
-<span class="sourceLineNo">636</span><a name="line.636"></a>
-<span class="sourceLineNo">637</span> public TEnvironment getEnvironment() {<a name="line.637"></a>
-<span class="sourceLineNo">638</span> return this.environment;<a name="line.638"></a>
-<span class="sourceLineNo">639</span> }<a name="line.639"></a>
-<span class="sourceLineNo">640</span><a name="line.640"></a>
-<span class="sourceLineNo">641</span> public ProcedureStore getStore() {<a name="line.641"></a>
-<span class="sourceLineNo">642</span> return this.store;<a name="line.642"></a>
+<span class="sourceLineNo">597</span> // stop the timeout executor<a name="line.597"></a>
+<span class="sourceLineNo">598</span> timeoutExecutor.awaitTermination();<a name="line.598"></a>
+<span class="sourceLineNo">599</span><a name="line.599"></a>
+<span class="sourceLineNo">600</span> // stop the worker threads<a name="line.600"></a>
+<span class="sourceLineNo">601</span> for (WorkerThread worker: workerThreads) {<a name="line.601"></a>
+<span class="sourceLineNo">602</span> worker.awaitTermination();<a name="line.602"></a>
+<span class="sourceLineNo">603</span> }<a name="line.603"></a>
+<span class="sourceLineNo">604</span><a name="line.604"></a>
+<span class="sourceLineNo">605</span> // Destroy the Thread Group for the executors<a name="line.605"></a>
+<span class="sourceLineNo">606</span> // TODO: Fix. #join is not place to destroy resources.<a name="line.606"></a>
+<span class="sourceLineNo">607</span> try {<a name="line.607"></a>
+<span class="sourceLineNo">608</span> threadGroup.destroy();<a name="line.608"></a>
+<span class="sourceLineNo">609</span> } catch (IllegalThreadStateException e) {<a name="line.609"></a>
+<span class="sourceLineNo">610</span> LOG.error("ThreadGroup {} contains running threads; {}: See STDOUT",<a name="line.610"></a>
+<span class="sourceLineNo">611</span> this.threadGroup, e.getMessage());<a name="line.611"></a>
+<span class="sourceLineNo">612</span> // This dumps list of threads on STDOUT.<a name="line.612"></a>
+<span class="sourceLineNo">613</span> this.threadGroup.list();<a name="line.613"></a>
+<span class="sourceLineNo">614</span> }<a name="line.614"></a>
+<span class="sourceLineNo">615</span><a name="line.615"></a>
+<span class="sourceLineNo">616</span> // reset the in-memory state for testing<a name="line.616"></a>
+<span class="sourceLineNo">617</span> completed.clear();<a name="line.617"></a>
+<span class="sourceLineNo">618</span> rollbackStack.clear();<a name="line.618"></a>
+<span class="sourceLineNo">619</span> procedures.clear();<a name="line.619"></a>
+<span class="sourceLineNo">620</span> nonceKeysToProcIdsMap.clear();<a name="line.620"></a>
+<span class="sourceLineNo">621</span> scheduler.clear();<a name="line.621"></a>
+<span class="sourceLineNo">622</span> lastProcId.set(-1);<a name="line.622"></a>
+<span class="sourceLineNo">623</span> }<a name="line.623"></a>
+<span class="sourceLineNo">624</span><a name="line.624"></a>
+<span class="sourceLineNo">625</span> public void refreshConfiguration(final Configuration conf) {<a name="line.625"></a>
+<span class="sourceLineNo">626</span> this.conf = conf;<a name="line.626"></a>
+<span class="sourceLineNo">627</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.627"></a>
+<span class="sourceLineNo">628</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.628"></a>
+<span class="sourceLineNo">629</span> }<a name="line.629"></a>
+<span class="sourceLineNo">630</span><a name="line.630"></a>
+<span class="sourceLineNo">631</span> // ==========================================================================<a name="line.631"></a>
+<span class="sourceLineNo">632</span> // Accessors<a name="line.632"></a>
+<span class="sourceLineNo">633</span> // ==========================================================================<a name="line.633"></a>
+<span class="sourceLineNo">634</span> public boolean isRunning() {<a name="line.634"></a>
+<span class="sourceLineNo">635</span> return running.get();<a name="line.635"></a>
+<span class="sourceLineNo">636</span> }<a name="line.636"></a>
+<span class="sourceLineNo">637</span><a name="line.637"></a>
+<span class="sourceLineNo">638</span> /**<a name="line.638"></a>
+<span class="sourceLineNo">639</span> * @return the current number of worker threads.<a name="line.639"></a>
+<span class="sourceLineNo">640</span> */<a name="line.640"></a>
+<span class="sourceLineNo">641</span> public int getWorkerThreadCount() {<a name="line.641"></a>
+<span class="sourceLineNo">642</span> return workerThreads.size();<a name="line.642"></a>
<span class="sourceLineNo">643</span> }<a name="line.643"></a>
<span class="sourceLineNo">644</span><a name="line.644"></a>
-<span class="sourceLineNo">645</span> ProcedureScheduler getScheduler() {<a name="line.645"></a>
-<span class="sourceLineNo">646</span> return scheduler;<a name="line.646"></a>
-<span class="sourceLineNo">647</span> }<a name="line.647"></a>
-<span class="sourceLineNo">648</span><a name="line.648"></a>
-<span class="sourceLineNo">649</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.649"></a>
-<span class="sourceLineNo">650</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.650"></a>
-<span class="sourceLineNo">651</span> this.scheduler.signalAll();<a name="line.651"></a>
-<span class="sourceLineNo">652</span> }<a name="line.652"></a>
-<span class="sourceLineNo">653</span><a name="line.653"></a>
-<span class="sourceLineNo">654</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.654"></a>
-<span class="sourceLineNo">655</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.655"></a>
-<span class="sourceLineNo">656</span> }<a name="line.656"></a>
-<span class="sourceLineNo">657</span><a name="line.657"></a>
-<span class="sourceLineNo">658</span> // ==========================================================================<a name="line.658"></a>
-<span class="sourceLineNo">659</span> // Submit/Remove Chores<a name="line.659"></a>
-<span class="sourceLineNo">660</span> // ==========================================================================<a name="line.660"></a>
-<span class="sourceLineNo">661</span><a name="line.661"></a>
-<span class="sourceLineNo">662</span> /**<a name="line.662"></a>
-<span class="sourceLineNo">663</span> * Add a chore procedure to the executor<a name="line.663"></a>
-<span class="sourceLineNo">664</span> * @param chore the chore to add<a name="line.664"></a>
-<span class="sourceLineNo">665</span> */<a name="line.665"></a>
-<span class="sourceLineNo">666</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.666"></a>
-<span class="sourceLineNo">667</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.667"></a>
-<span class="sourceLineNo">668</span> timeoutExecutor.add(chore);<a name="line.668"></a>
-<span class="sourceLineNo">669</span> }<a name="line.669"></a>
-<span class="sourceLineNo">670</span><a name="line.670"></a>
-<span class="sourceLineNo">671</span> /**<a name="line.671"></a>
-<span class="sourceLineNo">672</span> * Remove a chore procedure from the executor<a name="line.672"></a>
-<span class="sourceLineNo">673</span> * @param chore the chore to remove<a name="line.673"></a>
-<span class="sourceLineNo">674</span> * @return whether the chore is removed, or it will be removed later<a name="line.674"></a>
-<span class="sourceLineNo">675</span> */<a name="line.675"></a>
-<span class="sourceLineNo">676</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.676"></a>
-<span class="sourceLineNo">677</span> chore.setState(ProcedureState.SUCCESS);<a name="line.677"></a>
-<span class="sourceLineNo">678</span> return timeoutExecutor.remove(chore);<a name="line.678"></a>
-<span class="sourceLineNo">679</span> }<a name="line.679"></a>
+<span class="sourceLineNo">645</span> /**<a name="line.645"></a>
+<span class="sourceLineNo">646</span> * @return the core pool size settings.<a name="line.646"></a>
+<span class="sourceLineNo">647</span> */<a name="line.647"></a>
+<span class="sourceLineNo">648</span> public int getCorePoolSize() {<a name="line.648"></a>
+<span class="sourceLineNo">649</span> return corePoolSize;<a name="line.649"></a>
+<span class="sourceLineNo">650</span> }<a name="line.650"></a>
+<span class="sourceLineNo">651</span><a name="line.651"></a>
+<span class="sourceLineNo">652</span> public int getActiveExecutorCount() {<a name="line.652"></a>
+<span class="sourceLineNo">653</span> return activeExecutorCount.get();<a name="line.653"></a>
+<span class="sourceLineNo">654</span> }<a name="line.654"></a>
+<span class="sourceLineNo">655</span><a name="line.655"></a>
+<span class="sourceLineNo">656</span> public TEnvironment getEnvironment() {<a name="line.656"></a>
+<span class="sourceLineNo">657</span> return this.environment;<a name="line.657"></a>
+<span class="sourceLineNo">658</span> }<a name="line.658"></a>
+<span class="sourceLineNo">659</span><a name="line.659"></a>
+<span class="sourceLineNo">660</span> public ProcedureStore getStore() {<a name="line.660"></a>
+<span class="sourceLineNo">661</span> return this.store;<a name="line.661"></a>
+<span class="sourceLineNo">662</span> }<a name="line.662"></a>
+<span class="sourceLineNo">663</span><a name="line.663"></a>
+<span class="sourceLineNo">664</span> ProcedureScheduler getScheduler() {<a name="line.664"></a>
+<span class="sourceLineNo">665</span> return scheduler;<a name="line.665"></a>
+<span class="sourceLineNo">666</span> }<a name="line.666"></a>
+<span class="sourceLineNo">667</span><a name="line.667"></a>
+<span class="sourceLineNo">668</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.668"></a>
+<span class="sourceLineNo">669</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.669"></a>
+<span class="sourceLineNo">670</span> this.scheduler.signalAll();<a name="line.670"></a>
+<span class="sourceLineNo">671</span> }<a name="line.671"></a>
+<span class="sourceLineNo">672</span><a name="line.672"></a>
+<span class="sourceLineNo">673</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.673"></a>
+<span class="sourceLineNo">674</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.674"></a>
+<span class="sourceLineNo">675</span> }<a name="line.675"></a>
+<span class="sourceLineNo">676</span><a name="line.676"></a>
+<span class="sourceLineNo">677</span> // ==========================================================================<a name="line.677"></a>
+<span class="sourceLineNo">678</span> // Submit/Remove Chores<a name="line.678"></a>
+<span class="sourceLineNo">679</span> // ==========================================================================<a name="line.679"></a>
<span class="sourceLineNo">680</span><a name="line.680"></a>
-<span class="sourceLineNo">681</span> // ==========================================================================<a name="line.681"></a>
-<span class="sourceLineNo">682</span> // Nonce Procedure helpers<a name="line.682"></a>
-<span class="sourceLineNo">683</span> // ==========================================================================<a name="line.683"></a>
-<span class="sourceLineNo">684</span> /**<a name="line.684"></a>
-<span class="sourceLineNo">685</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.685"></a>
-<span class="sourceLineNo">686</span> * @param nonceGroup<a name="line.686"></a>
-<span class="sourceLineNo">687</span> * @param nonce<a name="line.687"></a>
-<span class="sourceLineNo">688</span> * @return the generated NonceKey<a name="line.688"></a>
-<span class="sourceLineNo">689</span> */<a name="line.689"></a>
-<span class="sourceLineNo">690</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.690"></a>
-<span class="sourceLineNo">691</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.691"></a>
-<span class="sourceLineNo">692</span> }<a name="line.692"></a>
-<span class="sourceLineNo">693</span><a name="line.693"></a>
-<span class="sourceLineNo">694</span> /**<a name="line.694"></a>
-<span class="sourceLineNo">695</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.695"></a>
-<span class="sourceLineNo">696</span> * A procId will be reserved and on submitProcedure(),<a name="line.696"></a>
-<span class="sourceLineNo">697</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.697"></a>
-<span class="sourceLineNo">698</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.698"></a>
-<span class="sourceLineNo">699</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.699"></a>
-<span class="sourceLineNo">700</span> * and submit the procedure.<a name="line.700"></a>
-<span class="sourceLineNo">701</span> *<a name="line.701"></a>
-<span class="sourceLineNo">702</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.702"></a>
-<span class="sourceLineNo">703</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.703"></a>
-<span class="sourceLineNo">704</span> */<a name="line.704"></a>
-<span class="sourceLineNo">705</span> public long registerNonce(final NonceKey nonceKey) {<a name="line.705"></a>
-<span class="sourceLineNo">706</span> if (nonceKey == null) return -1;<a name="line.706"></a>
-<span class="sourceLineNo">707</span><a name="line.707"></a>
-<span class="sourceLineNo">708</span> // check if we have already a Reserved ID for the nonce<a name="line.708"></a>
-<span class="sourceLineNo">709</span> Long oldProcId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.709"></a>
-<span class="sourceLineNo">710</span> if (oldProcId == null) {<a name="line.710"></a>
-<span class="sourceLineNo">711</span> // reserve a new Procedure ID, this will be associated with the nonce<a name="line.711"></a>
-<span class="sourceLineNo">712</span> // and the procedure submitted with the specified nonce will use this ID.<a name="line.712"></a>
-<span class="sourceLineNo">713</span> final long newProcId = nextProcId();<a name="line.713"></a>
-<span class="sourceLineNo">714</span> oldProcId = nonceKeysToProcIdsMap.putIfAbsent(nonceKey, newProcId);<a name="line.714"></a>
-<span class="sourceLineNo">715</span> if (oldProcId == null) return -1;<a name="line.715"></a>
-<span class="sourceLineNo">716</span> }<a name="line.716"></a>
-<span class="sourceLineNo">717</span><a name="line.717"></a>
-<span class="sourceLineNo">718</span> // we found a registered nonce, but the procedure may not have been submitted yet.<a name="line.718"></a>
-<span class="sourceLineNo">719</span> // since the client expect the procedure to be submitted, spin here until it is.<a name="line.719"></a>
-<span class="sourceLineNo">720</span> final boolean traceEnabled = LOG.isTraceEnabled();<a name="line.720"></a>
-<span class="sourceLineNo">721</span> while (isRunning() &&<a name="line.721"></a>
-<span class="sourceLineNo">722</span> !(procedures.containsKey(oldProcId) || completed.containsKey(oldProcId)) &&<a name="line.722"></a>
-<span class="sourceLineNo">723</span> nonceKeysToProcIdsMap.containsKey(nonceKey)) {<a name="line.723"></a>
-<span class="sourceLineNo">724</span> if (traceEnabled) {<a name="line.724"></a>
-<span class="sourceLineNo">725</span> LOG.trace("Waiting for pid=" + oldProcId.longValue() + " to be submitted");<a name="line.725"></a>
-<span class="sourceLineNo">726</span> }<a name="line.726"></a>
-<span class="sourceLineNo">727</span> Threads.sleep(100);<a name="line.727"></a>
-<span class="sourceLineNo">728</span> }<a name="line.728"></a>
-<span class="sourceLineNo">729</span> return oldProcId.longValue();<a name="line.729"></a>
-<span class="sourceLineNo">730</span> }<a name="line.730"></a>
-<span class="sourceLineNo">731</span><a name="line.731"></a>
-<span class="sourceLineNo">732</span> /**<a name="line.732"></a>
-<span class="sourceLineNo">733</span> * Remove the NonceKey if the procedure was not submitted to the executor.<a name="line.733"></a>
-<span class="sourceLineNo">734</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.734"></a>
-<span class="sourceLineNo">735</span> */<a name="line.735"></a>
-<span class="sourceLineNo">736</span> public void unregisterNonceIfProcedureWasNotSubmitted(final NonceKey nonceKey) {<a name="line.736"></a>
-<span class="sourceLineNo">737</span> if (nonceKey == null) return;<a name="line.737"></a>
-<span class="sourceLineNo">738</span><a name="line.738"></a>
-<span class="sourceLineNo">739</span> final Long procId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.739"></a>
-<span class="sourceLineNo">740</span> if (procId == null) return;<a name="line.740"></a>
-<span class="sourceLineNo">741</span><a name="line.741"></a>
-<span class="sourceLineNo">742</span> // if the procedure was not submitted, remove the nonce<a name="line.742"></a>
-<span class="sourceLineNo">743</span> if (!(procedures.containsKey(procId) || completed.containsKey(procId))) {<a name="line.743"></a>
-<span class="sourceLineNo">744</span> nonceKeysToProcIdsMap.remove(nonceKey);<a name="line.744"></a>
-<span class="sourceLineNo">745</span> }<a name="line.745"></a>
-<span class="sourceLineNo">746</span> }<a name="line.746"></a>
-<span class="sourceLineNo">747</span><a name="line.747"></a>
-<span class="sourceLineNo">748</span> public static class FailedProcedure<TEnvironment> extends Procedure<TEnvironment> {<a name="line.748"></a>
-<span class="sourceLineNo">749</span> private String procName;<a name="line.749"></a>
+<span class="sourceLineNo">681</span> /**<a name="line.681"></a>
+<span class="sourceLineNo">682</span> * Add a chore procedure to the executor<a name="line.682"></a>
+<span class="sourceLineNo">683</span> * @param chore the chore to add<a name="line.683"></a>
+<span class="sourceLineNo">684</span> */<a name="line.684"></a>
+<span class="sourceLineNo">685</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.685"></a>
+<span class="sourceLineNo">686</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.686"></a>
+<span class="sourceLineNo">687</span> timeoutExecutor.add(chore);<a name="line.687"></a>
+<span class="sourceLineNo">688</span> }<a name="line.688"></a>
+<span class="sourceLineNo">689</span><a name="line.689"></a>
+<span class="sourceLineNo">690</span> /**<a name="line.690"></a>
+<span class="sourceLineNo">691</span> * Remove a chore procedure from the executor<a name="line.691"></a>
+<span class="sourceLineNo">692</span> * @param chore the chore to remove<a name="line.692"></a>
+<span class="sourceLineNo">693</span> * @return whether the chore is removed, or it will be removed later<a name="line.693"></a>
+<span class="sourceLineNo">694</span> */<a name="line.694"></a>
+<span class="sourceLineNo">695</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.695"></a>
+<span class="sourceLineNo">696</span> chore.setState(ProcedureState.SUCCESS);<a name="line.696"></a>
+<span class="sourceLineNo">697</span> return timeoutExecutor.remove(chore);<a name="line.697"></a>
+<span class="sourceLineNo">698</span> }<a name="line.698"></a>
+<span class="sourceLineNo">699</span><a name="line.699"></a>
+<span class="sourceLineNo">700</span> // ==========================================================================<a name="line.700"></a>
+<span class="sourceLineNo">701</span> // Nonce Procedure helpers<a name="line.701"></a>
+<span class="sourceLineNo">702</span> // ==========================================================================<a name="line.702"></a>
+<span class="sourceLineNo">703</span> /**<a name="line.703"></a>
+<span class="sourceLineNo">704</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.704"></a>
+<span class="sourceLineNo">705</span> * @param nonceGroup<a name="line.705"></a>
+<span class="sourceLineNo">706</span> * @param nonce<a name="line.706"></a>
+<span class="sourceLineNo">707</span> * @return the generated NonceKey<a name="line.707"></a>
+<span class="sourceLineNo">708</span> */<a name="line.708"></a>
+<span class="sourceLineNo">709</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.709"></a>
+<span class="sourceLineNo">710</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.710"></a>
+<span class="sourceLineNo">711</span> }<a name="line.711"></a>
+<span class="sourceLineNo">712</span><a name="line.712"></a>
+<span class="sourceLineNo">713</span> /**<a name="line.713"></a>
+<span class="sourceLineNo">714</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.714"></a>
+<span class="sourceLineNo">715</span> * A procId will be reserved and on submitProcedure(),<a name="line.715"></a>
+<span class="sourceLineNo">716</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.716"></a>
+<span class="sourceLineNo">717</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.717"></a>
+<span class="sourceLineNo">718</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.718"></a>
+<span class="sourceLineNo">719</span> * and submit the procedure.<a name="line.719"></a>
+<span class="sourceLineNo">720</span> *<a name="line.720"></a>
+<span class="sourceLineNo">721</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.721"></a>
+<span class="sourceLineNo">722</span> * @return the procId associated
<TRUNCATED>
[07/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html
index 97aa79c..cc787e3 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.KeepAliveWorkerThread.html
@@ -272,1623 +272,1642 @@
<span class="sourceLineNo">264</span> private final CopyOnWriteArrayList<ProcedureExecutorListener> listeners = new CopyOnWriteArrayList<>();<a name="line.264"></a>
<span class="sourceLineNo">265</span><a name="line.265"></a>
<span class="sourceLineNo">266</span> private Configuration conf;<a name="line.266"></a>
-<span class="sourceLineNo">267</span> private ThreadGroup threadGroup;<a name="line.267"></a>
-<span class="sourceLineNo">268</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.268"></a>
-<span class="sourceLineNo">269</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.269"></a>
-<span class="sourceLineNo">270</span> private int corePoolSize;<a name="line.270"></a>
-<span class="sourceLineNo">271</span> private int maxPoolSize;<a name="line.271"></a>
-<span class="sourceLineNo">272</span><a name="line.272"></a>
-<span class="sourceLineNo">273</span> private volatile long keepAliveTime;<a name="line.273"></a>
-<span class="sourceLineNo">274</span><a name="line.274"></a>
-<span class="sourceLineNo">275</span> /**<a name="line.275"></a>
-<span class="sourceLineNo">276</span> * Scheduler/Queue that contains runnable procedures.<a name="line.276"></a>
-<span class="sourceLineNo">277</span> */<a name="line.277"></a>
-<span class="sourceLineNo">278</span> private final ProcedureScheduler scheduler;<a name="line.278"></a>
-<span class="sourceLineNo">279</span><a name="line.279"></a>
-<span class="sourceLineNo">280</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.280"></a>
-<span class="sourceLineNo">281</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.281"></a>
-<span class="sourceLineNo">282</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.282"></a>
-<span class="sourceLineNo">283</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.283"></a>
-<span class="sourceLineNo">284</span> private final TEnvironment environment;<a name="line.284"></a>
-<span class="sourceLineNo">285</span> private final ProcedureStore store;<a name="line.285"></a>
-<span class="sourceLineNo">286</span><a name="line.286"></a>
-<span class="sourceLineNo">287</span> private final boolean checkOwnerSet;<a name="line.287"></a>
-<span class="sourceLineNo">288</span><a name="line.288"></a>
-<span class="sourceLineNo">289</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.289"></a>
-<span class="sourceLineNo">290</span> final ProcedureStore store) {<a name="line.290"></a>
-<span class="sourceLineNo">291</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.291"></a>
-<span class="sourceLineNo">292</span> }<a name="line.292"></a>
-<span class="sourceLineNo">293</span><a name="line.293"></a>
-<span class="sourceLineNo">294</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.294"></a>
-<span class="sourceLineNo">295</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.295"></a>
-<span class="sourceLineNo">296</span> this.environment = environment;<a name="line.296"></a>
-<span class="sourceLineNo">297</span> this.scheduler = scheduler;<a name="line.297"></a>
-<span class="sourceLineNo">298</span> this.store = store;<a name="line.298"></a>
-<span class="sourceLineNo">299</span> this.conf = conf;<a name="line.299"></a>
-<span class="sourceLineNo">300</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.300"></a>
-<span class="sourceLineNo">301</span> refreshConfiguration(conf);<a name="line.301"></a>
-<span class="sourceLineNo">302</span> }<a name="line.302"></a>
-<span class="sourceLineNo">303</span><a name="line.303"></a>
-<span class="sourceLineNo">304</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.304"></a>
-<span class="sourceLineNo">305</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.305"></a>
-<span class="sourceLineNo">306</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.306"></a>
-<span class="sourceLineNo">307</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.307"></a>
-<span class="sourceLineNo">308</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.308"></a>
-<span class="sourceLineNo">309</span><a name="line.309"></a>
-<span class="sourceLineNo">310</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.310"></a>
-<span class="sourceLineNo">311</span> @Override<a name="line.311"></a>
-<span class="sourceLineNo">312</span> public void setMaxProcId(long maxProcId) {<a name="line.312"></a>
-<span class="sourceLineNo">313</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.313"></a>
-<span class="sourceLineNo">314</span> lastProcId.set(maxProcId);<a name="line.314"></a>
-<span class="sourceLineNo">315</span> }<a name="line.315"></a>
-<span class="sourceLineNo">316</span><a name="line.316"></a>
-<span class="sourceLineNo">317</span> @Override<a name="line.317"></a>
-<span class="sourceLineNo">318</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.318"></a>
-<span class="sourceLineNo">319</span> loadProcedures(procIter, abortOnCorruption);<a name="line.319"></a>
-<span class="sourceLineNo">320</span> }<a name="line.320"></a>
-<span class="sourceLineNo">321</span><a name="line.321"></a>
-<span class="sourceLineNo">322</span> @Override<a name="line.322"></a>
-<span class="sourceLineNo">323</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.323"></a>
-<span class="sourceLineNo">324</span> int corruptedCount = 0;<a name="line.324"></a>
-<span class="sourceLineNo">325</span> while (procIter.hasNext()) {<a name="line.325"></a>
-<span class="sourceLineNo">326</span> Procedure<?> proc = procIter.next();<a name="line.326"></a>
-<span class="sourceLineNo">327</span> LOG.error("Corrupt " + proc);<a name="line.327"></a>
-<span class="sourceLineNo">328</span> corruptedCount++;<a name="line.328"></a>
-<span class="sourceLineNo">329</span> }<a name="line.329"></a>
-<span class="sourceLineNo">330</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.330"></a>
-<span class="sourceLineNo">331</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.331"></a>
-<span class="sourceLineNo">332</span> }<a name="line.332"></a>
-<span class="sourceLineNo">333</span> }<a name="line.333"></a>
-<span class="sourceLineNo">334</span> });<a name="line.334"></a>
-<span class="sourceLineNo">335</span> }<a name="line.335"></a>
-<span class="sourceLineNo">336</span><a name="line.336"></a>
-<span class="sourceLineNo">337</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.337"></a>
-<span class="sourceLineNo">338</span> final boolean abortOnCorruption) throws IOException {<a name="line.338"></a>
-<span class="sourceLineNo">339</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.339"></a>
-<span class="sourceLineNo">340</span><a name="line.340"></a>
-<span class="sourceLineNo">341</span> // 1. Build the rollback stack<a name="line.341"></a>
-<span class="sourceLineNo">342</span> int runnablesCount = 0;<a name="line.342"></a>
-<span class="sourceLineNo">343</span> while (procIter.hasNext()) {<a name="line.343"></a>
-<span class="sourceLineNo">344</span> boolean finished = procIter.isNextFinished();<a name="line.344"></a>
-<span class="sourceLineNo">345</span> Procedure proc = procIter.next();<a name="line.345"></a>
-<span class="sourceLineNo">346</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.346"></a>
-<span class="sourceLineNo">347</span> long procId = proc.getProcId();<a name="line.347"></a>
-<span class="sourceLineNo">348</span><a name="line.348"></a>
-<span class="sourceLineNo">349</span> if (finished) {<a name="line.349"></a>
-<span class="sourceLineNo">350</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.350"></a>
-<span class="sourceLineNo">351</span> if (debugEnabled) {<a name="line.351"></a>
-<span class="sourceLineNo">352</span> LOG.debug("Completed " + proc);<a name="line.352"></a>
-<span class="sourceLineNo">353</span> }<a name="line.353"></a>
-<span class="sourceLineNo">354</span> } else {<a name="line.354"></a>
-<span class="sourceLineNo">355</span> if (!proc.hasParent()) {<a name="line.355"></a>
-<span class="sourceLineNo">356</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.356"></a>
-<span class="sourceLineNo">357</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.357"></a>
-<span class="sourceLineNo">358</span> }<a name="line.358"></a>
+<span class="sourceLineNo">267</span><a name="line.267"></a>
+<span class="sourceLineNo">268</span> /**<a name="line.268"></a>
+<span class="sourceLineNo">269</span> * Created in the {@link #start(int, boolean)} method. Destroyed in {@link #join()} (FIX! Doing<a name="line.269"></a>
+<span class="sourceLineNo">270</span> * resource handling rather than observing in a #join is unexpected).<a name="line.270"></a>
+<span class="sourceLineNo">271</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.271"></a>
+<span class="sourceLineNo">272</span> * (Should be ok).<a name="line.272"></a>
+<span class="sourceLineNo">273</span> */<a name="line.273"></a>
+<span class="sourceLineNo">274</span> private ThreadGroup threadGroup;<a name="line.274"></a>
+<span class="sourceLineNo">275</span><a name="line.275"></a>
+<span class="sourceLineNo">276</span> /**<a name="line.276"></a>
+<span class="sourceLineNo">277</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.277"></a>
+<span class="sourceLineNo">278</span> * resource handling rather than observing in a #join is unexpected).<a name="line.278"></a>
+<span class="sourceLineNo">279</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.279"></a>
+<span class="sourceLineNo">280</span> * (Should be ok).<a name="line.280"></a>
+<span class="sourceLineNo">281</span> */<a name="line.281"></a>
+<span class="sourceLineNo">282</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.282"></a>
+<span class="sourceLineNo">283</span><a name="line.283"></a>
+<span class="sourceLineNo">284</span> /**<a name="line.284"></a>
+<span class="sourceLineNo">285</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.285"></a>
+<span class="sourceLineNo">286</span> * resource handling rather than observing in a #join is unexpected).<a name="line.286"></a>
+<span class="sourceLineNo">287</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.287"></a>
+<span class="sourceLineNo">288</span> * (Should be ok).<a name="line.288"></a>
+<span class="sourceLineNo">289</span> */<a name="line.289"></a>
+<span class="sourceLineNo">290</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.290"></a>
+<span class="sourceLineNo">291</span><a name="line.291"></a>
+<span class="sourceLineNo">292</span> private int corePoolSize;<a name="line.292"></a>
+<span class="sourceLineNo">293</span> private int maxPoolSize;<a name="line.293"></a>
+<span class="sourceLineNo">294</span><a name="line.294"></a>
+<span class="sourceLineNo">295</span> private volatile long keepAliveTime;<a name="line.295"></a>
+<span class="sourceLineNo">296</span><a name="line.296"></a>
+<span class="sourceLineNo">297</span> /**<a name="line.297"></a>
+<span class="sourceLineNo">298</span> * Scheduler/Queue that contains runnable procedures.<a name="line.298"></a>
+<span class="sourceLineNo">299</span> */<a name="line.299"></a>
+<span class="sourceLineNo">300</span> private final ProcedureScheduler scheduler;<a name="line.300"></a>
+<span class="sourceLineNo">301</span><a name="line.301"></a>
+<span class="sourceLineNo">302</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.302"></a>
+<span class="sourceLineNo">303</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.303"></a>
+<span class="sourceLineNo">304</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.304"></a>
+<span class="sourceLineNo">305</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.305"></a>
+<span class="sourceLineNo">306</span> private final TEnvironment environment;<a name="line.306"></a>
+<span class="sourceLineNo">307</span> private final ProcedureStore store;<a name="line.307"></a>
+<span class="sourceLineNo">308</span><a name="line.308"></a>
+<span class="sourceLineNo">309</span> private final boolean checkOwnerSet;<a name="line.309"></a>
+<span class="sourceLineNo">310</span><a name="line.310"></a>
+<span class="sourceLineNo">311</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.311"></a>
+<span class="sourceLineNo">312</span> final ProcedureStore store) {<a name="line.312"></a>
+<span class="sourceLineNo">313</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.313"></a>
+<span class="sourceLineNo">314</span> }<a name="line.314"></a>
+<span class="sourceLineNo">315</span><a name="line.315"></a>
+<span class="sourceLineNo">316</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.316"></a>
+<span class="sourceLineNo">317</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.317"></a>
+<span class="sourceLineNo">318</span> this.environment = environment;<a name="line.318"></a>
+<span class="sourceLineNo">319</span> this.scheduler = scheduler;<a name="line.319"></a>
+<span class="sourceLineNo">320</span> this.store = store;<a name="line.320"></a>
+<span class="sourceLineNo">321</span> this.conf = conf;<a name="line.321"></a>
+<span class="sourceLineNo">322</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.322"></a>
+<span class="sourceLineNo">323</span> refreshConfiguration(conf);<a name="line.323"></a>
+<span class="sourceLineNo">324</span><a name="line.324"></a>
+<span class="sourceLineNo">325</span> }<a name="line.325"></a>
+<span class="sourceLineNo">326</span><a name="line.326"></a>
+<span class="sourceLineNo">327</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.327"></a>
+<span class="sourceLineNo">328</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.328"></a>
+<span class="sourceLineNo">329</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.329"></a>
+<span class="sourceLineNo">330</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.330"></a>
+<span class="sourceLineNo">331</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.331"></a>
+<span class="sourceLineNo">332</span><a name="line.332"></a>
+<span class="sourceLineNo">333</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.333"></a>
+<span class="sourceLineNo">334</span> @Override<a name="line.334"></a>
+<span class="sourceLineNo">335</span> public void setMaxProcId(long maxProcId) {<a name="line.335"></a>
+<span class="sourceLineNo">336</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.336"></a>
+<span class="sourceLineNo">337</span> lastProcId.set(maxProcId);<a name="line.337"></a>
+<span class="sourceLineNo">338</span> }<a name="line.338"></a>
+<span class="sourceLineNo">339</span><a name="line.339"></a>
+<span class="sourceLineNo">340</span> @Override<a name="line.340"></a>
+<span class="sourceLineNo">341</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.341"></a>
+<span class="sourceLineNo">342</span> loadProcedures(procIter, abortOnCorruption);<a name="line.342"></a>
+<span class="sourceLineNo">343</span> }<a name="line.343"></a>
+<span class="sourceLineNo">344</span><a name="line.344"></a>
+<span class="sourceLineNo">345</span> @Override<a name="line.345"></a>
+<span class="sourceLineNo">346</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.346"></a>
+<span class="sourceLineNo">347</span> int corruptedCount = 0;<a name="line.347"></a>
+<span class="sourceLineNo">348</span> while (procIter.hasNext()) {<a name="line.348"></a>
+<span class="sourceLineNo">349</span> Procedure<?> proc = procIter.next();<a name="line.349"></a>
+<span class="sourceLineNo">350</span> LOG.error("Corrupt " + proc);<a name="line.350"></a>
+<span class="sourceLineNo">351</span> corruptedCount++;<a name="line.351"></a>
+<span class="sourceLineNo">352</span> }<a name="line.352"></a>
+<span class="sourceLineNo">353</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.353"></a>
+<span class="sourceLineNo">354</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.354"></a>
+<span class="sourceLineNo">355</span> }<a name="line.355"></a>
+<span class="sourceLineNo">356</span> }<a name="line.356"></a>
+<span class="sourceLineNo">357</span> });<a name="line.357"></a>
+<span class="sourceLineNo">358</span> }<a name="line.358"></a>
<span class="sourceLineNo">359</span><a name="line.359"></a>
-<span class="sourceLineNo">360</span> // add the procedure to the map<a name="line.360"></a>
-<span class="sourceLineNo">361</span> proc.beforeReplay(getEnvironment());<a name="line.361"></a>
-<span class="sourceLineNo">362</span> procedures.put(proc.getProcId(), proc);<a name="line.362"></a>
+<span class="sourceLineNo">360</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.360"></a>
+<span class="sourceLineNo">361</span> final boolean abortOnCorruption) throws IOException {<a name="line.361"></a>
+<span class="sourceLineNo">362</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.362"></a>
<span class="sourceLineNo">363</span><a name="line.363"></a>
-<span class="sourceLineNo">364</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.364"></a>
-<span class="sourceLineNo">365</span> runnablesCount++;<a name="line.365"></a>
-<span class="sourceLineNo">366</span> }<a name="line.366"></a>
-<span class="sourceLineNo">367</span> }<a name="line.367"></a>
-<span class="sourceLineNo">368</span><a name="line.368"></a>
-<span class="sourceLineNo">369</span> // add the nonce to the map<a name="line.369"></a>
-<span class="sourceLineNo">370</span> if (nonceKey != null) {<a name="line.370"></a>
-<span class="sourceLineNo">371</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.371"></a>
-<span class="sourceLineNo">372</span> }<a name="line.372"></a>
-<span class="sourceLineNo">373</span> }<a name="line.373"></a>
-<span class="sourceLineNo">374</span><a name="line.374"></a>
-<span class="sourceLineNo">375</span> // 2. Initialize the stacks<a name="line.375"></a>
-<span class="sourceLineNo">376</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.376"></a>
-<span class="sourceLineNo">377</span> HashSet<Procedure> waitingSet = null;<a name="line.377"></a>
-<span class="sourceLineNo">378</span> procIter.reset();<a name="line.378"></a>
-<span class="sourceLineNo">379</span> while (procIter.hasNext()) {<a name="line.379"></a>
-<span class="sourceLineNo">380</span> if (procIter.isNextFinished()) {<a name="line.380"></a>
-<span class="sourceLineNo">381</span> procIter.skipNext();<a name="line.381"></a>
-<span class="sourceLineNo">382</span> continue;<a name="line.382"></a>
-<span class="sourceLineNo">383</span> }<a name="line.383"></a>
-<span class="sourceLineNo">384</span><a name="line.384"></a>
-<span class="sourceLineNo">385</span> Procedure proc = procIter.next();<a name="line.385"></a>
-<span class="sourceLineNo">386</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.386"></a>
-<span class="sourceLineNo">387</span><a name="line.387"></a>
-<span class="sourceLineNo">388</span> if (debugEnabled) {<a name="line.388"></a>
-<span class="sourceLineNo">389</span> LOG.debug(String.format("Loading %s", proc));<a name="line.389"></a>
+<span class="sourceLineNo">364</span> // 1. Build the rollback stack<a name="line.364"></a>
+<span class="sourceLineNo">365</span> int runnablesCount = 0;<a name="line.365"></a>
+<span class="sourceLineNo">366</span> while (procIter.hasNext()) {<a name="line.366"></a>
+<span class="sourceLineNo">367</span> boolean finished = procIter.isNextFinished();<a name="line.367"></a>
+<span class="sourceLineNo">368</span> Procedure proc = procIter.next();<a name="line.368"></a>
+<span class="sourceLineNo">369</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.369"></a>
+<span class="sourceLineNo">370</span> long procId = proc.getProcId();<a name="line.370"></a>
+<span class="sourceLineNo">371</span><a name="line.371"></a>
+<span class="sourceLineNo">372</span> if (finished) {<a name="line.372"></a>
+<span class="sourceLineNo">373</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.373"></a>
+<span class="sourceLineNo">374</span> if (debugEnabled) {<a name="line.374"></a>
+<span class="sourceLineNo">375</span> LOG.debug("Completed " + proc);<a name="line.375"></a>
+<span class="sourceLineNo">376</span> }<a name="line.376"></a>
+<span class="sourceLineNo">377</span> } else {<a name="line.377"></a>
+<span class="sourceLineNo">378</span> if (!proc.hasParent()) {<a name="line.378"></a>
+<span class="sourceLineNo">379</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.379"></a>
+<span class="sourceLineNo">380</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.380"></a>
+<span class="sourceLineNo">381</span> }<a name="line.381"></a>
+<span class="sourceLineNo">382</span><a name="line.382"></a>
+<span class="sourceLineNo">383</span> // add the procedure to the map<a name="line.383"></a>
+<span class="sourceLineNo">384</span> proc.beforeReplay(getEnvironment());<a name="line.384"></a>
+<span class="sourceLineNo">385</span> procedures.put(proc.getProcId(), proc);<a name="line.385"></a>
+<span class="sourceLineNo">386</span><a name="line.386"></a>
+<span class="sourceLineNo">387</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.387"></a>
+<span class="sourceLineNo">388</span> runnablesCount++;<a name="line.388"></a>
+<span class="sourceLineNo">389</span> }<a name="line.389"></a>
<span class="sourceLineNo">390</span> }<a name="line.390"></a>
<span class="sourceLineNo">391</span><a name="line.391"></a>
-<span class="sourceLineNo">392</span> Long rootProcId = getRootProcedureId(proc);<a name="line.392"></a>
-<span class="sourceLineNo">393</span> if (rootProcId == null) {<a name="line.393"></a>
-<span class="sourceLineNo">394</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.394"></a>
-<span class="sourceLineNo">395</span> scheduler.addBack(proc);<a name="line.395"></a>
-<span class="sourceLineNo">396</span> continue;<a name="line.396"></a>
-<span class="sourceLineNo">397</span> }<a name="line.397"></a>
-<span class="sourceLineNo">398</span><a name="line.398"></a>
-<span class="sourceLineNo">399</span> if (proc.hasParent()) {<a name="line.399"></a>
-<span class="sourceLineNo">400</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.400"></a>
-<span class="sourceLineNo">401</span> // corrupted procedures are handled later at step 3<a name="line.401"></a>
-<span class="sourceLineNo">402</span> if (parent != null && !proc.isFinished()) {<a name="line.402"></a>
-<span class="sourceLineNo">403</span> parent.incChildrenLatch();<a name="line.403"></a>
-<span class="sourceLineNo">404</span> }<a name="line.404"></a>
-<span class="sourceLineNo">405</span> }<a name="line.405"></a>
-<span class="sourceLineNo">406</span><a name="line.406"></a>
-<span class="sourceLineNo">407</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.407"></a>
-<span class="sourceLineNo">408</span> procStack.loadStack(proc);<a name="line.408"></a>
-<span class="sourceLineNo">409</span><a name="line.409"></a>
-<span class="sourceLineNo">410</span> proc.setRootProcId(rootProcId);<a name="line.410"></a>
-<span class="sourceLineNo">411</span> switch (proc.getState()) {<a name="line.411"></a>
-<span class="sourceLineNo">412</span> case RUNNABLE:<a name="line.412"></a>
-<span class="sourceLineNo">413</span> runnableList.add(proc);<a name="line.413"></a>
-<span class="sourceLineNo">414</span> break;<a name="line.414"></a>
-<span class="sourceLineNo">415</span> case WAITING:<a name="line.415"></a>
-<span class="sourceLineNo">416</span> if (!proc.hasChildren()) {<a name="line.416"></a>
-<span class="sourceLineNo">417</span> runnableList.add(proc);<a name="line.417"></a>
-<span class="sourceLineNo">418</span> }<a name="line.418"></a>
-<span class="sourceLineNo">419</span> break;<a name="line.419"></a>
-<span class="sourceLineNo">420</span> case WAITING_TIMEOUT:<a name="line.420"></a>
-<span class="sourceLineNo">421</span> if (waitingSet == null) {<a name="line.421"></a>
-<span class="sourceLineNo">422</span> waitingSet = new HashSet<>();<a name="line.422"></a>
-<span class="sourceLineNo">423</span> }<a name="line.423"></a>
-<span class="sourceLineNo">424</span> waitingSet.add(proc);<a name="line.424"></a>
-<span class="sourceLineNo">425</span> break;<a name="line.425"></a>
-<span class="sourceLineNo">426</span> case FAILED:<a name="line.426"></a>
-<span class="sourceLineNo">427</span> // add the proc to the scheduler to perform the rollback<a name="line.427"></a>
-<span class="sourceLineNo">428</span> scheduler.addBack(proc);<a name="line.428"></a>
-<span class="sourceLineNo">429</span> break;<a name="line.429"></a>
-<span class="sourceLineNo">430</span> case ROLLEDBACK:<a name="line.430"></a>
-<span class="sourceLineNo">431</span> case INITIALIZING:<a name="line.431"></a>
-<span class="sourceLineNo">432</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.432"></a>
-<span class="sourceLineNo">433</span> LOG.error(msg);<a name="line.433"></a>
-<span class="sourceLineNo">434</span> throw new UnsupportedOperationException(msg);<a name="line.434"></a>
-<span class="sourceLineNo">435</span> default:<a name="line.435"></a>
-<span class="sourceLineNo">436</span> break;<a name="line.436"></a>
-<span class="sourceLineNo">437</span> }<a name="line.437"></a>
-<span class="sourceLineNo">438</span> }<a name="line.438"></a>
-<span class="sourceLineNo">439</span><a name="line.439"></a>
-<span class="sourceLineNo">440</span> // 3. Validate the stacks<a name="line.440"></a>
-<span class="sourceLineNo">441</span> int corruptedCount = 0;<a name="line.441"></a>
-<span class="sourceLineNo">442</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.442"></a>
-<span class="sourceLineNo">443</span> while (itStack.hasNext()) {<a name="line.443"></a>
-<span class="sourceLineNo">444</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.444"></a>
-<span class="sourceLineNo">445</span> RootProcedureState procStack = entry.getValue();<a name="line.445"></a>
-<span class="sourceLineNo">446</span> if (procStack.isValid()) continue;<a name="line.446"></a>
-<span class="sourceLineNo">447</span><a name="line.447"></a>
-<span class="sourceLineNo">448</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.448"></a>
-<span class="sourceLineNo">449</span> LOG.error("Corrupted " + proc);<a name="line.449"></a>
-<span class="sourceLineNo">450</span> procedures.remove(proc.getProcId());<a name="line.450"></a>
-<span class="sourceLineNo">451</span> runnableList.remove(proc);<a name="line.451"></a>
-<span class="sourceLineNo">452</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.452"></a>
-<span class="sourceLineNo">453</span> corruptedCount++;<a name="line.453"></a>
-<span class="sourceLineNo">454</span> }<a name="line.454"></a>
-<span class="sourceLineNo">455</span> itStack.remove();<a name="line.455"></a>
-<span class="sourceLineNo">456</span> }<a name="line.456"></a>
-<span class="sourceLineNo">457</span><a name="line.457"></a>
-<span class="sourceLineNo">458</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.458"></a>
-<span class="sourceLineNo">459</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.459"></a>
-<span class="sourceLineNo">460</span> }<a name="line.460"></a>
-<span class="sourceLineNo">461</span><a name="line.461"></a>
-<span class="sourceLineNo">462</span> // 4. Push the procedures to the timeout executor<a name="line.462"></a>
-<span class="sourceLineNo">463</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.463"></a>
-<span class="sourceLineNo">464</span> for (Procedure proc: waitingSet) {<a name="line.464"></a>
-<span class="sourceLineNo">465</span> proc.afterReplay(getEnvironment());<a name="line.465"></a>
-<span class="sourceLineNo">466</span> timeoutExecutor.add(proc);<a name="line.466"></a>
-<span class="sourceLineNo">467</span> }<a name="line.467"></a>
-<span class="sourceLineNo">468</span> }<a name="line.468"></a>
-<span class="sourceLineNo">469</span><a name="line.469"></a>
-<span class="sourceLineNo">470</span> // 5. Push the procedure to the scheduler<a name="line.470"></a>
-<span class="sourceLineNo">471</span> if (!runnableList.isEmpty()) {<a name="line.471"></a>
-<span class="sourceLineNo">472</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.472"></a>
-<span class="sourceLineNo">473</span> // some procedure may be started way before this stuff.<a name="line.473"></a>
-<span class="sourceLineNo">474</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.474"></a>
-<span class="sourceLineNo">475</span> Procedure proc = runnableList.get(i);<a name="line.475"></a>
-<span class="sourceLineNo">476</span> proc.afterReplay(getEnvironment());<a name="line.476"></a>
-<span class="sourceLineNo">477</span> if (!proc.hasParent()) {<a name="line.477"></a>
-<span class="sourceLineNo">478</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.478"></a>
-<span class="sourceLineNo">479</span> }<a name="line.479"></a>
-<span class="sourceLineNo">480</span> if (proc.wasExecuted()) {<a name="line.480"></a>
-<span class="sourceLineNo">481</span> scheduler.addFront(proc);<a name="line.481"></a>
-<span class="sourceLineNo">482</span> } else {<a name="line.482"></a>
-<span class="sourceLineNo">483</span> // if it was not in execution, it can wait.<a name="line.483"></a>
-<span class="sourceLineNo">484</span> scheduler.addBack(proc);<a name="line.484"></a>
-<span class="sourceLineNo">485</span> }<a name="line.485"></a>
-<span class="sourceLineNo">486</span> }<a name="line.486"></a>
-<span class="sourceLineNo">487</span> }<a name="line.487"></a>
-<span class="sourceLineNo">488</span> }<a name="line.488"></a>
-<span class="sourceLineNo">489</span><a name="line.489"></a>
-<span class="sourceLineNo">490</span> /**<a name="line.490"></a>
-<span class="sourceLineNo">491</span> * Start the procedure executor.<a name="line.491"></a>
-<span class="sourceLineNo">492</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.492"></a>
-<span class="sourceLineNo">493</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.493"></a>
-<span class="sourceLineNo">494</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.494"></a>
-<span class="sourceLineNo">495</span> *<a name="line.495"></a>
-<span class="sourceLineNo">496</span> * @param numThreads number of threads available for procedure execution.<a name="line.496"></a>
-<span class="sourceLineNo">497</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.497"></a>
-<span class="sourceLineNo">498</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.498"></a>
-<span class="sourceLineNo">499</span> */<a name="line.499"></a>
-<span class="sourceLineNo">500</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.500"></a>
-<span class="sourceLineNo">501</span> if (!running.compareAndSet(false, true)) {<a name="line.501"></a>
-<span class="sourceLineNo">502</span> LOG.warn("Already running");<a name="line.502"></a>
-<span class="sourceLineNo">503</span> return;<a name="line.503"></a>
-<span class="sourceLineNo">504</span> }<a name="line.504"></a>
-<span class="sourceLineNo">505</span><a name="line.505"></a>
-<span class="sourceLineNo">506</span> // We have numThreads executor + one timer thread used for timing out<a name="line.506"></a>
-<span class="sourceLineNo">507</span> // procedures and triggering periodic procedures.<a name="line.507"></a>
-<span class="sourceLineNo">508</span> this.corePoolSize = numThreads;<a name="line.508"></a>
-<span class="sourceLineNo">509</span> this.maxPoolSize = 10 * numThreads;<a name="line.509"></a>
-<span class="sourceLineNo">510</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.510"></a>
-<span class="sourceLineNo">511</span> corePoolSize, maxPoolSize);<a name="line.511"></a>
+<span class="sourceLineNo">392</span> // add the nonce to the map<a name="line.392"></a>
+<span class="sourceLineNo">393</span> if (nonceKey != null) {<a name="line.393"></a>
+<span class="sourceLineNo">394</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.394"></a>
+<span class="sourceLineNo">395</span> }<a name="line.395"></a>
+<span class="sourceLineNo">396</span> }<a name="line.396"></a>
+<span class="sourceLineNo">397</span><a name="line.397"></a>
+<span class="sourceLineNo">398</span> // 2. Initialize the stacks<a name="line.398"></a>
+<span class="sourceLineNo">399</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.399"></a>
+<span class="sourceLineNo">400</span> HashSet<Procedure> waitingSet = null;<a name="line.400"></a>
+<span class="sourceLineNo">401</span> procIter.reset();<a name="line.401"></a>
+<span class="sourceLineNo">402</span> while (procIter.hasNext()) {<a name="line.402"></a>
+<span class="sourceLineNo">403</span> if (procIter.isNextFinished()) {<a name="line.403"></a>
+<span class="sourceLineNo">404</span> procIter.skipNext();<a name="line.404"></a>
+<span class="sourceLineNo">405</span> continue;<a name="line.405"></a>
+<span class="sourceLineNo">406</span> }<a name="line.406"></a>
+<span class="sourceLineNo">407</span><a name="line.407"></a>
+<span class="sourceLineNo">408</span> Procedure proc = procIter.next();<a name="line.408"></a>
+<span class="sourceLineNo">409</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.409"></a>
+<span class="sourceLineNo">410</span><a name="line.410"></a>
+<span class="sourceLineNo">411</span> if (debugEnabled) {<a name="line.411"></a>
+<span class="sourceLineNo">412</span> LOG.debug(String.format("Loading %s", proc));<a name="line.412"></a>
+<span class="sourceLineNo">413</span> }<a name="line.413"></a>
+<span class="sourceLineNo">414</span><a name="line.414"></a>
+<span class="sourceLineNo">415</span> Long rootProcId = getRootProcedureId(proc);<a name="line.415"></a>
+<span class="sourceLineNo">416</span> if (rootProcId == null) {<a name="line.416"></a>
+<span class="sourceLineNo">417</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.417"></a>
+<span class="sourceLineNo">418</span> scheduler.addBack(proc);<a name="line.418"></a>
+<span class="sourceLineNo">419</span> continue;<a name="line.419"></a>
+<span class="sourceLineNo">420</span> }<a name="line.420"></a>
+<span class="sourceLineNo">421</span><a name="line.421"></a>
+<span class="sourceLineNo">422</span> if (proc.hasParent()) {<a name="line.422"></a>
+<span class="sourceLineNo">423</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.423"></a>
+<span class="sourceLineNo">424</span> // corrupted procedures are handled later at step 3<a name="line.424"></a>
+<span class="sourceLineNo">425</span> if (parent != null && !proc.isFinished()) {<a name="line.425"></a>
+<span class="sourceLineNo">426</span> parent.incChildrenLatch();<a name="line.426"></a>
+<span class="sourceLineNo">427</span> }<a name="line.427"></a>
+<span class="sourceLineNo">428</span> }<a name="line.428"></a>
+<span class="sourceLineNo">429</span><a name="line.429"></a>
+<span class="sourceLineNo">430</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.430"></a>
+<span class="sourceLineNo">431</span> procStack.loadStack(proc);<a name="line.431"></a>
+<span class="sourceLineNo">432</span><a name="line.432"></a>
+<span class="sourceLineNo">433</span> proc.setRootProcId(rootProcId);<a name="line.433"></a>
+<span class="sourceLineNo">434</span> switch (proc.getState()) {<a name="line.434"></a>
+<span class="sourceLineNo">435</span> case RUNNABLE:<a name="line.435"></a>
+<span class="sourceLineNo">436</span> runnableList.add(proc);<a name="line.436"></a>
+<span class="sourceLineNo">437</span> break;<a name="line.437"></a>
+<span class="sourceLineNo">438</span> case WAITING:<a name="line.438"></a>
+<span class="sourceLineNo">439</span> if (!proc.hasChildren()) {<a name="line.439"></a>
+<span class="sourceLineNo">440</span> runnableList.add(proc);<a name="line.440"></a>
+<span class="sourceLineNo">441</span> }<a name="line.441"></a>
+<span class="sourceLineNo">442</span> break;<a name="line.442"></a>
+<span class="sourceLineNo">443</span> case WAITING_TIMEOUT:<a name="line.443"></a>
+<span class="sourceLineNo">444</span> if (waitingSet == null) {<a name="line.444"></a>
+<span class="sourceLineNo">445</span> waitingSet = new HashSet<>();<a name="line.445"></a>
+<span class="sourceLineNo">446</span> }<a name="line.446"></a>
+<span class="sourceLineNo">447</span> waitingSet.add(proc);<a name="line.447"></a>
+<span class="sourceLineNo">448</span> break;<a name="line.448"></a>
+<span class="sourceLineNo">449</span> case FAILED:<a name="line.449"></a>
+<span class="sourceLineNo">450</span> // add the proc to the scheduler to perform the rollback<a name="line.450"></a>
+<span class="sourceLineNo">451</span> scheduler.addBack(proc);<a name="line.451"></a>
+<span class="sourceLineNo">452</span> break;<a name="line.452"></a>
+<span class="sourceLineNo">453</span> case ROLLEDBACK:<a name="line.453"></a>
+<span class="sourceLineNo">454</span> case INITIALIZING:<a name="line.454"></a>
+<span class="sourceLineNo">455</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.455"></a>
+<span class="sourceLineNo">456</span> LOG.error(msg);<a name="line.456"></a>
+<span class="sourceLineNo">457</span> throw new UnsupportedOperationException(msg);<a name="line.457"></a>
+<span class="sourceLineNo">458</span> default:<a name="line.458"></a>
+<span class="sourceLineNo">459</span> break;<a name="line.459"></a>
+<span class="sourceLineNo">460</span> }<a name="line.460"></a>
+<span class="sourceLineNo">461</span> }<a name="line.461"></a>
+<span class="sourceLineNo">462</span><a name="line.462"></a>
+<span class="sourceLineNo">463</span> // 3. Validate the stacks<a name="line.463"></a>
+<span class="sourceLineNo">464</span> int corruptedCount = 0;<a name="line.464"></a>
+<span class="sourceLineNo">465</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.465"></a>
+<span class="sourceLineNo">466</span> while (itStack.hasNext()) {<a name="line.466"></a>
+<span class="sourceLineNo">467</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.467"></a>
+<span class="sourceLineNo">468</span> RootProcedureState procStack = entry.getValue();<a name="line.468"></a>
+<span class="sourceLineNo">469</span> if (procStack.isValid()) continue;<a name="line.469"></a>
+<span class="sourceLineNo">470</span><a name="line.470"></a>
+<span class="sourceLineNo">471</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.471"></a>
+<span class="sourceLineNo">472</span> LOG.error("Corrupted " + proc);<a name="line.472"></a>
+<span class="sourceLineNo">473</span> procedures.remove(proc.getProcId());<a name="line.473"></a>
+<span class="sourceLineNo">474</span> runnableList.remove(proc);<a name="line.474"></a>
+<span class="sourceLineNo">475</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.475"></a>
+<span class="sourceLineNo">476</span> corruptedCount++;<a name="line.476"></a>
+<span class="sourceLineNo">477</span> }<a name="line.477"></a>
+<span class="sourceLineNo">478</span> itStack.remove();<a name="line.478"></a>
+<span class="sourceLineNo">479</span> }<a name="line.479"></a>
+<span class="sourceLineNo">480</span><a name="line.480"></a>
+<span class="sourceLineNo">481</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.481"></a>
+<span class="sourceLineNo">482</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.482"></a>
+<span class="sourceLineNo">483</span> }<a name="line.483"></a>
+<span class="sourceLineNo">484</span><a name="line.484"></a>
+<span class="sourceLineNo">485</span> // 4. Push the procedures to the timeout executor<a name="line.485"></a>
+<span class="sourceLineNo">486</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.486"></a>
+<span class="sourceLineNo">487</span> for (Procedure proc: waitingSet) {<a name="line.487"></a>
+<span class="sourceLineNo">488</span> proc.afterReplay(getEnvironment());<a name="line.488"></a>
+<span class="sourceLineNo">489</span> timeoutExecutor.add(proc);<a name="line.489"></a>
+<span class="sourceLineNo">490</span> }<a name="line.490"></a>
+<span class="sourceLineNo">491</span> }<a name="line.491"></a>
+<span class="sourceLineNo">492</span><a name="line.492"></a>
+<span class="sourceLineNo">493</span> // 5. Push the procedure to the scheduler<a name="line.493"></a>
+<span class="sourceLineNo">494</span> if (!runnableList.isEmpty()) {<a name="line.494"></a>
+<span class="sourceLineNo">495</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.495"></a>
+<span class="sourceLineNo">496</span> // some procedure may be started way before this stuff.<a name="line.496"></a>
+<span class="sourceLineNo">497</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.497"></a>
+<span class="sourceLineNo">498</span> Procedure proc = runnableList.get(i);<a name="line.498"></a>
+<span class="sourceLineNo">499</span> proc.afterReplay(getEnvironment());<a name="line.499"></a>
+<span class="sourceLineNo">500</span> if (!proc.hasParent()) {<a name="line.500"></a>
+<span class="sourceLineNo">501</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.501"></a>
+<span class="sourceLineNo">502</span> }<a name="line.502"></a>
+<span class="sourceLineNo">503</span> if (proc.wasExecuted()) {<a name="line.503"></a>
+<span class="sourceLineNo">504</span> scheduler.addFront(proc);<a name="line.504"></a>
+<span class="sourceLineNo">505</span> } else {<a name="line.505"></a>
+<span class="sourceLineNo">506</span> // if it was not in execution, it can wait.<a name="line.506"></a>
+<span class="sourceLineNo">507</span> scheduler.addBack(proc);<a name="line.507"></a>
+<span class="sourceLineNo">508</span> }<a name="line.508"></a>
+<span class="sourceLineNo">509</span> }<a name="line.509"></a>
+<span class="sourceLineNo">510</span> }<a name="line.510"></a>
+<span class="sourceLineNo">511</span> }<a name="line.511"></a>
<span class="sourceLineNo">512</span><a name="line.512"></a>
-<span class="sourceLineNo">513</span> // Create the Thread Group for the executors<a name="line.513"></a>
-<span class="sourceLineNo">514</span> threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.514"></a>
-<span class="sourceLineNo">515</span><a name="line.515"></a>
-<span class="sourceLineNo">516</span> // Create the timeout executor<a name="line.516"></a>
-<span class="sourceLineNo">517</span> timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.517"></a>
-<span class="sourceLineNo">518</span><a name="line.518"></a>
-<span class="sourceLineNo">519</span> // Create the workers<a name="line.519"></a>
-<span class="sourceLineNo">520</span> workerId.set(0);<a name="line.520"></a>
-<span class="sourceLineNo">521</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.521"></a>
-<span class="sourceLineNo">522</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.522"></a>
-<span class="sourceLineNo">523</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.523"></a>
-<span class="sourceLineNo">524</span> }<a name="line.524"></a>
-<span class="sourceLineNo">525</span><a name="line.525"></a>
-<span class="sourceLineNo">526</span> long st, et;<a name="line.526"></a>
-<span class="sourceLineNo">527</span><a name="line.527"></a>
-<span class="sourceLineNo">528</span> // Acquire the store lease.<a name="line.528"></a>
-<span class="sourceLineNo">529</span> st = EnvironmentEdgeManager.currentTime();<a name="line.529"></a>
-<span class="sourceLineNo">530</span> store.recoverLease();<a name="line.530"></a>
-<span class="sourceLineNo">531</span> et = EnvironmentEdgeManager.currentTime();<a name="line.531"></a>
-<span class="sourceLineNo">532</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.532"></a>
-<span class="sourceLineNo">533</span> StringUtils.humanTimeDiff(et - st));<a name="line.533"></a>
-<span class="sourceLineNo">534</span><a name="line.534"></a>
-<span class="sourceLineNo">535</span> // start the procedure scheduler<a name="line.535"></a>
-<span class="sourceLineNo">536</span> scheduler.start();<a name="line.536"></a>
-<span class="sourceLineNo">537</span><a name="line.537"></a>
-<span class="sourceLineNo">538</span> // TODO: Split in two steps.<a name="line.538"></a>
-<span class="sourceLineNo">539</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.539"></a>
-<span class="sourceLineNo">540</span> // The first one will make sure that we have the latest id,<a name="line.540"></a>
-<span class="sourceLineNo">541</span> // so we can start the threads and accept new procedures.<a name="line.541"></a>
-<span class="sourceLineNo">542</span> // The second step will do the actual load of old procedures.<a name="line.542"></a>
-<span class="sourceLineNo">543</span> st = EnvironmentEdgeManager.currentTime();<a name="line.543"></a>
-<span class="sourceLineNo">544</span> load(abortOnCorruption);<a name="line.544"></a>
-<span class="sourceLineNo">545</span> et = EnvironmentEdgeManager.currentTime();<a name="line.545"></a>
-<span class="sourceLineNo">546</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.546"></a>
-<span class="sourceLineNo">547</span> StringUtils.humanTimeDiff(et - st));<a name="line.547"></a>
-<span class="sourceLineNo">548</span><a name="line.548"></a>
-<span class="sourceLineNo">549</span> // Start the executors. Here we must have the lastProcId set.<a name="line.549"></a>
-<span class="sourceLineNo">550</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.550"></a>
-<span class="sourceLineNo">551</span> timeoutExecutor.start();<a name="line.551"></a>
-<span class="sourceLineNo">552</span> for (WorkerThread worker: workerThreads) {<a name="line.552"></a>
-<span class="sourceLineNo">553</span> worker.start();<a name="line.553"></a>
-<span class="sourceLineNo">554</span> }<a name="line.554"></a>
-<span class="sourceLineNo">555</span><a name="line.555"></a>
-<span class="sourceLineNo">556</span> // Internal chores<a name="line.556"></a>
-<span class="sourceLineNo">557</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.557"></a>
-<span class="sourceLineNo">558</span><a name="line.558"></a>
-<span class="sourceLineNo">559</span> // Add completed cleaner chore<a name="line.559"></a>
-<span class="sourceLineNo">560</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.560"></a>
-<span class="sourceLineNo">561</span> }<a name="line.561"></a>
-<span class="sourceLineNo">562</span><a name="line.562"></a>
-<span class="sourceLineNo">563</span> public void stop() {<a name="line.563"></a>
-<span class="sourceLineNo">564</span> if (!running.getAndSet(false)) {<a name="line.564"></a>
-<span class="sourceLineNo">565</span> return;<a name="line.565"></a>
-<span class="sourceLineNo">566</span> }<a name="line.566"></a>
-<span class="sourceLineNo">567</span><a name="line.567"></a>
-<span class="sourceLineNo">568</span> LOG.info("Stopping");<a name="line.568"></a>
-<span class="sourceLineNo">569</span> scheduler.stop();<a name="line.569"></a>
-<span class="sourceLineNo">570</span> timeoutExecutor.sendStopSignal();<a name="line.570"></a>
-<span class="sourceLineNo">571</span> }<a name="line.571"></a>
-<span class="sourceLineNo">572</span><a name="line.572"></a>
-<span class="sourceLineNo">573</span> @VisibleForTesting<a name="line.573"></a>
-<span class="sourceLineNo">574</span> public void join() {<a name="line.574"></a>
-<span class="sourceLineNo">575</span> assert !isRunning() : "expected not running";<a name="line.575"></a>
-<span class="sourceLineNo">576</span><a name="line.576"></a>
-<span class="sourceLineNo">577</span> // stop the timeout executor<a name="line.577"></a>
-<span class="sourceLineNo">578</span> timeoutExecutor.awaitTermination();<a name="line.578"></a>
-<span class="sourceLineNo">579</span> timeoutExecutor = null;<a name="line.579"></a>
-<span class="sourceLineNo">580</span><a name="line.580"></a>
-<span class="sourceLineNo">581</span> // stop the worker threads<a name="line.581"></a>
-<span class="sourceLineNo">582</span> for (WorkerThread worker: workerThreads) {<a name="line.582"></a>
-<span class="sourceLineNo">583</span> worker.awaitTermination();<a name="line.583"></a>
-<span class="sourceLineNo">584</span> }<a name="line.584"></a>
-<span class="sourceLineNo">585</span> workerThreads = null;<a name="line.585"></a>
-<span class="sourceLineNo">586</span><a name="line.586"></a>
-<span class="sourceLineNo">587</span> // Destroy the Thread Group for the executors<a name="line.587"></a>
-<span class="sourceLineNo">588</span> try {<a name="line.588"></a>
-<span class="sourceLineNo">589</span> threadGroup.destroy();<a name="line.589"></a>
-<span class="sourceLineNo">590</span> } catch (IllegalThreadStateException e) {<a name="line.590"></a>
-<span class="sourceLineNo">591</span> LOG.error("ThreadGroup " + threadGroup + " contains running threads; " + e.getMessage());<a name="line.591"></a>
-<span class="sourceLineNo">592</span> threadGroup.list();<a name="line.592"></a>
-<span class="sourceLineNo">593</span> } finally {<a name="line.593"></a>
-<span class="sourceLineNo">594</span> threadGroup = null;<a name="line.594"></a>
-<span class="sourceLineNo">595</span> }<a name="line.595"></a>
+<span class="sourceLineNo">513</span> /**<a name="line.513"></a>
+<span class="sourceLineNo">514</span> * Start the procedure executor.<a name="line.514"></a>
+<span class="sourceLineNo">515</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.515"></a>
+<span class="sourceLineNo">516</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.516"></a>
+<span class="sourceLineNo">517</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.517"></a>
+<span class="sourceLineNo">518</span> *<a name="line.518"></a>
+<span class="sourceLineNo">519</span> * @param numThreads number of threads available for procedure execution.<a name="line.519"></a>
+<span class="sourceLineNo">520</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.520"></a>
+<span class="sourceLineNo">521</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.521"></a>
+<span class="sourceLineNo">522</span> */<a name="line.522"></a>
+<span class="sourceLineNo">523</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.523"></a>
+<span class="sourceLineNo">524</span> if (!running.compareAndSet(false, true)) {<a name="line.524"></a>
+<span class="sourceLineNo">525</span> LOG.warn("Already running");<a name="line.525"></a>
+<span class="sourceLineNo">526</span> return;<a name="line.526"></a>
+<span class="sourceLineNo">527</span> }<a name="line.527"></a>
+<span class="sourceLineNo">528</span><a name="line.528"></a>
+<span class="sourceLineNo">529</span> // We have numThreads executor + one timer thread used for timing out<a name="line.529"></a>
+<span class="sourceLineNo">530</span> // procedures and triggering periodic procedures.<a name="line.530"></a>
+<span class="sourceLineNo">531</span> this.corePoolSize = numThreads;<a name="line.531"></a>
+<span class="sourceLineNo">532</span> this.maxPoolSize = 10 * numThreads;<a name="line.532"></a>
+<span class="sourceLineNo">533</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.533"></a>
+<span class="sourceLineNo">534</span> corePoolSize, maxPoolSize);<a name="line.534"></a>
+<span class="sourceLineNo">535</span><a name="line.535"></a>
+<span class="sourceLineNo">536</span> this.threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.536"></a>
+<span class="sourceLineNo">537</span> this.timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.537"></a>
+<span class="sourceLineNo">538</span><a name="line.538"></a>
+<span class="sourceLineNo">539</span> // Create the workers<a name="line.539"></a>
+<span class="sourceLineNo">540</span> workerId.set(0);<a name="line.540"></a>
+<span class="sourceLineNo">541</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.541"></a>
+<span class="sourceLineNo">542</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.542"></a>
+<span class="sourceLineNo">543</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.543"></a>
+<span class="sourceLineNo">544</span> }<a name="line.544"></a>
+<span class="sourceLineNo">545</span><a name="line.545"></a>
+<span class="sourceLineNo">546</span> long st, et;<a name="line.546"></a>
+<span class="sourceLineNo">547</span><a name="line.547"></a>
+<span class="sourceLineNo">548</span> // Acquire the store lease.<a name="line.548"></a>
+<span class="sourceLineNo">549</span> st = EnvironmentEdgeManager.currentTime();<a name="line.549"></a>
+<span class="sourceLineNo">550</span> store.recoverLease();<a name="line.550"></a>
+<span class="sourceLineNo">551</span> et = EnvironmentEdgeManager.currentTime();<a name="line.551"></a>
+<span class="sourceLineNo">552</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.552"></a>
+<span class="sourceLineNo">553</span> StringUtils.humanTimeDiff(et - st));<a name="line.553"></a>
+<span class="sourceLineNo">554</span><a name="line.554"></a>
+<span class="sourceLineNo">555</span> // start the procedure scheduler<a name="line.555"></a>
+<span class="sourceLineNo">556</span> scheduler.start();<a name="line.556"></a>
+<span class="sourceLineNo">557</span><a name="line.557"></a>
+<span class="sourceLineNo">558</span> // TODO: Split in two steps.<a name="line.558"></a>
+<span class="sourceLineNo">559</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.559"></a>
+<span class="sourceLineNo">560</span> // The first one will make sure that we have the latest id,<a name="line.560"></a>
+<span class="sourceLineNo">561</span> // so we can start the threads and accept new procedures.<a name="line.561"></a>
+<span class="sourceLineNo">562</span> // The second step will do the actual load of old procedures.<a name="line.562"></a>
+<span class="sourceLineNo">563</span> st = EnvironmentEdgeManager.currentTime();<a name="line.563"></a>
+<span class="sourceLineNo">564</span> load(abortOnCorruption);<a name="line.564"></a>
+<span class="sourceLineNo">565</span> et = EnvironmentEdgeManager.currentTime();<a name="line.565"></a>
+<span class="sourceLineNo">566</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.566"></a>
+<span class="sourceLineNo">567</span> StringUtils.humanTimeDiff(et - st));<a name="line.567"></a>
+<span class="sourceLineNo">568</span><a name="line.568"></a>
+<span class="sourceLineNo">569</span> // Start the executors. Here we must have the lastProcId set.<a name="line.569"></a>
+<span class="sourceLineNo">570</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.570"></a>
+<span class="sourceLineNo">571</span> timeoutExecutor.start();<a name="line.571"></a>
+<span class="sourceLineNo">572</span> for (WorkerThread worker: workerThreads) {<a name="line.572"></a>
+<span class="sourceLineNo">573</span> worker.start();<a name="line.573"></a>
+<span class="sourceLineNo">574</span> }<a name="line.574"></a>
+<span class="sourceLineNo">575</span><a name="line.575"></a>
+<span class="sourceLineNo">576</span> // Internal chores<a name="line.576"></a>
+<span class="sourceLineNo">577</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.577"></a>
+<span class="sourceLineNo">578</span><a name="line.578"></a>
+<span class="sourceLineNo">579</span> // Add completed cleaner chore<a name="line.579"></a>
+<span class="sourceLineNo">580</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.580"></a>
+<span class="sourceLineNo">581</span> }<a name="line.581"></a>
+<span class="sourceLineNo">582</span><a name="line.582"></a>
+<span class="sourceLineNo">583</span> public void stop() {<a name="line.583"></a>
+<span class="sourceLineNo">584</span> if (!running.getAndSet(false)) {<a name="line.584"></a>
+<span class="sourceLineNo">585</span> return;<a name="line.585"></a>
+<span class="sourceLineNo">586</span> }<a name="line.586"></a>
+<span class="sourceLineNo">587</span><a name="line.587"></a>
+<span class="sourceLineNo">588</span> LOG.info("Stopping");<a name="line.588"></a>
+<span class="sourceLineNo">589</span> scheduler.stop();<a name="line.589"></a>
+<span class="sourceLineNo">590</span> timeoutExecutor.sendStopSignal();<a name="line.590"></a>
+<span class="sourceLineNo">591</span> }<a name="line.591"></a>
+<span class="sourceLineNo">592</span><a name="line.592"></a>
+<span class="sourceLineNo">593</span> @VisibleForTesting<a name="line.593"></a>
+<span class="sourceLineNo">594</span> public void join() {<a name="line.594"></a>
+<span class="sourceLineNo">595</span> assert !isRunning() : "expected not running";<a name="line.595"></a>
<span class="sourceLineNo">596</span><a name="line.596"></a>
-<span class="sourceLineNo">597</span> // reset the in-memory state for testing<a name="line.597"></a>
-<span class="sourceLineNo">598</span> completed.clear();<a name="line.598"></a>
-<span class="sourceLineNo">599</span> rollbackStack.clear();<a name="line.599"></a>
-<span class="sourceLineNo">600</span> procedures.clear();<a name="line.600"></a>
-<span class="sourceLineNo">601</span> nonceKeysToProcIdsMap.clear();<a name="line.601"></a>
-<span class="sourceLineNo">602</span> scheduler.clear();<a name="line.602"></a>
-<span class="sourceLineNo">603</span> lastProcId.set(-1);<a name="line.603"></a>
-<span class="sourceLineNo">604</span> }<a name="line.604"></a>
-<span class="sourceLineNo">605</span><a name="line.605"></a>
-<span class="sourceLineNo">606</span> public void refreshConfiguration(final Configuration conf) {<a name="line.606"></a>
-<span class="sourceLineNo">607</span> this.conf = conf;<a name="line.607"></a>
-<span class="sourceLineNo">608</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.608"></a>
-<span class="sourceLineNo">609</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.609"></a>
-<span class="sourceLineNo">610</span> }<a name="line.610"></a>
-<span class="sourceLineNo">611</span><a name="line.611"></a>
-<span class="sourceLineNo">612</span> // ==========================================================================<a name="line.612"></a>
-<span class="sourceLineNo">613</span> // Accessors<a name="line.613"></a>
-<span class="sourceLineNo">614</span> // ==========================================================================<a name="line.614"></a>
-<span class="sourceLineNo">615</span> public boolean isRunning() {<a name="line.615"></a>
-<span class="sourceLineNo">616</span> return running.get();<a name="line.616"></a>
-<span class="sourceLineNo">617</span> }<a name="line.617"></a>
-<span class="sourceLineNo">618</span><a name="line.618"></a>
-<span class="sourceLineNo">619</span> /**<a name="line.619"></a>
-<span class="sourceLineNo">620</span> * @return the current number of worker threads.<a name="line.620"></a>
-<span class="sourceLineNo">621</span> */<a name="line.621"></a>
-<span class="sourceLineNo">622</span> public int getWorkerThreadCount() {<a name="line.622"></a>
-<span class="sourceLineNo">623</span> return workerThreads.size();<a name="line.623"></a>
-<span class="sourceLineNo">624</span> }<a name="line.624"></a>
-<span class="sourceLineNo">625</span><a name="line.625"></a>
-<span class="sourceLineNo">626</span> /**<a name="line.626"></a>
-<span class="sourceLineNo">627</span> * @return the core pool size settings.<a name="line.627"></a>
-<span class="sourceLineNo">628</span> */<a name="line.628"></a>
-<span class="sourceLineNo">629</span> public int getCorePoolSize() {<a name="line.629"></a>
-<span class="sourceLineNo">630</span> return corePoolSize;<a name="line.630"></a>
-<span class="sourceLineNo">631</span> }<a name="line.631"></a>
-<span class="sourceLineNo">632</span><a name="line.632"></a>
-<span class="sourceLineNo">633</span> public int getActiveExecutorCount() {<a name="line.633"></a>
-<span class="sourceLineNo">634</span> return activeExecutorCount.get();<a name="line.634"></a>
-<span class="sourceLineNo">635</span> }<a name="line.635"></a>
-<span class="sourceLineNo">636</span><a name="line.636"></a>
-<span class="sourceLineNo">637</span> public TEnvironment getEnvironment() {<a name="line.637"></a>
-<span class="sourceLineNo">638</span> return this.environment;<a name="line.638"></a>
-<span class="sourceLineNo">639</span> }<a name="line.639"></a>
-<span class="sourceLineNo">640</span><a name="line.640"></a>
-<span class="sourceLineNo">641</span> public ProcedureStore getStore() {<a name="line.641"></a>
-<span class="sourceLineNo">642</span> return this.store;<a name="line.642"></a>
+<span class="sourceLineNo">597</span> // stop the timeout executor<a name="line.597"></a>
+<span class="sourceLineNo">598</span> timeoutExecutor.awaitTermination();<a name="line.598"></a>
+<span class="sourceLineNo">599</span><a name="line.599"></a>
+<span class="sourceLineNo">600</span> // stop the worker threads<a name="line.600"></a>
+<span class="sourceLineNo">601</span> for (WorkerThread worker: workerThreads) {<a name="line.601"></a>
+<span class="sourceLineNo">602</span> worker.awaitTermination();<a name="line.602"></a>
+<span class="sourceLineNo">603</span> }<a name="line.603"></a>
+<span class="sourceLineNo">604</span><a name="line.604"></a>
+<span class="sourceLineNo">605</span> // Destroy the Thread Group for the executors<a name="line.605"></a>
+<span class="sourceLineNo">606</span> // TODO: Fix. #join is not place to destroy resources.<a name="line.606"></a>
+<span class="sourceLineNo">607</span> try {<a name="line.607"></a>
+<span class="sourceLineNo">608</span> threadGroup.destroy();<a name="line.608"></a>
+<span class="sourceLineNo">609</span> } catch (IllegalThreadStateException e) {<a name="line.609"></a>
+<span class="sourceLineNo">610</span> LOG.error("ThreadGroup {} contains running threads; {}: See STDOUT",<a name="line.610"></a>
+<span class="sourceLineNo">611</span> this.threadGroup, e.getMessage());<a name="line.611"></a>
+<span class="sourceLineNo">612</span> // This dumps list of threads on STDOUT.<a name="line.612"></a>
+<span class="sourceLineNo">613</span> this.threadGroup.list();<a name="line.613"></a>
+<span class="sourceLineNo">614</span> }<a name="line.614"></a>
+<span class="sourceLineNo">615</span><a name="line.615"></a>
+<span class="sourceLineNo">616</span> // reset the in-memory state for testing<a name="line.616"></a>
+<span class="sourceLineNo">617</span> completed.clear();<a name="line.617"></a>
+<span class="sourceLineNo">618</span> rollbackStack.clear();<a name="line.618"></a>
+<span class="sourceLineNo">619</span> procedures.clear();<a name="line.619"></a>
+<span class="sourceLineNo">620</span> nonceKeysToProcIdsMap.clear();<a name="line.620"></a>
+<span class="sourceLineNo">621</span> scheduler.clear();<a name="line.621"></a>
+<span class="sourceLineNo">622</span> lastProcId.set(-1);<a name="line.622"></a>
+<span class="sourceLineNo">623</span> }<a name="line.623"></a>
+<span class="sourceLineNo">624</span><a name="line.624"></a>
+<span class="sourceLineNo">625</span> public void refreshConfiguration(final Configuration conf) {<a name="line.625"></a>
+<span class="sourceLineNo">626</span> this.conf = conf;<a name="line.626"></a>
+<span class="sourceLineNo">627</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.627"></a>
+<span class="sourceLineNo">628</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.628"></a>
+<span class="sourceLineNo">629</span> }<a name="line.629"></a>
+<span class="sourceLineNo">630</span><a name="line.630"></a>
+<span class="sourceLineNo">631</span> // ==========================================================================<a name="line.631"></a>
+<span class="sourceLineNo">632</span> // Accessors<a name="line.632"></a>
+<span class="sourceLineNo">633</span> // ==========================================================================<a name="line.633"></a>
+<span class="sourceLineNo">634</span> public boolean isRunning() {<a name="line.634"></a>
+<span class="sourceLineNo">635</span> return running.get();<a name="line.635"></a>
+<span class="sourceLineNo">636</span> }<a name="line.636"></a>
+<span class="sourceLineNo">637</span><a name="line.637"></a>
+<span class="sourceLineNo">638</span> /**<a name="line.638"></a>
+<span class="sourceLineNo">639</span> * @return the current number of worker threads.<a name="line.639"></a>
+<span class="sourceLineNo">640</span> */<a name="line.640"></a>
+<span class="sourceLineNo">641</span> public int getWorkerThreadCount() {<a name="line.641"></a>
+<span class="sourceLineNo">642</span> return workerThreads.size();<a name="line.642"></a>
<span class="sourceLineNo">643</span> }<a name="line.643"></a>
<span class="sourceLineNo">644</span><a name="line.644"></a>
-<span class="sourceLineNo">645</span> ProcedureScheduler getScheduler() {<a name="line.645"></a>
-<span class="sourceLineNo">646</span> return scheduler;<a name="line.646"></a>
-<span class="sourceLineNo">647</span> }<a name="line.647"></a>
-<span class="sourceLineNo">648</span><a name="line.648"></a>
-<span class="sourceLineNo">649</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.649"></a>
-<span class="sourceLineNo">650</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.650"></a>
-<span class="sourceLineNo">651</span> this.scheduler.signalAll();<a name="line.651"></a>
-<span class="sourceLineNo">652</span> }<a name="line.652"></a>
-<span class="sourceLineNo">653</span><a name="line.653"></a>
-<span class="sourceLineNo">654</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.654"></a>
-<span class="sourceLineNo">655</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.655"></a>
-<span class="sourceLineNo">656</span> }<a name="line.656"></a>
-<span class="sourceLineNo">657</span><a name="line.657"></a>
-<span class="sourceLineNo">658</span> // ==========================================================================<a name="line.658"></a>
-<span class="sourceLineNo">659</span> // Submit/Remove Chores<a name="line.659"></a>
-<span class="sourceLineNo">660</span> // ==========================================================================<a name="line.660"></a>
-<span class="sourceLineNo">661</span><a name="line.661"></a>
-<span class="sourceLineNo">662</span> /**<a name="line.662"></a>
-<span class="sourceLineNo">663</span> * Add a chore procedure to the executor<a name="line.663"></a>
-<span class="sourceLineNo">664</span> * @param chore the chore to add<a name="line.664"></a>
-<span class="sourceLineNo">665</span> */<a name="line.665"></a>
-<span class="sourceLineNo">666</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.666"></a>
-<span class="sourceLineNo">667</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.667"></a>
-<span class="sourceLineNo">668</span> timeoutExecutor.add(chore);<a name="line.668"></a>
-<span class="sourceLineNo">669</span> }<a name="line.669"></a>
-<span class="sourceLineNo">670</span><a name="line.670"></a>
-<span class="sourceLineNo">671</span> /**<a name="line.671"></a>
-<span class="sourceLineNo">672</span> * Remove a chore procedure from the executor<a name="line.672"></a>
-<span class="sourceLineNo">673</span> * @param chore the chore to remove<a name="line.673"></a>
-<span class="sourceLineNo">674</span> * @return whether the chore is removed, or it will be removed later<a name="line.674"></a>
-<span class="sourceLineNo">675</span> */<a name="line.675"></a>
-<span class="sourceLineNo">676</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.676"></a>
-<span class="sourceLineNo">677</span> chore.setState(ProcedureState.SUCCESS);<a name="line.677"></a>
-<span class="sourceLineNo">678</span> return timeoutExecutor.remove(chore);<a name="line.678"></a>
-<span class="sourceLineNo">679</span> }<a name="line.679"></a>
+<span class="sourceLineNo">645</span> /**<a name="line.645"></a>
+<span class="sourceLineNo">646</span> * @return the core pool size settings.<a name="line.646"></a>
+<span class="sourceLineNo">647</span> */<a name="line.647"></a>
+<span class="sourceLineNo">648</span> public int getCorePoolSize() {<a name="line.648"></a>
+<span class="sourceLineNo">649</span> return corePoolSize;<a name="line.649"></a>
+<span class="sourceLineNo">650</span> }<a name="line.650"></a>
+<span class="sourceLineNo">651</span><a name="line.651"></a>
+<span class="sourceLineNo">652</span> public int getActiveExecutorCount() {<a name="line.652"></a>
+<span class="sourceLineNo">653</span> return activeExecutorCount.get();<a name="line.653"></a>
+<span class="sourceLineNo">654</span> }<a name="line.654"></a>
+<span class="sourceLineNo">655</span><a name="line.655"></a>
+<span class="sourceLineNo">656</span> public TEnvironment getEnvironment() {<a name="line.656"></a>
+<span class="sourceLineNo">657</span> return this.environment;<a name="line.657"></a>
+<span class="sourceLineNo">658</span> }<a name="line.658"></a>
+<span class="sourceLineNo">659</span><a name="line.659"></a>
+<span class="sourceLineNo">660</span> public ProcedureStore getStore() {<a name="line.660"></a>
+<span class="sourceLineNo">661</span> return this.store;<a name="line.661"></a>
+<span class="sourceLineNo">662</span> }<a name="line.662"></a>
+<span class="sourceLineNo">663</span><a name="line.663"></a>
+<span class="sourceLineNo">664</span> ProcedureScheduler getScheduler() {<a name="line.664"></a>
+<span class="sourceLineNo">665</span> return scheduler;<a name="line.665"></a>
+<span class="sourceLineNo">666</span> }<a name="line.666"></a>
+<span class="sourceLineNo">667</span><a name="line.667"></a>
+<span class="sourceLineNo">668</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.668"></a>
+<span class="sourceLineNo">669</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.669"></a>
+<span class="sourceLineNo">670</span> this.scheduler.signalAll();<a name="line.670"></a>
+<span class="sourceLineNo">671</span> }<a name="line.671"></a>
+<span class="sourceLineNo">672</span><a name="line.672"></a>
+<span class="sourceLineNo">673</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.673"></a>
+<span class="sourceLineNo">674</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.674"></a>
+<span class="sourceLineNo">675</span> }<a name="line.675"></a>
+<span class="sourceLineNo">676</span><a name="line.676"></a>
+<span class="sourceLineNo">677</span> // ==========================================================================<a name="line.677"></a>
+<span class="sourceLineNo">678</span> // Submit/Remove Chores<a name="line.678"></a>
+<span class="sourceLineNo">679</span> // ==========================================================================<a name="line.679"></a>
<span class="sourceLineNo">680</span><a name="line.680"></a>
-<span class="sourceLineNo">681</span> // ==========================================================================<a name="line.681"></a>
-<span class="sourceLineNo">682</span> // Nonce Procedure helpers<a name="line.682"></a>
-<span class="sourceLineNo">683</span> // ==========================================================================<a name="line.683"></a>
-<span class="sourceLineNo">684</span> /**<a name="line.684"></a>
-<span class="sourceLineNo">685</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.685"></a>
-<span class="sourceLineNo">686</span> * @param nonceGroup<a name="line.686"></a>
-<span class="sourceLineNo">687</span> * @param nonce<a name="line.687"></a>
-<span class="sourceLineNo">688</span> * @return the generated NonceKey<a name="line.688"></a>
-<span class="sourceLineNo">689</span> */<a name="line.689"></a>
-<span class="sourceLineNo">690</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.690"></a>
-<span class="sourceLineNo">691</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.691"></a>
-<span class="sourceLineNo">692</span> }<a name="line.692"></a>
-<span class="sourceLineNo">693</span><a name="line.693"></a>
-<span class="sourceLineNo">694</span> /**<a name="line.694"></a>
-<span class="sourceLineNo">695</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.695"></a>
-<span class="sourceLineNo">696</span> * A procId will be reserved and on submitProcedure(),<a name="line.696"></a>
-<span class="sourceLineNo">697</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.697"></a>
-<span class="sourceLineNo">698</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.698"></a>
-<span class="sourceLineNo">699</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.699"></a>
-<span class="sourceLineNo">700</span> * and submit the procedure.<a name="line.700"></a>
-<span class="sourceLineNo">701</span> *<a name="line.701"></a>
-<span class="sourceLineNo">702</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.702"></a>
-<span class="sourceLineNo">703</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.703"></a>
-<span class="sourceLineNo">704</span> */<a name="line.704"></a>
-<span class="sourceLineNo">705</span> public long registerNonce(final NonceKey nonceKey) {<a name="line.705"></a>
-<span class="sourceLineNo">706</span> if (nonceKey == null) return -1;<a name="line.706"></a>
-<span class="sourceLineNo">707</span><a name="line.707"></a>
-<span class="sourceLineNo">708</span> // check if we have already a Reserved ID for the nonce<a name="line.708"></a>
-<span class="sourceLineNo">709</span> Long oldProcId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.709"></a>
-<span class="sourceLineNo">710</span> if (oldProcId == null) {<a name="line.710"></a>
-<span class="sourceLineNo">711</span> // reserve a new Procedure ID, this will be associated with the nonce<a name="line.711"></a>
-<span class="sourceLineNo">712</span> // and the procedure submitted with the specified nonce will use this ID.<a name="line.712"></a>
-<span class="sourceLineNo">713</span> final long newProcId = nextProcId();<a name="line.713"></a>
-<span class="sourceLineNo">714</span> oldProcId = nonceKeysToProcIdsMap.putIfAbsent(nonceKey, newProcId);<a name="line.714"></a>
-<span class="sourceLineNo">715</span> if (oldProcId == null) return -1;<a name="line.715"></a>
-<span class="sourceLineNo">716</span> }<a name="line.716"></a>
-<span class="sourceLineNo">717</span><a name="line.717"></a>
-<span class="sourceLineNo">718</span> // we found a registered nonce, but the procedure may not have been submitted yet.<a name="line.718"></a>
-<span class="sourceLineNo">719</span> // since the client expect the procedure to be submitted, spin here until it is.<a name="line.719"></a>
-<span class="sourceLineNo">720</span> final boolean traceEnabled = LOG.isTraceEnabled();<a name="line.720"></a>
-<span class="sourceLineNo">721</span> while (isRunning() &&<a name="line.721"></a>
-<span class="sourceLineNo">722</span> !(procedures.containsKey(oldProcId) || completed.containsKey(oldProcId)) &&<a name="line.722"></a>
-<span class="sourceLineNo">723</span> nonceKeysToProcIdsMap.containsKey(nonceKey)) {<a name="line.723"></a>
-<span class="sourceLineNo">724</span> if (traceEnabled) {<a name="line.724"></a>
-<span class="sourceLineNo">725</span> LOG.trace("Waiting for pid=" + oldProcId.longValue() + " to be submitted");<a name="line.725"></a>
-<span class="sourceLineNo">726</span> }<a name="line.726"></a>
-<span class="sourceLineNo">727</span> Threads.sleep(100);<a name="line.727"></a>
-<span class="sourceLineNo">728</span> }<a name="line.728"></a>
-<span class="sourceLineNo">729</span> return oldProcId.longValue();<a name="line.729"></a>
-<span class="sourceLineNo">730</span> }<a name="line.730"></a>
-<span class="sourceLineNo">731</span><a name="line.731"></a>
-<span class="sourceLineNo">732</span> /**<a name="line.732"></a>
-<span class="sourceLineNo">733</span> * Remove the NonceKey if the procedure was not submitted to the executor.<a name="line.733"></a>
-<span class="sourceLineNo">734</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.734"></a>
-<span class="sourceLineNo">735</span> */<a name="line.735"></a>
-<span class="sourceLineNo">736</span> public void unregisterNonceIfProcedureWasNotSubmitted(final NonceKey nonceKey) {<a name="line.736"></a>
-<span class="sourceLineNo">737</span> if (nonceKey == null) return;<a name="line.737"></a>
-<span class="sourceLineNo">738</span><a name="line.738"></a>
-<span class="sourceLineNo">739</span> final Long procId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.739"></a>
-<span class="sourceLineNo">740</span> if (procId == null) return;<a name="line.740"></a>
-<span class="sourceLineNo">741</span><a name="line.741"></a>
-<span class="sourceLineNo">742</span> // if the procedure was not submitted, remove the nonce<a name="line.742"></a>
-<span class="sourceLineNo">743</span> if (!(procedures.containsKey(procId) || completed.containsKey(procId))) {<a name="line.743"></a>
-<span class="sourceLineNo">744</span> nonceKeysToProcIdsMap.remove(nonceKey);<a name="line.744"></a>
-<span class="sourceLineNo">745</span> }<a name="line.745"></a>
-<span class="sourceLineNo">746</span> }<a name="line.746"></a>
-<span class="sourceLineNo">747</span><a name="line.747"></a>
-<span class="sourceLineNo">748</span> public static class FailedProcedure<TEnvironment> extends Procedure<TEnvironment> {<a name="line.748"></a>
-<span class="sourceLineNo">749</span> private String procName;<a name="line.749"></a>
+<span class="sourceLineNo">681</span> /**<a name="line.681"></a>
+<span class="sourceLineNo">682</span> * Add a chore procedure to the executor<a name="line.682"></a>
+<span class="sourceLineNo">683</span> * @param chore the chore to add<a name="line.683"></a>
+<span class="sourceLineNo">684</span> */<a name="line.684"></a>
+<span class="sourceLineNo">685</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.685"></a>
+<span class="sourceLineNo">686</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.686"></a>
+<span class="sourceLineNo">687</span> timeoutExecutor.add(chore);<a name="line.687"></a>
+<span class="sourceLineNo">688</span> }<a name="line.688"></a>
+<span class="sourceLineNo">689</span><a name="line.689"></a>
+<span class="sourceLineNo">690</span> /**<a name="line.690"></a>
+<span class="sourceLineNo">691</span> * Remove a chore procedure from the executor<a name="line.691"></a>
+<span class="sourceLineNo">692</span> * @param chore the chore to remove<a name="line.692"></a>
+<span class="sourceLineNo">693</span> * @return whether the chore is removed, or it will be removed later<a name="line.693"></a>
+<span class="sourceLineNo">694</span> */<a name="line.694"></a>
+<span class="sourceLineNo">695</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.695"></a>
+<span class="sourceLineNo">696</span> chore.setState(ProcedureState.SUCCESS);<a name="line.696"></a>
+<span class="sourceLineNo">697</span> return timeoutExecutor.remove(chore);<a name="line.697"></a>
+<span class="sourceLineNo">698</span> }<a name="line.698"></a>
+<span class="sourceLineNo">699</span><a name="line.699"></a>
+<span class="sourceLineNo">700</span> // ==========================================================================<a name="line.700"></a>
+<span class="sourceLineNo">701</span> // Nonce Procedure helpers<a name="line.701"></a>
+<span class="sourceLineNo">702</span> // ==========================================================================<a name="line.702"></a>
+<span class="sourceLineNo">703</span> /**<a name="line.703"></a>
+<span class="sourceLineNo">704</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.704"></a>
+<span class="sourceLineNo">705</span> * @param nonceGroup<a name="line.705"></a>
+<span class="sourceLineNo">706</span> * @param nonce<a name="line.706"></a>
+<span class="sourceLineNo">707</span> * @return the generated NonceKey<a name="line.707"></a>
+<span class="sourceLineNo">708</span> */<a name="line.708"></a>
+<span class="sourceLineNo">709</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.709"></a>
+<span class="sourceLineNo">710</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.710"></a>
+<span class="sourceLineNo">711</span> }<a name="line.711"></a>
+<span class="sourceLineNo">712</span><a name="line.712"></a>
+<span class="sourceLineNo">713</span> /**<a name="line.713"></a>
+<span class="sourceLineNo">714</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.714"></a>
+<span class="sourceLineNo">715</span> * A procId will be reserved and on submitProcedure(),<a name="line.715"></a>
+<span class="sourceLineNo">716</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.716"></a>
+<span class="sourceLineNo">717</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.717"></a>
+<span class="sourceLineNo">718</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.718"></a>
+<span class="sourceLineNo">719</span> * and submit the procedure.<a name="line.719"></a>
+<span class="sourceLineNo">720</span> *<a name="line.720"></a>
+<span class="sourceLineNo">721</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.721"></a>
+<span class="sourceLineNo">722</span> * @return the procId associated with the nonce, if any ot
<TRUNCATED>
[12/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionStat.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionStat.html b/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionStat.html
index 078e4b1..fe1e077 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionStat.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/master/assignment/AssignmentManager.RegionInTransitionStat.html
@@ -1265,700 +1265,701 @@
<span class="sourceLineNo">1257</span> } else if (localState == State.OFFLINE || regionInfo.isOffline()) {<a name="line.1257"></a>
<span class="sourceLineNo">1258</span> regionStates.addToOfflineRegions(regionNode);<a name="line.1258"></a>
<span class="sourceLineNo">1259</span> } else if (localState == State.CLOSED && getTableStateManager().<a name="line.1259"></a>
-<span class="sourceLineNo">1260</span> isTableState(regionNode.getTable(), TableState.State.DISABLED)) {<a name="line.1260"></a>
-<span class="sourceLineNo">1261</span> // The region is CLOSED and the table is DISABLED, there is nothing to schedule;<a name="line.1261"></a>
-<span class="sourceLineNo">1262</span> // the region is inert.<a name="line.1262"></a>
-<span class="sourceLineNo">1263</span> } else {<a name="line.1263"></a>
-<span class="sourceLineNo">1264</span> // These regions should have a procedure in replay<a name="line.1264"></a>
-<span class="sourceLineNo">1265</span> regionStates.addRegionInTransition(regionNode, null);<a name="line.1265"></a>
-<span class="sourceLineNo">1266</span> }<a name="line.1266"></a>
-<span class="sourceLineNo">1267</span> }<a name="line.1267"></a>
-<span class="sourceLineNo">1268</span> }<a name="line.1268"></a>
-<span class="sourceLineNo">1269</span> }<a name="line.1269"></a>
-<span class="sourceLineNo">1270</span> });<a name="line.1270"></a>
-<span class="sourceLineNo">1271</span><a name="line.1271"></a>
-<span class="sourceLineNo">1272</span> // every assignment is blocked until meta is loaded.<a name="line.1272"></a>
-<span class="sourceLineNo">1273</span> wakeMetaLoadedEvent();<a name="line.1273"></a>
-<span class="sourceLineNo">1274</span> }<a name="line.1274"></a>
-<span class="sourceLineNo">1275</span><a name="line.1275"></a>
-<span class="sourceLineNo">1276</span> /**<a name="line.1276"></a>
-<span class="sourceLineNo">1277</span> * Look at what is in meta and the list of servers that have checked in and make reconciliation.<a name="line.1277"></a>
-<span class="sourceLineNo">1278</span> * We cannot tell definitively the difference between a clean shutdown and a cluster that has<a name="line.1278"></a>
-<span class="sourceLineNo">1279</span> * been crashed down. At this stage of a Master startup, they look the same: they have the<a name="line.1279"></a>
-<span class="sourceLineNo">1280</span> * same state in hbase:meta. We could do detective work probing ZK and the FS for old WALs to<a name="line.1280"></a>
-<span class="sourceLineNo">1281</span> * split but SCP does this already so just let it do its job.<a name="line.1281"></a>
-<span class="sourceLineNo">1282</span> * <p>>The profiles of clean shutdown and cluster crash-down are the same because on clean<a name="line.1282"></a>
-<span class="sourceLineNo">1283</span> * shutdown currently, we do not update hbase:meta with region close state (In AMv2, region<a name="line.1283"></a>
-<span class="sourceLineNo">1284</span> * state is kept in hbse:meta). Usually the master runs all region transitions as of AMv2 but on<a name="line.1284"></a>
-<span class="sourceLineNo">1285</span> * cluster controlled shutdown, the RegionServers close all their regions only reporting the<a name="line.1285"></a>
-<span class="sourceLineNo">1286</span> * final change to the Master. Currently this report is ignored. Later we could take it and<a name="line.1286"></a>
-<span class="sourceLineNo">1287</span> * update as many regions as we can before hbase:meta goes down or have the master run the<a name="line.1287"></a>
-<span class="sourceLineNo">1288</span> * close of all regions out on the cluster but we may never be able to achieve the proper state on<a name="line.1288"></a>
-<span class="sourceLineNo">1289</span> * all regions (at least not w/o lots of painful manipulations and waiting) so clean shutdown<a name="line.1289"></a>
-<span class="sourceLineNo">1290</span> * might not be possible especially on big clusters.... And clean shutdown will take time. Given<a name="line.1290"></a>
-<span class="sourceLineNo">1291</span> * this current state of affairs, we just run ServerCrashProcedure in both cases. It will always<a name="line.1291"></a>
-<span class="sourceLineNo">1292</span> * do the right thing.<a name="line.1292"></a>
-<span class="sourceLineNo">1293</span> * @return True if for sure this is a failover where a Master is starting up into an already<a name="line.1293"></a>
-<span class="sourceLineNo">1294</span> * running cluster.<a name="line.1294"></a>
-<span class="sourceLineNo">1295</span> */<a name="line.1295"></a>
-<span class="sourceLineNo">1296</span> // The assumption here is that if RSs are crashing while we are executing this<a name="line.1296"></a>
-<span class="sourceLineNo">1297</span> // they will be handled by the SSH that are put in the ServerManager deadservers "queue".<a name="line.1297"></a>
-<span class="sourceLineNo">1298</span> private boolean processofflineServersWithOnlineRegions() {<a name="line.1298"></a>
-<span class="sourceLineNo">1299</span> boolean deadServers = !master.getServerManager().getDeadServers().isEmpty();<a name="line.1299"></a>
-<span class="sourceLineNo">1300</span> final Set<ServerName> offlineServersWithOnlineRegions = new HashSet<>();<a name="line.1300"></a>
-<span class="sourceLineNo">1301</span> int size = regionStates.getRegionStateNodes().size();<a name="line.1301"></a>
-<span class="sourceLineNo">1302</span> final List<RegionInfo> offlineRegionsToAssign = new ArrayList<>(size);<a name="line.1302"></a>
-<span class="sourceLineNo">1303</span> // If deadservers then its a failover, else, we are not sure yet.<a name="line.1303"></a>
-<span class="sourceLineNo">1304</span> boolean failover = deadServers;<a name="line.1304"></a>
-<span class="sourceLineNo">1305</span> for (RegionStateNode regionNode: regionStates.getRegionStateNodes()) {<a name="line.1305"></a>
-<span class="sourceLineNo">1306</span> // Region State can be OPEN even if we did controlled cluster shutdown; Master does not close<a name="line.1306"></a>
-<span class="sourceLineNo">1307</span> // the regions in this case. The RegionServer does the close so hbase:meta is state in<a name="line.1307"></a>
-<span class="sourceLineNo">1308</span> // hbase:meta is not updated -- Master does all updates -- and is left with OPEN as region<a name="line.1308"></a>
-<span class="sourceLineNo">1309</span> // state in meta. How to tell difference between ordered shutdown and crashed-down cluster<a name="line.1309"></a>
-<span class="sourceLineNo">1310</span> // then? We can't. Not currently. Perhaps if we updated hbase:meta with CLOSED on ordered<a name="line.1310"></a>
-<span class="sourceLineNo">1311</span> // shutdown. This would slow shutdown though and not all edits would make it in anyways.<a name="line.1311"></a>
-<span class="sourceLineNo">1312</span> // TODO: Examine.<a name="line.1312"></a>
-<span class="sourceLineNo">1313</span> // Because we can't be sure it an ordered shutdown, we run ServerCrashProcedure always.<a name="line.1313"></a>
-<span class="sourceLineNo">1314</span> // ServerCrashProcedure will try to retain old deploy when it goes to assign.<a name="line.1314"></a>
-<span class="sourceLineNo">1315</span> if (regionNode.getState() == State.OPEN) {<a name="line.1315"></a>
-<span class="sourceLineNo">1316</span> final ServerName serverName = regionNode.getRegionLocation();<a name="line.1316"></a>
-<span class="sourceLineNo">1317</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1317"></a>
-<span class="sourceLineNo">1318</span> offlineServersWithOnlineRegions.add(serverName);<a name="line.1318"></a>
-<span class="sourceLineNo">1319</span> } else {<a name="line.1319"></a>
-<span class="sourceLineNo">1320</span> // Server is online. This a failover. Master is starting into already-running cluster.<a name="line.1320"></a>
-<span class="sourceLineNo">1321</span> failover = true;<a name="line.1321"></a>
-<span class="sourceLineNo">1322</span> }<a name="line.1322"></a>
-<span class="sourceLineNo">1323</span> } else if (regionNode.getState() == State.OFFLINE) {<a name="line.1323"></a>
-<span class="sourceLineNo">1324</span> if (isTableEnabled(regionNode.getTable())) {<a name="line.1324"></a>
-<span class="sourceLineNo">1325</span> offlineRegionsToAssign.add(regionNode.getRegionInfo());<a name="line.1325"></a>
-<span class="sourceLineNo">1326</span> }<a name="line.1326"></a>
-<span class="sourceLineNo">1327</span> }<a name="line.1327"></a>
-<span class="sourceLineNo">1328</span> }<a name="line.1328"></a>
-<span class="sourceLineNo">1329</span> // Kill servers with online regions just-in-case. Runs ServerCrashProcedure.<a name="line.1329"></a>
-<span class="sourceLineNo">1330</span> for (ServerName serverName: offlineServersWithOnlineRegions) {<a name="line.1330"></a>
-<span class="sourceLineNo">1331</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1331"></a>
-<span class="sourceLineNo">1332</span> LOG.info("KILL RegionServer=" + serverName + " hosting regions but not online.");<a name="line.1332"></a>
-<span class="sourceLineNo">1333</span> killRegionServer(serverName);<a name="line.1333"></a>
-<span class="sourceLineNo">1334</span> }<a name="line.1334"></a>
-<span class="sourceLineNo">1335</span> }<a name="line.1335"></a>
-<span class="sourceLineNo">1336</span> setFailoverCleanupDone(true);<a name="line.1336"></a>
-<span class="sourceLineNo">1337</span><a name="line.1337"></a>
-<span class="sourceLineNo">1338</span> // Assign offline regions. Uses round-robin.<a name="line.1338"></a>
-<span class="sourceLineNo">1339</span> if (offlineRegionsToAssign.size() > 0) {<a name="line.1339"></a>
-<span class="sourceLineNo">1340</span> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().<a name="line.1340"></a>
-<span class="sourceLineNo">1341</span> createRoundRobinAssignProcedures(offlineRegionsToAssign));<a name="line.1341"></a>
-<span class="sourceLineNo">1342</span> }<a name="line.1342"></a>
-<span class="sourceLineNo">1343</span><a name="line.1343"></a>
-<span class="sourceLineNo">1344</span> return failover;<a name="line.1344"></a>
-<span class="sourceLineNo">1345</span> }<a name="line.1345"></a>
-<span class="sourceLineNo">1346</span><a name="line.1346"></a>
-<span class="sourceLineNo">1347</span> /**<a name="line.1347"></a>
-<span class="sourceLineNo">1348</span> * Used by ServerCrashProcedure to make sure AssignmentManager has completed<a name="line.1348"></a>
-<span class="sourceLineNo">1349</span> * the failover cleanup before re-assigning regions of dead servers. So that<a name="line.1349"></a>
-<span class="sourceLineNo">1350</span> * when re-assignment happens, AssignmentManager has proper region states.<a name="line.1350"></a>
-<span class="sourceLineNo">1351</span> */<a name="line.1351"></a>
-<span class="sourceLineNo">1352</span> public boolean isFailoverCleanupDone() {<a name="line.1352"></a>
-<span class="sourceLineNo">1353</span> return failoverCleanupDone.isReady();<a name="line.1353"></a>
-<span class="sourceLineNo">1354</span> }<a name="line.1354"></a>
-<span class="sourceLineNo">1355</span><a name="line.1355"></a>
-<span class="sourceLineNo">1356</span> /**<a name="line.1356"></a>
-<span class="sourceLineNo">1357</span> * Used by ServerCrashProcedure tests verify the ability to suspend the<a name="line.1357"></a>
-<span class="sourceLineNo">1358</span> * execution of the ServerCrashProcedure.<a name="line.1358"></a>
-<span class="sourceLineNo">1359</span> */<a name="line.1359"></a>
-<span class="sourceLineNo">1360</span> @VisibleForTesting<a name="line.1360"></a>
-<span class="sourceLineNo">1361</span> public void setFailoverCleanupDone(final boolean b) {<a name="line.1361"></a>
-<span class="sourceLineNo">1362</span> master.getMasterProcedureExecutor().getEnvironment()<a name="line.1362"></a>
-<span class="sourceLineNo">1363</span> .setEventReady(failoverCleanupDone, b);<a name="line.1363"></a>
-<span class="sourceLineNo">1364</span> }<a name="line.1364"></a>
-<span class="sourceLineNo">1365</span><a name="line.1365"></a>
-<span class="sourceLineNo">1366</span> public ProcedureEvent getFailoverCleanupEvent() {<a name="line.1366"></a>
-<span class="sourceLineNo">1367</span> return failoverCleanupDone;<a name="line.1367"></a>
-<span class="sourceLineNo">1368</span> }<a name="line.1368"></a>
-<span class="sourceLineNo">1369</span><a name="line.1369"></a>
-<span class="sourceLineNo">1370</span> /**<a name="line.1370"></a>
-<span class="sourceLineNo">1371</span> * Used to check if the failover cleanup is done.<a name="line.1371"></a>
-<span class="sourceLineNo">1372</span> * if not we throw PleaseHoldException since we are rebuilding the RegionStates<a name="line.1372"></a>
-<span class="sourceLineNo">1373</span> * @param hri region to check if it is already rebuild<a name="line.1373"></a>
-<span class="sourceLineNo">1374</span> * @throws PleaseHoldException if the failover cleanup is not completed<a name="line.1374"></a>
-<span class="sourceLineNo">1375</span> */<a name="line.1375"></a>
-<span class="sourceLineNo">1376</span> private void checkFailoverCleanupCompleted(final RegionInfo hri) throws PleaseHoldException {<a name="line.1376"></a>
-<span class="sourceLineNo">1377</span> if (!isRunning()) {<a name="line.1377"></a>
-<span class="sourceLineNo">1378</span> throw new PleaseHoldException("AssignmentManager not running");<a name="line.1378"></a>
-<span class="sourceLineNo">1379</span> }<a name="line.1379"></a>
-<span class="sourceLineNo">1380</span><a name="line.1380"></a>
-<span class="sourceLineNo">1381</span> // TODO: can we avoid throwing an exception if hri is already loaded?<a name="line.1381"></a>
-<span class="sourceLineNo">1382</span> // at the moment we bypass only meta<a name="line.1382"></a>
-<span class="sourceLineNo">1383</span> boolean meta = isMetaRegion(hri);<a name="line.1383"></a>
-<span class="sourceLineNo">1384</span> boolean cleanup = isFailoverCleanupDone();<a name="line.1384"></a>
-<span class="sourceLineNo">1385</span> if (!isMetaRegion(hri) && !isFailoverCleanupDone()) {<a name="line.1385"></a>
-<span class="sourceLineNo">1386</span> String msg = "Master not fully online; hbase:meta=" + meta + ", failoverCleanup=" + cleanup;<a name="line.1386"></a>
-<span class="sourceLineNo">1387</span> throw new PleaseHoldException(msg);<a name="line.1387"></a>
-<span class="sourceLineNo">1388</span> }<a name="line.1388"></a>
-<span class="sourceLineNo">1389</span> }<a name="line.1389"></a>
-<span class="sourceLineNo">1390</span><a name="line.1390"></a>
-<span class="sourceLineNo">1391</span> // ============================================================================================<a name="line.1391"></a>
-<span class="sourceLineNo">1392</span> // TODO: Metrics<a name="line.1392"></a>
-<span class="sourceLineNo">1393</span> // ============================================================================================<a name="line.1393"></a>
-<span class="sourceLineNo">1394</span> public int getNumRegionsOpened() {<a name="line.1394"></a>
-<span class="sourceLineNo">1395</span> // TODO: Used by TestRegionPlacement.java and assume monotonically increasing value<a name="line.1395"></a>
-<span class="sourceLineNo">1396</span> return 0;<a name="line.1396"></a>
-<span class="sourceLineNo">1397</span> }<a name="line.1397"></a>
-<span class="sourceLineNo">1398</span><a name="line.1398"></a>
-<span class="sourceLineNo">1399</span> public void submitServerCrash(final ServerName serverName, final boolean shouldSplitWal) {<a name="line.1399"></a>
-<span class="sourceLineNo">1400</span> boolean carryingMeta = isCarryingMeta(serverName);<a name="line.1400"></a>
-<span class="sourceLineNo">1401</span> ProcedureExecutor<MasterProcedureEnv> procExec = this.master.getMasterProcedureExecutor();<a name="line.1401"></a>
-<span class="sourceLineNo">1402</span> procExec.submitProcedure(new ServerCrashProcedure(procExec.getEnvironment(), serverName,<a name="line.1402"></a>
-<span class="sourceLineNo">1403</span> shouldSplitWal, carryingMeta));<a name="line.1403"></a>
-<span class="sourceLineNo">1404</span> LOG.debug("Added=" + serverName +<a name="line.1404"></a>
-<span class="sourceLineNo">1405</span> " to dead servers, submitted shutdown handler to be executed meta=" + carryingMeta);<a name="line.1405"></a>
-<span class="sourceLineNo">1406</span> }<a name="line.1406"></a>
-<span class="sourceLineNo">1407</span><a name="line.1407"></a>
-<span class="sourceLineNo">1408</span> public void offlineRegion(final RegionInfo regionInfo) {<a name="line.1408"></a>
-<span class="sourceLineNo">1409</span> // TODO used by MasterRpcServices ServerCrashProcedure<a name="line.1409"></a>
-<span class="sourceLineNo">1410</span> final RegionStateNode node = regionStates.getRegionStateNode(regionInfo);<a name="line.1410"></a>
-<span class="sourceLineNo">1411</span> if (node != null) node.offline();<a name="line.1411"></a>
-<span class="sourceLineNo">1412</span> }<a name="line.1412"></a>
-<span class="sourceLineNo">1413</span><a name="line.1413"></a>
-<span class="sourceLineNo">1414</span> public void onlineRegion(final RegionInfo regionInfo, final ServerName serverName) {<a name="line.1414"></a>
-<span class="sourceLineNo">1415</span> // TODO used by TestSplitTransactionOnCluster.java<a name="line.1415"></a>
-<span class="sourceLineNo">1416</span> }<a name="line.1416"></a>
-<span class="sourceLineNo">1417</span><a name="line.1417"></a>
-<span class="sourceLineNo">1418</span> public Map<ServerName, List<RegionInfo>> getSnapShotOfAssignment(<a name="line.1418"></a>
-<span class="sourceLineNo">1419</span> final Collection<RegionInfo> regions) {<a name="line.1419"></a>
-<span class="sourceLineNo">1420</span> return regionStates.getSnapShotOfAssignment(regions);<a name="line.1420"></a>
-<span class="sourceLineNo">1421</span> }<a name="line.1421"></a>
-<span class="sourceLineNo">1422</span><a name="line.1422"></a>
-<span class="sourceLineNo">1423</span> // ============================================================================================<a name="line.1423"></a>
-<span class="sourceLineNo">1424</span> // TODO: UTILS/HELPERS?<a name="line.1424"></a>
-<span class="sourceLineNo">1425</span> // ============================================================================================<a name="line.1425"></a>
-<span class="sourceLineNo">1426</span> /**<a name="line.1426"></a>
-<span class="sourceLineNo">1427</span> * Used by the client (via master) to identify if all regions have the schema updates<a name="line.1427"></a>
-<span class="sourceLineNo">1428</span> *<a name="line.1428"></a>
-<span class="sourceLineNo">1429</span> * @param tableName<a name="line.1429"></a>
-<span class="sourceLineNo">1430</span> * @return Pair indicating the status of the alter command (pending/total)<a name="line.1430"></a>
-<span class="sourceLineNo">1431</span> * @throws IOException<a name="line.1431"></a>
-<span class="sourceLineNo">1432</span> */<a name="line.1432"></a>
-<span class="sourceLineNo">1433</span> public Pair<Integer, Integer> getReopenStatus(TableName tableName) {<a name="line.1433"></a>
-<span class="sourceLineNo">1434</span> if (isTableDisabled(tableName)) return new Pair<Integer, Integer>(0, 0);<a name="line.1434"></a>
-<span class="sourceLineNo">1435</span><a name="line.1435"></a>
-<span class="sourceLineNo">1436</span> final List<RegionState> states = regionStates.getTableRegionStates(tableName);<a name="line.1436"></a>
-<span class="sourceLineNo">1437</span> int ritCount = 0;<a name="line.1437"></a>
-<span class="sourceLineNo">1438</span> for (RegionState regionState: states) {<a name="line.1438"></a>
-<span class="sourceLineNo">1439</span> if (!regionState.isOpened()) ritCount++;<a name="line.1439"></a>
-<span class="sourceLineNo">1440</span> }<a name="line.1440"></a>
-<span class="sourceLineNo">1441</span> return new Pair<Integer, Integer>(ritCount, states.size());<a name="line.1441"></a>
-<span class="sourceLineNo">1442</span> }<a name="line.1442"></a>
-<span class="sourceLineNo">1443</span><a name="line.1443"></a>
-<span class="sourceLineNo">1444</span> // ============================================================================================<a name="line.1444"></a>
-<span class="sourceLineNo">1445</span> // TODO: Region State In Transition<a name="line.1445"></a>
-<span class="sourceLineNo">1446</span> // ============================================================================================<a name="line.1446"></a>
-<span class="sourceLineNo">1447</span> protected boolean addRegionInTransition(final RegionStateNode regionNode,<a name="line.1447"></a>
-<span class="sourceLineNo">1448</span> final RegionTransitionProcedure procedure) {<a name="line.1448"></a>
-<span class="sourceLineNo">1449</span> return regionStates.addRegionInTransition(regionNode, procedure);<a name="line.1449"></a>
-<span class="sourceLineNo">1450</span> }<a name="line.1450"></a>
-<span class="sourceLineNo">1451</span><a name="line.1451"></a>
-<span class="sourceLineNo">1452</span> protected void removeRegionInTransition(final RegionStateNode regionNode,<a name="line.1452"></a>
-<span class="sourceLineNo">1453</span> final RegionTransitionProcedure procedure) {<a name="line.1453"></a>
-<span class="sourceLineNo">1454</span> regionStates.removeRegionInTransition(regionNode, procedure);<a name="line.1454"></a>
-<span class="sourceLineNo">1455</span> }<a name="line.1455"></a>
-<span class="sourceLineNo">1456</span><a name="line.1456"></a>
-<span class="sourceLineNo">1457</span> public boolean hasRegionsInTransition() {<a name="line.1457"></a>
-<span class="sourceLineNo">1458</span> return regionStates.hasRegionsInTransition();<a name="line.1458"></a>
-<span class="sourceLineNo">1459</span> }<a name="line.1459"></a>
-<span class="sourceLineNo">1460</span><a name="line.1460"></a>
-<span class="sourceLineNo">1461</span> public List<RegionStateNode> getRegionsInTransition() {<a name="line.1461"></a>
-<span class="sourceLineNo">1462</span> return regionStates.getRegionsInTransition();<a name="line.1462"></a>
-<span class="sourceLineNo">1463</span> }<a name="line.1463"></a>
-<span class="sourceLineNo">1464</span><a name="line.1464"></a>
-<span class="sourceLineNo">1465</span> public List<RegionInfo> getAssignedRegions() {<a name="line.1465"></a>
-<span class="sourceLineNo">1466</span> return regionStates.getAssignedRegions();<a name="line.1466"></a>
-<span class="sourceLineNo">1467</span> }<a name="line.1467"></a>
-<span class="sourceLineNo">1468</span><a name="line.1468"></a>
-<span class="sourceLineNo">1469</span> public RegionInfo getRegionInfo(final byte[] regionName) {<a name="line.1469"></a>
-<span class="sourceLineNo">1470</span> final RegionStateNode regionState = regionStates.getRegionStateNodeFromName(regionName);<a name="line.1470"></a>
-<span class="sourceLineNo">1471</span> return regionState != null ? regionState.getRegionInfo() : null;<a name="line.1471"></a>
-<span class="sourceLineNo">1472</span> }<a name="line.1472"></a>
-<span class="sourceLineNo">1473</span><a name="line.1473"></a>
-<span class="sourceLineNo">1474</span> // ============================================================================================<a name="line.1474"></a>
-<span class="sourceLineNo">1475</span> // TODO: Region Status update<a name="line.1475"></a>
-<span class="sourceLineNo">1476</span> // ============================================================================================<a name="line.1476"></a>
-<span class="sourceLineNo">1477</span> private void sendRegionOpenedNotification(final RegionInfo regionInfo,<a name="line.1477"></a>
-<span class="sourceLineNo">1478</span> final ServerName serverName) {<a name="line.1478"></a>
-<span class="sourceLineNo">1479</span> getBalancer().regionOnline(regionInfo, serverName);<a name="line.1479"></a>
-<span class="sourceLineNo">1480</span> if (!this.listeners.isEmpty()) {<a name="line.1480"></a>
-<span class="sourceLineNo">1481</span> for (AssignmentListener listener : this.listeners) {<a name="line.1481"></a>
-<span class="sourceLineNo">1482</span> listener.regionOpened(regionInfo, serverName);<a name="line.1482"></a>
-<span class="sourceLineNo">1483</span> }<a name="line.1483"></a>
-<span class="sourceLineNo">1484</span> }<a name="line.1484"></a>
-<span class="sourceLineNo">1485</span> }<a name="line.1485"></a>
-<span class="sourceLineNo">1486</span><a name="line.1486"></a>
-<span class="sourceLineNo">1487</span> private void sendRegionClosedNotification(final RegionInfo regionInfo) {<a name="line.1487"></a>
-<span class="sourceLineNo">1488</span> getBalancer().regionOffline(regionInfo);<a name="line.1488"></a>
-<span class="sourceLineNo">1489</span> if (!this.listeners.isEmpty()) {<a name="line.1489"></a>
-<span class="sourceLineNo">1490</span> for (AssignmentListener listener : this.listeners) {<a name="line.1490"></a>
-<span class="sourceLineNo">1491</span> listener.regionClosed(regionInfo);<a name="line.1491"></a>
-<span class="sourceLineNo">1492</span> }<a name="line.1492"></a>
-<span class="sourceLineNo">1493</span> }<a name="line.1493"></a>
-<span class="sourceLineNo">1494</span> }<a name="line.1494"></a>
-<span class="sourceLineNo">1495</span><a name="line.1495"></a>
-<span class="sourceLineNo">1496</span> public void markRegionAsOpening(final RegionStateNode regionNode) throws IOException {<a name="line.1496"></a>
-<span class="sourceLineNo">1497</span> synchronized (regionNode) {<a name="line.1497"></a>
-<span class="sourceLineNo">1498</span> regionNode.transitionState(State.OPENING, RegionStates.STATES_EXPECTED_ON_OPEN);<a name="line.1498"></a>
-<span class="sourceLineNo">1499</span> regionStates.addRegionToServer(regionNode);<a name="line.1499"></a>
-<span class="sourceLineNo">1500</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1500"></a>
-<span class="sourceLineNo">1501</span> }<a name="line.1501"></a>
-<span class="sourceLineNo">1502</span><a name="line.1502"></a>
-<span class="sourceLineNo">1503</span> // update the operation count metrics<a name="line.1503"></a>
-<span class="sourceLineNo">1504</span> metrics.incrementOperationCounter();<a name="line.1504"></a>
-<span class="sourceLineNo">1505</span> }<a name="line.1505"></a>
-<span class="sourceLineNo">1506</span><a name="line.1506"></a>
-<span class="sourceLineNo">1507</span> public void undoRegionAsOpening(final RegionStateNode regionNode) {<a name="line.1507"></a>
-<span class="sourceLineNo">1508</span> boolean opening = false;<a name="line.1508"></a>
-<span class="sourceLineNo">1509</span> synchronized (regionNode) {<a name="line.1509"></a>
-<span class="sourceLineNo">1510</span> if (regionNode.isInState(State.OPENING)) {<a name="line.1510"></a>
-<span class="sourceLineNo">1511</span> opening = true;<a name="line.1511"></a>
-<span class="sourceLineNo">1512</span> regionStates.removeRegionFromServer(regionNode.getRegionLocation(), regionNode);<a name="line.1512"></a>
-<span class="sourceLineNo">1513</span> }<a name="line.1513"></a>
-<span class="sourceLineNo">1514</span> // Should we update hbase:meta?<a name="line.1514"></a>
-<span class="sourceLineNo">1515</span> }<a name="line.1515"></a>
-<span class="sourceLineNo">1516</span> if (opening) {<a name="line.1516"></a>
-<span class="sourceLineNo">1517</span> // TODO: Metrics. Do opposite of metrics.incrementOperationCounter();<a name="line.1517"></a>
-<span class="sourceLineNo">1518</span> }<a name="line.1518"></a>
-<span class="sourceLineNo">1519</span> }<a name="line.1519"></a>
-<span class="sourceLineNo">1520</span><a name="line.1520"></a>
-<span class="sourceLineNo">1521</span> public void markRegionAsOpened(final RegionStateNode regionNode) throws IOException {<a name="line.1521"></a>
-<span class="sourceLineNo">1522</span> final RegionInfo hri = regionNode.getRegionInfo();<a name="line.1522"></a>
-<span class="sourceLineNo">1523</span> synchronized (regionNode) {<a name="line.1523"></a>
-<span class="sourceLineNo">1524</span> regionNode.transitionState(State.OPEN, RegionStates.STATES_EXPECTED_ON_OPEN);<a name="line.1524"></a>
-<span class="sourceLineNo">1525</span> if (isMetaRegion(hri)) {<a name="line.1525"></a>
-<span class="sourceLineNo">1526</span> // Usually we'd set a table ENABLED at this stage but hbase:meta is ALWAYs enabled, it<a name="line.1526"></a>
-<span class="sourceLineNo">1527</span> // can't be disabled -- so skip the RPC (besides... enabled is managed by TableStateManager<a name="line.1527"></a>
-<span class="sourceLineNo">1528</span> // which is backed by hbase:meta... Avoid setting ENABLED to avoid having to update state<a name="line.1528"></a>
-<span class="sourceLineNo">1529</span> // on table that contains state.<a name="line.1529"></a>
-<span class="sourceLineNo">1530</span> setMetaInitialized(hri, true);<a name="line.1530"></a>
-<span class="sourceLineNo">1531</span> }<a name="line.1531"></a>
-<span class="sourceLineNo">1532</span> regionStates.addRegionToServer(regionNode);<a name="line.1532"></a>
-<span class="sourceLineNo">1533</span> // TODO: OPENING Updates hbase:meta too... we need to do both here and there?<a name="line.1533"></a>
-<span class="sourceLineNo">1534</span> // That is a lot of hbase:meta writing.<a name="line.1534"></a>
-<span class="sourceLineNo">1535</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1535"></a>
-<span class="sourceLineNo">1536</span> sendRegionOpenedNotification(hri, regionNode.getRegionLocation());<a name="line.1536"></a>
-<span class="sourceLineNo">1537</span> }<a name="line.1537"></a>
-<span class="sourceLineNo">1538</span> }<a name="line.1538"></a>
-<span class="sourceLineNo">1539</span><a name="line.1539"></a>
-<span class="sourceLineNo">1540</span> public void markRegionAsClosing(final RegionStateNode regionNode) throws IOException {<a name="line.1540"></a>
-<span class="sourceLineNo">1541</span> final RegionInfo hri = regionNode.getRegionInfo();<a name="line.1541"></a>
-<span class="sourceLineNo">1542</span> synchronized (regionNode) {<a name="line.1542"></a>
-<span class="sourceLineNo">1543</span> regionNode.transitionState(State.CLOSING, RegionStates.STATES_EXPECTED_ON_CLOSE);<a name="line.1543"></a>
-<span class="sourceLineNo">1544</span> // Set meta has not initialized early. so people trying to create/edit tables will wait<a name="line.1544"></a>
-<span class="sourceLineNo">1545</span> if (isMetaRegion(hri)) {<a name="line.1545"></a>
-<span class="sourceLineNo">1546</span> setMetaInitialized(hri, false);<a name="line.1546"></a>
-<span class="sourceLineNo">1547</span> }<a name="line.1547"></a>
-<span class="sourceLineNo">1548</span> regionStates.addRegionToServer(regionNode);<a name="line.1548"></a>
-<span class="sourceLineNo">1549</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1549"></a>
-<span class="sourceLineNo">1550</span> }<a name="line.1550"></a>
-<span class="sourceLineNo">1551</span><a name="line.1551"></a>
-<span class="sourceLineNo">1552</span> // update the operation count metrics<a name="line.1552"></a>
-<span class="sourceLineNo">1553</span> metrics.incrementOperationCounter();<a name="line.1553"></a>
-<span class="sourceLineNo">1554</span> }<a name="line.1554"></a>
-<span class="sourceLineNo">1555</span><a name="line.1555"></a>
-<span class="sourceLineNo">1556</span> public void undoRegionAsClosing(final RegionStateNode regionNode) {<a name="line.1556"></a>
-<span class="sourceLineNo">1557</span> // TODO: Metrics. Do opposite of metrics.incrementOperationCounter();<a name="line.1557"></a>
-<span class="sourceLineNo">1558</span> // There is nothing to undo?<a name="line.1558"></a>
-<span class="sourceLineNo">1559</span> }<a name="line.1559"></a>
-<span class="sourceLineNo">1560</span><a name="line.1560"></a>
-<span class="sourceLineNo">1561</span> public void markRegionAsClosed(final RegionStateNode regionNode) throws IOException {<a name="line.1561"></a>
-<span class="sourceLineNo">1562</span> final RegionInfo hri = regionNode.getRegionInfo();<a name="line.1562"></a>
-<span class="sourceLineNo">1563</span> synchronized (regionNode) {<a name="line.1563"></a>
-<span class="sourceLineNo">1564</span> regionNode.transitionState(State.CLOSED, RegionStates.STATES_EXPECTED_ON_CLOSE);<a name="line.1564"></a>
-<span class="sourceLineNo">1565</span> regionStates.removeRegionFromServer(regionNode.getRegionLocation(), regionNode);<a name="line.1565"></a>
-<span class="sourceLineNo">1566</span> regionNode.setLastHost(regionNode.getRegionLocation());<a name="line.1566"></a>
-<span class="sourceLineNo">1567</span> regionNode.setRegionLocation(null);<a name="line.1567"></a>
-<span class="sourceLineNo">1568</span> regionStateStore.updateRegionLocation(regionNode);<a name="line.1568"></a>
-<span class="sourceLineNo">1569</span> sendRegionClosedNotification(hri);<a name="line.1569"></a>
-<span class="sourceLineNo">1570</span> }<a name="line.1570"></a>
-<span class="sourceLineNo">1571</span> }<a name="line.1571"></a>
-<span class="sourceLineNo">1572</span><a name="line.1572"></a>
-<span class="sourceLineNo">1573</span> public void markRegionAsSplit(final RegionInfo parent, final ServerName serverName,<a name="line.1573"></a>
-<span class="sourceLineNo">1574</span> final RegionInfo daughterA, final RegionInfo daughterB) throws IOException {<a name="line.1574"></a>
-<span class="sourceLineNo">1575</span> // Update hbase:meta. Parent will be marked offline and split up in hbase:meta.<a name="line.1575"></a>
-<span class="sourceLineNo">1576</span> // The parent stays in regionStates until cleared when removed by CatalogJanitor.<a name="line.1576"></a>
-<span class="sourceLineNo">1577</span> // Update its state in regionStates to it shows as offline and split when read<a name="line.1577"></a>
-<span class="sourceLineNo">1578</span> // later figuring what regions are in a table and what are not: see<a name="line.1578"></a>
-<span class="sourceLineNo">1579</span> // regionStates#getRegionsOfTable<a name="line.1579"></a>
-<span class="sourceLineNo">1580</span> final RegionStateNode node = regionStates.getOrCreateRegionStateNode(parent);<a name="line.1580"></a>
-<span class="sourceLineNo">1581</span> node.setState(State.SPLIT);<a name="line.1581"></a>
-<span class="sourceLineNo">1582</span> final RegionStateNode nodeA = regionStates.getOrCreateRegionStateNode(daughterA);<a name="line.1582"></a>
-<span class="sourceLineNo">1583</span> nodeA.setState(State.SPLITTING_NEW);<a name="line.1583"></a>
-<span class="sourceLineNo">1584</span> final RegionStateNode nodeB = regionStates.getOrCreateRegionStateNode(daughterB);<a name="line.1584"></a>
-<span class="sourceLineNo">1585</span> nodeB.setState(State.SPLITTING_NEW);<a name="line.1585"></a>
-<span class="sourceLineNo">1586</span><a name="line.1586"></a>
-<span class="sourceLineNo">1587</span> regionStateStore.splitRegion(parent, daughterA, daughterB, serverName);<a name="line.1587"></a>
-<span class="sourceLineNo">1588</span> if (shouldAssignFavoredNodes(parent)) {<a name="line.1588"></a>
-<span class="sourceLineNo">1589</span> List<ServerName> onlineServers = this.master.getServerManager().getOnlineServersList();<a name="line.1589"></a>
-<span class="sourceLineNo">1590</span> ((FavoredNodesPromoter)getBalancer()).<a name="line.1590"></a>
-<span class="sourceLineNo">1591</span> generateFavoredNodesForDaughter(onlineServers, parent, daughterA, daughterB);<a name="line.1591"></a>
-<span class="sourceLineNo">1592</span> }<a name="line.1592"></a>
-<span class="sourceLineNo">1593</span> }<a name="line.1593"></a>
-<span class="sourceLineNo">1594</span><a name="line.1594"></a>
-<span class="sourceLineNo">1595</span> /**<a name="line.1595"></a>
-<span class="sourceLineNo">1596</span> * When called here, the merge has happened. The two merged regions have been<a name="line.1596"></a>
-<span class="sourceLineNo">1597</span> * unassigned and the above markRegionClosed has been called on each so they have been<a name="line.1597"></a>
-<span class="sourceLineNo">1598</span> * disassociated from a hosting Server. The merged region will be open after this call. The<a name="line.1598"></a>
-<span class="sourceLineNo">1599</span> * merged regions are removed from hbase:meta below> Later they are deleted from the filesystem<a name="line.1599"></a>
-<span class="sourceLineNo">1600</span> * by the catalog janitor running against hbase:meta. It notices when the merged region no<a name="line.1600"></a>
-<span class="sourceLineNo">1601</span> * longer holds references to the old regions.<a name="line.1601"></a>
-<span class="sourceLineNo">1602</span> */<a name="line.1602"></a>
-<span class="sourceLineNo">1603</span> public void markRegionAsMerged(final RegionInfo child, final ServerName serverName,<a name="line.1603"></a>
-<span class="sourceLineNo">1604</span> final RegionInfo mother, final RegionInfo father) throws IOException {<a name="line.1604"></a>
-<span class="sourceLineNo">1605</span> final RegionStateNode node = regionStates.getOrCreateRegionStateNode(child);<a name="line.1605"></a>
-<span class="sourceLineNo">1606</span> node.setState(State.MERGED);<a name="line.1606"></a>
-<span class="sourceLineNo">1607</span> regionStates.deleteRegion(mother);<a name="line.1607"></a>
-<span class="sourceLineNo">1608</span> regionStates.deleteRegion(father);<a name="line.1608"></a>
-<span class="sourceLineNo">1609</span> regionStateStore.mergeRegions(child, mother, father, serverName);<a name="line.1609"></a>
-<span class="sourceLineNo">1610</span> if (shouldAssignFavoredNodes(child)) {<a name="line.1610"></a>
-<span class="sourceLineNo">1611</span> ((FavoredNodesPromoter)getBalancer()).<a name="line.1611"></a>
-<span class="sourceLineNo">1612</span> generateFavoredNodesForMergedRegion(child, mother, father);<a name="line.1612"></a>
-<span class="sourceLineNo">1613</span> }<a name="line.1613"></a>
-<span class="sourceLineNo">1614</span> }<a name="line.1614"></a>
-<span class="sourceLineNo">1615</span><a name="line.1615"></a>
-<span class="sourceLineNo">1616</span> /*<a name="line.1616"></a>
-<span class="sourceLineNo">1617</span> * Favored nodes should be applied only when FavoredNodes balancer is configured and the region<a name="line.1617"></a>
-<span class="sourceLineNo">1618</span> * belongs to a non-system table.<a name="line.1618"></a>
-<span class="sourceLineNo">1619</span> */<a name="line.1619"></a>
-<span class="sourceLineNo">1620</span> private boolean shouldAssignFavoredNodes(RegionInfo region) {<a name="line.1620"></a>
-<span class="sourceLineNo">1621</span> return this.shouldAssignRegionsWithFavoredNodes &&<a name="line.1621"></a>
-<span class="sourceLineNo">1622</span> FavoredNodesManager.isFavoredNodeApplicable(region);<a name="line.1622"></a>
-<span class="sourceLineNo">1623</span> }<a name="line.1623"></a>
-<span class="sourceLineNo">1624</span><a name="line.1624"></a>
-<span class="sourceLineNo">1625</span> // ============================================================================================<a name="line.1625"></a>
-<span class="sourceLineNo">1626</span> // Assign Queue (Assign/Balance)<a name="line.1626"></a>
-<span class="sourceLineNo">1627</span> // ============================================================================================<a name="line.1627"></a>
-<span class="sourceLineNo">1628</span> private final ArrayList<RegionStateNode> pendingAssignQueue = new ArrayList<RegionStateNode>();<a name="line.1628"></a>
-<span class="sourceLineNo">1629</span> private final ReentrantLock assignQueueLock = new ReentrantLock();<a name="line.1629"></a>
-<span class="sourceLineNo">1630</span> private final Condition assignQueueFullCond = assignQueueLock.newCondition();<a name="line.1630"></a>
-<span class="sourceLineNo">1631</span><a name="line.1631"></a>
-<span class="sourceLineNo">1632</span> /**<a name="line.1632"></a>
-<span class="sourceLineNo">1633</span> * Add the assign operation to the assignment queue.<a name="line.1633"></a>
-<span class="sourceLineNo">1634</span> * The pending assignment operation will be processed,<a name="line.1634"></a>
-<span class="sourceLineNo">1635</span> * and each region will be assigned by a server using the balancer.<a name="line.1635"></a>
-<span class="sourceLineNo">1636</span> */<a name="line.1636"></a>
-<span class="sourceLineNo">1637</span> protected void queueAssign(final RegionStateNode regionNode) {<a name="line.1637"></a>
-<span class="sourceLineNo">1638</span> regionNode.getProcedureEvent().suspend();<a name="line.1638"></a>
-<span class="sourceLineNo">1639</span><a name="line.1639"></a>
-<span class="sourceLineNo">1640</span> // TODO: quick-start for meta and the other sys-tables?<a name="line.1640"></a>
-<span class="sourceLineNo">1641</span> assignQueueLock.lock();<a name="line.1641"></a>
-<span class="sourceLineNo">1642</span> try {<a name="line.1642"></a>
-<span class="sourceLineNo">1643</span> pendingAssignQueue.add(regionNode);<a name="line.1643"></a>
-<span class="sourceLineNo">1644</span> if (regionNode.isSystemTable() ||<a name="line.1644"></a>
-<span class="sourceLineNo">1645</span> pendingAssignQueue.size() == 1 ||<a name="line.1645"></a>
-<span class="sourceLineNo">1646</span> pendingAssignQueue.size() >= assignDispatchWaitQueueMaxSize) {<a name="line.1646"></a>
-<span class="sourceLineNo">1647</span> assignQueueFullCond.signal();<a name="line.1647"></a>
-<span class="sourceLineNo">1648</span> }<a name="line.1648"></a>
-<span class="sourceLineNo">1649</span> } finally {<a name="line.1649"></a>
-<span class="sourceLineNo">1650</span> assignQueueLock.unlock();<a name="line.1650"></a>
-<span class="sourceLineNo">1651</span> }<a name="line.1651"></a>
-<span class="sourceLineNo">1652</span> }<a name="line.1652"></a>
-<span class="sourceLineNo">1653</span><a name="line.1653"></a>
-<span class="sourceLineNo">1654</span> private void startAssignmentThread() {<a name="line.1654"></a>
-<span class="sourceLineNo">1655</span> // Get Server Thread name. Sometimes the Server is mocked so may not implement HasThread.<a name="line.1655"></a>
-<span class="sourceLineNo">1656</span> // For example, in tests.<a name="line.1656"></a>
-<span class="sourceLineNo">1657</span> String name = master instanceof HasThread? ((HasThread)master).getName():<a name="line.1657"></a>
-<span class="sourceLineNo">1658</span> master.getServerName().toShortString();<a name="line.1658"></a>
-<span class="sourceLineNo">1659</span> assignThread = new Thread(name) {<a name="line.1659"></a>
-<span class="sourceLineNo">1660</span> @Override<a name="line.1660"></a>
-<span class="sourceLineNo">1661</span> public void run() {<a name="line.1661"></a>
-<span class="sourceLineNo">1662</span> while (isRunning()) {<a name="line.1662"></a>
-<span class="sourceLineNo">1663</span> processAssignQueue();<a name="line.1663"></a>
-<span class="sourceLineNo">1664</span> }<a name="line.1664"></a>
-<span class="sourceLineNo">1665</span> pendingAssignQueue.clear();<a name="line.1665"></a>
-<span class="sourceLineNo">1666</span> }<a name="line.1666"></a>
-<span class="sourceLineNo">1667</span> };<a name="line.1667"></a>
-<span class="sourceLineNo">1668</span> assignThread.setDaemon(true);<a name="line.1668"></a>
-<span class="sourceLineNo">1669</span> assignThread.start();<a name="line.1669"></a>
-<span class="sourceLineNo">1670</span> }<a name="line.1670"></a>
-<span class="sourceLineNo">1671</span><a name="line.1671"></a>
-<span class="sourceLineNo">1672</span> private void stopAssignmentThread() {<a name="line.1672"></a>
-<span class="sourceLineNo">1673</span> assignQueueSignal();<a name="line.1673"></a>
-<span class="sourceLineNo">1674</span> try {<a name="line.1674"></a>
-<span class="sourceLineNo">1675</span> while (assignThread.isAlive()) {<a name="line.1675"></a>
-<span class="sourceLineNo">1676</span> assignQueueSignal();<a name="line.1676"></a>
-<span class="sourceLineNo">1677</span> assignThread.join(250);<a name="line.1677"></a>
-<span class="sourceLineNo">1678</span> }<a name="line.1678"></a>
-<span class="sourceLineNo">1679</span> } catch (InterruptedException e) {<a name="line.1679"></a>
-<span class="sourceLineNo">1680</span> LOG.warn("join interrupted", e);<a name="line.1680"></a>
-<span class="sourceLineNo">1681</span> Thread.currentThread().interrupt();<a name="line.1681"></a>
-<span class="sourceLineNo">1682</span> }<a name="line.1682"></a>
-<span class="sourceLineNo">1683</span> }<a name="line.1683"></a>
-<span class="sourceLineNo">1684</span><a name="line.1684"></a>
-<span class="sourceLineNo">1685</span> private void assignQueueSignal() {<a name="line.1685"></a>
-<span class="sourceLineNo">1686</span> assignQueueLock.lock();<a name="line.1686"></a>
-<span class="sourceLineNo">1687</span> try {<a name="line.1687"></a>
-<span class="sourceLineNo">1688</span> assignQueueFullCond.signal();<a name="line.1688"></a>
-<span class="sourceLineNo">1689</span> } finally {<a name="line.1689"></a>
-<span class="sourceLineNo">1690</span> assignQueueLock.unlock();<a name="line.1690"></a>
-<span class="sourceLineNo">1691</span> }<a name="line.1691"></a>
-<span class="sourceLineNo">1692</span> }<a name="line.1692"></a>
-<span class="sourceLineNo">1693</span><a name="line.1693"></a>
-<span class="sourceLineNo">1694</span> @edu.umd.cs.findbugs.annotations.SuppressWarnings("WA_AWAIT_NOT_IN_LOOP")<a name="line.1694"></a>
-<span class="sourceLineNo">1695</span> private HashMap<RegionInfo, RegionStateNode> waitOnAssignQueue() {<a name="line.1695"></a>
-<span class="sourceLineNo">1696</span> HashMap<RegionInfo, RegionStateNode> regions = null;<a name="line.1696"></a>
-<span class="sourceLineNo">1697</span><a name="line.1697"></a>
-<span class="sourceLineNo">1698</span> assignQueueLock.lock();<a name="line.1698"></a>
-<span class="sourceLineNo">1699</span> try {<a name="line.1699"></a>
-<span class="sourceLineNo">1700</span> if (pendingAssignQueue.isEmpty() && isRunning()) {<a name="line.1700"></a>
-<span class="sourceLineNo">1701</span> assignQueueFullCond.await();<a name="line.1701"></a>
-<span class="sourceLineNo">1702</span> }<a name="line.1702"></a>
-<span class="sourceLineNo">1703</span><a name="line.1703"></a>
-<span class="sourceLineNo">1704</span> if (!isRunning()) return null;<a name="line.1704"></a>
-<span class="sourceLineNo">1705</span> assignQueueFullCond.await(assignDispatchWaitMillis, TimeUnit.MILLISECONDS);<a name="line.1705"></a>
-<span class="sourceLineNo">1706</span> regions = new HashMap<RegionInfo, RegionStateNode>(pendingAssignQueue.size());<a name="line.1706"></a>
-<span class="sourceLineNo">1707</span> for (RegionStateNode regionNode: pendingAssignQueue) {<a name="line.1707"></a>
-<span class="sourceLineNo">1708</span> regions.put(regionNode.getRegionInfo(), regionNode);<a name="line.1708"></a>
-<span class="sourceLineNo">1709</span> }<a name="line.1709"></a>
-<span class="sourceLineNo">1710</span> pendingAssignQueue.clear();<a name="line.1710"></a>
-<span class="sourceLineNo">1711</span> } catch (InterruptedException e) {<a name="line.1711"></a>
-<span class="sourceLineNo">1712</span> LOG.warn("got interrupted ", e);<a name="line.1712"></a>
-<span class="sourceLineNo">1713</span> Thread.currentThread().interrupt();<a name="line.1713"></a>
-<span class="sourceLineNo">1714</span> } finally {<a name="line.1714"></a>
-<span class="sourceLineNo">1715</span> assignQueueLock.unlock();<a name="line.1715"></a>
-<span class="sourceLineNo">1716</span> }<a name="line.1716"></a>
-<span class="sourceLineNo">1717</span> return regions;<a name="line.1717"></a>
-<span class="sourceLineNo">1718</span> }<a name="line.1718"></a>
-<span class="sourceLineNo">1719</span><a name="line.1719"></a>
-<span class="sourceLineNo">1720</span> private void processAssignQueue() {<a name="line.1720"></a>
-<span class="sourceLineNo">1721</span> final HashMap<RegionInfo, RegionStateNode> regions = waitOnAssignQueue();<a name="line.1721"></a>
-<span class="sourceLineNo">1722</span> if (regions == null || regions.size() == 0 || !isRunning()) {<a name="line.1722"></a>
-<span class="sourceLineNo">1723</span> return;<a name="line.1723"></a>
-<span class="sourceLineNo">1724</span> }<a name="line.1724"></a>
-<span class="sourceLineNo">1725</span><a name="line.1725"></a>
-<span class="sourceLineNo">1726</span> if (LOG.isTraceEnabled()) {<a name="line.1726"></a>
-<span class="sourceLineNo">1727</span> LOG.trace("PROCESS ASSIGN QUEUE regionCount=" + regions.size());<a name="line.1727"></a>
-<span class="sourceLineNo">1728</span> }<a name="line.1728"></a>
-<span class="sourceLineNo">1729</span><a name="line.1729"></a>
-<span class="sourceLineNo">1730</span> // TODO: Optimize balancer. pass a RegionPlan?<a name="line.1730"></a>
-<span class="sourceLineNo">1731</span> final HashMap<RegionInfo, ServerName> retainMap = new HashMap<>();<a name="line.1731"></a>
-<span class="sourceLineNo">1732</span> final List<RegionInfo> userHRIs = new ArrayList<>(regions.size());<a name="line.1732"></a>
-<span class="sourceLineNo">1733</span> // Regions for system tables requiring reassignment<a name="line.1733"></a>
-<span class="sourceLineNo">1734</span> final List<RegionInfo> systemHRIs = new ArrayList<>();<a name="line.1734"></a>
-<span class="sourceLineNo">1735</span> for (RegionStateNode regionStateNode: regions.values()) {<a name="line.1735"></a>
-<span class="sourceLineNo">1736</span> boolean sysTable = regionStateNode.isSystemTable();<a name="line.1736"></a>
-<span class="sourceLineNo">1737</span> final List<RegionInfo> hris = sysTable? systemHRIs: userHRIs;<a name="line.1737"></a>
-<span class="sourceLineNo">1738</span> if (regionStateNode.getRegionLocation() != null) {<a name="line.1738"></a>
-<span class="sourceLineNo">1739</span> retainMap.put(regionStateNode.getRegionInfo(), regionStateNode.getRegionLocation());<a name="line.1739"></a>
-<span class="sourceLineNo">1740</span> } else {<a name="line.1740"></a>
-<span class="sourceLineNo">1741</span> hris.add(regionStateNode.getRegionInfo());<a name="line.1741"></a>
-<span class="sourceLineNo">1742</span> }<a name="line.1742"></a>
-<span class="sourceLineNo">1743</span> }<a name="line.1743"></a>
-<span class="sourceLineNo">1744</span><a name="line.1744"></a>
-<span class="sourceLineNo">1745</span> // TODO: connect with the listener to invalidate the cache<a name="line.1745"></a>
-<span class="sourceLineNo">1746</span><a name="line.1746"></a>
-<span class="sourceLineNo">1747</span> // TODO use events<a name="line.1747"></a>
-<span class="sourceLineNo">1748</span> List<ServerName> servers = master.getServerManager().createDestinationServersList();<a name="line.1748"></a>
-<span class="sourceLineNo">1749</span> for (int i = 0; servers.size() < 1; ++i) {<a name="line.1749"></a>
-<span class="sourceLineNo">1750</span> // Report every fourth time around this loop; try not to flood log.<a name="line.1750"></a>
-<span class="sourceLineNo">1751</span> if (i % 4 == 0) {<a name="line.1751"></a>
-<span class="sourceLineNo">1752</span> LOG.warn("No servers available; cannot place " + regions.size() + " unassigned regions.");<a name="line.1752"></a>
-<span class="sourceLineNo">1753</span> }<a name="line.1753"></a>
-<span class="sourceLineNo">1754</span><a name="line.1754"></a>
-<span class="sourceLineNo">1755</span> if (!isRunning()) {<a name="line.1755"></a>
-<span class="sourceLineNo">1756</span> LOG.debug("Stopped! Dropping assign of " + regions.size() + " queued regions.");<a name="line.1756"></a>
-<span class="sourceLineNo">1757</span> return;<a name="line.1757"></a>
-<span class="sourceLineNo">1758</span> }<a name="line.1758"></a>
-<span class="sourceLineNo">1759</span> Threads.sleep(250);<a name="line.1759"></a>
-<span class="sourceLineNo">1760</span> servers = master.getServerManager().createDestinationServersList();<a name="line.1760"></a>
-<span class="sourceLineNo">1761</span> }<a name="line.1761"></a>
-<span class="sourceLineNo">1762</span><a name="line.1762"></a>
-<span class="sourceLineNo">1763</span> if (!systemHRIs.isEmpty()) {<a name="line.1763"></a>
-<span class="sourceLineNo">1764</span> // System table regions requiring reassignment are present, get region servers<a name="line.1764"></a>
-<span class="sourceLineNo">1765</span> // not available for system table regions<a name="line.1765"></a>
-<span class="sourceLineNo">1766</span> final List<ServerName> excludeServers = getExcludedServersForSystemTable();<a name="line.1766"></a>
-<span class="sourceLineNo">1767</span> List<ServerName> serversForSysTables = servers.stream()<a name="line.1767"></a>
-<span class="sourceLineNo">1768</span> .filter(s -> !excludeServers.contains(s)).collect(Collectors.toList());<a name="line.1768"></a>
-<span class="sourceLineNo">1769</span> if (serversForSysTables.isEmpty()) {<a name="line.1769"></a>
-<span class="sourceLineNo">1770</span> LOG.warn("Filtering old server versions and the excluded produced an empty set; " +<a name="line.1770"></a>
-<span class="sourceLineNo">1771</span> "instead considering all candidate servers!");<a name="line.1771"></a>
-<span class="sourceLineNo">1772</span> }<a name="line.1772"></a>
-<span class="sourceLineNo">1773</span> LOG.debug("Processing assignQueue; systemServersCount=" + serversForSysTables.size() +<a name="line.1773"></a>
-<span class="sourceLineNo">1774</span> ", allServersCount=" + servers.size());<a name="line.1774"></a>
-<span class="sourceLineNo">1775</span> processAssignmentPlans(regions, null, systemHRIs,<a name="line.1775"></a>
-<span class="sourceLineNo">1776</span> serversForSysTables.isEmpty()? servers: serversForSysTables);<a name="line.1776"></a>
-<span class="sourceLineNo">1777</span> }<a name="line.1777"></a>
-<span class="sourceLineNo">1778</span><a name="line.1778"></a>
-<span class="sourceLineNo">1779</span> processAssignmentPlans(regions, retainMap, userHRIs, servers);<a name="line.1779"></a>
-<span class="sourceLineNo">1780</span> }<a name="line.1780"></a>
-<span class="sourceLineNo">1781</span><a name="line.1781"></a>
-<span class="sourceLineNo">1782</span> private void processAssignmentPlans(final HashMap<RegionInfo, RegionStateNode> regions,<a name="line.1782"></a>
-<span class="sourceLineNo">1783</span> final HashMap<RegionInfo, ServerName> retainMap, final List<RegionInfo> hris,<a name="line.1783"></a>
-<span class="sourceLineNo">1784</span> final List<ServerName> servers) {<a name="line.1784"></a>
-<span class="sourceLineNo">1785</span> boolean isTraceEnabled = LOG.isTraceEnabled();<a name="line.1785"></a>
-<span class="sourceLineNo">1786</span> if (isTraceEnabled) {<a name="line.1786"></a>
-<span class="sourceLineNo">1787</span> LOG.trace("Available servers count=" + servers.size() + ": " + servers);<a name="line.1787"></a>
-<span class="sourceLineNo">1788</span> }<a name="line.1788"></a>
-<span class="sourceLineNo">1789</span><a name="line.1789"></a>
-<span class="sourceLineNo">1790</span> final LoadBalancer balancer = getBalancer();<a name="line.1790"></a>
-<span class="sourceLineNo">1791</span> // ask the balancer where to place regions<a name="line.1791"></a>
-<span class="sourceLineNo">1792</span> if (retainMap != null && !retainMap.isEmpty()) {<a name="line.1792"></a>
-<span class="sourceLineNo">1793</span> if (isTraceEnabled) {<a name="line.1793"></a>
-<span class="sourceLineNo">1794</span> LOG.trace("retain assign regions=" + retainMap);<a name="line.1794"></a>
-<span class="sourceLineNo">1795</span> }<a name="line.1795"></a>
-<span class="sourceLineNo">1796</span> try {<a name="line.1796"></a>
-<span class="sourceLineNo">1797</span> acceptPlan(regions, balancer.retainAssignment(retainMap, servers));<a name="line.1797"></a>
-<span class="sourceLineNo">1798</span> } catch (HBaseIOException e) {<a name="line.1798"></a>
-<span class="sourceLineNo">1799</span> LOG.warn("unable to retain assignment", e);<a name="line.1799"></a>
-<span class="sourceLineNo">1800</span> addToPendingAssignment(regions, retainMap.keySet());<a name="line.1800"></a>
-<span class="sourceLineNo">1801</span> }<a name="line.1801"></a>
-<span class="sourceLineNo">1802</span> }<a name="line.1802"></a>
-<span class="sourceLineNo">1803</span><a name="line.1803"></a>
-<span class="sourceLineNo">1804</span> // TODO: Do we need to split retain and round-robin?<a name="line.1804"></a>
-<span class="sourceLineNo">1805</span> // the retain seems to fallback to round-robin/random if the region is not in the map.<a name="line.1805"></a>
-<span class="sourceLineNo">1806</span> if (!hris.isEmpty()) {<a name="line.1806"></a>
-<span class="sourceLineNo">1807</span> Collections.sort(hris, RegionInfo.COMPARATOR);<a name="line.1807"></a>
-<span class="sourceLineNo">1808</span> if (isTraceEnabled) {<a name="line.1808"></a>
-<span class="sourceLineNo">1809</span> LOG.trace("round robin regions=" + hris);<a name="line.1809"></a>
-<span class="sourceLineNo">1810</span> }<a name="line.1810"></a>
-<span class="sourceLineNo">1811</span> try {<a name="line.1811"></a>
-<span class="sourceLineNo">1812</span> acceptPlan(regions, balancer.roundRobinAssignment(hris, servers));<a name="line.1812"></a>
-<span class="sourceLineNo">1813</span> } catch (HBaseIOException e) {<a name="line.1813"></a>
-<span class="sourceLineNo">1814</span> LOG.warn("unable to round-robin assignment", e);<a name="line.1814"></a>
-<span class="sourceLineNo">1815</span> addToPendingAssignment(regions, hris);<a name="line.1815"></a>
-<span class="sourceLineNo">1816</span> }<a name="line.1816"></a>
-<span class="sourceLineNo">1817</span> }<a name="line.1817"></a>
-<span class="sourceLineNo">1818</span> }<a name="line.1818"></a>
-<span class="sourceLineNo">1819</span><a name="line.1819"></a>
-<span class="sourceLineNo">1820</span> private void acceptPlan(final HashMap<RegionInfo, RegionStateNode> regions,<a name="line.1820"></a>
-<span class="sourceLineNo">1821</span> final Map<ServerName, List<RegionInfo>> plan) throws HBaseIOException {<a name="line.1821"></a>
-<span class="sourceLineNo">1822</span> final ProcedureEvent[] events = new ProcedureEvent[regions.size()];<a name="line.1822"></a>
-<span class="sourceLineNo">1823</span> final long st = System.currentTimeMillis();<a name="line.1823"></a>
-<span class="sourceLineNo">1824</span><a name="line.1824"></a>
-<span class="sourceLineNo">1825</span> if (plan == null) {<a name="line.1825"></a>
-<span class="sourceLineNo">1826</span> throw new HBaseIOException("unable to compute plans for regions=" + regions.size());<a name="line.1826"></a>
-<span class="sourceLineNo">1827</span> }<a name="line.1827"></a>
-<span class="sourceLineNo">1828</span><a name="line.1828"></a>
-<span class="sourceLineNo">1829</span> if (plan.isEmpty()) return;<a name="line.1829"></a>
-<span class="sourceLineNo">1830</span><a name="line.1830"></a>
-<span class="sourceLineNo">1831</span> int evcount = 0;<a name="line.1831"></a>
-<span class="sourceLineNo">1832</span> for (Map.Entry<ServerName, List<RegionInfo>> entry: plan.entrySet()) {<a name="line.1832"></a>
-<span class="sourceLineNo">1833</span> final ServerName server = entry.getKey();<a name="line.1833"></a>
-<span class="sourceLineNo">1834</span> for (RegionInfo hri: entry.getValue()) {<a name="line.1834"></a>
-<span class="sourceLineNo">1835</span> final RegionStateNode regionNode = regions.get(hri);<a name="line.1835"></a>
-<span class="sourceLineNo">1836</span> regionNode.setRegionLocation(server);<a name="line.1836"></a>
-<span class="sourceLineNo">1837</span> events[evcount++] = regionNode.getProcedureEvent();<a name="line.1837"></a>
-<span class="sourceLineNo">1838</span> }<a name="line.1838"></a>
-<span class="sourceLineNo">1839</span> }<a name="line.1839"></a>
-<span class="sourceLineNo">1840</span> ProcedureEvent.wakeEvents(getProcedureScheduler(), events);<a name="line.1840"></a>
-<span class="sourceLineNo">1841</span><a name="line.1841"></a>
-<span class="sourceLineNo">1842</span> final long et = System.currentTimeMillis();<a name="line.1842"></a>
-<span class="sourceLineNo">1843</span> if (LOG.isTraceEnabled()) {<a name="line.1843"></a>
-<span class="sourceLineNo">1844</span> LOG.trace("ASSIGN ACCEPT " + events.length + " -> " +<a name="line.1844"></a>
-<span class="sourceLineNo">1845</span> StringUtils.humanTimeDiff(et - st));<a name="line.1845"></a>
-<span class="sourceLineNo">1846</span> }<a name="line.1846"></a>
-<span class="sourceLineNo">1847</span> }<a name="line.1847"></a>
-<span class="sourceLineNo">1848</span><a name="line.1848"></a>
-<span class="sourceLineNo">1849</span> private void addToPendingAssignment(final HashMap<RegionInfo, RegionStateNode> regions,<a name="line.1849"></a>
-<span class="sourceLineNo">1850</span> final Collection<RegionInfo> pendingRegions) {<a name="line.1850"></a>
-<span class="sourceLineNo">1851</span> assignQueueLock.lock();<a name="line.1851"></a>
-<span class="sourceLineNo">1852</span> try {<a name="line.1852"></a>
-<span class="sourceLineNo">1853</span> for (RegionInfo hri: pendingRegions) {<a name="line.1853"></a>
-<span class="sourceLineNo">1854</span> pendingAssignQueue.add(regions.get(hri));<a name="line.1854"></a>
-<span class="sourceLineNo">1855</span> }<a name="line.1855"></a>
-<span class="sourceLineNo">1856</span> } finally {<a name="line.1856"></a>
-<span class="sourceLineNo">1857</span> assignQueueLock.unlock();<a name="line.1857"></a>
-<span class="sourceLineNo">1858</span> }<a name="line.1858"></a>
-<span class="sourceLineNo">1859</span> }<a name="line.1859"></a>
-<span class="sourceLineNo">1860</span><a name="line.1860"></a>
-<span class="sourceLineNo">1861</span> /**<a name="line.1861"></a>
-<span class="sourceLineNo">1862</span> * Get a list of servers that this region cannot be assigned to.<a name="line.1862"></a>
-<span class="sourceLineNo">1863</span> * For system tables, we must assign them to a server with highest version.<a name="line.1863"></a>
-<span class="sourceLineNo">1864</span> */<a name="line.1864"></a>
-<span class="sourceLineNo">1865</span> public List<ServerName> getExcludedServersForSystemTable() {<a name="line.1865"></a>
-<span class="sourceLineNo">1866</span> // TODO: This should be a cached list kept by the ServerManager rather than calculated on each<a name="line.1866"></a>
-<span class="sourceLineNo">1867</span> // move or system region assign. The RegionServerTracker keeps list of online Servers with<a name="line.1867"></a>
-<span class="sourceLineNo">1868</span> // RegionServerInfo that includes Version.<a name="line.1868"></a>
-<span class="sourceLineNo">1869</span> List<Pair<ServerName, String>> serverList = master.getServerManager().getOnlineServersList()<a name="line.1869"></a>
-<span class="sourceLineNo">1870</span> .stream()<a name="line.1870"></a>
-<span class="sourceLineNo">1871</span> .map((s)->new Pair<>(s, master.getRegionServerVersion(s)))<a name="line.1871"></a>
-<span class="sourceLineNo">1872</span> .collect(Collectors.toList());<a name="line.1872"></a>
-<span class="sourceLineNo">1873</span> if (serverList.isEmpty()) {<a name="line.1873"></a>
-<span class="sourceLineNo">1874</span> return Collections.EMPTY_LIST;<a name="line.1874"></a>
-<span class="sourceLineNo">1875</span> }<a name="line.1875"></a>
-<span class="sourceLineNo">1876</span> String highestVersion = Collections.max(serverList,<a name="line.1876"></a>
-<span class="sourceLineNo">1877</span> (o1, o2) -> VersionInfo.compareVersion(o1.getSecond(), o2.getSecond())).getSecond();<a name="line.1877"></a>
-<span class="sourceLineNo">1878</span> return serverList.stream()<a name="line.1878"></a>
-<span class="sourceLineNo">1879</span> .filter((p)->!p.getSecond().equals(highestVersion))<a name="line.1879"></a>
-<span class="sourceLineNo">1880</span> .map(Pair::getFirst)<a name="line.1880"></a>
-<span class="sourceLineNo">1881</span> .collect(Collectors.toList());<a name="line.1881"></a>
-<span class="sourceLineNo">1882</span> }<a name="line.1882"></a>
-<span class="sourceLineNo">1883</span><a name="line.1883"></a>
-<span class="sourceLineNo">1884</span> // ============================================================================================<a name="line.1884"></a>
-<span class="sourceLineNo">1885</span> // Server Helpers<a name="line.1885"></a>
-<span class="sourceLineNo">1886</span> // ============================================================================================<a name="line.1886"></a>
-<span class="sourceLineNo">1887</span> @Override<a name="line.1887"></a>
-<span class="sourceLineNo">1888</span> public void serverAdded(final ServerName serverName) {<a name="line.1888"></a>
-<span class="sourceLineNo">1889</span> }<a name="line.1889"></a>
-<span class="sourceLineNo">1890</span><a name="line.1890"></a>
-<span class="sourceLineNo">1891</span> @Override<a name="line.1891"></a>
-<span class="sourceLineNo">1892</span> public void serverRemoved(final ServerName serverName) {<a name="line.1892"></a>
-<span class="sourceLineNo">1893</span> final ServerStateNode serverNode = regionStates.getServerNode(serverName);<a name="line.1893"></a>
-<span class="sourceLineNo">1894</span> if (serverNode == null) return;<a name="line.1894"></a>
-<span class="sourceLineNo">1895</span><a name="line.1895"></a>
-<span class="sourceLineNo">1896</span> // just in case, wake procedures waiting for this server report<a name="line.1896"></a>
-<span class="sourceLineNo">1897</span> wakeServerReportEvent(serverNode);<a name="line.1897"></a>
-<span class="sourceLineNo">1898</span> }<a name="line.1898"></a>
-<span class="sourceLineNo">1899</span><a name="line.1899"></a>
-<span class="sourceLineNo">1900</span> public int getServerVersion(final ServerName serverName) {<a name="line.1900"></a>
-<span class="sourceLineNo">1901</span> final ServerStateNode node = regionStates.getServerNode(serverName);<a name="line.1901"></a>
-<span class="sourceLineNo">1902</span> return node != null ? node.getVersionNumber() : 0;<a name="line.1902"></a>
-<span class="sourceLineNo">1903</span> }<a name="line.1903"></a>
-<span class="sourceLineNo">1904</span><a name="line.1904"></a>
-<span class="sourceLineNo">1905</span> public void killRegionServer(final ServerName serverName) {<a name="line.1905"></a>
-<span class="sourceLineNo">1906</span> final ServerStateNode serverNode = regionStates.getServerNode(serverName);<a name="line.1906"></a>
-<span class="sourceLineNo">1907</span> killRegionServer(serverNode);<a name="line.1907"></a>
-<span class="sourceLineNo">1908</span> }<a name="line.1908"></a>
-<span class="sourceLineNo">1909</span><a name="line.1909"></a>
-<span class="sourceLineNo">1910</span> public void killRegionServer(final ServerStateNode serverNode) {<a name="line.1910"></a>
-<span class="sourceLineNo">1911</span> /** Don't do this. Messes up accounting. Let ServerCrashProcedure do this.<a name="line.1911"></a>
-<span class="sourceLineNo">1912</span> for (RegionStateNode regionNode: serverNode.getRegions()) {<a name="line.1912"></a>
-<span class="sourceLineNo">1913</span> regionNode.offline();<a name="line.1913"></a>
-<span class="sourceLineNo">1914</span> }*/<a name="line.1914"></a>
-<span class="sourceLineNo">1915</span> master.getServerManager().expireServer(serverNode.getServerName());<a name="line.1915"></a>
-<span class="sourceLineNo">1916</span> }<a name="line.1916"></a>
-<span class="sourceLineNo">1917</span><a name="line.1917"></a>
-<span class="sourceLineNo">1918</span> /**<a name="line.1918"></a>
-<span class="sourceLineNo">1919</span> * Handle RIT of meta region against crashed server.<a name="line.1919"></a>
-<span class="sourceLineNo">1920</span> * Only used when ServerCrashProcedure is not enabled.<a name="line.1920"></a>
-<span class="sourceLineNo">1921</span> * See handleRIT in ServerCrashProcedure for similar function.<a name="line.1921"></a>
-<span class="sourceLineNo">1922</span> *<a name="line.1922"></a>
-<span class="sourceLineNo">1923</span> * @param serverName Server that has already crashed<a name="line.1923"></a>
-<span class="sourceLineNo">1924</span> */<a name="line.1924"></a>
-<span class="sourceLineNo">1925</span> public void handleMetaRITOnCrashedServer(ServerName serverName) {<a name="line.1925"></a>
-<span class="sourceLineNo">1926</span> RegionInfo hri = RegionReplicaUtil<a name="line.1926"></a>
-<span class="sourceLineNo">1927</span> .getRegionInfoForReplica(RegionInfoBuilder.FIRST_META_REGIONINFO,<a name="line.1927"></a>
-<span class="sourceLineNo">1928</span> RegionInfo.DEFAULT_REPLICA_ID);<a name="line.1928"></a>
-<span class="sourceLineNo">1929</span> RegionState regionStateNode = getRegionStates().getRegionState(hri);<a name="line.1929"></a>
-<span class="sourceLineNo">1930</span> if (regionStateNode == null) {<a name="line.1930"></a>
-<span class="sourceLineNo">1931</span> LOG.warn("RegionStateNode is null for " + hri);<a name="line.1931"></a>
-<span class="sourceLineNo">1932</span> return;<a name="line.1932"></a>
-<span class="sourceLineNo">1933</span> }<a name="line.1933"></a>
-<span class="sourceLineNo">1934</span> ServerName rsnServerName = regionStateNode.getServerName();<a name="line.1934"></a>
-<span class="sourceLineNo">1935</span> if (rsnServerName != null && !rsnServerName.equals(serverName)) {<a name="line.1935"></a>
-<span class="sourceLineNo">1936</span> return;<a name="line.1936"></a>
-<span class="sourceLineNo">1937</span> } else if (rsnServerName == null) {<a name="line.1937"></a>
-<span class="sourceLineNo">1938</span> LOG.warn("Empty ServerName in RegionStateNode; proceeding anyways in case latched " +<a name="line.1938"></a>
-<span class="sourceLineNo">1939</span> "RecoverMetaProcedure so meta latch gets cleaned up.");<a name="line.1939"></a>
-<span class="sourceLineNo">1940</span> }<a name="line.1940"></a>
-<span class="sourceLineNo">1941</span> // meta has been assigned to crashed server.<a name="line.1941"></a>
-<span class="sourceLineNo">1942</span> LOG.info("Meta assigned to crashed " + serverName + "; reassigning...");<a name="line.1942"></a>
-<span class="sourceLineNo">1943</span> // Handle failure and wake event<a name="line.1943"></a>
-<span class="sourceLineNo">1944</span> RegionTransitionProcedure rtp = getRegionStates().getRegionTransitionProcedure(hri);<a name="line.1944"></a>
-<span class="sourceLineNo">1945</span> // Do not need to consider for REGION_TRANSITION_QUEUE step<a name="line.1945"></a>
-<span class="sourceLineNo">1946</span> if (rtp != null && rtp.isMeta() &&<a name="line.1946"></a>
-<span class="sourceLineNo">1947</span> rtp.getTransitionState() == RegionTransitionState.REGION_TRANSITION_DISPATCH) {<a name="line.1947"></a>
-<span class="sourceLineNo">1948</span> LOG.debug("Failing " + rtp.toString());<a name="line.1948"></a>
-<span class="sourceLineNo">1949</span> rtp.remoteCallFailed(master.getMasterProcedureExecutor().getEnvironment(), serverName,<a name="line.1949"></a>
-<span class="sourceLineNo">1950</span> new ServerCrashException(rtp.getProcId(), serverName));<a name="line.1950"></a>
-<span class="sourceLineNo">1951</span> }<a name="line.1951"></a>
-<span class="sourceLineNo">1952</span> }<a name="line.1952"></a>
-<span class="sourceLineNo">1953</span>}<a name="line.1953"></a>
+<span class="sourceLineNo">1260</span> isTableState(regionNode.getTable(), TableState.State.DISABLED,<a name="line.1260"></a>
+<span class="sourceLineNo">1261</span> TableState.State.DISABLING)) {<a name="line.1261"></a>
+<span class="sourceLineNo">1262</span> // The region is CLOSED and the table is DISABLED/ DISABLING, there is nothing to<a name="line.1262"></a>
+<span class="sourceLineNo">1263</span> // schedule; the region is inert.<a name="line.1263"></a>
+<span class="sourceLineNo">1264</span> } else {<a name="line.1264"></a>
+<span class="sourceLineNo">1265</span> // These regions should have a procedure in replay<a name="line.1265"></a>
+<span class="sourceLineNo">1266</span> regionStates.addRegionInTransition(regionNode, null);<a name="line.1266"></a>
+<span class="sourceLineNo">1267</span> }<a name="line.1267"></a>
+<span class="sourceLineNo">1268</span> }<a name="line.1268"></a>
+<span class="sourceLineNo">1269</span> }<a name="line.1269"></a>
+<span class="sourceLineNo">1270</span> }<a name="line.1270"></a>
+<span class="sourceLineNo">1271</span> });<a name="line.1271"></a>
+<span class="sourceLineNo">1272</span><a name="line.1272"></a>
+<span class="sourceLineNo">1273</span> // every assignment is blocked until meta is loaded.<a name="line.1273"></a>
+<span class="sourceLineNo">1274</span> wakeMetaLoadedEvent();<a name="line.1274"></a>
+<span class="sourceLineNo">1275</span> }<a name="line.1275"></a>
+<span class="sourceLineNo">1276</span><a name="line.1276"></a>
+<span class="sourceLineNo">1277</span> /**<a name="line.1277"></a>
+<span class="sourceLineNo">1278</span> * Look at what is in meta and the list of servers that have checked in and make reconciliation.<a name="line.1278"></a>
+<span class="sourceLineNo">1279</span> * We cannot tell definitively the difference between a clean shutdown and a cluster that has<a name="line.1279"></a>
+<span class="sourceLineNo">1280</span> * been crashed down. At this stage of a Master startup, they look the same: they have the<a name="line.1280"></a>
+<span class="sourceLineNo">1281</span> * same state in hbase:meta. We could do detective work probing ZK and the FS for old WALs to<a name="line.1281"></a>
+<span class="sourceLineNo">1282</span> * split but SCP does this already so just let it do its job.<a name="line.1282"></a>
+<span class="sourceLineNo">1283</span> * <p>>The profiles of clean shutdown and cluster crash-down are the same because on clean<a name="line.1283"></a>
+<span class="sourceLineNo">1284</span> * shutdown currently, we do not update hbase:meta with region close state (In AMv2, region<a name="line.1284"></a>
+<span class="sourceLineNo">1285</span> * state is kept in hbse:meta). Usually the master runs all region transitions as of AMv2 but on<a name="line.1285"></a>
+<span class="sourceLineNo">1286</span> * cluster controlled shutdown, the RegionServers close all their regions only reporting the<a name="line.1286"></a>
+<span class="sourceLineNo">1287</span> * final change to the Master. Currently this report is ignored. Later we could take it and<a name="line.1287"></a>
+<span class="sourceLineNo">1288</span> * update as many regions as we can before hbase:meta goes down or have the master run the<a name="line.1288"></a>
+<span class="sourceLineNo">1289</span> * close of all regions out on the cluster but we may never be able to achieve the proper state on<a name="line.1289"></a>
+<span class="sourceLineNo">1290</span> * all regions (at least not w/o lots of painful manipulations and waiting) so clean shutdown<a name="line.1290"></a>
+<span class="sourceLineNo">1291</span> * might not be possible especially on big clusters.... And clean shutdown will take time. Given<a name="line.1291"></a>
+<span class="sourceLineNo">1292</span> * this current state of affairs, we just run ServerCrashProcedure in both cases. It will always<a name="line.1292"></a>
+<span class="sourceLineNo">1293</span> * do the right thing.<a name="line.1293"></a>
+<span class="sourceLineNo">1294</span> * @return True if for sure this is a failover where a Master is starting up into an already<a name="line.1294"></a>
+<span class="sourceLineNo">1295</span> * running cluster.<a name="line.1295"></a>
+<span class="sourceLineNo">1296</span> */<a name="line.1296"></a>
+<span class="sourceLineNo">1297</span> // The assumption here is that if RSs are crashing while we are executing this<a name="line.1297"></a>
+<span class="sourceLineNo">1298</span> // they will be handled by the SSH that are put in the ServerManager deadservers "queue".<a name="line.1298"></a>
+<span class="sourceLineNo">1299</span> private boolean processofflineServersWithOnlineRegions() {<a name="line.1299"></a>
+<span class="sourceLineNo">1300</span> boolean deadServers = !master.getServerManager().getDeadServers().isEmpty();<a name="line.1300"></a>
+<span class="sourceLineNo">1301</span> final Set<ServerName> offlineServersWithOnlineRegions = new HashSet<>();<a name="line.1301"></a>
+<span class="sourceLineNo">1302</span> int size = regionStates.getRegionStateNodes().size();<a name="line.1302"></a>
+<span class="sourceLineNo">1303</span> final List<RegionInfo> offlineRegionsToAssign = new ArrayList<>(size);<a name="line.1303"></a>
+<span class="sourceLineNo">1304</span> // If deadservers then its a failover, else, we are not sure yet.<a name="line.1304"></a>
+<span class="sourceLineNo">1305</span> boolean failover = deadServers;<a name="line.1305"></a>
+<span class="sourceLineNo">1306</span> for (RegionStateNode regionNode: regionStates.getRegionStateNodes()) {<a name="line.1306"></a>
+<span class="sourceLineNo">1307</span> // Region State can be OPEN even if we did controlled cluster shutdown; Master does not close<a name="line.1307"></a>
+<span class="sourceLineNo">1308</span> // the regions in this case. The RegionServer does the close so hbase:meta is state in<a name="line.1308"></a>
+<span class="sourceLineNo">1309</span> // hbase:meta is not updated -- Master does all updates -- and is left with OPEN as region<a name="line.1309"></a>
+<span class="sourceLineNo">1310</span> // state in meta. How to tell difference between ordered shutdown and crashed-down cluster<a name="line.1310"></a>
+<span class="sourceLineNo">1311</span> // then? We can't. Not currently. Perhaps if we updated hbase:meta with CLOSED on ordered<a name="line.1311"></a>
+<span class="sourceLineNo">1312</span> // shutdown. This would slow shutdown though and not all edits would make it in anyways.<a name="line.1312"></a>
+<span class="sourceLineNo">1313</span> // TODO: Examine.<a name="line.1313"></a>
+<span class="sourceLineNo">1314</span> // Because we can't be sure it an ordered shutdown, we run ServerCrashProcedure always.<a name="line.1314"></a>
+<span class="sourceLineNo">1315</span> // ServerCrashProcedure will try to retain old deploy when it goes to assign.<a name="line.1315"></a>
+<span class="sourceLineNo">1316</span> if (regionNode.getState() == State.OPEN) {<a name="line.1316"></a>
+<span class="sourceLineNo">1317</span> final ServerName serverName = regionNode.getRegionLocation();<a name="line.1317"></a>
+<span class="sourceLineNo">1318</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1318"></a>
+<span class="sourceLineNo">1319</span> offlineServersWithOnlineRegions.add(serverName);<a name="line.1319"></a>
+<span class="sourceLineNo">1320</span> } else {<a name="line.1320"></a>
+<span class="sourceLineNo">1321</span> // Server is online. This a failover. Master is starting into already-running cluster.<a name="line.1321"></a>
+<span class="sourceLineNo">1322</span> failover = true;<a name="line.1322"></a>
+<span class="sourceLineNo">1323</span> }<a name="line.1323"></a>
+<span class="sourceLineNo">1324</span> } else if (regionNode.getState() == State.OFFLINE) {<a name="line.1324"></a>
+<span class="sourceLineNo">1325</span> if (isTableEnabled(regionNode.getTable())) {<a name="line.1325"></a>
+<span class="sourceLineNo">1326</span> offlineRegionsToAssign.add(regionNode.getRegionInfo());<a name="line.1326"></a>
+<span class="sourceLineNo">1327</span> }<a name="line.1327"></a>
+<span class="sourceLineNo">1328</span> }<a name="line.1328"></a>
+<span class="sourceLineNo">1329</span> }<a name="line.1329"></a>
+<span class="sourceLineNo">1330</span> // Kill servers with online regions just-in-case. Runs ServerCrashProcedure.<a name="line.1330"></a>
+<span class="sourceLineNo">1331</span> for (ServerName serverName: offlineServersWithOnlineRegions) {<a name="line.1331"></a>
+<span class="sourceLineNo">1332</span> if (!master.getServerManager().isServerOnline(serverName)) {<a name="line.1332"></a>
+<span class="sourceLineNo">1333</span> LOG.info("KILL RegionServer=" + serverName + " hosting regions but not online.");<a name="line.1333"></a>
+<span class="sourceLineNo">1334</span> killRegionServer(serverName);<a name="line.1334"></a>
+<span class="sourceLineNo">1335</span> }<a name="line.1335"></a>
+<span class="sourceLineNo">1336</span> }<a name="line.1336"></a>
+<span class="sourceLineNo">1337</span> setFailoverCleanupDone(true);<a name="line.1337"></a>
+<span class="sourceLineNo">1338</span><a name="line.1338"></a>
+<span class="sourceLineNo">1339</span> // Assign offline regions. Uses round-robin.<a name="line.1339"></a>
+<span class="sourceLineNo">1340</span> if (offlineRegionsToAssign.size() > 0) {<a name="line.1340"></a>
+<span class="sourceLineNo">1341</span> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().<a name="line.1341"></a>
+<span class="sourceLineNo">1342</span> createRoundRobinAssignProcedures(offlineRegionsToAssign));<a name="line.1342"></a>
+<span class="sourceLineNo">1343</span> }<a name="line.1343"></a>
+<span class="sourceLineNo">1344</span><a name="line.1344"></a>
+<span class="sourceLineNo">1345</span> return failover;<a name="line.1345"></a>
+<span class="sourceLineNo">1346</span> }<a name="line.1346"></a>
+<span class="sourceLineNo">1347</span><a name="line.1347"></a>
+<span class="sourceLineNo">1348</span> /**<a name="line.1348"></a>
+<span class="sourceLineNo">1349</span> * Used by ServerCrashProcedure to make sure AssignmentManager has completed<a name="line.1349"></a>
+<span class="sourceLineNo">1350</span> * the failover cleanup before re-assigning regions of dead servers. So that<a name="line.1350"></a>
+<span class="sourceLineNo">1351</span> * when re-assignment happens, AssignmentManager has proper region states.<a name="line.1351"></a>
+<span class="sourceLineNo">1352</span> */<a name="line.1352"></a>
+<span class="sourceLineNo">1353</span> public boolean isFailoverCleanupDone() {<a name="line.1353"></a>
+<span class="sourceLineNo">1354</span> return failoverCleanupDone.isReady();<a name="line.1354"></a>
+<span class="sourceLineNo">1355</span> }<a name="line.1355"></a>
+<span class="sourceLineNo">1356</span><a name="line.1356"></a>
+<span class="sourceLineNo">1357</span> /**<a name="line.1357"></a>
+<span class="sourceLineNo">1358</span> * Used by ServerCrashProcedure tests verify the ability to suspend the<a name="line.1358"></a>
+<span class="sourceLineNo">1359</span> * execution of the ServerCrashProcedure.<a name="line.1359"></a>
+<span class="sourceLineNo">1360</span> */<a name="line.1360"></a>
+<span class="sourceLineNo">1361</span> @VisibleForTesting<a name="line.1361"></a>
+<span class="sourceLineNo">1362</span> public void setFailoverCleanupDone(final boolean b) {<a name="line.1362"></a>
+<span class="sourceLineNo">1363</span> master.getMasterProcedureExecutor().getEnvironment()<a name="line.1363"></a>
+<span class="sourceLineNo">1364</span> .setEventReady(failoverCleanupDone, b);<a name="line.1364"></a>
+<span class="sourceLineNo">1365</span> }<a name="line.1365"></a>
+<span class="sourceLineNo">1366</span><a name="line.1366"></a>
+<span class="sourceLineNo">1367</span> public ProcedureEvent getFailoverCleanupEvent() {<a name="line.1367"></a>
+<span class="sourceLineNo">1368</span> return failoverCleanupDone;<a name="line.1368"></a>
+<span class="sourceLineNo">1369</span> }<a name="line.1369"></a>
+<span class="sourceLineNo">1370</span><a name="line.1370"></a>
+<span class="sourceLineNo">1371</span> /**<a name="line.1371"></a>
+<span class="sourceLineNo">1372</span> * Used to check if the failover cleanup is done.<a name="line.1372"></a>
+<span class="sourceLineNo">1373</span> * if not we throw PleaseHoldException since we are rebuilding the RegionStates<a name="line.1373"></a>
+<span class="sourceLineNo">1374</span> * @param hri region to check if it is already rebuild<a name="line.1374"></a>
+<span class="sourceLineNo">1375</span> * @throws PleaseHoldException if the failover cleanup is not completed<a name="line.1375"></a>
+<span class="sourceLineNo">1376</span> */<a name="line.1376"></a>
+<span class="sourceLineNo">1377</span> private void checkFailoverCleanupCompleted(final RegionInfo hri) throws PleaseHoldException {<a name="line.1377"></a>
+<span class="sourceLineNo">1378</span> if (!isRunning()) {<a name="line.1378"></a>
+<span class="sourceLineNo">1379</span> throw new PleaseHoldException("AssignmentManager not running");<a name="line.1379"></a>
+<span class="sourceLineNo">1380</span> }<a name="line.1380"></a>
+<span class="sourceLineNo">1381</span><a name="line.1381"></a>
+<span class="sourceLineNo">1382</span> // TODO: can we avoid throwing an exception if hri is already loaded?<a name="line.1382"></a>
+<span class="sourceLineNo">1383</span> // at the moment we bypass only meta<a name="line.1383"></a>
+<span class="sourceLineNo">1384</span> boolean meta = isMetaRegion(hri);<a name="line.1384"></a>
+<span class="sourceLineNo">1385</span> boolean cleanup = isFailoverCleanupDone();<a name="line.1385"></a>
+<span class="sourceLineNo">1386</span> if (!isMetaRegion(hri) && !isFailoverCleanupDone()) {<a name="line.1386"></a>
+<span class="sourceLineNo">1387</span> String msg = "Master not fully online; hbase:meta=" + meta + ", failoverCleanup=" + cleanup;<a name="line.1387"></a>
+<span class="sourceLineNo">1388</span> throw new PleaseHoldException(msg);<a name="line.1388"></a>
+<span class="sourceLineNo">1389</span> }<a name="line.1389"></a>
+<span class="sourceLineNo">1390</span> }<a name="line.1390"></a>
+<span clas
<TRUNCATED>
[04/16] hbase-site git commit: Published site at
4cb444e77b41cdb733544770a471068256d65bbe.
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/hbase-site/blob/2d03f412/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html
----------------------------------------------------------------------
diff --git a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html
index 97aa79c..cc787e3 100644
--- a/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html
+++ b/devapidocs/src-html/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.WorkerMonitor.html
@@ -272,1623 +272,1642 @@
<span class="sourceLineNo">264</span> private final CopyOnWriteArrayList<ProcedureExecutorListener> listeners = new CopyOnWriteArrayList<>();<a name="line.264"></a>
<span class="sourceLineNo">265</span><a name="line.265"></a>
<span class="sourceLineNo">266</span> private Configuration conf;<a name="line.266"></a>
-<span class="sourceLineNo">267</span> private ThreadGroup threadGroup;<a name="line.267"></a>
-<span class="sourceLineNo">268</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.268"></a>
-<span class="sourceLineNo">269</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.269"></a>
-<span class="sourceLineNo">270</span> private int corePoolSize;<a name="line.270"></a>
-<span class="sourceLineNo">271</span> private int maxPoolSize;<a name="line.271"></a>
-<span class="sourceLineNo">272</span><a name="line.272"></a>
-<span class="sourceLineNo">273</span> private volatile long keepAliveTime;<a name="line.273"></a>
-<span class="sourceLineNo">274</span><a name="line.274"></a>
-<span class="sourceLineNo">275</span> /**<a name="line.275"></a>
-<span class="sourceLineNo">276</span> * Scheduler/Queue that contains runnable procedures.<a name="line.276"></a>
-<span class="sourceLineNo">277</span> */<a name="line.277"></a>
-<span class="sourceLineNo">278</span> private final ProcedureScheduler scheduler;<a name="line.278"></a>
-<span class="sourceLineNo">279</span><a name="line.279"></a>
-<span class="sourceLineNo">280</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.280"></a>
-<span class="sourceLineNo">281</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.281"></a>
-<span class="sourceLineNo">282</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.282"></a>
-<span class="sourceLineNo">283</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.283"></a>
-<span class="sourceLineNo">284</span> private final TEnvironment environment;<a name="line.284"></a>
-<span class="sourceLineNo">285</span> private final ProcedureStore store;<a name="line.285"></a>
-<span class="sourceLineNo">286</span><a name="line.286"></a>
-<span class="sourceLineNo">287</span> private final boolean checkOwnerSet;<a name="line.287"></a>
-<span class="sourceLineNo">288</span><a name="line.288"></a>
-<span class="sourceLineNo">289</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.289"></a>
-<span class="sourceLineNo">290</span> final ProcedureStore store) {<a name="line.290"></a>
-<span class="sourceLineNo">291</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.291"></a>
-<span class="sourceLineNo">292</span> }<a name="line.292"></a>
-<span class="sourceLineNo">293</span><a name="line.293"></a>
-<span class="sourceLineNo">294</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.294"></a>
-<span class="sourceLineNo">295</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.295"></a>
-<span class="sourceLineNo">296</span> this.environment = environment;<a name="line.296"></a>
-<span class="sourceLineNo">297</span> this.scheduler = scheduler;<a name="line.297"></a>
-<span class="sourceLineNo">298</span> this.store = store;<a name="line.298"></a>
-<span class="sourceLineNo">299</span> this.conf = conf;<a name="line.299"></a>
-<span class="sourceLineNo">300</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.300"></a>
-<span class="sourceLineNo">301</span> refreshConfiguration(conf);<a name="line.301"></a>
-<span class="sourceLineNo">302</span> }<a name="line.302"></a>
-<span class="sourceLineNo">303</span><a name="line.303"></a>
-<span class="sourceLineNo">304</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.304"></a>
-<span class="sourceLineNo">305</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.305"></a>
-<span class="sourceLineNo">306</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.306"></a>
-<span class="sourceLineNo">307</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.307"></a>
-<span class="sourceLineNo">308</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.308"></a>
-<span class="sourceLineNo">309</span><a name="line.309"></a>
-<span class="sourceLineNo">310</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.310"></a>
-<span class="sourceLineNo">311</span> @Override<a name="line.311"></a>
-<span class="sourceLineNo">312</span> public void setMaxProcId(long maxProcId) {<a name="line.312"></a>
-<span class="sourceLineNo">313</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.313"></a>
-<span class="sourceLineNo">314</span> lastProcId.set(maxProcId);<a name="line.314"></a>
-<span class="sourceLineNo">315</span> }<a name="line.315"></a>
-<span class="sourceLineNo">316</span><a name="line.316"></a>
-<span class="sourceLineNo">317</span> @Override<a name="line.317"></a>
-<span class="sourceLineNo">318</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.318"></a>
-<span class="sourceLineNo">319</span> loadProcedures(procIter, abortOnCorruption);<a name="line.319"></a>
-<span class="sourceLineNo">320</span> }<a name="line.320"></a>
-<span class="sourceLineNo">321</span><a name="line.321"></a>
-<span class="sourceLineNo">322</span> @Override<a name="line.322"></a>
-<span class="sourceLineNo">323</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.323"></a>
-<span class="sourceLineNo">324</span> int corruptedCount = 0;<a name="line.324"></a>
-<span class="sourceLineNo">325</span> while (procIter.hasNext()) {<a name="line.325"></a>
-<span class="sourceLineNo">326</span> Procedure<?> proc = procIter.next();<a name="line.326"></a>
-<span class="sourceLineNo">327</span> LOG.error("Corrupt " + proc);<a name="line.327"></a>
-<span class="sourceLineNo">328</span> corruptedCount++;<a name="line.328"></a>
-<span class="sourceLineNo">329</span> }<a name="line.329"></a>
-<span class="sourceLineNo">330</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.330"></a>
-<span class="sourceLineNo">331</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.331"></a>
-<span class="sourceLineNo">332</span> }<a name="line.332"></a>
-<span class="sourceLineNo">333</span> }<a name="line.333"></a>
-<span class="sourceLineNo">334</span> });<a name="line.334"></a>
-<span class="sourceLineNo">335</span> }<a name="line.335"></a>
-<span class="sourceLineNo">336</span><a name="line.336"></a>
-<span class="sourceLineNo">337</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.337"></a>
-<span class="sourceLineNo">338</span> final boolean abortOnCorruption) throws IOException {<a name="line.338"></a>
-<span class="sourceLineNo">339</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.339"></a>
-<span class="sourceLineNo">340</span><a name="line.340"></a>
-<span class="sourceLineNo">341</span> // 1. Build the rollback stack<a name="line.341"></a>
-<span class="sourceLineNo">342</span> int runnablesCount = 0;<a name="line.342"></a>
-<span class="sourceLineNo">343</span> while (procIter.hasNext()) {<a name="line.343"></a>
-<span class="sourceLineNo">344</span> boolean finished = procIter.isNextFinished();<a name="line.344"></a>
-<span class="sourceLineNo">345</span> Procedure proc = procIter.next();<a name="line.345"></a>
-<span class="sourceLineNo">346</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.346"></a>
-<span class="sourceLineNo">347</span> long procId = proc.getProcId();<a name="line.347"></a>
-<span class="sourceLineNo">348</span><a name="line.348"></a>
-<span class="sourceLineNo">349</span> if (finished) {<a name="line.349"></a>
-<span class="sourceLineNo">350</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.350"></a>
-<span class="sourceLineNo">351</span> if (debugEnabled) {<a name="line.351"></a>
-<span class="sourceLineNo">352</span> LOG.debug("Completed " + proc);<a name="line.352"></a>
-<span class="sourceLineNo">353</span> }<a name="line.353"></a>
-<span class="sourceLineNo">354</span> } else {<a name="line.354"></a>
-<span class="sourceLineNo">355</span> if (!proc.hasParent()) {<a name="line.355"></a>
-<span class="sourceLineNo">356</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.356"></a>
-<span class="sourceLineNo">357</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.357"></a>
-<span class="sourceLineNo">358</span> }<a name="line.358"></a>
+<span class="sourceLineNo">267</span><a name="line.267"></a>
+<span class="sourceLineNo">268</span> /**<a name="line.268"></a>
+<span class="sourceLineNo">269</span> * Created in the {@link #start(int, boolean)} method. Destroyed in {@link #join()} (FIX! Doing<a name="line.269"></a>
+<span class="sourceLineNo">270</span> * resource handling rather than observing in a #join is unexpected).<a name="line.270"></a>
+<span class="sourceLineNo">271</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.271"></a>
+<span class="sourceLineNo">272</span> * (Should be ok).<a name="line.272"></a>
+<span class="sourceLineNo">273</span> */<a name="line.273"></a>
+<span class="sourceLineNo">274</span> private ThreadGroup threadGroup;<a name="line.274"></a>
+<span class="sourceLineNo">275</span><a name="line.275"></a>
+<span class="sourceLineNo">276</span> /**<a name="line.276"></a>
+<span class="sourceLineNo">277</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.277"></a>
+<span class="sourceLineNo">278</span> * resource handling rather than observing in a #join is unexpected).<a name="line.278"></a>
+<span class="sourceLineNo">279</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.279"></a>
+<span class="sourceLineNo">280</span> * (Should be ok).<a name="line.280"></a>
+<span class="sourceLineNo">281</span> */<a name="line.281"></a>
+<span class="sourceLineNo">282</span> private CopyOnWriteArrayList<WorkerThread> workerThreads;<a name="line.282"></a>
+<span class="sourceLineNo">283</span><a name="line.283"></a>
+<span class="sourceLineNo">284</span> /**<a name="line.284"></a>
+<span class="sourceLineNo">285</span> * Created in the {@link #start(int, boolean)} method. Terminated in {@link #join()} (FIX! Doing<a name="line.285"></a>
+<span class="sourceLineNo">286</span> * resource handling rather than observing in a #join is unexpected).<a name="line.286"></a>
+<span class="sourceLineNo">287</span> * Overridden when we do the ProcedureTestingUtility.testRecoveryAndDoubleExecution trickery<a name="line.287"></a>
+<span class="sourceLineNo">288</span> * (Should be ok).<a name="line.288"></a>
+<span class="sourceLineNo">289</span> */<a name="line.289"></a>
+<span class="sourceLineNo">290</span> private TimeoutExecutorThread timeoutExecutor;<a name="line.290"></a>
+<span class="sourceLineNo">291</span><a name="line.291"></a>
+<span class="sourceLineNo">292</span> private int corePoolSize;<a name="line.292"></a>
+<span class="sourceLineNo">293</span> private int maxPoolSize;<a name="line.293"></a>
+<span class="sourceLineNo">294</span><a name="line.294"></a>
+<span class="sourceLineNo">295</span> private volatile long keepAliveTime;<a name="line.295"></a>
+<span class="sourceLineNo">296</span><a name="line.296"></a>
+<span class="sourceLineNo">297</span> /**<a name="line.297"></a>
+<span class="sourceLineNo">298</span> * Scheduler/Queue that contains runnable procedures.<a name="line.298"></a>
+<span class="sourceLineNo">299</span> */<a name="line.299"></a>
+<span class="sourceLineNo">300</span> private final ProcedureScheduler scheduler;<a name="line.300"></a>
+<span class="sourceLineNo">301</span><a name="line.301"></a>
+<span class="sourceLineNo">302</span> private final AtomicLong lastProcId = new AtomicLong(-1);<a name="line.302"></a>
+<span class="sourceLineNo">303</span> private final AtomicLong workerId = new AtomicLong(0);<a name="line.303"></a>
+<span class="sourceLineNo">304</span> private final AtomicInteger activeExecutorCount = new AtomicInteger(0);<a name="line.304"></a>
+<span class="sourceLineNo">305</span> private final AtomicBoolean running = new AtomicBoolean(false);<a name="line.305"></a>
+<span class="sourceLineNo">306</span> private final TEnvironment environment;<a name="line.306"></a>
+<span class="sourceLineNo">307</span> private final ProcedureStore store;<a name="line.307"></a>
+<span class="sourceLineNo">308</span><a name="line.308"></a>
+<span class="sourceLineNo">309</span> private final boolean checkOwnerSet;<a name="line.309"></a>
+<span class="sourceLineNo">310</span><a name="line.310"></a>
+<span class="sourceLineNo">311</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.311"></a>
+<span class="sourceLineNo">312</span> final ProcedureStore store) {<a name="line.312"></a>
+<span class="sourceLineNo">313</span> this(conf, environment, store, new SimpleProcedureScheduler());<a name="line.313"></a>
+<span class="sourceLineNo">314</span> }<a name="line.314"></a>
+<span class="sourceLineNo">315</span><a name="line.315"></a>
+<span class="sourceLineNo">316</span> public ProcedureExecutor(final Configuration conf, final TEnvironment environment,<a name="line.316"></a>
+<span class="sourceLineNo">317</span> final ProcedureStore store, final ProcedureScheduler scheduler) {<a name="line.317"></a>
+<span class="sourceLineNo">318</span> this.environment = environment;<a name="line.318"></a>
+<span class="sourceLineNo">319</span> this.scheduler = scheduler;<a name="line.319"></a>
+<span class="sourceLineNo">320</span> this.store = store;<a name="line.320"></a>
+<span class="sourceLineNo">321</span> this.conf = conf;<a name="line.321"></a>
+<span class="sourceLineNo">322</span> this.checkOwnerSet = conf.getBoolean(CHECK_OWNER_SET_CONF_KEY, DEFAULT_CHECK_OWNER_SET);<a name="line.322"></a>
+<span class="sourceLineNo">323</span> refreshConfiguration(conf);<a name="line.323"></a>
+<span class="sourceLineNo">324</span><a name="line.324"></a>
+<span class="sourceLineNo">325</span> }<a name="line.325"></a>
+<span class="sourceLineNo">326</span><a name="line.326"></a>
+<span class="sourceLineNo">327</span> private void load(final boolean abortOnCorruption) throws IOException {<a name="line.327"></a>
+<span class="sourceLineNo">328</span> Preconditions.checkArgument(completed.isEmpty(), "completed not empty");<a name="line.328"></a>
+<span class="sourceLineNo">329</span> Preconditions.checkArgument(rollbackStack.isEmpty(), "rollback state not empty");<a name="line.329"></a>
+<span class="sourceLineNo">330</span> Preconditions.checkArgument(procedures.isEmpty(), "procedure map not empty");<a name="line.330"></a>
+<span class="sourceLineNo">331</span> Preconditions.checkArgument(scheduler.size() == 0, "run queue not empty");<a name="line.331"></a>
+<span class="sourceLineNo">332</span><a name="line.332"></a>
+<span class="sourceLineNo">333</span> store.load(new ProcedureStore.ProcedureLoader() {<a name="line.333"></a>
+<span class="sourceLineNo">334</span> @Override<a name="line.334"></a>
+<span class="sourceLineNo">335</span> public void setMaxProcId(long maxProcId) {<a name="line.335"></a>
+<span class="sourceLineNo">336</span> assert lastProcId.get() < 0 : "expected only one call to setMaxProcId()";<a name="line.336"></a>
+<span class="sourceLineNo">337</span> lastProcId.set(maxProcId);<a name="line.337"></a>
+<span class="sourceLineNo">338</span> }<a name="line.338"></a>
+<span class="sourceLineNo">339</span><a name="line.339"></a>
+<span class="sourceLineNo">340</span> @Override<a name="line.340"></a>
+<span class="sourceLineNo">341</span> public void load(ProcedureIterator procIter) throws IOException {<a name="line.341"></a>
+<span class="sourceLineNo">342</span> loadProcedures(procIter, abortOnCorruption);<a name="line.342"></a>
+<span class="sourceLineNo">343</span> }<a name="line.343"></a>
+<span class="sourceLineNo">344</span><a name="line.344"></a>
+<span class="sourceLineNo">345</span> @Override<a name="line.345"></a>
+<span class="sourceLineNo">346</span> public void handleCorrupted(ProcedureIterator procIter) throws IOException {<a name="line.346"></a>
+<span class="sourceLineNo">347</span> int corruptedCount = 0;<a name="line.347"></a>
+<span class="sourceLineNo">348</span> while (procIter.hasNext()) {<a name="line.348"></a>
+<span class="sourceLineNo">349</span> Procedure<?> proc = procIter.next();<a name="line.349"></a>
+<span class="sourceLineNo">350</span> LOG.error("Corrupt " + proc);<a name="line.350"></a>
+<span class="sourceLineNo">351</span> corruptedCount++;<a name="line.351"></a>
+<span class="sourceLineNo">352</span> }<a name="line.352"></a>
+<span class="sourceLineNo">353</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.353"></a>
+<span class="sourceLineNo">354</span> throw new IOException("found " + corruptedCount + " corrupted procedure(s) on replay");<a name="line.354"></a>
+<span class="sourceLineNo">355</span> }<a name="line.355"></a>
+<span class="sourceLineNo">356</span> }<a name="line.356"></a>
+<span class="sourceLineNo">357</span> });<a name="line.357"></a>
+<span class="sourceLineNo">358</span> }<a name="line.358"></a>
<span class="sourceLineNo">359</span><a name="line.359"></a>
-<span class="sourceLineNo">360</span> // add the procedure to the map<a name="line.360"></a>
-<span class="sourceLineNo">361</span> proc.beforeReplay(getEnvironment());<a name="line.361"></a>
-<span class="sourceLineNo">362</span> procedures.put(proc.getProcId(), proc);<a name="line.362"></a>
+<span class="sourceLineNo">360</span> private void loadProcedures(final ProcedureIterator procIter,<a name="line.360"></a>
+<span class="sourceLineNo">361</span> final boolean abortOnCorruption) throws IOException {<a name="line.361"></a>
+<span class="sourceLineNo">362</span> final boolean debugEnabled = LOG.isDebugEnabled();<a name="line.362"></a>
<span class="sourceLineNo">363</span><a name="line.363"></a>
-<span class="sourceLineNo">364</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.364"></a>
-<span class="sourceLineNo">365</span> runnablesCount++;<a name="line.365"></a>
-<span class="sourceLineNo">366</span> }<a name="line.366"></a>
-<span class="sourceLineNo">367</span> }<a name="line.367"></a>
-<span class="sourceLineNo">368</span><a name="line.368"></a>
-<span class="sourceLineNo">369</span> // add the nonce to the map<a name="line.369"></a>
-<span class="sourceLineNo">370</span> if (nonceKey != null) {<a name="line.370"></a>
-<span class="sourceLineNo">371</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.371"></a>
-<span class="sourceLineNo">372</span> }<a name="line.372"></a>
-<span class="sourceLineNo">373</span> }<a name="line.373"></a>
-<span class="sourceLineNo">374</span><a name="line.374"></a>
-<span class="sourceLineNo">375</span> // 2. Initialize the stacks<a name="line.375"></a>
-<span class="sourceLineNo">376</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.376"></a>
-<span class="sourceLineNo">377</span> HashSet<Procedure> waitingSet = null;<a name="line.377"></a>
-<span class="sourceLineNo">378</span> procIter.reset();<a name="line.378"></a>
-<span class="sourceLineNo">379</span> while (procIter.hasNext()) {<a name="line.379"></a>
-<span class="sourceLineNo">380</span> if (procIter.isNextFinished()) {<a name="line.380"></a>
-<span class="sourceLineNo">381</span> procIter.skipNext();<a name="line.381"></a>
-<span class="sourceLineNo">382</span> continue;<a name="line.382"></a>
-<span class="sourceLineNo">383</span> }<a name="line.383"></a>
-<span class="sourceLineNo">384</span><a name="line.384"></a>
-<span class="sourceLineNo">385</span> Procedure proc = procIter.next();<a name="line.385"></a>
-<span class="sourceLineNo">386</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.386"></a>
-<span class="sourceLineNo">387</span><a name="line.387"></a>
-<span class="sourceLineNo">388</span> if (debugEnabled) {<a name="line.388"></a>
-<span class="sourceLineNo">389</span> LOG.debug(String.format("Loading %s", proc));<a name="line.389"></a>
+<span class="sourceLineNo">364</span> // 1. Build the rollback stack<a name="line.364"></a>
+<span class="sourceLineNo">365</span> int runnablesCount = 0;<a name="line.365"></a>
+<span class="sourceLineNo">366</span> while (procIter.hasNext()) {<a name="line.366"></a>
+<span class="sourceLineNo">367</span> boolean finished = procIter.isNextFinished();<a name="line.367"></a>
+<span class="sourceLineNo">368</span> Procedure proc = procIter.next();<a name="line.368"></a>
+<span class="sourceLineNo">369</span> NonceKey nonceKey = proc.getNonceKey();<a name="line.369"></a>
+<span class="sourceLineNo">370</span> long procId = proc.getProcId();<a name="line.370"></a>
+<span class="sourceLineNo">371</span><a name="line.371"></a>
+<span class="sourceLineNo">372</span> if (finished) {<a name="line.372"></a>
+<span class="sourceLineNo">373</span> completed.put(proc.getProcId(), new CompletedProcedureRetainer(proc));<a name="line.373"></a>
+<span class="sourceLineNo">374</span> if (debugEnabled) {<a name="line.374"></a>
+<span class="sourceLineNo">375</span> LOG.debug("Completed " + proc);<a name="line.375"></a>
+<span class="sourceLineNo">376</span> }<a name="line.376"></a>
+<span class="sourceLineNo">377</span> } else {<a name="line.377"></a>
+<span class="sourceLineNo">378</span> if (!proc.hasParent()) {<a name="line.378"></a>
+<span class="sourceLineNo">379</span> assert !proc.isFinished() : "unexpected finished procedure";<a name="line.379"></a>
+<span class="sourceLineNo">380</span> rollbackStack.put(proc.getProcId(), new RootProcedureState());<a name="line.380"></a>
+<span class="sourceLineNo">381</span> }<a name="line.381"></a>
+<span class="sourceLineNo">382</span><a name="line.382"></a>
+<span class="sourceLineNo">383</span> // add the procedure to the map<a name="line.383"></a>
+<span class="sourceLineNo">384</span> proc.beforeReplay(getEnvironment());<a name="line.384"></a>
+<span class="sourceLineNo">385</span> procedures.put(proc.getProcId(), proc);<a name="line.385"></a>
+<span class="sourceLineNo">386</span><a name="line.386"></a>
+<span class="sourceLineNo">387</span> if (proc.getState() == ProcedureState.RUNNABLE) {<a name="line.387"></a>
+<span class="sourceLineNo">388</span> runnablesCount++;<a name="line.388"></a>
+<span class="sourceLineNo">389</span> }<a name="line.389"></a>
<span class="sourceLineNo">390</span> }<a name="line.390"></a>
<span class="sourceLineNo">391</span><a name="line.391"></a>
-<span class="sourceLineNo">392</span> Long rootProcId = getRootProcedureId(proc);<a name="line.392"></a>
-<span class="sourceLineNo">393</span> if (rootProcId == null) {<a name="line.393"></a>
-<span class="sourceLineNo">394</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.394"></a>
-<span class="sourceLineNo">395</span> scheduler.addBack(proc);<a name="line.395"></a>
-<span class="sourceLineNo">396</span> continue;<a name="line.396"></a>
-<span class="sourceLineNo">397</span> }<a name="line.397"></a>
-<span class="sourceLineNo">398</span><a name="line.398"></a>
-<span class="sourceLineNo">399</span> if (proc.hasParent()) {<a name="line.399"></a>
-<span class="sourceLineNo">400</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.400"></a>
-<span class="sourceLineNo">401</span> // corrupted procedures are handled later at step 3<a name="line.401"></a>
-<span class="sourceLineNo">402</span> if (parent != null && !proc.isFinished()) {<a name="line.402"></a>
-<span class="sourceLineNo">403</span> parent.incChildrenLatch();<a name="line.403"></a>
-<span class="sourceLineNo">404</span> }<a name="line.404"></a>
-<span class="sourceLineNo">405</span> }<a name="line.405"></a>
-<span class="sourceLineNo">406</span><a name="line.406"></a>
-<span class="sourceLineNo">407</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.407"></a>
-<span class="sourceLineNo">408</span> procStack.loadStack(proc);<a name="line.408"></a>
-<span class="sourceLineNo">409</span><a name="line.409"></a>
-<span class="sourceLineNo">410</span> proc.setRootProcId(rootProcId);<a name="line.410"></a>
-<span class="sourceLineNo">411</span> switch (proc.getState()) {<a name="line.411"></a>
-<span class="sourceLineNo">412</span> case RUNNABLE:<a name="line.412"></a>
-<span class="sourceLineNo">413</span> runnableList.add(proc);<a name="line.413"></a>
-<span class="sourceLineNo">414</span> break;<a name="line.414"></a>
-<span class="sourceLineNo">415</span> case WAITING:<a name="line.415"></a>
-<span class="sourceLineNo">416</span> if (!proc.hasChildren()) {<a name="line.416"></a>
-<span class="sourceLineNo">417</span> runnableList.add(proc);<a name="line.417"></a>
-<span class="sourceLineNo">418</span> }<a name="line.418"></a>
-<span class="sourceLineNo">419</span> break;<a name="line.419"></a>
-<span class="sourceLineNo">420</span> case WAITING_TIMEOUT:<a name="line.420"></a>
-<span class="sourceLineNo">421</span> if (waitingSet == null) {<a name="line.421"></a>
-<span class="sourceLineNo">422</span> waitingSet = new HashSet<>();<a name="line.422"></a>
-<span class="sourceLineNo">423</span> }<a name="line.423"></a>
-<span class="sourceLineNo">424</span> waitingSet.add(proc);<a name="line.424"></a>
-<span class="sourceLineNo">425</span> break;<a name="line.425"></a>
-<span class="sourceLineNo">426</span> case FAILED:<a name="line.426"></a>
-<span class="sourceLineNo">427</span> // add the proc to the scheduler to perform the rollback<a name="line.427"></a>
-<span class="sourceLineNo">428</span> scheduler.addBack(proc);<a name="line.428"></a>
-<span class="sourceLineNo">429</span> break;<a name="line.429"></a>
-<span class="sourceLineNo">430</span> case ROLLEDBACK:<a name="line.430"></a>
-<span class="sourceLineNo">431</span> case INITIALIZING:<a name="line.431"></a>
-<span class="sourceLineNo">432</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.432"></a>
-<span class="sourceLineNo">433</span> LOG.error(msg);<a name="line.433"></a>
-<span class="sourceLineNo">434</span> throw new UnsupportedOperationException(msg);<a name="line.434"></a>
-<span class="sourceLineNo">435</span> default:<a name="line.435"></a>
-<span class="sourceLineNo">436</span> break;<a name="line.436"></a>
-<span class="sourceLineNo">437</span> }<a name="line.437"></a>
-<span class="sourceLineNo">438</span> }<a name="line.438"></a>
-<span class="sourceLineNo">439</span><a name="line.439"></a>
-<span class="sourceLineNo">440</span> // 3. Validate the stacks<a name="line.440"></a>
-<span class="sourceLineNo">441</span> int corruptedCount = 0;<a name="line.441"></a>
-<span class="sourceLineNo">442</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.442"></a>
-<span class="sourceLineNo">443</span> while (itStack.hasNext()) {<a name="line.443"></a>
-<span class="sourceLineNo">444</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.444"></a>
-<span class="sourceLineNo">445</span> RootProcedureState procStack = entry.getValue();<a name="line.445"></a>
-<span class="sourceLineNo">446</span> if (procStack.isValid()) continue;<a name="line.446"></a>
-<span class="sourceLineNo">447</span><a name="line.447"></a>
-<span class="sourceLineNo">448</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.448"></a>
-<span class="sourceLineNo">449</span> LOG.error("Corrupted " + proc);<a name="line.449"></a>
-<span class="sourceLineNo">450</span> procedures.remove(proc.getProcId());<a name="line.450"></a>
-<span class="sourceLineNo">451</span> runnableList.remove(proc);<a name="line.451"></a>
-<span class="sourceLineNo">452</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.452"></a>
-<span class="sourceLineNo">453</span> corruptedCount++;<a name="line.453"></a>
-<span class="sourceLineNo">454</span> }<a name="line.454"></a>
-<span class="sourceLineNo">455</span> itStack.remove();<a name="line.455"></a>
-<span class="sourceLineNo">456</span> }<a name="line.456"></a>
-<span class="sourceLineNo">457</span><a name="line.457"></a>
-<span class="sourceLineNo">458</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.458"></a>
-<span class="sourceLineNo">459</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.459"></a>
-<span class="sourceLineNo">460</span> }<a name="line.460"></a>
-<span class="sourceLineNo">461</span><a name="line.461"></a>
-<span class="sourceLineNo">462</span> // 4. Push the procedures to the timeout executor<a name="line.462"></a>
-<span class="sourceLineNo">463</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.463"></a>
-<span class="sourceLineNo">464</span> for (Procedure proc: waitingSet) {<a name="line.464"></a>
-<span class="sourceLineNo">465</span> proc.afterReplay(getEnvironment());<a name="line.465"></a>
-<span class="sourceLineNo">466</span> timeoutExecutor.add(proc);<a name="line.466"></a>
-<span class="sourceLineNo">467</span> }<a name="line.467"></a>
-<span class="sourceLineNo">468</span> }<a name="line.468"></a>
-<span class="sourceLineNo">469</span><a name="line.469"></a>
-<span class="sourceLineNo">470</span> // 5. Push the procedure to the scheduler<a name="line.470"></a>
-<span class="sourceLineNo">471</span> if (!runnableList.isEmpty()) {<a name="line.471"></a>
-<span class="sourceLineNo">472</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.472"></a>
-<span class="sourceLineNo">473</span> // some procedure may be started way before this stuff.<a name="line.473"></a>
-<span class="sourceLineNo">474</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.474"></a>
-<span class="sourceLineNo">475</span> Procedure proc = runnableList.get(i);<a name="line.475"></a>
-<span class="sourceLineNo">476</span> proc.afterReplay(getEnvironment());<a name="line.476"></a>
-<span class="sourceLineNo">477</span> if (!proc.hasParent()) {<a name="line.477"></a>
-<span class="sourceLineNo">478</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.478"></a>
-<span class="sourceLineNo">479</span> }<a name="line.479"></a>
-<span class="sourceLineNo">480</span> if (proc.wasExecuted()) {<a name="line.480"></a>
-<span class="sourceLineNo">481</span> scheduler.addFront(proc);<a name="line.481"></a>
-<span class="sourceLineNo">482</span> } else {<a name="line.482"></a>
-<span class="sourceLineNo">483</span> // if it was not in execution, it can wait.<a name="line.483"></a>
-<span class="sourceLineNo">484</span> scheduler.addBack(proc);<a name="line.484"></a>
-<span class="sourceLineNo">485</span> }<a name="line.485"></a>
-<span class="sourceLineNo">486</span> }<a name="line.486"></a>
-<span class="sourceLineNo">487</span> }<a name="line.487"></a>
-<span class="sourceLineNo">488</span> }<a name="line.488"></a>
-<span class="sourceLineNo">489</span><a name="line.489"></a>
-<span class="sourceLineNo">490</span> /**<a name="line.490"></a>
-<span class="sourceLineNo">491</span> * Start the procedure executor.<a name="line.491"></a>
-<span class="sourceLineNo">492</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.492"></a>
-<span class="sourceLineNo">493</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.493"></a>
-<span class="sourceLineNo">494</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.494"></a>
-<span class="sourceLineNo">495</span> *<a name="line.495"></a>
-<span class="sourceLineNo">496</span> * @param numThreads number of threads available for procedure execution.<a name="line.496"></a>
-<span class="sourceLineNo">497</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.497"></a>
-<span class="sourceLineNo">498</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.498"></a>
-<span class="sourceLineNo">499</span> */<a name="line.499"></a>
-<span class="sourceLineNo">500</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.500"></a>
-<span class="sourceLineNo">501</span> if (!running.compareAndSet(false, true)) {<a name="line.501"></a>
-<span class="sourceLineNo">502</span> LOG.warn("Already running");<a name="line.502"></a>
-<span class="sourceLineNo">503</span> return;<a name="line.503"></a>
-<span class="sourceLineNo">504</span> }<a name="line.504"></a>
-<span class="sourceLineNo">505</span><a name="line.505"></a>
-<span class="sourceLineNo">506</span> // We have numThreads executor + one timer thread used for timing out<a name="line.506"></a>
-<span class="sourceLineNo">507</span> // procedures and triggering periodic procedures.<a name="line.507"></a>
-<span class="sourceLineNo">508</span> this.corePoolSize = numThreads;<a name="line.508"></a>
-<span class="sourceLineNo">509</span> this.maxPoolSize = 10 * numThreads;<a name="line.509"></a>
-<span class="sourceLineNo">510</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.510"></a>
-<span class="sourceLineNo">511</span> corePoolSize, maxPoolSize);<a name="line.511"></a>
+<span class="sourceLineNo">392</span> // add the nonce to the map<a name="line.392"></a>
+<span class="sourceLineNo">393</span> if (nonceKey != null) {<a name="line.393"></a>
+<span class="sourceLineNo">394</span> nonceKeysToProcIdsMap.put(nonceKey, procId);<a name="line.394"></a>
+<span class="sourceLineNo">395</span> }<a name="line.395"></a>
+<span class="sourceLineNo">396</span> }<a name="line.396"></a>
+<span class="sourceLineNo">397</span><a name="line.397"></a>
+<span class="sourceLineNo">398</span> // 2. Initialize the stacks<a name="line.398"></a>
+<span class="sourceLineNo">399</span> final ArrayList<Procedure> runnableList = new ArrayList(runnablesCount);<a name="line.399"></a>
+<span class="sourceLineNo">400</span> HashSet<Procedure> waitingSet = null;<a name="line.400"></a>
+<span class="sourceLineNo">401</span> procIter.reset();<a name="line.401"></a>
+<span class="sourceLineNo">402</span> while (procIter.hasNext()) {<a name="line.402"></a>
+<span class="sourceLineNo">403</span> if (procIter.isNextFinished()) {<a name="line.403"></a>
+<span class="sourceLineNo">404</span> procIter.skipNext();<a name="line.404"></a>
+<span class="sourceLineNo">405</span> continue;<a name="line.405"></a>
+<span class="sourceLineNo">406</span> }<a name="line.406"></a>
+<span class="sourceLineNo">407</span><a name="line.407"></a>
+<span class="sourceLineNo">408</span> Procedure proc = procIter.next();<a name="line.408"></a>
+<span class="sourceLineNo">409</span> assert !(proc.isFinished() && !proc.hasParent()) : "unexpected completed proc=" + proc;<a name="line.409"></a>
+<span class="sourceLineNo">410</span><a name="line.410"></a>
+<span class="sourceLineNo">411</span> if (debugEnabled) {<a name="line.411"></a>
+<span class="sourceLineNo">412</span> LOG.debug(String.format("Loading %s", proc));<a name="line.412"></a>
+<span class="sourceLineNo">413</span> }<a name="line.413"></a>
+<span class="sourceLineNo">414</span><a name="line.414"></a>
+<span class="sourceLineNo">415</span> Long rootProcId = getRootProcedureId(proc);<a name="line.415"></a>
+<span class="sourceLineNo">416</span> if (rootProcId == null) {<a name="line.416"></a>
+<span class="sourceLineNo">417</span> // The 'proc' was ready to run but the root procedure was rolledback?<a name="line.417"></a>
+<span class="sourceLineNo">418</span> scheduler.addBack(proc);<a name="line.418"></a>
+<span class="sourceLineNo">419</span> continue;<a name="line.419"></a>
+<span class="sourceLineNo">420</span> }<a name="line.420"></a>
+<span class="sourceLineNo">421</span><a name="line.421"></a>
+<span class="sourceLineNo">422</span> if (proc.hasParent()) {<a name="line.422"></a>
+<span class="sourceLineNo">423</span> Procedure parent = procedures.get(proc.getParentProcId());<a name="line.423"></a>
+<span class="sourceLineNo">424</span> // corrupted procedures are handled later at step 3<a name="line.424"></a>
+<span class="sourceLineNo">425</span> if (parent != null && !proc.isFinished()) {<a name="line.425"></a>
+<span class="sourceLineNo">426</span> parent.incChildrenLatch();<a name="line.426"></a>
+<span class="sourceLineNo">427</span> }<a name="line.427"></a>
+<span class="sourceLineNo">428</span> }<a name="line.428"></a>
+<span class="sourceLineNo">429</span><a name="line.429"></a>
+<span class="sourceLineNo">430</span> RootProcedureState procStack = rollbackStack.get(rootProcId);<a name="line.430"></a>
+<span class="sourceLineNo">431</span> procStack.loadStack(proc);<a name="line.431"></a>
+<span class="sourceLineNo">432</span><a name="line.432"></a>
+<span class="sourceLineNo">433</span> proc.setRootProcId(rootProcId);<a name="line.433"></a>
+<span class="sourceLineNo">434</span> switch (proc.getState()) {<a name="line.434"></a>
+<span class="sourceLineNo">435</span> case RUNNABLE:<a name="line.435"></a>
+<span class="sourceLineNo">436</span> runnableList.add(proc);<a name="line.436"></a>
+<span class="sourceLineNo">437</span> break;<a name="line.437"></a>
+<span class="sourceLineNo">438</span> case WAITING:<a name="line.438"></a>
+<span class="sourceLineNo">439</span> if (!proc.hasChildren()) {<a name="line.439"></a>
+<span class="sourceLineNo">440</span> runnableList.add(proc);<a name="line.440"></a>
+<span class="sourceLineNo">441</span> }<a name="line.441"></a>
+<span class="sourceLineNo">442</span> break;<a name="line.442"></a>
+<span class="sourceLineNo">443</span> case WAITING_TIMEOUT:<a name="line.443"></a>
+<span class="sourceLineNo">444</span> if (waitingSet == null) {<a name="line.444"></a>
+<span class="sourceLineNo">445</span> waitingSet = new HashSet<>();<a name="line.445"></a>
+<span class="sourceLineNo">446</span> }<a name="line.446"></a>
+<span class="sourceLineNo">447</span> waitingSet.add(proc);<a name="line.447"></a>
+<span class="sourceLineNo">448</span> break;<a name="line.448"></a>
+<span class="sourceLineNo">449</span> case FAILED:<a name="line.449"></a>
+<span class="sourceLineNo">450</span> // add the proc to the scheduler to perform the rollback<a name="line.450"></a>
+<span class="sourceLineNo">451</span> scheduler.addBack(proc);<a name="line.451"></a>
+<span class="sourceLineNo">452</span> break;<a name="line.452"></a>
+<span class="sourceLineNo">453</span> case ROLLEDBACK:<a name="line.453"></a>
+<span class="sourceLineNo">454</span> case INITIALIZING:<a name="line.454"></a>
+<span class="sourceLineNo">455</span> String msg = "Unexpected " + proc.getState() + " state for " + proc;<a name="line.455"></a>
+<span class="sourceLineNo">456</span> LOG.error(msg);<a name="line.456"></a>
+<span class="sourceLineNo">457</span> throw new UnsupportedOperationException(msg);<a name="line.457"></a>
+<span class="sourceLineNo">458</span> default:<a name="line.458"></a>
+<span class="sourceLineNo">459</span> break;<a name="line.459"></a>
+<span class="sourceLineNo">460</span> }<a name="line.460"></a>
+<span class="sourceLineNo">461</span> }<a name="line.461"></a>
+<span class="sourceLineNo">462</span><a name="line.462"></a>
+<span class="sourceLineNo">463</span> // 3. Validate the stacks<a name="line.463"></a>
+<span class="sourceLineNo">464</span> int corruptedCount = 0;<a name="line.464"></a>
+<span class="sourceLineNo">465</span> Iterator<Map.Entry<Long, RootProcedureState>> itStack = rollbackStack.entrySet().iterator();<a name="line.465"></a>
+<span class="sourceLineNo">466</span> while (itStack.hasNext()) {<a name="line.466"></a>
+<span class="sourceLineNo">467</span> Map.Entry<Long, RootProcedureState> entry = itStack.next();<a name="line.467"></a>
+<span class="sourceLineNo">468</span> RootProcedureState procStack = entry.getValue();<a name="line.468"></a>
+<span class="sourceLineNo">469</span> if (procStack.isValid()) continue;<a name="line.469"></a>
+<span class="sourceLineNo">470</span><a name="line.470"></a>
+<span class="sourceLineNo">471</span> for (Procedure proc: procStack.getSubproceduresStack()) {<a name="line.471"></a>
+<span class="sourceLineNo">472</span> LOG.error("Corrupted " + proc);<a name="line.472"></a>
+<span class="sourceLineNo">473</span> procedures.remove(proc.getProcId());<a name="line.473"></a>
+<span class="sourceLineNo">474</span> runnableList.remove(proc);<a name="line.474"></a>
+<span class="sourceLineNo">475</span> if (waitingSet != null) waitingSet.remove(proc);<a name="line.475"></a>
+<span class="sourceLineNo">476</span> corruptedCount++;<a name="line.476"></a>
+<span class="sourceLineNo">477</span> }<a name="line.477"></a>
+<span class="sourceLineNo">478</span> itStack.remove();<a name="line.478"></a>
+<span class="sourceLineNo">479</span> }<a name="line.479"></a>
+<span class="sourceLineNo">480</span><a name="line.480"></a>
+<span class="sourceLineNo">481</span> if (abortOnCorruption && corruptedCount > 0) {<a name="line.481"></a>
+<span class="sourceLineNo">482</span> throw new IOException("found " + corruptedCount + " procedures on replay");<a name="line.482"></a>
+<span class="sourceLineNo">483</span> }<a name="line.483"></a>
+<span class="sourceLineNo">484</span><a name="line.484"></a>
+<span class="sourceLineNo">485</span> // 4. Push the procedures to the timeout executor<a name="line.485"></a>
+<span class="sourceLineNo">486</span> if (waitingSet != null && !waitingSet.isEmpty()) {<a name="line.486"></a>
+<span class="sourceLineNo">487</span> for (Procedure proc: waitingSet) {<a name="line.487"></a>
+<span class="sourceLineNo">488</span> proc.afterReplay(getEnvironment());<a name="line.488"></a>
+<span class="sourceLineNo">489</span> timeoutExecutor.add(proc);<a name="line.489"></a>
+<span class="sourceLineNo">490</span> }<a name="line.490"></a>
+<span class="sourceLineNo">491</span> }<a name="line.491"></a>
+<span class="sourceLineNo">492</span><a name="line.492"></a>
+<span class="sourceLineNo">493</span> // 5. Push the procedure to the scheduler<a name="line.493"></a>
+<span class="sourceLineNo">494</span> if (!runnableList.isEmpty()) {<a name="line.494"></a>
+<span class="sourceLineNo">495</span> // TODO: See ProcedureWALFormatReader#hasFastStartSupport<a name="line.495"></a>
+<span class="sourceLineNo">496</span> // some procedure may be started way before this stuff.<a name="line.496"></a>
+<span class="sourceLineNo">497</span> for (int i = runnableList.size() - 1; i >= 0; --i) {<a name="line.497"></a>
+<span class="sourceLineNo">498</span> Procedure proc = runnableList.get(i);<a name="line.498"></a>
+<span class="sourceLineNo">499</span> proc.afterReplay(getEnvironment());<a name="line.499"></a>
+<span class="sourceLineNo">500</span> if (!proc.hasParent()) {<a name="line.500"></a>
+<span class="sourceLineNo">501</span> sendProcedureLoadedNotification(proc.getProcId());<a name="line.501"></a>
+<span class="sourceLineNo">502</span> }<a name="line.502"></a>
+<span class="sourceLineNo">503</span> if (proc.wasExecuted()) {<a name="line.503"></a>
+<span class="sourceLineNo">504</span> scheduler.addFront(proc);<a name="line.504"></a>
+<span class="sourceLineNo">505</span> } else {<a name="line.505"></a>
+<span class="sourceLineNo">506</span> // if it was not in execution, it can wait.<a name="line.506"></a>
+<span class="sourceLineNo">507</span> scheduler.addBack(proc);<a name="line.507"></a>
+<span class="sourceLineNo">508</span> }<a name="line.508"></a>
+<span class="sourceLineNo">509</span> }<a name="line.509"></a>
+<span class="sourceLineNo">510</span> }<a name="line.510"></a>
+<span class="sourceLineNo">511</span> }<a name="line.511"></a>
<span class="sourceLineNo">512</span><a name="line.512"></a>
-<span class="sourceLineNo">513</span> // Create the Thread Group for the executors<a name="line.513"></a>
-<span class="sourceLineNo">514</span> threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.514"></a>
-<span class="sourceLineNo">515</span><a name="line.515"></a>
-<span class="sourceLineNo">516</span> // Create the timeout executor<a name="line.516"></a>
-<span class="sourceLineNo">517</span> timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.517"></a>
-<span class="sourceLineNo">518</span><a name="line.518"></a>
-<span class="sourceLineNo">519</span> // Create the workers<a name="line.519"></a>
-<span class="sourceLineNo">520</span> workerId.set(0);<a name="line.520"></a>
-<span class="sourceLineNo">521</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.521"></a>
-<span class="sourceLineNo">522</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.522"></a>
-<span class="sourceLineNo">523</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.523"></a>
-<span class="sourceLineNo">524</span> }<a name="line.524"></a>
-<span class="sourceLineNo">525</span><a name="line.525"></a>
-<span class="sourceLineNo">526</span> long st, et;<a name="line.526"></a>
-<span class="sourceLineNo">527</span><a name="line.527"></a>
-<span class="sourceLineNo">528</span> // Acquire the store lease.<a name="line.528"></a>
-<span class="sourceLineNo">529</span> st = EnvironmentEdgeManager.currentTime();<a name="line.529"></a>
-<span class="sourceLineNo">530</span> store.recoverLease();<a name="line.530"></a>
-<span class="sourceLineNo">531</span> et = EnvironmentEdgeManager.currentTime();<a name="line.531"></a>
-<span class="sourceLineNo">532</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.532"></a>
-<span class="sourceLineNo">533</span> StringUtils.humanTimeDiff(et - st));<a name="line.533"></a>
-<span class="sourceLineNo">534</span><a name="line.534"></a>
-<span class="sourceLineNo">535</span> // start the procedure scheduler<a name="line.535"></a>
-<span class="sourceLineNo">536</span> scheduler.start();<a name="line.536"></a>
-<span class="sourceLineNo">537</span><a name="line.537"></a>
-<span class="sourceLineNo">538</span> // TODO: Split in two steps.<a name="line.538"></a>
-<span class="sourceLineNo">539</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.539"></a>
-<span class="sourceLineNo">540</span> // The first one will make sure that we have the latest id,<a name="line.540"></a>
-<span class="sourceLineNo">541</span> // so we can start the threads and accept new procedures.<a name="line.541"></a>
-<span class="sourceLineNo">542</span> // The second step will do the actual load of old procedures.<a name="line.542"></a>
-<span class="sourceLineNo">543</span> st = EnvironmentEdgeManager.currentTime();<a name="line.543"></a>
-<span class="sourceLineNo">544</span> load(abortOnCorruption);<a name="line.544"></a>
-<span class="sourceLineNo">545</span> et = EnvironmentEdgeManager.currentTime();<a name="line.545"></a>
-<span class="sourceLineNo">546</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.546"></a>
-<span class="sourceLineNo">547</span> StringUtils.humanTimeDiff(et - st));<a name="line.547"></a>
-<span class="sourceLineNo">548</span><a name="line.548"></a>
-<span class="sourceLineNo">549</span> // Start the executors. Here we must have the lastProcId set.<a name="line.549"></a>
-<span class="sourceLineNo">550</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.550"></a>
-<span class="sourceLineNo">551</span> timeoutExecutor.start();<a name="line.551"></a>
-<span class="sourceLineNo">552</span> for (WorkerThread worker: workerThreads) {<a name="line.552"></a>
-<span class="sourceLineNo">553</span> worker.start();<a name="line.553"></a>
-<span class="sourceLineNo">554</span> }<a name="line.554"></a>
-<span class="sourceLineNo">555</span><a name="line.555"></a>
-<span class="sourceLineNo">556</span> // Internal chores<a name="line.556"></a>
-<span class="sourceLineNo">557</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.557"></a>
-<span class="sourceLineNo">558</span><a name="line.558"></a>
-<span class="sourceLineNo">559</span> // Add completed cleaner chore<a name="line.559"></a>
-<span class="sourceLineNo">560</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.560"></a>
-<span class="sourceLineNo">561</span> }<a name="line.561"></a>
-<span class="sourceLineNo">562</span><a name="line.562"></a>
-<span class="sourceLineNo">563</span> public void stop() {<a name="line.563"></a>
-<span class="sourceLineNo">564</span> if (!running.getAndSet(false)) {<a name="line.564"></a>
-<span class="sourceLineNo">565</span> return;<a name="line.565"></a>
-<span class="sourceLineNo">566</span> }<a name="line.566"></a>
-<span class="sourceLineNo">567</span><a name="line.567"></a>
-<span class="sourceLineNo">568</span> LOG.info("Stopping");<a name="line.568"></a>
-<span class="sourceLineNo">569</span> scheduler.stop();<a name="line.569"></a>
-<span class="sourceLineNo">570</span> timeoutExecutor.sendStopSignal();<a name="line.570"></a>
-<span class="sourceLineNo">571</span> }<a name="line.571"></a>
-<span class="sourceLineNo">572</span><a name="line.572"></a>
-<span class="sourceLineNo">573</span> @VisibleForTesting<a name="line.573"></a>
-<span class="sourceLineNo">574</span> public void join() {<a name="line.574"></a>
-<span class="sourceLineNo">575</span> assert !isRunning() : "expected not running";<a name="line.575"></a>
-<span class="sourceLineNo">576</span><a name="line.576"></a>
-<span class="sourceLineNo">577</span> // stop the timeout executor<a name="line.577"></a>
-<span class="sourceLineNo">578</span> timeoutExecutor.awaitTermination();<a name="line.578"></a>
-<span class="sourceLineNo">579</span> timeoutExecutor = null;<a name="line.579"></a>
-<span class="sourceLineNo">580</span><a name="line.580"></a>
-<span class="sourceLineNo">581</span> // stop the worker threads<a name="line.581"></a>
-<span class="sourceLineNo">582</span> for (WorkerThread worker: workerThreads) {<a name="line.582"></a>
-<span class="sourceLineNo">583</span> worker.awaitTermination();<a name="line.583"></a>
-<span class="sourceLineNo">584</span> }<a name="line.584"></a>
-<span class="sourceLineNo">585</span> workerThreads = null;<a name="line.585"></a>
-<span class="sourceLineNo">586</span><a name="line.586"></a>
-<span class="sourceLineNo">587</span> // Destroy the Thread Group for the executors<a name="line.587"></a>
-<span class="sourceLineNo">588</span> try {<a name="line.588"></a>
-<span class="sourceLineNo">589</span> threadGroup.destroy();<a name="line.589"></a>
-<span class="sourceLineNo">590</span> } catch (IllegalThreadStateException e) {<a name="line.590"></a>
-<span class="sourceLineNo">591</span> LOG.error("ThreadGroup " + threadGroup + " contains running threads; " + e.getMessage());<a name="line.591"></a>
-<span class="sourceLineNo">592</span> threadGroup.list();<a name="line.592"></a>
-<span class="sourceLineNo">593</span> } finally {<a name="line.593"></a>
-<span class="sourceLineNo">594</span> threadGroup = null;<a name="line.594"></a>
-<span class="sourceLineNo">595</span> }<a name="line.595"></a>
+<span class="sourceLineNo">513</span> /**<a name="line.513"></a>
+<span class="sourceLineNo">514</span> * Start the procedure executor.<a name="line.514"></a>
+<span class="sourceLineNo">515</span> * It calls ProcedureStore.recoverLease() and ProcedureStore.load() to<a name="line.515"></a>
+<span class="sourceLineNo">516</span> * recover the lease, and ensure a single executor, and start the procedure<a name="line.516"></a>
+<span class="sourceLineNo">517</span> * replay to resume and recover the previous pending and in-progress perocedures.<a name="line.517"></a>
+<span class="sourceLineNo">518</span> *<a name="line.518"></a>
+<span class="sourceLineNo">519</span> * @param numThreads number of threads available for procedure execution.<a name="line.519"></a>
+<span class="sourceLineNo">520</span> * @param abortOnCorruption true if you want to abort your service in case<a name="line.520"></a>
+<span class="sourceLineNo">521</span> * a corrupted procedure is found on replay. otherwise false.<a name="line.521"></a>
+<span class="sourceLineNo">522</span> */<a name="line.522"></a>
+<span class="sourceLineNo">523</span> public void start(int numThreads, boolean abortOnCorruption) throws IOException {<a name="line.523"></a>
+<span class="sourceLineNo">524</span> if (!running.compareAndSet(false, true)) {<a name="line.524"></a>
+<span class="sourceLineNo">525</span> LOG.warn("Already running");<a name="line.525"></a>
+<span class="sourceLineNo">526</span> return;<a name="line.526"></a>
+<span class="sourceLineNo">527</span> }<a name="line.527"></a>
+<span class="sourceLineNo">528</span><a name="line.528"></a>
+<span class="sourceLineNo">529</span> // We have numThreads executor + one timer thread used for timing out<a name="line.529"></a>
+<span class="sourceLineNo">530</span> // procedures and triggering periodic procedures.<a name="line.530"></a>
+<span class="sourceLineNo">531</span> this.corePoolSize = numThreads;<a name="line.531"></a>
+<span class="sourceLineNo">532</span> this.maxPoolSize = 10 * numThreads;<a name="line.532"></a>
+<span class="sourceLineNo">533</span> LOG.info("Starting {} core workers (bigger of cpus/4 or 16) with max (burst) worker count={}",<a name="line.533"></a>
+<span class="sourceLineNo">534</span> corePoolSize, maxPoolSize);<a name="line.534"></a>
+<span class="sourceLineNo">535</span><a name="line.535"></a>
+<span class="sourceLineNo">536</span> this.threadGroup = new ThreadGroup("PEWorkerGroup");<a name="line.536"></a>
+<span class="sourceLineNo">537</span> this.timeoutExecutor = new TimeoutExecutorThread(this, threadGroup);<a name="line.537"></a>
+<span class="sourceLineNo">538</span><a name="line.538"></a>
+<span class="sourceLineNo">539</span> // Create the workers<a name="line.539"></a>
+<span class="sourceLineNo">540</span> workerId.set(0);<a name="line.540"></a>
+<span class="sourceLineNo">541</span> workerThreads = new CopyOnWriteArrayList<>();<a name="line.541"></a>
+<span class="sourceLineNo">542</span> for (int i = 0; i < corePoolSize; ++i) {<a name="line.542"></a>
+<span class="sourceLineNo">543</span> workerThreads.add(new WorkerThread(threadGroup));<a name="line.543"></a>
+<span class="sourceLineNo">544</span> }<a name="line.544"></a>
+<span class="sourceLineNo">545</span><a name="line.545"></a>
+<span class="sourceLineNo">546</span> long st, et;<a name="line.546"></a>
+<span class="sourceLineNo">547</span><a name="line.547"></a>
+<span class="sourceLineNo">548</span> // Acquire the store lease.<a name="line.548"></a>
+<span class="sourceLineNo">549</span> st = EnvironmentEdgeManager.currentTime();<a name="line.549"></a>
+<span class="sourceLineNo">550</span> store.recoverLease();<a name="line.550"></a>
+<span class="sourceLineNo">551</span> et = EnvironmentEdgeManager.currentTime();<a name="line.551"></a>
+<span class="sourceLineNo">552</span> LOG.info("Recovered {} lease in {}", store.getClass().getSimpleName(),<a name="line.552"></a>
+<span class="sourceLineNo">553</span> StringUtils.humanTimeDiff(et - st));<a name="line.553"></a>
+<span class="sourceLineNo">554</span><a name="line.554"></a>
+<span class="sourceLineNo">555</span> // start the procedure scheduler<a name="line.555"></a>
+<span class="sourceLineNo">556</span> scheduler.start();<a name="line.556"></a>
+<span class="sourceLineNo">557</span><a name="line.557"></a>
+<span class="sourceLineNo">558</span> // TODO: Split in two steps.<a name="line.558"></a>
+<span class="sourceLineNo">559</span> // TODO: Handle corrupted procedures (currently just a warn)<a name="line.559"></a>
+<span class="sourceLineNo">560</span> // The first one will make sure that we have the latest id,<a name="line.560"></a>
+<span class="sourceLineNo">561</span> // so we can start the threads and accept new procedures.<a name="line.561"></a>
+<span class="sourceLineNo">562</span> // The second step will do the actual load of old procedures.<a name="line.562"></a>
+<span class="sourceLineNo">563</span> st = EnvironmentEdgeManager.currentTime();<a name="line.563"></a>
+<span class="sourceLineNo">564</span> load(abortOnCorruption);<a name="line.564"></a>
+<span class="sourceLineNo">565</span> et = EnvironmentEdgeManager.currentTime();<a name="line.565"></a>
+<span class="sourceLineNo">566</span> LOG.info("Loaded {} in {}", store.getClass().getSimpleName(),<a name="line.566"></a>
+<span class="sourceLineNo">567</span> StringUtils.humanTimeDiff(et - st));<a name="line.567"></a>
+<span class="sourceLineNo">568</span><a name="line.568"></a>
+<span class="sourceLineNo">569</span> // Start the executors. Here we must have the lastProcId set.<a name="line.569"></a>
+<span class="sourceLineNo">570</span> LOG.trace("Start workers {}", workerThreads.size());<a name="line.570"></a>
+<span class="sourceLineNo">571</span> timeoutExecutor.start();<a name="line.571"></a>
+<span class="sourceLineNo">572</span> for (WorkerThread worker: workerThreads) {<a name="line.572"></a>
+<span class="sourceLineNo">573</span> worker.start();<a name="line.573"></a>
+<span class="sourceLineNo">574</span> }<a name="line.574"></a>
+<span class="sourceLineNo">575</span><a name="line.575"></a>
+<span class="sourceLineNo">576</span> // Internal chores<a name="line.576"></a>
+<span class="sourceLineNo">577</span> timeoutExecutor.add(new WorkerMonitor());<a name="line.577"></a>
+<span class="sourceLineNo">578</span><a name="line.578"></a>
+<span class="sourceLineNo">579</span> // Add completed cleaner chore<a name="line.579"></a>
+<span class="sourceLineNo">580</span> addChore(new CompletedProcedureCleaner(conf, store, completed, nonceKeysToProcIdsMap));<a name="line.580"></a>
+<span class="sourceLineNo">581</span> }<a name="line.581"></a>
+<span class="sourceLineNo">582</span><a name="line.582"></a>
+<span class="sourceLineNo">583</span> public void stop() {<a name="line.583"></a>
+<span class="sourceLineNo">584</span> if (!running.getAndSet(false)) {<a name="line.584"></a>
+<span class="sourceLineNo">585</span> return;<a name="line.585"></a>
+<span class="sourceLineNo">586</span> }<a name="line.586"></a>
+<span class="sourceLineNo">587</span><a name="line.587"></a>
+<span class="sourceLineNo">588</span> LOG.info("Stopping");<a name="line.588"></a>
+<span class="sourceLineNo">589</span> scheduler.stop();<a name="line.589"></a>
+<span class="sourceLineNo">590</span> timeoutExecutor.sendStopSignal();<a name="line.590"></a>
+<span class="sourceLineNo">591</span> }<a name="line.591"></a>
+<span class="sourceLineNo">592</span><a name="line.592"></a>
+<span class="sourceLineNo">593</span> @VisibleForTesting<a name="line.593"></a>
+<span class="sourceLineNo">594</span> public void join() {<a name="line.594"></a>
+<span class="sourceLineNo">595</span> assert !isRunning() : "expected not running";<a name="line.595"></a>
<span class="sourceLineNo">596</span><a name="line.596"></a>
-<span class="sourceLineNo">597</span> // reset the in-memory state for testing<a name="line.597"></a>
-<span class="sourceLineNo">598</span> completed.clear();<a name="line.598"></a>
-<span class="sourceLineNo">599</span> rollbackStack.clear();<a name="line.599"></a>
-<span class="sourceLineNo">600</span> procedures.clear();<a name="line.600"></a>
-<span class="sourceLineNo">601</span> nonceKeysToProcIdsMap.clear();<a name="line.601"></a>
-<span class="sourceLineNo">602</span> scheduler.clear();<a name="line.602"></a>
-<span class="sourceLineNo">603</span> lastProcId.set(-1);<a name="line.603"></a>
-<span class="sourceLineNo">604</span> }<a name="line.604"></a>
-<span class="sourceLineNo">605</span><a name="line.605"></a>
-<span class="sourceLineNo">606</span> public void refreshConfiguration(final Configuration conf) {<a name="line.606"></a>
-<span class="sourceLineNo">607</span> this.conf = conf;<a name="line.607"></a>
-<span class="sourceLineNo">608</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.608"></a>
-<span class="sourceLineNo">609</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.609"></a>
-<span class="sourceLineNo">610</span> }<a name="line.610"></a>
-<span class="sourceLineNo">611</span><a name="line.611"></a>
-<span class="sourceLineNo">612</span> // ==========================================================================<a name="line.612"></a>
-<span class="sourceLineNo">613</span> // Accessors<a name="line.613"></a>
-<span class="sourceLineNo">614</span> // ==========================================================================<a name="line.614"></a>
-<span class="sourceLineNo">615</span> public boolean isRunning() {<a name="line.615"></a>
-<span class="sourceLineNo">616</span> return running.get();<a name="line.616"></a>
-<span class="sourceLineNo">617</span> }<a name="line.617"></a>
-<span class="sourceLineNo">618</span><a name="line.618"></a>
-<span class="sourceLineNo">619</span> /**<a name="line.619"></a>
-<span class="sourceLineNo">620</span> * @return the current number of worker threads.<a name="line.620"></a>
-<span class="sourceLineNo">621</span> */<a name="line.621"></a>
-<span class="sourceLineNo">622</span> public int getWorkerThreadCount() {<a name="line.622"></a>
-<span class="sourceLineNo">623</span> return workerThreads.size();<a name="line.623"></a>
-<span class="sourceLineNo">624</span> }<a name="line.624"></a>
-<span class="sourceLineNo">625</span><a name="line.625"></a>
-<span class="sourceLineNo">626</span> /**<a name="line.626"></a>
-<span class="sourceLineNo">627</span> * @return the core pool size settings.<a name="line.627"></a>
-<span class="sourceLineNo">628</span> */<a name="line.628"></a>
-<span class="sourceLineNo">629</span> public int getCorePoolSize() {<a name="line.629"></a>
-<span class="sourceLineNo">630</span> return corePoolSize;<a name="line.630"></a>
-<span class="sourceLineNo">631</span> }<a name="line.631"></a>
-<span class="sourceLineNo">632</span><a name="line.632"></a>
-<span class="sourceLineNo">633</span> public int getActiveExecutorCount() {<a name="line.633"></a>
-<span class="sourceLineNo">634</span> return activeExecutorCount.get();<a name="line.634"></a>
-<span class="sourceLineNo">635</span> }<a name="line.635"></a>
-<span class="sourceLineNo">636</span><a name="line.636"></a>
-<span class="sourceLineNo">637</span> public TEnvironment getEnvironment() {<a name="line.637"></a>
-<span class="sourceLineNo">638</span> return this.environment;<a name="line.638"></a>
-<span class="sourceLineNo">639</span> }<a name="line.639"></a>
-<span class="sourceLineNo">640</span><a name="line.640"></a>
-<span class="sourceLineNo">641</span> public ProcedureStore getStore() {<a name="line.641"></a>
-<span class="sourceLineNo">642</span> return this.store;<a name="line.642"></a>
+<span class="sourceLineNo">597</span> // stop the timeout executor<a name="line.597"></a>
+<span class="sourceLineNo">598</span> timeoutExecutor.awaitTermination();<a name="line.598"></a>
+<span class="sourceLineNo">599</span><a name="line.599"></a>
+<span class="sourceLineNo">600</span> // stop the worker threads<a name="line.600"></a>
+<span class="sourceLineNo">601</span> for (WorkerThread worker: workerThreads) {<a name="line.601"></a>
+<span class="sourceLineNo">602</span> worker.awaitTermination();<a name="line.602"></a>
+<span class="sourceLineNo">603</span> }<a name="line.603"></a>
+<span class="sourceLineNo">604</span><a name="line.604"></a>
+<span class="sourceLineNo">605</span> // Destroy the Thread Group for the executors<a name="line.605"></a>
+<span class="sourceLineNo">606</span> // TODO: Fix. #join is not place to destroy resources.<a name="line.606"></a>
+<span class="sourceLineNo">607</span> try {<a name="line.607"></a>
+<span class="sourceLineNo">608</span> threadGroup.destroy();<a name="line.608"></a>
+<span class="sourceLineNo">609</span> } catch (IllegalThreadStateException e) {<a name="line.609"></a>
+<span class="sourceLineNo">610</span> LOG.error("ThreadGroup {} contains running threads; {}: See STDOUT",<a name="line.610"></a>
+<span class="sourceLineNo">611</span> this.threadGroup, e.getMessage());<a name="line.611"></a>
+<span class="sourceLineNo">612</span> // This dumps list of threads on STDOUT.<a name="line.612"></a>
+<span class="sourceLineNo">613</span> this.threadGroup.list();<a name="line.613"></a>
+<span class="sourceLineNo">614</span> }<a name="line.614"></a>
+<span class="sourceLineNo">615</span><a name="line.615"></a>
+<span class="sourceLineNo">616</span> // reset the in-memory state for testing<a name="line.616"></a>
+<span class="sourceLineNo">617</span> completed.clear();<a name="line.617"></a>
+<span class="sourceLineNo">618</span> rollbackStack.clear();<a name="line.618"></a>
+<span class="sourceLineNo">619</span> procedures.clear();<a name="line.619"></a>
+<span class="sourceLineNo">620</span> nonceKeysToProcIdsMap.clear();<a name="line.620"></a>
+<span class="sourceLineNo">621</span> scheduler.clear();<a name="line.621"></a>
+<span class="sourceLineNo">622</span> lastProcId.set(-1);<a name="line.622"></a>
+<span class="sourceLineNo">623</span> }<a name="line.623"></a>
+<span class="sourceLineNo">624</span><a name="line.624"></a>
+<span class="sourceLineNo">625</span> public void refreshConfiguration(final Configuration conf) {<a name="line.625"></a>
+<span class="sourceLineNo">626</span> this.conf = conf;<a name="line.626"></a>
+<span class="sourceLineNo">627</span> setKeepAliveTime(conf.getLong(WORKER_KEEP_ALIVE_TIME_CONF_KEY,<a name="line.627"></a>
+<span class="sourceLineNo">628</span> DEFAULT_WORKER_KEEP_ALIVE_TIME), TimeUnit.MILLISECONDS);<a name="line.628"></a>
+<span class="sourceLineNo">629</span> }<a name="line.629"></a>
+<span class="sourceLineNo">630</span><a name="line.630"></a>
+<span class="sourceLineNo">631</span> // ==========================================================================<a name="line.631"></a>
+<span class="sourceLineNo">632</span> // Accessors<a name="line.632"></a>
+<span class="sourceLineNo">633</span> // ==========================================================================<a name="line.633"></a>
+<span class="sourceLineNo">634</span> public boolean isRunning() {<a name="line.634"></a>
+<span class="sourceLineNo">635</span> return running.get();<a name="line.635"></a>
+<span class="sourceLineNo">636</span> }<a name="line.636"></a>
+<span class="sourceLineNo">637</span><a name="line.637"></a>
+<span class="sourceLineNo">638</span> /**<a name="line.638"></a>
+<span class="sourceLineNo">639</span> * @return the current number of worker threads.<a name="line.639"></a>
+<span class="sourceLineNo">640</span> */<a name="line.640"></a>
+<span class="sourceLineNo">641</span> public int getWorkerThreadCount() {<a name="line.641"></a>
+<span class="sourceLineNo">642</span> return workerThreads.size();<a name="line.642"></a>
<span class="sourceLineNo">643</span> }<a name="line.643"></a>
<span class="sourceLineNo">644</span><a name="line.644"></a>
-<span class="sourceLineNo">645</span> ProcedureScheduler getScheduler() {<a name="line.645"></a>
-<span class="sourceLineNo">646</span> return scheduler;<a name="line.646"></a>
-<span class="sourceLineNo">647</span> }<a name="line.647"></a>
-<span class="sourceLineNo">648</span><a name="line.648"></a>
-<span class="sourceLineNo">649</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.649"></a>
-<span class="sourceLineNo">650</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.650"></a>
-<span class="sourceLineNo">651</span> this.scheduler.signalAll();<a name="line.651"></a>
-<span class="sourceLineNo">652</span> }<a name="line.652"></a>
-<span class="sourceLineNo">653</span><a name="line.653"></a>
-<span class="sourceLineNo">654</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.654"></a>
-<span class="sourceLineNo">655</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.655"></a>
-<span class="sourceLineNo">656</span> }<a name="line.656"></a>
-<span class="sourceLineNo">657</span><a name="line.657"></a>
-<span class="sourceLineNo">658</span> // ==========================================================================<a name="line.658"></a>
-<span class="sourceLineNo">659</span> // Submit/Remove Chores<a name="line.659"></a>
-<span class="sourceLineNo">660</span> // ==========================================================================<a name="line.660"></a>
-<span class="sourceLineNo">661</span><a name="line.661"></a>
-<span class="sourceLineNo">662</span> /**<a name="line.662"></a>
-<span class="sourceLineNo">663</span> * Add a chore procedure to the executor<a name="line.663"></a>
-<span class="sourceLineNo">664</span> * @param chore the chore to add<a name="line.664"></a>
-<span class="sourceLineNo">665</span> */<a name="line.665"></a>
-<span class="sourceLineNo">666</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.666"></a>
-<span class="sourceLineNo">667</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.667"></a>
-<span class="sourceLineNo">668</span> timeoutExecutor.add(chore);<a name="line.668"></a>
-<span class="sourceLineNo">669</span> }<a name="line.669"></a>
-<span class="sourceLineNo">670</span><a name="line.670"></a>
-<span class="sourceLineNo">671</span> /**<a name="line.671"></a>
-<span class="sourceLineNo">672</span> * Remove a chore procedure from the executor<a name="line.672"></a>
-<span class="sourceLineNo">673</span> * @param chore the chore to remove<a name="line.673"></a>
-<span class="sourceLineNo">674</span> * @return whether the chore is removed, or it will be removed later<a name="line.674"></a>
-<span class="sourceLineNo">675</span> */<a name="line.675"></a>
-<span class="sourceLineNo">676</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.676"></a>
-<span class="sourceLineNo">677</span> chore.setState(ProcedureState.SUCCESS);<a name="line.677"></a>
-<span class="sourceLineNo">678</span> return timeoutExecutor.remove(chore);<a name="line.678"></a>
-<span class="sourceLineNo">679</span> }<a name="line.679"></a>
+<span class="sourceLineNo">645</span> /**<a name="line.645"></a>
+<span class="sourceLineNo">646</span> * @return the core pool size settings.<a name="line.646"></a>
+<span class="sourceLineNo">647</span> */<a name="line.647"></a>
+<span class="sourceLineNo">648</span> public int getCorePoolSize() {<a name="line.648"></a>
+<span class="sourceLineNo">649</span> return corePoolSize;<a name="line.649"></a>
+<span class="sourceLineNo">650</span> }<a name="line.650"></a>
+<span class="sourceLineNo">651</span><a name="line.651"></a>
+<span class="sourceLineNo">652</span> public int getActiveExecutorCount() {<a name="line.652"></a>
+<span class="sourceLineNo">653</span> return activeExecutorCount.get();<a name="line.653"></a>
+<span class="sourceLineNo">654</span> }<a name="line.654"></a>
+<span class="sourceLineNo">655</span><a name="line.655"></a>
+<span class="sourceLineNo">656</span> public TEnvironment getEnvironment() {<a name="line.656"></a>
+<span class="sourceLineNo">657</span> return this.environment;<a name="line.657"></a>
+<span class="sourceLineNo">658</span> }<a name="line.658"></a>
+<span class="sourceLineNo">659</span><a name="line.659"></a>
+<span class="sourceLineNo">660</span> public ProcedureStore getStore() {<a name="line.660"></a>
+<span class="sourceLineNo">661</span> return this.store;<a name="line.661"></a>
+<span class="sourceLineNo">662</span> }<a name="line.662"></a>
+<span class="sourceLineNo">663</span><a name="line.663"></a>
+<span class="sourceLineNo">664</span> ProcedureScheduler getScheduler() {<a name="line.664"></a>
+<span class="sourceLineNo">665</span> return scheduler;<a name="line.665"></a>
+<span class="sourceLineNo">666</span> }<a name="line.666"></a>
+<span class="sourceLineNo">667</span><a name="line.667"></a>
+<span class="sourceLineNo">668</span> public void setKeepAliveTime(final long keepAliveTime, final TimeUnit timeUnit) {<a name="line.668"></a>
+<span class="sourceLineNo">669</span> this.keepAliveTime = timeUnit.toMillis(keepAliveTime);<a name="line.669"></a>
+<span class="sourceLineNo">670</span> this.scheduler.signalAll();<a name="line.670"></a>
+<span class="sourceLineNo">671</span> }<a name="line.671"></a>
+<span class="sourceLineNo">672</span><a name="line.672"></a>
+<span class="sourceLineNo">673</span> public long getKeepAliveTime(final TimeUnit timeUnit) {<a name="line.673"></a>
+<span class="sourceLineNo">674</span> return timeUnit.convert(keepAliveTime, TimeUnit.MILLISECONDS);<a name="line.674"></a>
+<span class="sourceLineNo">675</span> }<a name="line.675"></a>
+<span class="sourceLineNo">676</span><a name="line.676"></a>
+<span class="sourceLineNo">677</span> // ==========================================================================<a name="line.677"></a>
+<span class="sourceLineNo">678</span> // Submit/Remove Chores<a name="line.678"></a>
+<span class="sourceLineNo">679</span> // ==========================================================================<a name="line.679"></a>
<span class="sourceLineNo">680</span><a name="line.680"></a>
-<span class="sourceLineNo">681</span> // ==========================================================================<a name="line.681"></a>
-<span class="sourceLineNo">682</span> // Nonce Procedure helpers<a name="line.682"></a>
-<span class="sourceLineNo">683</span> // ==========================================================================<a name="line.683"></a>
-<span class="sourceLineNo">684</span> /**<a name="line.684"></a>
-<span class="sourceLineNo">685</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.685"></a>
-<span class="sourceLineNo">686</span> * @param nonceGroup<a name="line.686"></a>
-<span class="sourceLineNo">687</span> * @param nonce<a name="line.687"></a>
-<span class="sourceLineNo">688</span> * @return the generated NonceKey<a name="line.688"></a>
-<span class="sourceLineNo">689</span> */<a name="line.689"></a>
-<span class="sourceLineNo">690</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.690"></a>
-<span class="sourceLineNo">691</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.691"></a>
-<span class="sourceLineNo">692</span> }<a name="line.692"></a>
-<span class="sourceLineNo">693</span><a name="line.693"></a>
-<span class="sourceLineNo">694</span> /**<a name="line.694"></a>
-<span class="sourceLineNo">695</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.695"></a>
-<span class="sourceLineNo">696</span> * A procId will be reserved and on submitProcedure(),<a name="line.696"></a>
-<span class="sourceLineNo">697</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.697"></a>
-<span class="sourceLineNo">698</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.698"></a>
-<span class="sourceLineNo">699</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.699"></a>
-<span class="sourceLineNo">700</span> * and submit the procedure.<a name="line.700"></a>
-<span class="sourceLineNo">701</span> *<a name="line.701"></a>
-<span class="sourceLineNo">702</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.702"></a>
-<span class="sourceLineNo">703</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.703"></a>
-<span class="sourceLineNo">704</span> */<a name="line.704"></a>
-<span class="sourceLineNo">705</span> public long registerNonce(final NonceKey nonceKey) {<a name="line.705"></a>
-<span class="sourceLineNo">706</span> if (nonceKey == null) return -1;<a name="line.706"></a>
-<span class="sourceLineNo">707</span><a name="line.707"></a>
-<span class="sourceLineNo">708</span> // check if we have already a Reserved ID for the nonce<a name="line.708"></a>
-<span class="sourceLineNo">709</span> Long oldProcId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.709"></a>
-<span class="sourceLineNo">710</span> if (oldProcId == null) {<a name="line.710"></a>
-<span class="sourceLineNo">711</span> // reserve a new Procedure ID, this will be associated with the nonce<a name="line.711"></a>
-<span class="sourceLineNo">712</span> // and the procedure submitted with the specified nonce will use this ID.<a name="line.712"></a>
-<span class="sourceLineNo">713</span> final long newProcId = nextProcId();<a name="line.713"></a>
-<span class="sourceLineNo">714</span> oldProcId = nonceKeysToProcIdsMap.putIfAbsent(nonceKey, newProcId);<a name="line.714"></a>
-<span class="sourceLineNo">715</span> if (oldProcId == null) return -1;<a name="line.715"></a>
-<span class="sourceLineNo">716</span> }<a name="line.716"></a>
-<span class="sourceLineNo">717</span><a name="line.717"></a>
-<span class="sourceLineNo">718</span> // we found a registered nonce, but the procedure may not have been submitted yet.<a name="line.718"></a>
-<span class="sourceLineNo">719</span> // since the client expect the procedure to be submitted, spin here until it is.<a name="line.719"></a>
-<span class="sourceLineNo">720</span> final boolean traceEnabled = LOG.isTraceEnabled();<a name="line.720"></a>
-<span class="sourceLineNo">721</span> while (isRunning() &&<a name="line.721"></a>
-<span class="sourceLineNo">722</span> !(procedures.containsKey(oldProcId) || completed.containsKey(oldProcId)) &&<a name="line.722"></a>
-<span class="sourceLineNo">723</span> nonceKeysToProcIdsMap.containsKey(nonceKey)) {<a name="line.723"></a>
-<span class="sourceLineNo">724</span> if (traceEnabled) {<a name="line.724"></a>
-<span class="sourceLineNo">725</span> LOG.trace("Waiting for pid=" + oldProcId.longValue() + " to be submitted");<a name="line.725"></a>
-<span class="sourceLineNo">726</span> }<a name="line.726"></a>
-<span class="sourceLineNo">727</span> Threads.sleep(100);<a name="line.727"></a>
-<span class="sourceLineNo">728</span> }<a name="line.728"></a>
-<span class="sourceLineNo">729</span> return oldProcId.longValue();<a name="line.729"></a>
-<span class="sourceLineNo">730</span> }<a name="line.730"></a>
-<span class="sourceLineNo">731</span><a name="line.731"></a>
-<span class="sourceLineNo">732</span> /**<a name="line.732"></a>
-<span class="sourceLineNo">733</span> * Remove the NonceKey if the procedure was not submitted to the executor.<a name="line.733"></a>
-<span class="sourceLineNo">734</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.734"></a>
-<span class="sourceLineNo">735</span> */<a name="line.735"></a>
-<span class="sourceLineNo">736</span> public void unregisterNonceIfProcedureWasNotSubmitted(final NonceKey nonceKey) {<a name="line.736"></a>
-<span class="sourceLineNo">737</span> if (nonceKey == null) return;<a name="line.737"></a>
-<span class="sourceLineNo">738</span><a name="line.738"></a>
-<span class="sourceLineNo">739</span> final Long procId = nonceKeysToProcIdsMap.get(nonceKey);<a name="line.739"></a>
-<span class="sourceLineNo">740</span> if (procId == null) return;<a name="line.740"></a>
-<span class="sourceLineNo">741</span><a name="line.741"></a>
-<span class="sourceLineNo">742</span> // if the procedure was not submitted, remove the nonce<a name="line.742"></a>
-<span class="sourceLineNo">743</span> if (!(procedures.containsKey(procId) || completed.containsKey(procId))) {<a name="line.743"></a>
-<span class="sourceLineNo">744</span> nonceKeysToProcIdsMap.remove(nonceKey);<a name="line.744"></a>
-<span class="sourceLineNo">745</span> }<a name="line.745"></a>
-<span class="sourceLineNo">746</span> }<a name="line.746"></a>
-<span class="sourceLineNo">747</span><a name="line.747"></a>
-<span class="sourceLineNo">748</span> public static class FailedProcedure<TEnvironment> extends Procedure<TEnvironment> {<a name="line.748"></a>
-<span class="sourceLineNo">749</span> private String procName;<a name="line.749"></a>
+<span class="sourceLineNo">681</span> /**<a name="line.681"></a>
+<span class="sourceLineNo">682</span> * Add a chore procedure to the executor<a name="line.682"></a>
+<span class="sourceLineNo">683</span> * @param chore the chore to add<a name="line.683"></a>
+<span class="sourceLineNo">684</span> */<a name="line.684"></a>
+<span class="sourceLineNo">685</span> public void addChore(final ProcedureInMemoryChore chore) {<a name="line.685"></a>
+<span class="sourceLineNo">686</span> chore.setState(ProcedureState.WAITING_TIMEOUT);<a name="line.686"></a>
+<span class="sourceLineNo">687</span> timeoutExecutor.add(chore);<a name="line.687"></a>
+<span class="sourceLineNo">688</span> }<a name="line.688"></a>
+<span class="sourceLineNo">689</span><a name="line.689"></a>
+<span class="sourceLineNo">690</span> /**<a name="line.690"></a>
+<span class="sourceLineNo">691</span> * Remove a chore procedure from the executor<a name="line.691"></a>
+<span class="sourceLineNo">692</span> * @param chore the chore to remove<a name="line.692"></a>
+<span class="sourceLineNo">693</span> * @return whether the chore is removed, or it will be removed later<a name="line.693"></a>
+<span class="sourceLineNo">694</span> */<a name="line.694"></a>
+<span class="sourceLineNo">695</span> public boolean removeChore(final ProcedureInMemoryChore chore) {<a name="line.695"></a>
+<span class="sourceLineNo">696</span> chore.setState(ProcedureState.SUCCESS);<a name="line.696"></a>
+<span class="sourceLineNo">697</span> return timeoutExecutor.remove(chore);<a name="line.697"></a>
+<span class="sourceLineNo">698</span> }<a name="line.698"></a>
+<span class="sourceLineNo">699</span><a name="line.699"></a>
+<span class="sourceLineNo">700</span> // ==========================================================================<a name="line.700"></a>
+<span class="sourceLineNo">701</span> // Nonce Procedure helpers<a name="line.701"></a>
+<span class="sourceLineNo">702</span> // ==========================================================================<a name="line.702"></a>
+<span class="sourceLineNo">703</span> /**<a name="line.703"></a>
+<span class="sourceLineNo">704</span> * Create a NoneKey from the specified nonceGroup and nonce.<a name="line.704"></a>
+<span class="sourceLineNo">705</span> * @param nonceGroup<a name="line.705"></a>
+<span class="sourceLineNo">706</span> * @param nonce<a name="line.706"></a>
+<span class="sourceLineNo">707</span> * @return the generated NonceKey<a name="line.707"></a>
+<span class="sourceLineNo">708</span> */<a name="line.708"></a>
+<span class="sourceLineNo">709</span> public NonceKey createNonceKey(final long nonceGroup, final long nonce) {<a name="line.709"></a>
+<span class="sourceLineNo">710</span> return (nonce == HConstants.NO_NONCE) ? null : new NonceKey(nonceGroup, nonce);<a name="line.710"></a>
+<span class="sourceLineNo">711</span> }<a name="line.711"></a>
+<span class="sourceLineNo">712</span><a name="line.712"></a>
+<span class="sourceLineNo">713</span> /**<a name="line.713"></a>
+<span class="sourceLineNo">714</span> * Register a nonce for a procedure that is going to be submitted.<a name="line.714"></a>
+<span class="sourceLineNo">715</span> * A procId will be reserved and on submitProcedure(),<a name="line.715"></a>
+<span class="sourceLineNo">716</span> * the procedure with the specified nonce will take the reserved ProcId.<a name="line.716"></a>
+<span class="sourceLineNo">717</span> * If someone already reserved the nonce, this method will return the procId reserved,<a name="line.717"></a>
+<span class="sourceLineNo">718</span> * otherwise an invalid procId will be returned. and the caller should procede<a name="line.718"></a>
+<span class="sourceLineNo">719</span> * and submit the procedure.<a name="line.719"></a>
+<span class="sourceLineNo">720</span> *<a name="line.720"></a>
+<span class="sourceLineNo">721</span> * @param nonceKey A unique identifier for this operation from the client or process.<a name="line.721"></a>
+<span class="sourceLineNo">722</span> * @return the procId associated with the nonce, if any otherwise an invalid procId.<a name="line.
<TRUNCATED>