You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/03/27 16:38:19 UTC
[GitHub] [hbase] saintstack opened a new pull request #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
saintstack opened a new pull request #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373
Add check for stopped server at a few more points in Master startup.
Defend against NPE in RSProcedureDispatcher; log and retun instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#issuecomment-605313605
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
| -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 1s | branch-2.3 passed |
| +1 :green_heart: | compile | 0m 59s | branch-2.3 passed |
| +1 :green_heart: | shadedjars | 4m 47s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 38s | branch-2.3 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 22s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 33s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 68m 42s | hbase-server in the patch passed. |
| | | 95m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1373 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 78e0c2f102ce 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / c06af5d05d |
| Default Java | 1.8.0_232 |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/testReport/ |
| Max. process+thread count | 5336 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#issuecomment-605216595
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 42s | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 37s | branch-2.3 passed |
| +1 :green_heart: | compile | 1m 10s | branch-2.3 passed |
| +1 :green_heart: | shadedjars | 5m 29s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 42s | hbase-server in branch-2.3 failed. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 16s | the patch passed |
| +1 :green_heart: | compile | 1m 8s | the patch passed |
| +1 :green_heart: | javac | 1m 8s | the patch passed |
| +1 :green_heart: | shadedjars | 6m 40s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 1m 6s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| -0 :warning: | unit | 107m 51s | hbase-server in the patch failed. |
| | | 140m 9s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1373 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 67da76e9155e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / c06af5d05d |
| Default Java | 2020-01-14 |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/testReport/ |
| Max. process+thread count | 5247 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#issuecomment-605130958
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ branch-2.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 45s | branch-2.3 passed |
| +1 :green_heart: | checkstyle | 1m 11s | branch-2.3 passed |
| +1 :green_heart: | spotbugs | 1m 59s | branch-2.3 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 7s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 155 unchanged - 1 fixed = 155 total (was 156) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 10m 33s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. |
| +1 :green_heart: | spotbugs | 2m 19s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 35m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1373 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 495e24ea124a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / c06af5d05d |
| Max. process+thread count | 93 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
saintstack commented on issue #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#issuecomment-606297703
Signed-off-by: Bharath Vissapragada <bh...@apache.org>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1373:
HBASE-24052 Add debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
saintstack commented on a change in pull request #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#discussion_r399491856
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##########
@@ -156,48 +151,19 @@ public void testMasterShutdownBeforeStartingAnyRegionServer() throws Exception {
hbaseCluster = new LocalHBaseCluster(htu.getConfiguration(), options.getNumMasters(),
options.getNumRegionServers(), options.getMasterClass(), options.getRsClass());
final MasterThread masterThread = hbaseCluster.getMasters().get(0);
-
- final CompletableFuture<Void> shutdownFuture = CompletableFuture.runAsync(() -> {
- // Switching to master registry exacerbated a race in the master bootstrap that can result
- // in a lost shutdown command (HBASE-8422, HBASE-23836). The race is essentially because
- // the server manager in HMaster is not initialized by the time shutdown() RPC (below) is
- // made to the master. The suspected reason as to why it was uncommon before HBASE-18095
- // is because the connection creation with ZK registry is so slow that by then the server
- // manager is usually init'ed in time for the RPC to be made. For now, adding an explicit
- // wait() in the test, waiting for the server manager to become available.
- final long timeout = TimeUnit.MINUTES.toMillis(10);
- assertNotEquals("timeout waiting for server manager to become available.",
- -1, Waiter.waitFor(htu.getConfiguration(), timeout,
- () -> masterThread.getMaster().getServerManager() != null));
-
- // Master has come up far enough that we can terminate it without creating a zombie.
- final long result = Waiter.waitFor(htu.getConfiguration(), timeout, 500, () -> {
- final Configuration conf = createResponsiveZkConfig(htu.getConfiguration());
- LOG.debug("Attempting to establish connection.");
- final CompletableFuture<AsyncConnection> connFuture =
- ConnectionFactory.createAsyncConnection(conf);
- try (final AsyncConnection conn = connFuture.join()) {
- LOG.debug("Sending shutdown RPC.");
- try {
- conn.getAdmin().shutdown().join();
- LOG.debug("Shutdown RPC sent.");
- return true;
- } catch (CompletionException e) {
- LOG.debug("Failure sending shutdown RPC.");
- }
- } catch (IOException|CompletionException e) {
- LOG.debug("Failed to establish connection.");
- } catch (Throwable e) {
- LOG.info("Something unexpected happened.", e);
- }
- return false;
- });
- assertNotEquals("Failed to issue shutdown RPC after " + Duration.ofMillis(timeout),
- -1, result);
- });
-
masterThread.start();
- shutdownFuture.join();
+ // Switching to master registry exacerbated a race in the master bootstrap that can result
+ // in a lost shutdown command (HBASE-8422, HBASE-23836). The race is essentially because
+ // the server manager in HMaster is not initialized by the time shutdown() RPC (below) is
+ // made to the master. The suspected reason as to why it was uncommon before HBASE-18095
+ // is because the connection creation with ZK registry is so slow that by then the server
+ // manager is usually init'ed in time for the RPC to be made. For now, adding an explicit
+ // wait() in the test, waiting for the server manager to become available.
+ final long timeout = TimeUnit.MINUTES.toMillis(10);
+ assertNotEquals("Timeout waiting for server manager to become available.",
+ -1, Waiter.waitFor(htu.getConfiguration(), timeout,
+ () -> masterThread.getMaster().getServerManager() != null));
+ htu.getConnection().getAdmin().shutdown();
Review comment:
Thanks for taking a look @bharathv .
Yeah, something up. Looking at what is happening inline in the rpc request seems minimal -- setting flags -- but something is not right... I filed HBASE-24070, an issue to look at this, as you suggest above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#issuecomment-605298182
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 2s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
||| _ branch-2.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 8m 1s | branch-2.3 passed |
| +1 :green_heart: | checkstyle | 1m 57s | branch-2.3 passed |
| +1 :green_heart: | spotbugs | 3m 12s | branch-2.3 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 8m 0s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 55s | hbase-server: The patch generated 0 new + 155 unchanged - 1 fixed = 155 total (was 156) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | hadoopcheck | 16m 10s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2. |
| +1 :green_heart: | spotbugs | 3m 26s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 16s | The patch does not generate ASF License warnings. |
| | | 54m 3s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1373 |
| Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle |
| uname | Linux 2abaa2225f3c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / c06af5d05d |
| Max. process+thread count | 94 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1373:
HBASE-24052 Add debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#discussion_r399460335
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##########
@@ -156,48 +151,19 @@ public void testMasterShutdownBeforeStartingAnyRegionServer() throws Exception {
hbaseCluster = new LocalHBaseCluster(htu.getConfiguration(), options.getNumMasters(),
options.getNumRegionServers(), options.getMasterClass(), options.getRsClass());
final MasterThread masterThread = hbaseCluster.getMasters().get(0);
-
- final CompletableFuture<Void> shutdownFuture = CompletableFuture.runAsync(() -> {
- // Switching to master registry exacerbated a race in the master bootstrap that can result
- // in a lost shutdown command (HBASE-8422, HBASE-23836). The race is essentially because
- // the server manager in HMaster is not initialized by the time shutdown() RPC (below) is
- // made to the master. The suspected reason as to why it was uncommon before HBASE-18095
- // is because the connection creation with ZK registry is so slow that by then the server
- // manager is usually init'ed in time for the RPC to be made. For now, adding an explicit
- // wait() in the test, waiting for the server manager to become available.
- final long timeout = TimeUnit.MINUTES.toMillis(10);
- assertNotEquals("timeout waiting for server manager to become available.",
- -1, Waiter.waitFor(htu.getConfiguration(), timeout,
- () -> masterThread.getMaster().getServerManager() != null));
-
- // Master has come up far enough that we can terminate it without creating a zombie.
- final long result = Waiter.waitFor(htu.getConfiguration(), timeout, 500, () -> {
- final Configuration conf = createResponsiveZkConfig(htu.getConfiguration());
- LOG.debug("Attempting to establish connection.");
- final CompletableFuture<AsyncConnection> connFuture =
- ConnectionFactory.createAsyncConnection(conf);
- try (final AsyncConnection conn = connFuture.join()) {
- LOG.debug("Sending shutdown RPC.");
- try {
- conn.getAdmin().shutdown().join();
- LOG.debug("Shutdown RPC sent.");
- return true;
- } catch (CompletionException e) {
- LOG.debug("Failure sending shutdown RPC.");
- }
- } catch (IOException|CompletionException e) {
- LOG.debug("Failed to establish connection.");
- } catch (Throwable e) {
- LOG.info("Something unexpected happened.", e);
- }
- return false;
- });
- assertNotEquals("Failed to issue shutdown RPC after " + Duration.ofMillis(timeout),
- -1, result);
- });
-
masterThread.start();
- shutdownFuture.join();
+ // Switching to master registry exacerbated a race in the master bootstrap that can result
+ // in a lost shutdown command (HBASE-8422, HBASE-23836). The race is essentially because
+ // the server manager in HMaster is not initialized by the time shutdown() RPC (below) is
+ // made to the master. The suspected reason as to why it was uncommon before HBASE-18095
+ // is because the connection creation with ZK registry is so slow that by then the server
+ // manager is usually init'ed in time for the RPC to be made. For now, adding an explicit
+ // wait() in the test, waiting for the server manager to become available.
+ final long timeout = TimeUnit.MINUTES.toMillis(10);
+ assertNotEquals("Timeout waiting for server manager to become available.",
+ -1, Waiter.waitFor(htu.getConfiguration(), timeout,
+ () -> masterThread.getMaster().getServerManager() != null));
+ htu.getConnection().getAdmin().shutdown();
Review comment:
I see what you did here..I'm ok with this to stabilize the test.
I think internally it is masking the problem of "shutdown()" implementation being synchronous on the _server_ side. Ideally, I'd think it should be async on the server side too, meaning the rpc call just sets a flag (startShuttingDown = true) which is read by multiple running threads and they begin the tear down rather than doing it in the context of the rpc thread. We already do this partially but the main driver of master shutdown still runs in the rpc context. Don't think this should block the test, but just something for the future. Thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
saintstack commented on issue #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#issuecomment-605259615
Rerunning tests to see if failures related.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] bharathv commented on a change in pull request #1373:
HBASE-24052 Add debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
bharathv commented on a change in pull request #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#discussion_r399546754
##########
File path: hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
##########
@@ -156,48 +151,19 @@ public void testMasterShutdownBeforeStartingAnyRegionServer() throws Exception {
hbaseCluster = new LocalHBaseCluster(htu.getConfiguration(), options.getNumMasters(),
options.getNumRegionServers(), options.getMasterClass(), options.getRsClass());
final MasterThread masterThread = hbaseCluster.getMasters().get(0);
-
- final CompletableFuture<Void> shutdownFuture = CompletableFuture.runAsync(() -> {
- // Switching to master registry exacerbated a race in the master bootstrap that can result
- // in a lost shutdown command (HBASE-8422, HBASE-23836). The race is essentially because
- // the server manager in HMaster is not initialized by the time shutdown() RPC (below) is
- // made to the master. The suspected reason as to why it was uncommon before HBASE-18095
- // is because the connection creation with ZK registry is so slow that by then the server
- // manager is usually init'ed in time for the RPC to be made. For now, adding an explicit
- // wait() in the test, waiting for the server manager to become available.
- final long timeout = TimeUnit.MINUTES.toMillis(10);
- assertNotEquals("timeout waiting for server manager to become available.",
- -1, Waiter.waitFor(htu.getConfiguration(), timeout,
- () -> masterThread.getMaster().getServerManager() != null));
-
- // Master has come up far enough that we can terminate it without creating a zombie.
- final long result = Waiter.waitFor(htu.getConfiguration(), timeout, 500, () -> {
- final Configuration conf = createResponsiveZkConfig(htu.getConfiguration());
- LOG.debug("Attempting to establish connection.");
- final CompletableFuture<AsyncConnection> connFuture =
- ConnectionFactory.createAsyncConnection(conf);
- try (final AsyncConnection conn = connFuture.join()) {
- LOG.debug("Sending shutdown RPC.");
- try {
- conn.getAdmin().shutdown().join();
- LOG.debug("Shutdown RPC sent.");
- return true;
- } catch (CompletionException e) {
- LOG.debug("Failure sending shutdown RPC.");
- }
- } catch (IOException|CompletionException e) {
- LOG.debug("Failed to establish connection.");
- } catch (Throwable e) {
- LOG.info("Something unexpected happened.", e);
- }
- return false;
- });
- assertNotEquals("Failed to issue shutdown RPC after " + Duration.ofMillis(timeout),
- -1, result);
- });
-
masterThread.start();
- shutdownFuture.join();
+ // Switching to master registry exacerbated a race in the master bootstrap that can result
+ // in a lost shutdown command (HBASE-8422, HBASE-23836). The race is essentially because
+ // the server manager in HMaster is not initialized by the time shutdown() RPC (below) is
+ // made to the master. The suspected reason as to why it was uncommon before HBASE-18095
+ // is because the connection creation with ZK registry is so slow that by then the server
+ // manager is usually init'ed in time for the RPC to be made. For now, adding an explicit
+ // wait() in the test, waiting for the server manager to become available.
+ final long timeout = TimeUnit.MINUTES.toMillis(10);
+ assertNotEquals("Timeout waiting for server manager to become available.",
+ -1, Waiter.waitFor(htu.getConfiguration(), timeout,
+ () -> masterThread.getMaster().getServerManager() != null));
+ htu.getConnection().getAdmin().shutdown();
Review comment:
Thanks for the jira, I'll try to poke around too a little bit to see if I can find something. Fwiw, I think a small comment detailing this finding with a jira reference would be helpful. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack merged pull request #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
saintstack merged pull request #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#issuecomment-605328370
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 59s | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 8m 20s | branch-2.3 passed |
| +1 :green_heart: | compile | 1m 10s | branch-2.3 passed |
| +1 :green_heart: | shadedjars | 5m 52s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 41s | hbase-server in branch-2.3 failed. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 52s | the patch passed |
| +1 :green_heart: | compile | 1m 11s | the patch passed |
| +1 :green_heart: | javac | 1m 11s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 51s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. |
||| _ Other Tests _ |
| -0 :warning: | unit | 104m 16s | hbase-server in the patch failed. |
| | | 137m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1373 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 36e9dfdcfd5a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / c06af5d05d |
| Default Java | 2020-01-14 |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt |
| javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/testReport/ |
| Max. process+thread count | 4100 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/2/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1373: HBASE-24052 Add
debug+fix to TestMasterShutdown
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #1373: HBASE-24052 Add debug+fix to TestMasterShutdown
URL: https://github.com/apache/hbase/pull/1373#issuecomment-605213430
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 42s | Docker mode activated. |
| -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ branch-2.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 45s | branch-2.3 passed |
| +1 :green_heart: | compile | 0m 57s | branch-2.3 passed |
| +1 :green_heart: | shadedjars | 4m 36s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 40s | branch-2.3 passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 20s | the patch passed |
| +1 :green_heart: | compile | 1m 1s | the patch passed |
| +1 :green_heart: | javac | 1m 1s | the patch passed |
| +1 :green_heart: | shadedjars | 4m 30s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 106m 0s | hbase-server in the patch failed. |
| | | 132m 40s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/1373 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 35a219320503 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | branch-2.3 / c06af5d05d |
| Default Java | 1.8.0_232 |
| unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/testReport/ |
| Max. process+thread count | 5031 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1373/1/console |
| versions | git=2.17.1 maven=2018-06-17T18:33:14Z) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services