[GitHub] sijie opened a new pull request #1543: Issue 1540: Bookie/BookieServer components shutdown will fail to end exit the BookieProcess

[GitBox <gi...@apache.org>]

sijie opened a new pull request #1543: Issue 1540: Bookie/BookieServer components shutdown will fail to end exit the BookieProcess
URL: https://github.com/apache/bookkeeper/pull/1543
 
 
   Descriptions of the changes in this PR:
   
    ### Motivation
   
   Fixes the issue at #1540.
   
   If Bookie/BookieServer components are shutdown internally because of any fatal errors
   (ExitCode - INVALID_CONF, SERVER_EXCEPTION, ZK_EXPIRED, ZK_REG_FAIL, BOOKIE_EXCEPTION) then
   it will go through shutdown method logic and shutdowns components internal to Bookie/BookieServer
   but it will not succeed in bringing down the bookie process.
   
   This is because in BookieServer.main / server.Main.doMain it would wait for the startComponent
   future to complete
   http://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/server/Main.java#L227 .
   The startComponent future will be market complete only in runtime shutdownhook -
   https://github.com/apache/bookkeeper/blob/master/bookkeeper-common/src/main/java/org/apache/bookkeeper/common/component/ComponentStarter.java#L66.
   
   But the problem is nowhere in Bookie/BookieProcess shutdown we are calling System.exit() and hence
   the runtime shutdownhook is not executed to mark the startComponent future to complete. Hence
   Main.doMain will wait forever on this future though Bookie/BookieServer components are shutdown
   because of known fatal errors.
   
    ### Regression
   
   Issue #508 introduced this regression. Before this change, the main thread is blocking using `BookieServer#join()`.
   When bookie is dead for any reason, the DeathWatchThread will kill the bookie and bookie server. so the main thread will quite.
   However after #508 is introduced, the lifecycle management is disconnected from the bookie and bookie server. so when they are dead,
   lifecycle management is unaware of the situation and the main thread doesn't quite.
   
    ### Changes
   
   - Add `UncaughtExceptionHandler` to lifecycle components
   - When a lifecycle component hits an error, it can use `UncaughtExceptionHandler` to notify lifecycle component stack to shutdown the whole stack
   
   
   
   Master Issue: #1540 
   
   > ---
   > In order to uphold a high standard for quality for code contributions, Apache BookKeeper runs various precommit
   > checks for pull requests. A pull request can only be merged when it passes precommit checks. However running all
   > the precommit checks can take a long time, some trivial changes don't need to run all the precommit checks. You
   > can check following list to skip the tests that don't need to run for your pull request. Leave them unchecked if
   > you are not sure, committers will help you:
   >
   > - [ ] [skip bookkeeper-server bookie tests]: skip testing `org.apache.bookkeeper.bookie` in bookkeeper-server module.
   > - [ ] [skip bookkeeper-server client tests]: skip testing `org.apache.bookkeeper.client` in bookkeeper-server module.
   > - [ ] [skip bookkeeper-server replication tests]: skip testing `org.apache.bookkeeper.replication` in bookkeeper-server module.
   > - [ ] [skip bookkeeper-server tls tests]: skip testing `org.apache.bookkeeper.tls` in bookkeeper-server module.
   > - [ ] [skip bookkeeper-server remaining tests]: skip testing all other tests in bookkeeper-server module.
   > - [ ] [skip integration tests]: skip docker based integration tests. if you make java code changes, you shouldn't skip integration tests.
   > - [ ] [skip build java8]: skip build on java8. *ONLY* skip this when *ONLY* changing files under documentation under `site`.
   > - [ ] [skip build java9]: skip build on java9. *ONLY* skip this when *ONLY* changing files under documentation under `site`.
   > ---
   
   > ---
   > Be sure to do all of the following to help us incorporate your contribution
   > quickly and easily:
   >
   > If this PR is a BookKeeper Proposal (BP):
   >
   > - [ ] Make sure the PR title is formatted like:
   >     `<BP-#>: Description of bookkeeper proposal`
   >     `e.g. BP-1: 64 bits ledger is support`
   > - [ ] Attach the master issue link in the description of this PR.
   > - [ ] Attach the google doc link if the BP is written in Google Doc.
   >
   > Otherwise:
   > 
   > - [ ] Make sure the PR title is formatted like:
   >     `<Issue #>: Description of pull request`
   >     `e.g. Issue 123: Description ...`
   > - [ ] Make sure tests pass via `mvn clean apache-rat:check install spotbugs:check`.
   > - [ ] Replace `<Issue #>` in the title with the actual Issue number.
   > 
   > ---
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services