You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/12/15 13:23:53 UTC
[GitHub] [bookkeeper] wenbingshen opened a new pull request #2942: [BUGFIX] Fix the infinite waiting for shutdown due to throttler limit
wenbingshen opened a new pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942
Descriptions of the changes in this PR:
### Motivation
If the compactor is limited, the shutdown priority should be higher than waiting for RateLimiter.acquire.
### Changes
Since the GC thread in `EntryLogCompactor` if interrupted by Interrupt, it may bring corrupted messages, so `EntryLogCompactor` seems to be unable to avoid this problem; but `TransactionalEntryLogCompactor` phased compact has been carried out, so we can add some `synchronization` states in these stages to avoid long wait due to `RateLimiter.acquire`.
Master Issue: #2941
> ---
> In order to uphold a high standard for quality for code contributions, Apache BookKeeper runs various precommit
> checks for pull requests. A pull request can only be merged when it passes precommit checks.
>
> ---
> Be sure to do all of the following to help us incorporate your contribution
> quickly and easily:
>
> If this PR is a BookKeeper Proposal (BP):
>
> - [ ] Make sure the PR title is formatted like:
> `<BP-#>: Description of bookkeeper proposal`
> `e.g. BP-1: 64 bits ledger is support`
> - [ ] Attach the master issue link in the description of this PR.
> - [ ] Attach the google doc link if the BP is written in Google Doc.
>
> Otherwise:
>
> - [ ] Make sure the PR title is formatted like:
> `<Issue #>: Description of pull request`
> `e.g. Issue 123: Description ...`
> - [ ] Make sure tests pass via `mvn clean apache-rat:check install spotbugs:check`.
> - [ ] Replace `<Issue #>` in the title with the actual Issue number.
>
> ---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 commented on pull request #2942: [BUGFIX] Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-996540389
@wenbingshen Thanks for you enormous contribution.
The root cause of preventing GarbageCollectorThread shutdown is that the `compacting` flag only control one compact entry phase.
https://github.com/apache/bookkeeper/blob/7385d90189b2aa9ae11418a7c1c0fa60e5d0ea25/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java#L558-L579
So, it will wait for one entry log file compact complete before GarbageCollectorThread shutdown. If we throttle the compaction in low throughput, it will take multi minutes to compaction 2GB entry log file.
IMO, we'd better introduce a new compact flag for `AbstractLogCompactor` and check the status on `newScanner.process` method, if the status is false, just throw an IOException, and stop the compact immediately. It can work for both `EntryLogCompactor` and `TransactionalEntryLogCompactor`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #2942: Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-1011621108
@pkumar-singh It's been a long time. Can you take a look at this pr? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #2942: [WIP] Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-1001464462
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #2942: [BUGFIX] Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-994789977
@eolivelli @merlimat @codelipenghui Can you take a look at this pr. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #2942: [WIP] Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-1001484963
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #2942: Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-1004929578
ping @eolivelli PTAL, Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #2942: [WIP] Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-1002097339
@hangc0276 PTAL, thanks. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #2942: [BUGFIX] Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-996703443
@hangc0276 Gotcha! Thanks for your review. I will implement it soon. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #2942: [WIP] Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-1001901788
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #2942: Fix the infinite waiting for shutdown due to throttler limit
Posted by GitBox <gi...@apache.org>.
wenbingshen commented on pull request #2942:
URL: https://github.com/apache/bookkeeper/pull/2942#issuecomment-1006491002
@eolivelli PTAL Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org