You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/08/19 09:47:10 UTC
[GitHub] [bookkeeper] gaozhangmin opened a new pull request #2767: skip replicating ledger whose bookkeeper server is online
gaozhangmin opened a new pull request #2767:
URL: https://github.com/apache/bookkeeper/pull/2767
Fixes #2766
### Motivation
When 2 or more bookies in the cluster are down, the auto-recovery gets triggered, but as these come back online, the rereplication worker should Ideally skip the rereplication of the ledgers that are marked as underreplicated. But the ledgers are rereplicated instead
### Changes
skip replicated ledger when bookkeeper server is online.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on pull request #2767: skip replicating ledger when bookkeeper server is online
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #2767:
URL: https://github.com/apache/bookkeeper/pull/2767#issuecomment-906274203
This is is interesting.
Thank you for sharing your needs and your idea for fixing the problem
It is better to start a discussion on dev@bookkeeper.apache.org in order to have a broader audience.
I would add a configuration flag to explicitly enable this behaviour.
Some quick notes (but really, it is better to start a discussion on the ML):
- if the bookie has thousands of ledgers (this is very common) we are going to perform many writes to ZK, we should add some rate limiter and also probably save us from writing to ZK if the ledger is not yet marked as underreplicated
- the fact that one bookies contains the data is not enough to say that the ledger is "underreplicated" (if you want 5 copies and 4 bookies are lost you are still underreplicated even in case of the only one remaining bookie is up and running)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] gaozhangmin commented on pull request #2767: skip replicating ledger when bookkeeper server is online
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #2767:
URL: https://github.com/apache/bookkeeper/pull/2767#issuecomment-906879392
> You can subscribe [dev@bookkeeper.apache.org](mailto:dev@bookkeeper.apache.org) here
> https://bookkeeper.apache.org/community/mailing-lists/
>
> search for "dev@" in the page and follow the instructions.
>
> Feel free to reach out to me on slack, I will be happy to help you in subscribing
> https://bookkeeper.apache.org/community/slack/
A discussion started here, http://mail-archives.apache.org/mod_mbox/bookkeeper-dev/202108.mbox/browser
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on pull request #2767: skip replicating ledger when bookkeeper server is online
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #2767:
URL: https://github.com/apache/bookkeeper/pull/2767#issuecomment-906363373
You can subscribe dev@bookkeeper.apache.org here
https://bookkeeper.apache.org/community/mailing-lists/
search for "dev@" in the page and follow the instructions.
Feel free to reach out to me on slack, I will be happy to help you in subscribing
https://bookkeeper.apache.org/community/slack/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] gaozhangmin commented on pull request #2767: skip replicating ledger when bookkeeper server is online
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #2767:
URL: https://github.com/apache/bookkeeper/pull/2767#issuecomment-906298492
> This is is interesting.
> Thank you for sharing your needs and your idea for fixing the problem
>
> It is better to start a discussion on [dev@bookkeeper.apache.org](mailto:dev@bookkeeper.apache.org) in order to have a broader audience.
>
> I would add a configuration flag to explicitly enable this behaviour.
>
> Some quick notes (but really, it is better to start a discussion on the ML):
>
> * if the bookie has thousands of ledgers (this is very common) we are going to perform many writes to ZK, we should add some rate limiter and also probably save us from writing to ZK if the ledger is not yet marked as underreplicated
> * the fact that one bookies contains the data is not enough to say that the ledger is "underreplicated" (if you want 5 copies and 4 bookies are lost you are still underreplicated even in case of the only one remaining bookie is up and running)
where to start a discussion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] gaozhangmin commented on pull request #2767: skip replicating ledger when bookkeeper server is online
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #2767:
URL: https://github.com/apache/bookkeeper/pull/2767#issuecomment-906314762
> This is is interesting.
> Thank you for sharing your needs and your idea for fixing the problem
>
> It is better to start a discussion on [dev@bookkeeper.apache.org](mailto:dev@bookkeeper.apache.org) in order to have a broader audience.
>
> I would add a configuration flag to explicitly enable this behaviour.
>
> Some quick notes (but really, it is better to start a discussion on the ML):
>
> * if the bookie has thousands of ledgers (this is very common) we are going to perform many writes to ZK, we should add some rate limiter and also probably save us from writing to ZK if the ledger is not yet marked as underreplicated
> * the fact that one bookies contains the data is not enough to say that the ledger is "underreplicated" (if you want 5 copies and 4 bookies are lost you are still underreplicated even in case of the only one remaining bookie is up and running)
For situation 2, I think is ok, because auditor would bring ledger to the underreplicated again, after this pr marked it replicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] gaozhangmin commented on pull request #2767: skip replicating ledger when bookkeeper server is online
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #2767:
URL: https://github.com/apache/bookkeeper/pull/2767#issuecomment-906150370
@hangc0276 @codelipenghui @merlimat PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] gaozhangmin closed pull request #2767: skip replicating ledger when bookkeeper server is online
Posted by GitBox <gi...@apache.org>.
gaozhangmin closed pull request #2767:
URL: https://github.com/apache/bookkeeper/pull/2767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] gaozhangmin commented on pull request #2767: skip replicating ledger when bookkeeper server is online
Posted by GitBox <gi...@apache.org>.
gaozhangmin commented on pull request #2767:
URL: https://github.com/apache/bookkeeper/pull/2767#issuecomment-907629259
@eolivelli
Maybe the idea remove ledger from underreplicated when bookie starts up has some problem, What important is, replicate legder puts so much heavy loads on cluster. I want to reduce the affect.
suppose there are 5 copies and 4 bookies are lost, when one lost bookie is online again, I don't simply mark the ledger replicated. When the replicateWork gets the failed bookies, it will ignore the one which online again. So, this will reduce replicating one copy of data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org