You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Matteo Merli <mm...@apache.org> on 2021/12/14 18:03:20 UTC
[DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
https://github.com/apache/pulsar/issues/13304
Pasted below for quoting convenience.
-------
## Motivation
After all the work done for PIP-45 that was already included in 2.8 and 2.9
releases, it enabled the concept of re-acquirable resource locks and leader
election.
Another important change was to avoid doing any deferrable metadata operation
when we know that we are not currently connected to the metadata service.
Finally, that enabled stabilization in 2.9 the configuration setting that allows
brokers to continue operating in a safe mode when the session with ZooKeeper
expires.
The way it works is that, when we lose a ZooKeeper session, the data plane will
continue to work undisturbed, relying on the BookKeeper fencing to avoid any
inconsistencies.
New topics are not able to get started, but existing topics will see no
impact.
The original intention for shutting down the brokers was to ensure that we
would automatically go back to a consistent state, with respect to which
resources are "owned" in ZooKeeper by a given broker.
With the re-acquirable resource locks, that problem was solved and thoroughly
tested to be robust.
## Proposed changes
In 2.10 release, for the setting:
```properties
# There are two policies to apply when a broker metadata session
expires: session expired happens, "shutdown" or "reconnect".
# With "shutdown", the broker will be restarted.
# With "reconnect", the broker will keep serving the topics, while
attempting to recreate a new session.
zookeeperSessionExpiredPolicy=shutdown
```
Change its default value to `reconnect`.
--
Matteo Merli
<mm...@apache.org>
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by Hang Chen <ch...@apache.org>.
+1
Thanks,
Hang
PengHui Li <pe...@apache.org> 于2021年12月15日周三 07:59写道：
>
> +1
>
> Penghui
>
> On Wed, Dec 15, 2021 at 2:03 AM Matteo Merli <mm...@apache.org> wrote:
>
> > https://github.com/apache/pulsar/issues/13304
> >
> >
> > Pasted below for quoting convenience.
> >
> > -------
> >
> >
> > ## Motivation
> >
> > After all the work done for PIP-45 that was already included in 2.8 and 2.9
> > releases, it enabled the concept of re-acquirable resource locks and leader
> > election.
> >
> > Another important change was to avoid doing any deferrable metadata
> > operation
> > when we know that we are not currently connected to the metadata service.
> >
> > Finally, that enabled stabilization in 2.9 the configuration setting that
> > allows
> > brokers to continue operating in a safe mode when the session with
> > ZooKeeper
> > expires.
> >
> > The way it works is that, when we lose a ZooKeeper session, the data plane
> > will
> > continue to work undisturbed, relying on the BookKeeper fencing to avoid
> > any
> > inconsistencies.
> >
> > New topics are not able to get started, but existing topics will see no
> > impact.
> >
> > The original intention for shutting down the brokers was to ensure that we
> > would automatically go back to a consistent state, with respect to which
> > resources are "owned" in ZooKeeper by a given broker.
> >
> > With the re-acquirable resource locks, that problem was solved and
> > thoroughly
> > tested to be robust.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # There are two policies to apply when a broker metadata session
> > expires: session expired happens, "shutdown" or "reconnect".
> > # With "shutdown", the broker will be restarted.
> > # With "reconnect", the broker will keep serving the topics, while
> > attempting to recreate a new session.
> > zookeeperSessionExpiredPolicy=shutdown
> > ```
> >
> > Change its default value to `reconnect`.
> >
> >
> > --
> > Matteo Merli
> > <mm...@apache.org>
> >
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by PengHui Li <pe...@apache.org>.
+1
Penghui
On Wed, Dec 15, 2021 at 2:03 AM Matteo Merli <mm...@apache.org> wrote:
> https://github.com/apache/pulsar/issues/13304
>
>
> Pasted below for quoting convenience.
>
> -------
>
>
> ## Motivation
>
> After all the work done for PIP-45 that was already included in 2.8 and 2.9
> releases, it enabled the concept of re-acquirable resource locks and leader
> election.
>
> Another important change was to avoid doing any deferrable metadata
> operation
> when we know that we are not currently connected to the metadata service.
>
> Finally, that enabled stabilization in 2.9 the configuration setting that
> allows
> brokers to continue operating in a safe mode when the session with
> ZooKeeper
> expires.
>
> The way it works is that, when we lose a ZooKeeper session, the data plane
> will
> continue to work undisturbed, relying on the BookKeeper fencing to avoid
> any
> inconsistencies.
>
> New topics are not able to get started, but existing topics will see no
> impact.
>
> The original intention for shutting down the brokers was to ensure that we
> would automatically go back to a consistent state, with respect to which
> resources are "owned" in ZooKeeper by a given broker.
>
> With the re-acquirable resource locks, that problem was solved and
> thoroughly
> tested to be robust.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # There are two policies to apply when a broker metadata session
> expires: session expired happens, "shutdown" or "reconnect".
> # With "shutdown", the broker will be restarted.
> # With "reconnect", the broker will keep serving the topics, while
> attempting to recreate a new session.
> zookeeperSessionExpiredPolicy=shutdown
> ```
>
> Change its default value to `reconnect`.
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by Matteo Merli <ma...@gmail.com>.
On Mon, Dec 20, 2021 at 2:24 PM Michael Marshall <mm...@apache.org> wrote:
> I haven't been able to follow the PIP 45 work, so I have some high
> level questions. Do we have documentation on the implementation of
> these re-acquirable resource locks? I'd like to understand how these
> work.
There is admittedly not much high-level doc on the locks. The
implementation class is the best reference atm:
https://github.com/apache/pulsar/blob/master/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java
Similarly, for leader election implementation:
https://github.com/apache/pulsar/blob/master/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/LeaderElectionImpl.java
> Also, are you able to describe the thorough testing? Were these
> tests manual or are they already part of our GitHub CI or both?
It's a combination of those. In the unit tests, through fault
injections, we're reproducing all the possible states.
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by Michael Marshall <mm...@apache.org>.
+1 sounds like a great improvement.
> With the re-acquirable resource locks, that problem was solved and thoroughly
> tested to be robust.
I haven't been able to follow the PIP 45 work, so I have some high
level questions. Do we have documentation on the implementation of
these re-acquirable resource locks? I'd like to understand how these
work. Also, are you able to describe the thorough testing? Were these
tests manual or are they already part of our GitHub CI or both?
Thanks,
Michael
On Tue, Dec 14, 2021 at 12:03 PM Matteo Merli <mm...@apache.org> wrote:
>
> https://github.com/apache/pulsar/issues/13304
>
>
> Pasted below for quoting convenience.
>
> -------
>
>
> ## Motivation
>
> After all the work done for PIP-45 that was already included in 2.8 and 2.9
> releases, it enabled the concept of re-acquirable resource locks and leader
> election.
>
> Another important change was to avoid doing any deferrable metadata operation
> when we know that we are not currently connected to the metadata service.
>
> Finally, that enabled stabilization in 2.9 the configuration setting that allows
> brokers to continue operating in a safe mode when the session with ZooKeeper
> expires.
>
> The way it works is that, when we lose a ZooKeeper session, the data plane will
> continue to work undisturbed, relying on the BookKeeper fencing to avoid any
> inconsistencies.
>
> New topics are not able to get started, but existing topics will see no
> impact.
>
> The original intention for shutting down the brokers was to ensure that we
> would automatically go back to a consistent state, with respect to which
> resources are "owned" in ZooKeeper by a given broker.
>
> With the re-acquirable resource locks, that problem was solved and thoroughly
> tested to be robust.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # There are two policies to apply when a broker metadata session
> expires: session expired happens, "shutdown" or "reconnect".
> # With "shutdown", the broker will be restarted.
> # With "reconnect", the broker will keep serving the topics, while
> attempting to recreate a new session.
> zookeeperSessionExpiredPolicy=shutdown
> ```
>
> Change its default value to `reconnect`.
>
>
> --
> Matteo Merli
> <mm...@apache.org>
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by PengHui Li <pe...@apache.org>.
Hi Lin Lin
Which version you are working on the test?
Do you have any steps to reproduce the issue?
We'd better fix the problem in 2.10.
Regards,
Penghui
On Tue, Dec 21, 2021 at 11:06 AM Lin Lin <li...@apache.org> wrote:
>
>
> On 2021/12/14 18:03:20 Matteo Merli wrote:
> > https://github.com/apache/pulsar/issues/13304
> >
> >
> > Pasted below for quoting convenience.
> >
> > -------
> >
> >
> > ## Motivation
> >
> > After all the work done for PIP-45 that was already included in 2.8 and
> 2.9
> > releases, it enabled the concept of re-acquirable resource locks and
> leader
> > election.
> >
> > Another important change was to avoid doing any deferrable metadata
> operation
> > when we know that we are not currently connected to the metadata service.
> >
> > Finally, that enabled stabilization in 2.9 the configuration setting
> that allows
> > brokers to continue operating in a safe mode when the session with
> ZooKeeper
> > expires.
> >
> > The way it works is that, when we lose a ZooKeeper session, the data
> plane will
> > continue to work undisturbed, relying on the BookKeeper fencing to avoid
> any
> > inconsistencies.
> >
> > New topics are not able to get started, but existing topics will see no
> > impact.
> >
> > The original intention for shutting down the brokers was to ensure that
> we
> > would automatically go back to a consistent state, with respect to which
> > resources are "owned" in ZooKeeper by a given broker.
> >
> > With the re-acquirable resource locks, that problem was solved and
> thoroughly
> > tested to be robust.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # There are two policies to apply when a broker metadata session
> > expires: session expired happens, "shutdown" or "reconnect".
> > # With "shutdown", the broker will be restarted.
> > # With "reconnect", the broker will keep serving the topics, while
> > attempting to recreate a new session.
> > zookeeperSessionExpiredPolicy=shutdown
> > ```
> >
> > Change its default value to `reconnect`.
> >
> >
> > --
> > Matteo Merli
> > <mm...@apache.org>
> >
>
> +1
>
> Please pay attention to the following scenarios:
> I tried to use `Reconnect` in the production environment, and there was a
> serious online failure: two Brokers kept opening and closing the Ledger of
> the same topic, and the Ledger of BK kept throwing fencing exceptions.
>
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by Lin Lin <li...@apache.org>.
On 2021/12/14 18:03:20 Matteo Merli wrote:
> https://github.com/apache/pulsar/issues/13304
>
>
> Pasted below for quoting convenience.
>
> -------
>
>
> ## Motivation
>
> After all the work done for PIP-45 that was already included in 2.8 and 2.9
> releases, it enabled the concept of re-acquirable resource locks and leader
> election.
>
> Another important change was to avoid doing any deferrable metadata operation
> when we know that we are not currently connected to the metadata service.
>
> Finally, that enabled stabilization in 2.9 the configuration setting that allows
> brokers to continue operating in a safe mode when the session with ZooKeeper
> expires.
>
> The way it works is that, when we lose a ZooKeeper session, the data plane will
> continue to work undisturbed, relying on the BookKeeper fencing to avoid any
> inconsistencies.
>
> New topics are not able to get started, but existing topics will see no
> impact.
>
> The original intention for shutting down the brokers was to ensure that we
> would automatically go back to a consistent state, with respect to which
> resources are "owned" in ZooKeeper by a given broker.
>
> With the re-acquirable resource locks, that problem was solved and thoroughly
> tested to be robust.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # There are two policies to apply when a broker metadata session
> expires: session expired happens, "shutdown" or "reconnect".
> # With "shutdown", the broker will be restarted.
> # With "reconnect", the broker will keep serving the topics, while
> attempting to recreate a new session.
> zookeeperSessionExpiredPolicy=shutdown
> ```
>
> Change its default value to `reconnect`.
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>
+1
Please pay attention to the following scenarios:
I tried to use `Reconnect` in the production environment, and there was a serious online failure: two Brokers kept opening and closing the Ledger of the same topic, and the Ledger of BK kept throwing fencing exceptions.
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by Matteo Merli <ma...@gmail.com>.
On Wed, Dec 15, 2021 at 5:47 PM Haiting Jiang <ji...@apache.org> wrote:
> Small question about new parameter "zookeeperSessionExpiredPolicy", now we have Etcd implementation for metadata store, is it better to use "metadataStoreExpiredPolicy" ?
Good point, we should also use this occasion to deprecate the
`zookeeperSessionExpiredPolicy` key (and moving it to the end of the
config files, in the deprecated section) and add the setting.
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by Haiting Jiang <ji...@apache.org>.
+1
Great PIP. Long expected.
Small question about new parameter "zookeeperSessionExpiredPolicy", now we have Etcd implementation for metadata store, is it better to use "metadataStoreExpiredPolicy" ?
On 2021/12/14 18:03:20 Matteo Merli wrote:
> https://github.com/apache/pulsar/issues/13304
>
>
> Pasted below for quoting convenience.
>
> -------
>
>
> ## Motivation
>
> After all the work done for PIP-45 that was already included in 2.8 and 2.9
> releases, it enabled the concept of re-acquirable resource locks and leader
> election.
>
> Another important change was to avoid doing any deferrable metadata operation
> when we know that we are not currently connected to the metadata service.
>
> Finally, that enabled stabilization in 2.9 the configuration setting that allows
> brokers to continue operating in a safe mode when the session with ZooKeeper
> expires.
>
> The way it works is that, when we lose a ZooKeeper session, the data plane will
> continue to work undisturbed, relying on the BookKeeper fencing to avoid any
> inconsistencies.
>
> New topics are not able to get started, but existing topics will see no
> impact.
>
> The original intention for shutting down the brokers was to ensure that we
> would automatically go back to a consistent state, with respect to which
> resources are "owned" in ZooKeeper by a given broker.
>
> With the re-acquirable resource locks, that problem was solved and thoroughly
> tested to be robust.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # There are two policies to apply when a broker metadata session
> expires: session expired happens, "shutdown" or "reconnect".
> # With "shutdown", the broker will be restarted.
> # With "reconnect", the broker will keep serving the topics, while
> attempting to recreate a new session.
> zookeeperSessionExpiredPolicy=shutdown
> ```
>
> Change its default value to `reconnect`.
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by Enrico Olivelli <eo...@gmail.com>.
+1
Enrico
Il Ven 17 Dic 2021, 05:36 Sijie Guo <gu...@gmail.com> ha scritto:
> +1
>
> On Tue, Dec 14, 2021 at 10:03 AM Matteo Merli <mm...@apache.org> wrote:
>
> > https://github.com/apache/pulsar/issues/13304
> >
> >
> > Pasted below for quoting convenience.
> >
> > -------
> >
> >
> > ## Motivation
> >
> > After all the work done for PIP-45 that was already included in 2.8 and
> 2.9
> > releases, it enabled the concept of re-acquirable resource locks and
> leader
> > election.
> >
> > Another important change was to avoid doing any deferrable metadata
> > operation
> > when we know that we are not currently connected to the metadata service.
> >
> > Finally, that enabled stabilization in 2.9 the configuration setting that
> > allows
> > brokers to continue operating in a safe mode when the session with
> > ZooKeeper
> > expires.
> >
> > The way it works is that, when we lose a ZooKeeper session, the data
> plane
> > will
> > continue to work undisturbed, relying on the BookKeeper fencing to avoid
> > any
> > inconsistencies.
> >
> > New topics are not able to get started, but existing topics will see no
> > impact.
> >
> > The original intention for shutting down the brokers was to ensure that
> we
> > would automatically go back to a consistent state, with respect to which
> > resources are "owned" in ZooKeeper by a given broker.
> >
> > With the re-acquirable resource locks, that problem was solved and
> > thoroughly
> > tested to be robust.
> >
> > ## Proposed changes
> >
> > In 2.10 release, for the setting:
> >
> > ```properties
> > # There are two policies to apply when a broker metadata session
> > expires: session expired happens, "shutdown" or "reconnect".
> > # With "shutdown", the broker will be restarted.
> > # With "reconnect", the broker will keep serving the topics, while
> > attempting to recreate a new session.
> > zookeeperSessionExpiredPolicy=shutdown
> > ```
> >
> > Change its default value to `reconnect`.
> >
> >
> > --
> > Matteo Merli
> > <mm...@apache.org>
> >
>
Re: [DISCUSSION] PIP-118: Do not restart brokers when ZooKeeper session expires
Posted by Sijie Guo <gu...@gmail.com>.
+1
On Tue, Dec 14, 2021 at 10:03 AM Matteo Merli <mm...@apache.org> wrote:
> https://github.com/apache/pulsar/issues/13304
>
>
> Pasted below for quoting convenience.
>
> -------
>
>
> ## Motivation
>
> After all the work done for PIP-45 that was already included in 2.8 and 2.9
> releases, it enabled the concept of re-acquirable resource locks and leader
> election.
>
> Another important change was to avoid doing any deferrable metadata
> operation
> when we know that we are not currently connected to the metadata service.
>
> Finally, that enabled stabilization in 2.9 the configuration setting that
> allows
> brokers to continue operating in a safe mode when the session with
> ZooKeeper
> expires.
>
> The way it works is that, when we lose a ZooKeeper session, the data plane
> will
> continue to work undisturbed, relying on the BookKeeper fencing to avoid
> any
> inconsistencies.
>
> New topics are not able to get started, but existing topics will see no
> impact.
>
> The original intention for shutting down the brokers was to ensure that we
> would automatically go back to a consistent state, with respect to which
> resources are "owned" in ZooKeeper by a given broker.
>
> With the re-acquirable resource locks, that problem was solved and
> thoroughly
> tested to be robust.
>
> ## Proposed changes
>
> In 2.10 release, for the setting:
>
> ```properties
> # There are two policies to apply when a broker metadata session
> expires: session expired happens, "shutdown" or "reconnect".
> # With "shutdown", the broker will be restarted.
> # With "reconnect", the broker will keep serving the topics, while
> attempting to recreate a new session.
> zookeeperSessionExpiredPolicy=shutdown
> ```
>
> Change its default value to `reconnect`.
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>