You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Matteo Merli <ma...@gmail.com> on 2022/01/14 22:51:57 UTC
[VOTE] PIP-135: Include MetadataStore backend for Etcd
https://github.com/apache/pulsar/issues/13717
-----
## Motivation
Since all the pieces that composed the proposal in PIP-45 were finally merged
and are currently ready for 2.10 release, it is now possible to add other
metadata backends that can be used to support a BookKeeper + Pulsar cluster.
One of the popular systems that is most commonly used as an alternative of
ZooKeeper is Etcd, thus it makes sense to have this as the first non-zookeeper
implementation.
## Goal
Provide an Etcd implementation for the `MetadataStore` API. This will allow
users to deploy Pulsar clusters using Etcd service for the metadata and it will
not require the presence of ZooKeeper.
## Implementation
* Use the existing JEtcd Java client library for Etcd
* Extends the `AbstractBatchedMetadataStore` class, in order to reuse the
transparent batching logic that will be shared with the ZooKeeper
implementation.
Work in progress: https://github.com/apache/pulsar/pull/13225
--
Matteo Merli
<ma...@gmail.com>
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
Posted by Enrico Olivelli <eo...@gmail.com>.
+1 (binding)
Thanks
Enrico
Il Ven 14 Gen 2022, 23:52 Matteo Merli <ma...@gmail.com> ha scritto:
> https://github.com/apache/pulsar/issues/13717
>
> -----
>
> ## Motivation
>
> Since all the pieces that composed the proposal in PIP-45 were finally
> merged
> and are currently ready for 2.10 release, it is now possible to add other
> metadata backends that can be used to support a BookKeeper + Pulsar
> cluster.
>
> One of the popular systems that is most commonly used as an alternative of
> ZooKeeper is Etcd, thus it makes sense to have this as the first
> non-zookeeper
> implementation.
>
> ## Goal
>
> Provide an Etcd implementation for the `MetadataStore` API. This will allow
> users to deploy Pulsar clusters using Etcd service for the metadata and it
> will
> not require the presence of ZooKeeper.
>
>
> ## Implementation
>
> * Use the existing JEtcd Java client library for Etcd
> * Extends the `AbstractBatchedMetadataStore` class, in order to reuse the
> transparent batching logic that will be shared with the ZooKeeper
> implementation.
>
> Work in progress: https://github.com/apache/pulsar/pull/13225
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
Posted by Neng Lu <fr...@gmail.com>.
+1 (non-binding)
This is very interesting
On Mon, Jan 17, 2022 at 5:08 AM Lan Liang <li...@163.com> wrote:
> +1, Thanks for your work!
>
>
> - lan.liang
>
>
> On 01/17/2022 20:50，ZhangJian He<sh...@gmail.com> wrote：
> +1
>
> PengHui Li <pe...@apache.org> 于2022年1月17日周一 14:01写道：
>
> +1 (binding)
>
> Regards,
> Penghui
>
> On Sat, Jan 15, 2022 at 9:24 PM Joe F <jo...@gmail.com> wrote:
>
> +1 (binding)
>
> On Sat, Jan 15, 2022 at 4:46 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> Il Sab 15 Gen 2022, 09:10 tamer Abdlatif <ta...@gmail.com> ha
> scritto:
>
> Will that affect the existing ZK metadata in a pulsar instance , When
> we
> upgrade from a lower version to 2.10 ? In other words,
> Do we need a metadate migration to switch from ZK to Etcd ?
>
>
> There is no need to migrate.
>
> Most probably the first release will bring this feature as non
> production
> ready and it will take some time to stabilise
>
> Enrico
>
>
>
>
> Thanks
> Tamer
>
>
>
> On Fri, 14 Jan 2022, 22:52 Matteo Merli, <ma...@gmail.com>
> wrote:
>
> https://github.com/apache/pulsar/issues/13717
>
> -----
>
> ## Motivation
>
> Since all the pieces that composed the proposal in PIP-45 were
> finally
> merged
> and are currently ready for 2.10 release, it is now possible to add
> other
> metadata backends that can be used to support a BookKeeper + Pulsar
> cluster.
>
> One of the popular systems that is most commonly used as an
> alternative
> of
> ZooKeeper is Etcd, thus it makes sense to have this as the first
> non-zookeeper
> implementation.
>
> ## Goal
>
> Provide an Etcd implementation for the `MetadataStore` API. This
> will
> allow
> users to deploy Pulsar clusters using Etcd service for the metadata
> and
> it
> will
> not require the presence of ZooKeeper.
>
>
> ## Implementation
>
> * Use the existing JEtcd Java client library for Etcd
> * Extends the `AbstractBatchedMetadataStore` class, in order to
> reuse
> the
> transparent batching logic that will be shared with the
> ZooKeeper
> implementation.
>
> Work in progress: https://github.com/apache/pulsar/pull/13225
>
> --
> Matteo Merli
> <ma...@gmail.com>
>
>
>
>
>
>
--
Best Regards,
Neng
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
Posted by Lan Liang <li...@163.com>.
+1, Thanks for your work!
- lan.liang
On 01/17/2022 20:50，ZhangJian He<sh...@gmail.com> wrote：
+1
PengHui Li <pe...@apache.org> 于2022年1月17日周一 14:01写道：
+1 (binding)
Regards,
Penghui
On Sat, Jan 15, 2022 at 9:24 PM Joe F <jo...@gmail.com> wrote:
+1 (binding)
On Sat, Jan 15, 2022 at 4:46 AM Enrico Olivelli <eo...@gmail.com>
wrote:
Il Sab 15 Gen 2022, 09:10 tamer Abdlatif <ta...@gmail.com> ha
scritto:
Will that affect the existing ZK metadata in a pulsar instance , When
we
upgrade from a lower version to 2.10 ? In other words,
Do we need a metadate migration to switch from ZK to Etcd ?
There is no need to migrate.
Most probably the first release will bring this feature as non
production
ready and it will take some time to stabilise
Enrico
Thanks
Tamer
On Fri, 14 Jan 2022, 22:52 Matteo Merli, <ma...@gmail.com>
wrote:
https://github.com/apache/pulsar/issues/13717
-----
## Motivation
Since all the pieces that composed the proposal in PIP-45 were
finally
merged
and are currently ready for 2.10 release, it is now possible to add
other
metadata backends that can be used to support a BookKeeper + Pulsar
cluster.
One of the popular systems that is most commonly used as an
alternative
of
ZooKeeper is Etcd, thus it makes sense to have this as the first
non-zookeeper
implementation.
## Goal
Provide an Etcd implementation for the `MetadataStore` API. This
will
allow
users to deploy Pulsar clusters using Etcd service for the metadata
and
it
will
not require the presence of ZooKeeper.
## Implementation
* Use the existing JEtcd Java client library for Etcd
* Extends the `AbstractBatchedMetadataStore` class, in order to
reuse
the
transparent batching logic that will be shared with the
ZooKeeper
implementation.
Work in progress: https://github.com/apache/pulsar/pull/13225
--
Matteo Merli
<ma...@gmail.com>
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
Posted by ZhangJian He <sh...@gmail.com>.
+1
PengHui Li <pe...@apache.org> 于2022年1月17日周一 14:01写道：
> +1 (binding)
>
> Regards,
> Penghui
>
> On Sat, Jan 15, 2022 at 9:24 PM Joe F <jo...@gmail.com> wrote:
>
> > +1 (binding)
> >
> > On Sat, Jan 15, 2022 at 4:46 AM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Il Sab 15 Gen 2022, 09:10 tamer Abdlatif <ta...@gmail.com> ha
> > > scritto:
> > >
> > > > Will that affect the existing ZK metadata in a pulsar instance , When
> > we
> > > > upgrade from a lower version to 2.10 ? In other words,
> > > > Do we need a metadate migration to switch from ZK to Etcd ?
> > > >
> > >
> > > There is no need to migrate.
> > >
> > > Most probably the first release will bring this feature as non
> production
> > > ready and it will take some time to stabilise
> > >
> > > Enrico
> > >
> > >
> > >
> > > >
> > > > Thanks
> > > > Tamer
> > > >
> > > >
> > > >
> > > > On Fri, 14 Jan 2022, 22:52 Matteo Merli, <ma...@gmail.com>
> > wrote:
> > > >
> > > > > https://github.com/apache/pulsar/issues/13717
> > > > >
> > > > > -----
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > Since all the pieces that composed the proposal in PIP-45 were
> > finally
> > > > > merged
> > > > > and are currently ready for 2.10 release, it is now possible to add
> > > other
> > > > > metadata backends that can be used to support a BookKeeper + Pulsar
> > > > > cluster.
> > > > >
> > > > > One of the popular systems that is most commonly used as an
> > alternative
> > > > of
> > > > > ZooKeeper is Etcd, thus it makes sense to have this as the first
> > > > > non-zookeeper
> > > > > implementation.
> > > > >
> > > > > ## Goal
> > > > >
> > > > > Provide an Etcd implementation for the `MetadataStore` API. This
> will
> > > > allow
> > > > > users to deploy Pulsar clusters using Etcd service for the metadata
> > and
> > > > it
> > > > > will
> > > > > not require the presence of ZooKeeper.
> > > > >
> > > > >
> > > > > ## Implementation
> > > > >
> > > > > * Use the existing JEtcd Java client library for Etcd
> > > > > * Extends the `AbstractBatchedMetadataStore` class, in order to
> > reuse
> > > > the
> > > > > transparent batching logic that will be shared with the
> ZooKeeper
> > > > > implementation.
> > > > >
> > > > > Work in progress: https://github.com/apache/pulsar/pull/13225
> > > > >
> > > > > --
> > > > > Matteo Merli
> > > > > <ma...@gmail.com>
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
Posted by PengHui Li <pe...@apache.org>.
+1 (binding)
Regards,
Penghui
On Sat, Jan 15, 2022 at 9:24 PM Joe F <jo...@gmail.com> wrote:
> +1 (binding)
>
> On Sat, Jan 15, 2022 at 4:46 AM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Il Sab 15 Gen 2022, 09:10 tamer Abdlatif <ta...@gmail.com> ha
> > scritto:
> >
> > > Will that affect the existing ZK metadata in a pulsar instance , When
> we
> > > upgrade from a lower version to 2.10 ? In other words,
> > > Do we need a metadate migration to switch from ZK to Etcd ?
> > >
> >
> > There is no need to migrate.
> >
> > Most probably the first release will bring this feature as non production
> > ready and it will take some time to stabilise
> >
> > Enrico
> >
> >
> >
> > >
> > > Thanks
> > > Tamer
> > >
> > >
> > >
> > > On Fri, 14 Jan 2022, 22:52 Matteo Merli, <ma...@gmail.com>
> wrote:
> > >
> > > > https://github.com/apache/pulsar/issues/13717
> > > >
> > > > -----
> > > >
> > > > ## Motivation
> > > >
> > > > Since all the pieces that composed the proposal in PIP-45 were
> finally
> > > > merged
> > > > and are currently ready for 2.10 release, it is now possible to add
> > other
> > > > metadata backends that can be used to support a BookKeeper + Pulsar
> > > > cluster.
> > > >
> > > > One of the popular systems that is most commonly used as an
> alternative
> > > of
> > > > ZooKeeper is Etcd, thus it makes sense to have this as the first
> > > > non-zookeeper
> > > > implementation.
> > > >
> > > > ## Goal
> > > >
> > > > Provide an Etcd implementation for the `MetadataStore` API. This will
> > > allow
> > > > users to deploy Pulsar clusters using Etcd service for the metadata
> and
> > > it
> > > > will
> > > > not require the presence of ZooKeeper.
> > > >
> > > >
> > > > ## Implementation
> > > >
> > > > * Use the existing JEtcd Java client library for Etcd
> > > > * Extends the `AbstractBatchedMetadataStore` class, in order to
> reuse
> > > the
> > > > transparent batching logic that will be shared with the ZooKeeper
> > > > implementation.
> > > >
> > > > Work in progress: https://github.com/apache/pulsar/pull/13225
> > > >
> > > > --
> > > > Matteo Merli
> > > > <ma...@gmail.com>
> > > >
> > >
> >
>
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
Posted by Joe F <jo...@gmail.com>.
+1 (binding)
On Sat, Jan 15, 2022 at 4:46 AM Enrico Olivelli <eo...@gmail.com> wrote:
> Il Sab 15 Gen 2022, 09:10 tamer Abdlatif <ta...@gmail.com> ha
> scritto:
>
> > Will that affect the existing ZK metadata in a pulsar instance , When we
> > upgrade from a lower version to 2.10 ? In other words,
> > Do we need a metadate migration to switch from ZK to Etcd ?
> >
>
> There is no need to migrate.
>
> Most probably the first release will bring this feature as non production
> ready and it will take some time to stabilise
>
> Enrico
>
>
>
> >
> > Thanks
> > Tamer
> >
> >
> >
> > On Fri, 14 Jan 2022, 22:52 Matteo Merli, <ma...@gmail.com> wrote:
> >
> > > https://github.com/apache/pulsar/issues/13717
> > >
> > > -----
> > >
> > > ## Motivation
> > >
> > > Since all the pieces that composed the proposal in PIP-45 were finally
> > > merged
> > > and are currently ready for 2.10 release, it is now possible to add
> other
> > > metadata backends that can be used to support a BookKeeper + Pulsar
> > > cluster.
> > >
> > > One of the popular systems that is most commonly used as an alternative
> > of
> > > ZooKeeper is Etcd, thus it makes sense to have this as the first
> > > non-zookeeper
> > > implementation.
> > >
> > > ## Goal
> > >
> > > Provide an Etcd implementation for the `MetadataStore` API. This will
> > allow
> > > users to deploy Pulsar clusters using Etcd service for the metadata and
> > it
> > > will
> > > not require the presence of ZooKeeper.
> > >
> > >
> > > ## Implementation
> > >
> > > * Use the existing JEtcd Java client library for Etcd
> > > * Extends the `AbstractBatchedMetadataStore` class, in order to reuse
> > the
> > > transparent batching logic that will be shared with the ZooKeeper
> > > implementation.
> > >
> > > Work in progress: https://github.com/apache/pulsar/pull/13225
> > >
> > > --
> > > Matteo Merli
> > > <ma...@gmail.com>
> > >
> >
>
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
Posted by Enrico Olivelli <eo...@gmail.com>.
Il Sab 15 Gen 2022, 09:10 tamer Abdlatif <ta...@gmail.com> ha
scritto:
> Will that affect the existing ZK metadata in a pulsar instance , When we
> upgrade from a lower version to 2.10 ? In other words,
> Do we need a metadate migration to switch from ZK to Etcd ?
>
There is no need to migrate.
Most probably the first release will bring this feature as non production
ready and it will take some time to stabilise
Enrico
>
> Thanks
> Tamer
>
>
>
> On Fri, 14 Jan 2022, 22:52 Matteo Merli, <ma...@gmail.com> wrote:
>
> > https://github.com/apache/pulsar/issues/13717
> >
> > -----
> >
> > ## Motivation
> >
> > Since all the pieces that composed the proposal in PIP-45 were finally
> > merged
> > and are currently ready for 2.10 release, it is now possible to add other
> > metadata backends that can be used to support a BookKeeper + Pulsar
> > cluster.
> >
> > One of the popular systems that is most commonly used as an alternative
> of
> > ZooKeeper is Etcd, thus it makes sense to have this as the first
> > non-zookeeper
> > implementation.
> >
> > ## Goal
> >
> > Provide an Etcd implementation for the `MetadataStore` API. This will
> allow
> > users to deploy Pulsar clusters using Etcd service for the metadata and
> it
> > will
> > not require the presence of ZooKeeper.
> >
> >
> > ## Implementation
> >
> > * Use the existing JEtcd Java client library for Etcd
> > * Extends the `AbstractBatchedMetadataStore` class, in order to reuse
> the
> > transparent batching logic that will be shared with the ZooKeeper
> > implementation.
> >
> > Work in progress: https://github.com/apache/pulsar/pull/13225
> >
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
>
Re: [VOTE] PIP-135: Include MetadataStore backend for Etcd
Posted by tamer Abdlatif <ta...@gmail.com>.
Will that affect the existing ZK metadata in a pulsar instance , When we
upgrade from a lower version to 2.10 ? In other words,
Do we need a metadate migration to switch from ZK to Etcd ?
Thanks
Tamer
On Fri, 14 Jan 2022, 22:52 Matteo Merli, <ma...@gmail.com> wrote:
> https://github.com/apache/pulsar/issues/13717
>
> -----
>
> ## Motivation
>
> Since all the pieces that composed the proposal in PIP-45 were finally
> merged
> and are currently ready for 2.10 release, it is now possible to add other
> metadata backends that can be used to support a BookKeeper + Pulsar
> cluster.
>
> One of the popular systems that is most commonly used as an alternative of
> ZooKeeper is Etcd, thus it makes sense to have this as the first
> non-zookeeper
> implementation.
>
> ## Goal
>
> Provide an Etcd implementation for the `MetadataStore` API. This will allow
> users to deploy Pulsar clusters using Etcd service for the metadata and it
> will
> not require the presence of ZooKeeper.
>
>
> ## Implementation
>
> * Use the existing JEtcd Java client library for Etcd
> * Extends the `AbstractBatchedMetadataStore` class, in order to reuse the
> transparent batching logic that will be shared with the ZooKeeper
> implementation.
>
> Work in progress: https://github.com/apache/pulsar/pull/13225
>
> --
> Matteo Merli
> <ma...@gmail.com>
>