You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Enrico Olivelli <eo...@gmail.com> on 2021/09/12 09:24:36 UTC
Topic metadata
Hello,
I would like to have the ability to store metadata about topics.
This would be very useful as with metadata you could add labels and other
pieces of information that would allow to define the purpose of a topic,
custom application level properties.
This feature will allow application level diagnostic tools and maintenance
tools to not need external databases to store such metadata.
I imagine that we could add a simple key value map (String keys and String
values) to the topic.
These metadata could be set during topic creation and also updated.
I would store these metadata on the Metadata service (zookkeeper).
Is there any interest or any ongoing effort in this direction?
Best regards
Enrico
Re: Topic metadata
Posted by Enrico Olivelli <eo...@gmail.com>.
Joe,
Il giorno gio 23 set 2021 alle ore 04:22 PengHui Li <pe...@apache.org> ha
scritto:
> Another option is to store the custom properties of the topic to the system
> topic as the topic policy does.
> The limitation is if using the system topic, we are not able to guarantee
> the atomic operation for the topic creation and properties adding,
> But it is required in some cases.
>
> Yes, the managed ledger already supports store custom properties, looks
> like we need to add new properties support for the partitioned topic?
> I think it does not make sense to copy the properties to each partition.
>
> Penghui
>
> On Thu, Sep 23, 2021 at 6:44 AM Joe F <jo...@gmail.com> wrote:
>
> > I don't think pulsar should increase its dependency on ZK. Can we store
> it
> > somewhere else?
>
IIUC As we already have such support in ML we are not adding more
dependencies on ZK from a code perspective.
Regarding the impact on ZK I believe that such metadata will usually be
like 1 or 2 entries per topic, so it should not add a significant load.
Enrico
> >
> > On Wed, Sep 22, 2021, 3:30 PM Matteo Merli <ma...@gmail.com>
> wrote:
> >
> > > We already have a mechanism to store custom properties in managed
> > > ledgers. So we don't need to store it separately.
> > > --
> > > Matteo Merli
> > > <ma...@gmail.com>
> > >
> > > On Wed, Sep 22, 2021 at 3:21 PM Michael Marshall <
> mikemarsh17@gmail.com>
> > > wrote:
> > > >
> > > > Hi Enrico,
> > > >
> > > > This sounds like a useful feature. Do you expect to use it internally
> > in
> > > > pulsar or should it be limited to external applications?
> > > >
> > > > In thinking about authorization, what level of permission should be
> > > > required to create/update/delete these labels? It seems to me that
> > these
> > > > should require some kind of admin permission.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > On Sun, Sep 19, 2021 at 7:21 PM PengHui Li <pe...@apache.org>
> wrote:
> > > >
> > > > > Hi enrico
> > > > >
> > > > > +1, sorry I missed the email before. It should be a useful feature
> > for
> > > > > Pulsar users and I have a discussion with the Flink Pulsar
> connector
> > > > > authors several days ago, they also looking for this feature.
> > > > >
> > > > > Thanks
> > > > > Penghui
> > > > >
> > > > > Enrico Olivelli <eo...@gmail.com>于2021年9月12日 周日17:24写道：
> > > > >
> > > > > > Hello,
> > > > > > I would like to have the ability to store metadata about topics.
> > > > > >
> > > > > > This would be very useful as with metadata you could add labels
> and
> > > other
> > > > > > pieces of information that would allow to define the purpose of a
> > > topic,
> > > > > > custom application level properties.
> > > > > > This feature will allow application level diagnostic tools and
> > > > > maintenance
> > > > > > tools to not need external databases to store such metadata.
> > > > > >
> > > > > > I imagine that we could add a simple key value map (String keys
> and
> > > > > String
> > > > > > values) to the topic.
> > > > > > These metadata could be set during topic creation and also
> updated.
> > > > > >
> > > > > > I would store these metadata on the Metadata service
> (zookkeeper).
> > > > > >
> > > > > > Is there any interest or any ongoing effort in this direction?
> > > > > >
> > > > > >
> > > > > > Best regards
> > > > > > Enrico
> > > > > >
> > > > >
> > >
> >
>
Re: Topic metadata
Posted by PengHui Li <pe...@apache.org>.
Another option is to store the custom properties of the topic to the system
topic as the topic policy does.
The limitation is if using the system topic, we are not able to guarantee
the atomic operation for the topic creation and properties adding,
But it is required in some cases.
Yes, the managed ledger already supports store custom properties, looks
like we need to add new properties support for the partitioned topic?
I think it does not make sense to copy the properties to each partition.
Penghui
On Thu, Sep 23, 2021 at 6:44 AM Joe F <jo...@gmail.com> wrote:
> I don't think pulsar should increase its dependency on ZK. Can we store it
> somewhere else?
>
> On Wed, Sep 22, 2021, 3:30 PM Matteo Merli <ma...@gmail.com> wrote:
>
> > We already have a mechanism to store custom properties in managed
> > ledgers. So we don't need to store it separately.
> > --
> > Matteo Merli
> > <ma...@gmail.com>
> >
> > On Wed, Sep 22, 2021 at 3:21 PM Michael Marshall <mi...@gmail.com>
> > wrote:
> > >
> > > Hi Enrico,
> > >
> > > This sounds like a useful feature. Do you expect to use it internally
> in
> > > pulsar or should it be limited to external applications?
> > >
> > > In thinking about authorization, what level of permission should be
> > > required to create/update/delete these labels? It seems to me that
> these
> > > should require some kind of admin permission.
> > >
> > > Thanks,
> > > Michael
> > >
> > > On Sun, Sep 19, 2021 at 7:21 PM PengHui Li <pe...@apache.org> wrote:
> > >
> > > > Hi enrico
> > > >
> > > > +1, sorry I missed the email before. It should be a useful feature
> for
> > > > Pulsar users and I have a discussion with the Flink Pulsar connector
> > > > authors several days ago, they also looking for this feature.
> > > >
> > > > Thanks
> > > > Penghui
> > > >
> > > > Enrico Olivelli <eo...@gmail.com>于2021年9月12日 周日17:24写道：
> > > >
> > > > > Hello,
> > > > > I would like to have the ability to store metadata about topics.
> > > > >
> > > > > This would be very useful as with metadata you could add labels and
> > other
> > > > > pieces of information that would allow to define the purpose of a
> > topic,
> > > > > custom application level properties.
> > > > > This feature will allow application level diagnostic tools and
> > > > maintenance
> > > > > tools to not need external databases to store such metadata.
> > > > >
> > > > > I imagine that we could add a simple key value map (String keys and
> > > > String
> > > > > values) to the topic.
> > > > > These metadata could be set during topic creation and also updated.
> > > > >
> > > > > I would store these metadata on the Metadata service (zookkeeper).
> > > > >
> > > > > Is there any interest or any ongoing effort in this direction?
> > > > >
> > > > >
> > > > > Best regards
> > > > > Enrico
> > > > >
> > > >
> >
>
Re: Topic metadata
Posted by Joe F <jo...@gmail.com>.
I don't think pulsar should increase its dependency on ZK. Can we store it
somewhere else?
On Wed, Sep 22, 2021, 3:30 PM Matteo Merli <ma...@gmail.com> wrote:
> We already have a mechanism to store custom properties in managed
> ledgers. So we don't need to store it separately.
> --
> Matteo Merli
> <ma...@gmail.com>
>
> On Wed, Sep 22, 2021 at 3:21 PM Michael Marshall <mi...@gmail.com>
> wrote:
> >
> > Hi Enrico,
> >
> > This sounds like a useful feature. Do you expect to use it internally in
> > pulsar or should it be limited to external applications?
> >
> > In thinking about authorization, what level of permission should be
> > required to create/update/delete these labels? It seems to me that these
> > should require some kind of admin permission.
> >
> > Thanks,
> > Michael
> >
> > On Sun, Sep 19, 2021 at 7:21 PM PengHui Li <pe...@apache.org> wrote:
> >
> > > Hi enrico
> > >
> > > +1, sorry I missed the email before. It should be a useful feature for
> > > Pulsar users and I have a discussion with the Flink Pulsar connector
> > > authors several days ago, they also looking for this feature.
> > >
> > > Thanks
> > > Penghui
> > >
> > > Enrico Olivelli <eo...@gmail.com>于2021年9月12日 周日17:24写道：
> > >
> > > > Hello,
> > > > I would like to have the ability to store metadata about topics.
> > > >
> > > > This would be very useful as with metadata you could add labels and
> other
> > > > pieces of information that would allow to define the purpose of a
> topic,
> > > > custom application level properties.
> > > > This feature will allow application level diagnostic tools and
> > > maintenance
> > > > tools to not need external databases to store such metadata.
> > > >
> > > > I imagine that we could add a simple key value map (String keys and
> > > String
> > > > values) to the topic.
> > > > These metadata could be set during topic creation and also updated.
> > > >
> > > > I would store these metadata on the Metadata service (zookkeeper).
> > > >
> > > > Is there any interest or any ongoing effort in this direction?
> > > >
> > > >
> > > > Best regards
> > > > Enrico
> > > >
> > >
>
Re: Topic metadata
Posted by Matteo Merli <ma...@gmail.com>.
We already have a mechanism to store custom properties in managed
ledgers. So we don't need to store it separately.
--
Matteo Merli
<ma...@gmail.com>
On Wed, Sep 22, 2021 at 3:21 PM Michael Marshall <mi...@gmail.com> wrote:
>
> Hi Enrico,
>
> This sounds like a useful feature. Do you expect to use it internally in
> pulsar or should it be limited to external applications?
>
> In thinking about authorization, what level of permission should be
> required to create/update/delete these labels? It seems to me that these
> should require some kind of admin permission.
>
> Thanks,
> Michael
>
> On Sun, Sep 19, 2021 at 7:21 PM PengHui Li <pe...@apache.org> wrote:
>
> > Hi enrico
> >
> > +1, sorry I missed the email before. It should be a useful feature for
> > Pulsar users and I have a discussion with the Flink Pulsar connector
> > authors several days ago, they also looking for this feature.
> >
> > Thanks
> > Penghui
> >
> > Enrico Olivelli <eo...@gmail.com>于2021年9月12日 周日17:24写道：
> >
> > > Hello,
> > > I would like to have the ability to store metadata about topics.
> > >
> > > This would be very useful as with metadata you could add labels and other
> > > pieces of information that would allow to define the purpose of a topic,
> > > custom application level properties.
> > > This feature will allow application level diagnostic tools and
> > maintenance
> > > tools to not need external databases to store such metadata.
> > >
> > > I imagine that we could add a simple key value map (String keys and
> > String
> > > values) to the topic.
> > > These metadata could be set during topic creation and also updated.
> > >
> > > I would store these metadata on the Metadata service (zookkeeper).
> > >
> > > Is there any interest or any ongoing effort in this direction?
> > >
> > >
> > > Best regards
> > > Enrico
> > >
> >
Re: Topic metadata
Posted by Michael Marshall <mi...@gmail.com>.
Hi Enrico,
This sounds like a useful feature. Do you expect to use it internally in
pulsar or should it be limited to external applications?
In thinking about authorization, what level of permission should be
required to create/update/delete these labels? It seems to me that these
should require some kind of admin permission.
Thanks,
Michael
On Sun, Sep 19, 2021 at 7:21 PM PengHui Li <pe...@apache.org> wrote:
> Hi enrico
>
> +1, sorry I missed the email before. It should be a useful feature for
> Pulsar users and I have a discussion with the Flink Pulsar connector
> authors several days ago, they also looking for this feature.
>
> Thanks
> Penghui
>
> Enrico Olivelli <eo...@gmail.com>于2021年9月12日 周日17:24写道：
>
> > Hello,
> > I would like to have the ability to store metadata about topics.
> >
> > This would be very useful as with metadata you could add labels and other
> > pieces of information that would allow to define the purpose of a topic,
> > custom application level properties.
> > This feature will allow application level diagnostic tools and
> maintenance
> > tools to not need external databases to store such metadata.
> >
> > I imagine that we could add a simple key value map (String keys and
> String
> > values) to the topic.
> > These metadata could be set during topic creation and also updated.
> >
> > I would store these metadata on the Metadata service (zookkeeper).
> >
> > Is there any interest or any ongoing effort in this direction?
> >
> >
> > Best regards
> > Enrico
> >
>
Re: Topic metadata
Posted by PengHui Li <pe...@apache.org>.
Hi enrico
+1, sorry I missed the email before. It should be a useful feature for
Pulsar users and I have a discussion with the Flink Pulsar connector
authors several days ago, they also looking for this feature.
Thanks
Penghui
Enrico Olivelli <eo...@gmail.com>于2021年9月12日 周日17:24写道：
> Hello,
> I would like to have the ability to store metadata about topics.
>
> This would be very useful as with metadata you could add labels and other
> pieces of information that would allow to define the purpose of a topic,
> custom application level properties.
> This feature will allow application level diagnostic tools and maintenance
> tools to not need external databases to store such metadata.
>
> I imagine that we could add a simple key value map (String keys and String
> values) to the topic.
> These metadata could be set during topic creation and also updated.
>
> I would store these metadata on the Metadata service (zookkeeper).
>
> Is there any interest or any ongoing effort in this direction?
>
>
> Best regards
> Enrico
>