You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Garmes Amine <am...@yahoo.fr.INVALID> on 2021/06/11 13:03:50 UTC
[idea] Kafka topic metadata
Hello Apache Kafka community,
In the driection to make Kafka more cloud friendly, do you think it make sense to have key-velue metadata in the topic definition,
this will allow to search, filter, select ... topics not only by name but also by metadata
Ex:
topic name: test
topic metadata:
- namespace: space1 (will allow to have multi tenant Kafka cluster)
- project: test
- phase: dev
- type: json
- area: logging
Best regards,
Garmes
Re: [idea] Kafka topic metadata
Posted by Ivan Yurchenko <iv...@gmail.com>.
Hi Garmes,
I think it's worth writing a separate email for this as described in
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-WhoshouldinitiatetheKIP?
(2nd paragraph).
Best,
Ivan
On Tue, 15 Jun 2021 at 12:04, Garmes Amine <am...@yahoo.fr.invalid>
wrote:
> Hello Israel, Ivan
>
> thank you for your feedback.
>
> Can I have the rights to create the KIP page please.
>
> Bests
> Garmes
>
> > On 14 Jun 2021, at 19:27, Israel Ekpo <is...@gmail.com> wrote:
> >
> > Sorry Ivan and Garmes,
> >
> > I misunderstood the suggestion earlier. I think this will be a great idea
> > for a KIP.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > You were referring to metadata for the actual topic and not its contents.
> >
> > Sorry about that confusion.
> >
> >
> >
> > On Mon, Jun 14, 2021 at 1:05 PM Ivan Yurchenko <ivan0yurchenko@gmail.com
> >
> > wrote:
> >
> >> Hi,
> >>
> >> Having metadata for topics seems pretty useful. Currently, one has to
> use
> >> external storage for this (e.g. a database) and the question of keeping
> >> topic and metadata in sync exists: A topic is deleted, how to delete its
> >> metadata? How to deal with delete-then-recreate scenarios (well, we have
> >> topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.
> >>
> >> Ivan
> >>
> >>
> >> On Mon, 14 Jun 2021 at 19:55, Israel Ekpo <is...@gmail.com> wrote:
> >>
> >>> Garmes,
> >>>
> >>> I had similar questions in the past but @Matthias J. Sax
> >>> <mj...@apache.org> pointed
> >>> me to this
> >>>
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> >>>
> >>> With the headers, you can filter based on the header content and not
> just
> >>> the contents of the topic.
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine
> >> <aminegarmes@yahoo.fr.invalid
> >>>>
> >>> wrote:
> >>>
> >>>> Hello Apache Kafka community,
> >>>>
> >>>> In the driection to make Kafka more cloud friendly, do you think it
> >> make
> >>>> sense to have key-velue metadata in the topic definition,
> >>>> this will allow to search, filter, select ... topics not only by name
> >> but
> >>>> also by metadata
> >>>>
> >>>> Ex:
> >>>> topic name: test
> >>>> topic metadata:
> >>>> - namespace: space1 (will allow to have multi tenant Kafka cluster)
> >>>> - project: test
> >>>> - phase: dev
> >>>> - type: json
> >>>> - area: logging
> >>>>
> >>>>
> >>>> Best regards,
> >>>> Garmes
> >>>
> >>
>
>
Re: [idea] Kafka topic metadata
Posted by Ivan Yurchenko <iv...@gmail.com>.
Hi Garmes,
I think it's worth writing a separate email for this as described in
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-WhoshouldinitiatetheKIP?
(2nd paragraph).
Best,
Ivan
On Tue, 15 Jun 2021 at 12:04, Garmes Amine <am...@yahoo.fr.invalid>
wrote:
> Hello Israel, Ivan
>
> thank you for your feedback.
>
> Can I have the rights to create the KIP page please.
>
> Bests
> Garmes
>
> > On 14 Jun 2021, at 19:27, Israel Ekpo <is...@gmail.com> wrote:
> >
> > Sorry Ivan and Garmes,
> >
> > I misunderstood the suggestion earlier. I think this will be a great idea
> > for a KIP.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > You were referring to metadata for the actual topic and not its contents.
> >
> > Sorry about that confusion.
> >
> >
> >
> > On Mon, Jun 14, 2021 at 1:05 PM Ivan Yurchenko <ivan0yurchenko@gmail.com
> >
> > wrote:
> >
> >> Hi,
> >>
> >> Having metadata for topics seems pretty useful. Currently, one has to
> use
> >> external storage for this (e.g. a database) and the question of keeping
> >> topic and metadata in sync exists: A topic is deleted, how to delete its
> >> metadata? How to deal with delete-then-recreate scenarios (well, we have
> >> topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.
> >>
> >> Ivan
> >>
> >>
> >> On Mon, 14 Jun 2021 at 19:55, Israel Ekpo <is...@gmail.com> wrote:
> >>
> >>> Garmes,
> >>>
> >>> I had similar questions in the past but @Matthias J. Sax
> >>> <mj...@apache.org> pointed
> >>> me to this
> >>>
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> >>>
> >>> With the headers, you can filter based on the header content and not
> just
> >>> the contents of the topic.
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine
> >> <aminegarmes@yahoo.fr.invalid
> >>>>
> >>> wrote:
> >>>
> >>>> Hello Apache Kafka community,
> >>>>
> >>>> In the driection to make Kafka more cloud friendly, do you think it
> >> make
> >>>> sense to have key-velue metadata in the topic definition,
> >>>> this will allow to search, filter, select ... topics not only by name
> >> but
> >>>> also by metadata
> >>>>
> >>>> Ex:
> >>>> topic name: test
> >>>> topic metadata:
> >>>> - namespace: space1 (will allow to have multi tenant Kafka cluster)
> >>>> - project: test
> >>>> - phase: dev
> >>>> - type: json
> >>>> - area: logging
> >>>>
> >>>>
> >>>> Best regards,
> >>>> Garmes
> >>>
> >>
>
>
Re: [idea] Kafka topic metadata
Posted by Garmes Amine <am...@yahoo.fr.INVALID>.
Hello Israel, Ivan
thank you for your feedback.
Can I have the rights to create the KIP page please.
Bests
Garmes
> On 14 Jun 2021, at 19:27, Israel Ekpo <is...@gmail.com> wrote:
>
> Sorry Ivan and Garmes,
>
> I misunderstood the suggestion earlier. I think this will be a great idea
> for a KIP.
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> You were referring to metadata for the actual topic and not its contents.
>
> Sorry about that confusion.
>
>
>
> On Mon, Jun 14, 2021 at 1:05 PM Ivan Yurchenko <iv...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Having metadata for topics seems pretty useful. Currently, one has to use
>> external storage for this (e.g. a database) and the question of keeping
>> topic and metadata in sync exists: A topic is deleted, how to delete its
>> metadata? How to deal with delete-then-recreate scenarios (well, we have
>> topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.
>>
>> Ivan
>>
>>
>> On Mon, 14 Jun 2021 at 19:55, Israel Ekpo <is...@gmail.com> wrote:
>>
>>> Garmes,
>>>
>>> I had similar questions in the past but @Matthias J. Sax
>>> <mj...@apache.org> pointed
>>> me to this
>>>
>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
>>>
>>> With the headers, you can filter based on the header content and not just
>>> the contents of the topic.
>>>
>>>
>>>
>>>
>>> On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine
>> <aminegarmes@yahoo.fr.invalid
>>>>
>>> wrote:
>>>
>>>> Hello Apache Kafka community,
>>>>
>>>> In the driection to make Kafka more cloud friendly, do you think it
>> make
>>>> sense to have key-velue metadata in the topic definition,
>>>> this will allow to search, filter, select ... topics not only by name
>> but
>>>> also by metadata
>>>>
>>>> Ex:
>>>> topic name: test
>>>> topic metadata:
>>>> - namespace: space1 (will allow to have multi tenant Kafka cluster)
>>>> - project: test
>>>> - phase: dev
>>>> - type: json
>>>> - area: logging
>>>>
>>>>
>>>> Best regards,
>>>> Garmes
>>>
>>
Re: [idea] Kafka topic metadata
Posted by Garmes Amine <am...@yahoo.fr.INVALID>.
Hello Israel, Ivan
thank you for your feedback.
Can I have the rights to create the KIP page please.
Bests
Garmes
> On 14 Jun 2021, at 19:27, Israel Ekpo <is...@gmail.com> wrote:
>
> Sorry Ivan and Garmes,
>
> I misunderstood the suggestion earlier. I think this will be a great idea
> for a KIP.
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> You were referring to metadata for the actual topic and not its contents.
>
> Sorry about that confusion.
>
>
>
> On Mon, Jun 14, 2021 at 1:05 PM Ivan Yurchenko <iv...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Having metadata for topics seems pretty useful. Currently, one has to use
>> external storage for this (e.g. a database) and the question of keeping
>> topic and metadata in sync exists: A topic is deleted, how to delete its
>> metadata? How to deal with delete-then-recreate scenarios (well, we have
>> topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.
>>
>> Ivan
>>
>>
>> On Mon, 14 Jun 2021 at 19:55, Israel Ekpo <is...@gmail.com> wrote:
>>
>>> Garmes,
>>>
>>> I had similar questions in the past but @Matthias J. Sax
>>> <mj...@apache.org> pointed
>>> me to this
>>>
>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
>>>
>>> With the headers, you can filter based on the header content and not just
>>> the contents of the topic.
>>>
>>>
>>>
>>>
>>> On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine
>> <aminegarmes@yahoo.fr.invalid
>>>>
>>> wrote:
>>>
>>>> Hello Apache Kafka community,
>>>>
>>>> In the driection to make Kafka more cloud friendly, do you think it
>> make
>>>> sense to have key-velue metadata in the topic definition,
>>>> this will allow to search, filter, select ... topics not only by name
>> but
>>>> also by metadata
>>>>
>>>> Ex:
>>>> topic name: test
>>>> topic metadata:
>>>> - namespace: space1 (will allow to have multi tenant Kafka cluster)
>>>> - project: test
>>>> - phase: dev
>>>> - type: json
>>>> - area: logging
>>>>
>>>>
>>>> Best regards,
>>>> Garmes
>>>
>>
Re: [idea] Kafka topic metadata
Posted by Israel Ekpo <is...@gmail.com>.
Sorry Ivan and Garmes,
I misunderstood the suggestion earlier. I think this will be a great idea
for a KIP.
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
You were referring to metadata for the actual topic and not its contents.
Sorry about that confusion.
On Mon, Jun 14, 2021 at 1:05 PM Ivan Yurchenko <iv...@gmail.com>
wrote:
> Hi,
>
> Having metadata for topics seems pretty useful. Currently, one has to use
> external storage for this (e.g. a database) and the question of keeping
> topic and metadata in sync exists: A topic is deleted, how to delete its
> metadata? How to deal with delete-then-recreate scenarios (well, we have
> topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.
>
> Ivan
>
>
> On Mon, 14 Jun 2021 at 19:55, Israel Ekpo <is...@gmail.com> wrote:
>
> > Garmes,
> >
> > I had similar questions in the past but @Matthias J. Sax
> > <mj...@apache.org> pointed
> > me to this
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> >
> > With the headers, you can filter based on the header content and not just
> > the contents of the topic.
> >
> >
> >
> >
> > On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine
> <aminegarmes@yahoo.fr.invalid
> > >
> > wrote:
> >
> > > Hello Apache Kafka community,
> > >
> > > In the driection to make Kafka more cloud friendly, do you think it
> make
> > > sense to have key-velue metadata in the topic definition,
> > > this will allow to search, filter, select ... topics not only by name
> but
> > > also by metadata
> > >
> > > Ex:
> > > topic name: test
> > > topic metadata:
> > > - namespace: space1 (will allow to have multi tenant Kafka cluster)
> > > - project: test
> > > - phase: dev
> > > - type: json
> > > - area: logging
> > >
> > >
> > > Best regards,
> > > Garmes
> >
>
Re: [idea] Kafka topic metadata
Posted by Israel Ekpo <is...@gmail.com>.
Sorry Ivan and Garmes,
I misunderstood the suggestion earlier. I think this will be a great idea
for a KIP.
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
You were referring to metadata for the actual topic and not its contents.
Sorry about that confusion.
On Mon, Jun 14, 2021 at 1:05 PM Ivan Yurchenko <iv...@gmail.com>
wrote:
> Hi,
>
> Having metadata for topics seems pretty useful. Currently, one has to use
> external storage for this (e.g. a database) and the question of keeping
> topic and metadata in sync exists: A topic is deleted, how to delete its
> metadata? How to deal with delete-then-recreate scenarios (well, we have
> topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.
>
> Ivan
>
>
> On Mon, 14 Jun 2021 at 19:55, Israel Ekpo <is...@gmail.com> wrote:
>
> > Garmes,
> >
> > I had similar questions in the past but @Matthias J. Sax
> > <mj...@apache.org> pointed
> > me to this
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> >
> > With the headers, you can filter based on the header content and not just
> > the contents of the topic.
> >
> >
> >
> >
> > On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine
> <aminegarmes@yahoo.fr.invalid
> > >
> > wrote:
> >
> > > Hello Apache Kafka community,
> > >
> > > In the driection to make Kafka more cloud friendly, do you think it
> make
> > > sense to have key-velue metadata in the topic definition,
> > > this will allow to search, filter, select ... topics not only by name
> but
> > > also by metadata
> > >
> > > Ex:
> > > topic name: test
> > > topic metadata:
> > > - namespace: space1 (will allow to have multi tenant Kafka cluster)
> > > - project: test
> > > - phase: dev
> > > - type: json
> > > - area: logging
> > >
> > >
> > > Best regards,
> > > Garmes
> >
>
Re: [idea] Kafka topic metadata
Posted by Ivan Yurchenko <iv...@gmail.com>.
Hi,
Having metadata for topics seems pretty useful. Currently, one has to use
external storage for this (e.g. a database) and the question of keeping
topic and metadata in sync exists: A topic is deleted, how to delete its
metadata? How to deal with delete-then-recreate scenarios (well, we have
topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.
Ivan
On Mon, 14 Jun 2021 at 19:55, Israel Ekpo <is...@gmail.com> wrote:
> Garmes,
>
> I had similar questions in the past but @Matthias J. Sax
> <mj...@apache.org> pointed
> me to this
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
>
> With the headers, you can filter based on the header content and not just
> the contents of the topic.
>
>
>
>
> On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine <aminegarmes@yahoo.fr.invalid
> >
> wrote:
>
> > Hello Apache Kafka community,
> >
> > In the driection to make Kafka more cloud friendly, do you think it make
> > sense to have key-velue metadata in the topic definition,
> > this will allow to search, filter, select ... topics not only by name but
> > also by metadata
> >
> > Ex:
> > topic name: test
> > topic metadata:
> > - namespace: space1 (will allow to have multi tenant Kafka cluster)
> > - project: test
> > - phase: dev
> > - type: json
> > - area: logging
> >
> >
> > Best regards,
> > Garmes
>
Re: [idea] Kafka topic metadata
Posted by Ivan Yurchenko <iv...@gmail.com>.
Hi,
Having metadata for topics seems pretty useful. Currently, one has to use
external storage for this (e.g. a database) and the question of keeping
topic and metadata in sync exists: A topic is deleted, how to delete its
metadata? How to deal with delete-then-recreate scenarios (well, we have
topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.
Ivan
On Mon, 14 Jun 2021 at 19:55, Israel Ekpo <is...@gmail.com> wrote:
> Garmes,
>
> I had similar questions in the past but @Matthias J. Sax
> <mj...@apache.org> pointed
> me to this
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
>
> With the headers, you can filter based on the header content and not just
> the contents of the topic.
>
>
>
>
> On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine <aminegarmes@yahoo.fr.invalid
> >
> wrote:
>
> > Hello Apache Kafka community,
> >
> > In the driection to make Kafka more cloud friendly, do you think it make
> > sense to have key-velue metadata in the topic definition,
> > this will allow to search, filter, select ... topics not only by name but
> > also by metadata
> >
> > Ex:
> > topic name: test
> > topic metadata:
> > - namespace: space1 (will allow to have multi tenant Kafka cluster)
> > - project: test
> > - phase: dev
> > - type: json
> > - area: logging
> >
> >
> > Best regards,
> > Garmes
>
Re: [idea] Kafka topic metadata
Posted by Israel Ekpo <is...@gmail.com>.
Garmes,
I had similar questions in the past but @Matthias J. Sax
<mj...@apache.org> pointed
me to this
https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
With the headers, you can filter based on the header content and not just
the contents of the topic.
On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine <am...@yahoo.fr.invalid>
wrote:
> Hello Apache Kafka community,
>
> In the driection to make Kafka more cloud friendly, do you think it make
> sense to have key-velue metadata in the topic definition,
> this will allow to search, filter, select ... topics not only by name but
> also by metadata
>
> Ex:
> topic name: test
> topic metadata:
> - namespace: space1 (will allow to have multi tenant Kafka cluster)
> - project: test
> - phase: dev
> - type: json
> - area: logging
>
>
> Best regards,
> Garmes
Re: [idea] Kafka topic metadata
Posted by Israel Ekpo <is...@gmail.com>.
Garmes,
I had similar questions in the past but @Matthias J. Sax
<mj...@apache.org> pointed
me to this
https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
With the headers, you can filter based on the header content and not just
the contents of the topic.
On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine <am...@yahoo.fr.invalid>
wrote:
> Hello Apache Kafka community,
>
> In the driection to make Kafka more cloud friendly, do you think it make
> sense to have key-velue metadata in the topic definition,
> this will allow to search, filter, select ... topics not only by name but
> also by metadata
>
> Ex:
> topic name: test
> topic metadata:
> - namespace: space1 (will allow to have multi tenant Kafka cluster)
> - project: test
> - phase: dev
> - type: json
> - area: logging
>
>
> Best regards,
> Garmes