[DISCUSS]

[Anton Agestam <an...@aiven.io.INVALID>]

Hello everyone,

I'm hoping to start a discussion to get some feedback on a topic, I assume
this is something that would require a KIP, but wanted to start out with a
discussion to gather some early feedback.

We have a use case internally for creating topics with a given topic ID.
The need for this arises because we're working on the capability to delete
a cluster and later recreate its tiered storage topics in an entirely new
cluster. For that to work, since the organization of the tiered data is
dependent on the topic ID, the topic created in the fresh cluster must have
the same ID as the original one in the deleted cluster.

We have a small patch only used for internal testing so far, showing that
adding TopicID as a tagged field on CreateTopicsRequest and implementing
the capability is a small and uncomplicated change:
https://github.com/aiven/kafka/pull/48/commits/fa5ebbca642391af630f1c7f7cc013fe56856e13
.

We also foresee that the same need will arise from tools that replicate
topics between clusters, such as mirrormaker2.

Are there any thoughts about this proposal, or any pointers for moving
forward formalizing this in a KIP?

Best regards,
Anton

-- 
[image: Aiven] <https://www.aiven.io/>
*Anton Agestam* (he/him or they/them)
Software Engineer, *Aiven*
anton.agestam@aiven.io   |   +46 704 486 289
aiven.io <https://www.aiven.io/>   |
<https://www.facebook.com/aivencloud>
<https://www.linkedin.com/company/aiven/>    <https://twitter.com/aiven_io>

Re: [DISCUSS]

[Justine Olshan <jo...@confluent.io.INVALID>]

Hey there,

Thanks for bringing this discussion to the mailing list. I think this is an
interesting one.

I am little concerned about the ability to add any topic ID to the create
topic request. It means that we can no longer rely on topic IDs being
globally unique. I worry folks may misunderstand and think they should be
creating their own IDs for topics and getting into sticky situations.

I wonder if there is a way to encapsulate what you want to do into an API
(ie some sort of formalized migration of a topic). This would make it
clearer that this is meant to be the "same" topic.
I also wonder if this is something that would be on the tiered storage
side. Is the use case combining two topics or recreating the topic and
using the same tiered storage? I'm curious about this use case and if the
topic is intended to be the "same" topic on the recreated cluster.

Thanks,
Justine

On Mon, Feb 26, 2024 at 5:39 AM Anton Agestam
<an...@aiven.io.invalid> wrote:

> Hello everyone,
>
> I'm hoping to start a discussion to get some feedback on a topic, I assume
> this is something that would require a KIP, but wanted to start out with a
> discussion to gather some early feedback.
>
> We have a use case internally for creating topics with a given topic ID.
> The need for this arises because we're working on the capability to delete
> a cluster and later recreate its tiered storage topics in an entirely new
> cluster. For that to work, since the organization of the tiered data is
> dependent on the topic ID, the topic created in the fresh cluster must have
> the same ID as the original one in the deleted cluster.
>
> We have a small patch only used for internal testing so far, showing that
> adding TopicID as a tagged field on CreateTopicsRequest and implementing
> the capability is a small and uncomplicated change:
>
> https://github.com/aiven/kafka/pull/48/commits/fa5ebbca642391af630f1c7f7cc013fe56856e13
> .
>
> We also foresee that the same need will arise from tools that replicate
> topics between clusters, such as mirrormaker2.
>
> Are there any thoughts about this proposal, or any pointers for moving
> forward formalizing this in a KIP?
>
> Best regards,
> Anton
>
> --
> [image: Aiven] <https://www.aiven.io/>
> *Anton Agestam* (he/him or they/them)
> Software Engineer, *Aiven*
> anton.agestam@aiven.io   |   +46 704 486 289
> aiven.io <https://www.aiven.io/>   |
> <https://www.facebook.com/aivencloud>
> <https://www.linkedin.com/company/aiven/>    <https://twitter.com/aiven_io
> >
>