[GitHub] [pulsar] oversearch created a discussion: Question: copy or move topics between Pulsar clusters

[GitBox <gi...@apache.org>]

GitHub user oversearch created a discussion: Question: copy or move topics between Pulsar clusters

**Is your feature request related to a problem? Please describe.**
I'm wondering if it's currently possible to copy or move a topic that exists in one Pulsar cluster to another independent cluster?  

For example, say I have a topic with a long retention time (perhaps backed by tiered storage in S3) associated with some entity in my business logic that exists specifically in one AWS region in a multi-region setup, with a cluster per region.  Let's say that management of that entity needs to move to a different region, and thus I'd need to move that topic and its producers and consumers to another region to maintain locality.  

Is there an official / easy way to do this?

**Describe the solution you'd like**

Ideally there would be an admin command to safely copy a complete topic as-is from one cluster to another, with metadata and consumer positions preserved (assuming that an identical namespace existed in the destination cluster).  But a documented workflow would satisfy my needs.

**Describe alternatives you've considered**

I could of course simply open a reader to the source topic and a producer on the destination, and then stream the messages over.  This has two major shortcomings though: 1) Some of the metadata will presumably change, such as the message IDs and the "publish timestamp", which I don't think I could force to be preserved. 2) If a regex consumer on the destination region picks up the newly created topic shortly after it's created, it will presumably start receiving the copied messages as if they're brand new, which is undesirable.  While it's possible to manually expire messages with the admin API, that would require a great deal of coordination if the process is happening live, and I'm not sure that it could be guaranteed that none of the copied messages would be delivered to existing consumers on the destination.

I also considered the possibility that if the topic existed in its own namespace, I might be able to set up cross-region replication and have it replicated to the destination.  But I'm not sure if messages already marked "received" would be replicated in this situation?  The docs don't seem to be clear on this, but now that I'm re-reading them, the answer seems to be "no"...

Beyond that I was thinking about manipulating the Bookkeeper ledger files directly, but that seems to be getting way into the weeds...

Any advice you could provide would be helpful, thank you!


GitHub link: https://github.com/apache/pulsar/discussions/18461

----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org

[GitHub] [pulsar] codelipenghui added a comment to the discussion: Question: copy or move topics between Pulsar clusters

[GitBox <gi...@apache.org>]

GitHub user codelipenghui added a comment to the discussion: Question: copy or move topics between Pulsar clusters

The issue had no activity for 30 days, mark with Stale label.

GitHub link: https://github.com/apache/pulsar/discussions/18461#discussioncomment-4136189

----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org