You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "kparisa (via GitHub)" <gi...@apache.org> on 2026/07/28 06:36:06 UTC

[I] feat(connectors): Iggy Source Connector: Native Cluster-to-Cluster Replication [iggy]

kparisa opened a new issue, #3764:
URL: https://github.com/apache/iggy/issues/3764

   ### Description
   
   **Summary**
   
   Implement an Iggy Source Connector that consumes data from an upstream Iggy cluster and publishes it into a downstream Iggy cluster.
   
   This enables seamless cluster-to-cluster replication and provides the foundation for multi-region deployments, disaster recovery, active-active architectures, edge-to-cloud streaming, and hierarchical streaming topologies.
   
   Unlike traditional application-level replication, this connector allows operators to replicate selected streams, partitions, or namespaces without requiring producers or consumers to change.
   
   **Motivation**
   
   Many production deployments require data to exist in multiple locations.
   
   Today, applications would need to publish to multiple Iggy clusters directly, increasing complexity and coupling replication logic into business applications.
   
   A native Iggy Source Connector would make replication a first-class capability by continuously consuming from one cluster and publishing to another while preserving ordering and delivery guarantees.
   
                Producers
                    │
                    ▼
           +------------------+
           |  Iggy Cluster A  |
           |   (Primary)      |
           +------------------+
                    │
         Iggy Source Connector
                    │
                    ▼
           +------------------+
           |  Iggy Cluster B  |
           |  (Replica)       |
           +------------------+
                    │
             Local Consumers
   
   The connector should behave like any other Iggy connector:
   
   * Source = upstream Iggy
   * Sink = downstream Iggy
   
   ### Affected area / component
   
   Connectors
   
   ### Proposed solution
   
   _No response_
   
   ### Alternatives considered
   
   _No response_
   
   ### Contribution
   
   - [ ] I'm willing to submit a pull request to implement this feature
   
   ### Good first issue
   
   - [x] I think this could be a good first issue for a new contributor


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "jiengup (via GitHub)" <gi...@apache.org>.
jiengup commented on issue #3764:
URL: https://github.com/apache/iggy/issues/3764#issuecomment-5302393307

   @slbotbm @kparisa 
   
   I pulled a source connector implementation. Please have a look.
   Now the source connector is a partition-agnostic design, with per-partition offsets persisted in the connector state. And it doesn't rely on server-side offset preservation. 
   
   What do you think about it? I will continue to push forward and publish a sink implementation if it makes sense.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "raman20 (via GitHub)" <gi...@apache.org>.
raman20 commented on issue #3764:
URL: https://github.com/apache/iggy/issues/3764#issuecomment-5191059718

   please remove me from this issue....


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on issue #3764:
URL: https://github.com/apache/iggy/issues/3764#issuecomment-6773803491

   the connectors moved to [apache/iggy-connectors](https://github.com/apache/iggy-connectors), and this issue moved with them: apache/iggy-connectors#31. please continue there.
   
   @jiengup: github can't carry your assignment over, because it only lets us assign people who commented on an issue. please leave a short comment on the new issue, so a maintainer can assign you there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "jiengup (via GitHub)" <gi...@apache.org>.
jiengup commented on issue #3764:
URL: https://github.com/apache/iggy/issues/3764#issuecomment-5199810348

   Then I want to take this.
   I'm getting familiar with the ASR cluster implementation these days.
   And I'm interested in this proposal, I will track the following development in this direction.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "raman20 (via GitHub)" <gi...@apache.org>.
raman20 commented on issue #3764:
URL: https://github.com/apache/iggy/issues/3764#issuecomment-5149013127

   iggy_sink pr is out.... 
   releasing iggy_source pr shortly...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on issue #3764:
URL: https://github.com/apache/iggy/issues/3764#issuecomment-5112110632

   @raman20 - When you raise PR, please raise separate PR for Sink and source. Please ping me on discord, I want to share a work in progress blog article for developing connectors for Iggy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio closed issue #3764: feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication
URL: https://github.com/apache/iggy/issues/3764


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "slbotbm (via GitHub)" <gi...@apache.org>.
slbotbm commented on issue #3764:
URL: https://github.com/apache/iggy/issues/3764#issuecomment-5202238868

   @jiengup assigned


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "raman20 (via GitHub)" <gi...@apache.org>.
raman20 commented on issue #3764:
URL: https://github.com/apache/iggy/issues/3764#issuecomment-5100886021

   hey @kparisa i would like to work on this...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] feat(connectors): Iggy Source Connector: Native Cluster-to-Cluster Replication [iggy]

Posted by "kparisa (via GitHub)" <gi...@apache.org>.
kparisa commented on issue #3764:
URL: https://github.com/apache/iggy/issues/3764#issuecomment-5100932277

   @raman20 Thank you! Assigned to you.
   This will be a great feature!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org