Message Routing between several locations

[andi welchlin <an...@gmail.com>]

Hello all,

I am new to Apache Kafka but used Apache Qpid, before.

I have a question regarding the broker setup:

Is it possible to run several brokers in a Kafka cluster while they are not
all running on the same zookeeper?

I want to reach the following target:

Several organizations are running one Kafka broker each. These brokers are
connected to the Kafka brokers in the cloud. The brokers in the cloud are
running on one zookeeper and are maintained by one organization. But all
the other connected organizations run their own broker in their own
environment.

With QPID brokers this would be possible by defining routes between these
brokers.

Can someone tell me if this is possible with Apache Kafka?

Kind Regards,
Andreas

Re: Message Routing between several locations

[andi welchlin <an...@gmail.com>]

Hello Jakub,

nice to also see you on this list :-)

Thank you for your answer.

Kind Regards,
Andreas


On Wed, Nov 22, 2017 at 4:18 PM, Jakub Scholz <ja...@scholz.cz> wrote:

> Hi Andreas,
>
> What you are describing is basically one central cluster (managed by the
> central org.) and a set of "satelite" clusters (which are managed by
> different org. each). These will never form a single cluster and you cannot
> use features such as replication to mirror the messages.
>
> But you might be able to use a tool such as MirrorMaker (
> http://kafka.apache.org/documentation/#basic_ops_mirror_maker /
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330)
> to link the "satelite" clusters with the main central cluster and forward
> the messages. How good will such solution be probably depends on your use
> cases and topic configuration. For example if you just publish to the
> satelite clusters and want to get these data mirrored to the central
> cluster it should work fine. The same if you have topic from which you just
> read. But if you try to do a full synchronization (i.e. all satelites
> publishing to the same topic but also consuming from it) it might be a bit
> more tricky I guess.
>
> Regards
> Jakub
>
> On Wed, Nov 22, 2017 at 3:49 PM, andi welchlin <an...@gmail.com>
> wrote:
>
> > Hello all,
> >
> > I am new to Apache Kafka but used Apache Qpid, before.
> >
> > I have a question regarding the broker setup:
> >
> > Is it possible to run several brokers in a Kafka cluster while they are
> not
> > all running on the same zookeeper?
> >
> > I want to reach the following target:
> >
> > Several organizations are running one Kafka broker each. These brokers
> are
> > connected to the Kafka brokers in the cloud. The brokers in the cloud are
> > running on one zookeeper and are maintained by one organization. But all
> > the other connected organizations run their own broker in their own
> > environment.
> >
> > With QPID brokers this would be possible by defining routes between these
> > brokers.
> >
> > Can someone tell me if this is possible with Apache Kafka?
> >
> > Kind Regards,
> > Andreas
> >
>

Re: Message Routing between several locations

[Jakub Scholz <ja...@scholz.cz>]

Hi Andreas,

What you are describing is basically one central cluster (managed by the
central org.) and a set of "satelite" clusters (which are managed by
different org. each). These will never form a single cluster and you cannot
use features such as replication to mirror the messages.

But you might be able to use a tool such as MirrorMaker (
http://kafka.apache.org/documentation/#basic_ops_mirror_maker /
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330)
to link the "satelite" clusters with the main central cluster and forward
the messages. How good will such solution be probably depends on your use
cases and topic configuration. For example if you just publish to the
satelite clusters and want to get these data mirrored to the central
cluster it should work fine. The same if you have topic from which you just
read. But if you try to do a full synchronization (i.e. all satelites
publishing to the same topic but also consuming from it) it might be a bit
more tricky I guess.

Regards
Jakub

On Wed, Nov 22, 2017 at 3:49 PM, andi welchlin <an...@gmail.com>
wrote:

> Hello all,
>
> I am new to Apache Kafka but used Apache Qpid, before.
>
> I have a question regarding the broker setup:
>
> Is it possible to run several brokers in a Kafka cluster while they are not
> all running on the same zookeeper?
>
> I want to reach the following target:
>
> Several organizations are running one Kafka broker each. These brokers are
> connected to the Kafka brokers in the cloud. The brokers in the cloud are
> running on one zookeeper and are maintained by one organization. But all
> the other connected organizations run their own broker in their own
> environment.
>
> With QPID brokers this would be possible by defining routes between these
> brokers.
>
> Can someone tell me if this is possible with Apache Kafka?
>
> Kind Regards,
> Andreas
>