mirroring - can we mirror a broker directly

[xiaoyu wang <xi...@gmail.com>]

Hello,

In the mirroring setup, the consumer configuration takes only zookeeper
connection string. I am wondering whether it can point directly to a
broker.

Here is what I am trying to do.

1. run my application server and a kafka broker on the same machine. My
application sends logs to the kafka broker
2. have an agggregate kafka cluster which collects logs from the kafka
brokers on all my servers through mirroring.

Zookeeper is not useful on my application servers since each has only one
broker running. But I have to run it for mirroring to work. Anyway to avoid
running zookeeper on my application server?

Thanks,

-Xiaoyu

Re: mirroring - can we mirror a broker directly

[Joel Koshy <jj...@gmail.com>]

The mirroring process (both the embedded consumer and the mirror-maker
tool) needs a zookeeper connection string. It should be possible to write a
mirroring tool that talks to a broker, but you would most likely then need
to have an alternate mechanism to do some load-balancing across the
mirror's consumers. In that case, you could have something as simple as
rsync'ing your partitions from each source broker to designated brokers in
your mirror cluster (I guess for that to work you would need a consistent
mapping). The easiest approach of course would be to have all your
application servers talk to a common Kafka cluster that would be an
aggregate, but you probably have a good reason to keep them separate.

Thanks,

Joel


> In the mirroring setup, the consumer configuration takes only zookeeper
> connection string. I am wondering whether it can point directly to a
> broker.


> Here is what I am trying to do.
>
> 1. run my application server and a kafka broker on the same machine. My
> application sends logs to the kafka broker
> 2. have an agggregate kafka cluster which collects logs from the kafka
> brokers on all my servers through mirroring.
>
> Zookeeper is not useful on my application servers since each has only one
> broker running. But I have to run it for mirroring to work. Anyway to avoid
> running zookeeper on my application server?
>
> Thanks,
>
> -Xiaoyu
>