Migrating both Zookeeper ensemble and Kafka cluster please confirm steps

[John Yost <ho...@gmail.com>]

Hi Everyone,

I need to migrate my organization's Kafka cluster along with the underlying
Zookeeper ensemble (cluster) from one set of racks to another within our
data center. I am pretty sure I have the steps correct, but I need to
confirm just to ensure I am not missing anything.

Here's what I think I need to do in the following order:

1. Configure new Zookeeper ensemble
2. Configure brokers for new Kafka cluster, with zookeeper.connect list of
new Zookeeper ensemble nodes
3. Startup new Zookeeper ensemble
4. Startup new Kafka cluster
5. Shutdown data feeds to old Kafka cluster and reconfigure for new cluster
6. Migrate topics from old Kafka brokers to new brokers via
kafka-reassign-partitions.sh script and the instructions here:
http://kafka.apache.org/082/documentation.html#basic_ops_cluster_expansion
7. Export consumer offsets from old Zookeeper ensemble and import into new
Zookeeper ensemble
8. Shutdown old Kafka cluster
9. Restart data feeds into new Kafka cluster

The Kafka documentation is great and I've tested out the topic reassignment
and consumer offset import and export, but, again, just want to ensure I am
not missing anything.

Thanks

--John

Re: Migrating both Zookeeper ensemble and Kafka cluster please confirm steps

[John Yost <ho...@gmail.com>]

Hi Tulio,

Thanks a bunch for responding to my question! We basically need to migrate
data from brokers hosted on racks being decommisioned, but the new and old
brokers are actually within the same cluster. The new brokers are
configured to use the new Zookeeper ensemble, but, again, they are intended
to be brokers within the same cluster. Once the data is moved to the new
brokers, the old brokers will be deleted, or at least that's what I am
intending to do.

Please confirm if my approach makes sense or if there is a problem and/or a
better way to do it.

Thanks

--John

On Mon, Nov 2, 2015 at 2:20 PM, Tulio Ballari <tu...@gmail.com>
wrote:

> I'm not really sure but I think kafka-reassing-partitions.sh only works to
> move partitions between instanses of the same cluster
>
> If you have two separete clusters, it is not going to work. Are you using
> two different zookeeper clusters, or simply adding instance to the existing
> one?
>
>
>
> On Sun, Nov 1, 2015 at 1:18 PM, John Yost <ho...@gmail.com> wrote:
>
> > Hi Everyone,
> >
> > I need to migrate my organization's Kafka cluster along with the
> underlying
> > Zookeeper ensemble (cluster) from one set of racks to another within our
> > data center. I am pretty sure I have the steps correct, but I need to
> > confirm just to ensure I am not missing anything.
> >
> > Here's what I think I need to do in the following order:
> >
> > 1. Configure new Zookeeper ensemble
> > 2. Configure brokers for new Kafka cluster, with zookeeper.connect list
> of
> > new Zookeeper ensemble nodes
> > 3. Startup new Zookeeper ensemble
> > 4. Startup new Kafka cluster
> > 5. Shutdown data feeds to old Kafka cluster and reconfigure for new
> cluster
> > 6. Migrate topics from old Kafka brokers to new brokers via
> > kafka-reassign-partitions.sh script and the instructions here:
> >
> http://kafka.apache.org/082/documentation.html#basic_ops_cluster_expansion
> > 7. Export consumer offsets from old Zookeeper ensemble and import into
> new
> > Zookeeper ensemble
> > 8. Shutdown old Kafka cluster
> > 9. Restart data feeds into new Kafka cluster
> >
> > The Kafka documentation is great and I've tested out the topic
> reassignment
> > and consumer offset import and export, but, again, just want to ensure I
> am
> > not missing anything.
> >
> > Thanks
> >
> > --John
> >
>

Re: Migrating both Zookeeper ensemble and Kafka cluster please confirm steps

[Tulio Ballari <tu...@gmail.com>]

I'm not really sure but I think kafka-reassing-partitions.sh only works to
move partitions between instanses of the same cluster

If you have two separete clusters, it is not going to work. Are you using
two different zookeeper clusters, or simply adding instance to the existing
one?



On Sun, Nov 1, 2015 at 1:18 PM, John Yost <ho...@gmail.com> wrote:

> Hi Everyone,
>
> I need to migrate my organization's Kafka cluster along with the underlying
> Zookeeper ensemble (cluster) from one set of racks to another within our
> data center. I am pretty sure I have the steps correct, but I need to
> confirm just to ensure I am not missing anything.
>
> Here's what I think I need to do in the following order:
>
> 1. Configure new Zookeeper ensemble
> 2. Configure brokers for new Kafka cluster, with zookeeper.connect list of
> new Zookeeper ensemble nodes
> 3. Startup new Zookeeper ensemble
> 4. Startup new Kafka cluster
> 5. Shutdown data feeds to old Kafka cluster and reconfigure for new cluster
> 6. Migrate topics from old Kafka brokers to new brokers via
> kafka-reassign-partitions.sh script and the instructions here:
> http://kafka.apache.org/082/documentation.html#basic_ops_cluster_expansion
> 7. Export consumer offsets from old Zookeeper ensemble and import into new
> Zookeeper ensemble
> 8. Shutdown old Kafka cluster
> 9. Restart data feeds into new Kafka cluster
>
> The Kafka documentation is great and I've tested out the topic reassignment
> and consumer offset import and export, but, again, just want to ensure I am
> not missing anything.
>
> Thanks
>
> --John
>