Upgrade by replacing brokers?

[Andrew Otto <ot...@wikimedia.org>]

​Hi all,

We’re planning a big upgrade from 0.9.0.1 to 0.11.  As part of this
upgrade, we’ll be replacing the all the hardware in the cluster.

We are considering doing this as follows:  One by one, we’d shut down an
original broker machine, and then start up a broker​ on a new node with the
same broker.id as the one just shutdown.  I believe that this would cause
the newly started broker to replicate all of its partitions from the other
brokers.  If needed, we could also copy over the original broker log dirs
before starting up the new broker.

Would this work as I expect it to?  Is there anything I’m missing?  Are
there gotchas related with on disk log file formats that might cause some
issues?

Thanks!
- Andrew Otto
  Systems Engineer, Wikimedia Foundation

Re: Upgrade by replacing brokers?

[Matt Andruff <ma...@gmail.com>]

I would not mess with internals like this.  It's safer if you treat this
like a special case of a Broker expansion.

Don't forget if you are going to have mixed lineage brokers:

*From the documentation*

   1. Update server.properties file on all brokers and add the following
   properties:
      - inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 0.8.2 or
      0.9.0.0).
      - log.message.format.version=CURRENT_KAFKA_VERSION (See potential
      performance impact following the upgrade
      <https://kafka.apache.org/0100/documentation.html#upgrade_10_performance_impact>
for
      the details on what this configuration does.)
   2. Upgrade the brokers. This can be done a broker at a time by simply
   bringing it down, updating the code, and restarting it.
   3. Once the entire cluster is upgraded, bump the protocol version by
   editing inter.broker.protocol.version and setting it to 0.11.0.0. NOTE: You
   shouldn't touch log.message.format.version yet - this parameter should only
   change once all consumers have been upgraded to 0.11.0.0
   4. Restart the brokers one by one for the new protocol version to take
   effect.
   5. Once all consumers have been upgraded to 0.11.0, change
   log.message.format.version to 0.11.0 on each broker and restart them one by
   one.

Ok now that we've discussed that, what you want to do is a special type of
expansion.  Where you essentially create a partition plan that removes the
old broker from the reassignment strategy.  (
https://kafka.apache.org/documentation/#basic_ops_cluster_expansion) but
before you run off and do it the hard way consider using
https://github.com/linkedin/kafka-tools/wiki/Kafka-Assigner which helps
with features like removing a broker.

Hope this helps.

Matt



On Wed, 26 Jul 2017 at 08:19 Andrew Otto <ot...@wikimedia.org> wrote:

> ​Hi all,
>
> We’re planning a big upgrade from 0.9.0.1 to 0.11.  As part of this
> upgrade, we’ll be replacing the all the hardware in the cluster.
>
> We are considering doing this as follows:  One by one, we’d shut down an
> original broker machine, and then start up a broker​ on a new node with the
> same broker.id as the one just shutdown.  I believe that this would cause
> the newly started broker to replicate all of its partitions from the other
> brokers.  If needed, we could also copy over the original broker log dirs
> before starting up the new broker.
>
> Would this work as I expect it to?  Is there anything I’m missing?  Are
> there gotchas related with on disk log file formats that might cause some
> issues?
>
> Thanks!
> - Andrew Otto
>   Systems Engineer, Wikimedia Foundation
>