Upgrade questions

[Fernando Vega <fe...@turn.com>]

I have a few questions regarding an upgrade that Im attempting to perform.

- Currently we are running version 0.8.2_beta.
- We are trying to upgrade to 10.1.1
- Our setup uses the following path
   /server/kafka
   where kafka is a symlink to kafka-{version}
- I attempt to perform the upgrade as you guys specify in the documentation:
http://kafka.apache.org/0100/documentation.html#upgrade

However when I attempt this my server.properties file that was in
kafka-0.8.2_beta was gone and a new file was under kafka-0.10.1.1.

I started the server without noticing this at first and it started normally.
When I realize about it I stop it and copy the server.properties file from
another host, running the old version into this server in the new directory
and adding the 2 lines mentioned in the instructions above as well as
changing its broker id of course, when I do this the server does not start.

So my questions are:

Are any of the config files different from one version to another?
Does the options from one version to another are the same?

This is a sample of the file that the old cluster uses and I will like to
know if this options are still valid for the new version:

## See http://kafka.apache.org/documentation.html#brokerconfigs for default
values.

# The id of the broker. This must be set to a unique integer for each
broker.
broker.id=6

# The port the socket server listens on
port=9092

# A comma seperated list of directories under which to store log files
log.dirs=/data/1/kafka/datalog,/data/2/kafka/datalog,/data/3/kafka/datalog,/data/4/kafka/datalog,/data/5/kafka/datalog,/data/6/kafka/datalog,/data/7/kafka/datalog,/data/8/kafka/datalog

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.

zookeeper.connect=dwh-pipeline001:2181,dwh-pipeline002:2181,dwh-pipeline003:2181,dwh-pipeline004:2181,dwh-pipeline005:2181/kafka/data

# Additional configuration options may follow here
auto.leader.rebalance.enable=true
delete.topic.enable=true
log.retention.size=20000000000
socket.receive.buffer.bytes=1048576
socket.send.buffer.bytes=1048576
default.replication.factor=2
auto.create.topics.enable=true
num.partitions=100
num.network.threads=3
num.io.threads=40
log.retention.hours=24
log.roll.hours=1
num.replica.fetchers=8
zookeeper.connection.timeout.ms=30000
zookeeper.session.timeout.ms=30000

Also What are the steps or ways I can check the update work fine and that
Im running x or y version or to determine things are working fine

Thank you

[image: Turn] <http://www.turn.com/>

*Fernando Vega*
Sr. Operations Engineer
*cell* (415) 810-0242
901 Marshall St, Suite 200, Redwood City, CA 94063


turn.com <http://www.turn.com/>   |   @TurnPlatform
<ht...@TurnPlatform>

This message is Turn Confidential, except for information included that is
already available to the public. If this message was sent to you
accidentally, please delete it.

Re: Upgrade questions

[Ewen Cheslack-Postava <ew...@confluent.io>]

Note that the documentation that you linked to for upgrades specifically
lists configs that you need to be careful to adjust in your
server.properties.

In fact, the server.properties shipped with Kafka is meant for testing
only. There are some configs in the example server.properties that are not
safe to use in production, for example log.dirs, which I see you've
adjusted.

That said, we are careful about not changing default values in the actual
service without warning. If we change them, it would be during major
version changes and we would still warn you about it in the release notes.
You are making a big jump (0.8.2-beta to 0.10.x) so you'd want to check the
0.8.2, 0.9, and 0.10 release notes. The most obvious thing I'd expect to be
an issue would be the log format changes and inter protocol version
settings. These are settings you need to be careful about setting correctly
during upgrades and are clearly described in the docs you already linked:
http://kafka.apache.org/0100/documentation.html#upgrade

-Ewen

On Thu, Jan 26, 2017 at 3:22 PM, Fernando Vega <fe...@turn.com>
wrote:

> I have a few questions regarding an upgrade that Im attempting to perform.
>
> - Currently we are running version 0.8.2_beta.
> - We are trying to upgrade to 10.1.1
> - Our setup uses the following path
>    /server/kafka
>    where kafka is a symlink to kafka-{version}
> - I attempt to perform the upgrade as you guys specify in the
> documentation:
> http://kafka.apache.org/0100/documentation.html#upgrade
>
> However when I attempt this my server.properties file that was in
> kafka-0.8.2_beta was gone and a new file was under kafka-0.10.1.1.
>
> I started the server without noticing this at first and it started
> normally.
> When I realize about it I stop it and copy the server.properties file from
> another host, running the old version into this server in the new directory
> and adding the 2 lines mentioned in the instructions above as well as
> changing its broker id of course, when I do this the server does not start.
>
> So my questions are:
>
> Are any of the config files different from one version to another?
> Does the options from one version to another are the same?
>
> This is a sample of the file that the old cluster uses and I will like to
> know if this options are still valid for the new version:
>
> ## See http://kafka.apache.org/documentation.html#brokerconfigs for
> default
> values.
>
> # The id of the broker. This must be set to a unique integer for each
> broker.
> broker.id=6
>
> # The port the socket server listens on
> port=9092
>
> # A comma seperated list of directories under which to store log files
> log.dirs=/data/1/kafka/datalog,/data/2/kafka/datalog,/data/
> 3/kafka/datalog,/data/4/kafka/datalog,/data/5/kafka/datalog,
> /data/6/kafka/datalog,/data/7/kafka/datalog,/data/8/kafka/datalog
>
> # Zookeeper connection string (see zookeeper docs for details).
> # This is a comma separated host:port pairs, each corresponding to a zk
> # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
> # You can also append an optional chroot string to the urls to specify the
> # root directory for all kafka znodes.
>
> zookeeper.connect=dwh-pipeline001:2181,dwh-pipeline002:2181,
> dwh-pipeline003:2181,dwh-pipeline004:2181,dwh-pipeline005:2181/kafka/data
>
> # Additional configuration options may follow here
> auto.leader.rebalance.enable=true
> delete.topic.enable=true
> log.retention.size=20000000000
> socket.receive.buffer.bytes=1048576
> socket.send.buffer.bytes=1048576
> default.replication.factor=2
> auto.create.topics.enable=true
> num.partitions=100
> num.network.threads=3
> num.io.threads=40
> log.retention.hours=24
> log.roll.hours=1
> num.replica.fetchers=8
> zookeeper.connection.timeout.ms=30000
> zookeeper.session.timeout.ms=30000
>
> Also What are the steps or ways I can check the update work fine and that
> Im running x or y version or to determine things are working fine
>
> Thank you
>
> [image: Turn] <http://www.turn.com/>
>
> *Fernando Vega*
> Sr. Operations Engineer
> *cell* (415) 810-0242
> 901 Marshall St, Suite 200, Redwood City, CA 94063
>
>
> turn.com <http://www.turn.com/>   |   @TurnPlatform
> <ht...@TurnPlatform>
>
> This message is Turn Confidential, except for information included that is
> already available to the public. If this message was sent to you
> accidentally, please delete it.
>