You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by adrien ruffie <ad...@hotmail.fr> on 2018/02/20 20:45:48 UTC
broker properties explanations
Hello all,
after reading several properties in Kafka documentations, I asked mysleft some questions ...
these 2 following options are available:
log.dir The directory in which the log data is kept (supplemental for log.dirs property) string /tmp/kafka-logs high
log.dirs The directories in which the log data is kept. If not set, the value in log.dir is used
But in fact, if is the same thing, why only "log.dirs" isn't kept ? What's difference in usage ?
Also I noticed that the "data" of logs partition and also the "application" logs of Kafka are written in the same directory.
Is a mistake of my part ? Because it's very strange for me to log error, debug, warn application message in the same location of my data ...
After that I don't very understand why the log.flush.interval.messages have a so big default value ???
log.flush.interval.messages The number of messages accumulated on a log partition before messages are flushed to disk long 9223372036854775807
And the log.flush.interval.ms is by default null ... ?
It means that until there are so many messages (9223372036854775807) in my topics, they will not be flushed to disk ? It can be very long for a default value ....
Best regards,
Adrien
RE: broker properties explanations
Posted by adrien ruffie <ad...@hotmail.fr>.
It's really help me, thank Thomas !
Just a jot of another question,
I don't understand correctly this property:
leader.imbalance.per.broker.percentage The ratio of leader imbalance allowed per broker. The controller would trigger a leader balance if it goes above this value per broker. The value is specified in percentage. int 10
How the ratio is calculated ? The imbalance broker compare to what ? Moreover the 10% by default value is very small, no ?
Thank again Thomas,
Adrien
________________________________
De : Thomas Aley <Th...@ibm.com>
Envoyé : mercredi 21 février 2018 09:24:13
À : users@kafka.apache.org
Objet : Re: broker properties explanations
Hi Adrien,
log.dirs exists to facilitate multiple data directories which allows more
than one disk to be used without the need for RAID. This increases
throughput but beware of naive load balancing that may fill up one disk
way before another.
When log.flush.interval.ms is null the log.flush.interval.messages
property is not used. With default settings, messages are written to disk
immediately.
Hope this helps.
Tom Aley
thomas.aley@ibm.com
From: adrien ruffie <ad...@hotmail.fr>
To: "users@kafka.apache.org" <us...@kafka.apache.org>
Date: 20/02/2018 20:46
Subject: broker properties explanations
Hello all,
after reading several properties in Kafka documentations, I asked mysleft
some questions ...
these 2 following options are available:
log.dir The directory in which the log data is kept (supplemental for
log.dirs property) string /tmp/kafka-logs high
log.dirs The directories in which the log data is kept. If not set,
the value in log.dir is used
But in fact, if is the same thing, why only "log.dirs" isn't kept ? What's
difference in usage ?
Also I noticed that the "data" of logs partition and also the
"application" logs of Kafka are written in the same directory.
Is a mistake of my part ? Because it's very strange for me to log error,
debug, warn application message in the same location of my data ...
After that I don't very understand why the log.flush.interval.messages
have a so big default value ???
log.flush.interval.messages The number of messages accumulated on a
log partition before messages are flushed to disk long
9223372036854775807
And the log.flush.interval.ms is by default null ... ?
It means that until there are so many messages (9223372036854775807) in my
topics, they will not be flushed to disk ? It can be very long for a
default value ....
Best regards,
Adrien
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: broker properties explanations
Posted by Thomas Aley <Th...@ibm.com>.
Hi Adrien,
log.dirs exists to facilitate multiple data directories which allows more
than one disk to be used without the need for RAID. This increases
throughput but beware of naive load balancing that may fill up one disk
way before another.
When log.flush.interval.ms is null the log.flush.interval.messages
property is not used. With default settings, messages are written to disk
immediately.
Hope this helps.
Tom Aley
thomas.aley@ibm.com
From: adrien ruffie <ad...@hotmail.fr>
To: "users@kafka.apache.org" <us...@kafka.apache.org>
Date: 20/02/2018 20:46
Subject: broker properties explanations
Hello all,
after reading several properties in Kafka documentations, I asked mysleft
some questions ...
these 2 following options are available:
log.dir The directory in which the log data is kept (supplemental for
log.dirs property) string /tmp/kafka-logs high
log.dirs The directories in which the log data is kept. If not set,
the value in log.dir is used
But in fact, if is the same thing, why only "log.dirs" isn't kept ? What's
difference in usage ?
Also I noticed that the "data" of logs partition and also the
"application" logs of Kafka are written in the same directory.
Is a mistake of my part ? Because it's very strange for me to log error,
debug, warn application message in the same location of my data ...
After that I don't very understand why the log.flush.interval.messages
have a so big default value ???
log.flush.interval.messages The number of messages accumulated on a
log partition before messages are flushed to disk long
9223372036854775807
And the log.flush.interval.ms is by default null ... ?
It means that until there are so many messages (9223372036854775807) in my
topics, they will not be flushed to disk ? It can be very long for a
default value ....
Best regards,
Adrien
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU