You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kane Kim <ka...@gmail.com> on 2017/11/29 06:19:11 UTC
kafka compacted topic
How does kafka log compaction work?
Does it compact all of the log files periodically against new changes?
Re: kafka compacted topic
Posted by Ben Stopford <be...@confluent.io>.
The CPU/IO required to complete a compaction phase will grow as the log
grows but you can manage this via the cleaner's various configs. Check out
properties starting log.cleaner in the docs (
https://kafka.apache.org/documentation). All databases that implement LSM
storage have a similar overhead (Cassandra, HBase, RocksDB etc). Note the
first (active) segment is never compacted.
On Thu, Nov 30, 2017 at 6:59 AM Kane Kim <ka...@gmail.com> wrote:
> I want to confirm if kafka has to re-compact all log segments, as log grows
> doesn't it become slower as well?
>
> On Tue, Nov 28, 2017 at 11:33 PM, Jakub Scholz <ja...@scholz.cz> wrote:
>
> > There is quite a nice section on this in the documentation -
> > http://kafka.apache.org/documentation/#compaction ... I think it should
> > answer your questions.
> >
> > On Wed, Nov 29, 2017 at 7:19 AM, Kane Kim <ka...@gmail.com> wrote:
> >
> > > How does kafka log compaction work?
> > > Does it compact all of the log files periodically against new changes?
> > >
> >
>
Re: kafka compacted topic
Posted by Kane Kim <ka...@gmail.com>.
I want to confirm if kafka has to re-compact all log segments, as log grows
doesn't it become slower as well?
On Tue, Nov 28, 2017 at 11:33 PM, Jakub Scholz <ja...@scholz.cz> wrote:
> There is quite a nice section on this in the documentation -
> http://kafka.apache.org/documentation/#compaction ... I think it should
> answer your questions.
>
> On Wed, Nov 29, 2017 at 7:19 AM, Kane Kim <ka...@gmail.com> wrote:
>
> > How does kafka log compaction work?
> > Does it compact all of the log files periodically against new changes?
> >
>
Re: kafka compacted topic
Posted by Jakub Scholz <ja...@scholz.cz>.
There is quite a nice section on this in the documentation -
http://kafka.apache.org/documentation/#compaction ... I think it should
answer your questions.
On Wed, Nov 29, 2017 at 7:19 AM, Kane Kim <ka...@gmail.com> wrote:
> How does kafka log compaction work?
> Does it compact all of the log files periodically against new changes?
>