offset topics growing huge

[Tobias Adamson <to...@stargazer.com.sg>]

Hi
We are using Kafka 0.10.1 with offsets commits being stored inside of Kafka
After a while these topics become extremely large and we are wondering if we 
need to enable log.cleaner.enable=true (currently false) to make sure the internal
offset topics get compacted and keep their size down?

Regards
T

Re: offset topics growing huge

[Tom Crayford <tc...@heroku.com>]

On Mon, Oct 3, 2016 at 5:38 PM, Ali Akhtar <al...@gmail.com> wrote:

> Newbie question, but what exactly does log.cleaner.enable=true do, and how
> do I know if I need to set it to be true?
>

If you're using any compacted topics (including __consumer_offsets), it
needs to be on.


>
> Also, if config changes like that need to be made once a cluster is up and
> running, what's the recommended way to do that? Do you killall -12 kafka
> and then make the change, and then start kafka again, one broker at a time?
>

Roughly yes. You should probably orchestrate it and run e.g. health checks
and such before restarting the next node.

Thanks

Tom Crayford
Heroku Kafka


>
> On Mon, Oct 3, 2016 at 9:27 PM, Tom Crayford <tc...@heroku.com> wrote:
>
> > Yes, offset topic compaction is just the normal compaction.
> >
> > Thanks
> >
> > Tom Crayford
> > Heroku Kafka
> >
> > On Monday, 3 October 2016, Tobias Adamson <to...@stargazer.com.sg>
> wrote:
> >
> > > Hi
> > > We are using Kafka 0.10.1 with offsets commits being stored inside of
> > Kafka
> > > After a while these topics become extremely large and we are wondering
> if
> > > we
> > > need to enable log.cleaner.enable=true (currently false) to make sure
> the
> > > internal
> > > offset topics get compacted and keep their size down?
> > >
> > > Regards
> > > T
> >
>

Re: offset topics growing huge

[Ali Akhtar <al...@gmail.com>]

Newbie question, but what exactly does log.cleaner.enable=true do, and how
do I know if I need to set it to be true?

Also, if config changes like that need to be made once a cluster is up and
running, what's the recommended way to do that? Do you killall -12 kafka
and then make the change, and then start kafka again, one broker at a time?

On Mon, Oct 3, 2016 at 9:27 PM, Tom Crayford <tc...@heroku.com> wrote:

> Yes, offset topic compaction is just the normal compaction.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Monday, 3 October 2016, Tobias Adamson <to...@stargazer.com.sg> wrote:
>
> > Hi
> > We are using Kafka 0.10.1 with offsets commits being stored inside of
> Kafka
> > After a while these topics become extremely large and we are wondering if
> > we
> > need to enable log.cleaner.enable=true (currently false) to make sure the
> > internal
> > offset topics get compacted and keep their size down?
> >
> > Regards
> > T
>

Re: offset topics growing huge

[Tom Crayford <tc...@heroku.com>]

Yes, offset topic compaction is just the normal compaction.

Thanks

Tom Crayford
Heroku Kafka

On Monday, 3 October 2016, Tobias Adamson <to...@stargazer.com.sg> wrote:

> Hi
> We are using Kafka 0.10.1 with offsets commits being stored inside of Kafka
> After a while these topics become extremely large and we are wondering if
> we
> need to enable log.cleaner.enable=true (currently false) to make sure the
> internal
> offset topics get compacted and keep their size down?
>
> Regards
> T