Process to enable log compaction on a cluster

[Sathyakumar Seshachalam <sa...@trimble.com>]

Ok, Am in situation where all kafka nodes are going to run out of space.
This is because I had been running uncompacted __consumer_offset topic and
everything retained topics .

Now at a place, where I can afford to compact __consumer_offset topic and
also delete certain topics. I would like to know the right process to do
this.

Now since I am having close 1.8T of data on __consumer_offset topic and
more in the topics data, any log compaction and log deletion/trunction is
going to take time. Should I do this node by node. Will Kafka's replication
come in the way. (I have read that uncompacted data from the leader is sent
to the followers.)

Is there a clean process for this for a 3 node Kafka cluster ? Last time I
triggered a log compaction in all the 3 node simultaneously, all consumers
broke (I raised this in the same email group and got an answer to increase
the memory). Eventually they self-healed, but this caused some serious
disruption to the service, so before trying I want to make sure, there is a
cleaner process here.

Any help/pointers will be greatly appreciated.

Thanks,
Sathya

Re: Process to enable log compaction on a cluster

[Mario Ricci <ma...@trimble.com>]

Sathya,

Did you ever figure out what to do here?

On Mon, Jul 4, 2016 at 12:19 AM Sathyakumar Seshachalam <
sathyakumar_seshachalam@trimble.com> wrote:

> My another followup question is that Am I right in assuming that per topic
> retention minutes or clean up policy, they all have any effect only when
> you log.cleaner.enable-false ?
> So In other words, I choose to truncate topic data, then __consumer_offsets
> topic will also be either deleted or compacted ?
>
> On Mon, Jul 4, 2016 at 11:38 AM, Sathyakumar Seshachalam <
> sathyakumar_seshachalam@trimble.com> wrote:
>
> > Ok, Am in situation where all kafka nodes are going to run out of space.
> > This is because I had been running uncompacted __consumer_offset topic
> and
> > everything retained topics .
> >
> > Now at a place, where I can afford to compact __consumer_offset topic and
> > also delete certain topics. I would like to know the right process to do
> > this.
> >
> > Now since I am having close 1.8T of data on __consumer_offset topic and
> > more in the topics data, any log compaction and log deletion/trunction is
> > going to take time. Should I do this node by node. Will Kafka's
> replication
> > come in the way. (I have read that uncompacted data from the leader is
> sent
> > to the followers.)
> >
> > Is there a clean process for this for a 3 node Kafka cluster ? Last time
> I
> > triggered a log compaction in all the 3 node simultaneously, all
> consumers
> > broke (I raised this in the same email group and got an answer to
> increase
> > the memory). Eventually they self-healed, but this caused some serious
> > disruption to the service, so before trying I want to make sure, there
> is a
> > cleaner process here.
> >
> > Any help/pointers will be greatly appreciated.
> >
> > Thanks,
> > Sathya
> >
> >
> >
>
-- 

Mario Ricci | Software Engineer | Trimble Navigation Limited | VirtualSite
Solutions | Office: +1 (303) 635-8604 / x228604

Re: Process to enable log compaction on a cluster

[Sathyakumar Seshachalam <sa...@trimble.com>]

My another followup question is that Am I right in assuming that per topic
retention minutes or clean up policy, they all have any effect only when
you log.cleaner.enable-false ?
So In other words, I choose to truncate topic data, then __consumer_offsets
topic will also be either deleted or compacted ?

On Mon, Jul 4, 2016 at 11:38 AM, Sathyakumar Seshachalam <
sathyakumar_seshachalam@trimble.com> wrote:

> Ok, Am in situation where all kafka nodes are going to run out of space.
> This is because I had been running uncompacted __consumer_offset topic and
> everything retained topics .
>
> Now at a place, where I can afford to compact __consumer_offset topic and
> also delete certain topics. I would like to know the right process to do
> this.
>
> Now since I am having close 1.8T of data on __consumer_offset topic and
> more in the topics data, any log compaction and log deletion/trunction is
> going to take time. Should I do this node by node. Will Kafka's replication
> come in the way. (I have read that uncompacted data from the leader is sent
> to the followers.)
>
> Is there a clean process for this for a 3 node Kafka cluster ? Last time I
> triggered a log compaction in all the 3 node simultaneously, all consumers
> broke (I raised this in the same email group and got an answer to increase
> the memory). Eventually they self-healed, but this caused some serious
> disruption to the service, so before trying I want to make sure, there is a
> cleaner process here.
>
> Any help/pointers will be greatly appreciated.
>
> Thanks,
> Sathya
>
>
>