You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Francesco laTorre <fr...@openbet.com> on 2016/11/02 12:55:00 UTC
log compaction
Hi,
We want to enable log compaction on an existing topic (in production).
Is it a safe operation or there are things to take into consideration ?
Kafka version 0.8
Cheers,
Francesco
--
<http://www.openbet.com/> Francesco laTorre
Senior Developer
T: +44 208 742 1600
+44 203 249 8394
E: francesco.latorre@openbet.com
W: www.openbet.com
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster@openbet.com and delete it from your system as well as any
copies. The content of e-mails as well as traffic data may be monitored by
OpenBet for employment and security purposes. To protect the environment
please do not print this e-mail unless necessary. OpenBet Ltd. Registered
Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
United Kingdom. A company registered in England and Wales. Registered no.
3134634. VAT no. GB927523612
Re: log compaction
Posted by Francesco laTorre <fr...@openbet.com>.
Hi Becket,
I can confirm :
1. All messages are keyed
2. Generous memory settings already in place
3. The topic is not compacted
4. Luckily we haven't experienced this issue. We are testing 0.10.0 and
hope to make the jump shortly.
Thanks you very much for the support, very much appreciated :)
Cheers,
Francesco
On 4 November 2016 at 01:38, Becket Qin <be...@gmail.com> wrote:
> Hi Francesco,
>
> There are a few things to think about before turning on log compaction for
> a topic.
>
> 1. Does the topic have non-keyed message? Log compaction only works if all
> the messages have a key.
> 2. The log cleaner needs some memory to build the offset map for log
> compaction, so the memory consumption may be higher.
>
> Given you are still running on 0.8, there are a few additional things to be
> aware of:
> 3. Log compaction doesn't work with compressed topics until Kafka 0.9.0.
> 4. Some other potential issues that may caused by log compaction. (e.g.
> KAFKA-2024).
>
> Jiangjie (Becket) Qin
>
>
>
> On Wed, Nov 2, 2016 at 5:55 AM, Francesco laTorre <
> francesco.latorre@openbet.com> wrote:
>
> > Hi,
> >
> > We want to enable log compaction on an existing topic (in production).
> > Is it a safe operation or there are things to take into consideration ?
> >
> > Kafka version 0.8
> >
> > Cheers,
> > Francesco
> >
> > --
> > <http://www.openbet.com/> Francesco laTorre
> > Senior Developer
> > T: +44 208 742 1600
> > +44 203 249 8394
> >
> > E: francesco.latorre@openbet.com
> > W: www.openbet.com
> > OpenBet Ltd
> > Chiswick Park Building 9
> > 566 Chiswick High Rd
> > London
> > W4 5XT
> > <https://www.openbet.com/email_promo>
> > This message is confidential and intended only for the addressee. If you
> > have received this message in error, please immediately notify the
> > postmaster@openbet.com and delete it from your system as well as any
> > copies. The content of e-mails as well as traffic data may be monitored
> by
> > OpenBet for employment and security purposes. To protect the environment
> > please do not print this e-mail unless necessary. OpenBet Ltd. Registered
> > Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
> > United Kingdom. A company registered in England and Wales. Registered no.
> > 3134634. VAT no. GB927523612
> >
>
--
<http://www.openbet.com/> Francesco laTorre
Senior Developer
T: +44 208 742 1600
+44 203 249 8394
E: francesco.latorre@openbet.com
W: www.openbet.com
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster@openbet.com and delete it from your system as well as any
copies. The content of e-mails as well as traffic data may be monitored by
OpenBet for employment and security purposes. To protect the environment
please do not print this e-mail unless necessary. OpenBet Ltd. Registered
Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
United Kingdom. A company registered in England and Wales. Registered no.
3134634. VAT no. GB927523612
Re: log compaction
Posted by Becket Qin <be...@gmail.com>.
Hi Francesco,
There are a few things to think about before turning on log compaction for
a topic.
1. Does the topic have non-keyed message? Log compaction only works if all
the messages have a key.
2. The log cleaner needs some memory to build the offset map for log
compaction, so the memory consumption may be higher.
Given you are still running on 0.8, there are a few additional things to be
aware of:
3. Log compaction doesn't work with compressed topics until Kafka 0.9.0.
4. Some other potential issues that may caused by log compaction. (e.g.
KAFKA-2024).
Jiangjie (Becket) Qin
On Wed, Nov 2, 2016 at 5:55 AM, Francesco laTorre <
francesco.latorre@openbet.com> wrote:
> Hi,
>
> We want to enable log compaction on an existing topic (in production).
> Is it a safe operation or there are things to take into consideration ?
>
> Kafka version 0.8
>
> Cheers,
> Francesco
>
> --
> <http://www.openbet.com/> Francesco laTorre
> Senior Developer
> T: +44 208 742 1600
> +44 203 249 8394
>
> E: francesco.latorre@openbet.com
> W: www.openbet.com
> OpenBet Ltd
> Chiswick Park Building 9
> 566 Chiswick High Rd
> London
> W4 5XT
> <https://www.openbet.com/email_promo>
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> postmaster@openbet.com and delete it from your system as well as any
> copies. The content of e-mails as well as traffic data may be monitored by
> OpenBet for employment and security purposes. To protect the environment
> please do not print this e-mail unless necessary. OpenBet Ltd. Registered
> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
> United Kingdom. A company registered in England and Wales. Registered no.
> 3134634. VAT no. GB927523612
>