You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Manikumar <ma...@gmail.com> on 2019/11/14 18:40:58 UTC
Preliminary blog post about the Apache Kafka 2.4.0 release
Hi all,
I've prepared a preliminary blog post about the upcoming Apache Kafka 2.4.0
release.
Please take a look and let me know if you want to add/modify details.
Thanks to all who contributed to this blog post.
https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
Thanks,
Manikumar
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Manikumar <ma...@gmail.com>.
Sure, I will mention the details about ZK upgrade to 3.5
Thanks.
On Tue, Nov 26, 2019 at 1:57 PM Sönke Liebau
<so...@opencore.com.invalid> wrote:
> Hi Manikumar,
>
> looks great, thanks for the effort!
>
> Like Ismael, I think it would make sense to mention the ZK upgrade to 3.5 I
> know a few people that are waiting for this because of the SSL support so
> there is probably more of them out there :)
>
> Best regards,
> Sönke
>
>
> On Mon, 25 Nov 2019 at 20:11, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Manikumar,
> >
> > One thing I had previously missed, should we mention the upgrade to ZK
> 3.5?
> >
> > Ismael
> >
> > On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > 2.4.0
> > > release.
> > > Please take a look and let me know if you want to add/modify details.
> > > Thanks to all who contributed to this blog post.
> > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Manikumar <ma...@gmail.com>.
Sure, I will mention the details about ZK upgrade to 3.5
Thanks.
On Tue, Nov 26, 2019 at 1:57 PM Sönke Liebau
<so...@opencore.com.invalid> wrote:
> Hi Manikumar,
>
> looks great, thanks for the effort!
>
> Like Ismael, I think it would make sense to mention the ZK upgrade to 3.5 I
> know a few people that are waiting for this because of the SSL support so
> there is probably more of them out there :)
>
> Best regards,
> Sönke
>
>
> On Mon, 25 Nov 2019 at 20:11, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Manikumar,
> >
> > One thing I had previously missed, should we mention the upgrade to ZK
> 3.5?
> >
> > Ismael
> >
> > On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > 2.4.0
> > > release.
> > > Please take a look and let me know if you want to add/modify details.
> > > Thanks to all who contributed to this blog post.
> > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Sönke Liebau <so...@opencore.com.INVALID>.
Hi Manikumar,
looks great, thanks for the effort!
Like Ismael, I think it would make sense to mention the ZK upgrade to 3.5 I
know a few people that are waiting for this because of the SSL support so
there is probably more of them out there :)
Best regards,
Sönke
On Mon, 25 Nov 2019 at 20:11, Ismael Juma <is...@juma.me.uk> wrote:
> Manikumar,
>
> One thing I had previously missed, should we mention the upgrade to ZK 3.5?
>
> Ismael
>
> On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I've prepared a preliminary blog post about the upcoming Apache Kafka
> 2.4.0
> > release.
> > Please take a look and let me know if you want to add/modify details.
> > Thanks to all who contributed to this blog post.
> >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> >
> > Thanks,
> > Manikumar
> >
>
--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Sönke Liebau <so...@opencore.com.INVALID>.
Hi Manikumar,
looks great, thanks for the effort!
Like Ismael, I think it would make sense to mention the ZK upgrade to 3.5 I
know a few people that are waiting for this because of the SSL support so
there is probably more of them out there :)
Best regards,
Sönke
On Mon, 25 Nov 2019 at 20:11, Ismael Juma <is...@juma.me.uk> wrote:
> Manikumar,
>
> One thing I had previously missed, should we mention the upgrade to ZK 3.5?
>
> Ismael
>
> On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I've prepared a preliminary blog post about the upcoming Apache Kafka
> 2.4.0
> > release.
> > Please take a look and let me know if you want to add/modify details.
> > Thanks to all who contributed to this blog post.
> >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> >
> > Thanks,
> > Manikumar
> >
>
--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Ismael Juma <is...@juma.me.uk>.
Manikumar,
One thing I had previously missed, should we mention the upgrade to ZK 3.5?
Ismael
On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
wrote:
> Hi all,
>
> I've prepared a preliminary blog post about the upcoming Apache Kafka 2.4.0
> release.
> Please take a look and let me know if you want to add/modify details.
> Thanks to all who contributed to this blog post.
>
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
>
> Thanks,
> Manikumar
>
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Manikumar <ma...@gmail.com>.
Hi Sean,
Thanks for the summary. Will include in the blogpost.
On Mon, Nov 18, 2019 at 11:04 PM Sean Glover <se...@lightbend.com>
wrote:
> Here's a summary that can go under the section "What’s new in Kafka broker,
> producer, and consumer" as an "improvement". Feel free to rephrase as you
> see fit.
>
> When a partition is paused by the user in the consumer the partition is
> > considered "unfetchable". When the consumer has already fetched data
> for a
> > partition, and then the partition is paused, then in the next consumer
> poll
> > all data from "unfetchable" partitions will be discarded. In use cases
> > where pausing and resuming partitions is common during regular operation
> of
> > the consumer this can result in discarding pre-fetched data when it's not
> > necessary. Once the partition is resumed then new fetch requests will be
> > generated and sent to the broker to get the same partition data again.
> > Depending on the frequency of pausing and resuming of partitions this can
> > impact a number of different aspects of consumer polling including:
> > broker/consumer throughput, number of consumer fetch requests, and
> > NIO-related GC concerns for regularly dereferenced byte buffers of
> > partition data. This issue is now resolved by retaining completed fetch
> > data for partitions that are paused so that it may be returned in a
> future
> > consumer poll once the partition is resumed by the user.
> >
>
>
> See [KAFKA-7548](https://issues.apache.org/jira/browse/KAFKA-7548) for
> more
> > details.
>
>
> Regards,
> Sean
>
> On Mon, Nov 18, 2019 at 11:45 AM Ismael Juma <is...@juma.me.uk> wrote:
>
> > That makes sense to me.
> >
> > Ismael
> >
> > On Mon, Nov 18, 2019 at 8:40 AM Sean Glover <se...@lightbend.com>
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > I'm putting together an akka.io blog post regarding [KAFKA-7548] -
> > > KafkaConsumer should not throw away already fetched data for paused
> > > partitions. Since it doesn't change any user-facing APIs it has no
> KIP,
> > > but it has a significant impact on consumer use cases that frequently
> > pause
> > > and resume partitions, such as in Alpakka Kafka. I can provide a small
> > > summary for you to include in your blog post if you think it's
> > appropriate.
> > >
> > > Regards,
> > > Sean
> > >
> > > On Mon, Nov 18, 2019 at 11:25 AM Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Chris. will update the blog content.
> > > >
> > > > On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton <ch...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hi Manikumar,
> > > > >
> > > > > It looks like the header for KIP-440 is accurate ("KIP-440: Extend
> > > > Connect
> > > > > Converter to support headers") but the content appears to
> correspond
> > to
> > > > > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON")
> > > instead.
> > > > > Could we double-check and make sure that the summary for KIP-440
> > > matches
> > > > > what was contributed for it (and it nothing was, alter the summary
> to
> > > > more
> > > > > closely reflect what KIP-440 accomplished)?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Thu, Nov 14, 2019 at 10:41 AM Manikumar <
> > manikumar.reddy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I've prepared a preliminary blog post about the upcoming Apache
> > Kafka
> > > > > 2.4.0
> > > > > > release.
> > > > > > Please take a look and let me know if you want to add/modify
> > details.
> > > > > > Thanks to all who contributed to this blog post.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > > > > >
> > > > > > Thanks,
> > > > > > Manikumar
> > > > > >
> > > > >
> > > >
> > >
>
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Sean Glover <se...@lightbend.com>.
Here's a summary that can go under the section "What’s new in Kafka broker,
producer, and consumer" as an "improvement". Feel free to rephrase as you
see fit.
When a partition is paused by the user in the consumer the partition is
> considered "unfetchable". When the consumer has already fetched data for a
> partition, and then the partition is paused, then in the next consumer poll
> all data from "unfetchable" partitions will be discarded. In use cases
> where pausing and resuming partitions is common during regular operation of
> the consumer this can result in discarding pre-fetched data when it's not
> necessary. Once the partition is resumed then new fetch requests will be
> generated and sent to the broker to get the same partition data again.
> Depending on the frequency of pausing and resuming of partitions this can
> impact a number of different aspects of consumer polling including:
> broker/consumer throughput, number of consumer fetch requests, and
> NIO-related GC concerns for regularly dereferenced byte buffers of
> partition data. This issue is now resolved by retaining completed fetch
> data for partitions that are paused so that it may be returned in a future
> consumer poll once the partition is resumed by the user.
>
See [KAFKA-7548](https://issues.apache.org/jira/browse/KAFKA-7548) for more
> details.
Regards,
Sean
On Mon, Nov 18, 2019 at 11:45 AM Ismael Juma <is...@juma.me.uk> wrote:
> That makes sense to me.
>
> Ismael
>
> On Mon, Nov 18, 2019 at 8:40 AM Sean Glover <se...@lightbend.com>
> wrote:
>
> > Hi Manikumar,
> >
> > I'm putting together an akka.io blog post regarding [KAFKA-7548] -
> > KafkaConsumer should not throw away already fetched data for paused
> > partitions. Since it doesn't change any user-facing APIs it has no KIP,
> > but it has a significant impact on consumer use cases that frequently
> pause
> > and resume partitions, such as in Alpakka Kafka. I can provide a small
> > summary for you to include in your blog post if you think it's
> appropriate.
> >
> > Regards,
> > Sean
> >
> > On Mon, Nov 18, 2019 at 11:25 AM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Thanks Chris. will update the blog content.
> > >
> > > On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton <ch...@confluent.io>
> > > wrote:
> > >
> > > > Hi Manikumar,
> > > >
> > > > It looks like the header for KIP-440 is accurate ("KIP-440: Extend
> > > Connect
> > > > Converter to support headers") but the content appears to correspond
> to
> > > > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON")
> > instead.
> > > > Could we double-check and make sure that the summary for KIP-440
> > matches
> > > > what was contributed for it (and it nothing was, alter the summary to
> > > more
> > > > closely reflect what KIP-440 accomplished)?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Thu, Nov 14, 2019 at 10:41 AM Manikumar <
> manikumar.reddy@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I've prepared a preliminary blog post about the upcoming Apache
> Kafka
> > > > 2.4.0
> > > > > release.
> > > > > Please take a look and let me know if you want to add/modify
> details.
> > > > > Thanks to all who contributed to this blog post.
> > > > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> >
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Ismael Juma <is...@juma.me.uk>.
That makes sense to me.
Ismael
On Mon, Nov 18, 2019 at 8:40 AM Sean Glover <se...@lightbend.com>
wrote:
> Hi Manikumar,
>
> I'm putting together an akka.io blog post regarding [KAFKA-7548] -
> KafkaConsumer should not throw away already fetched data for paused
> partitions. Since it doesn't change any user-facing APIs it has no KIP,
> but it has a significant impact on consumer use cases that frequently pause
> and resume partitions, such as in Alpakka Kafka. I can provide a small
> summary for you to include in your blog post if you think it's appropriate.
>
> Regards,
> Sean
>
> On Mon, Nov 18, 2019 at 11:25 AM Manikumar <ma...@gmail.com>
> wrote:
>
> > Thanks Chris. will update the blog content.
> >
> > On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton <ch...@confluent.io>
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > It looks like the header for KIP-440 is accurate ("KIP-440: Extend
> > Connect
> > > Converter to support headers") but the content appears to correspond to
> > > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON")
> instead.
> > > Could we double-check and make sure that the summary for KIP-440
> matches
> > > what was contributed for it (and it nothing was, alter the summary to
> > more
> > > closely reflect what KIP-440 accomplished)?
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > > 2.4.0
> > > > release.
> > > > Please take a look and let me know if you want to add/modify details.
> > > > Thanks to all who contributed to this blog post.
> > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > >
> >
>
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Sean Glover <se...@lightbend.com>.
Hi Manikumar,
I'm putting together an akka.io blog post regarding [KAFKA-7548] -
KafkaConsumer should not throw away already fetched data for paused
partitions. Since it doesn't change any user-facing APIs it has no KIP,
but it has a significant impact on consumer use cases that frequently pause
and resume partitions, such as in Alpakka Kafka. I can provide a small
summary for you to include in your blog post if you think it's appropriate.
Regards,
Sean
On Mon, Nov 18, 2019 at 11:25 AM Manikumar <ma...@gmail.com>
wrote:
> Thanks Chris. will update the blog content.
>
> On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton <ch...@confluent.io>
> wrote:
>
> > Hi Manikumar,
> >
> > It looks like the header for KIP-440 is accurate ("KIP-440: Extend
> Connect
> > Converter to support headers") but the content appears to correspond to
> > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON") instead.
> > Could we double-check and make sure that the summary for KIP-440 matches
> > what was contributed for it (and it nothing was, alter the summary to
> more
> > closely reflect what KIP-440 accomplished)?
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I've prepared a preliminary blog post about the upcoming Apache Kafka
> > 2.4.0
> > > release.
> > > Please take a look and let me know if you want to add/modify details.
> > > Thanks to all who contributed to this blog post.
> > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Manikumar <ma...@gmail.com>.
Thanks Chris. will update the blog content.
On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton <ch...@confluent.io> wrote:
> Hi Manikumar,
>
> It looks like the header for KIP-440 is accurate ("KIP-440: Extend Connect
> Converter to support headers") but the content appears to correspond to
> KIP-481 ("SerDe Improvements for Connect Decimal type in JSON") instead.
> Could we double-check and make sure that the summary for KIP-440 matches
> what was contributed for it (and it nothing was, alter the summary to more
> closely reflect what KIP-440 accomplished)?
>
> Cheers,
>
> Chris
>
> On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I've prepared a preliminary blog post about the upcoming Apache Kafka
> 2.4.0
> > release.
> > Please take a look and let me know if you want to add/modify details.
> > Thanks to all who contributed to this blog post.
> >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> >
> > Thanks,
> > Manikumar
> >
>
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Chris Egerton <ch...@confluent.io>.
Hi Manikumar,
It looks like the header for KIP-440 is accurate ("KIP-440: Extend Connect
Converter to support headers") but the content appears to correspond to
KIP-481 ("SerDe Improvements for Connect Decimal type in JSON") instead.
Could we double-check and make sure that the summary for KIP-440 matches
what was contributed for it (and it nothing was, alter the summary to more
closely reflect what KIP-440 accomplished)?
Cheers,
Chris
On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
wrote:
> Hi all,
>
> I've prepared a preliminary blog post about the upcoming Apache Kafka 2.4.0
> release.
> Please take a look and let me know if you want to add/modify details.
> Thanks to all who contributed to this blog post.
>
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
>
> Thanks,
> Manikumar
>
Re: Preliminary blog post about the Apache Kafka 2.4.0 release
Posted by Ismael Juma <is...@juma.me.uk>.
Manikumar,
One thing I had previously missed, should we mention the upgrade to ZK 3.5?
Ismael
On Thu, Nov 14, 2019 at 10:41 AM Manikumar <ma...@gmail.com>
wrote:
> Hi all,
>
> I've prepared a preliminary blog post about the upcoming Apache Kafka 2.4.0
> release.
> Please take a look and let me know if you want to add/modify details.
> Thanks to all who contributed to this blog post.
>
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
>
> Thanks,
> Manikumar
>