You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Krishna Agarwal <kr...@gmail.com> on 2023/09/08 07:17:13 UTC
[DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Hi,
I want to submit a KIP to deliver an experimental Apache Kafka docker image.
The proposed docker image can launch brokers with sub-second startup time
and minimal memory footprint by leveraging a GraalVM based native Kafka
binary.
KIP-974: Docker Image for GraalVM based Native Kafka Broker
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker>
Regards,
Krishna
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi Ismael,
Thanks for the feedback.
I noticed that the graalvm-reachability-metadata project now includes
> kafka-clients[1]. Worth checking if the metadata over there matches what
> we've observed for our own attempt.
>
After reviewing the graalvm-reachability-metadata, I observed that they
have provided configurations solely for the kafka-clients. However, we
require configurations for the server (the core module).
To obtain these configurations, I've leveraged the existing Apache Kafka
system tests. I'm executing the tests using GraalVM JIT with the
Native-Image Agent attached. I merge the configurations once all the tests
have been completed.
Can we please add peak memory usage as one of the columns of the table?
> I see that the information is covered elsewhere - it would be nice to have
> it as part of the table too.
>
Done
GraalVM for JDK 21 was recently released, can we use that? Does it improve
> the numbers in a meaningful way?
Sure, we will be using the Image corresponding to 21. The community version
Image:
https://github.com/graalvm/container/pkgs/container/graalvm-community/129309492?tag=21
It would be good to include details on the naming - how will this image
> be named to differentiate itself from the non graalvm one?
>
Added a section in the respective KIPs.
In summary, we propose naming the image as
*<image-name>:<kafka-version>-<optional-suffix>*.
For the native AK docker image, we are considering '*kafka-local*' as it
clearly signifies that this image is intended exclusively for local
development purposes. Another option under consideration is '*kafka-native*
'.
Can we talk a bit more about the compatibility guarantees while this
> image is still experimental?
Do you here mean the functionalities that this native AK docker image will
be able to support?
I have updated the KIP with the above points.
Regards,
Krishna
On Wed, Sep 20, 2023 at 11:39 PM Ismael Juma <me...@ismaeljuma.com> wrote:
> Hi Krishna,
>
> Thanks for the KIP. The results are impressive! A few quick comments:
>
> 1. I noticed that the graalvm-reachability-metadata project now includes
> kafka-clients[1]. Worth checking if the metadata over there matches what
> we've observed for our own attempt.
> 2. Can we please add peak memory usage as one of the columns of the table?
> I see that the information is covered elsewhere - it would be nice to have
> it as part of the table too.
> 3. GraalVM for JDK 21 was recently released, can we use that? Does it
> improve the numbers in a meaningful way?
> 4. Can we talk a bit more about the compatibility guarantees while this
> image is still experimental?
> 5. It would be good to include details on the naming - how will this image
> be named to differentiate itself from the non graalvm one?
>
> [1]
>
> https://github.com/oracle/graalvm-reachability-metadata/tree/df53148e4b2f730ad692daa956d72ab0cd7d43e3/metadata/org.apache.kafka/kafka-clients/3.5.1
>
> Ismael
>
> On Fri, Sep 8, 2023 at 12:17 AM Krishna Agarwal <
> krishna0608agarwal@gmail.com> wrote:
>
> > Hi,
> > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > image.
> > The proposed docker image can launch brokers with sub-second startup time
> > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > binary.
> >
> > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > >
> >
> > Regards,
> > Krishna
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Ismael Juma <me...@ismaeljuma.com>.
Hi Krishna,
Thanks for the KIP. The results are impressive! A few quick comments:
1. I noticed that the graalvm-reachability-metadata project now includes
kafka-clients[1]. Worth checking if the metadata over there matches what
we've observed for our own attempt.
2. Can we please add peak memory usage as one of the columns of the table?
I see that the information is covered elsewhere - it would be nice to have
it as part of the table too.
3. GraalVM for JDK 21 was recently released, can we use that? Does it
improve the numbers in a meaningful way?
4. Can we talk a bit more about the compatibility guarantees while this
image is still experimental?
5. It would be good to include details on the naming - how will this image
be named to differentiate itself from the non graalvm one?
[1]
https://github.com/oracle/graalvm-reachability-metadata/tree/df53148e4b2f730ad692daa956d72ab0cd7d43e3/metadata/org.apache.kafka/kafka-clients/3.5.1
Ismael
On Fri, Sep 8, 2023 at 12:17 AM Krishna Agarwal <
krishna0608agarwal@gmail.com> wrote:
> Hi,
> I want to submit a KIP to deliver an experimental Apache Kafka docker
> image.
> The proposed docker image can launch brokers with sub-second startup time
> and minimal memory footprint by leveraging a GraalVM based native Kafka
> binary.
>
> KIP-974: Docker Image for GraalVM based Native Kafka Broker
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> >
>
> Regards,
> Krishna
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi Justine,
Thanks for the feedback.
1. I have added the name of the other image in the KIP.
2. By experimental we mean the docker image is intended for local
development and testing usage.
GraalVM Native-Image tool is still in maturing stage, hence the usage of
this image for production can’t be recommended.
Regarding the upgrade, we will be running the existing System Tests
covering upgrade(JVM to Native) and downgrade.
Regards,
Krishna
On Wed, Nov 22, 2023 at 6:56 AM Justine Olshan <jo...@confluent.io.invalid>
wrote:
> Hey -- just catching up here, since I saw the vote thread. I had 2
> questions that I'm not sure got answered from the previous discussion.
>
> 1. Can we update the KIP to include the name of the other image so if
> someone stumbles across this KIP they know the name of the other one?
> 2. Did we cover what "experimental" means here? I think Ismael asked
> > Can we talk a bit more about the compatibility guarantees while this
> image is still experimental?
> I took this to mean should we be able to upgrade to or from this image or
> if clusters running on it can only run on it? Or if there are no
> guarantees about the upgrade/downgrade story.
>
> Thanks,
> Justine
>
> On Sun, Nov 19, 2023 at 7:54 PM Krishna Agarwal <
> krishna0608agarwal@gmail.com> wrote:
>
> > Hi,
> > Thanks for the insightful feedback on this KIP.
> > As there are no ongoing discussions, I'm considering moving into the
> voting
> > process.
> > Your continued input is greatly appreciated!
> >
> > Regards,
> > Krishna
> >
> > On Fri, Sep 8, 2023 at 12:47 PM Krishna Agarwal <
> > krishna0608agarwal@gmail.com> wrote:
> >
> > > Hi,
> > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > > image.
> > > The proposed docker image can launch brokers with sub-second startup
> time
> > > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > > binary.
> > >
> > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > >
> > >
> > > Regards,
> > > Krishna
> > >
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Justine Olshan <jo...@confluent.io.INVALID>.
Hey -- just catching up here, since I saw the vote thread. I had 2
questions that I'm not sure got answered from the previous discussion.
1. Can we update the KIP to include the name of the other image so if
someone stumbles across this KIP they know the name of the other one?
2. Did we cover what "experimental" means here? I think Ismael asked
> Can we talk a bit more about the compatibility guarantees while this
image is still experimental?
I took this to mean should we be able to upgrade to or from this image or
if clusters running on it can only run on it? Or if there are no
guarantees about the upgrade/downgrade story.
Thanks,
Justine
On Sun, Nov 19, 2023 at 7:54 PM Krishna Agarwal <
krishna0608agarwal@gmail.com> wrote:
> Hi,
> Thanks for the insightful feedback on this KIP.
> As there are no ongoing discussions, I'm considering moving into the voting
> process.
> Your continued input is greatly appreciated!
>
> Regards,
> Krishna
>
> On Fri, Sep 8, 2023 at 12:47 PM Krishna Agarwal <
> krishna0608agarwal@gmail.com> wrote:
>
> > Hi,
> > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > image.
> > The proposed docker image can launch brokers with sub-second startup time
> > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > binary.
> >
> > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> >
> >
> > Regards,
> > Krishna
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi,
Thanks for the insightful feedback on this KIP.
As there are no ongoing discussions, I'm considering moving into the voting
process.
Your continued input is greatly appreciated!
Regards,
Krishna
On Fri, Sep 8, 2023 at 12:47 PM Krishna Agarwal <
krishna0608agarwal@gmail.com> wrote:
> Hi,
> I want to submit a KIP to deliver an experimental Apache Kafka docker
> image.
> The proposed docker image can launch brokers with sub-second startup time
> and minimal memory footprint by leveraging a GraalVM based native Kafka
> binary.
>
> KIP-974: Docker Image for GraalVM based Native Kafka Broker
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker>
>
> Regards,
> Krishna
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Ismael Juma <me...@ismaeljuma.com>.
Hi Federico,
I suggest taking this conversation to KIP-975 and keeping this one to the
graalvm specifics.
Ismael
On Fri, Sep 8, 2023 at 2:38 AM Federico Valeri <fe...@gmail.com> wrote:
> Hi Krishna, thanks for opening this discussion.
>
> I see you created two separate KIPs (974 and 975), but there are some
> common points (build system and test plan).
>
> Currently, the Docker image used for system tests is only supported in
> that limited scope, so the maintenance burden is minimal. Providing
> official Kafka images would be much more complicated. Have you
> considered how the image rebuild process would work in case a high
> severity CVE comes out for a non Kafka image dependency? In that case,
> there will be no Kafka release.
>
> Br
> Fede
>
> On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> <kr...@gmail.com> wrote:
> >
> > Hi,
> > I want to submit a KIP to deliver an experimental Apache Kafka docker
> image.
> > The proposed docker image can launch brokers with sub-second startup time
> > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > binary.
> >
> > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> >
> >
> > Regards,
> > Krishna
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Thanks for the feedback.
I have updated the KIP with "kafka-native" as the accepted docker image
name.
Regards,
Krishna
On Sun, Oct 29, 2023 at 10:42 PM Ismael Juma <me...@ismaeljuma.com> wrote:
> I think kafka-native is clearer. Over time, the graalvm images may be used
> for production too.
>
> Ismael
>
> On Sat, Oct 28, 2023, 11:52 PM Manikumar <ma...@gmail.com>
> wrote:
>
> > Thanks for the explanation. I am fine with using ""kafka-local" as the
> > image name.
> >
> > On Fri, Oct 27, 2023 at 11:47 AM Krishna Agarwal <
> > krishna0608agarwal@gmail.com> wrote:
> >
> > > Hi Manikumar,
> > > Thanks for the feedback.
> > >
> > > This image signifies 2 things:
> > >
> > > 1. Image should be used for the local development and testing
> purposes
> > > with fast startup times. (kafka-local)
> > > 2. To achieve (1) - we are providing a native executable for Apache
> > > Kafka in the docker image. (kafka-native)
> > >
> > > While "graalvm" is the underlying tool enabling this, I'm unsure if we
> > > should explicitly mention it in the name.
> > > I'd love to hear your thoughts on this. Do you prefer "kafka-native"
> > > instead of "kafka-local"?
> > >
> > > Regards,
> > > Krishna
> > >
> > > On Fri, Oct 20, 2023 at 3:32 PM Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > > For the native AK docker image, we are considering '*kafka-local*'
> as
> > > it
> > > > clearly signifies that this image is intended exclusively for local
> > > >
> > > > I am not sure, if there is any naming pattern for graalvm based
> images.
> > > Can
> > > > we include "graalvm" to the image name like "kafka-graalvm-native".
> > > > This will clearly indicate this is graalvm based image.
> > > >
> > > >
> > > > Thanks. Regards
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> > > > krishna0608agarwal@gmail.com> wrote:
> > > >
> > > > > Hi Federico,
> > > > > Thanks for the feedback and apologies for the delay.
> > > > >
> > > > > I've included a section in the KIP on the release process. I would
> > > > greatly
> > > > > appreciate your insights after reviewing it.
> > > > >
> > > > > Regards,
> > > > > Krishna
> > > > >
> > > > > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri <
> fedevaleri@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Krishna, thanks for opening this discussion.
> > > > > >
> > > > > > I see you created two separate KIPs (974 and 975), but there are
> > some
> > > > > > common points (build system and test plan).
> > > > > >
> > > > > > Currently, the Docker image used for system tests is only
> supported
> > > in
> > > > > > that limited scope, so the maintenance burden is minimal.
> Providing
> > > > > > official Kafka images would be much more complicated. Have you
> > > > > > considered how the image rebuild process would work in case a
> high
> > > > > > severity CVE comes out for a non Kafka image dependency? In that
> > > case,
> > > > > > there will be no Kafka release.
> > > > > >
> > > > > > Br
> > > > > > Fede
> > > > > >
> > > > > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > > > > > <kr...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > > I want to submit a KIP to deliver an experimental Apache Kafka
> > > docker
> > > > > > image.
> > > > > > > The proposed docker image can launch brokers with sub-second
> > > startup
> > > > > time
> > > > > > > and minimal memory footprint by leveraging a GraalVM based
> native
> > > > Kafka
> > > > > > > binary.
> > > > > > >
> > > > > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Krishna
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Ismael Juma <me...@ismaeljuma.com>.
I think kafka-native is clearer. Over time, the graalvm images may be used
for production too.
Ismael
On Sat, Oct 28, 2023, 11:52 PM Manikumar <ma...@gmail.com> wrote:
> Thanks for the explanation. I am fine with using ""kafka-local" as the
> image name.
>
> On Fri, Oct 27, 2023 at 11:47 AM Krishna Agarwal <
> krishna0608agarwal@gmail.com> wrote:
>
> > Hi Manikumar,
> > Thanks for the feedback.
> >
> > This image signifies 2 things:
> >
> > 1. Image should be used for the local development and testing purposes
> > with fast startup times. (kafka-local)
> > 2. To achieve (1) - we are providing a native executable for Apache
> > Kafka in the docker image. (kafka-native)
> >
> > While "graalvm" is the underlying tool enabling this, I'm unsure if we
> > should explicitly mention it in the name.
> > I'd love to hear your thoughts on this. Do you prefer "kafka-native"
> > instead of "kafka-local"?
> >
> > Regards,
> > Krishna
> >
> > On Fri, Oct 20, 2023 at 3:32 PM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > > For the native AK docker image, we are considering '*kafka-local*' as
> > it
> > > clearly signifies that this image is intended exclusively for local
> > >
> > > I am not sure, if there is any naming pattern for graalvm based images.
> > Can
> > > we include "graalvm" to the image name like "kafka-graalvm-native".
> > > This will clearly indicate this is graalvm based image.
> > >
> > >
> > > Thanks. Regards
> > >
> > >
> > >
> > >
> > > On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> > > krishna0608agarwal@gmail.com> wrote:
> > >
> > > > Hi Federico,
> > > > Thanks for the feedback and apologies for the delay.
> > > >
> > > > I've included a section in the KIP on the release process. I would
> > > greatly
> > > > appreciate your insights after reviewing it.
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > > > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri <fedevaleri@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi Krishna, thanks for opening this discussion.
> > > > >
> > > > > I see you created two separate KIPs (974 and 975), but there are
> some
> > > > > common points (build system and test plan).
> > > > >
> > > > > Currently, the Docker image used for system tests is only supported
> > in
> > > > > that limited scope, so the maintenance burden is minimal. Providing
> > > > > official Kafka images would be much more complicated. Have you
> > > > > considered how the image rebuild process would work in case a high
> > > > > severity CVE comes out for a non Kafka image dependency? In that
> > case,
> > > > > there will be no Kafka release.
> > > > >
> > > > > Br
> > > > > Fede
> > > > >
> > > > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > > > > <kr...@gmail.com> wrote:
> > > > > >
> > > > > > Hi,
> > > > > > I want to submit a KIP to deliver an experimental Apache Kafka
> > docker
> > > > > image.
> > > > > > The proposed docker image can launch brokers with sub-second
> > startup
> > > > time
> > > > > > and minimal memory footprint by leveraging a GraalVM based native
> > > Kafka
> > > > > > binary.
> > > > > >
> > > > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Krishna
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Manikumar <ma...@gmail.com>.
Thanks for the explanation. I am fine with using ""kafka-local" as the
image name.
On Fri, Oct 27, 2023 at 11:47 AM Krishna Agarwal <
krishna0608agarwal@gmail.com> wrote:
> Hi Manikumar,
> Thanks for the feedback.
>
> This image signifies 2 things:
>
> 1. Image should be used for the local development and testing purposes
> with fast startup times. (kafka-local)
> 2. To achieve (1) - we are providing a native executable for Apache
> Kafka in the docker image. (kafka-native)
>
> While "graalvm" is the underlying tool enabling this, I'm unsure if we
> should explicitly mention it in the name.
> I'd love to hear your thoughts on this. Do you prefer "kafka-native"
> instead of "kafka-local"?
>
> Regards,
> Krishna
>
> On Fri, Oct 20, 2023 at 3:32 PM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi,
> >
> > > For the native AK docker image, we are considering '*kafka-local*' as
> it
> > clearly signifies that this image is intended exclusively for local
> >
> > I am not sure, if there is any naming pattern for graalvm based images.
> Can
> > we include "graalvm" to the image name like "kafka-graalvm-native".
> > This will clearly indicate this is graalvm based image.
> >
> >
> > Thanks. Regards
> >
> >
> >
> >
> > On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> > krishna0608agarwal@gmail.com> wrote:
> >
> > > Hi Federico,
> > > Thanks for the feedback and apologies for the delay.
> > >
> > > I've included a section in the KIP on the release process. I would
> > greatly
> > > appreciate your insights after reviewing it.
> > >
> > > Regards,
> > > Krishna
> > >
> > > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri <fe...@gmail.com>
> > > wrote:
> > >
> > > > Hi Krishna, thanks for opening this discussion.
> > > >
> > > > I see you created two separate KIPs (974 and 975), but there are some
> > > > common points (build system and test plan).
> > > >
> > > > Currently, the Docker image used for system tests is only supported
> in
> > > > that limited scope, so the maintenance burden is minimal. Providing
> > > > official Kafka images would be much more complicated. Have you
> > > > considered how the image rebuild process would work in case a high
> > > > severity CVE comes out for a non Kafka image dependency? In that
> case,
> > > > there will be no Kafka release.
> > > >
> > > > Br
> > > > Fede
> > > >
> > > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > > > <kr...@gmail.com> wrote:
> > > > >
> > > > > Hi,
> > > > > I want to submit a KIP to deliver an experimental Apache Kafka
> docker
> > > > image.
> > > > > The proposed docker image can launch brokers with sub-second
> startup
> > > time
> > > > > and minimal memory footprint by leveraging a GraalVM based native
> > Kafka
> > > > > binary.
> > > > >
> > > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > > >
> > > > >
> > > > > Regards,
> > > > > Krishna
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi Manikumar,
Thanks for the feedback.
This image signifies 2 things:
1. Image should be used for the local development and testing purposes
with fast startup times. (kafka-local)
2. To achieve (1) - we are providing a native executable for Apache
Kafka in the docker image. (kafka-native)
While "graalvm" is the underlying tool enabling this, I'm unsure if we
should explicitly mention it in the name.
I'd love to hear your thoughts on this. Do you prefer "kafka-native"
instead of "kafka-local"?
Regards,
Krishna
On Fri, Oct 20, 2023 at 3:32 PM Manikumar <ma...@gmail.com> wrote:
> Hi,
>
> > For the native AK docker image, we are considering '*kafka-local*' as it
> clearly signifies that this image is intended exclusively for local
>
> I am not sure, if there is any naming pattern for graalvm based images. Can
> we include "graalvm" to the image name like "kafka-graalvm-native".
> This will clearly indicate this is graalvm based image.
>
>
> Thanks. Regards
>
>
>
>
> On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> krishna0608agarwal@gmail.com> wrote:
>
> > Hi Federico,
> > Thanks for the feedback and apologies for the delay.
> >
> > I've included a section in the KIP on the release process. I would
> greatly
> > appreciate your insights after reviewing it.
> >
> > Regards,
> > Krishna
> >
> > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri <fe...@gmail.com>
> > wrote:
> >
> > > Hi Krishna, thanks for opening this discussion.
> > >
> > > I see you created two separate KIPs (974 and 975), but there are some
> > > common points (build system and test plan).
> > >
> > > Currently, the Docker image used for system tests is only supported in
> > > that limited scope, so the maintenance burden is minimal. Providing
> > > official Kafka images would be much more complicated. Have you
> > > considered how the image rebuild process would work in case a high
> > > severity CVE comes out for a non Kafka image dependency? In that case,
> > > there will be no Kafka release.
> > >
> > > Br
> > > Fede
> > >
> > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > > <kr...@gmail.com> wrote:
> > > >
> > > > Hi,
> > > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > > image.
> > > > The proposed docker image can launch brokers with sub-second startup
> > time
> > > > and minimal memory footprint by leveraging a GraalVM based native
> Kafka
> > > > binary.
> > > >
> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > >
> > > >
> > > > Regards,
> > > > Krishna
> > >
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Federico Valeri <fe...@gmail.com>.
Hi Krishna, thanks for the updates, LGTM.
On Mon, Oct 30, 2023 at 3:36 PM Krishna Agarwal
<kr...@gmail.com> wrote:
>
> Hi Federico,
> Thanks for the feedback.
>
> 1. Yes, we will add the building, testing and scanning automation for
> this Docker Image along with the flow mentioned in KIP-975. (Updated in the
> KIP)
> 2. Added the other alternatives to the "rejected alternatives" section,
> instead of the main sections. (Updated in the KIP)
> 3. Regarding the release process- In the KIP-975, it was concluded that
> there shouldn't be any docker specific release process. If there is a high
> severity CVE, we should release a new version of Kafka for the affected
> branch. It would include the latest Kafka code from the branch. In my
> opinion we should keep the same release process here for consistency.
> (Updated in the KIP)
> KIP-975 Release Process:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-ReleaseProcess
> Discussion thread for the same:
> https://lists.apache.org/thread/05t8ccvhp3fotfftgm7dzn8wobkl59l4
>
>
> Regards,
> Krishna
>
> On Wed, Oct 25, 2023 at 9:50 PM Federico Valeri <fe...@gmail.com>
> wrote:
>
> > Hi Krishna, thanks for updating the KIP and all the work you are
> > putting into that.
> >
> > The release process LGTM. In the other KIP I see that there will be
> > some automation for building, testing and scanning for CVEs. Is this
> > also true for native images?
> >
> > I see you are proposing to use Alpine as the base image. I would add
> > Distroless to the rejected alternatives with the motivation. Maybe we
> > can do the same for the GraalVM distribution of choice.
> >
> > On Fri, Oct 20, 2023 at 12:02 PM Manikumar <ma...@gmail.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > > For the native AK docker image, we are considering '*kafka-local*' as
> > it
> > > clearly signifies that this image is intended exclusively for local
> > >
> > > I am not sure, if there is any naming pattern for graalvm based images.
> > Can
> > > we include "graalvm" to the image name like "kafka-graalvm-native".
> > > This will clearly indicate this is graalvm based image.
> > >
> > >
> > > Thanks. Regards
> > >
> > >
> > >
> > >
> > > On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> > > krishna0608agarwal@gmail.com> wrote:
> > >
> > > > Hi Federico,
> > > > Thanks for the feedback and apologies for the delay.
> > > >
> > > > I've included a section in the KIP on the release process. I would
> > greatly
> > > > appreciate your insights after reviewing it.
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > > > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri <fe...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Krishna, thanks for opening this discussion.
> > > > >
> > > > > I see you created two separate KIPs (974 and 975), but there are some
> > > > > common points (build system and test plan).
> > > > >
> > > > > Currently, the Docker image used for system tests is only supported
> > in
> > > > > that limited scope, so the maintenance burden is minimal. Providing
> > > > > official Kafka images would be much more complicated. Have you
> > > > > considered how the image rebuild process would work in case a high
> > > > > severity CVE comes out for a non Kafka image dependency? In that
> > case,
> > > > > there will be no Kafka release.
> > > > >
> > > > > Br
> > > > > Fede
> > > > >
> > > > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > > > > <kr...@gmail.com> wrote:
> > > > > >
> > > > > > Hi,
> > > > > > I want to submit a KIP to deliver an experimental Apache Kafka
> > docker
> > > > > image.
> > > > > > The proposed docker image can launch brokers with sub-second
> > startup
> > > > time
> > > > > > and minimal memory footprint by leveraging a GraalVM based native
> > Kafka
> > > > > > binary.
> > > > > >
> > > > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > > > <
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Krishna
> > > > >
> > > >
> >
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi Federico,
Thanks for the feedback.
1. Yes, we will add the building, testing and scanning automation for
this Docker Image along with the flow mentioned in KIP-975. (Updated in the
KIP)
2. Added the other alternatives to the "rejected alternatives" section,
instead of the main sections. (Updated in the KIP)
3. Regarding the release process- In the KIP-975, it was concluded that
there shouldn't be any docker specific release process. If there is a high
severity CVE, we should release a new version of Kafka for the affected
branch. It would include the latest Kafka code from the branch. In my
opinion we should keep the same release process here for consistency.
(Updated in the KIP)
KIP-975 Release Process:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-ReleaseProcess
Discussion thread for the same:
https://lists.apache.org/thread/05t8ccvhp3fotfftgm7dzn8wobkl59l4
Regards,
Krishna
On Wed, Oct 25, 2023 at 9:50 PM Federico Valeri <fe...@gmail.com>
wrote:
> Hi Krishna, thanks for updating the KIP and all the work you are
> putting into that.
>
> The release process LGTM. In the other KIP I see that there will be
> some automation for building, testing and scanning for CVEs. Is this
> also true for native images?
>
> I see you are proposing to use Alpine as the base image. I would add
> Distroless to the rejected alternatives with the motivation. Maybe we
> can do the same for the GraalVM distribution of choice.
>
> On Fri, Oct 20, 2023 at 12:02 PM Manikumar <ma...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > > For the native AK docker image, we are considering '*kafka-local*' as
> it
> > clearly signifies that this image is intended exclusively for local
> >
> > I am not sure, if there is any naming pattern for graalvm based images.
> Can
> > we include "graalvm" to the image name like "kafka-graalvm-native".
> > This will clearly indicate this is graalvm based image.
> >
> >
> > Thanks. Regards
> >
> >
> >
> >
> > On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> > krishna0608agarwal@gmail.com> wrote:
> >
> > > Hi Federico,
> > > Thanks for the feedback and apologies for the delay.
> > >
> > > I've included a section in the KIP on the release process. I would
> greatly
> > > appreciate your insights after reviewing it.
> > >
> > > Regards,
> > > Krishna
> > >
> > > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri <fe...@gmail.com>
> > > wrote:
> > >
> > > > Hi Krishna, thanks for opening this discussion.
> > > >
> > > > I see you created two separate KIPs (974 and 975), but there are some
> > > > common points (build system and test plan).
> > > >
> > > > Currently, the Docker image used for system tests is only supported
> in
> > > > that limited scope, so the maintenance burden is minimal. Providing
> > > > official Kafka images would be much more complicated. Have you
> > > > considered how the image rebuild process would work in case a high
> > > > severity CVE comes out for a non Kafka image dependency? In that
> case,
> > > > there will be no Kafka release.
> > > >
> > > > Br
> > > > Fede
> > > >
> > > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > > > <kr...@gmail.com> wrote:
> > > > >
> > > > > Hi,
> > > > > I want to submit a KIP to deliver an experimental Apache Kafka
> docker
> > > > image.
> > > > > The proposed docker image can launch brokers with sub-second
> startup
> > > time
> > > > > and minimal memory footprint by leveraging a GraalVM based native
> Kafka
> > > > > binary.
> > > > >
> > > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > > >
> > > > >
> > > > > Regards,
> > > > > Krishna
> > > >
> > >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Federico Valeri <fe...@gmail.com>.
Hi Krishna, thanks for updating the KIP and all the work you are
putting into that.
The release process LGTM. In the other KIP I see that there will be
some automation for building, testing and scanning for CVEs. Is this
also true for native images?
I see you are proposing to use Alpine as the base image. I would add
Distroless to the rejected alternatives with the motivation. Maybe we
can do the same for the GraalVM distribution of choice.
On Fri, Oct 20, 2023 at 12:02 PM Manikumar <ma...@gmail.com> wrote:
>
> Hi,
>
> > For the native AK docker image, we are considering '*kafka-local*' as it
> clearly signifies that this image is intended exclusively for local
>
> I am not sure, if there is any naming pattern for graalvm based images. Can
> we include "graalvm" to the image name like "kafka-graalvm-native".
> This will clearly indicate this is graalvm based image.
>
>
> Thanks. Regards
>
>
>
>
> On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> krishna0608agarwal@gmail.com> wrote:
>
> > Hi Federico,
> > Thanks for the feedback and apologies for the delay.
> >
> > I've included a section in the KIP on the release process. I would greatly
> > appreciate your insights after reviewing it.
> >
> > Regards,
> > Krishna
> >
> > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri <fe...@gmail.com>
> > wrote:
> >
> > > Hi Krishna, thanks for opening this discussion.
> > >
> > > I see you created two separate KIPs (974 and 975), but there are some
> > > common points (build system and test plan).
> > >
> > > Currently, the Docker image used for system tests is only supported in
> > > that limited scope, so the maintenance burden is minimal. Providing
> > > official Kafka images would be much more complicated. Have you
> > > considered how the image rebuild process would work in case a high
> > > severity CVE comes out for a non Kafka image dependency? In that case,
> > > there will be no Kafka release.
> > >
> > > Br
> > > Fede
> > >
> > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > > <kr...@gmail.com> wrote:
> > > >
> > > > Hi,
> > > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > > image.
> > > > The proposed docker image can launch brokers with sub-second startup
> > time
> > > > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > > > binary.
> > > >
> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > <
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > >
> > > >
> > > > Regards,
> > > > Krishna
> > >
> >
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Manikumar <ma...@gmail.com>.
Hi,
> For the native AK docker image, we are considering '*kafka-local*' as it
clearly signifies that this image is intended exclusively for local
I am not sure, if there is any naming pattern for graalvm based images. Can
we include "graalvm" to the image name like "kafka-graalvm-native".
This will clearly indicate this is graalvm based image.
Thanks. Regards
On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
krishna0608agarwal@gmail.com> wrote:
> Hi Federico,
> Thanks for the feedback and apologies for the delay.
>
> I've included a section in the KIP on the release process. I would greatly
> appreciate your insights after reviewing it.
>
> Regards,
> Krishna
>
> On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri <fe...@gmail.com>
> wrote:
>
> > Hi Krishna, thanks for opening this discussion.
> >
> > I see you created two separate KIPs (974 and 975), but there are some
> > common points (build system and test plan).
> >
> > Currently, the Docker image used for system tests is only supported in
> > that limited scope, so the maintenance burden is minimal. Providing
> > official Kafka images would be much more complicated. Have you
> > considered how the image rebuild process would work in case a high
> > severity CVE comes out for a non Kafka image dependency? In that case,
> > there will be no Kafka release.
> >
> > Br
> > Fede
> >
> > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > <kr...@gmail.com> wrote:
> > >
> > > Hi,
> > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > image.
> > > The proposed docker image can launch brokers with sub-second startup
> time
> > > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > > binary.
> > >
> > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > >
> > >
> > > Regards,
> > > Krishna
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi Federico,
Thanks for the feedback and apologies for the delay.
I've included a section in the KIP on the release process. I would greatly
appreciate your insights after reviewing it.
Regards,
Krishna
On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri <fe...@gmail.com> wrote:
> Hi Krishna, thanks for opening this discussion.
>
> I see you created two separate KIPs (974 and 975), but there are some
> common points (build system and test plan).
>
> Currently, the Docker image used for system tests is only supported in
> that limited scope, so the maintenance burden is minimal. Providing
> official Kafka images would be much more complicated. Have you
> considered how the image rebuild process would work in case a high
> severity CVE comes out for a non Kafka image dependency? In that case,
> there will be no Kafka release.
>
> Br
> Fede
>
> On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> <kr...@gmail.com> wrote:
> >
> > Hi,
> > I want to submit a KIP to deliver an experimental Apache Kafka docker
> image.
> > The proposed docker image can launch brokers with sub-second startup time
> > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > binary.
> >
> > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> >
> >
> > Regards,
> > Krishna
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Federico Valeri <fe...@gmail.com>.
Hi Krishna, thanks for opening this discussion.
I see you created two separate KIPs (974 and 975), but there are some
common points (build system and test plan).
Currently, the Docker image used for system tests is only supported in
that limited scope, so the maintenance burden is minimal. Providing
official Kafka images would be much more complicated. Have you
considered how the image rebuild process would work in case a high
severity CVE comes out for a non Kafka image dependency? In that case,
there will be no Kafka release.
Br
Fede
On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
<kr...@gmail.com> wrote:
>
> Hi,
> I want to submit a KIP to deliver an experimental Apache Kafka docker image.
> The proposed docker image can launch brokers with sub-second startup time
> and minimal memory footprint by leveraging a GraalVM based native Kafka
> binary.
>
> KIP-974: Docker Image for GraalVM based Native Kafka Broker
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker>
>
> Regards,
> Krishna
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Justine Olshan <jo...@confluent.io.INVALID>.
Thanks Krishna. I will go ahead and vote.
Justine
On Fri, Feb 2, 2024 at 1:30 AM Krishna Agarwal <kr...@gmail.com>
wrote:
> Hi Justine,
> Thanks for the feedback.
>
> I have added the above comparison results to the KIP.
>
> Given the widespread use of Alpine, the likelihood of encountering issues
> is minimal. Ubuntu, being a more comprehensive image, is expected to be
> lesser prone to such challenges.
> Considering that we are initially releasing a GraalVM-based Docker image
> for local usage, I am inclined to prioritise minimising its size, which can
> be achieved by using the Alpine image. Should future instances suggest the
> need for a different Docker base image for optimisation, we can explore
> alternatives at that time.
>
> Regards,
> Krishna
>
> On Thu, Feb 1, 2024 at 12:32 AM Justine Olshan
> <jo...@confluent.io.invalid>
> wrote:
>
> > Hey Krishna,
> >
> > Can we include the perf results between the distroless/alpine/ubuntu
> images
> > in the KIP?
> >
> > I also noticed
> >
> > - Alpine employs the apk package manager, which, being relatively less
> > popular, may pose challenges in the future. There's a potential risk
> > that
> > certain libraries we might need could lack support from apk
> >
> >
> > Is this a concern we would have with the other images?
> >
> > Thanks,
> > Justine
> >
> > On Tue, Dec 12, 2023 at 9:34 AM Krishna Agarwal <
> > krishna0608agarwal@gmail.com> wrote:
> >
> > > Hi Ismael,
> > > Would you happen to have any remaining concerns regarding the selection
> > of
> > > the base Docker image?
> > > Alternatively, do you have any additional suggestions or insights?
> > >
> > > Regards,
> > > Krishna
> > >
> > >
> > > On Fri, Nov 24, 2023 at 1:16 AM Krishna Agarwal <
> > > krishna0608agarwal@gmail.com> wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > In my pursuit of a lightweight base image, I initially considered
> > Alpine
> > > > and Distroless
> > > >
> > > > 1. The next best option I explored is the Ubuntu Docker image(
> > > > https://hub.docker.com/_/ubuntu/tags) which is a more complete
> > image.
> > > > It has a size of 70MB compared to the 15MB of the Alpine image
> > > > (post-installation of glibc and bash), resulting in a difference
> of
> > > 55MB.
> > > > 2. To assess performance, I executed produce/consume performance
> > > > scripts on the Kafka native Docker image using both Alpine and
> > > Ubuntu, and
> > > > the results indicated comparable performance between the two.
> > > >
> > > > I wanted to check if there's any other image you'd like me to assess
> > for
> > > > consideration. Your input would be greatly appreciated.
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > > > On Thu, Nov 23, 2023 at 2:31 AM Ismael Juma <me...@ismaeljuma.com>
> wrote:
> > > >
> > > >> Hi Krishna,
> > > >>
> > > >> I am still finding it difficult to evaluate this choice. A couple of
> > > >> things
> > > >> would help:
> > > >>
> > > >> 1. How much smaller is the alpine image compared to the best
> > > alternative?
> > > >> 2. Is there any performance impact of going with Alpine?
> > > >>
> > > >> Ismael
> > > >>
> > > >>
> > > >> On Wed, Nov 22, 2023, 8:42 AM Krishna Agarwal <
> > > >> krishna0608agarwal@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Ismael,
> > > >> > Thanks for the feedback.
> > > >> >
> > > >> > The alpine image does present a few drawbacks, such as the use of
> > musl
> > > >> libc
> > > >> > instead of glibc, the absence of bash, and reliance on the less
> > > popular
> > > >> > package manager "apk". Considering the advantage of a smaller
> image
> > > size
> > > >> > and installing the missing packages(glibc and bash), I have
> proposed
> > > the
> > > >> > alpine image as the base image. Let me know if you have any
> > > suggestions.
> > > >> > I have added a detailed section for the same in the KIP.
> > > >> >
> > > >> > Regards,
> > > >> > Krishna
> > > >> >
> > > >> > On Wed, Nov 22, 2023 at 8:08 PM Ismael Juma <me...@ismaeljuma.com>
> > > wrote:
> > > >> >
> > > >> > > Hi,
> > > >> > >
> > > >> > > One question I have is regarding the choice to use alpine - it
> > would
> > > >> be
> > > >> > > good to clarify if there are downsides (the upside was
> explained -
> > > >> images
> > > >> > > are smaller).
> > > >> > >
> > > >> > > Ismael
> > > >> > >
> > > >> > > On Fri, Sep 8, 2023, 12:17 AM Krishna Agarwal <
> > > >> > > krishna0608agarwal@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi,
> > > >> > > > I want to submit a KIP to deliver an experimental Apache Kafka
> > > >> docker
> > > >> > > > image.
> > > >> > > > The proposed docker image can launch brokers with sub-second
> > > startup
> > > >> > time
> > > >> > > > and minimal memory footprint by leveraging a GraalVM based
> > native
> > > >> Kafka
> > > >> > > > binary.
> > > >> > > >
> > > >> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > >> > > > <
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > >> > > > >
> > > >> > > >
> > > >> > > > Regards,
> > > >> > > > Krishna
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi Justine,
Thanks for the feedback.
I have added the above comparison results to the KIP.
Given the widespread use of Alpine, the likelihood of encountering issues
is minimal. Ubuntu, being a more comprehensive image, is expected to be
lesser prone to such challenges.
Considering that we are initially releasing a GraalVM-based Docker image
for local usage, I am inclined to prioritise minimising its size, which can
be achieved by using the Alpine image. Should future instances suggest the
need for a different Docker base image for optimisation, we can explore
alternatives at that time.
Regards,
Krishna
On Thu, Feb 1, 2024 at 12:32 AM Justine Olshan <jo...@confluent.io.invalid>
wrote:
> Hey Krishna,
>
> Can we include the perf results between the distroless/alpine/ubuntu images
> in the KIP?
>
> I also noticed
>
> - Alpine employs the apk package manager, which, being relatively less
> popular, may pose challenges in the future. There's a potential risk
> that
> certain libraries we might need could lack support from apk
>
>
> Is this a concern we would have with the other images?
>
> Thanks,
> Justine
>
> On Tue, Dec 12, 2023 at 9:34 AM Krishna Agarwal <
> krishna0608agarwal@gmail.com> wrote:
>
> > Hi Ismael,
> > Would you happen to have any remaining concerns regarding the selection
> of
> > the base Docker image?
> > Alternatively, do you have any additional suggestions or insights?
> >
> > Regards,
> > Krishna
> >
> >
> > On Fri, Nov 24, 2023 at 1:16 AM Krishna Agarwal <
> > krishna0608agarwal@gmail.com> wrote:
> >
> > > Hi Ismael,
> > >
> > > In my pursuit of a lightweight base image, I initially considered
> Alpine
> > > and Distroless
> > >
> > > 1. The next best option I explored is the Ubuntu Docker image(
> > > https://hub.docker.com/_/ubuntu/tags) which is a more complete
> image.
> > > It has a size of 70MB compared to the 15MB of the Alpine image
> > > (post-installation of glibc and bash), resulting in a difference of
> > 55MB.
> > > 2. To assess performance, I executed produce/consume performance
> > > scripts on the Kafka native Docker image using both Alpine and
> > Ubuntu, and
> > > the results indicated comparable performance between the two.
> > >
> > > I wanted to check if there's any other image you'd like me to assess
> for
> > > consideration. Your input would be greatly appreciated.
> > >
> > > Regards,
> > > Krishna
> > >
> > > On Thu, Nov 23, 2023 at 2:31 AM Ismael Juma <me...@ismaeljuma.com> wrote:
> > >
> > >> Hi Krishna,
> > >>
> > >> I am still finding it difficult to evaluate this choice. A couple of
> > >> things
> > >> would help:
> > >>
> > >> 1. How much smaller is the alpine image compared to the best
> > alternative?
> > >> 2. Is there any performance impact of going with Alpine?
> > >>
> > >> Ismael
> > >>
> > >>
> > >> On Wed, Nov 22, 2023, 8:42 AM Krishna Agarwal <
> > >> krishna0608agarwal@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Ismael,
> > >> > Thanks for the feedback.
> > >> >
> > >> > The alpine image does present a few drawbacks, such as the use of
> musl
> > >> libc
> > >> > instead of glibc, the absence of bash, and reliance on the less
> > popular
> > >> > package manager "apk". Considering the advantage of a smaller image
> > size
> > >> > and installing the missing packages(glibc and bash), I have proposed
> > the
> > >> > alpine image as the base image. Let me know if you have any
> > suggestions.
> > >> > I have added a detailed section for the same in the KIP.
> > >> >
> > >> > Regards,
> > >> > Krishna
> > >> >
> > >> > On Wed, Nov 22, 2023 at 8:08 PM Ismael Juma <me...@ismaeljuma.com>
> > wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > One question I have is regarding the choice to use alpine - it
> would
> > >> be
> > >> > > good to clarify if there are downsides (the upside was explained -
> > >> images
> > >> > > are smaller).
> > >> > >
> > >> > > Ismael
> > >> > >
> > >> > > On Fri, Sep 8, 2023, 12:17 AM Krishna Agarwal <
> > >> > > krishna0608agarwal@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi,
> > >> > > > I want to submit a KIP to deliver an experimental Apache Kafka
> > >> docker
> > >> > > > image.
> > >> > > > The proposed docker image can launch brokers with sub-second
> > startup
> > >> > time
> > >> > > > and minimal memory footprint by leveraging a GraalVM based
> native
> > >> Kafka
> > >> > > > binary.
> > >> > > >
> > >> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > >> > > > <
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > >> > > > >
> > >> > > >
> > >> > > > Regards,
> > >> > > > Krishna
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Justine Olshan <jo...@confluent.io.INVALID>.
Hey Krishna,
Can we include the perf results between the distroless/alpine/ubuntu images
in the KIP?
I also noticed
- Alpine employs the apk package manager, which, being relatively less
popular, may pose challenges in the future. There's a potential risk that
certain libraries we might need could lack support from apk
Is this a concern we would have with the other images?
Thanks,
Justine
On Tue, Dec 12, 2023 at 9:34 AM Krishna Agarwal <
krishna0608agarwal@gmail.com> wrote:
> Hi Ismael,
> Would you happen to have any remaining concerns regarding the selection of
> the base Docker image?
> Alternatively, do you have any additional suggestions or insights?
>
> Regards,
> Krishna
>
>
> On Fri, Nov 24, 2023 at 1:16 AM Krishna Agarwal <
> krishna0608agarwal@gmail.com> wrote:
>
> > Hi Ismael,
> >
> > In my pursuit of a lightweight base image, I initially considered Alpine
> > and Distroless
> >
> > 1. The next best option I explored is the Ubuntu Docker image(
> > https://hub.docker.com/_/ubuntu/tags) which is a more complete image.
> > It has a size of 70MB compared to the 15MB of the Alpine image
> > (post-installation of glibc and bash), resulting in a difference of
> 55MB.
> > 2. To assess performance, I executed produce/consume performance
> > scripts on the Kafka native Docker image using both Alpine and
> Ubuntu, and
> > the results indicated comparable performance between the two.
> >
> > I wanted to check if there's any other image you'd like me to assess for
> > consideration. Your input would be greatly appreciated.
> >
> > Regards,
> > Krishna
> >
> > On Thu, Nov 23, 2023 at 2:31 AM Ismael Juma <me...@ismaeljuma.com> wrote:
> >
> >> Hi Krishna,
> >>
> >> I am still finding it difficult to evaluate this choice. A couple of
> >> things
> >> would help:
> >>
> >> 1. How much smaller is the alpine image compared to the best
> alternative?
> >> 2. Is there any performance impact of going with Alpine?
> >>
> >> Ismael
> >>
> >>
> >> On Wed, Nov 22, 2023, 8:42 AM Krishna Agarwal <
> >> krishna0608agarwal@gmail.com>
> >> wrote:
> >>
> >> > Hi Ismael,
> >> > Thanks for the feedback.
> >> >
> >> > The alpine image does present a few drawbacks, such as the use of musl
> >> libc
> >> > instead of glibc, the absence of bash, and reliance on the less
> popular
> >> > package manager "apk". Considering the advantage of a smaller image
> size
> >> > and installing the missing packages(glibc and bash), I have proposed
> the
> >> > alpine image as the base image. Let me know if you have any
> suggestions.
> >> > I have added a detailed section for the same in the KIP.
> >> >
> >> > Regards,
> >> > Krishna
> >> >
> >> > On Wed, Nov 22, 2023 at 8:08 PM Ismael Juma <me...@ismaeljuma.com>
> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > One question I have is regarding the choice to use alpine - it would
> >> be
> >> > > good to clarify if there are downsides (the upside was explained -
> >> images
> >> > > are smaller).
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Fri, Sep 8, 2023, 12:17 AM Krishna Agarwal <
> >> > > krishna0608agarwal@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi,
> >> > > > I want to submit a KIP to deliver an experimental Apache Kafka
> >> docker
> >> > > > image.
> >> > > > The proposed docker image can launch brokers with sub-second
> startup
> >> > time
> >> > > > and minimal memory footprint by leveraging a GraalVM based native
> >> Kafka
> >> > > > binary.
> >> > > >
> >> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> >> > > > <
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> >> > > > >
> >> > > >
> >> > > > Regards,
> >> > > > Krishna
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi Ismael,
Would you happen to have any remaining concerns regarding the selection of
the base Docker image?
Alternatively, do you have any additional suggestions or insights?
Regards,
Krishna
On Fri, Nov 24, 2023 at 1:16 AM Krishna Agarwal <
krishna0608agarwal@gmail.com> wrote:
> Hi Ismael,
>
> In my pursuit of a lightweight base image, I initially considered Alpine
> and Distroless
>
> 1. The next best option I explored is the Ubuntu Docker image(
> https://hub.docker.com/_/ubuntu/tags) which is a more complete image.
> It has a size of 70MB compared to the 15MB of the Alpine image
> (post-installation of glibc and bash), resulting in a difference of 55MB.
> 2. To assess performance, I executed produce/consume performance
> scripts on the Kafka native Docker image using both Alpine and Ubuntu, and
> the results indicated comparable performance between the two.
>
> I wanted to check if there's any other image you'd like me to assess for
> consideration. Your input would be greatly appreciated.
>
> Regards,
> Krishna
>
> On Thu, Nov 23, 2023 at 2:31 AM Ismael Juma <me...@ismaeljuma.com> wrote:
>
>> Hi Krishna,
>>
>> I am still finding it difficult to evaluate this choice. A couple of
>> things
>> would help:
>>
>> 1. How much smaller is the alpine image compared to the best alternative?
>> 2. Is there any performance impact of going with Alpine?
>>
>> Ismael
>>
>>
>> On Wed, Nov 22, 2023, 8:42 AM Krishna Agarwal <
>> krishna0608agarwal@gmail.com>
>> wrote:
>>
>> > Hi Ismael,
>> > Thanks for the feedback.
>> >
>> > The alpine image does present a few drawbacks, such as the use of musl
>> libc
>> > instead of glibc, the absence of bash, and reliance on the less popular
>> > package manager "apk". Considering the advantage of a smaller image size
>> > and installing the missing packages(glibc and bash), I have proposed the
>> > alpine image as the base image. Let me know if you have any suggestions.
>> > I have added a detailed section for the same in the KIP.
>> >
>> > Regards,
>> > Krishna
>> >
>> > On Wed, Nov 22, 2023 at 8:08 PM Ismael Juma <me...@ismaeljuma.com> wrote:
>> >
>> > > Hi,
>> > >
>> > > One question I have is regarding the choice to use alpine - it would
>> be
>> > > good to clarify if there are downsides (the upside was explained -
>> images
>> > > are smaller).
>> > >
>> > > Ismael
>> > >
>> > > On Fri, Sep 8, 2023, 12:17 AM Krishna Agarwal <
>> > > krishna0608agarwal@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > > I want to submit a KIP to deliver an experimental Apache Kafka
>> docker
>> > > > image.
>> > > > The proposed docker image can launch brokers with sub-second startup
>> > time
>> > > > and minimal memory footprint by leveraging a GraalVM based native
>> Kafka
>> > > > binary.
>> > > >
>> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
>> > > > <
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
>> > > > >
>> > > >
>> > > > Regards,
>> > > > Krishna
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi Ismael,
In my pursuit of a lightweight base image, I initially considered Alpine
and Distroless
1. The next best option I explored is the Ubuntu Docker image(
https://hub.docker.com/_/ubuntu/tags) which is a more complete image.
It has a size of 70MB compared to the 15MB of the Alpine image
(post-installation of glibc and bash), resulting in a difference of 55MB.
2. To assess performance, I executed produce/consume performance scripts
on the Kafka native Docker image using both Alpine and Ubuntu, and the
results indicated comparable performance between the two.
I wanted to check if there's any other image you'd like me to assess for
consideration. Your input would be greatly appreciated.
Regards,
Krishna
On Thu, Nov 23, 2023 at 2:31 AM Ismael Juma <me...@ismaeljuma.com> wrote:
> Hi Krishna,
>
> I am still finding it difficult to evaluate this choice. A couple of things
> would help:
>
> 1. How much smaller is the alpine image compared to the best alternative?
> 2. Is there any performance impact of going with Alpine?
>
> Ismael
>
>
> On Wed, Nov 22, 2023, 8:42 AM Krishna Agarwal <
> krishna0608agarwal@gmail.com>
> wrote:
>
> > Hi Ismael,
> > Thanks for the feedback.
> >
> > The alpine image does present a few drawbacks, such as the use of musl
> libc
> > instead of glibc, the absence of bash, and reliance on the less popular
> > package manager "apk". Considering the advantage of a smaller image size
> > and installing the missing packages(glibc and bash), I have proposed the
> > alpine image as the base image. Let me know if you have any suggestions.
> > I have added a detailed section for the same in the KIP.
> >
> > Regards,
> > Krishna
> >
> > On Wed, Nov 22, 2023 at 8:08 PM Ismael Juma <me...@ismaeljuma.com> wrote:
> >
> > > Hi,
> > >
> > > One question I have is regarding the choice to use alpine - it would be
> > > good to clarify if there are downsides (the upside was explained -
> images
> > > are smaller).
> > >
> > > Ismael
> > >
> > > On Fri, Sep 8, 2023, 12:17 AM Krishna Agarwal <
> > > krishna0608agarwal@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > > > image.
> > > > The proposed docker image can launch brokers with sub-second startup
> > time
> > > > and minimal memory footprint by leveraging a GraalVM based native
> Kafka
> > > > binary.
> > > >
> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > > >
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Ismael Juma <me...@ismaeljuma.com>.
Hi Krishna,
I am still finding it difficult to evaluate this choice. A couple of things
would help:
1. How much smaller is the alpine image compared to the best alternative?
2. Is there any performance impact of going with Alpine?
Ismael
On Wed, Nov 22, 2023, 8:42 AM Krishna Agarwal <kr...@gmail.com>
wrote:
> Hi Ismael,
> Thanks for the feedback.
>
> The alpine image does present a few drawbacks, such as the use of musl libc
> instead of glibc, the absence of bash, and reliance on the less popular
> package manager "apk". Considering the advantage of a smaller image size
> and installing the missing packages(glibc and bash), I have proposed the
> alpine image as the base image. Let me know if you have any suggestions.
> I have added a detailed section for the same in the KIP.
>
> Regards,
> Krishna
>
> On Wed, Nov 22, 2023 at 8:08 PM Ismael Juma <me...@ismaeljuma.com> wrote:
>
> > Hi,
> >
> > One question I have is regarding the choice to use alpine - it would be
> > good to clarify if there are downsides (the upside was explained - images
> > are smaller).
> >
> > Ismael
> >
> > On Fri, Sep 8, 2023, 12:17 AM Krishna Agarwal <
> > krishna0608agarwal@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > > image.
> > > The proposed docker image can launch brokers with sub-second startup
> time
> > > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > > binary.
> > >
> > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > >
> > >
> > > Regards,
> > > Krishna
> > >
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Krishna Agarwal <kr...@gmail.com>.
Hi Ismael,
Thanks for the feedback.
The alpine image does present a few drawbacks, such as the use of musl libc
instead of glibc, the absence of bash, and reliance on the less popular
package manager "apk". Considering the advantage of a smaller image size
and installing the missing packages(glibc and bash), I have proposed the
alpine image as the base image. Let me know if you have any suggestions.
I have added a detailed section for the same in the KIP.
Regards,
Krishna
On Wed, Nov 22, 2023 at 8:08 PM Ismael Juma <me...@ismaeljuma.com> wrote:
> Hi,
>
> One question I have is regarding the choice to use alpine - it would be
> good to clarify if there are downsides (the upside was explained - images
> are smaller).
>
> Ismael
>
> On Fri, Sep 8, 2023, 12:17 AM Krishna Agarwal <
> krishna0608agarwal@gmail.com>
> wrote:
>
> > Hi,
> > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > image.
> > The proposed docker image can launch brokers with sub-second startup time
> > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > binary.
> >
> > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > >
> >
> > Regards,
> > Krishna
> >
>
Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker
Posted by Ismael Juma <me...@ismaeljuma.com>.
Hi,
One question I have is regarding the choice to use alpine - it would be
good to clarify if there are downsides (the upside was explained - images
are smaller).
Ismael
On Fri, Sep 8, 2023, 12:17 AM Krishna Agarwal <kr...@gmail.com>
wrote:
> Hi,
> I want to submit a KIP to deliver an experimental Apache Kafka docker
> image.
> The proposed docker image can launch brokers with sub-second startup time
> and minimal memory footprint by leveraging a GraalVM based native Kafka
> binary.
>
> KIP-974: Docker Image for GraalVM based Native Kafka Broker
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> >
>
> Regards,
> Krishna
>