[DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Krish Vora <kr...@gmail.com>]

Hi everyone,

I would like to start the discussion on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka
 .

This KIP aims to introduce JVM based Docker Official Image (DOI) for Apache
Kafka.

Regards,
Krish.

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Vedarth Sharma <ve...@gmail.com>]

Thanks for providing feedback and helping in improving this KIP.

We have done a very thorough discussion on this KIP and I feel it's now
ready for the voting process.

Thanks and regards,
Vedarth


On Tue, 14 May 2024 at 20:43, Chris Egerton <ch...@aiven.io.invalid> wrote:

> Hi Vedarth,
>
> This looks great, thank you for helping me thoroughly understand the
> motivation and benefits for the KIP. Looks good to me.
>
> Regarding public interface for the images--everything in the "Public
> Interface" section in KIP-975 does qualify as public interface for the
> images IMO, but I don't think it's comprehensive. If we were asked to, for
> example, change the port in the EXPOSE directive in our Dockerfile, that
> would probably qualify as a change to public interface too. With the strict
> language in the latest draft of this KIP that ensures that any functional
> changes to our Docker images go through another follow-up KIP, we should be
> fine without having to identify a comprehensive list of everything that
> constitutes public interface for our images.
>
> Cheers, and thanks again for the KIP,
>
> Chris
>
> On Mon, May 13, 2024 at 3:07 PM Vedarth Sharma <ve...@gmail.com>
> wrote:
>
> > Hey Chris,
> >
> > Once we provide the definitions to docker, they should take care of
> > everything from there. They mentioned here
> > <
> >
> https://github.com/docker-library/official-images?tab=readme-ov-file#library-definition-files
> > >
> > that
> > the image will be rebuilt when the base image is updated. Hence active
> > rebuilds won't require any changes from our side.
> > If we are packaging something which may contain a CVE, like some jar,
> then
> > the onus will be on us to patch it, but it will be upto us whether we
> > consider the threat severe enough to fix and when we want to provide the
> > fixed version. Having Docker Official Image will not impact the frequency
> > of our releases. It will be the Apache Kafka community's call on when a
> > release goes and Docker Official Image will be released accordingly as
> per
> > the KIP. source
> > <https://github.com/docker-library/faq?tab=readme-ov-file#image-building
> >
> >
> > As mentioned in Docker's documentation as well "In essence we strive to
> > heed upstream's recommendations on how they intend for their software to
> be
> > consumed." source
> > <
> >
> https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
> > >
> > Docker Official Image will rely on upstream's recommendation for
> > functionality. But I do agree that since Docker's stance on this might
> > change in future it makes sense to put a safeguard that will not allow
> any
> > functionality changes get incorporated as part of the vetting process. I
> > have updated the KIP to reflect the same.
> >
> > KIP-975 has a well defined public interface based on how configs can be
> > supplied and how it can be used. I am not sure if we put that label on it
> > during discussions. I am happy to have a separate email thread on it to
> > iron things out.
> >
> > I hope this addresses all of your concerns!
> >
> > Thanks and regards,
> > Vedarth
> >
> > On Mon, May 13, 2024 at 10:55 PM Chris Egerton <ch...@aiven.io.invalid>
> > wrote:
> >
> > > Thanks both for your responses! Friendly reminder: again, better to
> > provide
> > > a quote instead of just a link :)
> > >
> > > I've seen a bit about image rebuilding to handle CVEs but I'm a little
> > > unclear on how this would work in practice, and I couldn't find any
> > > concrete details in any of the links. Does Docker do this automatically
> > for
> > > DOIs? Or will the onus be on us to put out patched images? Would this
> > lead
> > > to us putting out images more quickly than we put out standard
> releases?
> > As
> > > a plus, it does look like DOIs get the benefit of Docker Scout [1] for
> > > free, which is nice, but it's still unclear who'd be doing the rest of
> > the
> > > work on that front.
> > >
> > > As far as this point from Vedarth goes:
> > >
> > > > By incorporating the source code of the Docker Official Image into
> our
> > > > AK ecosystem, we gain control over its functionality, ensuring
> > alignment
> > > > with the OSS Docker image. This ensures a seamless experience for
> users
> > > who
> > > > may need to transition between these images.
> > >
> > > This captures my concern with the KIP pretty well. If there's any
> > > significant divergence in behavior (not just build methodology) between
> > the
> > > apache/kafka image and what Docker requires for a Kafka DOI, how are we
> > > going to vet these changes moving forward? Under the "Post Release
> > Process
> > > - if Dockerhub folks suggest changes to the Dockerfiles:" header, this
> > KIP
> > > proposes that we port all suggested changes for the DOI to
> > > the docker/jvm/Dockerfile image, but this seems a bit too permissive.
> As
> > an
> > > alternative, we could state that all build-related changes can be done
> > with
> > > a PR on the apache/kafka GitHub repo (which will require approval from
> a
> > > single committer), but any functional changes will require a KIP.
> > >
> > > Finally, during KIP-975 was there discussion on what we would count as
> > the
> > > public interface for the apache/kafka image? If not, it'd be nice to
> get
> > > that ironed out since it may make future discussions around our Docker
> > > images quicker, but I don't think this is necessary for KIP-1028.
> > >
> > > [1] - https://www.docker.com/products/docker-scout/
> > >
> > > On Mon, May 13, 2024 at 4:37 AM Prabha Manepalli
> > > <mp...@confluent.io.invalid> wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > Sharing the requested links explaining why Docker Official images are
> > > > considered more secure -
> > > >
> > > >
> > >
> >
> https://www.docker.com/blog/enhancing-security-and-transparency-with-docker-official-images/
> > > > and
> > > >
> > > >
> > >
> >
> https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-cves
> > > >
> > > > I hope these links help you understand why we need Docker Official
> > images
> > > > for organisations with stringent security compliance requirements for
> > > their
> > > > Kafka workloads.
> > > >
> > > > Thank you.
> > > >
> > > >
> > > >
> > > > On Sun, May 12, 2024 at 3:33 PM Vedarth Sharma <
> > vedarth.sharma@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hey Chris!
> > > > >
> > > > > Functionality wise, we don't intend to have any differences between
> > OSS
> > > > > Docker Image and Docker Official Image.
> > > > > The Docker Official Image will be the recommended one.
> > > > > Since the Docker Official Image might be delayed due to review done
> > by
> > > > > Docker, images on apache/kafka (OSS Docker Image) can be used by
> > users.
> > > > >
> > > > > 1) I read https://docs.docker.com/trusted-content/official-images/
> > and
> > > > > > didn't find anything on that page or immediately around it that
> > > > explains
> > > > > > what compliance requirements might be satisfied by a DOI that
> > > couldn't
> > > > be
> > > > > > satisfied by the existing apache/kafka image. Can you elaborate?
> > Feel
> > > > > free
> > > > > > to provide another link, but please also quote the relevant
> > sections
> > > > from
> > > > > > it (as StackOverflow likes to say, links can grow stale over
> time).
> > > > >
> > > > >
> > > > >    - If a user's selection is confined solely to Docker Official
> > > Images,
> > > > >    this Docker Image will ensure their access to Kafka.
> > > > >    - Details on specific advantages of Docker Official Images can
> be
> > > > found
> > > > >    at
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
> > > > >    .
> > > > >    - The Docker Official Image will be actively rebuilt for updates
> > and
> > > > >    security fixes.
> > > > >    - It's true we can provide exactly the same benefits in the OSS
> > > Docker
> > > > >    Image as well. But it won't have the Docker Official Image badge
> > and
> > > > it
> > > > >    will add additional overhead for Apache Kafka community.
> > > > >    - The fact that it will have Docker Official Image badge will
> set
> > it
> > > > >    apart from the OSS Docker Image.
> > > > >    - Also the ability to do just docker pull kafka to get the kafka
> > > > docker
> > > > >    image will only be possible with Docker Official Image.
> > > > >
> > > > >
> > > > > 2) It would be great to see a brief summary of the differences in
> > these
> > > > > > images included in the KIP, in order to try to gauge how this
> would
> > > > look
> > > > > to
> > > > > > our users.
> > > > >
> > > > >
> > > > >    - Functionally, both Docker images will remain identical.
> > > > >    - The variance lies primarily in the methodologies of building
> and
> > > > >    validation, as outlined in the updated KIP.
> > > > >
> > > > >
> > > > > 3) What I suggested last time was not a separate
> apache/apache-docker
> > > > > > repository, but a repository controlled entirely by Docker. The
> DOI
> > > > docs
> > > > > > [1] state that "While it's preferable to have upstream software
> > > authors
> > > > > > maintaining their Docker Official Images, this isn't a strict
> > > > > requirement",
> > > > > > which I take to mean that it's not required for an Apache Kafka
> DOI
> > > to
> > > > > live
> > > > > > under the apache organization on GitHub. It also seems like
> there's
> > > > > > precedent for this: images for MySQL [2] and PHP [3] already
> exist
> > > > under
> > > > > > the control of Docker. The reason I think this is worth
> considering
> > > is
> > > > > that
> > > > > > Docker can arbitrarily change the eligibility requirements for
> > their
> > > > > > official images at any time, and it doesn't seem like there's a
> > clear
> > > > > > process in the KIP for judging how we should respond to these
> > changes
> > > > (in
> > > > > > fact, it seems like the idea in the KIP is that we should make
> any
> > > > change
> > > > > > required with no further vetting beyond possibly a pull request
> on
> > > > > > apache/kafka, which would require approval by a committer). By
> > > hosting
> > > > > the
> > > > > > DOI definitions ourselves (either in apache/kafka, or in a
> > > theoretical
> > > > > > apache/docker-kafka repository), we take responsibility for the
> > > image,
> > > > > even
> > > > > > if the owner on Docker Hub is Docker, not Apache. If the code
> lives
> > > > > > elsewhere, then (as long as basic trademark and possibly security
> > > > > > guidelines are respected) Apache doesn't have to concern itself
> at
> > > all
> > > > > with
> > > > > > the image and the maintainers are free to make whatever changes
> > they
> > > > want
> > > > > > to it in order to meet Docker's requirements.
> > > > >
> > > > >
> > > > >    - By incorporating the source code of the Docker Official Image
> > into
> > > > our
> > > > >    AK ecosystem, we gain control over its functionality, ensuring
> > > > alignment
> > > > >    with the OSS Docker image. This ensures a seamless experience
> for
> > > > users
> > > > > who
> > > > >    may need to transition between these images.
> > > > >    - Maintaining both images within the same community facilitates
> > ease
> > > > of
> > > > >    management and fosters a singular source of truth.
> > > > >    - While Apache may not retain ownership of the hosted Docker
> > > Official
> > > > >    Image, we are, in essence, providing Docker with a foundation
> that
> > > > > aligns
> > > > >    with their established guidelines as well as remains consistent
> > with
> > > > OSS
> > > > >    Docker Image apache/kafka.
> > > > >    - Any future alterations to the functionality can be seamlessly
> > > > >    propagated across both the OSS and Official Docker Images.
> > > > >
> > > > > I think these reasons must be why a lot of the Apache projects
> choose
> > > to
> > > > > host the docker definitions themselves. The responsibility of
> owning
> > > the
> > > > > definitions comes with benefits as well that we should also
> consider.
> > > > >
> > > > > Let us know if you have any further questions!
> > > > >
> > > > > Thanks and regards,
> > > > > Vedarth
> > > > >
> > > > > On Fri, May 10, 2024 at 7:56 PM Chris Egerton
> > <chrise@aiven.io.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Krish and Prabha,
> > > > > >
> > > > > > Thanks for your replies. I still have some follow-up questions:
> > > > > >
> > > > > > 1) I read
> https://docs.docker.com/trusted-content/official-images/
> > > and
> > > > > > didn't find anything on that page or immediately around it that
> > > > explains
> > > > > > what compliance requirements might be satisfied by a DOI that
> > > couldn't
> > > > be
> > > > > > satisfied by the existing apache/kafka image. Can you elaborate?
> > Feel
> > > > > free
> > > > > > to provide another link, but please also quote the relevant
> > sections
> > > > from
> > > > > > it (as StackOverflow likes to say, links can grow stale over
> time).
> > > > > >
> > > > > > 2) It would be great to see a brief summary of the differences in
> > > these
> > > > > > images included in the KIP, in order to try to gauge how this
> would
> > > > look
> > > > > to
> > > > > > our users.
> > > > > >
> > > > > > 3) What I suggested last time was not a separate
> > apache/apache-docker
> > > > > > repository, but a repository controlled entirely by Docker. The
> DOI
> > > > docs
> > > > > > [1] state that "While it's preferable to have upstream software
> > > authors
> > > > > > maintaining their Docker Official Images, this isn't a strict
> > > > > requirement",
> > > > > > which I take to mean that it's not required for an Apache Kafka
> DOI
> > > to
> > > > > live
> > > > > > under the apache organization on GitHub. It also seems like
> there's
> > > > > > precedent for this: images for MySQL [2] and PHP [3] already
> exist
> > > > under
> > > > > > the control of Docker. The reason I think this is worth
> considering
> > > is
> > > > > that
> > > > > > Docker can arbitrarily change the eligibility requirements for
> > their
> > > > > > official images at any time, and it doesn't seem like there's a
> > clear
> > > > > > process in the KIP for judging how we should respond to these
> > changes
> > > > (in
> > > > > > fact, it seems like the idea in the KIP is that we should make
> any
> > > > change
> > > > > > required with no further vetting beyond possibly a pull request
> on
> > > > > > apache/kafka, which would require approval by a committer). By
> > > hosting
> > > > > the
> > > > > > DOI definitions ourselves (either in apache/kafka, or in a
> > > theoretical
> > > > > > apache/docker-kafka repository), we take responsibility for the
> > > image,
> > > > > even
> > > > > > if the owner on Docker Hub is Docker, not Apache. If the code
> lives
> > > > > > elsewhere, then (as long as basic trademark and possibly security
> > > > > > guidelines are respected) Apache doesn't have to concern itself
> at
> > > all
> > > > > with
> > > > > > the image and the maintainers are free to make whatever changes
> > they
> > > > want
> > > > > > to it in order to meet Docker's requirements.
> > > > > >
> > > > > > [1] -
> > > > > >
> > > https://docs.docker.com/trusted-content/official-images/contributing/,
> > > > > > second paragraph under "Contributing to Docker Official Images"
> > > > > > [2] - https://github.com/docker-library/mysql
> > > > > > [3] - https://github.com/docker-library/php
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Fri, May 10, 2024 at 7:46 AM Krish Vora <
> krishvora01@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hey Chris,
> > > > > > >
> > > > > > > We have responded to the initial round of queries. And as you
> > > > mentioned
> > > > > > in
> > > > > > > your comment, you may have more questions related to this KIP.
> > > Please
> > > > > let
> > > > > > > us know if you have any.
> > > > > > > We want to start the voting for this KIP soon, as we intend to
> > > > include
> > > > > it
> > > > > > > in the 3.8.0 release.
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Krish.
> > > > > > >
> > > > > > > On Wed, May 8, 2024 at 6:19 PM Krish Vora <
> krishvora01@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Chris. Thanks for the questions.
> > > > > > > >
> > > > > > > > 3. Would a separate Docker-owned repository be out of the
> > > question?
> > > > > I'm
> > > > > > > >> guessing there are some trademark issues that might get in
> the
> > > > way,
> > > > > > but
> > > > > > > >> it's worth exploring since the entire purpose of this KIP
> > seems
> > > to
> > > > > be
> > > > > > to
> > > > > > > >> provide images that are vetted and designed by Docker more
> > than
> > > by
> > > > > the
> > > > > > > >> Apache Kafka contributors/committers/PMC.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >    - The process for introducing a Docker Official Image
> > involves
> > > > > > > >       - Hosting the Dockerfile in the Apache Kafka repository
> > and
> > > > > > > >       - Providing the path to this Dockerfile to Docker Hub
> in
> > > > Docker
> > > > > > > >       Hub’s own repo
> > > > > > > >       <
> > > > > > >
> > > >
> https://github.com/docker-library/official-images/tree/master/library>
> > > > > > > >       .
> > > > > > > >    - This ensures that any updates to the Dockerfile in the
> AK
> > > > > > repository
> > > > > > > >    are directly applicable to the docker official images
> > > available
> > > > on
> > > > > > > Docker
> > > > > > > >    Hub.
> > > > > > > >
> > > > > > > >
> > > > > > > >    - We also did not find any added advantage to create a
> > > separate
> > > > > > > >    repository named apache-docker within the Apache GitHub
> > > > > > organization.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Krish.
> > > > > > > >
> > > > > > > > On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli
> > > > > > > > <mp...@confluent.io.invalid> wrote:
> > > > > > > >
> > > > > > > >> Hi Chris,  I would like to add more context to this KIP's
> > > > > motivation.
> > > > > > > >> Vedarth and Krish, please weigh in with your inputs.
> > > > > > > >>
> > > > > > > >> In the motivation section it's stated that "Several other
> > Apache
> > > > > > > projects,
> > > > > > > >> > like Flink, Spark, Solr, have already released Docker
> > Official
> > > > > > Images,
> > > > > > > >> with
> > > > > > > >> > download figures ranging from 50 million to over 1
> billion.
> > > > These
> > > > > > > >> numbers
> > > > > > > >> > highlight the significant demand among users." But then
> > > > > immediately
> > > > > > > >> > afterwards, we learn that "Also the Docker Official Images
> > are
> > > > > > always
> > > > > > > >> the
> > > > > > > >> > top 1 search result, irrespective of the number of
> > downloads."
> > > > > > > Wouldn't
> > > > > > > >> a
> > > > > > > >> > high number of downloads for an image naturally follow
> from
> > > > being
> > > > > > the
> > > > > > > >> top
> > > > > > > >> > search result? It seems like we can't necessarily assume
> > that
> > > > > Docker
> > > > > > > >> > Official Images are inherently more desirable for users
> > based
> > > > > solely
> > > > > > > on
> > > > > > > >> > download statistics.
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >> *My thoughts: *Unlike the Sponsored OSS image, the Docker
> > > Official
> > > > > > image
> > > > > > > >> is
> > > > > > > >> more desirable for workloads that have stringent compliance
> > > > > > > requirements.
> > > > > > > >> More details on why official images are more trusted are
> > > > documented
> > > > > > here
> > > > > > > >> <https://docs.docker.com/trusted-content/official-images/>.
> > The
> > > > > > Docker
> > > > > > > >> Official image would also help an absolutely new Kafka
> > beginner
> > > > who
> > > > > > > might
> > > > > > > >> not know about Apache or the concept of Sponsored images. We
> > > want
> > > > to
> > > > > > > make
> > > > > > > >> it easier for Kafka beginners to discover the Kafka image
> > > through
> > > > > > > >> DockerHub.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Can you elaborate on the value that these new images would
> add
> > > > from
> > > > > a
> > > > > > > >> > user's perspective? I'm hesitant to introduce another
> image,
> > > > since
> > > > > > it
> > > > > > > >> adds
> > > > > > > >> > to the cognitive burden of people who will inevitably have
> > to
> > > > > answer
> > > > > > > the
> > > > > > > >> > question of "What are the differences between all of the
> > > > available
> > > > > > > >> images
> > > > > > > >> > and which one is best for my use case?"
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> *My thoughts: *This is a valid concern to address. The
> > response
> > > to
> > > > > the
> > > > > > > >> above question addresses the value-add this new Docker
> > Official
> > > > > image
> > > > > > > >> would
> > > > > > > >> provide. I also agree we need a clear distinction between
> each
> > > of
> > > > > > these
> > > > > > > >> images to be well documented. We plan to update the AK
> website
> > > > with
> > > > > > > >> details
> > > > > > > >> on how, why, and when a developer would want to use each of
> > > these
> > > > > > > >> particular images(KIP-974,975,1028).
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Prabha.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton
> > > > > <chrise@aiven.io.invalid
> > > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Hi Vedarth and Krish,
> > > > > > > >> >
> > > > > > > >> > Thanks for the KIP! I have to admit I'm a little
> skeptical;
> > > > > > hopefully
> > > > > > > >> you
> > > > > > > >> > can help me understand the need for these additional
> images.
> > > > > > > >> >
> > > > > > > >> > 1) In the motivation section it's stated that "Several
> other
> > > > > Apache
> > > > > > > >> > projects, like Flink, Spark, Solr, have already released
> > > Docker
> > > > > > > Official
> > > > > > > >> > Images, with download figures ranging from 50 million to
> > over
> > > 1
> > > > > > > billion.
> > > > > > > >> > These numbers highlight the significant demand among
> users."
> > > But
> > > > > > then
> > > > > > > >> > immediately afterwards, we learn that "Also the Docker
> > > Official
> > > > > > Images
> > > > > > > >> are
> > > > > > > >> > always the top 1 search result, irrespective of the number
> > of
> > > > > > > >> downloads."
> > > > > > > >> > Wouldn't a high number of downloads for an image naturally
> > > > follow
> > > > > > from
> > > > > > > >> > being the top search result? It seems like we can't
> > > necessarily
> > > > > > assume
> > > > > > > >> that
> > > > > > > >> > Docker Official Images are inherently more desirable for
> > users
> > > > > based
> > > > > > > >> solely
> > > > > > > >> > on download statistics.
> > > > > > > >> >
> > > > > > > >> > 2) Can you elaborate on the value that these new images
> > would
> > > > add
> > > > > > > from a
> > > > > > > >> > user's perspective? I'm hesitant to introduce another
> image,
> > > > since
> > > > > > it
> > > > > > > >> adds
> > > > > > > >> > to the cognitive burden of people who will inevitably have
> > to
> > > > > answer
> > > > > > > the
> > > > > > > >> > question of "What are the differences between all of the
> > > > available
> > > > > > > >> images
> > > > > > > >> > and which one is best for my use case?"
> > > > > > > >> >
> > > > > > > >> > 3) Would a separate Docker-owned repository be out of the
> > > > > question?
> > > > > > > I'm
> > > > > > > >> > guessing there are some trademark issues that might get in
> > the
> > > > > way,
> > > > > > > but
> > > > > > > >> > it's worth exploring since the entire purpose of this KIP
> > > seems
> > > > to
> > > > > > be
> > > > > > > to
> > > > > > > >> > provide images that are vetted and designed by Docker more
> > > than
> > > > by
> > > > > > the
> > > > > > > >> > Apache Kafka contributors/committers/PMC.
> > > > > > > >> >
> > > > > > > >> > I may have more questions later but wanted to get this
> > initial
> > > > > round
> > > > > > > out
> > > > > > > >> > now without trying to list everything first.
> > > > > > > >> >
> > > > > > > >> > Looking forward to your thoughts!
> > > > > > > >> >
> > > > > > > >> > Cheers,
> > > > > > > >> >
> > > > > > > >> > Chris
> > > > > > > >> >
> > > > > > > >> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <
> > > > > > > >> vedarth.sharma@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hey folks,
> > > > > > > >> > >
> > > > > > > >> > > Thanks a lot for reviewing the KIP and providing
> feedback.
> > > > > > > >> > > The discussion thread seems resolved and KIP has been
> > > updated
> > > > > > > >> > accordingly.
> > > > > > > >> > > We will be starting the voting thread for this KIP in
> the
> > > next
> > > > > few
> > > > > > > >> days.
> > > > > > > >> > > Please take a look at the KIP and let us know if any
> > further
> > > > > > > >> discussion
> > > > > > > >> > > is needed.
> > > > > > > >> > >
> > > > > > > >> > > Thanks and regards,
> > > > > > > >> > > Vedarth
> > > > > > > >> > >
> > > > > > > >> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <
> > > > > > > manikumar.reddy@gmail.com>
> > > > > > > >> > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Thanks Krish. KIP looks good to me.
> > > > > > > >> > > >
> > > > > > > >> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <
> > > > > > krishvora01@gmail.com
> > > > > > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > Hi Manikumar,
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks for the comments.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Maybe as part of the release process, RM can create
> a
> > > JIRA
> > > > > for
> > > > > > > >> this
> > > > > > > >> > > > > > task. This can be taken by RM or any comitter or
> any
> > > > > > > contributor
> > > > > > > >> > > (with
> > > > > > > >> > > > > > some help from commiters to run "Docker Image
> > > > Preparation
> > > > > > via
> > > > > > > >> > GitHub
> > > > > > > >> > > > > > Actions:"
> > > > > > > >> > > > >
> > > > > > > >> > > > > This sounds like a good idea. This step would be
> > > > beneficial.
> > > > > > By
> > > > > > > >> > > creating
> > > > > > > >> > > > a
> > > > > > > >> > > > > JIRA ticket, it will also serve as a reminder to
> > > complete
> > > > > the
> > > > > > > >> > > > post-release
> > > > > > > >> > > > > steps for the Docker official images. Have updated
> the
> > > KIP
> > > > > > with
> > > > > > > >> this
> > > > > > > >> > > > step.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Is this using GitHub Actions workflow? or manual
> > > testing?
> > > > > > > >> > > > >
> > > > > > > >> > > > > This will be done by a Github Actions workflow,
> which
> > > will
> > > > > > test
> > > > > > > >> the
> > > > > > > >> > > > static
> > > > > > > >> > > > > Docker Official Image assets for a specific release
> > > > version.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Is it mandatory for RM/comitters to raise the PR to
> > > Docker
> > > > > > Hub’s
> > > > > > > >> > > > > > official images repository (or) can it be done by
> > any
> > > > > > > >> contributor.
> > > > > > > >> > > > >
> > > > > > > >> > > > > I believe that it can be done by any contributor
> (ref:
> > > > This
> > > > > > link
> > > > > > > >> > > > > <
> > > > > > > >> >
> > > > > >
> > > https://docs.docker.com/trusted-content/official-images/contributing/
> > > > > > > >> > > >
> > > > > > > >> > > > > quotes "*Anyone can provide feedback, contribute
> code,
> > > > > suggest
> > > > > > > >> > process
> > > > > > > >> > > > > changes, or even propose a new Official Image.*")
> > > > > > > >> > > > >
> > > > > > > >> > > > > Also I was thinking, once the KIP gets voted, we
> > should
> > > > try
> > > > > to
> > > > > > > >> > release
> > > > > > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This
> > > will
> > > > > help
> > > > > > > us
> > > > > > > >> to
> > > > > > > >> > > > > > validate the process and allow us to fix any
> changes
> > > > > > suggested
> > > > > > > >> by
> > > > > > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > > > > > >> > > > >
> > > > > > > >> > > > > This sounds like a great idea. This KIP proposes
> > release
> > > > of
> > > > > > DOI
> > > > > > > >> as a
> > > > > > > >> > > > > post-release process, which can be done anytime post
> > > > > release.
> > > > > > > >> Since
> > > > > > > >> > > 3.7.0
> > > > > > > >> > > > > is already released, we can perform these steps for
> > that
> > > > > > release
> > > > > > > >> too.
> > > > > > > >> > > By
> > > > > > > >> > > > > the time the KIP gets implemented, if 3.7.1 is
> > released,
> > > > we
> > > > > > > could
> > > > > > > >> do
> > > > > > > >> > > > these
> > > > > > > >> > > > > steps for 3.7.1, instead of 3.7.0. This would allow
> us
> > > to
> > > > > make
> > > > > > > >> > changes
> > > > > > > >> > > to
> > > > > > > >> > > > > the Dockerfiles and other assets based on feedback
> > from
> > > > > Docker
> > > > > > > Hub
> > > > > > > >> > > before
> > > > > > > >> > > > > the release of version 3.8.0.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > Krish.
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
> > > > > > > >> > manikumar.reddy@gmail.com>
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Hi Krish,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Thanks for the updated KIP. a few comments below.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > "These actions can be carried out by the RM or
> any
> > > > > > > contributor
> > > > > > > >> > post
> > > > > > > >> > > > the
> > > > > > > >> > > > > > release process."
> > > > > > > >> > > > > > Maybe as part of the release process, RM can
> create
> > a
> > > > JIRA
> > > > > > for
> > > > > > > >> this
> > > > > > > >> > > > > > task. This can be taken by RM or any comitter or
> any
> > > > > > > contributor
> > > > > > > >> > > (with
> > > > > > > >> > > > > > some help from commiters to run "Docker Image
> > > > Preparation
> > > > > > via
> > > > > > > >> > GitHub
> > > > > > > >> > > > > > Actions:"
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > "Perform Docker build tests to ensure image
> > > integrity"
> > > > > > > >> > > > > > Is this using GitHub Actions workflow? or manual
> > > > testing?
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > "The RM will manually raise the final PR to
> Docker
> > > > Hub’s
> > > > > > > >> official
> > > > > > > >> > > > images
> > > > > > > >> > > > > > repository using the contents of the generated
> file"
> > > > > > > >> > > > > >  Is it mandatory for RM/comitters to raise the PR
> to
> > > > > Docker
> > > > > > > >> Hub’s
> > > > > > > >> > > > > > official images repository (or) can it be done by
> > any
> > > > > > > >> contributor.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Also I was thinking, once the KIP gets voted, we
> > > should
> > > > > try
> > > > > > to
> > > > > > > >> > > release
> > > > > > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This
> > > will
> > > > > help
> > > > > > > us
> > > > > > > >> to
> > > > > > > >> > > > > > validate the process and allow us to fix any
> changes
> > > > > > suggested
> > > > > > > >> by
> > > > > > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <
> > > > > > > >> krishvora01@gmail.com>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Hi Manikumar and Luke.
> > > > > > > >> > > > > > > Thanks for the questions.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 1. No, the Docker inventory files and
> > configurations
> > > > > will
> > > > > > > not
> > > > > > > >> be
> > > > > > > >> > > the
> > > > > > > >> > > > same
> > > > > > > >> > > > > > > for Open Source Software (OSS) Images and Docker
> > > > > Official
> > > > > > > >> Images
> > > > > > > >> > > > (DOI).
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > For OSS images, the Dockerfile located in
> > > > > > > >> docker/jvm/dockerfile
> > > > > > > >> > is
> > > > > > > >> > > > > > > utilized. This process is integrated with the
> > > existing
> > > > > > > release
> > > > > > > >> > > > pipeline
> > > > > > > >> > > > > > as
> > > > > > > >> > > > > > > outlined in KIP-975
> > > > > > > >> > > > > > > <
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > > > > > > >> > > > > > >,
> > > > > > > >> > > > > > > where the Kafka URL is provided as a build
> > argument.
> > > > > This
> > > > > > > >> method
> > > > > > > >> > > > allows
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > > building, testing, and releasing OSS images
> > > > dynamically.
> > > > > > The
> > > > > > > >> OSS
> > > > > > > >> > > > images
> > > > > > > >> > > > > > > will continue to be released under the standard
> > > > release
> > > > > > > >> process .
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > In contrast, the release process for DOIs
> requires
> > > > > > providing
> > > > > > > >> the
> > > > > > > >> > > > Docker
> > > > > > > >> > > > > > Hub
> > > > > > > >> > > > > > > team with a specific directory for each version
> > > > release
> > > > > > that
> > > > > > > >> > > > contains a
> > > > > > > >> > > > > > > standalone Dockerfile. These Dockerfiles are
> > > designed
> > > > to
> > > > > > be
> > > > > > > >> > > > > > > self-sufficient, hence require hardcoded values
> > > > instead
> > > > > of
> > > > > > > >> > relying
> > > > > > > >> > > on
> > > > > > > >> > > > > > build
> > > > > > > >> > > > > > > arguments. To accommodate this, in our proposed
> > > > > approach,
> > > > > > a
> > > > > > > >> new
> > > > > > > >> > > > directory
> > > > > > > >> > > > > > > named docker_official_images has been created.
> > This
> > > > > > > directory
> > > > > > > >> > > > contains
> > > > > > > >> > > > > > > version-specific directories, having Dockerfiles
> > > with
> > > > > > > >> hardcoded
> > > > > > > >> > > > > > > configurations for each release, acting as the
> > > source
> > > > of
> > > > > > > truth
> > > > > > > >> > for
> > > > > > > >> > > > DOI
> > > > > > > >> > > > > > > releases. The hardcoded dockerfiles will be
> > created
> > > > > using
> > > > > > > the
> > > > > > > >> > > > > > > docker/jvm/dockerfile as a template. Thus, as
> part
> > > of
> > > > > post
> > > > > > > >> > release
> > > > > > > >> > > we
> > > > > > > >> > > > > > will
> > > > > > > >> > > > > > > be creating a Dockerfile that will be reviewed
> by
> > > the
> > > > > > > >> Dockerhub
> > > > > > > >> > > > community
> > > > > > > >> > > > > > > and might need changes as per their review. This
> > > > > approach
> > > > > > > >> ensures
> > > > > > > >> > > > that
> > > > > > > >> > > > > > DOIs
> > > > > > > >> > > > > > > are built consistently and meet the specific
> > > > > requirements
> > > > > > > set
> > > > > > > >> by
> > > > > > > >> > > > Docker
> > > > > > > >> > > > > > Hub.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 2. Yes Manikumar, transitioning the release of
> > > Docker
> > > > > > > Official
> > > > > > > >> > > Images
> > > > > > > >> > > > > > (DOI)
> > > > > > > >> > > > > > > to a post-release activity does address the
> > concerns
> > > > > about
> > > > > > > >> > > > complicating
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > release process. Initially, we considered
> > > > incorporating
> > > > > > DOI
> > > > > > > >> > release
> > > > > > > >> > > > > > > directly into Kafka's release workflow. However,
> > > this
> > > > > > > approach
> > > > > > > >> > > > > > > significantly increased the RMs workload due to
> > the
> > > > > > addition
> > > > > > > >> of
> > > > > > > >> > > > numerous
> > > > > > > >> > > > > > > steps, complicating the process. By designating
> > the
> > > > DOI
> > > > > > > >> release
> > > > > > > >> > as
> > > > > > > >> > > a
> > > > > > > >> > > > > > > post-release task, we maintain the original
> > release
> > > > > > process.
> > > > > > > >> This
> > > > > > > >> > > > > > > adjustment allows for the DOI release to be done
> > > after
> > > > > the
> > > > > > > >> main
> > > > > > > >> > > > release.
> > > > > > > >> > > > > > We
> > > > > > > >> > > > > > > have revised the KIP to reflect that DOI
> releases
> > > will
> > > > > now
> > > > > > > >> occur
> > > > > > > >> > > > after
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > main release phase. Please review the updated
> > > document
> > > > > and
> > > > > > > >> > provide
> > > > > > > >> > > > any
> > > > > > > >> > > > > > > feedback you might have.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Thanks,
> > > > > > > >> > > > > > > Krish.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <
> > > > > > showuon@gmail.com
> > > > > > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > Hi Krishna,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > I also have the same question as Manikumar
> > raised:
> > > > > > > >> > > > > > > > 1. Will the Docker inventory files/etc are the
> > > same
> > > > > for
> > > > > > > OSS
> > > > > > > >> > Image
> > > > > > > >> > > > and
> > > > > > > >> > > > > > > > Docker Official Images?
> > > > > > > >> > > > > > > > If no, then why not? Could we make them
> > identical
> > > so
> > > > > > that
> > > > > > > we
> > > > > > > >> > > don't
> > > > > > > >> > > > > > have to
> > > > > > > >> > > > > > > > build 2 images for each release?
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Thank you.
> > > > > > > >> > > > > > > > Luke
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> > > > > > > >> > > > manikumar.reddy@gmail.com>
> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > Hi Krishna,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thanks for the KIP.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > I think Docker Official Images will be
> > > beneficial
> > > > to
> > > > > > the
> > > > > > > >> > Kafka
> > > > > > > >> > > > > > community.
> > > > > > > >> > > > > > > > > Few queries below.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > 1. Will the Docker inventory files/etc are
> the
> > > > same
> > > > > > for
> > > > > > > >> OSS
> > > > > > > >> > > > Image and
> > > > > > > >> > > > > > > > > Docker Official Images
> > > > > > > >> > > > > > > > > 2. I am a bit worried about the new steps to
> > the
> > > > > > release
> > > > > > > >> > > process.
> > > > > > > >> > > > > > Maybe
> > > > > > > >> > > > > > > > we
> > > > > > > >> > > > > > > > > should consider Docker Official Images
> release
> > > as
> > > > > > > >> > Post-Release
> > > > > > > >> > > > > > activity.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> > > > > > > >> > > > krishvora01@gmail.com>
> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > Hi Hector,
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Thanks for reaching out. This KIP builds
> on
> > > top
> > > > of
> > > > > > > >> KIP-975
> > > > > > > >> > > > > > > > > > <
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > and
> > > > > > > >> > > > > > > > > > aims to introduce a JVM-based Docker
> > Official
> > > > > Image
> > > > > > > (DOI
> > > > > > > >> > > > > > > > > > <
> > > > > > > >> https://docs.docker.com/trusted-content/official-images/
> > > > > > > >> > >)
> > > > > > > >> > > > for
> > > > > > > >> > > > > > Apache
> > > > > > > >> > > > > > > > > > Kafka that will be visible under Docker
> > > Official
> > > > > > > Images
> > > > > > > >> > > > > > > > > > <
> > > > > > > https://hub.docker.com/search?image_filter=official&q=
> > > > > > > >> >.
> > > > > > > >> > > Once
> > > > > > > >> > > > > > > > > implemented
> > > > > > > >> > > > > > > > > > for Apache Kafka, for each release, there
> > will
> > > > be
> > > > > > one
> > > > > > > >> more
> > > > > > > >> > > > > > JVM-based
> > > > > > > >> > > > > > > > > Docker
> > > > > > > >> > > > > > > > > > image available to users.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Currently, we already have an OSS
> sponsored
> > > > image,
> > > > > > > which
> > > > > > > >> > was
> > > > > > > >> > > > > > introduced
> > > > > > > >> > > > > > > > > via
> > > > > > > >> > > > > > > > > > KIP-975 (apache/kafka <
> > > > > > > >> > > > https://hub.docker.com/r/apache/kafka/tags
> > > > > > > >> > > > > > >)
> > > > > > > >> > > > > > > > > which
> > > > > > > >> > > > > > > > > > comes under The Apache Software
> Foundation <
> > > > > > > >> > > > > > > > > > https://hub.docker.com/u/apache> in
> > > > > > > >> > > > > > > > > > Docker Hub. The new Docker Image is the
> > Docker
> > > > > > > Official
> > > > > > > >> > Image
> > > > > > > >> > > > > > (DOI),
> > > > > > > >> > > > > > > > > which
> > > > > > > >> > > > > > > > > > will be built and maintained by Docker
> > > > Community.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > For example, for a release version like
> > 3.8.0
> > > we
> > > > > > will
> > > > > > > >> have
> > > > > > > >> > > two
> > > > > > > >> > > > JVM
> > > > > > > >> > > > > > > > based
> > > > > > > >> > > > > > > > > > docker images:-
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored
> > image)
> > > > > > > >> > > > > > > > > >    - kafka:3.8.0 (Docker Official image)
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > I have added the same in the KIP too for
> > > > > everyone's
> > > > > > > >> > > reference.
> > > > > > > >> > > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > > Krish.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector
> > > Geraldino
> > > > > > > >> > (BLOOMBERG/
> > > > > > > >> > > > 919
> > > > > > > >> > > > > > 3RD
> > > > > > > >> > > > > > > > A) <
> > > > > > > >> > > > > > > > > > hgeraldino@bloomberg.net> wrote:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Hi,
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > What is the difference between this KIP
> > and
> > > > > > KIP-975:
> > > > > > > >> > Docker
> > > > > > > >> > > > > > Image for
> > > > > > > >> > > > > > > > > > > Apache Kafka?
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24
> > > > > 07:30:07
> > > > > > > >> > > UTC-4:00To:
> > > > > > > >> > > > > > > > > > > dev@kafka.apache.org
> > > > > > > >> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker
> > Official
> > > > > Image
> > > > > > > for
> > > > > > > >> > > Apache
> > > > > > > >> > > > > > Kafka
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Hi everyone,
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > I would like to start the discussion on
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > > > >> > > > > > > > > > > age+for+Apache+Kafka
> > > > > > > >> > > > > > > > > > >  .
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > This KIP aims to introduce JVM based
> > Docker
> > > > > > Official
> > > > > > > >> > Image
> > > > > > > >> > > > (DOI)
> > > > > > > >> > > > > > for
> > > > > > > >> > > > > > > > > > Apache
> > > > > > > >> > > > > > > > > > > Kafka.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Regards,
> > > > > > > >> > > > > > > > > > > Krish.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >>
> > > > > > > >> [image: Confluent] <https://www.confluent.io>
> > > > > > > >> Prabha Manepalli
> > > > > > > >> Product Manager for Confluent Platform Security, Docker
> > > > > > > >> linkedin.com/in/prabhamanepalli
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > [image: Confluent] <https://www.confluent.io>
> > > > Prabha Manepalli
> > > > Product Manager for Confluent Platform Security, Docker Containers
> > > > linkedin.com/in/prabhamanepalli
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Chris Egerton <ch...@aiven.io.INVALID>]

Hi Vedarth,

This looks great, thank you for helping me thoroughly understand the
motivation and benefits for the KIP. Looks good to me.

Regarding public interface for the images--everything in the "Public
Interface" section in KIP-975 does qualify as public interface for the
images IMO, but I don't think it's comprehensive. If we were asked to, for
example, change the port in the EXPOSE directive in our Dockerfile, that
would probably qualify as a change to public interface too. With the strict
language in the latest draft of this KIP that ensures that any functional
changes to our Docker images go through another follow-up KIP, we should be
fine without having to identify a comprehensive list of everything that
constitutes public interface for our images.

Cheers, and thanks again for the KIP,

Chris

On Mon, May 13, 2024 at 3:07 PM Vedarth Sharma <ve...@gmail.com>
wrote:

> Hey Chris,
>
> Once we provide the definitions to docker, they should take care of
> everything from there. They mentioned here
> <
> https://github.com/docker-library/official-images?tab=readme-ov-file#library-definition-files
> >
> that
> the image will be rebuilt when the base image is updated. Hence active
> rebuilds won't require any changes from our side.
> If we are packaging something which may contain a CVE, like some jar, then
> the onus will be on us to patch it, but it will be upto us whether we
> consider the threat severe enough to fix and when we want to provide the
> fixed version. Having Docker Official Image will not impact the frequency
> of our releases. It will be the Apache Kafka community's call on when a
> release goes and Docker Official Image will be released accordingly as per
> the KIP. source
> <https://github.com/docker-library/faq?tab=readme-ov-file#image-building>
>
> As mentioned in Docker's documentation as well "In essence we strive to
> heed upstream's recommendations on how they intend for their software to be
> consumed." source
> <
> https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
> >
> Docker Official Image will rely on upstream's recommendation for
> functionality. But I do agree that since Docker's stance on this might
> change in future it makes sense to put a safeguard that will not allow any
> functionality changes get incorporated as part of the vetting process. I
> have updated the KIP to reflect the same.
>
> KIP-975 has a well defined public interface based on how configs can be
> supplied and how it can be used. I am not sure if we put that label on it
> during discussions. I am happy to have a separate email thread on it to
> iron things out.
>
> I hope this addresses all of your concerns!
>
> Thanks and regards,
> Vedarth
>
> On Mon, May 13, 2024 at 10:55 PM Chris Egerton <ch...@aiven.io.invalid>
> wrote:
>
> > Thanks both for your responses! Friendly reminder: again, better to
> provide
> > a quote instead of just a link :)
> >
> > I've seen a bit about image rebuilding to handle CVEs but I'm a little
> > unclear on how this would work in practice, and I couldn't find any
> > concrete details in any of the links. Does Docker do this automatically
> for
> > DOIs? Or will the onus be on us to put out patched images? Would this
> lead
> > to us putting out images more quickly than we put out standard releases?
> As
> > a plus, it does look like DOIs get the benefit of Docker Scout [1] for
> > free, which is nice, but it's still unclear who'd be doing the rest of
> the
> > work on that front.
> >
> > As far as this point from Vedarth goes:
> >
> > > By incorporating the source code of the Docker Official Image into our
> > > AK ecosystem, we gain control over its functionality, ensuring
> alignment
> > > with the OSS Docker image. This ensures a seamless experience for users
> > who
> > > may need to transition between these images.
> >
> > This captures my concern with the KIP pretty well. If there's any
> > significant divergence in behavior (not just build methodology) between
> the
> > apache/kafka image and what Docker requires for a Kafka DOI, how are we
> > going to vet these changes moving forward? Under the "Post Release
> Process
> > - if Dockerhub folks suggest changes to the Dockerfiles:" header, this
> KIP
> > proposes that we port all suggested changes for the DOI to
> > the docker/jvm/Dockerfile image, but this seems a bit too permissive. As
> an
> > alternative, we could state that all build-related changes can be done
> with
> > a PR on the apache/kafka GitHub repo (which will require approval from a
> > single committer), but any functional changes will require a KIP.
> >
> > Finally, during KIP-975 was there discussion on what we would count as
> the
> > public interface for the apache/kafka image? If not, it'd be nice to get
> > that ironed out since it may make future discussions around our Docker
> > images quicker, but I don't think this is necessary for KIP-1028.
> >
> > [1] - https://www.docker.com/products/docker-scout/
> >
> > On Mon, May 13, 2024 at 4:37 AM Prabha Manepalli
> > <mp...@confluent.io.invalid> wrote:
> >
> > > Hi Chris,
> > >
> > > Sharing the requested links explaining why Docker Official images are
> > > considered more secure -
> > >
> > >
> >
> https://www.docker.com/blog/enhancing-security-and-transparency-with-docker-official-images/
> > > and
> > >
> > >
> >
> https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-cves
> > >
> > > I hope these links help you understand why we need Docker Official
> images
> > > for organisations with stringent security compliance requirements for
> > their
> > > Kafka workloads.
> > >
> > > Thank you.
> > >
> > >
> > >
> > > On Sun, May 12, 2024 at 3:33 PM Vedarth Sharma <
> vedarth.sharma@gmail.com
> > >
> > > wrote:
> > >
> > > > Hey Chris!
> > > >
> > > > Functionality wise, we don't intend to have any differences between
> OSS
> > > > Docker Image and Docker Official Image.
> > > > The Docker Official Image will be the recommended one.
> > > > Since the Docker Official Image might be delayed due to review done
> by
> > > > Docker, images on apache/kafka (OSS Docker Image) can be used by
> users.
> > > >
> > > > 1) I read https://docs.docker.com/trusted-content/official-images/
> and
> > > > > didn't find anything on that page or immediately around it that
> > > explains
> > > > > what compliance requirements might be satisfied by a DOI that
> > couldn't
> > > be
> > > > > satisfied by the existing apache/kafka image. Can you elaborate?
> Feel
> > > > free
> > > > > to provide another link, but please also quote the relevant
> sections
> > > from
> > > > > it (as StackOverflow likes to say, links can grow stale over time).
> > > >
> > > >
> > > >    - If a user's selection is confined solely to Docker Official
> > Images,
> > > >    this Docker Image will ensure their access to Kafka.
> > > >    - Details on specific advantages of Docker Official Images can be
> > > found
> > > >    at
> > > >
> > > >
> > >
> >
> https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
> > > >    .
> > > >    - The Docker Official Image will be actively rebuilt for updates
> and
> > > >    security fixes.
> > > >    - It's true we can provide exactly the same benefits in the OSS
> > Docker
> > > >    Image as well. But it won't have the Docker Official Image badge
> and
> > > it
> > > >    will add additional overhead for Apache Kafka community.
> > > >    - The fact that it will have Docker Official Image badge will set
> it
> > > >    apart from the OSS Docker Image.
> > > >    - Also the ability to do just docker pull kafka to get the kafka
> > > docker
> > > >    image will only be possible with Docker Official Image.
> > > >
> > > >
> > > > 2) It would be great to see a brief summary of the differences in
> these
> > > > > images included in the KIP, in order to try to gauge how this would
> > > look
> > > > to
> > > > > our users.
> > > >
> > > >
> > > >    - Functionally, both Docker images will remain identical.
> > > >    - The variance lies primarily in the methodologies of building and
> > > >    validation, as outlined in the updated KIP.
> > > >
> > > >
> > > > 3) What I suggested last time was not a separate apache/apache-docker
> > > > > repository, but a repository controlled entirely by Docker. The DOI
> > > docs
> > > > > [1] state that "While it's preferable to have upstream software
> > authors
> > > > > maintaining their Docker Official Images, this isn't a strict
> > > > requirement",
> > > > > which I take to mean that it's not required for an Apache Kafka DOI
> > to
> > > > live
> > > > > under the apache organization on GitHub. It also seems like there's
> > > > > precedent for this: images for MySQL [2] and PHP [3] already exist
> > > under
> > > > > the control of Docker. The reason I think this is worth considering
> > is
> > > > that
> > > > > Docker can arbitrarily change the eligibility requirements for
> their
> > > > > official images at any time, and it doesn't seem like there's a
> clear
> > > > > process in the KIP for judging how we should respond to these
> changes
> > > (in
> > > > > fact, it seems like the idea in the KIP is that we should make any
> > > change
> > > > > required with no further vetting beyond possibly a pull request on
> > > > > apache/kafka, which would require approval by a committer). By
> > hosting
> > > > the
> > > > > DOI definitions ourselves (either in apache/kafka, or in a
> > theoretical
> > > > > apache/docker-kafka repository), we take responsibility for the
> > image,
> > > > even
> > > > > if the owner on Docker Hub is Docker, not Apache. If the code lives
> > > > > elsewhere, then (as long as basic trademark and possibly security
> > > > > guidelines are respected) Apache doesn't have to concern itself at
> > all
> > > > with
> > > > > the image and the maintainers are free to make whatever changes
> they
> > > want
> > > > > to it in order to meet Docker's requirements.
> > > >
> > > >
> > > >    - By incorporating the source code of the Docker Official Image
> into
> > > our
> > > >    AK ecosystem, we gain control over its functionality, ensuring
> > > alignment
> > > >    with the OSS Docker image. This ensures a seamless experience for
> > > users
> > > > who
> > > >    may need to transition between these images.
> > > >    - Maintaining both images within the same community facilitates
> ease
> > > of
> > > >    management and fosters a singular source of truth.
> > > >    - While Apache may not retain ownership of the hosted Docker
> > Official
> > > >    Image, we are, in essence, providing Docker with a foundation that
> > > > aligns
> > > >    with their established guidelines as well as remains consistent
> with
> > > OSS
> > > >    Docker Image apache/kafka.
> > > >    - Any future alterations to the functionality can be seamlessly
> > > >    propagated across both the OSS and Official Docker Images.
> > > >
> > > > I think these reasons must be why a lot of the Apache projects choose
> > to
> > > > host the docker definitions themselves. The responsibility of owning
> > the
> > > > definitions comes with benefits as well that we should also consider.
> > > >
> > > > Let us know if you have any further questions!
> > > >
> > > > Thanks and regards,
> > > > Vedarth
> > > >
> > > > On Fri, May 10, 2024 at 7:56 PM Chris Egerton
> <chrise@aiven.io.invalid
> > >
> > > > wrote:
> > > >
> > > > > Hi Krish and Prabha,
> > > > >
> > > > > Thanks for your replies. I still have some follow-up questions:
> > > > >
> > > > > 1) I read https://docs.docker.com/trusted-content/official-images/
> > and
> > > > > didn't find anything on that page or immediately around it that
> > > explains
> > > > > what compliance requirements might be satisfied by a DOI that
> > couldn't
> > > be
> > > > > satisfied by the existing apache/kafka image. Can you elaborate?
> Feel
> > > > free
> > > > > to provide another link, but please also quote the relevant
> sections
> > > from
> > > > > it (as StackOverflow likes to say, links can grow stale over time).
> > > > >
> > > > > 2) It would be great to see a brief summary of the differences in
> > these
> > > > > images included in the KIP, in order to try to gauge how this would
> > > look
> > > > to
> > > > > our users.
> > > > >
> > > > > 3) What I suggested last time was not a separate
> apache/apache-docker
> > > > > repository, but a repository controlled entirely by Docker. The DOI
> > > docs
> > > > > [1] state that "While it's preferable to have upstream software
> > authors
> > > > > maintaining their Docker Official Images, this isn't a strict
> > > > requirement",
> > > > > which I take to mean that it's not required for an Apache Kafka DOI
> > to
> > > > live
> > > > > under the apache organization on GitHub. It also seems like there's
> > > > > precedent for this: images for MySQL [2] and PHP [3] already exist
> > > under
> > > > > the control of Docker. The reason I think this is worth considering
> > is
> > > > that
> > > > > Docker can arbitrarily change the eligibility requirements for
> their
> > > > > official images at any time, and it doesn't seem like there's a
> clear
> > > > > process in the KIP for judging how we should respond to these
> changes
> > > (in
> > > > > fact, it seems like the idea in the KIP is that we should make any
> > > change
> > > > > required with no further vetting beyond possibly a pull request on
> > > > > apache/kafka, which would require approval by a committer). By
> > hosting
> > > > the
> > > > > DOI definitions ourselves (either in apache/kafka, or in a
> > theoretical
> > > > > apache/docker-kafka repository), we take responsibility for the
> > image,
> > > > even
> > > > > if the owner on Docker Hub is Docker, not Apache. If the code lives
> > > > > elsewhere, then (as long as basic trademark and possibly security
> > > > > guidelines are respected) Apache doesn't have to concern itself at
> > all
> > > > with
> > > > > the image and the maintainers are free to make whatever changes
> they
> > > want
> > > > > to it in order to meet Docker's requirements.
> > > > >
> > > > > [1] -
> > > > >
> > https://docs.docker.com/trusted-content/official-images/contributing/,
> > > > > second paragraph under "Contributing to Docker Official Images"
> > > > > [2] - https://github.com/docker-library/mysql
> > > > > [3] - https://github.com/docker-library/php
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Fri, May 10, 2024 at 7:46 AM Krish Vora <kr...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hey Chris,
> > > > > >
> > > > > > We have responded to the initial round of queries. And as you
> > > mentioned
> > > > > in
> > > > > > your comment, you may have more questions related to this KIP.
> > Please
> > > > let
> > > > > > us know if you have any.
> > > > > > We want to start the voting for this KIP soon, as we intend to
> > > include
> > > > it
> > > > > > in the 3.8.0 release.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Krish.
> > > > > >
> > > > > > On Wed, May 8, 2024 at 6:19 PM Krish Vora <krishvora01@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi Chris. Thanks for the questions.
> > > > > > >
> > > > > > > 3. Would a separate Docker-owned repository be out of the
> > question?
> > > > I'm
> > > > > > >> guessing there are some trademark issues that might get in the
> > > way,
> > > > > but
> > > > > > >> it's worth exploring since the entire purpose of this KIP
> seems
> > to
> > > > be
> > > > > to
> > > > > > >> provide images that are vetted and designed by Docker more
> than
> > by
> > > > the
> > > > > > >> Apache Kafka contributors/committers/PMC.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >    - The process for introducing a Docker Official Image
> involves
> > > > > > >       - Hosting the Dockerfile in the Apache Kafka repository
> and
> > > > > > >       - Providing the path to this Dockerfile to Docker Hub in
> > > Docker
> > > > > > >       Hub’s own repo
> > > > > > >       <
> > > > > >
> > > https://github.com/docker-library/official-images/tree/master/library>
> > > > > > >       .
> > > > > > >    - This ensures that any updates to the Dockerfile in the AK
> > > > > repository
> > > > > > >    are directly applicable to the docker official images
> > available
> > > on
> > > > > > Docker
> > > > > > >    Hub.
> > > > > > >
> > > > > > >
> > > > > > >    - We also did not find any added advantage to create a
> > separate
> > > > > > >    repository named apache-docker within the Apache GitHub
> > > > > organization.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Krish.
> > > > > > >
> > > > > > > On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli
> > > > > > > <mp...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > >> Hi Chris,  I would like to add more context to this KIP's
> > > > motivation.
> > > > > > >> Vedarth and Krish, please weigh in with your inputs.
> > > > > > >>
> > > > > > >> In the motivation section it's stated that "Several other
> Apache
> > > > > > projects,
> > > > > > >> > like Flink, Spark, Solr, have already released Docker
> Official
> > > > > Images,
> > > > > > >> with
> > > > > > >> > download figures ranging from 50 million to over 1 billion.
> > > These
> > > > > > >> numbers
> > > > > > >> > highlight the significant demand among users." But then
> > > > immediately
> > > > > > >> > afterwards, we learn that "Also the Docker Official Images
> are
> > > > > always
> > > > > > >> the
> > > > > > >> > top 1 search result, irrespective of the number of
> downloads."
> > > > > > Wouldn't
> > > > > > >> a
> > > > > > >> > high number of downloads for an image naturally follow from
> > > being
> > > > > the
> > > > > > >> top
> > > > > > >> > search result? It seems like we can't necessarily assume
> that
> > > > Docker
> > > > > > >> > Official Images are inherently more desirable for users
> based
> > > > solely
> > > > > > on
> > > > > > >> > download statistics.
> > > > > > >> >
> > > > > > >>
> > > > > > >> *My thoughts: *Unlike the Sponsored OSS image, the Docker
> > Official
> > > > > image
> > > > > > >> is
> > > > > > >> more desirable for workloads that have stringent compliance
> > > > > > requirements.
> > > > > > >> More details on why official images are more trusted are
> > > documented
> > > > > here
> > > > > > >> <https://docs.docker.com/trusted-content/official-images/>.
> The
> > > > > Docker
> > > > > > >> Official image would also help an absolutely new Kafka
> beginner
> > > who
> > > > > > might
> > > > > > >> not know about Apache or the concept of Sponsored images. We
> > want
> > > to
> > > > > > make
> > > > > > >> it easier for Kafka beginners to discover the Kafka image
> > through
> > > > > > >> DockerHub.
> > > > > > >>
> > > > > > >>
> > > > > > >> Can you elaborate on the value that these new images would add
> > > from
> > > > a
> > > > > > >> > user's perspective? I'm hesitant to introduce another image,
> > > since
> > > > > it
> > > > > > >> adds
> > > > > > >> > to the cognitive burden of people who will inevitably have
> to
> > > > answer
> > > > > > the
> > > > > > >> > question of "What are the differences between all of the
> > > available
> > > > > > >> images
> > > > > > >> > and which one is best for my use case?"
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >> *My thoughts: *This is a valid concern to address. The
> response
> > to
> > > > the
> > > > > > >> above question addresses the value-add this new Docker
> Official
> > > > image
> > > > > > >> would
> > > > > > >> provide. I also agree we need a clear distinction between each
> > of
> > > > > these
> > > > > > >> images to be well documented. We plan to update the AK website
> > > with
> > > > > > >> details
> > > > > > >> on how, why, and when a developer would want to use each of
> > these
> > > > > > >> particular images(KIP-974,975,1028).
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Prabha.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton
> > > > <chrise@aiven.io.invalid
> > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Vedarth and Krish,
> > > > > > >> >
> > > > > > >> > Thanks for the KIP! I have to admit I'm a little skeptical;
> > > > > hopefully
> > > > > > >> you
> > > > > > >> > can help me understand the need for these additional images.
> > > > > > >> >
> > > > > > >> > 1) In the motivation section it's stated that "Several other
> > > > Apache
> > > > > > >> > projects, like Flink, Spark, Solr, have already released
> > Docker
> > > > > > Official
> > > > > > >> > Images, with download figures ranging from 50 million to
> over
> > 1
> > > > > > billion.
> > > > > > >> > These numbers highlight the significant demand among users."
> > But
> > > > > then
> > > > > > >> > immediately afterwards, we learn that "Also the Docker
> > Official
> > > > > Images
> > > > > > >> are
> > > > > > >> > always the top 1 search result, irrespective of the number
> of
> > > > > > >> downloads."
> > > > > > >> > Wouldn't a high number of downloads for an image naturally
> > > follow
> > > > > from
> > > > > > >> > being the top search result? It seems like we can't
> > necessarily
> > > > > assume
> > > > > > >> that
> > > > > > >> > Docker Official Images are inherently more desirable for
> users
> > > > based
> > > > > > >> solely
> > > > > > >> > on download statistics.
> > > > > > >> >
> > > > > > >> > 2) Can you elaborate on the value that these new images
> would
> > > add
> > > > > > from a
> > > > > > >> > user's perspective? I'm hesitant to introduce another image,
> > > since
> > > > > it
> > > > > > >> adds
> > > > > > >> > to the cognitive burden of people who will inevitably have
> to
> > > > answer
> > > > > > the
> > > > > > >> > question of "What are the differences between all of the
> > > available
> > > > > > >> images
> > > > > > >> > and which one is best for my use case?"
> > > > > > >> >
> > > > > > >> > 3) Would a separate Docker-owned repository be out of the
> > > > question?
> > > > > > I'm
> > > > > > >> > guessing there are some trademark issues that might get in
> the
> > > > way,
> > > > > > but
> > > > > > >> > it's worth exploring since the entire purpose of this KIP
> > seems
> > > to
> > > > > be
> > > > > > to
> > > > > > >> > provide images that are vetted and designed by Docker more
> > than
> > > by
> > > > > the
> > > > > > >> > Apache Kafka contributors/committers/PMC.
> > > > > > >> >
> > > > > > >> > I may have more questions later but wanted to get this
> initial
> > > > round
> > > > > > out
> > > > > > >> > now without trying to list everything first.
> > > > > > >> >
> > > > > > >> > Looking forward to your thoughts!
> > > > > > >> >
> > > > > > >> > Cheers,
> > > > > > >> >
> > > > > > >> > Chris
> > > > > > >> >
> > > > > > >> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <
> > > > > > >> vedarth.sharma@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hey folks,
> > > > > > >> > >
> > > > > > >> > > Thanks a lot for reviewing the KIP and providing feedback.
> > > > > > >> > > The discussion thread seems resolved and KIP has been
> > updated
> > > > > > >> > accordingly.
> > > > > > >> > > We will be starting the voting thread for this KIP in the
> > next
> > > > few
> > > > > > >> days.
> > > > > > >> > > Please take a look at the KIP and let us know if any
> further
> > > > > > >> discussion
> > > > > > >> > > is needed.
> > > > > > >> > >
> > > > > > >> > > Thanks and regards,
> > > > > > >> > > Vedarth
> > > > > > >> > >
> > > > > > >> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <
> > > > > > manikumar.reddy@gmail.com>
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Thanks Krish. KIP looks good to me.
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <
> > > > > krishvora01@gmail.com
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > Hi Manikumar,
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks for the comments.
> > > > > > >> > > > >
> > > > > > >> > > > > Maybe as part of the release process, RM can create a
> > JIRA
> > > > for
> > > > > > >> this
> > > > > > >> > > > > > task. This can be taken by RM or any comitter or any
> > > > > > contributor
> > > > > > >> > > (with
> > > > > > >> > > > > > some help from commiters to run "Docker Image
> > > Preparation
> > > > > via
> > > > > > >> > GitHub
> > > > > > >> > > > > > Actions:"
> > > > > > >> > > > >
> > > > > > >> > > > > This sounds like a good idea. This step would be
> > > beneficial.
> > > > > By
> > > > > > >> > > creating
> > > > > > >> > > > a
> > > > > > >> > > > > JIRA ticket, it will also serve as a reminder to
> > complete
> > > > the
> > > > > > >> > > > post-release
> > > > > > >> > > > > steps for the Docker official images. Have updated the
> > KIP
> > > > > with
> > > > > > >> this
> > > > > > >> > > > step.
> > > > > > >> > > > >
> > > > > > >> > > > > Is this using GitHub Actions workflow? or manual
> > testing?
> > > > > > >> > > > >
> > > > > > >> > > > > This will be done by a Github Actions workflow, which
> > will
> > > > > test
> > > > > > >> the
> > > > > > >> > > > static
> > > > > > >> > > > > Docker Official Image assets for a specific release
> > > version.
> > > > > > >> > > > >
> > > > > > >> > > > > Is it mandatory for RM/comitters to raise the PR to
> > Docker
> > > > > Hub’s
> > > > > > >> > > > > > official images repository (or) can it be done by
> any
> > > > > > >> contributor.
> > > > > > >> > > > >
> > > > > > >> > > > > I believe that it can be done by any contributor (ref:
> > > This
> > > > > link
> > > > > > >> > > > > <
> > > > > > >> >
> > > > >
> > https://docs.docker.com/trusted-content/official-images/contributing/
> > > > > > >> > > >
> > > > > > >> > > > > quotes "*Anyone can provide feedback, contribute code,
> > > > suggest
> > > > > > >> > process
> > > > > > >> > > > > changes, or even propose a new Official Image.*")
> > > > > > >> > > > >
> > > > > > >> > > > > Also I was thinking, once the KIP gets voted, we
> should
> > > try
> > > > to
> > > > > > >> > release
> > > > > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This
> > will
> > > > help
> > > > > > us
> > > > > > >> to
> > > > > > >> > > > > > validate the process and allow us to fix any changes
> > > > > suggested
> > > > > > >> by
> > > > > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > > > > >> > > > >
> > > > > > >> > > > > This sounds like a great idea. This KIP proposes
> release
> > > of
> > > > > DOI
> > > > > > >> as a
> > > > > > >> > > > > post-release process, which can be done anytime post
> > > > release.
> > > > > > >> Since
> > > > > > >> > > 3.7.0
> > > > > > >> > > > > is already released, we can perform these steps for
> that
> > > > > release
> > > > > > >> too.
> > > > > > >> > > By
> > > > > > >> > > > > the time the KIP gets implemented, if 3.7.1 is
> released,
> > > we
> > > > > > could
> > > > > > >> do
> > > > > > >> > > > these
> > > > > > >> > > > > steps for 3.7.1, instead of 3.7.0. This would allow us
> > to
> > > > make
> > > > > > >> > changes
> > > > > > >> > > to
> > > > > > >> > > > > the Dockerfiles and other assets based on feedback
> from
> > > > Docker
> > > > > > Hub
> > > > > > >> > > before
> > > > > > >> > > > > the release of version 3.8.0.
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Krish.
> > > > > > >> > > > >
> > > > > > >> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
> > > > > > >> > manikumar.reddy@gmail.com>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hi Krish,
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks for the updated KIP. a few comments below.
> > > > > > >> > > > > >
> > > > > > >> > > > > > > "These actions can be carried out by the RM or any
> > > > > > contributor
> > > > > > >> > post
> > > > > > >> > > > the
> > > > > > >> > > > > > release process."
> > > > > > >> > > > > > Maybe as part of the release process, RM can create
> a
> > > JIRA
> > > > > for
> > > > > > >> this
> > > > > > >> > > > > > task. This can be taken by RM or any comitter or any
> > > > > > contributor
> > > > > > >> > > (with
> > > > > > >> > > > > > some help from commiters to run "Docker Image
> > > Preparation
> > > > > via
> > > > > > >> > GitHub
> > > > > > >> > > > > > Actions:"
> > > > > > >> > > > > >
> > > > > > >> > > > > > > "Perform Docker build tests to ensure image
> > integrity"
> > > > > > >> > > > > > Is this using GitHub Actions workflow? or manual
> > > testing?
> > > > > > >> > > > > >
> > > > > > >> > > > > > > "The RM will manually raise the final PR to Docker
> > > Hub’s
> > > > > > >> official
> > > > > > >> > > > images
> > > > > > >> > > > > > repository using the contents of the generated file"
> > > > > > >> > > > > >  Is it mandatory for RM/comitters to raise the PR to
> > > > Docker
> > > > > > >> Hub’s
> > > > > > >> > > > > > official images repository (or) can it be done by
> any
> > > > > > >> contributor.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Also I was thinking, once the KIP gets voted, we
> > should
> > > > try
> > > > > to
> > > > > > >> > > release
> > > > > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This
> > will
> > > > help
> > > > > > us
> > > > > > >> to
> > > > > > >> > > > > > validate the process and allow us to fix any changes
> > > > > suggested
> > > > > > >> by
> > > > > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks,
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <
> > > > > > >> krishvora01@gmail.com>
> > > > > > >> > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Hi Manikumar and Luke.
> > > > > > >> > > > > > > Thanks for the questions.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 1. No, the Docker inventory files and
> configurations
> > > > will
> > > > > > not
> > > > > > >> be
> > > > > > >> > > the
> > > > > > >> > > > same
> > > > > > >> > > > > > > for Open Source Software (OSS) Images and Docker
> > > > Official
> > > > > > >> Images
> > > > > > >> > > > (DOI).
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > For OSS images, the Dockerfile located in
> > > > > > >> docker/jvm/dockerfile
> > > > > > >> > is
> > > > > > >> > > > > > > utilized. This process is integrated with the
> > existing
> > > > > > release
> > > > > > >> > > > pipeline
> > > > > > >> > > > > > as
> > > > > > >> > > > > > > outlined in KIP-975
> > > > > > >> > > > > > > <
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > > > > > >> > > > > > >,
> > > > > > >> > > > > > > where the Kafka URL is provided as a build
> argument.
> > > > This
> > > > > > >> method
> > > > > > >> > > > allows
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > building, testing, and releasing OSS images
> > > dynamically.
> > > > > The
> > > > > > >> OSS
> > > > > > >> > > > images
> > > > > > >> > > > > > > will continue to be released under the standard
> > > release
> > > > > > >> process .
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > In contrast, the release process for DOIs requires
> > > > > providing
> > > > > > >> the
> > > > > > >> > > > Docker
> > > > > > >> > > > > > Hub
> > > > > > >> > > > > > > team with a specific directory for each version
> > > release
> > > > > that
> > > > > > >> > > > contains a
> > > > > > >> > > > > > > standalone Dockerfile. These Dockerfiles are
> > designed
> > > to
> > > > > be
> > > > > > >> > > > > > > self-sufficient, hence require hardcoded values
> > > instead
> > > > of
> > > > > > >> > relying
> > > > > > >> > > on
> > > > > > >> > > > > > build
> > > > > > >> > > > > > > arguments. To accommodate this, in our proposed
> > > > approach,
> > > > > a
> > > > > > >> new
> > > > > > >> > > > directory
> > > > > > >> > > > > > > named docker_official_images has been created.
> This
> > > > > > directory
> > > > > > >> > > > contains
> > > > > > >> > > > > > > version-specific directories, having Dockerfiles
> > with
> > > > > > >> hardcoded
> > > > > > >> > > > > > > configurations for each release, acting as the
> > source
> > > of
> > > > > > truth
> > > > > > >> > for
> > > > > > >> > > > DOI
> > > > > > >> > > > > > > releases. The hardcoded dockerfiles will be
> created
> > > > using
> > > > > > the
> > > > > > >> > > > > > > docker/jvm/dockerfile as a template. Thus, as part
> > of
> > > > post
> > > > > > >> > release
> > > > > > >> > > we
> > > > > > >> > > > > > will
> > > > > > >> > > > > > > be creating a Dockerfile that will be reviewed by
> > the
> > > > > > >> Dockerhub
> > > > > > >> > > > community
> > > > > > >> > > > > > > and might need changes as per their review. This
> > > > approach
> > > > > > >> ensures
> > > > > > >> > > > that
> > > > > > >> > > > > > DOIs
> > > > > > >> > > > > > > are built consistently and meet the specific
> > > > requirements
> > > > > > set
> > > > > > >> by
> > > > > > >> > > > Docker
> > > > > > >> > > > > > Hub.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 2. Yes Manikumar, transitioning the release of
> > Docker
> > > > > > Official
> > > > > > >> > > Images
> > > > > > >> > > > > > (DOI)
> > > > > > >> > > > > > > to a post-release activity does address the
> concerns
> > > > about
> > > > > > >> > > > complicating
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > release process. Initially, we considered
> > > incorporating
> > > > > DOI
> > > > > > >> > release
> > > > > > >> > > > > > > directly into Kafka's release workflow. However,
> > this
> > > > > > approach
> > > > > > >> > > > > > > significantly increased the RMs workload due to
> the
> > > > > addition
> > > > > > >> of
> > > > > > >> > > > numerous
> > > > > > >> > > > > > > steps, complicating the process. By designating
> the
> > > DOI
> > > > > > >> release
> > > > > > >> > as
> > > > > > >> > > a
> > > > > > >> > > > > > > post-release task, we maintain the original
> release
> > > > > process.
> > > > > > >> This
> > > > > > >> > > > > > > adjustment allows for the DOI release to be done
> > after
> > > > the
> > > > > > >> main
> > > > > > >> > > > release.
> > > > > > >> > > > > > We
> > > > > > >> > > > > > > have revised the KIP to reflect that DOI releases
> > will
> > > > now
> > > > > > >> occur
> > > > > > >> > > > after
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > main release phase. Please review the updated
> > document
> > > > and
> > > > > > >> > provide
> > > > > > >> > > > any
> > > > > > >> > > > > > > feedback you might have.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks,
> > > > > > >> > > > > > > Krish.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <
> > > > > showuon@gmail.com
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi Krishna,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > I also have the same question as Manikumar
> raised:
> > > > > > >> > > > > > > > 1. Will the Docker inventory files/etc are the
> > same
> > > > for
> > > > > > OSS
> > > > > > >> > Image
> > > > > > >> > > > and
> > > > > > >> > > > > > > > Docker Official Images?
> > > > > > >> > > > > > > > If no, then why not? Could we make them
> identical
> > so
> > > > > that
> > > > > > we
> > > > > > >> > > don't
> > > > > > >> > > > > > have to
> > > > > > >> > > > > > > > build 2 images for each release?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thank you.
> > > > > > >> > > > > > > > Luke
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> > > > > > >> > > > manikumar.reddy@gmail.com>
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Hi Krishna,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks for the KIP.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I think Docker Official Images will be
> > beneficial
> > > to
> > > > > the
> > > > > > >> > Kafka
> > > > > > >> > > > > > community.
> > > > > > >> > > > > > > > > Few queries below.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > 1. Will the Docker inventory files/etc are the
> > > same
> > > > > for
> > > > > > >> OSS
> > > > > > >> > > > Image and
> > > > > > >> > > > > > > > > Docker Official Images
> > > > > > >> > > > > > > > > 2. I am a bit worried about the new steps to
> the
> > > > > release
> > > > > > >> > > process.
> > > > > > >> > > > > > Maybe
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > should consider Docker Official Images release
> > as
> > > > > > >> > Post-Release
> > > > > > >> > > > > > activity.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> > > > > > >> > > > krishvora01@gmail.com>
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > Hi Hector,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks for reaching out. This KIP builds on
> > top
> > > of
> > > > > > >> KIP-975
> > > > > > >> > > > > > > > > > <
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > and
> > > > > > >> > > > > > > > > > aims to introduce a JVM-based Docker
> Official
> > > > Image
> > > > > > (DOI
> > > > > > >> > > > > > > > > > <
> > > > > > >> https://docs.docker.com/trusted-content/official-images/
> > > > > > >> > >)
> > > > > > >> > > > for
> > > > > > >> > > > > > Apache
> > > > > > >> > > > > > > > > > Kafka that will be visible under Docker
> > Official
> > > > > > Images
> > > > > > >> > > > > > > > > > <
> > > > > > https://hub.docker.com/search?image_filter=official&q=
> > > > > > >> >.
> > > > > > >> > > Once
> > > > > > >> > > > > > > > > implemented
> > > > > > >> > > > > > > > > > for Apache Kafka, for each release, there
> will
> > > be
> > > > > one
> > > > > > >> more
> > > > > > >> > > > > > JVM-based
> > > > > > >> > > > > > > > > Docker
> > > > > > >> > > > > > > > > > image available to users.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Currently, we already have an OSS sponsored
> > > image,
> > > > > > which
> > > > > > >> > was
> > > > > > >> > > > > > introduced
> > > > > > >> > > > > > > > > via
> > > > > > >> > > > > > > > > > KIP-975 (apache/kafka <
> > > > > > >> > > > https://hub.docker.com/r/apache/kafka/tags
> > > > > > >> > > > > > >)
> > > > > > >> > > > > > > > > which
> > > > > > >> > > > > > > > > > comes under The Apache Software Foundation <
> > > > > > >> > > > > > > > > > https://hub.docker.com/u/apache> in
> > > > > > >> > > > > > > > > > Docker Hub. The new Docker Image is the
> Docker
> > > > > > Official
> > > > > > >> > Image
> > > > > > >> > > > > > (DOI),
> > > > > > >> > > > > > > > > which
> > > > > > >> > > > > > > > > > will be built and maintained by Docker
> > > Community.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > For example, for a release version like
> 3.8.0
> > we
> > > > > will
> > > > > > >> have
> > > > > > >> > > two
> > > > > > >> > > > JVM
> > > > > > >> > > > > > > > based
> > > > > > >> > > > > > > > > > docker images:-
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored
> image)
> > > > > > >> > > > > > > > > >    - kafka:3.8.0 (Docker Official image)
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > I have added the same in the KIP too for
> > > > everyone's
> > > > > > >> > > reference.
> > > > > > >> > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > Krish.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector
> > Geraldino
> > > > > > >> > (BLOOMBERG/
> > > > > > >> > > > 919
> > > > > > >> > > > > > 3RD
> > > > > > >> > > > > > > > A) <
> > > > > > >> > > > > > > > > > hgeraldino@bloomberg.net> wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > Hi,
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > What is the difference between this KIP
> and
> > > > > KIP-975:
> > > > > > >> > Docker
> > > > > > >> > > > > > Image for
> > > > > > >> > > > > > > > > > > Apache Kafka?
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24
> > > > 07:30:07
> > > > > > >> > > UTC-4:00To:
> > > > > > >> > > > > > > > > > > dev@kafka.apache.org
> > > > > > >> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker
> Official
> > > > Image
> > > > > > for
> > > > > > >> > > Apache
> > > > > > >> > > > > > Kafka
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Hi everyone,
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > I would like to start the discussion on
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > > >> > > > > > > > > > > age+for+Apache+Kafka
> > > > > > >> > > > > > > > > > >  .
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > This KIP aims to introduce JVM based
> Docker
> > > > > Official
> > > > > > >> > Image
> > > > > > >> > > > (DOI)
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > > > > Apache
> > > > > > >> > > > > > > > > > > Kafka.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Regards,
> > > > > > >> > > > > > > > > > > Krish.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >>
> > > > > > >> [image: Confluent] <https://www.confluent.io>
> > > > > > >> Prabha Manepalli
> > > > > > >> Product Manager for Confluent Platform Security, Docker
> > > > > > >> linkedin.com/in/prabhamanepalli
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > [image: Confluent] <https://www.confluent.io>
> > > Prabha Manepalli
> > > Product Manager for Confluent Platform Security, Docker Containers
> > > linkedin.com/in/prabhamanepalli
> > >
> >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Vedarth Sharma <ve...@gmail.com>]

Hey Chris,

Once we provide the definitions to docker, they should take care of
everything from there. They mentioned here
<https://github.com/docker-library/official-images?tab=readme-ov-file#library-definition-files>
that
the image will be rebuilt when the base image is updated. Hence active
rebuilds won't require any changes from our side.
If we are packaging something which may contain a CVE, like some jar, then
the onus will be on us to patch it, but it will be upto us whether we
consider the threat severe enough to fix and when we want to provide the
fixed version. Having Docker Official Image will not impact the frequency
of our releases. It will be the Apache Kafka community's call on when a
release goes and Docker Official Image will be released accordingly as per
the KIP. source
<https://github.com/docker-library/faq?tab=readme-ov-file#image-building>

As mentioned in Docker's documentation as well "In essence we strive to
heed upstream's recommendations on how they intend for their software to be
consumed." source
<https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images>
Docker Official Image will rely on upstream's recommendation for
functionality. But I do agree that since Docker's stance on this might
change in future it makes sense to put a safeguard that will not allow any
functionality changes get incorporated as part of the vetting process. I
have updated the KIP to reflect the same.

KIP-975 has a well defined public interface based on how configs can be
supplied and how it can be used. I am not sure if we put that label on it
during discussions. I am happy to have a separate email thread on it to
iron things out.

I hope this addresses all of your concerns!

Thanks and regards,
Vedarth

On Mon, May 13, 2024 at 10:55 PM Chris Egerton <ch...@aiven.io.invalid>
wrote:

> Thanks both for your responses! Friendly reminder: again, better to provide
> a quote instead of just a link :)
>
> I've seen a bit about image rebuilding to handle CVEs but I'm a little
> unclear on how this would work in practice, and I couldn't find any
> concrete details in any of the links. Does Docker do this automatically for
> DOIs? Or will the onus be on us to put out patched images? Would this lead
> to us putting out images more quickly than we put out standard releases? As
> a plus, it does look like DOIs get the benefit of Docker Scout [1] for
> free, which is nice, but it's still unclear who'd be doing the rest of the
> work on that front.
>
> As far as this point from Vedarth goes:
>
> > By incorporating the source code of the Docker Official Image into our
> > AK ecosystem, we gain control over its functionality, ensuring alignment
> > with the OSS Docker image. This ensures a seamless experience for users
> who
> > may need to transition between these images.
>
> This captures my concern with the KIP pretty well. If there's any
> significant divergence in behavior (not just build methodology) between the
> apache/kafka image and what Docker requires for a Kafka DOI, how are we
> going to vet these changes moving forward? Under the "Post Release Process
> - if Dockerhub folks suggest changes to the Dockerfiles:" header, this KIP
> proposes that we port all suggested changes for the DOI to
> the docker/jvm/Dockerfile image, but this seems a bit too permissive. As an
> alternative, we could state that all build-related changes can be done with
> a PR on the apache/kafka GitHub repo (which will require approval from a
> single committer), but any functional changes will require a KIP.
>
> Finally, during KIP-975 was there discussion on what we would count as the
> public interface for the apache/kafka image? If not, it'd be nice to get
> that ironed out since it may make future discussions around our Docker
> images quicker, but I don't think this is necessary for KIP-1028.
>
> [1] - https://www.docker.com/products/docker-scout/
>
> On Mon, May 13, 2024 at 4:37 AM Prabha Manepalli
> <mp...@confluent.io.invalid> wrote:
>
> > Hi Chris,
> >
> > Sharing the requested links explaining why Docker Official images are
> > considered more secure -
> >
> >
> https://www.docker.com/blog/enhancing-security-and-transparency-with-docker-official-images/
> > and
> >
> >
> https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-cves
> >
> > I hope these links help you understand why we need Docker Official images
> > for organisations with stringent security compliance requirements for
> their
> > Kafka workloads.
> >
> > Thank you.
> >
> >
> >
> > On Sun, May 12, 2024 at 3:33 PM Vedarth Sharma <vedarth.sharma@gmail.com
> >
> > wrote:
> >
> > > Hey Chris!
> > >
> > > Functionality wise, we don't intend to have any differences between OSS
> > > Docker Image and Docker Official Image.
> > > The Docker Official Image will be the recommended one.
> > > Since the Docker Official Image might be delayed due to review done by
> > > Docker, images on apache/kafka (OSS Docker Image) can be used by users.
> > >
> > > 1) I read https://docs.docker.com/trusted-content/official-images/ and
> > > > didn't find anything on that page or immediately around it that
> > explains
> > > > what compliance requirements might be satisfied by a DOI that
> couldn't
> > be
> > > > satisfied by the existing apache/kafka image. Can you elaborate? Feel
> > > free
> > > > to provide another link, but please also quote the relevant sections
> > from
> > > > it (as StackOverflow likes to say, links can grow stale over time).
> > >
> > >
> > >    - If a user's selection is confined solely to Docker Official
> Images,
> > >    this Docker Image will ensure their access to Kafka.
> > >    - Details on specific advantages of Docker Official Images can be
> > found
> > >    at
> > >
> > >
> >
> https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
> > >    .
> > >    - The Docker Official Image will be actively rebuilt for updates and
> > >    security fixes.
> > >    - It's true we can provide exactly the same benefits in the OSS
> Docker
> > >    Image as well. But it won't have the Docker Official Image badge and
> > it
> > >    will add additional overhead for Apache Kafka community.
> > >    - The fact that it will have Docker Official Image badge will set it
> > >    apart from the OSS Docker Image.
> > >    - Also the ability to do just docker pull kafka to get the kafka
> > docker
> > >    image will only be possible with Docker Official Image.
> > >
> > >
> > > 2) It would be great to see a brief summary of the differences in these
> > > > images included in the KIP, in order to try to gauge how this would
> > look
> > > to
> > > > our users.
> > >
> > >
> > >    - Functionally, both Docker images will remain identical.
> > >    - The variance lies primarily in the methodologies of building and
> > >    validation, as outlined in the updated KIP.
> > >
> > >
> > > 3) What I suggested last time was not a separate apache/apache-docker
> > > > repository, but a repository controlled entirely by Docker. The DOI
> > docs
> > > > [1] state that "While it's preferable to have upstream software
> authors
> > > > maintaining their Docker Official Images, this isn't a strict
> > > requirement",
> > > > which I take to mean that it's not required for an Apache Kafka DOI
> to
> > > live
> > > > under the apache organization on GitHub. It also seems like there's
> > > > precedent for this: images for MySQL [2] and PHP [3] already exist
> > under
> > > > the control of Docker. The reason I think this is worth considering
> is
> > > that
> > > > Docker can arbitrarily change the eligibility requirements for their
> > > > official images at any time, and it doesn't seem like there's a clear
> > > > process in the KIP for judging how we should respond to these changes
> > (in
> > > > fact, it seems like the idea in the KIP is that we should make any
> > change
> > > > required with no further vetting beyond possibly a pull request on
> > > > apache/kafka, which would require approval by a committer). By
> hosting
> > > the
> > > > DOI definitions ourselves (either in apache/kafka, or in a
> theoretical
> > > > apache/docker-kafka repository), we take responsibility for the
> image,
> > > even
> > > > if the owner on Docker Hub is Docker, not Apache. If the code lives
> > > > elsewhere, then (as long as basic trademark and possibly security
> > > > guidelines are respected) Apache doesn't have to concern itself at
> all
> > > with
> > > > the image and the maintainers are free to make whatever changes they
> > want
> > > > to it in order to meet Docker's requirements.
> > >
> > >
> > >    - By incorporating the source code of the Docker Official Image into
> > our
> > >    AK ecosystem, we gain control over its functionality, ensuring
> > alignment
> > >    with the OSS Docker image. This ensures a seamless experience for
> > users
> > > who
> > >    may need to transition between these images.
> > >    - Maintaining both images within the same community facilitates ease
> > of
> > >    management and fosters a singular source of truth.
> > >    - While Apache may not retain ownership of the hosted Docker
> Official
> > >    Image, we are, in essence, providing Docker with a foundation that
> > > aligns
> > >    with their established guidelines as well as remains consistent with
> > OSS
> > >    Docker Image apache/kafka.
> > >    - Any future alterations to the functionality can be seamlessly
> > >    propagated across both the OSS and Official Docker Images.
> > >
> > > I think these reasons must be why a lot of the Apache projects choose
> to
> > > host the docker definitions themselves. The responsibility of owning
> the
> > > definitions comes with benefits as well that we should also consider.
> > >
> > > Let us know if you have any further questions!
> > >
> > > Thanks and regards,
> > > Vedarth
> > >
> > > On Fri, May 10, 2024 at 7:56 PM Chris Egerton <chrise@aiven.io.invalid
> >
> > > wrote:
> > >
> > > > Hi Krish and Prabha,
> > > >
> > > > Thanks for your replies. I still have some follow-up questions:
> > > >
> > > > 1) I read https://docs.docker.com/trusted-content/official-images/
> and
> > > > didn't find anything on that page or immediately around it that
> > explains
> > > > what compliance requirements might be satisfied by a DOI that
> couldn't
> > be
> > > > satisfied by the existing apache/kafka image. Can you elaborate? Feel
> > > free
> > > > to provide another link, but please also quote the relevant sections
> > from
> > > > it (as StackOverflow likes to say, links can grow stale over time).
> > > >
> > > > 2) It would be great to see a brief summary of the differences in
> these
> > > > images included in the KIP, in order to try to gauge how this would
> > look
> > > to
> > > > our users.
> > > >
> > > > 3) What I suggested last time was not a separate apache/apache-docker
> > > > repository, but a repository controlled entirely by Docker. The DOI
> > docs
> > > > [1] state that "While it's preferable to have upstream software
> authors
> > > > maintaining their Docker Official Images, this isn't a strict
> > > requirement",
> > > > which I take to mean that it's not required for an Apache Kafka DOI
> to
> > > live
> > > > under the apache organization on GitHub. It also seems like there's
> > > > precedent for this: images for MySQL [2] and PHP [3] already exist
> > under
> > > > the control of Docker. The reason I think this is worth considering
> is
> > > that
> > > > Docker can arbitrarily change the eligibility requirements for their
> > > > official images at any time, and it doesn't seem like there's a clear
> > > > process in the KIP for judging how we should respond to these changes
> > (in
> > > > fact, it seems like the idea in the KIP is that we should make any
> > change
> > > > required with no further vetting beyond possibly a pull request on
> > > > apache/kafka, which would require approval by a committer). By
> hosting
> > > the
> > > > DOI definitions ourselves (either in apache/kafka, or in a
> theoretical
> > > > apache/docker-kafka repository), we take responsibility for the
> image,
> > > even
> > > > if the owner on Docker Hub is Docker, not Apache. If the code lives
> > > > elsewhere, then (as long as basic trademark and possibly security
> > > > guidelines are respected) Apache doesn't have to concern itself at
> all
> > > with
> > > > the image and the maintainers are free to make whatever changes they
> > want
> > > > to it in order to meet Docker's requirements.
> > > >
> > > > [1] -
> > > >
> https://docs.docker.com/trusted-content/official-images/contributing/,
> > > > second paragraph under "Contributing to Docker Official Images"
> > > > [2] - https://github.com/docker-library/mysql
> > > > [3] - https://github.com/docker-library/php
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Fri, May 10, 2024 at 7:46 AM Krish Vora <kr...@gmail.com>
> > > wrote:
> > > >
> > > > > Hey Chris,
> > > > >
> > > > > We have responded to the initial round of queries. And as you
> > mentioned
> > > > in
> > > > > your comment, you may have more questions related to this KIP.
> Please
> > > let
> > > > > us know if you have any.
> > > > > We want to start the voting for this KIP soon, as we intend to
> > include
> > > it
> > > > > in the 3.8.0 release.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Krish.
> > > > >
> > > > > On Wed, May 8, 2024 at 6:19 PM Krish Vora <kr...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Chris. Thanks for the questions.
> > > > > >
> > > > > > 3. Would a separate Docker-owned repository be out of the
> question?
> > > I'm
> > > > > >> guessing there are some trademark issues that might get in the
> > way,
> > > > but
> > > > > >> it's worth exploring since the entire purpose of this KIP seems
> to
> > > be
> > > > to
> > > > > >> provide images that are vetted and designed by Docker more than
> by
> > > the
> > > > > >> Apache Kafka contributors/committers/PMC.
> > > > > >
> > > > > >
> > > > > >
> > > > > >    - The process for introducing a Docker Official Image involves
> > > > > >       - Hosting the Dockerfile in the Apache Kafka repository and
> > > > > >       - Providing the path to this Dockerfile to Docker Hub in
> > Docker
> > > > > >       Hub’s own repo
> > > > > >       <
> > > > >
> > https://github.com/docker-library/official-images/tree/master/library>
> > > > > >       .
> > > > > >    - This ensures that any updates to the Dockerfile in the AK
> > > > repository
> > > > > >    are directly applicable to the docker official images
> available
> > on
> > > > > Docker
> > > > > >    Hub.
> > > > > >
> > > > > >
> > > > > >    - We also did not find any added advantage to create a
> separate
> > > > > >    repository named apache-docker within the Apache GitHub
> > > > organization.
> > > > > >
> > > > > > Thanks,
> > > > > > Krish.
> > > > > >
> > > > > > On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli
> > > > > > <mp...@confluent.io.invalid> wrote:
> > > > > >
> > > > > >> Hi Chris,  I would like to add more context to this KIP's
> > > motivation.
> > > > > >> Vedarth and Krish, please weigh in with your inputs.
> > > > > >>
> > > > > >> In the motivation section it's stated that "Several other Apache
> > > > > projects,
> > > > > >> > like Flink, Spark, Solr, have already released Docker Official
> > > > Images,
> > > > > >> with
> > > > > >> > download figures ranging from 50 million to over 1 billion.
> > These
> > > > > >> numbers
> > > > > >> > highlight the significant demand among users." But then
> > > immediately
> > > > > >> > afterwards, we learn that "Also the Docker Official Images are
> > > > always
> > > > > >> the
> > > > > >> > top 1 search result, irrespective of the number of downloads."
> > > > > Wouldn't
> > > > > >> a
> > > > > >> > high number of downloads for an image naturally follow from
> > being
> > > > the
> > > > > >> top
> > > > > >> > search result? It seems like we can't necessarily assume that
> > > Docker
> > > > > >> > Official Images are inherently more desirable for users based
> > > solely
> > > > > on
> > > > > >> > download statistics.
> > > > > >> >
> > > > > >>
> > > > > >> *My thoughts: *Unlike the Sponsored OSS image, the Docker
> Official
> > > > image
> > > > > >> is
> > > > > >> more desirable for workloads that have stringent compliance
> > > > > requirements.
> > > > > >> More details on why official images are more trusted are
> > documented
> > > > here
> > > > > >> <https://docs.docker.com/trusted-content/official-images/>. The
> > > > Docker
> > > > > >> Official image would also help an absolutely new Kafka beginner
> > who
> > > > > might
> > > > > >> not know about Apache or the concept of Sponsored images. We
> want
> > to
> > > > > make
> > > > > >> it easier for Kafka beginners to discover the Kafka image
> through
> > > > > >> DockerHub.
> > > > > >>
> > > > > >>
> > > > > >> Can you elaborate on the value that these new images would add
> > from
> > > a
> > > > > >> > user's perspective? I'm hesitant to introduce another image,
> > since
> > > > it
> > > > > >> adds
> > > > > >> > to the cognitive burden of people who will inevitably have to
> > > answer
> > > > > the
> > > > > >> > question of "What are the differences between all of the
> > available
> > > > > >> images
> > > > > >> > and which one is best for my use case?"
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> *My thoughts: *This is a valid concern to address. The response
> to
> > > the
> > > > > >> above question addresses the value-add this new Docker Official
> > > image
> > > > > >> would
> > > > > >> provide. I also agree we need a clear distinction between each
> of
> > > > these
> > > > > >> images to be well documented. We plan to update the AK website
> > with
> > > > > >> details
> > > > > >> on how, why, and when a developer would want to use each of
> these
> > > > > >> particular images(KIP-974,975,1028).
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Prabha.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton
> > > <chrise@aiven.io.invalid
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Vedarth and Krish,
> > > > > >> >
> > > > > >> > Thanks for the KIP! I have to admit I'm a little skeptical;
> > > > hopefully
> > > > > >> you
> > > > > >> > can help me understand the need for these additional images.
> > > > > >> >
> > > > > >> > 1) In the motivation section it's stated that "Several other
> > > Apache
> > > > > >> > projects, like Flink, Spark, Solr, have already released
> Docker
> > > > > Official
> > > > > >> > Images, with download figures ranging from 50 million to over
> 1
> > > > > billion.
> > > > > >> > These numbers highlight the significant demand among users."
> But
> > > > then
> > > > > >> > immediately afterwards, we learn that "Also the Docker
> Official
> > > > Images
> > > > > >> are
> > > > > >> > always the top 1 search result, irrespective of the number of
> > > > > >> downloads."
> > > > > >> > Wouldn't a high number of downloads for an image naturally
> > follow
> > > > from
> > > > > >> > being the top search result? It seems like we can't
> necessarily
> > > > assume
> > > > > >> that
> > > > > >> > Docker Official Images are inherently more desirable for users
> > > based
> > > > > >> solely
> > > > > >> > on download statistics.
> > > > > >> >
> > > > > >> > 2) Can you elaborate on the value that these new images would
> > add
> > > > > from a
> > > > > >> > user's perspective? I'm hesitant to introduce another image,
> > since
> > > > it
> > > > > >> adds
> > > > > >> > to the cognitive burden of people who will inevitably have to
> > > answer
> > > > > the
> > > > > >> > question of "What are the differences between all of the
> > available
> > > > > >> images
> > > > > >> > and which one is best for my use case?"
> > > > > >> >
> > > > > >> > 3) Would a separate Docker-owned repository be out of the
> > > question?
> > > > > I'm
> > > > > >> > guessing there are some trademark issues that might get in the
> > > way,
> > > > > but
> > > > > >> > it's worth exploring since the entire purpose of this KIP
> seems
> > to
> > > > be
> > > > > to
> > > > > >> > provide images that are vetted and designed by Docker more
> than
> > by
> > > > the
> > > > > >> > Apache Kafka contributors/committers/PMC.
> > > > > >> >
> > > > > >> > I may have more questions later but wanted to get this initial
> > > round
> > > > > out
> > > > > >> > now without trying to list everything first.
> > > > > >> >
> > > > > >> > Looking forward to your thoughts!
> > > > > >> >
> > > > > >> > Cheers,
> > > > > >> >
> > > > > >> > Chris
> > > > > >> >
> > > > > >> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <
> > > > > >> vedarth.sharma@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hey folks,
> > > > > >> > >
> > > > > >> > > Thanks a lot for reviewing the KIP and providing feedback.
> > > > > >> > > The discussion thread seems resolved and KIP has been
> updated
> > > > > >> > accordingly.
> > > > > >> > > We will be starting the voting thread for this KIP in the
> next
> > > few
> > > > > >> days.
> > > > > >> > > Please take a look at the KIP and let us know if any further
> > > > > >> discussion
> > > > > >> > > is needed.
> > > > > >> > >
> > > > > >> > > Thanks and regards,
> > > > > >> > > Vedarth
> > > > > >> > >
> > > > > >> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <
> > > > > manikumar.reddy@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Thanks Krish. KIP looks good to me.
> > > > > >> > > >
> > > > > >> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <
> > > > krishvora01@gmail.com
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > Hi Manikumar,
> > > > > >> > > > >
> > > > > >> > > > > Thanks for the comments.
> > > > > >> > > > >
> > > > > >> > > > > Maybe as part of the release process, RM can create a
> JIRA
> > > for
> > > > > >> this
> > > > > >> > > > > > task. This can be taken by RM or any comitter or any
> > > > > contributor
> > > > > >> > > (with
> > > > > >> > > > > > some help from commiters to run "Docker Image
> > Preparation
> > > > via
> > > > > >> > GitHub
> > > > > >> > > > > > Actions:"
> > > > > >> > > > >
> > > > > >> > > > > This sounds like a good idea. This step would be
> > beneficial.
> > > > By
> > > > > >> > > creating
> > > > > >> > > > a
> > > > > >> > > > > JIRA ticket, it will also serve as a reminder to
> complete
> > > the
> > > > > >> > > > post-release
> > > > > >> > > > > steps for the Docker official images. Have updated the
> KIP
> > > > with
> > > > > >> this
> > > > > >> > > > step.
> > > > > >> > > > >
> > > > > >> > > > > Is this using GitHub Actions workflow? or manual
> testing?
> > > > > >> > > > >
> > > > > >> > > > > This will be done by a Github Actions workflow, which
> will
> > > > test
> > > > > >> the
> > > > > >> > > > static
> > > > > >> > > > > Docker Official Image assets for a specific release
> > version.
> > > > > >> > > > >
> > > > > >> > > > > Is it mandatory for RM/comitters to raise the PR to
> Docker
> > > > Hub’s
> > > > > >> > > > > > official images repository (or) can it be done by any
> > > > > >> contributor.
> > > > > >> > > > >
> > > > > >> > > > > I believe that it can be done by any contributor (ref:
> > This
> > > > link
> > > > > >> > > > > <
> > > > > >> >
> > > >
> https://docs.docker.com/trusted-content/official-images/contributing/
> > > > > >> > > >
> > > > > >> > > > > quotes "*Anyone can provide feedback, contribute code,
> > > suggest
> > > > > >> > process
> > > > > >> > > > > changes, or even propose a new Official Image.*")
> > > > > >> > > > >
> > > > > >> > > > > Also I was thinking, once the KIP gets voted, we should
> > try
> > > to
> > > > > >> > release
> > > > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This
> will
> > > help
> > > > > us
> > > > > >> to
> > > > > >> > > > > > validate the process and allow us to fix any changes
> > > > suggested
> > > > > >> by
> > > > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > > > >> > > > >
> > > > > >> > > > > This sounds like a great idea. This KIP proposes release
> > of
> > > > DOI
> > > > > >> as a
> > > > > >> > > > > post-release process, which can be done anytime post
> > > release.
> > > > > >> Since
> > > > > >> > > 3.7.0
> > > > > >> > > > > is already released, we can perform these steps for that
> > > > release
> > > > > >> too.
> > > > > >> > > By
> > > > > >> > > > > the time the KIP gets implemented, if 3.7.1 is released,
> > we
> > > > > could
> > > > > >> do
> > > > > >> > > > these
> > > > > >> > > > > steps for 3.7.1, instead of 3.7.0. This would allow us
> to
> > > make
> > > > > >> > changes
> > > > > >> > > to
> > > > > >> > > > > the Dockerfiles and other assets based on feedback from
> > > Docker
> > > > > Hub
> > > > > >> > > before
> > > > > >> > > > > the release of version 3.8.0.
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Krish.
> > > > > >> > > > >
> > > > > >> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
> > > > > >> > manikumar.reddy@gmail.com>
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi Krish,
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks for the updated KIP. a few comments below.
> > > > > >> > > > > >
> > > > > >> > > > > > > "These actions can be carried out by the RM or any
> > > > > contributor
> > > > > >> > post
> > > > > >> > > > the
> > > > > >> > > > > > release process."
> > > > > >> > > > > > Maybe as part of the release process, RM can create a
> > JIRA
> > > > for
> > > > > >> this
> > > > > >> > > > > > task. This can be taken by RM or any comitter or any
> > > > > contributor
> > > > > >> > > (with
> > > > > >> > > > > > some help from commiters to run "Docker Image
> > Preparation
> > > > via
> > > > > >> > GitHub
> > > > > >> > > > > > Actions:"
> > > > > >> > > > > >
> > > > > >> > > > > > > "Perform Docker build tests to ensure image
> integrity"
> > > > > >> > > > > > Is this using GitHub Actions workflow? or manual
> > testing?
> > > > > >> > > > > >
> > > > > >> > > > > > > "The RM will manually raise the final PR to Docker
> > Hub’s
> > > > > >> official
> > > > > >> > > > images
> > > > > >> > > > > > repository using the contents of the generated file"
> > > > > >> > > > > >  Is it mandatory for RM/comitters to raise the PR to
> > > Docker
> > > > > >> Hub’s
> > > > > >> > > > > > official images repository (or) can it be done by any
> > > > > >> contributor.
> > > > > >> > > > > >
> > > > > >> > > > > > Also I was thinking, once the KIP gets voted, we
> should
> > > try
> > > > to
> > > > > >> > > release
> > > > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This
> will
> > > help
> > > > > us
> > > > > >> to
> > > > > >> > > > > > validate the process and allow us to fix any changes
> > > > suggested
> > > > > >> by
> > > > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks,
> > > > > >> > > > > >
> > > > > >> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <
> > > > > >> krishvora01@gmail.com>
> > > > > >> > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > Hi Manikumar and Luke.
> > > > > >> > > > > > > Thanks for the questions.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 1. No, the Docker inventory files and configurations
> > > will
> > > > > not
> > > > > >> be
> > > > > >> > > the
> > > > > >> > > > same
> > > > > >> > > > > > > for Open Source Software (OSS) Images and Docker
> > > Official
> > > > > >> Images
> > > > > >> > > > (DOI).
> > > > > >> > > > > > >
> > > > > >> > > > > > > For OSS images, the Dockerfile located in
> > > > > >> docker/jvm/dockerfile
> > > > > >> > is
> > > > > >> > > > > > > utilized. This process is integrated with the
> existing
> > > > > release
> > > > > >> > > > pipeline
> > > > > >> > > > > > as
> > > > > >> > > > > > > outlined in KIP-975
> > > > > >> > > > > > > <
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > > > > >> > > > > > >,
> > > > > >> > > > > > > where the Kafka URL is provided as a build argument.
> > > This
> > > > > >> method
> > > > > >> > > > allows
> > > > > >> > > > > > for
> > > > > >> > > > > > > building, testing, and releasing OSS images
> > dynamically.
> > > > The
> > > > > >> OSS
> > > > > >> > > > images
> > > > > >> > > > > > > will continue to be released under the standard
> > release
> > > > > >> process .
> > > > > >> > > > > > >
> > > > > >> > > > > > > In contrast, the release process for DOIs requires
> > > > providing
> > > > > >> the
> > > > > >> > > > Docker
> > > > > >> > > > > > Hub
> > > > > >> > > > > > > team with a specific directory for each version
> > release
> > > > that
> > > > > >> > > > contains a
> > > > > >> > > > > > > standalone Dockerfile. These Dockerfiles are
> designed
> > to
> > > > be
> > > > > >> > > > > > > self-sufficient, hence require hardcoded values
> > instead
> > > of
> > > > > >> > relying
> > > > > >> > > on
> > > > > >> > > > > > build
> > > > > >> > > > > > > arguments. To accommodate this, in our proposed
> > > approach,
> > > > a
> > > > > >> new
> > > > > >> > > > directory
> > > > > >> > > > > > > named docker_official_images has been created. This
> > > > > directory
> > > > > >> > > > contains
> > > > > >> > > > > > > version-specific directories, having Dockerfiles
> with
> > > > > >> hardcoded
> > > > > >> > > > > > > configurations for each release, acting as the
> source
> > of
> > > > > truth
> > > > > >> > for
> > > > > >> > > > DOI
> > > > > >> > > > > > > releases. The hardcoded dockerfiles will be created
> > > using
> > > > > the
> > > > > >> > > > > > > docker/jvm/dockerfile as a template. Thus, as part
> of
> > > post
> > > > > >> > release
> > > > > >> > > we
> > > > > >> > > > > > will
> > > > > >> > > > > > > be creating a Dockerfile that will be reviewed by
> the
> > > > > >> Dockerhub
> > > > > >> > > > community
> > > > > >> > > > > > > and might need changes as per their review. This
> > > approach
> > > > > >> ensures
> > > > > >> > > > that
> > > > > >> > > > > > DOIs
> > > > > >> > > > > > > are built consistently and meet the specific
> > > requirements
> > > > > set
> > > > > >> by
> > > > > >> > > > Docker
> > > > > >> > > > > > Hub.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 2. Yes Manikumar, transitioning the release of
> Docker
> > > > > Official
> > > > > >> > > Images
> > > > > >> > > > > > (DOI)
> > > > > >> > > > > > > to a post-release activity does address the concerns
> > > about
> > > > > >> > > > complicating
> > > > > >> > > > > > the
> > > > > >> > > > > > > release process. Initially, we considered
> > incorporating
> > > > DOI
> > > > > >> > release
> > > > > >> > > > > > > directly into Kafka's release workflow. However,
> this
> > > > > approach
> > > > > >> > > > > > > significantly increased the RMs workload due to the
> > > > addition
> > > > > >> of
> > > > > >> > > > numerous
> > > > > >> > > > > > > steps, complicating the process. By designating the
> > DOI
> > > > > >> release
> > > > > >> > as
> > > > > >> > > a
> > > > > >> > > > > > > post-release task, we maintain the original release
> > > > process.
> > > > > >> This
> > > > > >> > > > > > > adjustment allows for the DOI release to be done
> after
> > > the
> > > > > >> main
> > > > > >> > > > release.
> > > > > >> > > > > > We
> > > > > >> > > > > > > have revised the KIP to reflect that DOI releases
> will
> > > now
> > > > > >> occur
> > > > > >> > > > after
> > > > > >> > > > > > the
> > > > > >> > > > > > > main release phase. Please review the updated
> document
> > > and
> > > > > >> > provide
> > > > > >> > > > any
> > > > > >> > > > > > > feedback you might have.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks,
> > > > > >> > > > > > > Krish.
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <
> > > > showuon@gmail.com
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Krishna,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I also have the same question as Manikumar raised:
> > > > > >> > > > > > > > 1. Will the Docker inventory files/etc are the
> same
> > > for
> > > > > OSS
> > > > > >> > Image
> > > > > >> > > > and
> > > > > >> > > > > > > > Docker Official Images?
> > > > > >> > > > > > > > If no, then why not? Could we make them identical
> so
> > > > that
> > > > > we
> > > > > >> > > don't
> > > > > >> > > > > > have to
> > > > > >> > > > > > > > build 2 images for each release?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Thank you.
> > > > > >> > > > > > > > Luke
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> > > > > >> > > > manikumar.reddy@gmail.com>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Hi Krishna,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks for the KIP.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I think Docker Official Images will be
> beneficial
> > to
> > > > the
> > > > > >> > Kafka
> > > > > >> > > > > > community.
> > > > > >> > > > > > > > > Few queries below.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > 1. Will the Docker inventory files/etc are the
> > same
> > > > for
> > > > > >> OSS
> > > > > >> > > > Image and
> > > > > >> > > > > > > > > Docker Official Images
> > > > > >> > > > > > > > > 2. I am a bit worried about the new steps to the
> > > > release
> > > > > >> > > process.
> > > > > >> > > > > > Maybe
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > should consider Docker Official Images release
> as
> > > > > >> > Post-Release
> > > > > >> > > > > > activity.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> > > > > >> > > > krishvora01@gmail.com>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Hi Hector,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks for reaching out. This KIP builds on
> top
> > of
> > > > > >> KIP-975
> > > > > >> > > > > > > > > > <
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > and
> > > > > >> > > > > > > > > > aims to introduce a JVM-based Docker Official
> > > Image
> > > > > (DOI
> > > > > >> > > > > > > > > > <
> > > > > >> https://docs.docker.com/trusted-content/official-images/
> > > > > >> > >)
> > > > > >> > > > for
> > > > > >> > > > > > Apache
> > > > > >> > > > > > > > > > Kafka that will be visible under Docker
> Official
> > > > > Images
> > > > > >> > > > > > > > > > <
> > > > > https://hub.docker.com/search?image_filter=official&q=
> > > > > >> >.
> > > > > >> > > Once
> > > > > >> > > > > > > > > implemented
> > > > > >> > > > > > > > > > for Apache Kafka, for each release, there will
> > be
> > > > one
> > > > > >> more
> > > > > >> > > > > > JVM-based
> > > > > >> > > > > > > > > Docker
> > > > > >> > > > > > > > > > image available to users.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Currently, we already have an OSS sponsored
> > image,
> > > > > which
> > > > > >> > was
> > > > > >> > > > > > introduced
> > > > > >> > > > > > > > > via
> > > > > >> > > > > > > > > > KIP-975 (apache/kafka <
> > > > > >> > > > https://hub.docker.com/r/apache/kafka/tags
> > > > > >> > > > > > >)
> > > > > >> > > > > > > > > which
> > > > > >> > > > > > > > > > comes under The Apache Software Foundation <
> > > > > >> > > > > > > > > > https://hub.docker.com/u/apache> in
> > > > > >> > > > > > > > > > Docker Hub. The new Docker Image is the Docker
> > > > > Official
> > > > > >> > Image
> > > > > >> > > > > > (DOI),
> > > > > >> > > > > > > > > which
> > > > > >> > > > > > > > > > will be built and maintained by Docker
> > Community.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > For example, for a release version like 3.8.0
> we
> > > > will
> > > > > >> have
> > > > > >> > > two
> > > > > >> > > > JVM
> > > > > >> > > > > > > > based
> > > > > >> > > > > > > > > > docker images:-
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > > > >> > > > > > > > > >    - kafka:3.8.0 (Docker Official image)
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > I have added the same in the KIP too for
> > > everyone's
> > > > > >> > > reference.
> > > > > >> > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > Krish.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector
> Geraldino
> > > > > >> > (BLOOMBERG/
> > > > > >> > > > 919
> > > > > >> > > > > > 3RD
> > > > > >> > > > > > > > A) <
> > > > > >> > > > > > > > > > hgeraldino@bloomberg.net> wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > Hi,
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > What is the difference between this KIP and
> > > > KIP-975:
> > > > > >> > Docker
> > > > > >> > > > > > Image for
> > > > > >> > > > > > > > > > > Apache Kafka?
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24
> > > 07:30:07
> > > > > >> > > UTC-4:00To:
> > > > > >> > > > > > > > > > > dev@kafka.apache.org
> > > > > >> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official
> > > Image
> > > > > for
> > > > > >> > > Apache
> > > > > >> > > > > > Kafka
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Hi everyone,
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > I would like to start the discussion on
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > >> > > > > > > > > > > age+for+Apache+Kafka
> > > > > >> > > > > > > > > > >  .
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > This KIP aims to introduce JVM based Docker
> > > > Official
> > > > > >> > Image
> > > > > >> > > > (DOI)
> > > > > >> > > > > > for
> > > > > >> > > > > > > > > > Apache
> > > > > >> > > > > > > > > > > Kafka.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Regards,
> > > > > >> > > > > > > > > > > Krish.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >>
> > > > > >> [image: Confluent] <https://www.confluent.io>
> > > > > >> Prabha Manepalli
> > > > > >> Product Manager for Confluent Platform Security, Docker
> > > > > >> linkedin.com/in/prabhamanepalli
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > [image: Confluent] <https://www.confluent.io>
> > Prabha Manepalli
> > Product Manager for Confluent Platform Security, Docker Containers
> > linkedin.com/in/prabhamanepalli
> >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Chris Egerton <ch...@aiven.io.INVALID>]

Thanks both for your responses! Friendly reminder: again, better to provide
a quote instead of just a link :)

I've seen a bit about image rebuilding to handle CVEs but I'm a little
unclear on how this would work in practice, and I couldn't find any
concrete details in any of the links. Does Docker do this automatically for
DOIs? Or will the onus be on us to put out patched images? Would this lead
to us putting out images more quickly than we put out standard releases? As
a plus, it does look like DOIs get the benefit of Docker Scout [1] for
free, which is nice, but it's still unclear who'd be doing the rest of the
work on that front.

As far as this point from Vedarth goes:

> By incorporating the source code of the Docker Official Image into our
> AK ecosystem, we gain control over its functionality, ensuring alignment
> with the OSS Docker image. This ensures a seamless experience for users
who
> may need to transition between these images.

This captures my concern with the KIP pretty well. If there's any
significant divergence in behavior (not just build methodology) between the
apache/kafka image and what Docker requires for a Kafka DOI, how are we
going to vet these changes moving forward? Under the "Post Release Process
- if Dockerhub folks suggest changes to the Dockerfiles:" header, this KIP
proposes that we port all suggested changes for the DOI to
the docker/jvm/Dockerfile image, but this seems a bit too permissive. As an
alternative, we could state that all build-related changes can be done with
a PR on the apache/kafka GitHub repo (which will require approval from a
single committer), but any functional changes will require a KIP.

Finally, during KIP-975 was there discussion on what we would count as the
public interface for the apache/kafka image? If not, it'd be nice to get
that ironed out since it may make future discussions around our Docker
images quicker, but I don't think this is necessary for KIP-1028.

[1] - https://www.docker.com/products/docker-scout/

On Mon, May 13, 2024 at 4:37 AM Prabha Manepalli
<mp...@confluent.io.invalid> wrote:

> Hi Chris,
>
> Sharing the requested links explaining why Docker Official images are
> considered more secure -
>
> https://www.docker.com/blog/enhancing-security-and-transparency-with-docker-official-images/
> and
>
> https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-cves
>
> I hope these links help you understand why we need Docker Official images
> for organisations with stringent security compliance requirements for their
> Kafka workloads.
>
> Thank you.
>
>
>
> On Sun, May 12, 2024 at 3:33 PM Vedarth Sharma <ve...@gmail.com>
> wrote:
>
> > Hey Chris!
> >
> > Functionality wise, we don't intend to have any differences between OSS
> > Docker Image and Docker Official Image.
> > The Docker Official Image will be the recommended one.
> > Since the Docker Official Image might be delayed due to review done by
> > Docker, images on apache/kafka (OSS Docker Image) can be used by users.
> >
> > 1) I read https://docs.docker.com/trusted-content/official-images/ and
> > > didn't find anything on that page or immediately around it that
> explains
> > > what compliance requirements might be satisfied by a DOI that couldn't
> be
> > > satisfied by the existing apache/kafka image. Can you elaborate? Feel
> > free
> > > to provide another link, but please also quote the relevant sections
> from
> > > it (as StackOverflow likes to say, links can grow stale over time).
> >
> >
> >    - If a user's selection is confined solely to Docker Official Images,
> >    this Docker Image will ensure their access to Kafka.
> >    - Details on specific advantages of Docker Official Images can be
> found
> >    at
> >
> >
> https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
> >    .
> >    - The Docker Official Image will be actively rebuilt for updates and
> >    security fixes.
> >    - It's true we can provide exactly the same benefits in the OSS Docker
> >    Image as well. But it won't have the Docker Official Image badge and
> it
> >    will add additional overhead for Apache Kafka community.
> >    - The fact that it will have Docker Official Image badge will set it
> >    apart from the OSS Docker Image.
> >    - Also the ability to do just docker pull kafka to get the kafka
> docker
> >    image will only be possible with Docker Official Image.
> >
> >
> > 2) It would be great to see a brief summary of the differences in these
> > > images included in the KIP, in order to try to gauge how this would
> look
> > to
> > > our users.
> >
> >
> >    - Functionally, both Docker images will remain identical.
> >    - The variance lies primarily in the methodologies of building and
> >    validation, as outlined in the updated KIP.
> >
> >
> > 3) What I suggested last time was not a separate apache/apache-docker
> > > repository, but a repository controlled entirely by Docker. The DOI
> docs
> > > [1] state that "While it's preferable to have upstream software authors
> > > maintaining their Docker Official Images, this isn't a strict
> > requirement",
> > > which I take to mean that it's not required for an Apache Kafka DOI to
> > live
> > > under the apache organization on GitHub. It also seems like there's
> > > precedent for this: images for MySQL [2] and PHP [3] already exist
> under
> > > the control of Docker. The reason I think this is worth considering is
> > that
> > > Docker can arbitrarily change the eligibility requirements for their
> > > official images at any time, and it doesn't seem like there's a clear
> > > process in the KIP for judging how we should respond to these changes
> (in
> > > fact, it seems like the idea in the KIP is that we should make any
> change
> > > required with no further vetting beyond possibly a pull request on
> > > apache/kafka, which would require approval by a committer). By hosting
> > the
> > > DOI definitions ourselves (either in apache/kafka, or in a theoretical
> > > apache/docker-kafka repository), we take responsibility for the image,
> > even
> > > if the owner on Docker Hub is Docker, not Apache. If the code lives
> > > elsewhere, then (as long as basic trademark and possibly security
> > > guidelines are respected) Apache doesn't have to concern itself at all
> > with
> > > the image and the maintainers are free to make whatever changes they
> want
> > > to it in order to meet Docker's requirements.
> >
> >
> >    - By incorporating the source code of the Docker Official Image into
> our
> >    AK ecosystem, we gain control over its functionality, ensuring
> alignment
> >    with the OSS Docker image. This ensures a seamless experience for
> users
> > who
> >    may need to transition between these images.
> >    - Maintaining both images within the same community facilitates ease
> of
> >    management and fosters a singular source of truth.
> >    - While Apache may not retain ownership of the hosted Docker Official
> >    Image, we are, in essence, providing Docker with a foundation that
> > aligns
> >    with their established guidelines as well as remains consistent with
> OSS
> >    Docker Image apache/kafka.
> >    - Any future alterations to the functionality can be seamlessly
> >    propagated across both the OSS and Official Docker Images.
> >
> > I think these reasons must be why a lot of the Apache projects choose to
> > host the docker definitions themselves. The responsibility of owning the
> > definitions comes with benefits as well that we should also consider.
> >
> > Let us know if you have any further questions!
> >
> > Thanks and regards,
> > Vedarth
> >
> > On Fri, May 10, 2024 at 7:56 PM Chris Egerton <ch...@aiven.io.invalid>
> > wrote:
> >
> > > Hi Krish and Prabha,
> > >
> > > Thanks for your replies. I still have some follow-up questions:
> > >
> > > 1) I read https://docs.docker.com/trusted-content/official-images/ and
> > > didn't find anything on that page or immediately around it that
> explains
> > > what compliance requirements might be satisfied by a DOI that couldn't
> be
> > > satisfied by the existing apache/kafka image. Can you elaborate? Feel
> > free
> > > to provide another link, but please also quote the relevant sections
> from
> > > it (as StackOverflow likes to say, links can grow stale over time).
> > >
> > > 2) It would be great to see a brief summary of the differences in these
> > > images included in the KIP, in order to try to gauge how this would
> look
> > to
> > > our users.
> > >
> > > 3) What I suggested last time was not a separate apache/apache-docker
> > > repository, but a repository controlled entirely by Docker. The DOI
> docs
> > > [1] state that "While it's preferable to have upstream software authors
> > > maintaining their Docker Official Images, this isn't a strict
> > requirement",
> > > which I take to mean that it's not required for an Apache Kafka DOI to
> > live
> > > under the apache organization on GitHub. It also seems like there's
> > > precedent for this: images for MySQL [2] and PHP [3] already exist
> under
> > > the control of Docker. The reason I think this is worth considering is
> > that
> > > Docker can arbitrarily change the eligibility requirements for their
> > > official images at any time, and it doesn't seem like there's a clear
> > > process in the KIP for judging how we should respond to these changes
> (in
> > > fact, it seems like the idea in the KIP is that we should make any
> change
> > > required with no further vetting beyond possibly a pull request on
> > > apache/kafka, which would require approval by a committer). By hosting
> > the
> > > DOI definitions ourselves (either in apache/kafka, or in a theoretical
> > > apache/docker-kafka repository), we take responsibility for the image,
> > even
> > > if the owner on Docker Hub is Docker, not Apache. If the code lives
> > > elsewhere, then (as long as basic trademark and possibly security
> > > guidelines are respected) Apache doesn't have to concern itself at all
> > with
> > > the image and the maintainers are free to make whatever changes they
> want
> > > to it in order to meet Docker's requirements.
> > >
> > > [1] -
> > > https://docs.docker.com/trusted-content/official-images/contributing/,
> > > second paragraph under "Contributing to Docker Official Images"
> > > [2] - https://github.com/docker-library/mysql
> > > [3] - https://github.com/docker-library/php
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Fri, May 10, 2024 at 7:46 AM Krish Vora <kr...@gmail.com>
> > wrote:
> > >
> > > > Hey Chris,
> > > >
> > > > We have responded to the initial round of queries. And as you
> mentioned
> > > in
> > > > your comment, you may have more questions related to this KIP. Please
> > let
> > > > us know if you have any.
> > > > We want to start the voting for this KIP soon, as we intend to
> include
> > it
> > > > in the 3.8.0 release.
> > > >
> > > > Thanks.
> > > >
> > > > Regards,
> > > >
> > > > Krish.
> > > >
> > > > On Wed, May 8, 2024 at 6:19 PM Krish Vora <kr...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Chris. Thanks for the questions.
> > > > >
> > > > > 3. Would a separate Docker-owned repository be out of the question?
> > I'm
> > > > >> guessing there are some trademark issues that might get in the
> way,
> > > but
> > > > >> it's worth exploring since the entire purpose of this KIP seems to
> > be
> > > to
> > > > >> provide images that are vetted and designed by Docker more than by
> > the
> > > > >> Apache Kafka contributors/committers/PMC.
> > > > >
> > > > >
> > > > >
> > > > >    - The process for introducing a Docker Official Image involves
> > > > >       - Hosting the Dockerfile in the Apache Kafka repository and
> > > > >       - Providing the path to this Dockerfile to Docker Hub in
> Docker
> > > > >       Hub’s own repo
> > > > >       <
> > > >
> https://github.com/docker-library/official-images/tree/master/library>
> > > > >       .
> > > > >    - This ensures that any updates to the Dockerfile in the AK
> > > repository
> > > > >    are directly applicable to the docker official images available
> on
> > > > Docker
> > > > >    Hub.
> > > > >
> > > > >
> > > > >    - We also did not find any added advantage to create a separate
> > > > >    repository named apache-docker within the Apache GitHub
> > > organization.
> > > > >
> > > > > Thanks,
> > > > > Krish.
> > > > >
> > > > > On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli
> > > > > <mp...@confluent.io.invalid> wrote:
> > > > >
> > > > >> Hi Chris,  I would like to add more context to this KIP's
> > motivation.
> > > > >> Vedarth and Krish, please weigh in with your inputs.
> > > > >>
> > > > >> In the motivation section it's stated that "Several other Apache
> > > > projects,
> > > > >> > like Flink, Spark, Solr, have already released Docker Official
> > > Images,
> > > > >> with
> > > > >> > download figures ranging from 50 million to over 1 billion.
> These
> > > > >> numbers
> > > > >> > highlight the significant demand among users." But then
> > immediately
> > > > >> > afterwards, we learn that "Also the Docker Official Images are
> > > always
> > > > >> the
> > > > >> > top 1 search result, irrespective of the number of downloads."
> > > > Wouldn't
> > > > >> a
> > > > >> > high number of downloads for an image naturally follow from
> being
> > > the
> > > > >> top
> > > > >> > search result? It seems like we can't necessarily assume that
> > Docker
> > > > >> > Official Images are inherently more desirable for users based
> > solely
> > > > on
> > > > >> > download statistics.
> > > > >> >
> > > > >>
> > > > >> *My thoughts: *Unlike the Sponsored OSS image, the Docker Official
> > > image
> > > > >> is
> > > > >> more desirable for workloads that have stringent compliance
> > > > requirements.
> > > > >> More details on why official images are more trusted are
> documented
> > > here
> > > > >> <https://docs.docker.com/trusted-content/official-images/>. The
> > > Docker
> > > > >> Official image would also help an absolutely new Kafka beginner
> who
> > > > might
> > > > >> not know about Apache or the concept of Sponsored images. We want
> to
> > > > make
> > > > >> it easier for Kafka beginners to discover the Kafka image through
> > > > >> DockerHub.
> > > > >>
> > > > >>
> > > > >> Can you elaborate on the value that these new images would add
> from
> > a
> > > > >> > user's perspective? I'm hesitant to introduce another image,
> since
> > > it
> > > > >> adds
> > > > >> > to the cognitive burden of people who will inevitably have to
> > answer
> > > > the
> > > > >> > question of "What are the differences between all of the
> available
> > > > >> images
> > > > >> > and which one is best for my use case?"
> > > > >> >
> > > > >>
> > > > >>
> > > > >> *My thoughts: *This is a valid concern to address. The response to
> > the
> > > > >> above question addresses the value-add this new Docker Official
> > image
> > > > >> would
> > > > >> provide. I also agree we need a clear distinction between each of
> > > these
> > > > >> images to be well documented. We plan to update the AK website
> with
> > > > >> details
> > > > >> on how, why, and when a developer would want to use each of these
> > > > >> particular images(KIP-974,975,1028).
> > > > >>
> > > > >> Thanks,
> > > > >> Prabha.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton
> > <chrise@aiven.io.invalid
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Vedarth and Krish,
> > > > >> >
> > > > >> > Thanks for the KIP! I have to admit I'm a little skeptical;
> > > hopefully
> > > > >> you
> > > > >> > can help me understand the need for these additional images.
> > > > >> >
> > > > >> > 1) In the motivation section it's stated that "Several other
> > Apache
> > > > >> > projects, like Flink, Spark, Solr, have already released Docker
> > > > Official
> > > > >> > Images, with download figures ranging from 50 million to over 1
> > > > billion.
> > > > >> > These numbers highlight the significant demand among users." But
> > > then
> > > > >> > immediately afterwards, we learn that "Also the Docker Official
> > > Images
> > > > >> are
> > > > >> > always the top 1 search result, irrespective of the number of
> > > > >> downloads."
> > > > >> > Wouldn't a high number of downloads for an image naturally
> follow
> > > from
> > > > >> > being the top search result? It seems like we can't necessarily
> > > assume
> > > > >> that
> > > > >> > Docker Official Images are inherently more desirable for users
> > based
> > > > >> solely
> > > > >> > on download statistics.
> > > > >> >
> > > > >> > 2) Can you elaborate on the value that these new images would
> add
> > > > from a
> > > > >> > user's perspective? I'm hesitant to introduce another image,
> since
> > > it
> > > > >> adds
> > > > >> > to the cognitive burden of people who will inevitably have to
> > answer
> > > > the
> > > > >> > question of "What are the differences between all of the
> available
> > > > >> images
> > > > >> > and which one is best for my use case?"
> > > > >> >
> > > > >> > 3) Would a separate Docker-owned repository be out of the
> > question?
> > > > I'm
> > > > >> > guessing there are some trademark issues that might get in the
> > way,
> > > > but
> > > > >> > it's worth exploring since the entire purpose of this KIP seems
> to
> > > be
> > > > to
> > > > >> > provide images that are vetted and designed by Docker more than
> by
> > > the
> > > > >> > Apache Kafka contributors/committers/PMC.
> > > > >> >
> > > > >> > I may have more questions later but wanted to get this initial
> > round
> > > > out
> > > > >> > now without trying to list everything first.
> > > > >> >
> > > > >> > Looking forward to your thoughts!
> > > > >> >
> > > > >> > Cheers,
> > > > >> >
> > > > >> > Chris
> > > > >> >
> > > > >> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <
> > > > >> vedarth.sharma@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hey folks,
> > > > >> > >
> > > > >> > > Thanks a lot for reviewing the KIP and providing feedback.
> > > > >> > > The discussion thread seems resolved and KIP has been updated
> > > > >> > accordingly.
> > > > >> > > We will be starting the voting thread for this KIP in the next
> > few
> > > > >> days.
> > > > >> > > Please take a look at the KIP and let us know if any further
> > > > >> discussion
> > > > >> > > is needed.
> > > > >> > >
> > > > >> > > Thanks and regards,
> > > > >> > > Vedarth
> > > > >> > >
> > > > >> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <
> > > > manikumar.reddy@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Thanks Krish. KIP looks good to me.
> > > > >> > > >
> > > > >> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <
> > > krishvora01@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > Hi Manikumar,
> > > > >> > > > >
> > > > >> > > > > Thanks for the comments.
> > > > >> > > > >
> > > > >> > > > > Maybe as part of the release process, RM can create a JIRA
> > for
> > > > >> this
> > > > >> > > > > > task. This can be taken by RM or any comitter or any
> > > > contributor
> > > > >> > > (with
> > > > >> > > > > > some help from commiters to run "Docker Image
> Preparation
> > > via
> > > > >> > GitHub
> > > > >> > > > > > Actions:"
> > > > >> > > > >
> > > > >> > > > > This sounds like a good idea. This step would be
> beneficial.
> > > By
> > > > >> > > creating
> > > > >> > > > a
> > > > >> > > > > JIRA ticket, it will also serve as a reminder to complete
> > the
> > > > >> > > > post-release
> > > > >> > > > > steps for the Docker official images. Have updated the KIP
> > > with
> > > > >> this
> > > > >> > > > step.
> > > > >> > > > >
> > > > >> > > > > Is this using GitHub Actions workflow? or manual testing?
> > > > >> > > > >
> > > > >> > > > > This will be done by a Github Actions workflow, which will
> > > test
> > > > >> the
> > > > >> > > > static
> > > > >> > > > > Docker Official Image assets for a specific release
> version.
> > > > >> > > > >
> > > > >> > > > > Is it mandatory for RM/comitters to raise the PR to Docker
> > > Hub’s
> > > > >> > > > > > official images repository (or) can it be done by any
> > > > >> contributor.
> > > > >> > > > >
> > > > >> > > > > I believe that it can be done by any contributor (ref:
> This
> > > link
> > > > >> > > > > <
> > > > >> >
> > > https://docs.docker.com/trusted-content/official-images/contributing/
> > > > >> > > >
> > > > >> > > > > quotes "*Anyone can provide feedback, contribute code,
> > suggest
> > > > >> > process
> > > > >> > > > > changes, or even propose a new Official Image.*")
> > > > >> > > > >
> > > > >> > > > > Also I was thinking, once the KIP gets voted, we should
> try
> > to
> > > > >> > release
> > > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will
> > help
> > > > us
> > > > >> to
> > > > >> > > > > > validate the process and allow us to fix any changes
> > > suggested
> > > > >> by
> > > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > > >> > > > >
> > > > >> > > > > This sounds like a great idea. This KIP proposes release
> of
> > > DOI
> > > > >> as a
> > > > >> > > > > post-release process, which can be done anytime post
> > release.
> > > > >> Since
> > > > >> > > 3.7.0
> > > > >> > > > > is already released, we can perform these steps for that
> > > release
> > > > >> too.
> > > > >> > > By
> > > > >> > > > > the time the KIP gets implemented, if 3.7.1 is released,
> we
> > > > could
> > > > >> do
> > > > >> > > > these
> > > > >> > > > > steps for 3.7.1, instead of 3.7.0. This would allow us to
> > make
> > > > >> > changes
> > > > >> > > to
> > > > >> > > > > the Dockerfiles and other assets based on feedback from
> > Docker
> > > > Hub
> > > > >> > > before
> > > > >> > > > > the release of version 3.8.0.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Krish.
> > > > >> > > > >
> > > > >> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
> > > > >> > manikumar.reddy@gmail.com>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi Krish,
> > > > >> > > > > >
> > > > >> > > > > > Thanks for the updated KIP. a few comments below.
> > > > >> > > > > >
> > > > >> > > > > > > "These actions can be carried out by the RM or any
> > > > contributor
> > > > >> > post
> > > > >> > > > the
> > > > >> > > > > > release process."
> > > > >> > > > > > Maybe as part of the release process, RM can create a
> JIRA
> > > for
> > > > >> this
> > > > >> > > > > > task. This can be taken by RM or any comitter or any
> > > > contributor
> > > > >> > > (with
> > > > >> > > > > > some help from commiters to run "Docker Image
> Preparation
> > > via
> > > > >> > GitHub
> > > > >> > > > > > Actions:"
> > > > >> > > > > >
> > > > >> > > > > > > "Perform Docker build tests to ensure image integrity"
> > > > >> > > > > > Is this using GitHub Actions workflow? or manual
> testing?
> > > > >> > > > > >
> > > > >> > > > > > > "The RM will manually raise the final PR to Docker
> Hub’s
> > > > >> official
> > > > >> > > > images
> > > > >> > > > > > repository using the contents of the generated file"
> > > > >> > > > > >  Is it mandatory for RM/comitters to raise the PR to
> > Docker
> > > > >> Hub’s
> > > > >> > > > > > official images repository (or) can it be done by any
> > > > >> contributor.
> > > > >> > > > > >
> > > > >> > > > > > Also I was thinking, once the KIP gets voted, we should
> > try
> > > to
> > > > >> > > release
> > > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will
> > help
> > > > us
> > > > >> to
> > > > >> > > > > > validate the process and allow us to fix any changes
> > > suggested
> > > > >> by
> > > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > >
> > > > >> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <
> > > > >> krishvora01@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > Hi Manikumar and Luke.
> > > > >> > > > > > > Thanks for the questions.
> > > > >> > > > > > >
> > > > >> > > > > > > 1. No, the Docker inventory files and configurations
> > will
> > > > not
> > > > >> be
> > > > >> > > the
> > > > >> > > > same
> > > > >> > > > > > > for Open Source Software (OSS) Images and Docker
> > Official
> > > > >> Images
> > > > >> > > > (DOI).
> > > > >> > > > > > >
> > > > >> > > > > > > For OSS images, the Dockerfile located in
> > > > >> docker/jvm/dockerfile
> > > > >> > is
> > > > >> > > > > > > utilized. This process is integrated with the existing
> > > > release
> > > > >> > > > pipeline
> > > > >> > > > > > as
> > > > >> > > > > > > outlined in KIP-975
> > > > >> > > > > > > <
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > > > >> > > > > > >,
> > > > >> > > > > > > where the Kafka URL is provided as a build argument.
> > This
> > > > >> method
> > > > >> > > > allows
> > > > >> > > > > > for
> > > > >> > > > > > > building, testing, and releasing OSS images
> dynamically.
> > > The
> > > > >> OSS
> > > > >> > > > images
> > > > >> > > > > > > will continue to be released under the standard
> release
> > > > >> process .
> > > > >> > > > > > >
> > > > >> > > > > > > In contrast, the release process for DOIs requires
> > > providing
> > > > >> the
> > > > >> > > > Docker
> > > > >> > > > > > Hub
> > > > >> > > > > > > team with a specific directory for each version
> release
> > > that
> > > > >> > > > contains a
> > > > >> > > > > > > standalone Dockerfile. These Dockerfiles are designed
> to
> > > be
> > > > >> > > > > > > self-sufficient, hence require hardcoded values
> instead
> > of
> > > > >> > relying
> > > > >> > > on
> > > > >> > > > > > build
> > > > >> > > > > > > arguments. To accommodate this, in our proposed
> > approach,
> > > a
> > > > >> new
> > > > >> > > > directory
> > > > >> > > > > > > named docker_official_images has been created. This
> > > > directory
> > > > >> > > > contains
> > > > >> > > > > > > version-specific directories, having Dockerfiles with
> > > > >> hardcoded
> > > > >> > > > > > > configurations for each release, acting as the source
> of
> > > > truth
> > > > >> > for
> > > > >> > > > DOI
> > > > >> > > > > > > releases. The hardcoded dockerfiles will be created
> > using
> > > > the
> > > > >> > > > > > > docker/jvm/dockerfile as a template. Thus, as part of
> > post
> > > > >> > release
> > > > >> > > we
> > > > >> > > > > > will
> > > > >> > > > > > > be creating a Dockerfile that will be reviewed by the
> > > > >> Dockerhub
> > > > >> > > > community
> > > > >> > > > > > > and might need changes as per their review. This
> > approach
> > > > >> ensures
> > > > >> > > > that
> > > > >> > > > > > DOIs
> > > > >> > > > > > > are built consistently and meet the specific
> > requirements
> > > > set
> > > > >> by
> > > > >> > > > Docker
> > > > >> > > > > > Hub.
> > > > >> > > > > > >
> > > > >> > > > > > > 2. Yes Manikumar, transitioning the release of Docker
> > > > Official
> > > > >> > > Images
> > > > >> > > > > > (DOI)
> > > > >> > > > > > > to a post-release activity does address the concerns
> > about
> > > > >> > > > complicating
> > > > >> > > > > > the
> > > > >> > > > > > > release process. Initially, we considered
> incorporating
> > > DOI
> > > > >> > release
> > > > >> > > > > > > directly into Kafka's release workflow. However, this
> > > > approach
> > > > >> > > > > > > significantly increased the RMs workload due to the
> > > addition
> > > > >> of
> > > > >> > > > numerous
> > > > >> > > > > > > steps, complicating the process. By designating the
> DOI
> > > > >> release
> > > > >> > as
> > > > >> > > a
> > > > >> > > > > > > post-release task, we maintain the original release
> > > process.
> > > > >> This
> > > > >> > > > > > > adjustment allows for the DOI release to be done after
> > the
> > > > >> main
> > > > >> > > > release.
> > > > >> > > > > > We
> > > > >> > > > > > > have revised the KIP to reflect that DOI releases will
> > now
> > > > >> occur
> > > > >> > > > after
> > > > >> > > > > > the
> > > > >> > > > > > > main release phase. Please review the updated document
> > and
> > > > >> > provide
> > > > >> > > > any
> > > > >> > > > > > > feedback you might have.
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks,
> > > > >> > > > > > > Krish.
> > > > >> > > > > > >
> > > > >> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <
> > > showuon@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi Krishna,
> > > > >> > > > > > > >
> > > > >> > > > > > > > I also have the same question as Manikumar raised:
> > > > >> > > > > > > > 1. Will the Docker inventory files/etc are the same
> > for
> > > > OSS
> > > > >> > Image
> > > > >> > > > and
> > > > >> > > > > > > > Docker Official Images?
> > > > >> > > > > > > > If no, then why not? Could we make them identical so
> > > that
> > > > we
> > > > >> > > don't
> > > > >> > > > > > have to
> > > > >> > > > > > > > build 2 images for each release?
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thank you.
> > > > >> > > > > > > > Luke
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> > > > >> > > > manikumar.reddy@gmail.com>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi Krishna,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks for the KIP.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I think Docker Official Images will be beneficial
> to
> > > the
> > > > >> > Kafka
> > > > >> > > > > > community.
> > > > >> > > > > > > > > Few queries below.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > 1. Will the Docker inventory files/etc are the
> same
> > > for
> > > > >> OSS
> > > > >> > > > Image and
> > > > >> > > > > > > > > Docker Official Images
> > > > >> > > > > > > > > 2. I am a bit worried about the new steps to the
> > > release
> > > > >> > > process.
> > > > >> > > > > > Maybe
> > > > >> > > > > > > > we
> > > > >> > > > > > > > > should consider Docker Official Images release as
> > > > >> > Post-Release
> > > > >> > > > > > activity.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> > > > >> > > > krishvora01@gmail.com>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Hi Hector,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks for reaching out. This KIP builds on top
> of
> > > > >> KIP-975
> > > > >> > > > > > > > > > <
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > and
> > > > >> > > > > > > > > > aims to introduce a JVM-based Docker Official
> > Image
> > > > (DOI
> > > > >> > > > > > > > > > <
> > > > >> https://docs.docker.com/trusted-content/official-images/
> > > > >> > >)
> > > > >> > > > for
> > > > >> > > > > > Apache
> > > > >> > > > > > > > > > Kafka that will be visible under Docker Official
> > > > Images
> > > > >> > > > > > > > > > <
> > > > https://hub.docker.com/search?image_filter=official&q=
> > > > >> >.
> > > > >> > > Once
> > > > >> > > > > > > > > implemented
> > > > >> > > > > > > > > > for Apache Kafka, for each release, there will
> be
> > > one
> > > > >> more
> > > > >> > > > > > JVM-based
> > > > >> > > > > > > > > Docker
> > > > >> > > > > > > > > > image available to users.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Currently, we already have an OSS sponsored
> image,
> > > > which
> > > > >> > was
> > > > >> > > > > > introduced
> > > > >> > > > > > > > > via
> > > > >> > > > > > > > > > KIP-975 (apache/kafka <
> > > > >> > > > https://hub.docker.com/r/apache/kafka/tags
> > > > >> > > > > > >)
> > > > >> > > > > > > > > which
> > > > >> > > > > > > > > > comes under The Apache Software Foundation <
> > > > >> > > > > > > > > > https://hub.docker.com/u/apache> in
> > > > >> > > > > > > > > > Docker Hub. The new Docker Image is the Docker
> > > > Official
> > > > >> > Image
> > > > >> > > > > > (DOI),
> > > > >> > > > > > > > > which
> > > > >> > > > > > > > > > will be built and maintained by Docker
> Community.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > For example, for a release version like 3.8.0 we
> > > will
> > > > >> have
> > > > >> > > two
> > > > >> > > > JVM
> > > > >> > > > > > > > based
> > > > >> > > > > > > > > > docker images:-
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > > >> > > > > > > > > >    - kafka:3.8.0 (Docker Official image)
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > I have added the same in the KIP too for
> > everyone's
> > > > >> > > reference.
> > > > >> > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > Krish.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino
> > > > >> > (BLOOMBERG/
> > > > >> > > > 919
> > > > >> > > > > > 3RD
> > > > >> > > > > > > > A) <
> > > > >> > > > > > > > > > hgeraldino@bloomberg.net> wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > Hi,
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > What is the difference between this KIP and
> > > KIP-975:
> > > > >> > Docker
> > > > >> > > > > > Image for
> > > > >> > > > > > > > > > > Apache Kafka?
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24
> > 07:30:07
> > > > >> > > UTC-4:00To:
> > > > >> > > > > > > > > > > dev@kafka.apache.org
> > > > >> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official
> > Image
> > > > for
> > > > >> > > Apache
> > > > >> > > > > > Kafka
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Hi everyone,
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > I would like to start the discussion on
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > >> > > > > > > > > > > age+for+Apache+Kafka
> > > > >> > > > > > > > > > >  .
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > This KIP aims to introduce JVM based Docker
> > > Official
> > > > >> > Image
> > > > >> > > > (DOI)
> > > > >> > > > > > for
> > > > >> > > > > > > > > > Apache
> > > > >> > > > > > > > > > > Kafka.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Regards,
> > > > >> > > > > > > > > > > Krish.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> [image: Confluent] <https://www.confluent.io>
> > > > >> Prabha Manepalli
> > > > >> Product Manager for Confluent Platform Security, Docker
> > > > >> linkedin.com/in/prabhamanepalli
> > > > >>
> > > > >
> > > >
> > >
> >
>
>
> --
>
> [image: Confluent] <https://www.confluent.io>
> Prabha Manepalli
> Product Manager for Confluent Platform Security, Docker Containers
> linkedin.com/in/prabhamanepalli
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Prabha Manepalli <mp...@confluent.io.INVALID>]

Hi Chris,

Sharing the requested links explaining why Docker Official images are
considered more secure -
https://www.docker.com/blog/enhancing-security-and-transparency-with-docker-official-images/
and
https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-cves

I hope these links help you understand why we need Docker Official images
for organisations with stringent security compliance requirements for their
Kafka workloads.

Thank you.



On Sun, May 12, 2024 at 3:33 PM Vedarth Sharma <ve...@gmail.com>
wrote:

> Hey Chris!
>
> Functionality wise, we don't intend to have any differences between OSS
> Docker Image and Docker Official Image.
> The Docker Official Image will be the recommended one.
> Since the Docker Official Image might be delayed due to review done by
> Docker, images on apache/kafka (OSS Docker Image) can be used by users.
>
> 1) I read https://docs.docker.com/trusted-content/official-images/ and
> > didn't find anything on that page or immediately around it that explains
> > what compliance requirements might be satisfied by a DOI that couldn't be
> > satisfied by the existing apache/kafka image. Can you elaborate? Feel
> free
> > to provide another link, but please also quote the relevant sections from
> > it (as StackOverflow likes to say, links can grow stale over time).
>
>
>    - If a user's selection is confined solely to Docker Official Images,
>    this Docker Image will ensure their access to Kafka.
>    - Details on specific advantages of Docker Official Images can be found
>    at
>
> https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
>    .
>    - The Docker Official Image will be actively rebuilt for updates and
>    security fixes.
>    - It's true we can provide exactly the same benefits in the OSS Docker
>    Image as well. But it won't have the Docker Official Image badge and it
>    will add additional overhead for Apache Kafka community.
>    - The fact that it will have Docker Official Image badge will set it
>    apart from the OSS Docker Image.
>    - Also the ability to do just docker pull kafka to get the kafka docker
>    image will only be possible with Docker Official Image.
>
>
> 2) It would be great to see a brief summary of the differences in these
> > images included in the KIP, in order to try to gauge how this would look
> to
> > our users.
>
>
>    - Functionally, both Docker images will remain identical.
>    - The variance lies primarily in the methodologies of building and
>    validation, as outlined in the updated KIP.
>
>
> 3) What I suggested last time was not a separate apache/apache-docker
> > repository, but a repository controlled entirely by Docker. The DOI docs
> > [1] state that "While it's preferable to have upstream software authors
> > maintaining their Docker Official Images, this isn't a strict
> requirement",
> > which I take to mean that it's not required for an Apache Kafka DOI to
> live
> > under the apache organization on GitHub. It also seems like there's
> > precedent for this: images for MySQL [2] and PHP [3] already exist under
> > the control of Docker. The reason I think this is worth considering is
> that
> > Docker can arbitrarily change the eligibility requirements for their
> > official images at any time, and it doesn't seem like there's a clear
> > process in the KIP for judging how we should respond to these changes (in
> > fact, it seems like the idea in the KIP is that we should make any change
> > required with no further vetting beyond possibly a pull request on
> > apache/kafka, which would require approval by a committer). By hosting
> the
> > DOI definitions ourselves (either in apache/kafka, or in a theoretical
> > apache/docker-kafka repository), we take responsibility for the image,
> even
> > if the owner on Docker Hub is Docker, not Apache. If the code lives
> > elsewhere, then (as long as basic trademark and possibly security
> > guidelines are respected) Apache doesn't have to concern itself at all
> with
> > the image and the maintainers are free to make whatever changes they want
> > to it in order to meet Docker's requirements.
>
>
>    - By incorporating the source code of the Docker Official Image into our
>    AK ecosystem, we gain control over its functionality, ensuring alignment
>    with the OSS Docker image. This ensures a seamless experience for users
> who
>    may need to transition between these images.
>    - Maintaining both images within the same community facilitates ease of
>    management and fosters a singular source of truth.
>    - While Apache may not retain ownership of the hosted Docker Official
>    Image, we are, in essence, providing Docker with a foundation that
> aligns
>    with their established guidelines as well as remains consistent with OSS
>    Docker Image apache/kafka.
>    - Any future alterations to the functionality can be seamlessly
>    propagated across both the OSS and Official Docker Images.
>
> I think these reasons must be why a lot of the Apache projects choose to
> host the docker definitions themselves. The responsibility of owning the
> definitions comes with benefits as well that we should also consider.
>
> Let us know if you have any further questions!
>
> Thanks and regards,
> Vedarth
>
> On Fri, May 10, 2024 at 7:56 PM Chris Egerton <ch...@aiven.io.invalid>
> wrote:
>
> > Hi Krish and Prabha,
> >
> > Thanks for your replies. I still have some follow-up questions:
> >
> > 1) I read https://docs.docker.com/trusted-content/official-images/ and
> > didn't find anything on that page or immediately around it that explains
> > what compliance requirements might be satisfied by a DOI that couldn't be
> > satisfied by the existing apache/kafka image. Can you elaborate? Feel
> free
> > to provide another link, but please also quote the relevant sections from
> > it (as StackOverflow likes to say, links can grow stale over time).
> >
> > 2) It would be great to see a brief summary of the differences in these
> > images included in the KIP, in order to try to gauge how this would look
> to
> > our users.
> >
> > 3) What I suggested last time was not a separate apache/apache-docker
> > repository, but a repository controlled entirely by Docker. The DOI docs
> > [1] state that "While it's preferable to have upstream software authors
> > maintaining their Docker Official Images, this isn't a strict
> requirement",
> > which I take to mean that it's not required for an Apache Kafka DOI to
> live
> > under the apache organization on GitHub. It also seems like there's
> > precedent for this: images for MySQL [2] and PHP [3] already exist under
> > the control of Docker. The reason I think this is worth considering is
> that
> > Docker can arbitrarily change the eligibility requirements for their
> > official images at any time, and it doesn't seem like there's a clear
> > process in the KIP for judging how we should respond to these changes (in
> > fact, it seems like the idea in the KIP is that we should make any change
> > required with no further vetting beyond possibly a pull request on
> > apache/kafka, which would require approval by a committer). By hosting
> the
> > DOI definitions ourselves (either in apache/kafka, or in a theoretical
> > apache/docker-kafka repository), we take responsibility for the image,
> even
> > if the owner on Docker Hub is Docker, not Apache. If the code lives
> > elsewhere, then (as long as basic trademark and possibly security
> > guidelines are respected) Apache doesn't have to concern itself at all
> with
> > the image and the maintainers are free to make whatever changes they want
> > to it in order to meet Docker's requirements.
> >
> > [1] -
> > https://docs.docker.com/trusted-content/official-images/contributing/,
> > second paragraph under "Contributing to Docker Official Images"
> > [2] - https://github.com/docker-library/mysql
> > [3] - https://github.com/docker-library/php
> >
> > Cheers,
> >
> > Chris
> >
> > On Fri, May 10, 2024 at 7:46 AM Krish Vora <kr...@gmail.com>
> wrote:
> >
> > > Hey Chris,
> > >
> > > We have responded to the initial round of queries. And as you mentioned
> > in
> > > your comment, you may have more questions related to this KIP. Please
> let
> > > us know if you have any.
> > > We want to start the voting for this KIP soon, as we intend to include
> it
> > > in the 3.8.0 release.
> > >
> > > Thanks.
> > >
> > > Regards,
> > >
> > > Krish.
> > >
> > > On Wed, May 8, 2024 at 6:19 PM Krish Vora <kr...@gmail.com>
> wrote:
> > >
> > > > Hi Chris. Thanks for the questions.
> > > >
> > > > 3. Would a separate Docker-owned repository be out of the question?
> I'm
> > > >> guessing there are some trademark issues that might get in the way,
> > but
> > > >> it's worth exploring since the entire purpose of this KIP seems to
> be
> > to
> > > >> provide images that are vetted and designed by Docker more than by
> the
> > > >> Apache Kafka contributors/committers/PMC.
> > > >
> > > >
> > > >
> > > >    - The process for introducing a Docker Official Image involves
> > > >       - Hosting the Dockerfile in the Apache Kafka repository and
> > > >       - Providing the path to this Dockerfile to Docker Hub in Docker
> > > >       Hub’s own repo
> > > >       <
> > > https://github.com/docker-library/official-images/tree/master/library>
> > > >       .
> > > >    - This ensures that any updates to the Dockerfile in the AK
> > repository
> > > >    are directly applicable to the docker official images available on
> > > Docker
> > > >    Hub.
> > > >
> > > >
> > > >    - We also did not find any added advantage to create a separate
> > > >    repository named apache-docker within the Apache GitHub
> > organization.
> > > >
> > > > Thanks,
> > > > Krish.
> > > >
> > > > On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli
> > > > <mp...@confluent.io.invalid> wrote:
> > > >
> > > >> Hi Chris,  I would like to add more context to this KIP's
> motivation.
> > > >> Vedarth and Krish, please weigh in with your inputs.
> > > >>
> > > >> In the motivation section it's stated that "Several other Apache
> > > projects,
> > > >> > like Flink, Spark, Solr, have already released Docker Official
> > Images,
> > > >> with
> > > >> > download figures ranging from 50 million to over 1 billion. These
> > > >> numbers
> > > >> > highlight the significant demand among users." But then
> immediately
> > > >> > afterwards, we learn that "Also the Docker Official Images are
> > always
> > > >> the
> > > >> > top 1 search result, irrespective of the number of downloads."
> > > Wouldn't
> > > >> a
> > > >> > high number of downloads for an image naturally follow from being
> > the
> > > >> top
> > > >> > search result? It seems like we can't necessarily assume that
> Docker
> > > >> > Official Images are inherently more desirable for users based
> solely
> > > on
> > > >> > download statistics.
> > > >> >
> > > >>
> > > >> *My thoughts: *Unlike the Sponsored OSS image, the Docker Official
> > image
> > > >> is
> > > >> more desirable for workloads that have stringent compliance
> > > requirements.
> > > >> More details on why official images are more trusted are documented
> > here
> > > >> <https://docs.docker.com/trusted-content/official-images/>. The
> > Docker
> > > >> Official image would also help an absolutely new Kafka beginner who
> > > might
> > > >> not know about Apache or the concept of Sponsored images. We want to
> > > make
> > > >> it easier for Kafka beginners to discover the Kafka image through
> > > >> DockerHub.
> > > >>
> > > >>
> > > >> Can you elaborate on the value that these new images would add from
> a
> > > >> > user's perspective? I'm hesitant to introduce another image, since
> > it
> > > >> adds
> > > >> > to the cognitive burden of people who will inevitably have to
> answer
> > > the
> > > >> > question of "What are the differences between all of the available
> > > >> images
> > > >> > and which one is best for my use case?"
> > > >> >
> > > >>
> > > >>
> > > >> *My thoughts: *This is a valid concern to address. The response to
> the
> > > >> above question addresses the value-add this new Docker Official
> image
> > > >> would
> > > >> provide. I also agree we need a clear distinction between each of
> > these
> > > >> images to be well documented. We plan to update the AK website with
> > > >> details
> > > >> on how, why, and when a developer would want to use each of these
> > > >> particular images(KIP-974,975,1028).
> > > >>
> > > >> Thanks,
> > > >> Prabha.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton
> <chrise@aiven.io.invalid
> > >
> > > >> wrote:
> > > >>
> > > >> > Hi Vedarth and Krish,
> > > >> >
> > > >> > Thanks for the KIP! I have to admit I'm a little skeptical;
> > hopefully
> > > >> you
> > > >> > can help me understand the need for these additional images.
> > > >> >
> > > >> > 1) In the motivation section it's stated that "Several other
> Apache
> > > >> > projects, like Flink, Spark, Solr, have already released Docker
> > > Official
> > > >> > Images, with download figures ranging from 50 million to over 1
> > > billion.
> > > >> > These numbers highlight the significant demand among users." But
> > then
> > > >> > immediately afterwards, we learn that "Also the Docker Official
> > Images
> > > >> are
> > > >> > always the top 1 search result, irrespective of the number of
> > > >> downloads."
> > > >> > Wouldn't a high number of downloads for an image naturally follow
> > from
> > > >> > being the top search result? It seems like we can't necessarily
> > assume
> > > >> that
> > > >> > Docker Official Images are inherently more desirable for users
> based
> > > >> solely
> > > >> > on download statistics.
> > > >> >
> > > >> > 2) Can you elaborate on the value that these new images would add
> > > from a
> > > >> > user's perspective? I'm hesitant to introduce another image, since
> > it
> > > >> adds
> > > >> > to the cognitive burden of people who will inevitably have to
> answer
> > > the
> > > >> > question of "What are the differences between all of the available
> > > >> images
> > > >> > and which one is best for my use case?"
> > > >> >
> > > >> > 3) Would a separate Docker-owned repository be out of the
> question?
> > > I'm
> > > >> > guessing there are some trademark issues that might get in the
> way,
> > > but
> > > >> > it's worth exploring since the entire purpose of this KIP seems to
> > be
> > > to
> > > >> > provide images that are vetted and designed by Docker more than by
> > the
> > > >> > Apache Kafka contributors/committers/PMC.
> > > >> >
> > > >> > I may have more questions later but wanted to get this initial
> round
> > > out
> > > >> > now without trying to list everything first.
> > > >> >
> > > >> > Looking forward to your thoughts!
> > > >> >
> > > >> > Cheers,
> > > >> >
> > > >> > Chris
> > > >> >
> > > >> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <
> > > >> vedarth.sharma@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hey folks,
> > > >> > >
> > > >> > > Thanks a lot for reviewing the KIP and providing feedback.
> > > >> > > The discussion thread seems resolved and KIP has been updated
> > > >> > accordingly.
> > > >> > > We will be starting the voting thread for this KIP in the next
> few
> > > >> days.
> > > >> > > Please take a look at the KIP and let us know if any further
> > > >> discussion
> > > >> > > is needed.
> > > >> > >
> > > >> > > Thanks and regards,
> > > >> > > Vedarth
> > > >> > >
> > > >> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <
> > > manikumar.reddy@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Thanks Krish. KIP looks good to me.
> > > >> > > >
> > > >> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <
> > krishvora01@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > Hi Manikumar,
> > > >> > > > >
> > > >> > > > > Thanks for the comments.
> > > >> > > > >
> > > >> > > > > Maybe as part of the release process, RM can create a JIRA
> for
> > > >> this
> > > >> > > > > > task. This can be taken by RM or any comitter or any
> > > contributor
> > > >> > > (with
> > > >> > > > > > some help from commiters to run "Docker Image Preparation
> > via
> > > >> > GitHub
> > > >> > > > > > Actions:"
> > > >> > > > >
> > > >> > > > > This sounds like a good idea. This step would be beneficial.
> > By
> > > >> > > creating
> > > >> > > > a
> > > >> > > > > JIRA ticket, it will also serve as a reminder to complete
> the
> > > >> > > > post-release
> > > >> > > > > steps for the Docker official images. Have updated the KIP
> > with
> > > >> this
> > > >> > > > step.
> > > >> > > > >
> > > >> > > > > Is this using GitHub Actions workflow? or manual testing?
> > > >> > > > >
> > > >> > > > > This will be done by a Github Actions workflow, which will
> > test
> > > >> the
> > > >> > > > static
> > > >> > > > > Docker Official Image assets for a specific release version.
> > > >> > > > >
> > > >> > > > > Is it mandatory for RM/comitters to raise the PR to Docker
> > Hub’s
> > > >> > > > > > official images repository (or) can it be done by any
> > > >> contributor.
> > > >> > > > >
> > > >> > > > > I believe that it can be done by any contributor (ref: This
> > link
> > > >> > > > > <
> > > >> >
> > https://docs.docker.com/trusted-content/official-images/contributing/
> > > >> > > >
> > > >> > > > > quotes "*Anyone can provide feedback, contribute code,
> suggest
> > > >> > process
> > > >> > > > > changes, or even propose a new Official Image.*")
> > > >> > > > >
> > > >> > > > > Also I was thinking, once the KIP gets voted, we should try
> to
> > > >> > release
> > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will
> help
> > > us
> > > >> to
> > > >> > > > > > validate the process and allow us to fix any changes
> > suggested
> > > >> by
> > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > >> > > > >
> > > >> > > > > This sounds like a great idea. This KIP proposes release of
> > DOI
> > > >> as a
> > > >> > > > > post-release process, which can be done anytime post
> release.
> > > >> Since
> > > >> > > 3.7.0
> > > >> > > > > is already released, we can perform these steps for that
> > release
> > > >> too.
> > > >> > > By
> > > >> > > > > the time the KIP gets implemented, if 3.7.1 is released, we
> > > could
> > > >> do
> > > >> > > > these
> > > >> > > > > steps for 3.7.1, instead of 3.7.0. This would allow us to
> make
> > > >> > changes
> > > >> > > to
> > > >> > > > > the Dockerfiles and other assets based on feedback from
> Docker
> > > Hub
> > > >> > > before
> > > >> > > > > the release of version 3.8.0.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Krish.
> > > >> > > > >
> > > >> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
> > > >> > manikumar.reddy@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi Krish,
> > > >> > > > > >
> > > >> > > > > > Thanks for the updated KIP. a few comments below.
> > > >> > > > > >
> > > >> > > > > > > "These actions can be carried out by the RM or any
> > > contributor
> > > >> > post
> > > >> > > > the
> > > >> > > > > > release process."
> > > >> > > > > > Maybe as part of the release process, RM can create a JIRA
> > for
> > > >> this
> > > >> > > > > > task. This can be taken by RM or any comitter or any
> > > contributor
> > > >> > > (with
> > > >> > > > > > some help from commiters to run "Docker Image Preparation
> > via
> > > >> > GitHub
> > > >> > > > > > Actions:"
> > > >> > > > > >
> > > >> > > > > > > "Perform Docker build tests to ensure image integrity"
> > > >> > > > > > Is this using GitHub Actions workflow? or manual testing?
> > > >> > > > > >
> > > >> > > > > > > "The RM will manually raise the final PR to Docker Hub’s
> > > >> official
> > > >> > > > images
> > > >> > > > > > repository using the contents of the generated file"
> > > >> > > > > >  Is it mandatory for RM/comitters to raise the PR to
> Docker
> > > >> Hub’s
> > > >> > > > > > official images repository (or) can it be done by any
> > > >> contributor.
> > > >> > > > > >
> > > >> > > > > > Also I was thinking, once the KIP gets voted, we should
> try
> > to
> > > >> > > release
> > > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will
> help
> > > us
> > > >> to
> > > >> > > > > > validate the process and allow us to fix any changes
> > suggested
> > > >> by
> > > >> > > > > > Dockerhub before the 3.8.0 release.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > >
> > > >> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <
> > > >> krishvora01@gmail.com>
> > > >> > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > Hi Manikumar and Luke.
> > > >> > > > > > > Thanks for the questions.
> > > >> > > > > > >
> > > >> > > > > > > 1. No, the Docker inventory files and configurations
> will
> > > not
> > > >> be
> > > >> > > the
> > > >> > > > same
> > > >> > > > > > > for Open Source Software (OSS) Images and Docker
> Official
> > > >> Images
> > > >> > > > (DOI).
> > > >> > > > > > >
> > > >> > > > > > > For OSS images, the Dockerfile located in
> > > >> docker/jvm/dockerfile
> > > >> > is
> > > >> > > > > > > utilized. This process is integrated with the existing
> > > release
> > > >> > > > pipeline
> > > >> > > > > > as
> > > >> > > > > > > outlined in KIP-975
> > > >> > > > > > > <
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > > >> > > > > > >,
> > > >> > > > > > > where the Kafka URL is provided as a build argument.
> This
> > > >> method
> > > >> > > > allows
> > > >> > > > > > for
> > > >> > > > > > > building, testing, and releasing OSS images dynamically.
> > The
> > > >> OSS
> > > >> > > > images
> > > >> > > > > > > will continue to be released under the standard release
> > > >> process .
> > > >> > > > > > >
> > > >> > > > > > > In contrast, the release process for DOIs requires
> > providing
> > > >> the
> > > >> > > > Docker
> > > >> > > > > > Hub
> > > >> > > > > > > team with a specific directory for each version release
> > that
> > > >> > > > contains a
> > > >> > > > > > > standalone Dockerfile. These Dockerfiles are designed to
> > be
> > > >> > > > > > > self-sufficient, hence require hardcoded values instead
> of
> > > >> > relying
> > > >> > > on
> > > >> > > > > > build
> > > >> > > > > > > arguments. To accommodate this, in our proposed
> approach,
> > a
> > > >> new
> > > >> > > > directory
> > > >> > > > > > > named docker_official_images has been created. This
> > > directory
> > > >> > > > contains
> > > >> > > > > > > version-specific directories, having Dockerfiles with
> > > >> hardcoded
> > > >> > > > > > > configurations for each release, acting as the source of
> > > truth
> > > >> > for
> > > >> > > > DOI
> > > >> > > > > > > releases. The hardcoded dockerfiles will be created
> using
> > > the
> > > >> > > > > > > docker/jvm/dockerfile as a template. Thus, as part of
> post
> > > >> > release
> > > >> > > we
> > > >> > > > > > will
> > > >> > > > > > > be creating a Dockerfile that will be reviewed by the
> > > >> Dockerhub
> > > >> > > > community
> > > >> > > > > > > and might need changes as per their review. This
> approach
> > > >> ensures
> > > >> > > > that
> > > >> > > > > > DOIs
> > > >> > > > > > > are built consistently and meet the specific
> requirements
> > > set
> > > >> by
> > > >> > > > Docker
> > > >> > > > > > Hub.
> > > >> > > > > > >
> > > >> > > > > > > 2. Yes Manikumar, transitioning the release of Docker
> > > Official
> > > >> > > Images
> > > >> > > > > > (DOI)
> > > >> > > > > > > to a post-release activity does address the concerns
> about
> > > >> > > > complicating
> > > >> > > > > > the
> > > >> > > > > > > release process. Initially, we considered incorporating
> > DOI
> > > >> > release
> > > >> > > > > > > directly into Kafka's release workflow. However, this
> > > approach
> > > >> > > > > > > significantly increased the RMs workload due to the
> > addition
> > > >> of
> > > >> > > > numerous
> > > >> > > > > > > steps, complicating the process. By designating the DOI
> > > >> release
> > > >> > as
> > > >> > > a
> > > >> > > > > > > post-release task, we maintain the original release
> > process.
> > > >> This
> > > >> > > > > > > adjustment allows for the DOI release to be done after
> the
> > > >> main
> > > >> > > > release.
> > > >> > > > > > We
> > > >> > > > > > > have revised the KIP to reflect that DOI releases will
> now
> > > >> occur
> > > >> > > > after
> > > >> > > > > > the
> > > >> > > > > > > main release phase. Please review the updated document
> and
> > > >> > provide
> > > >> > > > any
> > > >> > > > > > > feedback you might have.
> > > >> > > > > > >
> > > >> > > > > > > Thanks,
> > > >> > > > > > > Krish.
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <
> > showuon@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi Krishna,
> > > >> > > > > > > >
> > > >> > > > > > > > I also have the same question as Manikumar raised:
> > > >> > > > > > > > 1. Will the Docker inventory files/etc are the same
> for
> > > OSS
> > > >> > Image
> > > >> > > > and
> > > >> > > > > > > > Docker Official Images?
> > > >> > > > > > > > If no, then why not? Could we make them identical so
> > that
> > > we
> > > >> > > don't
> > > >> > > > > > have to
> > > >> > > > > > > > build 2 images for each release?
> > > >> > > > > > > >
> > > >> > > > > > > > Thank you.
> > > >> > > > > > > > Luke
> > > >> > > > > > > >
> > > >> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> > > >> > > > manikumar.reddy@gmail.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Krishna,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks for the KIP.
> > > >> > > > > > > > >
> > > >> > > > > > > > > I think Docker Official Images will be beneficial to
> > the
> > > >> > Kafka
> > > >> > > > > > community.
> > > >> > > > > > > > > Few queries below.
> > > >> > > > > > > > >
> > > >> > > > > > > > > 1. Will the Docker inventory files/etc are the same
> > for
> > > >> OSS
> > > >> > > > Image and
> > > >> > > > > > > > > Docker Official Images
> > > >> > > > > > > > > 2. I am a bit worried about the new steps to the
> > release
> > > >> > > process.
> > > >> > > > > > Maybe
> > > >> > > > > > > > we
> > > >> > > > > > > > > should consider Docker Official Images release as
> > > >> > Post-Release
> > > >> > > > > > activity.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> > > >> > > > krishvora01@gmail.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Hi Hector,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for reaching out. This KIP builds on top of
> > > >> KIP-975
> > > >> > > > > > > > > > <
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > and
> > > >> > > > > > > > > > aims to introduce a JVM-based Docker Official
> Image
> > > (DOI
> > > >> > > > > > > > > > <
> > > >> https://docs.docker.com/trusted-content/official-images/
> > > >> > >)
> > > >> > > > for
> > > >> > > > > > Apache
> > > >> > > > > > > > > > Kafka that will be visible under Docker Official
> > > Images
> > > >> > > > > > > > > > <
> > > https://hub.docker.com/search?image_filter=official&q=
> > > >> >.
> > > >> > > Once
> > > >> > > > > > > > > implemented
> > > >> > > > > > > > > > for Apache Kafka, for each release, there will be
> > one
> > > >> more
> > > >> > > > > > JVM-based
> > > >> > > > > > > > > Docker
> > > >> > > > > > > > > > image available to users.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Currently, we already have an OSS sponsored image,
> > > which
> > > >> > was
> > > >> > > > > > introduced
> > > >> > > > > > > > > via
> > > >> > > > > > > > > > KIP-975 (apache/kafka <
> > > >> > > > https://hub.docker.com/r/apache/kafka/tags
> > > >> > > > > > >)
> > > >> > > > > > > > > which
> > > >> > > > > > > > > > comes under The Apache Software Foundation <
> > > >> > > > > > > > > > https://hub.docker.com/u/apache> in
> > > >> > > > > > > > > > Docker Hub. The new Docker Image is the Docker
> > > Official
> > > >> > Image
> > > >> > > > > > (DOI),
> > > >> > > > > > > > > which
> > > >> > > > > > > > > > will be built and maintained by Docker Community.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > For example, for a release version like 3.8.0 we
> > will
> > > >> have
> > > >> > > two
> > > >> > > > JVM
> > > >> > > > > > > > based
> > > >> > > > > > > > > > docker images:-
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > >> > > > > > > > > >    - kafka:3.8.0 (Docker Official image)
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > I have added the same in the KIP too for
> everyone's
> > > >> > > reference.
> > > >> > > > > > > > > > Thanks,
> > > >> > > > > > > > > > Krish.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino
> > > >> > (BLOOMBERG/
> > > >> > > > 919
> > > >> > > > > > 3RD
> > > >> > > > > > > > A) <
> > > >> > > > > > > > > > hgeraldino@bloomberg.net> wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > What is the difference between this KIP and
> > KIP-975:
> > > >> > Docker
> > > >> > > > > > Image for
> > > >> > > > > > > > > > > Apache Kafka?
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24
> 07:30:07
> > > >> > > UTC-4:00To:
> > > >> > > > > > > > > > > dev@kafka.apache.org
> > > >> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official
> Image
> > > for
> > > >> > > Apache
> > > >> > > > > > Kafka
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Hi everyone,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I would like to start the discussion on
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > >> > > > > > > > > > > age+for+Apache+Kafka
> > > >> > > > > > > > > > >  .
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > This KIP aims to introduce JVM based Docker
> > Official
> > > >> > Image
> > > >> > > > (DOI)
> > > >> > > > > > for
> > > >> > > > > > > > > > Apache
> > > >> > > > > > > > > > > Kafka.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Regards,
> > > >> > > > > > > > > > > Krish.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> [image: Confluent] <https://www.confluent.io>
> > > >> Prabha Manepalli
> > > >> Product Manager for Confluent Platform Security, Docker
> > > >> linkedin.com/in/prabhamanepalli
> > > >>
> > > >
> > >
> >
>


-- 

[image: Confluent] <https://www.confluent.io>
Prabha Manepalli
Product Manager for Confluent Platform Security, Docker Containers
linkedin.com/in/prabhamanepalli

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Vedarth Sharma <ve...@gmail.com>]

Hey Chris!

Functionality wise, we don't intend to have any differences between OSS
Docker Image and Docker Official Image.
The Docker Official Image will be the recommended one.
Since the Docker Official Image might be delayed due to review done by
Docker, images on apache/kafka (OSS Docker Image) can be used by users.

1) I read https://docs.docker.com/trusted-content/official-images/ and
> didn't find anything on that page or immediately around it that explains
> what compliance requirements might be satisfied by a DOI that couldn't be
> satisfied by the existing apache/kafka image. Can you elaborate? Feel free
> to provide another link, but please also quote the relevant sections from
> it (as StackOverflow likes to say, links can grow stale over time).


   - If a user's selection is confined solely to Docker Official Images,
   this Docker Image will ensure their access to Kafka.
   - Details on specific advantages of Docker Official Images can be found
   at
   https://github.com/docker-library/official-images?tab=readme-ov-file#what-are-official-images
   .
   - The Docker Official Image will be actively rebuilt for updates and
   security fixes.
   - It's true we can provide exactly the same benefits in the OSS Docker
   Image as well. But it won't have the Docker Official Image badge and it
   will add additional overhead for Apache Kafka community.
   - The fact that it will have Docker Official Image badge will set it
   apart from the OSS Docker Image.
   - Also the ability to do just docker pull kafka to get the kafka docker
   image will only be possible with Docker Official Image.


2) It would be great to see a brief summary of the differences in these
> images included in the KIP, in order to try to gauge how this would look to
> our users.


   - Functionally, both Docker images will remain identical.
   - The variance lies primarily in the methodologies of building and
   validation, as outlined in the updated KIP.


3) What I suggested last time was not a separate apache/apache-docker
> repository, but a repository controlled entirely by Docker. The DOI docs
> [1] state that "While it's preferable to have upstream software authors
> maintaining their Docker Official Images, this isn't a strict requirement",
> which I take to mean that it's not required for an Apache Kafka DOI to live
> under the apache organization on GitHub. It also seems like there's
> precedent for this: images for MySQL [2] and PHP [3] already exist under
> the control of Docker. The reason I think this is worth considering is that
> Docker can arbitrarily change the eligibility requirements for their
> official images at any time, and it doesn't seem like there's a clear
> process in the KIP for judging how we should respond to these changes (in
> fact, it seems like the idea in the KIP is that we should make any change
> required with no further vetting beyond possibly a pull request on
> apache/kafka, which would require approval by a committer). By hosting the
> DOI definitions ourselves (either in apache/kafka, or in a theoretical
> apache/docker-kafka repository), we take responsibility for the image, even
> if the owner on Docker Hub is Docker, not Apache. If the code lives
> elsewhere, then (as long as basic trademark and possibly security
> guidelines are respected) Apache doesn't have to concern itself at all with
> the image and the maintainers are free to make whatever changes they want
> to it in order to meet Docker's requirements.


   - By incorporating the source code of the Docker Official Image into our
   AK ecosystem, we gain control over its functionality, ensuring alignment
   with the OSS Docker image. This ensures a seamless experience for users who
   may need to transition between these images.
   - Maintaining both images within the same community facilitates ease of
   management and fosters a singular source of truth.
   - While Apache may not retain ownership of the hosted Docker Official
   Image, we are, in essence, providing Docker with a foundation that aligns
   with their established guidelines as well as remains consistent with OSS
   Docker Image apache/kafka.
   - Any future alterations to the functionality can be seamlessly
   propagated across both the OSS and Official Docker Images.

I think these reasons must be why a lot of the Apache projects choose to
host the docker definitions themselves. The responsibility of owning the
definitions comes with benefits as well that we should also consider.

Let us know if you have any further questions!

Thanks and regards,
Vedarth

On Fri, May 10, 2024 at 7:56 PM Chris Egerton <ch...@aiven.io.invalid>
wrote:

> Hi Krish and Prabha,
>
> Thanks for your replies. I still have some follow-up questions:
>
> 1) I read https://docs.docker.com/trusted-content/official-images/ and
> didn't find anything on that page or immediately around it that explains
> what compliance requirements might be satisfied by a DOI that couldn't be
> satisfied by the existing apache/kafka image. Can you elaborate? Feel free
> to provide another link, but please also quote the relevant sections from
> it (as StackOverflow likes to say, links can grow stale over time).
>
> 2) It would be great to see a brief summary of the differences in these
> images included in the KIP, in order to try to gauge how this would look to
> our users.
>
> 3) What I suggested last time was not a separate apache/apache-docker
> repository, but a repository controlled entirely by Docker. The DOI docs
> [1] state that "While it's preferable to have upstream software authors
> maintaining their Docker Official Images, this isn't a strict requirement",
> which I take to mean that it's not required for an Apache Kafka DOI to live
> under the apache organization on GitHub. It also seems like there's
> precedent for this: images for MySQL [2] and PHP [3] already exist under
> the control of Docker. The reason I think this is worth considering is that
> Docker can arbitrarily change the eligibility requirements for their
> official images at any time, and it doesn't seem like there's a clear
> process in the KIP for judging how we should respond to these changes (in
> fact, it seems like the idea in the KIP is that we should make any change
> required with no further vetting beyond possibly a pull request on
> apache/kafka, which would require approval by a committer). By hosting the
> DOI definitions ourselves (either in apache/kafka, or in a theoretical
> apache/docker-kafka repository), we take responsibility for the image, even
> if the owner on Docker Hub is Docker, not Apache. If the code lives
> elsewhere, then (as long as basic trademark and possibly security
> guidelines are respected) Apache doesn't have to concern itself at all with
> the image and the maintainers are free to make whatever changes they want
> to it in order to meet Docker's requirements.
>
> [1] -
> https://docs.docker.com/trusted-content/official-images/contributing/,
> second paragraph under "Contributing to Docker Official Images"
> [2] - https://github.com/docker-library/mysql
> [3] - https://github.com/docker-library/php
>
> Cheers,
>
> Chris
>
> On Fri, May 10, 2024 at 7:46 AM Krish Vora <kr...@gmail.com> wrote:
>
> > Hey Chris,
> >
> > We have responded to the initial round of queries. And as you mentioned
> in
> > your comment, you may have more questions related to this KIP. Please let
> > us know if you have any.
> > We want to start the voting for this KIP soon, as we intend to include it
> > in the 3.8.0 release.
> >
> > Thanks.
> >
> > Regards,
> >
> > Krish.
> >
> > On Wed, May 8, 2024 at 6:19 PM Krish Vora <kr...@gmail.com> wrote:
> >
> > > Hi Chris. Thanks for the questions.
> > >
> > > 3. Would a separate Docker-owned repository be out of the question? I'm
> > >> guessing there are some trademark issues that might get in the way,
> but
> > >> it's worth exploring since the entire purpose of this KIP seems to be
> to
> > >> provide images that are vetted and designed by Docker more than by the
> > >> Apache Kafka contributors/committers/PMC.
> > >
> > >
> > >
> > >    - The process for introducing a Docker Official Image involves
> > >       - Hosting the Dockerfile in the Apache Kafka repository and
> > >       - Providing the path to this Dockerfile to Docker Hub in Docker
> > >       Hub’s own repo
> > >       <
> > https://github.com/docker-library/official-images/tree/master/library>
> > >       .
> > >    - This ensures that any updates to the Dockerfile in the AK
> repository
> > >    are directly applicable to the docker official images available on
> > Docker
> > >    Hub.
> > >
> > >
> > >    - We also did not find any added advantage to create a separate
> > >    repository named apache-docker within the Apache GitHub
> organization.
> > >
> > > Thanks,
> > > Krish.
> > >
> > > On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli
> > > <mp...@confluent.io.invalid> wrote:
> > >
> > >> Hi Chris,  I would like to add more context to this KIP's motivation.
> > >> Vedarth and Krish, please weigh in with your inputs.
> > >>
> > >> In the motivation section it's stated that "Several other Apache
> > projects,
> > >> > like Flink, Spark, Solr, have already released Docker Official
> Images,
> > >> with
> > >> > download figures ranging from 50 million to over 1 billion. These
> > >> numbers
> > >> > highlight the significant demand among users." But then immediately
> > >> > afterwards, we learn that "Also the Docker Official Images are
> always
> > >> the
> > >> > top 1 search result, irrespective of the number of downloads."
> > Wouldn't
> > >> a
> > >> > high number of downloads for an image naturally follow from being
> the
> > >> top
> > >> > search result? It seems like we can't necessarily assume that Docker
> > >> > Official Images are inherently more desirable for users based solely
> > on
> > >> > download statistics.
> > >> >
> > >>
> > >> *My thoughts: *Unlike the Sponsored OSS image, the Docker Official
> image
> > >> is
> > >> more desirable for workloads that have stringent compliance
> > requirements.
> > >> More details on why official images are more trusted are documented
> here
> > >> <https://docs.docker.com/trusted-content/official-images/>. The
> Docker
> > >> Official image would also help an absolutely new Kafka beginner who
> > might
> > >> not know about Apache or the concept of Sponsored images. We want to
> > make
> > >> it easier for Kafka beginners to discover the Kafka image through
> > >> DockerHub.
> > >>
> > >>
> > >> Can you elaborate on the value that these new images would add from a
> > >> > user's perspective? I'm hesitant to introduce another image, since
> it
> > >> adds
> > >> > to the cognitive burden of people who will inevitably have to answer
> > the
> > >> > question of "What are the differences between all of the available
> > >> images
> > >> > and which one is best for my use case?"
> > >> >
> > >>
> > >>
> > >> *My thoughts: *This is a valid concern to address. The response to the
> > >> above question addresses the value-add this new Docker Official image
> > >> would
> > >> provide. I also agree we need a clear distinction between each of
> these
> > >> images to be well documented. We plan to update the AK website with
> > >> details
> > >> on how, why, and when a developer would want to use each of these
> > >> particular images(KIP-974,975,1028).
> > >>
> > >> Thanks,
> > >> Prabha.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton <chrise@aiven.io.invalid
> >
> > >> wrote:
> > >>
> > >> > Hi Vedarth and Krish,
> > >> >
> > >> > Thanks for the KIP! I have to admit I'm a little skeptical;
> hopefully
> > >> you
> > >> > can help me understand the need for these additional images.
> > >> >
> > >> > 1) In the motivation section it's stated that "Several other Apache
> > >> > projects, like Flink, Spark, Solr, have already released Docker
> > Official
> > >> > Images, with download figures ranging from 50 million to over 1
> > billion.
> > >> > These numbers highlight the significant demand among users." But
> then
> > >> > immediately afterwards, we learn that "Also the Docker Official
> Images
> > >> are
> > >> > always the top 1 search result, irrespective of the number of
> > >> downloads."
> > >> > Wouldn't a high number of downloads for an image naturally follow
> from
> > >> > being the top search result? It seems like we can't necessarily
> assume
> > >> that
> > >> > Docker Official Images are inherently more desirable for users based
> > >> solely
> > >> > on download statistics.
> > >> >
> > >> > 2) Can you elaborate on the value that these new images would add
> > from a
> > >> > user's perspective? I'm hesitant to introduce another image, since
> it
> > >> adds
> > >> > to the cognitive burden of people who will inevitably have to answer
> > the
> > >> > question of "What are the differences between all of the available
> > >> images
> > >> > and which one is best for my use case?"
> > >> >
> > >> > 3) Would a separate Docker-owned repository be out of the question?
> > I'm
> > >> > guessing there are some trademark issues that might get in the way,
> > but
> > >> > it's worth exploring since the entire purpose of this KIP seems to
> be
> > to
> > >> > provide images that are vetted and designed by Docker more than by
> the
> > >> > Apache Kafka contributors/committers/PMC.
> > >> >
> > >> > I may have more questions later but wanted to get this initial round
> > out
> > >> > now without trying to list everything first.
> > >> >
> > >> > Looking forward to your thoughts!
> > >> >
> > >> > Cheers,
> > >> >
> > >> > Chris
> > >> >
> > >> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <
> > >> vedarth.sharma@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hey folks,
> > >> > >
> > >> > > Thanks a lot for reviewing the KIP and providing feedback.
> > >> > > The discussion thread seems resolved and KIP has been updated
> > >> > accordingly.
> > >> > > We will be starting the voting thread for this KIP in the next few
> > >> days.
> > >> > > Please take a look at the KIP and let us know if any further
> > >> discussion
> > >> > > is needed.
> > >> > >
> > >> > > Thanks and regards,
> > >> > > Vedarth
> > >> > >
> > >> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <
> > manikumar.reddy@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Thanks Krish. KIP looks good to me.
> > >> > > >
> > >> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <
> krishvora01@gmail.com
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > Hi Manikumar,
> > >> > > > >
> > >> > > > > Thanks for the comments.
> > >> > > > >
> > >> > > > > Maybe as part of the release process, RM can create a JIRA for
> > >> this
> > >> > > > > > task. This can be taken by RM or any comitter or any
> > contributor
> > >> > > (with
> > >> > > > > > some help from commiters to run "Docker Image Preparation
> via
> > >> > GitHub
> > >> > > > > > Actions:"
> > >> > > > >
> > >> > > > > This sounds like a good idea. This step would be beneficial.
> By
> > >> > > creating
> > >> > > > a
> > >> > > > > JIRA ticket, it will also serve as a reminder to complete the
> > >> > > > post-release
> > >> > > > > steps for the Docker official images. Have updated the KIP
> with
> > >> this
> > >> > > > step.
> > >> > > > >
> > >> > > > > Is this using GitHub Actions workflow? or manual testing?
> > >> > > > >
> > >> > > > > This will be done by a Github Actions workflow, which will
> test
> > >> the
> > >> > > > static
> > >> > > > > Docker Official Image assets for a specific release version.
> > >> > > > >
> > >> > > > > Is it mandatory for RM/comitters to raise the PR to Docker
> Hub’s
> > >> > > > > > official images repository (or) can it be done by any
> > >> contributor.
> > >> > > > >
> > >> > > > > I believe that it can be done by any contributor (ref: This
> link
> > >> > > > > <
> > >> >
> https://docs.docker.com/trusted-content/official-images/contributing/
> > >> > > >
> > >> > > > > quotes "*Anyone can provide feedback, contribute code, suggest
> > >> > process
> > >> > > > > changes, or even propose a new Official Image.*")
> > >> > > > >
> > >> > > > > Also I was thinking, once the KIP gets voted, we should try to
> > >> > release
> > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help
> > us
> > >> to
> > >> > > > > > validate the process and allow us to fix any changes
> suggested
> > >> by
> > >> > > > > > Dockerhub before the 3.8.0 release.
> > >> > > > >
> > >> > > > > This sounds like a great idea. This KIP proposes release of
> DOI
> > >> as a
> > >> > > > > post-release process, which can be done anytime post release.
> > >> Since
> > >> > > 3.7.0
> > >> > > > > is already released, we can perform these steps for that
> release
> > >> too.
> > >> > > By
> > >> > > > > the time the KIP gets implemented, if 3.7.1 is released, we
> > could
> > >> do
> > >> > > > these
> > >> > > > > steps for 3.7.1, instead of 3.7.0. This would allow us to make
> > >> > changes
> > >> > > to
> > >> > > > > the Dockerfiles and other assets based on feedback from Docker
> > Hub
> > >> > > before
> > >> > > > > the release of version 3.8.0.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Krish.
> > >> > > > >
> > >> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
> > >> > manikumar.reddy@gmail.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi Krish,
> > >> > > > > >
> > >> > > > > > Thanks for the updated KIP. a few comments below.
> > >> > > > > >
> > >> > > > > > > "These actions can be carried out by the RM or any
> > contributor
> > >> > post
> > >> > > > the
> > >> > > > > > release process."
> > >> > > > > > Maybe as part of the release process, RM can create a JIRA
> for
> > >> this
> > >> > > > > > task. This can be taken by RM or any comitter or any
> > contributor
> > >> > > (with
> > >> > > > > > some help from commiters to run "Docker Image Preparation
> via
> > >> > GitHub
> > >> > > > > > Actions:"
> > >> > > > > >
> > >> > > > > > > "Perform Docker build tests to ensure image integrity"
> > >> > > > > > Is this using GitHub Actions workflow? or manual testing?
> > >> > > > > >
> > >> > > > > > > "The RM will manually raise the final PR to Docker Hub’s
> > >> official
> > >> > > > images
> > >> > > > > > repository using the contents of the generated file"
> > >> > > > > >  Is it mandatory for RM/comitters to raise the PR to Docker
> > >> Hub’s
> > >> > > > > > official images repository (or) can it be done by any
> > >> contributor.
> > >> > > > > >
> > >> > > > > > Also I was thinking, once the KIP gets voted, we should try
> to
> > >> > > release
> > >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help
> > us
> > >> to
> > >> > > > > > validate the process and allow us to fix any changes
> suggested
> > >> by
> > >> > > > > > Dockerhub before the 3.8.0 release.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > >
> > >> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <
> > >> krishvora01@gmail.com>
> > >> > > > wrote:
> > >> > > > > > >
> > >> > > > > > > Hi Manikumar and Luke.
> > >> > > > > > > Thanks for the questions.
> > >> > > > > > >
> > >> > > > > > > 1. No, the Docker inventory files and configurations will
> > not
> > >> be
> > >> > > the
> > >> > > > same
> > >> > > > > > > for Open Source Software (OSS) Images and Docker Official
> > >> Images
> > >> > > > (DOI).
> > >> > > > > > >
> > >> > > > > > > For OSS images, the Dockerfile located in
> > >> docker/jvm/dockerfile
> > >> > is
> > >> > > > > > > utilized. This process is integrated with the existing
> > release
> > >> > > > pipeline
> > >> > > > > > as
> > >> > > > > > > outlined in KIP-975
> > >> > > > > > > <
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > >> > > > > > >,
> > >> > > > > > > where the Kafka URL is provided as a build argument. This
> > >> method
> > >> > > > allows
> > >> > > > > > for
> > >> > > > > > > building, testing, and releasing OSS images dynamically.
> The
> > >> OSS
> > >> > > > images
> > >> > > > > > > will continue to be released under the standard release
> > >> process .
> > >> > > > > > >
> > >> > > > > > > In contrast, the release process for DOIs requires
> providing
> > >> the
> > >> > > > Docker
> > >> > > > > > Hub
> > >> > > > > > > team with a specific directory for each version release
> that
> > >> > > > contains a
> > >> > > > > > > standalone Dockerfile. These Dockerfiles are designed to
> be
> > >> > > > > > > self-sufficient, hence require hardcoded values instead of
> > >> > relying
> > >> > > on
> > >> > > > > > build
> > >> > > > > > > arguments. To accommodate this, in our proposed approach,
> a
> > >> new
> > >> > > > directory
> > >> > > > > > > named docker_official_images has been created. This
> > directory
> > >> > > > contains
> > >> > > > > > > version-specific directories, having Dockerfiles with
> > >> hardcoded
> > >> > > > > > > configurations for each release, acting as the source of
> > truth
> > >> > for
> > >> > > > DOI
> > >> > > > > > > releases. The hardcoded dockerfiles will be created using
> > the
> > >> > > > > > > docker/jvm/dockerfile as a template. Thus, as part of post
> > >> > release
> > >> > > we
> > >> > > > > > will
> > >> > > > > > > be creating a Dockerfile that will be reviewed by the
> > >> Dockerhub
> > >> > > > community
> > >> > > > > > > and might need changes as per their review. This approach
> > >> ensures
> > >> > > > that
> > >> > > > > > DOIs
> > >> > > > > > > are built consistently and meet the specific requirements
> > set
> > >> by
> > >> > > > Docker
> > >> > > > > > Hub.
> > >> > > > > > >
> > >> > > > > > > 2. Yes Manikumar, transitioning the release of Docker
> > Official
> > >> > > Images
> > >> > > > > > (DOI)
> > >> > > > > > > to a post-release activity does address the concerns about
> > >> > > > complicating
> > >> > > > > > the
> > >> > > > > > > release process. Initially, we considered incorporating
> DOI
> > >> > release
> > >> > > > > > > directly into Kafka's release workflow. However, this
> > approach
> > >> > > > > > > significantly increased the RMs workload due to the
> addition
> > >> of
> > >> > > > numerous
> > >> > > > > > > steps, complicating the process. By designating the DOI
> > >> release
> > >> > as
> > >> > > a
> > >> > > > > > > post-release task, we maintain the original release
> process.
> > >> This
> > >> > > > > > > adjustment allows for the DOI release to be done after the
> > >> main
> > >> > > > release.
> > >> > > > > > We
> > >> > > > > > > have revised the KIP to reflect that DOI releases will now
> > >> occur
> > >> > > > after
> > >> > > > > > the
> > >> > > > > > > main release phase. Please review the updated document and
> > >> > provide
> > >> > > > any
> > >> > > > > > > feedback you might have.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > Krish.
> > >> > > > > > >
> > >> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <
> showuon@gmail.com
> > >
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi Krishna,
> > >> > > > > > > >
> > >> > > > > > > > I also have the same question as Manikumar raised:
> > >> > > > > > > > 1. Will the Docker inventory files/etc are the same for
> > OSS
> > >> > Image
> > >> > > > and
> > >> > > > > > > > Docker Official Images?
> > >> > > > > > > > If no, then why not? Could we make them identical so
> that
> > we
> > >> > > don't
> > >> > > > > > have to
> > >> > > > > > > > build 2 images for each release?
> > >> > > > > > > >
> > >> > > > > > > > Thank you.
> > >> > > > > > > > Luke
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> > >> > > > manikumar.reddy@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi Krishna,
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks for the KIP.
> > >> > > > > > > > >
> > >> > > > > > > > > I think Docker Official Images will be beneficial to
> the
> > >> > Kafka
> > >> > > > > > community.
> > >> > > > > > > > > Few queries below.
> > >> > > > > > > > >
> > >> > > > > > > > > 1. Will the Docker inventory files/etc are the same
> for
> > >> OSS
> > >> > > > Image and
> > >> > > > > > > > > Docker Official Images
> > >> > > > > > > > > 2. I am a bit worried about the new steps to the
> release
> > >> > > process.
> > >> > > > > > Maybe
> > >> > > > > > > > we
> > >> > > > > > > > > should consider Docker Official Images release as
> > >> > Post-Release
> > >> > > > > > activity.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > >
> > >> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> > >> > > > krishvora01@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi Hector,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for reaching out. This KIP builds on top of
> > >> KIP-975
> > >> > > > > > > > > > <
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > >> > > > > > > > > > >
> > >> > > > > > > > > > and
> > >> > > > > > > > > > aims to introduce a JVM-based Docker Official Image
> > (DOI
> > >> > > > > > > > > > <
> > >> https://docs.docker.com/trusted-content/official-images/
> > >> > >)
> > >> > > > for
> > >> > > > > > Apache
> > >> > > > > > > > > > Kafka that will be visible under Docker Official
> > Images
> > >> > > > > > > > > > <
> > https://hub.docker.com/search?image_filter=official&q=
> > >> >.
> > >> > > Once
> > >> > > > > > > > > implemented
> > >> > > > > > > > > > for Apache Kafka, for each release, there will be
> one
> > >> more
> > >> > > > > > JVM-based
> > >> > > > > > > > > Docker
> > >> > > > > > > > > > image available to users.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Currently, we already have an OSS sponsored image,
> > which
> > >> > was
> > >> > > > > > introduced
> > >> > > > > > > > > via
> > >> > > > > > > > > > KIP-975 (apache/kafka <
> > >> > > > https://hub.docker.com/r/apache/kafka/tags
> > >> > > > > > >)
> > >> > > > > > > > > which
> > >> > > > > > > > > > comes under The Apache Software Foundation <
> > >> > > > > > > > > > https://hub.docker.com/u/apache> in
> > >> > > > > > > > > > Docker Hub. The new Docker Image is the Docker
> > Official
> > >> > Image
> > >> > > > > > (DOI),
> > >> > > > > > > > > which
> > >> > > > > > > > > > will be built and maintained by Docker Community.
> > >> > > > > > > > > >
> > >> > > > > > > > > > For example, for a release version like 3.8.0 we
> will
> > >> have
> > >> > > two
> > >> > > > JVM
> > >> > > > > > > > based
> > >> > > > > > > > > > docker images:-
> > >> > > > > > > > > >
> > >> > > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > >> > > > > > > > > >    - kafka:3.8.0 (Docker Official image)
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > I have added the same in the KIP too for everyone's
> > >> > > reference.
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Krish.
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino
> > >> > (BLOOMBERG/
> > >> > > > 919
> > >> > > > > > 3RD
> > >> > > > > > > > A) <
> > >> > > > > > > > > > hgeraldino@bloomberg.net> wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > What is the difference between this KIP and
> KIP-975:
> > >> > Docker
> > >> > > > > > Image for
> > >> > > > > > > > > > > Apache Kafka?
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07
> > >> > > UTC-4:00To:
> > >> > > > > > > > > > > dev@kafka.apache.org
> > >> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image
> > for
> > >> > > Apache
> > >> > > > > > Kafka
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Hi everyone,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I would like to start the discussion on
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > >> > > > > > > > > > > age+for+Apache+Kafka
> > >> > > > > > > > > > >  .
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > This KIP aims to introduce JVM based Docker
> Official
> > >> > Image
> > >> > > > (DOI)
> > >> > > > > > for
> > >> > > > > > > > > > Apache
> > >> > > > > > > > > > > Kafka.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Regards,
> > >> > > > > > > > > > > Krish.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >>
> > >> [image: Confluent] <https://www.confluent.io>
> > >> Prabha Manepalli
> > >> Product Manager for Confluent Platform Security, Docker
> > >> linkedin.com/in/prabhamanepalli
> > >>
> > >
> >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Chris Egerton <ch...@aiven.io.INVALID>]

Hi Krish and Prabha,

Thanks for your replies. I still have some follow-up questions:

1) I read https://docs.docker.com/trusted-content/official-images/ and
didn't find anything on that page or immediately around it that explains
what compliance requirements might be satisfied by a DOI that couldn't be
satisfied by the existing apache/kafka image. Can you elaborate? Feel free
to provide another link, but please also quote the relevant sections from
it (as StackOverflow likes to say, links can grow stale over time).

2) It would be great to see a brief summary of the differences in these
images included in the KIP, in order to try to gauge how this would look to
our users.

3) What I suggested last time was not a separate apache/apache-docker
repository, but a repository controlled entirely by Docker. The DOI docs
[1] state that "While it's preferable to have upstream software authors
maintaining their Docker Official Images, this isn't a strict requirement",
which I take to mean that it's not required for an Apache Kafka DOI to live
under the apache organization on GitHub. It also seems like there's
precedent for this: images for MySQL [2] and PHP [3] already exist under
the control of Docker. The reason I think this is worth considering is that
Docker can arbitrarily change the eligibility requirements for their
official images at any time, and it doesn't seem like there's a clear
process in the KIP for judging how we should respond to these changes (in
fact, it seems like the idea in the KIP is that we should make any change
required with no further vetting beyond possibly a pull request on
apache/kafka, which would require approval by a committer). By hosting the
DOI definitions ourselves (either in apache/kafka, or in a theoretical
apache/docker-kafka repository), we take responsibility for the image, even
if the owner on Docker Hub is Docker, not Apache. If the code lives
elsewhere, then (as long as basic trademark and possibly security
guidelines are respected) Apache doesn't have to concern itself at all with
the image and the maintainers are free to make whatever changes they want
to it in order to meet Docker's requirements.

[1] - https://docs.docker.com/trusted-content/official-images/contributing/,
second paragraph under "Contributing to Docker Official Images"
[2] - https://github.com/docker-library/mysql
[3] - https://github.com/docker-library/php

Cheers,

Chris

On Fri, May 10, 2024 at 7:46 AM Krish Vora <kr...@gmail.com> wrote:

> Hey Chris,
>
> We have responded to the initial round of queries. And as you mentioned in
> your comment, you may have more questions related to this KIP. Please let
> us know if you have any.
> We want to start the voting for this KIP soon, as we intend to include it
> in the 3.8.0 release.
>
> Thanks.
>
> Regards,
>
> Krish.
>
> On Wed, May 8, 2024 at 6:19 PM Krish Vora <kr...@gmail.com> wrote:
>
> > Hi Chris. Thanks for the questions.
> >
> > 3. Would a separate Docker-owned repository be out of the question? I'm
> >> guessing there are some trademark issues that might get in the way, but
> >> it's worth exploring since the entire purpose of this KIP seems to be to
> >> provide images that are vetted and designed by Docker more than by the
> >> Apache Kafka contributors/committers/PMC.
> >
> >
> >
> >    - The process for introducing a Docker Official Image involves
> >       - Hosting the Dockerfile in the Apache Kafka repository and
> >       - Providing the path to this Dockerfile to Docker Hub in Docker
> >       Hub’s own repo
> >       <
> https://github.com/docker-library/official-images/tree/master/library>
> >       .
> >    - This ensures that any updates to the Dockerfile in the AK repository
> >    are directly applicable to the docker official images available on
> Docker
> >    Hub.
> >
> >
> >    - We also did not find any added advantage to create a separate
> >    repository named apache-docker within the Apache GitHub organization.
> >
> > Thanks,
> > Krish.
> >
> > On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli
> > <mp...@confluent.io.invalid> wrote:
> >
> >> Hi Chris,  I would like to add more context to this KIP's motivation.
> >> Vedarth and Krish, please weigh in with your inputs.
> >>
> >> In the motivation section it's stated that "Several other Apache
> projects,
> >> > like Flink, Spark, Solr, have already released Docker Official Images,
> >> with
> >> > download figures ranging from 50 million to over 1 billion. These
> >> numbers
> >> > highlight the significant demand among users." But then immediately
> >> > afterwards, we learn that "Also the Docker Official Images are always
> >> the
> >> > top 1 search result, irrespective of the number of downloads."
> Wouldn't
> >> a
> >> > high number of downloads for an image naturally follow from being the
> >> top
> >> > search result? It seems like we can't necessarily assume that Docker
> >> > Official Images are inherently more desirable for users based solely
> on
> >> > download statistics.
> >> >
> >>
> >> *My thoughts: *Unlike the Sponsored OSS image, the Docker Official image
> >> is
> >> more desirable for workloads that have stringent compliance
> requirements.
> >> More details on why official images are more trusted are documented here
> >> <https://docs.docker.com/trusted-content/official-images/>. The Docker
> >> Official image would also help an absolutely new Kafka beginner who
> might
> >> not know about Apache or the concept of Sponsored images. We want to
> make
> >> it easier for Kafka beginners to discover the Kafka image through
> >> DockerHub.
> >>
> >>
> >> Can you elaborate on the value that these new images would add from a
> >> > user's perspective? I'm hesitant to introduce another image, since it
> >> adds
> >> > to the cognitive burden of people who will inevitably have to answer
> the
> >> > question of "What are the differences between all of the available
> >> images
> >> > and which one is best for my use case?"
> >> >
> >>
> >>
> >> *My thoughts: *This is a valid concern to address. The response to the
> >> above question addresses the value-add this new Docker Official image
> >> would
> >> provide. I also agree we need a clear distinction between each of these
> >> images to be well documented. We plan to update the AK website with
> >> details
> >> on how, why, and when a developer would want to use each of these
> >> particular images(KIP-974,975,1028).
> >>
> >> Thanks,
> >> Prabha.
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton <ch...@aiven.io.invalid>
> >> wrote:
> >>
> >> > Hi Vedarth and Krish,
> >> >
> >> > Thanks for the KIP! I have to admit I'm a little skeptical; hopefully
> >> you
> >> > can help me understand the need for these additional images.
> >> >
> >> > 1) In the motivation section it's stated that "Several other Apache
> >> > projects, like Flink, Spark, Solr, have already released Docker
> Official
> >> > Images, with download figures ranging from 50 million to over 1
> billion.
> >> > These numbers highlight the significant demand among users." But then
> >> > immediately afterwards, we learn that "Also the Docker Official Images
> >> are
> >> > always the top 1 search result, irrespective of the number of
> >> downloads."
> >> > Wouldn't a high number of downloads for an image naturally follow from
> >> > being the top search result? It seems like we can't necessarily assume
> >> that
> >> > Docker Official Images are inherently more desirable for users based
> >> solely
> >> > on download statistics.
> >> >
> >> > 2) Can you elaborate on the value that these new images would add
> from a
> >> > user's perspective? I'm hesitant to introduce another image, since it
> >> adds
> >> > to the cognitive burden of people who will inevitably have to answer
> the
> >> > question of "What are the differences between all of the available
> >> images
> >> > and which one is best for my use case?"
> >> >
> >> > 3) Would a separate Docker-owned repository be out of the question?
> I'm
> >> > guessing there are some trademark issues that might get in the way,
> but
> >> > it's worth exploring since the entire purpose of this KIP seems to be
> to
> >> > provide images that are vetted and designed by Docker more than by the
> >> > Apache Kafka contributors/committers/PMC.
> >> >
> >> > I may have more questions later but wanted to get this initial round
> out
> >> > now without trying to list everything first.
> >> >
> >> > Looking forward to your thoughts!
> >> >
> >> > Cheers,
> >> >
> >> > Chris
> >> >
> >> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <
> >> vedarth.sharma@gmail.com>
> >> > wrote:
> >> >
> >> > > Hey folks,
> >> > >
> >> > > Thanks a lot for reviewing the KIP and providing feedback.
> >> > > The discussion thread seems resolved and KIP has been updated
> >> > accordingly.
> >> > > We will be starting the voting thread for this KIP in the next few
> >> days.
> >> > > Please take a look at the KIP and let us know if any further
> >> discussion
> >> > > is needed.
> >> > >
> >> > > Thanks and regards,
> >> > > Vedarth
> >> > >
> >> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <
> manikumar.reddy@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Thanks Krish. KIP looks good to me.
> >> > > >
> >> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <krishvora01@gmail.com
> >
> >> > > wrote:
> >> > > > >
> >> > > > > Hi Manikumar,
> >> > > > >
> >> > > > > Thanks for the comments.
> >> > > > >
> >> > > > > Maybe as part of the release process, RM can create a JIRA for
> >> this
> >> > > > > > task. This can be taken by RM or any comitter or any
> contributor
> >> > > (with
> >> > > > > > some help from commiters to run "Docker Image Preparation via
> >> > GitHub
> >> > > > > > Actions:"
> >> > > > >
> >> > > > > This sounds like a good idea. This step would be beneficial. By
> >> > > creating
> >> > > > a
> >> > > > > JIRA ticket, it will also serve as a reminder to complete the
> >> > > > post-release
> >> > > > > steps for the Docker official images. Have updated the KIP with
> >> this
> >> > > > step.
> >> > > > >
> >> > > > > Is this using GitHub Actions workflow? or manual testing?
> >> > > > >
> >> > > > > This will be done by a Github Actions workflow, which will test
> >> the
> >> > > > static
> >> > > > > Docker Official Image assets for a specific release version.
> >> > > > >
> >> > > > > Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> >> > > > > > official images repository (or) can it be done by any
> >> contributor.
> >> > > > >
> >> > > > > I believe that it can be done by any contributor (ref: This link
> >> > > > > <
> >> > https://docs.docker.com/trusted-content/official-images/contributing/
> >> > > >
> >> > > > > quotes "*Anyone can provide feedback, contribute code, suggest
> >> > process
> >> > > > > changes, or even propose a new Official Image.*")
> >> > > > >
> >> > > > > Also I was thinking, once the KIP gets voted, we should try to
> >> > release
> >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help
> us
> >> to
> >> > > > > > validate the process and allow us to fix any changes suggested
> >> by
> >> > > > > > Dockerhub before the 3.8.0 release.
> >> > > > >
> >> > > > > This sounds like a great idea. This KIP proposes release of DOI
> >> as a
> >> > > > > post-release process, which can be done anytime post release.
> >> Since
> >> > > 3.7.0
> >> > > > > is already released, we can perform these steps for that release
> >> too.
> >> > > By
> >> > > > > the time the KIP gets implemented, if 3.7.1 is released, we
> could
> >> do
> >> > > > these
> >> > > > > steps for 3.7.1, instead of 3.7.0. This would allow us to make
> >> > changes
> >> > > to
> >> > > > > the Dockerfiles and other assets based on feedback from Docker
> Hub
> >> > > before
> >> > > > > the release of version 3.8.0.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Krish.
> >> > > > >
> >> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
> >> > manikumar.reddy@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi Krish,
> >> > > > > >
> >> > > > > > Thanks for the updated KIP. a few comments below.
> >> > > > > >
> >> > > > > > > "These actions can be carried out by the RM or any
> contributor
> >> > post
> >> > > > the
> >> > > > > > release process."
> >> > > > > > Maybe as part of the release process, RM can create a JIRA for
> >> this
> >> > > > > > task. This can be taken by RM or any comitter or any
> contributor
> >> > > (with
> >> > > > > > some help from commiters to run "Docker Image Preparation via
> >> > GitHub
> >> > > > > > Actions:"
> >> > > > > >
> >> > > > > > > "Perform Docker build tests to ensure image integrity"
> >> > > > > > Is this using GitHub Actions workflow? or manual testing?
> >> > > > > >
> >> > > > > > > "The RM will manually raise the final PR to Docker Hub’s
> >> official
> >> > > > images
> >> > > > > > repository using the contents of the generated file"
> >> > > > > >  Is it mandatory for RM/comitters to raise the PR to Docker
> >> Hub’s
> >> > > > > > official images repository (or) can it be done by any
> >> contributor.
> >> > > > > >
> >> > > > > > Also I was thinking, once the KIP gets voted, we should try to
> >> > > release
> >> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help
> us
> >> to
> >> > > > > > validate the process and allow us to fix any changes suggested
> >> by
> >> > > > > > Dockerhub before the 3.8.0 release.
> >> > > > > >
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <
> >> krishvora01@gmail.com>
> >> > > > wrote:
> >> > > > > > >
> >> > > > > > > Hi Manikumar and Luke.
> >> > > > > > > Thanks for the questions.
> >> > > > > > >
> >> > > > > > > 1. No, the Docker inventory files and configurations will
> not
> >> be
> >> > > the
> >> > > > same
> >> > > > > > > for Open Source Software (OSS) Images and Docker Official
> >> Images
> >> > > > (DOI).
> >> > > > > > >
> >> > > > > > > For OSS images, the Dockerfile located in
> >> docker/jvm/dockerfile
> >> > is
> >> > > > > > > utilized. This process is integrated with the existing
> release
> >> > > > pipeline
> >> > > > > > as
> >> > > > > > > outlined in KIP-975
> >> > > > > > > <
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> >> > > > > > >,
> >> > > > > > > where the Kafka URL is provided as a build argument. This
> >> method
> >> > > > allows
> >> > > > > > for
> >> > > > > > > building, testing, and releasing OSS images dynamically. The
> >> OSS
> >> > > > images
> >> > > > > > > will continue to be released under the standard release
> >> process .
> >> > > > > > >
> >> > > > > > > In contrast, the release process for DOIs requires providing
> >> the
> >> > > > Docker
> >> > > > > > Hub
> >> > > > > > > team with a specific directory for each version release that
> >> > > > contains a
> >> > > > > > > standalone Dockerfile. These Dockerfiles are designed to be
> >> > > > > > > self-sufficient, hence require hardcoded values instead of
> >> > relying
> >> > > on
> >> > > > > > build
> >> > > > > > > arguments. To accommodate this, in our proposed approach, a
> >> new
> >> > > > directory
> >> > > > > > > named docker_official_images has been created. This
> directory
> >> > > > contains
> >> > > > > > > version-specific directories, having Dockerfiles with
> >> hardcoded
> >> > > > > > > configurations for each release, acting as the source of
> truth
> >> > for
> >> > > > DOI
> >> > > > > > > releases. The hardcoded dockerfiles will be created using
> the
> >> > > > > > > docker/jvm/dockerfile as a template. Thus, as part of post
> >> > release
> >> > > we
> >> > > > > > will
> >> > > > > > > be creating a Dockerfile that will be reviewed by the
> >> Dockerhub
> >> > > > community
> >> > > > > > > and might need changes as per their review. This approach
> >> ensures
> >> > > > that
> >> > > > > > DOIs
> >> > > > > > > are built consistently and meet the specific requirements
> set
> >> by
> >> > > > Docker
> >> > > > > > Hub.
> >> > > > > > >
> >> > > > > > > 2. Yes Manikumar, transitioning the release of Docker
> Official
> >> > > Images
> >> > > > > > (DOI)
> >> > > > > > > to a post-release activity does address the concerns about
> >> > > > complicating
> >> > > > > > the
> >> > > > > > > release process. Initially, we considered incorporating DOI
> >> > release
> >> > > > > > > directly into Kafka's release workflow. However, this
> approach
> >> > > > > > > significantly increased the RMs workload due to the addition
> >> of
> >> > > > numerous
> >> > > > > > > steps, complicating the process. By designating the DOI
> >> release
> >> > as
> >> > > a
> >> > > > > > > post-release task, we maintain the original release process.
> >> This
> >> > > > > > > adjustment allows for the DOI release to be done after the
> >> main
> >> > > > release.
> >> > > > > > We
> >> > > > > > > have revised the KIP to reflect that DOI releases will now
> >> occur
> >> > > > after
> >> > > > > > the
> >> > > > > > > main release phase. Please review the updated document and
> >> > provide
> >> > > > any
> >> > > > > > > feedback you might have.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > Krish.
> >> > > > > > >
> >> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <showuon@gmail.com
> >
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi Krishna,
> >> > > > > > > >
> >> > > > > > > > I also have the same question as Manikumar raised:
> >> > > > > > > > 1. Will the Docker inventory files/etc are the same for
> OSS
> >> > Image
> >> > > > and
> >> > > > > > > > Docker Official Images?
> >> > > > > > > > If no, then why not? Could we make them identical so that
> we
> >> > > don't
> >> > > > > > have to
> >> > > > > > > > build 2 images for each release?
> >> > > > > > > >
> >> > > > > > > > Thank you.
> >> > > > > > > > Luke
> >> > > > > > > >
> >> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> >> > > > manikumar.reddy@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Krishna,
> >> > > > > > > > >
> >> > > > > > > > > Thanks for the KIP.
> >> > > > > > > > >
> >> > > > > > > > > I think Docker Official Images will be beneficial to the
> >> > Kafka
> >> > > > > > community.
> >> > > > > > > > > Few queries below.
> >> > > > > > > > >
> >> > > > > > > > > 1. Will the Docker inventory files/etc are the same for
> >> OSS
> >> > > > Image and
> >> > > > > > > > > Docker Official Images
> >> > > > > > > > > 2. I am a bit worried about the new steps to the release
> >> > > process.
> >> > > > > > Maybe
> >> > > > > > > > we
> >> > > > > > > > > should consider Docker Official Images release as
> >> > Post-Release
> >> > > > > > activity.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> >> > > > krishvora01@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi Hector,
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for reaching out. This KIP builds on top of
> >> KIP-975
> >> > > > > > > > > > <
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> >> > > > > > > > > > >
> >> > > > > > > > > > and
> >> > > > > > > > > > aims to introduce a JVM-based Docker Official Image
> (DOI
> >> > > > > > > > > > <
> >> https://docs.docker.com/trusted-content/official-images/
> >> > >)
> >> > > > for
> >> > > > > > Apache
> >> > > > > > > > > > Kafka that will be visible under Docker Official
> Images
> >> > > > > > > > > > <
> https://hub.docker.com/search?image_filter=official&q=
> >> >.
> >> > > Once
> >> > > > > > > > > implemented
> >> > > > > > > > > > for Apache Kafka, for each release, there will be one
> >> more
> >> > > > > > JVM-based
> >> > > > > > > > > Docker
> >> > > > > > > > > > image available to users.
> >> > > > > > > > > >
> >> > > > > > > > > > Currently, we already have an OSS sponsored image,
> which
> >> > was
> >> > > > > > introduced
> >> > > > > > > > > via
> >> > > > > > > > > > KIP-975 (apache/kafka <
> >> > > > https://hub.docker.com/r/apache/kafka/tags
> >> > > > > > >)
> >> > > > > > > > > which
> >> > > > > > > > > > comes under The Apache Software Foundation <
> >> > > > > > > > > > https://hub.docker.com/u/apache> in
> >> > > > > > > > > > Docker Hub. The new Docker Image is the Docker
> Official
> >> > Image
> >> > > > > > (DOI),
> >> > > > > > > > > which
> >> > > > > > > > > > will be built and maintained by Docker Community.
> >> > > > > > > > > >
> >> > > > > > > > > > For example, for a release version like 3.8.0 we will
> >> have
> >> > > two
> >> > > > JVM
> >> > > > > > > > based
> >> > > > > > > > > > docker images:-
> >> > > > > > > > > >
> >> > > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> >> > > > > > > > > >    - kafka:3.8.0 (Docker Official image)
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > I have added the same in the KIP too for everyone's
> >> > > reference.
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Krish.
> >> > > > > > > > > >
> >> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino
> >> > (BLOOMBERG/
> >> > > > 919
> >> > > > > > 3RD
> >> > > > > > > > A) <
> >> > > > > > > > > > hgeraldino@bloomberg.net> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi,
> >> > > > > > > > > > >
> >> > > > > > > > > > > What is the difference between this KIP and KIP-975:
> >> > Docker
> >> > > > > > Image for
> >> > > > > > > > > > > Apache Kafka?
> >> > > > > > > > > > >
> >> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07
> >> > > UTC-4:00To:
> >> > > > > > > > > > > dev@kafka.apache.org
> >> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image
> for
> >> > > Apache
> >> > > > > > Kafka
> >> > > > > > > > > > >
> >> > > > > > > > > > > Hi everyone,
> >> > > > > > > > > > >
> >> > > > > > > > > > > I would like to start the discussion on
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> >> > > > > > > > > > > age+for+Apache+Kafka
> >> > > > > > > > > > >  .
> >> > > > > > > > > > >
> >> > > > > > > > > > > This KIP aims to introduce JVM based Docker Official
> >> > Image
> >> > > > (DOI)
> >> > > > > > for
> >> > > > > > > > > > Apache
> >> > > > > > > > > > > Kafka.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Regards,
> >> > > > > > > > > > > Krish.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >>
> >> [image: Confluent] <https://www.confluent.io>
> >> Prabha Manepalli
> >> Product Manager for Confluent Platform Security, Docker
> >> linkedin.com/in/prabhamanepalli
> >>
> >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Krish Vora <kr...@gmail.com>]

Hey Chris,

We have responded to the initial round of queries. And as you mentioned in
your comment, you may have more questions related to this KIP. Please let
us know if you have any.
We want to start the voting for this KIP soon, as we intend to include it
in the 3.8.0 release.

Thanks.

Regards,

Krish.

On Wed, May 8, 2024 at 6:19 PM Krish Vora <kr...@gmail.com> wrote:

> Hi Chris. Thanks for the questions.
>
> 3. Would a separate Docker-owned repository be out of the question? I'm
>> guessing there are some trademark issues that might get in the way, but
>> it's worth exploring since the entire purpose of this KIP seems to be to
>> provide images that are vetted and designed by Docker more than by the
>> Apache Kafka contributors/committers/PMC.
>
>
>
>    - The process for introducing a Docker Official Image involves
>       - Hosting the Dockerfile in the Apache Kafka repository and
>       - Providing the path to this Dockerfile to Docker Hub in Docker
>       Hub’s own repo
>       <https://github.com/docker-library/official-images/tree/master/library>
>       .
>    - This ensures that any updates to the Dockerfile in the AK repository
>    are directly applicable to the docker official images available on Docker
>    Hub.
>
>
>    - We also did not find any added advantage to create a separate
>    repository named apache-docker within the Apache GitHub organization.
>
> Thanks,
> Krish.
>
> On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli
> <mp...@confluent.io.invalid> wrote:
>
>> Hi Chris,  I would like to add more context to this KIP's motivation.
>> Vedarth and Krish, please weigh in with your inputs.
>>
>> In the motivation section it's stated that "Several other Apache projects,
>> > like Flink, Spark, Solr, have already released Docker Official Images,
>> with
>> > download figures ranging from 50 million to over 1 billion. These
>> numbers
>> > highlight the significant demand among users." But then immediately
>> > afterwards, we learn that "Also the Docker Official Images are always
>> the
>> > top 1 search result, irrespective of the number of downloads." Wouldn't
>> a
>> > high number of downloads for an image naturally follow from being the
>> top
>> > search result? It seems like we can't necessarily assume that Docker
>> > Official Images are inherently more desirable for users based solely on
>> > download statistics.
>> >
>>
>> *My thoughts: *Unlike the Sponsored OSS image, the Docker Official image
>> is
>> more desirable for workloads that have stringent compliance requirements.
>> More details on why official images are more trusted are documented here
>> <https://docs.docker.com/trusted-content/official-images/>. The Docker
>> Official image would also help an absolutely new Kafka beginner who might
>> not know about Apache or the concept of Sponsored images. We want to make
>> it easier for Kafka beginners to discover the Kafka image through
>> DockerHub.
>>
>>
>> Can you elaborate on the value that these new images would add from a
>> > user's perspective? I'm hesitant to introduce another image, since it
>> adds
>> > to the cognitive burden of people who will inevitably have to answer the
>> > question of "What are the differences between all of the available
>> images
>> > and which one is best for my use case?"
>> >
>>
>>
>> *My thoughts: *This is a valid concern to address. The response to the
>> above question addresses the value-add this new Docker Official image
>> would
>> provide. I also agree we need a clear distinction between each of these
>> images to be well documented. We plan to update the AK website with
>> details
>> on how, why, and when a developer would want to use each of these
>> particular images(KIP-974,975,1028).
>>
>> Thanks,
>> Prabha.
>>
>>
>>
>>
>>
>> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton <ch...@aiven.io.invalid>
>> wrote:
>>
>> > Hi Vedarth and Krish,
>> >
>> > Thanks for the KIP! I have to admit I'm a little skeptical; hopefully
>> you
>> > can help me understand the need for these additional images.
>> >
>> > 1) In the motivation section it's stated that "Several other Apache
>> > projects, like Flink, Spark, Solr, have already released Docker Official
>> > Images, with download figures ranging from 50 million to over 1 billion.
>> > These numbers highlight the significant demand among users." But then
>> > immediately afterwards, we learn that "Also the Docker Official Images
>> are
>> > always the top 1 search result, irrespective of the number of
>> downloads."
>> > Wouldn't a high number of downloads for an image naturally follow from
>> > being the top search result? It seems like we can't necessarily assume
>> that
>> > Docker Official Images are inherently more desirable for users based
>> solely
>> > on download statistics.
>> >
>> > 2) Can you elaborate on the value that these new images would add from a
>> > user's perspective? I'm hesitant to introduce another image, since it
>> adds
>> > to the cognitive burden of people who will inevitably have to answer the
>> > question of "What are the differences between all of the available
>> images
>> > and which one is best for my use case?"
>> >
>> > 3) Would a separate Docker-owned repository be out of the question? I'm
>> > guessing there are some trademark issues that might get in the way, but
>> > it's worth exploring since the entire purpose of this KIP seems to be to
>> > provide images that are vetted and designed by Docker more than by the
>> > Apache Kafka contributors/committers/PMC.
>> >
>> > I may have more questions later but wanted to get this initial round out
>> > now without trying to list everything first.
>> >
>> > Looking forward to your thoughts!
>> >
>> > Cheers,
>> >
>> > Chris
>> >
>> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <
>> vedarth.sharma@gmail.com>
>> > wrote:
>> >
>> > > Hey folks,
>> > >
>> > > Thanks a lot for reviewing the KIP and providing feedback.
>> > > The discussion thread seems resolved and KIP has been updated
>> > accordingly.
>> > > We will be starting the voting thread for this KIP in the next few
>> days.
>> > > Please take a look at the KIP and let us know if any further
>> discussion
>> > > is needed.
>> > >
>> > > Thanks and regards,
>> > > Vedarth
>> > >
>> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <ma...@gmail.com>
>> > > wrote:
>> > >
>> > > > Thanks Krish. KIP looks good to me.
>> > > >
>> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <kr...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > Hi Manikumar,
>> > > > >
>> > > > > Thanks for the comments.
>> > > > >
>> > > > > Maybe as part of the release process, RM can create a JIRA for
>> this
>> > > > > > task. This can be taken by RM or any comitter or any contributor
>> > > (with
>> > > > > > some help from commiters to run "Docker Image Preparation via
>> > GitHub
>> > > > > > Actions:"
>> > > > >
>> > > > > This sounds like a good idea. This step would be beneficial. By
>> > > creating
>> > > > a
>> > > > > JIRA ticket, it will also serve as a reminder to complete the
>> > > > post-release
>> > > > > steps for the Docker official images. Have updated the KIP with
>> this
>> > > > step.
>> > > > >
>> > > > > Is this using GitHub Actions workflow? or manual testing?
>> > > > >
>> > > > > This will be done by a Github Actions workflow, which will test
>> the
>> > > > static
>> > > > > Docker Official Image assets for a specific release version.
>> > > > >
>> > > > > Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
>> > > > > > official images repository (or) can it be done by any
>> contributor.
>> > > > >
>> > > > > I believe that it can be done by any contributor (ref: This link
>> > > > > <
>> > https://docs.docker.com/trusted-content/official-images/contributing/
>> > > >
>> > > > > quotes "*Anyone can provide feedback, contribute code, suggest
>> > process
>> > > > > changes, or even propose a new Official Image.*")
>> > > > >
>> > > > > Also I was thinking, once the KIP gets voted, we should try to
>> > release
>> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us
>> to
>> > > > > > validate the process and allow us to fix any changes suggested
>> by
>> > > > > > Dockerhub before the 3.8.0 release.
>> > > > >
>> > > > > This sounds like a great idea. This KIP proposes release of DOI
>> as a
>> > > > > post-release process, which can be done anytime post release.
>> Since
>> > > 3.7.0
>> > > > > is already released, we can perform these steps for that release
>> too.
>> > > By
>> > > > > the time the KIP gets implemented, if 3.7.1 is released, we could
>> do
>> > > > these
>> > > > > steps for 3.7.1, instead of 3.7.0. This would allow us to make
>> > changes
>> > > to
>> > > > > the Dockerfiles and other assets based on feedback from Docker Hub
>> > > before
>> > > > > the release of version 3.8.0.
>> > > > >
>> > > > > Thanks,
>> > > > > Krish.
>> > > > >
>> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
>> > manikumar.reddy@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Krish,
>> > > > > >
>> > > > > > Thanks for the updated KIP. a few comments below.
>> > > > > >
>> > > > > > > "These actions can be carried out by the RM or any contributor
>> > post
>> > > > the
>> > > > > > release process."
>> > > > > > Maybe as part of the release process, RM can create a JIRA for
>> this
>> > > > > > task. This can be taken by RM or any comitter or any contributor
>> > > (with
>> > > > > > some help from commiters to run "Docker Image Preparation via
>> > GitHub
>> > > > > > Actions:"
>> > > > > >
>> > > > > > > "Perform Docker build tests to ensure image integrity"
>> > > > > > Is this using GitHub Actions workflow? or manual testing?
>> > > > > >
>> > > > > > > "The RM will manually raise the final PR to Docker Hub’s
>> official
>> > > > images
>> > > > > > repository using the contents of the generated file"
>> > > > > >  Is it mandatory for RM/comitters to raise the PR to Docker
>> Hub’s
>> > > > > > official images repository (or) can it be done by any
>> contributor.
>> > > > > >
>> > > > > > Also I was thinking, once the KIP gets voted, we should try to
>> > > release
>> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us
>> to
>> > > > > > validate the process and allow us to fix any changes suggested
>> by
>> > > > > > Dockerhub before the 3.8.0 release.
>> > > > > >
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <
>> krishvora01@gmail.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > > Hi Manikumar and Luke.
>> > > > > > > Thanks for the questions.
>> > > > > > >
>> > > > > > > 1. No, the Docker inventory files and configurations will not
>> be
>> > > the
>> > > > same
>> > > > > > > for Open Source Software (OSS) Images and Docker Official
>> Images
>> > > > (DOI).
>> > > > > > >
>> > > > > > > For OSS images, the Dockerfile located in
>> docker/jvm/dockerfile
>> > is
>> > > > > > > utilized. This process is integrated with the existing release
>> > > > pipeline
>> > > > > > as
>> > > > > > > outlined in KIP-975
>> > > > > > > <
>> > > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
>> > > > > > >,
>> > > > > > > where the Kafka URL is provided as a build argument. This
>> method
>> > > > allows
>> > > > > > for
>> > > > > > > building, testing, and releasing OSS images dynamically. The
>> OSS
>> > > > images
>> > > > > > > will continue to be released under the standard release
>> process .
>> > > > > > >
>> > > > > > > In contrast, the release process for DOIs requires providing
>> the
>> > > > Docker
>> > > > > > Hub
>> > > > > > > team with a specific directory for each version release that
>> > > > contains a
>> > > > > > > standalone Dockerfile. These Dockerfiles are designed to be
>> > > > > > > self-sufficient, hence require hardcoded values instead of
>> > relying
>> > > on
>> > > > > > build
>> > > > > > > arguments. To accommodate this, in our proposed approach, a
>> new
>> > > > directory
>> > > > > > > named docker_official_images has been created. This directory
>> > > > contains
>> > > > > > > version-specific directories, having Dockerfiles with
>> hardcoded
>> > > > > > > configurations for each release, acting as the source of truth
>> > for
>> > > > DOI
>> > > > > > > releases. The hardcoded dockerfiles will be created using the
>> > > > > > > docker/jvm/dockerfile as a template. Thus, as part of post
>> > release
>> > > we
>> > > > > > will
>> > > > > > > be creating a Dockerfile that will be reviewed by the
>> Dockerhub
>> > > > community
>> > > > > > > and might need changes as per their review. This approach
>> ensures
>> > > > that
>> > > > > > DOIs
>> > > > > > > are built consistently and meet the specific requirements set
>> by
>> > > > Docker
>> > > > > > Hub.
>> > > > > > >
>> > > > > > > 2. Yes Manikumar, transitioning the release of Docker Official
>> > > Images
>> > > > > > (DOI)
>> > > > > > > to a post-release activity does address the concerns about
>> > > > complicating
>> > > > > > the
>> > > > > > > release process. Initially, we considered incorporating DOI
>> > release
>> > > > > > > directly into Kafka's release workflow. However, this approach
>> > > > > > > significantly increased the RMs workload due to the addition
>> of
>> > > > numerous
>> > > > > > > steps, complicating the process. By designating the DOI
>> release
>> > as
>> > > a
>> > > > > > > post-release task, we maintain the original release process.
>> This
>> > > > > > > adjustment allows for the DOI release to be done after the
>> main
>> > > > release.
>> > > > > > We
>> > > > > > > have revised the KIP to reflect that DOI releases will now
>> occur
>> > > > after
>> > > > > > the
>> > > > > > > main release phase. Please review the updated document and
>> > provide
>> > > > any
>> > > > > > > feedback you might have.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Krish.
>> > > > > > >
>> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <sh...@gmail.com>
>> > > wrote:
>> > > > > > >
>> > > > > > > > Hi Krishna,
>> > > > > > > >
>> > > > > > > > I also have the same question as Manikumar raised:
>> > > > > > > > 1. Will the Docker inventory files/etc are the same for OSS
>> > Image
>> > > > and
>> > > > > > > > Docker Official Images?
>> > > > > > > > If no, then why not? Could we make them identical so that we
>> > > don't
>> > > > > > have to
>> > > > > > > > build 2 images for each release?
>> > > > > > > >
>> > > > > > > > Thank you.
>> > > > > > > > Luke
>> > > > > > > >
>> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
>> > > > manikumar.reddy@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Krishna,
>> > > > > > > > >
>> > > > > > > > > Thanks for the KIP.
>> > > > > > > > >
>> > > > > > > > > I think Docker Official Images will be beneficial to the
>> > Kafka
>> > > > > > community.
>> > > > > > > > > Few queries below.
>> > > > > > > > >
>> > > > > > > > > 1. Will the Docker inventory files/etc are the same for
>> OSS
>> > > > Image and
>> > > > > > > > > Docker Official Images
>> > > > > > > > > 2. I am a bit worried about the new steps to the release
>> > > process.
>> > > > > > Maybe
>> > > > > > > > we
>> > > > > > > > > should consider Docker Official Images release as
>> > Post-Release
>> > > > > > activity.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
>> > > > krishvora01@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Hector,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for reaching out. This KIP builds on top of
>> KIP-975
>> > > > > > > > > > <
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
>> > > > > > > > > > >
>> > > > > > > > > > and
>> > > > > > > > > > aims to introduce a JVM-based Docker Official Image (DOI
>> > > > > > > > > > <
>> https://docs.docker.com/trusted-content/official-images/
>> > >)
>> > > > for
>> > > > > > Apache
>> > > > > > > > > > Kafka that will be visible under Docker Official Images
>> > > > > > > > > > <https://hub.docker.com/search?image_filter=official&q=
>> >.
>> > > Once
>> > > > > > > > > implemented
>> > > > > > > > > > for Apache Kafka, for each release, there will be one
>> more
>> > > > > > JVM-based
>> > > > > > > > > Docker
>> > > > > > > > > > image available to users.
>> > > > > > > > > >
>> > > > > > > > > > Currently, we already have an OSS sponsored image, which
>> > was
>> > > > > > introduced
>> > > > > > > > > via
>> > > > > > > > > > KIP-975 (apache/kafka <
>> > > > https://hub.docker.com/r/apache/kafka/tags
>> > > > > > >)
>> > > > > > > > > which
>> > > > > > > > > > comes under The Apache Software Foundation <
>> > > > > > > > > > https://hub.docker.com/u/apache> in
>> > > > > > > > > > Docker Hub. The new Docker Image is the Docker Official
>> > Image
>> > > > > > (DOI),
>> > > > > > > > > which
>> > > > > > > > > > will be built and maintained by Docker Community.
>> > > > > > > > > >
>> > > > > > > > > > For example, for a release version like 3.8.0 we will
>> have
>> > > two
>> > > > JVM
>> > > > > > > > based
>> > > > > > > > > > docker images:-
>> > > > > > > > > >
>> > > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
>> > > > > > > > > >    - kafka:3.8.0 (Docker Official image)
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > I have added the same in the KIP too for everyone's
>> > > reference.
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Krish.
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino
>> > (BLOOMBERG/
>> > > > 919
>> > > > > > 3RD
>> > > > > > > > A) <
>> > > > > > > > > > hgeraldino@bloomberg.net> wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi,
>> > > > > > > > > > >
>> > > > > > > > > > > What is the difference between this KIP and KIP-975:
>> > Docker
>> > > > > > Image for
>> > > > > > > > > > > Apache Kafka?
>> > > > > > > > > > >
>> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07
>> > > UTC-4:00To:
>> > > > > > > > > > > dev@kafka.apache.org
>> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image for
>> > > Apache
>> > > > > > Kafka
>> > > > > > > > > > >
>> > > > > > > > > > > Hi everyone,
>> > > > > > > > > > >
>> > > > > > > > > > > I would like to start the discussion on
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
>> > > > > > > > > > > age+for+Apache+Kafka
>> > > > > > > > > > >  .
>> > > > > > > > > > >
>> > > > > > > > > > > This KIP aims to introduce JVM based Docker Official
>> > Image
>> > > > (DOI)
>> > > > > > for
>> > > > > > > > > > Apache
>> > > > > > > > > > > Kafka.
>> > > > > > > > > > >
>> > > > > > > > > > > Regards,
>> > > > > > > > > > > Krish.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>>
>> [image: Confluent] <https://www.confluent.io>
>> Prabha Manepalli
>> Product Manager for Confluent Platform Security, Docker
>> linkedin.com/in/prabhamanepalli
>>
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Krish Vora <kr...@gmail.com>]

Hi Chris. Thanks for the questions.

3. Would a separate Docker-owned repository be out of the question? I'm
> guessing there are some trademark issues that might get in the way, but
> it's worth exploring since the entire purpose of this KIP seems to be to
> provide images that are vetted and designed by Docker more than by the
> Apache Kafka contributors/committers/PMC.



   - The process for introducing a Docker Official Image involves
      - Hosting the Dockerfile in the Apache Kafka repository and
      - Providing the path to this Dockerfile to Docker Hub in Docker Hub’s
      own repo
      <https://github.com/docker-library/official-images/tree/master/library>
      .
   - This ensures that any updates to the Dockerfile in the AK repository
   are directly applicable to the docker official images available on Docker
   Hub.


   - We also did not find any added advantage to create a separate
   repository named apache-docker within the Apache GitHub organization.

Thanks,
Krish.

On Wed, May 8, 2024 at 6:05 PM Prabha Manepalli
<mp...@confluent.io.invalid> wrote:

> Hi Chris,  I would like to add more context to this KIP's motivation.
> Vedarth and Krish, please weigh in with your inputs.
>
> In the motivation section it's stated that "Several other Apache projects,
> > like Flink, Spark, Solr, have already released Docker Official Images,
> with
> > download figures ranging from 50 million to over 1 billion. These numbers
> > highlight the significant demand among users." But then immediately
> > afterwards, we learn that "Also the Docker Official Images are always the
> > top 1 search result, irrespective of the number of downloads." Wouldn't a
> > high number of downloads for an image naturally follow from being the top
> > search result? It seems like we can't necessarily assume that Docker
> > Official Images are inherently more desirable for users based solely on
> > download statistics.
> >
>
> *My thoughts: *Unlike the Sponsored OSS image, the Docker Official image is
> more desirable for workloads that have stringent compliance requirements.
> More details on why official images are more trusted are documented here
> <https://docs.docker.com/trusted-content/official-images/>. The Docker
> Official image would also help an absolutely new Kafka beginner who might
> not know about Apache or the concept of Sponsored images. We want to make
> it easier for Kafka beginners to discover the Kafka image through
> DockerHub.
>
>
> Can you elaborate on the value that these new images would add from a
> > user's perspective? I'm hesitant to introduce another image, since it
> adds
> > to the cognitive burden of people who will inevitably have to answer the
> > question of "What are the differences between all of the available images
> > and which one is best for my use case?"
> >
>
>
> *My thoughts: *This is a valid concern to address. The response to the
> above question addresses the value-add this new Docker Official image would
> provide. I also agree we need a clear distinction between each of these
> images to be well documented. We plan to update the AK website with details
> on how, why, and when a developer would want to use each of these
> particular images(KIP-974,975,1028).
>
> Thanks,
> Prabha.
>
>
>
>
>
> On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton <ch...@aiven.io.invalid>
> wrote:
>
> > Hi Vedarth and Krish,
> >
> > Thanks for the KIP! I have to admit I'm a little skeptical; hopefully you
> > can help me understand the need for these additional images.
> >
> > 1) In the motivation section it's stated that "Several other Apache
> > projects, like Flink, Spark, Solr, have already released Docker Official
> > Images, with download figures ranging from 50 million to over 1 billion.
> > These numbers highlight the significant demand among users." But then
> > immediately afterwards, we learn that "Also the Docker Official Images
> are
> > always the top 1 search result, irrespective of the number of downloads."
> > Wouldn't a high number of downloads for an image naturally follow from
> > being the top search result? It seems like we can't necessarily assume
> that
> > Docker Official Images are inherently more desirable for users based
> solely
> > on download statistics.
> >
> > 2) Can you elaborate on the value that these new images would add from a
> > user's perspective? I'm hesitant to introduce another image, since it
> adds
> > to the cognitive burden of people who will inevitably have to answer the
> > question of "What are the differences between all of the available images
> > and which one is best for my use case?"
> >
> > 3) Would a separate Docker-owned repository be out of the question? I'm
> > guessing there are some trademark issues that might get in the way, but
> > it's worth exploring since the entire purpose of this KIP seems to be to
> > provide images that are vetted and designed by Docker more than by the
> > Apache Kafka contributors/committers/PMC.
> >
> > I may have more questions later but wanted to get this initial round out
> > now without trying to list everything first.
> >
> > Looking forward to your thoughts!
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <vedarth.sharma@gmail.com
> >
> > wrote:
> >
> > > Hey folks,
> > >
> > > Thanks a lot for reviewing the KIP and providing feedback.
> > > The discussion thread seems resolved and KIP has been updated
> > accordingly.
> > > We will be starting the voting thread for this KIP in the next few
> days.
> > > Please take a look at the KIP and let us know if any further discussion
> > > is needed.
> > >
> > > Thanks and regards,
> > > Vedarth
> > >
> > > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Krish. KIP looks good to me.
> > > >
> > > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <kr...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hi Manikumar,
> > > > >
> > > > > Thanks for the comments.
> > > > >
> > > > > Maybe as part of the release process, RM can create a JIRA for this
> > > > > > task. This can be taken by RM or any comitter or any contributor
> > > (with
> > > > > > some help from commiters to run "Docker Image Preparation via
> > GitHub
> > > > > > Actions:"
> > > > >
> > > > > This sounds like a good idea. This step would be beneficial. By
> > > creating
> > > > a
> > > > > JIRA ticket, it will also serve as a reminder to complete the
> > > > post-release
> > > > > steps for the Docker official images. Have updated the KIP with
> this
> > > > step.
> > > > >
> > > > > Is this using GitHub Actions workflow? or manual testing?
> > > > >
> > > > > This will be done by a Github Actions workflow, which will test the
> > > > static
> > > > > Docker Official Image assets for a specific release version.
> > > > >
> > > > > Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > > > > > official images repository (or) can it be done by any
> contributor.
> > > > >
> > > > > I believe that it can be done by any contributor (ref: This link
> > > > > <
> > https://docs.docker.com/trusted-content/official-images/contributing/
> > > >
> > > > > quotes "*Anyone can provide feedback, contribute code, suggest
> > process
> > > > > changes, or even propose a new Official Image.*")
> > > > >
> > > > > Also I was thinking, once the KIP gets voted, we should try to
> > release
> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us
> to
> > > > > > validate the process and allow us to fix any changes suggested by
> > > > > > Dockerhub before the 3.8.0 release.
> > > > >
> > > > > This sounds like a great idea. This KIP proposes release of DOI as
> a
> > > > > post-release process, which can be done anytime post release. Since
> > > 3.7.0
> > > > > is already released, we can perform these steps for that release
> too.
> > > By
> > > > > the time the KIP gets implemented, if 3.7.1 is released, we could
> do
> > > > these
> > > > > steps for 3.7.1, instead of 3.7.0. This would allow us to make
> > changes
> > > to
> > > > > the Dockerfiles and other assets based on feedback from Docker Hub
> > > before
> > > > > the release of version 3.8.0.
> > > > >
> > > > > Thanks,
> > > > > Krish.
> > > > >
> > > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
> > manikumar.reddy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Krish,
> > > > > >
> > > > > > Thanks for the updated KIP. a few comments below.
> > > > > >
> > > > > > > "These actions can be carried out by the RM or any contributor
> > post
> > > > the
> > > > > > release process."
> > > > > > Maybe as part of the release process, RM can create a JIRA for
> this
> > > > > > task. This can be taken by RM or any comitter or any contributor
> > > (with
> > > > > > some help from commiters to run "Docker Image Preparation via
> > GitHub
> > > > > > Actions:"
> > > > > >
> > > > > > > "Perform Docker build tests to ensure image integrity"
> > > > > > Is this using GitHub Actions workflow? or manual testing?
> > > > > >
> > > > > > > "The RM will manually raise the final PR to Docker Hub’s
> official
> > > > images
> > > > > > repository using the contents of the generated file"
> > > > > >  Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > > > > > official images repository (or) can it be done by any
> contributor.
> > > > > >
> > > > > > Also I was thinking, once the KIP gets voted, we should try to
> > > release
> > > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us
> to
> > > > > > validate the process and allow us to fix any changes suggested by
> > > > > > Dockerhub before the 3.8.0 release.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <krishvora01@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > Hi Manikumar and Luke.
> > > > > > > Thanks for the questions.
> > > > > > >
> > > > > > > 1. No, the Docker inventory files and configurations will not
> be
> > > the
> > > > same
> > > > > > > for Open Source Software (OSS) Images and Docker Official
> Images
> > > > (DOI).
> > > > > > >
> > > > > > > For OSS images, the Dockerfile located in docker/jvm/dockerfile
> > is
> > > > > > > utilized. This process is integrated with the existing release
> > > > pipeline
> > > > > > as
> > > > > > > outlined in KIP-975
> > > > > > > <
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > > > > > >,
> > > > > > > where the Kafka URL is provided as a build argument. This
> method
> > > > allows
> > > > > > for
> > > > > > > building, testing, and releasing OSS images dynamically. The
> OSS
> > > > images
> > > > > > > will continue to be released under the standard release
> process .
> > > > > > >
> > > > > > > In contrast, the release process for DOIs requires providing
> the
> > > > Docker
> > > > > > Hub
> > > > > > > team with a specific directory for each version release that
> > > > contains a
> > > > > > > standalone Dockerfile. These Dockerfiles are designed to be
> > > > > > > self-sufficient, hence require hardcoded values instead of
> > relying
> > > on
> > > > > > build
> > > > > > > arguments. To accommodate this, in our proposed approach, a new
> > > > directory
> > > > > > > named docker_official_images has been created. This directory
> > > > contains
> > > > > > > version-specific directories, having Dockerfiles with hardcoded
> > > > > > > configurations for each release, acting as the source of truth
> > for
> > > > DOI
> > > > > > > releases. The hardcoded dockerfiles will be created using the
> > > > > > > docker/jvm/dockerfile as a template. Thus, as part of post
> > release
> > > we
> > > > > > will
> > > > > > > be creating a Dockerfile that will be reviewed by the Dockerhub
> > > > community
> > > > > > > and might need changes as per their review. This approach
> ensures
> > > > that
> > > > > > DOIs
> > > > > > > are built consistently and meet the specific requirements set
> by
> > > > Docker
> > > > > > Hub.
> > > > > > >
> > > > > > > 2. Yes Manikumar, transitioning the release of Docker Official
> > > Images
> > > > > > (DOI)
> > > > > > > to a post-release activity does address the concerns about
> > > > complicating
> > > > > > the
> > > > > > > release process. Initially, we considered incorporating DOI
> > release
> > > > > > > directly into Kafka's release workflow. However, this approach
> > > > > > > significantly increased the RMs workload due to the addition of
> > > > numerous
> > > > > > > steps, complicating the process. By designating the DOI release
> > as
> > > a
> > > > > > > post-release task, we maintain the original release process.
> This
> > > > > > > adjustment allows for the DOI release to be done after the main
> > > > release.
> > > > > > We
> > > > > > > have revised the KIP to reflect that DOI releases will now
> occur
> > > > after
> > > > > > the
> > > > > > > main release phase. Please review the updated document and
> > provide
> > > > any
> > > > > > > feedback you might have.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Krish.
> > > > > > >
> > > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <sh...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Krishna,
> > > > > > > >
> > > > > > > > I also have the same question as Manikumar raised:
> > > > > > > > 1. Will the Docker inventory files/etc are the same for OSS
> > Image
> > > > and
> > > > > > > > Docker Official Images?
> > > > > > > > If no, then why not? Could we make them identical so that we
> > > don't
> > > > > > have to
> > > > > > > > build 2 images for each release?
> > > > > > > >
> > > > > > > > Thank you.
> > > > > > > > Luke
> > > > > > > >
> > > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> > > > manikumar.reddy@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Krishna,
> > > > > > > > >
> > > > > > > > > Thanks for the KIP.
> > > > > > > > >
> > > > > > > > > I think Docker Official Images will be beneficial to the
> > Kafka
> > > > > > community.
> > > > > > > > > Few queries below.
> > > > > > > > >
> > > > > > > > > 1. Will the Docker inventory files/etc are the same for OSS
> > > > Image and
> > > > > > > > > Docker Official Images
> > > > > > > > > 2. I am a bit worried about the new steps to the release
> > > process.
> > > > > > Maybe
> > > > > > > > we
> > > > > > > > > should consider Docker Official Images release as
> > Post-Release
> > > > > > activity.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> > > > krishvora01@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Hector,
> > > > > > > > > >
> > > > > > > > > > Thanks for reaching out. This KIP builds on top of
> KIP-975
> > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > > > > > > >
> > > > > > > > > > and
> > > > > > > > > > aims to introduce a JVM-based Docker Official Image (DOI
> > > > > > > > > > <
> https://docs.docker.com/trusted-content/official-images/
> > >)
> > > > for
> > > > > > Apache
> > > > > > > > > > Kafka that will be visible under Docker Official Images
> > > > > > > > > > <https://hub.docker.com/search?image_filter=official&q=
> >.
> > > Once
> > > > > > > > > implemented
> > > > > > > > > > for Apache Kafka, for each release, there will be one
> more
> > > > > > JVM-based
> > > > > > > > > Docker
> > > > > > > > > > image available to users.
> > > > > > > > > >
> > > > > > > > > > Currently, we already have an OSS sponsored image, which
> > was
> > > > > > introduced
> > > > > > > > > via
> > > > > > > > > > KIP-975 (apache/kafka <
> > > > https://hub.docker.com/r/apache/kafka/tags
> > > > > > >)
> > > > > > > > > which
> > > > > > > > > > comes under The Apache Software Foundation <
> > > > > > > > > > https://hub.docker.com/u/apache> in
> > > > > > > > > > Docker Hub. The new Docker Image is the Docker Official
> > Image
> > > > > > (DOI),
> > > > > > > > > which
> > > > > > > > > > will be built and maintained by Docker Community.
> > > > > > > > > >
> > > > > > > > > > For example, for a release version like 3.8.0 we will
> have
> > > two
> > > > JVM
> > > > > > > > based
> > > > > > > > > > docker images:-
> > > > > > > > > >
> > > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > > > > > > > >    - kafka:3.8.0 (Docker Official image)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I have added the same in the KIP too for everyone's
> > > reference.
> > > > > > > > > > Thanks,
> > > > > > > > > > Krish.
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino
> > (BLOOMBERG/
> > > > 919
> > > > > > 3RD
> > > > > > > > A) <
> > > > > > > > > > hgeraldino@bloomberg.net> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > What is the difference between this KIP and KIP-975:
> > Docker
> > > > > > Image for
> > > > > > > > > > > Apache Kafka?
> > > > > > > > > > >
> > > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07
> > > UTC-4:00To:
> > > > > > > > > > > dev@kafka.apache.org
> > > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image for
> > > Apache
> > > > > > Kafka
> > > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > I would like to start the discussion on
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > > > > > > > age+for+Apache+Kafka
> > > > > > > > > > >  .
> > > > > > > > > > >
> > > > > > > > > > > This KIP aims to introduce JVM based Docker Official
> > Image
> > > > (DOI)
> > > > > > for
> > > > > > > > > > Apache
> > > > > > > > > > > Kafka.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Krish.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> >
>
>
> --
>
> [image: Confluent] <https://www.confluent.io>
> Prabha Manepalli
> Product Manager for Confluent Platform Security, Docker
> linkedin.com/in/prabhamanepalli
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Prabha Manepalli <mp...@confluent.io.INVALID>]

Hi Chris,  I would like to add more context to this KIP's motivation.
Vedarth and Krish, please weigh in with your inputs.

In the motivation section it's stated that "Several other Apache projects,
> like Flink, Spark, Solr, have already released Docker Official Images, with
> download figures ranging from 50 million to over 1 billion. These numbers
> highlight the significant demand among users." But then immediately
> afterwards, we learn that "Also the Docker Official Images are always the
> top 1 search result, irrespective of the number of downloads." Wouldn't a
> high number of downloads for an image naturally follow from being the top
> search result? It seems like we can't necessarily assume that Docker
> Official Images are inherently more desirable for users based solely on
> download statistics.
>

*My thoughts: *Unlike the Sponsored OSS image, the Docker Official image is
more desirable for workloads that have stringent compliance requirements.
More details on why official images are more trusted are documented here
<https://docs.docker.com/trusted-content/official-images/>. The Docker
Official image would also help an absolutely new Kafka beginner who might
not know about Apache or the concept of Sponsored images. We want to make
it easier for Kafka beginners to discover the Kafka image through DockerHub.


Can you elaborate on the value that these new images would add from a
> user's perspective? I'm hesitant to introduce another image, since it adds
> to the cognitive burden of people who will inevitably have to answer the
> question of "What are the differences between all of the available images
> and which one is best for my use case?"
>


*My thoughts: *This is a valid concern to address. The response to the
above question addresses the value-add this new Docker Official image would
provide. I also agree we need a clear distinction between each of these
images to be well documented. We plan to update the AK website with details
on how, why, and when a developer would want to use each of these
particular images(KIP-974,975,1028).

Thanks,
Prabha.





On Tue, Apr 30, 2024 at 9:41 PM Chris Egerton <ch...@aiven.io.invalid>
wrote:

> Hi Vedarth and Krish,
>
> Thanks for the KIP! I have to admit I'm a little skeptical; hopefully you
> can help me understand the need for these additional images.
>
> 1) In the motivation section it's stated that "Several other Apache
> projects, like Flink, Spark, Solr, have already released Docker Official
> Images, with download figures ranging from 50 million to over 1 billion.
> These numbers highlight the significant demand among users." But then
> immediately afterwards, we learn that "Also the Docker Official Images are
> always the top 1 search result, irrespective of the number of downloads."
> Wouldn't a high number of downloads for an image naturally follow from
> being the top search result? It seems like we can't necessarily assume that
> Docker Official Images are inherently more desirable for users based solely
> on download statistics.
>
> 2) Can you elaborate on the value that these new images would add from a
> user's perspective? I'm hesitant to introduce another image, since it adds
> to the cognitive burden of people who will inevitably have to answer the
> question of "What are the differences between all of the available images
> and which one is best for my use case?"
>
> 3) Would a separate Docker-owned repository be out of the question? I'm
> guessing there are some trademark issues that might get in the way, but
> it's worth exploring since the entire purpose of this KIP seems to be to
> provide images that are vetted and designed by Docker more than by the
> Apache Kafka contributors/committers/PMC.
>
> I may have more questions later but wanted to get this initial round out
> now without trying to list everything first.
>
> Looking forward to your thoughts!
>
> Cheers,
>
> Chris
>
> On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <ve...@gmail.com>
> wrote:
>
> > Hey folks,
> >
> > Thanks a lot for reviewing the KIP and providing feedback.
> > The discussion thread seems resolved and KIP has been updated
> accordingly.
> > We will be starting the voting thread for this KIP in the next few days.
> > Please take a look at the KIP and let us know if any further discussion
> > is needed.
> >
> > Thanks and regards,
> > Vedarth
> >
> > On Fri, Apr 19, 2024 at 1:33 PM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Thanks Krish. KIP looks good to me.
> > >
> > > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <kr...@gmail.com>
> > wrote:
> > > >
> > > > Hi Manikumar,
> > > >
> > > > Thanks for the comments.
> > > >
> > > > Maybe as part of the release process, RM can create a JIRA for this
> > > > > task. This can be taken by RM or any comitter or any contributor
> > (with
> > > > > some help from commiters to run "Docker Image Preparation via
> GitHub
> > > > > Actions:"
> > > >
> > > > This sounds like a good idea. This step would be beneficial. By
> > creating
> > > a
> > > > JIRA ticket, it will also serve as a reminder to complete the
> > > post-release
> > > > steps for the Docker official images. Have updated the KIP with this
> > > step.
> > > >
> > > > Is this using GitHub Actions workflow? or manual testing?
> > > >
> > > > This will be done by a Github Actions workflow, which will test the
> > > static
> > > > Docker Official Image assets for a specific release version.
> > > >
> > > > Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > > > > official images repository (or) can it be done by any contributor.
> > > >
> > > > I believe that it can be done by any contributor (ref: This link
> > > > <
> https://docs.docker.com/trusted-content/official-images/contributing/
> > >
> > > > quotes "*Anyone can provide feedback, contribute code, suggest
> process
> > > > changes, or even propose a new Official Image.*")
> > > >
> > > > Also I was thinking, once the KIP gets voted, we should try to
> release
> > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> > > > > validate the process and allow us to fix any changes suggested by
> > > > > Dockerhub before the 3.8.0 release.
> > > >
> > > > This sounds like a great idea. This KIP proposes release of DOI as a
> > > > post-release process, which can be done anytime post release. Since
> > 3.7.0
> > > > is already released, we can perform these steps for that release too.
> > By
> > > > the time the KIP gets implemented, if 3.7.1 is released, we could do
> > > these
> > > > steps for 3.7.1, instead of 3.7.0. This would allow us to make
> changes
> > to
> > > > the Dockerfiles and other assets based on feedback from Docker Hub
> > before
> > > > the release of version 3.8.0.
> > > >
> > > > Thanks,
> > > > Krish.
> > > >
> > > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <
> manikumar.reddy@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Krish,
> > > > >
> > > > > Thanks for the updated KIP. a few comments below.
> > > > >
> > > > > > "These actions can be carried out by the RM or any contributor
> post
> > > the
> > > > > release process."
> > > > > Maybe as part of the release process, RM can create a JIRA for this
> > > > > task. This can be taken by RM or any comitter or any contributor
> > (with
> > > > > some help from commiters to run "Docker Image Preparation via
> GitHub
> > > > > Actions:"
> > > > >
> > > > > > "Perform Docker build tests to ensure image integrity"
> > > > > Is this using GitHub Actions workflow? or manual testing?
> > > > >
> > > > > > "The RM will manually raise the final PR to Docker Hub’s official
> > > images
> > > > > repository using the contents of the generated file"
> > > > >  Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > > > > official images repository (or) can it be done by any contributor.
> > > > >
> > > > > Also I was thinking, once the KIP gets voted, we should try to
> > release
> > > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> > > > > validate the process and allow us to fix any changes suggested by
> > > > > Dockerhub before the 3.8.0 release.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <kr...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi Manikumar and Luke.
> > > > > > Thanks for the questions.
> > > > > >
> > > > > > 1. No, the Docker inventory files and configurations will not be
> > the
> > > same
> > > > > > for Open Source Software (OSS) Images and Docker Official Images
> > > (DOI).
> > > > > >
> > > > > > For OSS images, the Dockerfile located in docker/jvm/dockerfile
> is
> > > > > > utilized. This process is integrated with the existing release
> > > pipeline
> > > > > as
> > > > > > outlined in KIP-975
> > > > > > <
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > > > > >,
> > > > > > where the Kafka URL is provided as a build argument. This method
> > > allows
> > > > > for
> > > > > > building, testing, and releasing OSS images dynamically. The OSS
> > > images
> > > > > > will continue to be released under the standard release process .
> > > > > >
> > > > > > In contrast, the release process for DOIs requires providing the
> > > Docker
> > > > > Hub
> > > > > > team with a specific directory for each version release that
> > > contains a
> > > > > > standalone Dockerfile. These Dockerfiles are designed to be
> > > > > > self-sufficient, hence require hardcoded values instead of
> relying
> > on
> > > > > build
> > > > > > arguments. To accommodate this, in our proposed approach, a new
> > > directory
> > > > > > named docker_official_images has been created. This directory
> > > contains
> > > > > > version-specific directories, having Dockerfiles with hardcoded
> > > > > > configurations for each release, acting as the source of truth
> for
> > > DOI
> > > > > > releases. The hardcoded dockerfiles will be created using the
> > > > > > docker/jvm/dockerfile as a template. Thus, as part of post
> release
> > we
> > > > > will
> > > > > > be creating a Dockerfile that will be reviewed by the Dockerhub
> > > community
> > > > > > and might need changes as per their review. This approach ensures
> > > that
> > > > > DOIs
> > > > > > are built consistently and meet the specific requirements set by
> > > Docker
> > > > > Hub.
> > > > > >
> > > > > > 2. Yes Manikumar, transitioning the release of Docker Official
> > Images
> > > > > (DOI)
> > > > > > to a post-release activity does address the concerns about
> > > complicating
> > > > > the
> > > > > > release process. Initially, we considered incorporating DOI
> release
> > > > > > directly into Kafka's release workflow. However, this approach
> > > > > > significantly increased the RMs workload due to the addition of
> > > numerous
> > > > > > steps, complicating the process. By designating the DOI release
> as
> > a
> > > > > > post-release task, we maintain the original release process. This
> > > > > > adjustment allows for the DOI release to be done after the main
> > > release.
> > > > > We
> > > > > > have revised the KIP to reflect that DOI releases will now occur
> > > after
> > > > > the
> > > > > > main release phase. Please review the updated document and
> provide
> > > any
> > > > > > feedback you might have.
> > > > > >
> > > > > > Thanks,
> > > > > > Krish.
> > > > > >
> > > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <sh...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Krishna,
> > > > > > >
> > > > > > > I also have the same question as Manikumar raised:
> > > > > > > 1. Will the Docker inventory files/etc are the same for OSS
> Image
> > > and
> > > > > > > Docker Official Images?
> > > > > > > If no, then why not? Could we make them identical so that we
> > don't
> > > > > have to
> > > > > > > build 2 images for each release?
> > > > > > >
> > > > > > > Thank you.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> > > manikumar.reddy@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Krishna,
> > > > > > > >
> > > > > > > > Thanks for the KIP.
> > > > > > > >
> > > > > > > > I think Docker Official Images will be beneficial to the
> Kafka
> > > > > community.
> > > > > > > > Few queries below.
> > > > > > > >
> > > > > > > > 1. Will the Docker inventory files/etc are the same for OSS
> > > Image and
> > > > > > > > Docker Official Images
> > > > > > > > 2. I am a bit worried about the new steps to the release
> > process.
> > > > > Maybe
> > > > > > > we
> > > > > > > > should consider Docker Official Images release as
> Post-Release
> > > > > activity.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> > > krishvora01@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Hector,
> > > > > > > > >
> > > > > > > > > Thanks for reaching out. This KIP builds on top of KIP-975
> > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > > > > > >
> > > > > > > > > and
> > > > > > > > > aims to introduce a JVM-based Docker Official Image (DOI
> > > > > > > > > <https://docs.docker.com/trusted-content/official-images/
> >)
> > > for
> > > > > Apache
> > > > > > > > > Kafka that will be visible under Docker Official Images
> > > > > > > > > <https://hub.docker.com/search?image_filter=official&q=>.
> > Once
> > > > > > > > implemented
> > > > > > > > > for Apache Kafka, for each release, there will be one more
> > > > > JVM-based
> > > > > > > > Docker
> > > > > > > > > image available to users.
> > > > > > > > >
> > > > > > > > > Currently, we already have an OSS sponsored image, which
> was
> > > > > introduced
> > > > > > > > via
> > > > > > > > > KIP-975 (apache/kafka <
> > > https://hub.docker.com/r/apache/kafka/tags
> > > > > >)
> > > > > > > > which
> > > > > > > > > comes under The Apache Software Foundation <
> > > > > > > > > https://hub.docker.com/u/apache> in
> > > > > > > > > Docker Hub. The new Docker Image is the Docker Official
> Image
> > > > > (DOI),
> > > > > > > > which
> > > > > > > > > will be built and maintained by Docker Community.
> > > > > > > > >
> > > > > > > > > For example, for a release version like 3.8.0 we will have
> > two
> > > JVM
> > > > > > > based
> > > > > > > > > docker images:-
> > > > > > > > >
> > > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > > > > > > >    - kafka:3.8.0 (Docker Official image)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I have added the same in the KIP too for everyone's
> > reference.
> > > > > > > > > Thanks,
> > > > > > > > > Krish.
> > > > > > > > >
> > > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino
> (BLOOMBERG/
> > > 919
> > > > > 3RD
> > > > > > > A) <
> > > > > > > > > hgeraldino@bloomberg.net> wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > What is the difference between this KIP and KIP-975:
> Docker
> > > > > Image for
> > > > > > > > > > Apache Kafka?
> > > > > > > > > >
> > > > > > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07
> > UTC-4:00To:
> > > > > > > > > > dev@kafka.apache.org
> > > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image for
> > Apache
> > > > > Kafka
> > > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > I would like to start the discussion on
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > > > > > > age+for+Apache+Kafka
> > > > > > > > > >  .
> > > > > > > > > >
> > > > > > > > > > This KIP aims to introduce JVM based Docker Official
> Image
> > > (DOI)
> > > > > for
> > > > > > > > > Apache
> > > > > > > > > > Kafka.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Krish.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
>


-- 

[image: Confluent] <https://www.confluent.io>
Prabha Manepalli
Product Manager for Confluent Platform Security, Docker
linkedin.com/in/prabhamanepalli

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Chris Egerton <ch...@aiven.io.INVALID>]

Hi Vedarth and Krish,

Thanks for the KIP! I have to admit I'm a little skeptical; hopefully you
can help me understand the need for these additional images.

1) In the motivation section it's stated that "Several other Apache
projects, like Flink, Spark, Solr, have already released Docker Official
Images, with download figures ranging from 50 million to over 1 billion.
These numbers highlight the significant demand among users." But then
immediately afterwards, we learn that "Also the Docker Official Images are
always the top 1 search result, irrespective of the number of downloads."
Wouldn't a high number of downloads for an image naturally follow from
being the top search result? It seems like we can't necessarily assume that
Docker Official Images are inherently more desirable for users based solely
on download statistics.

2) Can you elaborate on the value that these new images would add from a
user's perspective? I'm hesitant to introduce another image, since it adds
to the cognitive burden of people who will inevitably have to answer the
question of "What are the differences between all of the available images
and which one is best for my use case?"

3) Would a separate Docker-owned repository be out of the question? I'm
guessing there are some trademark issues that might get in the way, but
it's worth exploring since the entire purpose of this KIP seems to be to
provide images that are vetted and designed by Docker more than by the
Apache Kafka contributors/committers/PMC.

I may have more questions later but wanted to get this initial round out
now without trying to list everything first.

Looking forward to your thoughts!

Cheers,

Chris

On Mon, Apr 22, 2024 at 2:14 PM Vedarth Sharma <ve...@gmail.com>
wrote:

> Hey folks,
>
> Thanks a lot for reviewing the KIP and providing feedback.
> The discussion thread seems resolved and KIP has been updated accordingly.
> We will be starting the voting thread for this KIP in the next few days.
> Please take a look at the KIP and let us know if any further discussion
> is needed.
>
> Thanks and regards,
> Vedarth
>
> On Fri, Apr 19, 2024 at 1:33 PM Manikumar <ma...@gmail.com>
> wrote:
>
> > Thanks Krish. KIP looks good to me.
> >
> > On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <kr...@gmail.com>
> wrote:
> > >
> > > Hi Manikumar,
> > >
> > > Thanks for the comments.
> > >
> > > Maybe as part of the release process, RM can create a JIRA for this
> > > > task. This can be taken by RM or any comitter or any contributor
> (with
> > > > some help from commiters to run "Docker Image Preparation via GitHub
> > > > Actions:"
> > >
> > > This sounds like a good idea. This step would be beneficial. By
> creating
> > a
> > > JIRA ticket, it will also serve as a reminder to complete the
> > post-release
> > > steps for the Docker official images. Have updated the KIP with this
> > step.
> > >
> > > Is this using GitHub Actions workflow? or manual testing?
> > >
> > > This will be done by a Github Actions workflow, which will test the
> > static
> > > Docker Official Image assets for a specific release version.
> > >
> > > Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > > > official images repository (or) can it be done by any contributor.
> > >
> > > I believe that it can be done by any contributor (ref: This link
> > > <https://docs.docker.com/trusted-content/official-images/contributing/
> >
> > > quotes "*Anyone can provide feedback, contribute code, suggest process
> > > changes, or even propose a new Official Image.*")
> > >
> > > Also I was thinking, once the KIP gets voted, we should try to release
> > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> > > > validate the process and allow us to fix any changes suggested by
> > > > Dockerhub before the 3.8.0 release.
> > >
> > > This sounds like a great idea. This KIP proposes release of DOI as a
> > > post-release process, which can be done anytime post release. Since
> 3.7.0
> > > is already released, we can perform these steps for that release too.
> By
> > > the time the KIP gets implemented, if 3.7.1 is released, we could do
> > these
> > > steps for 3.7.1, instead of 3.7.0. This would allow us to make changes
> to
> > > the Dockerfiles and other assets based on feedback from Docker Hub
> before
> > > the release of version 3.8.0.
> > >
> > > Thanks,
> > > Krish.
> > >
> > > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi Krish,
> > > >
> > > > Thanks for the updated KIP. a few comments below.
> > > >
> > > > > "These actions can be carried out by the RM or any contributor post
> > the
> > > > release process."
> > > > Maybe as part of the release process, RM can create a JIRA for this
> > > > task. This can be taken by RM or any comitter or any contributor
> (with
> > > > some help from commiters to run "Docker Image Preparation via GitHub
> > > > Actions:"
> > > >
> > > > > "Perform Docker build tests to ensure image integrity"
> > > > Is this using GitHub Actions workflow? or manual testing?
> > > >
> > > > > "The RM will manually raise the final PR to Docker Hub’s official
> > images
> > > > repository using the contents of the generated file"
> > > >  Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > > > official images repository (or) can it be done by any contributor.
> > > >
> > > > Also I was thinking, once the KIP gets voted, we should try to
> release
> > > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> > > > validate the process and allow us to fix any changes suggested by
> > > > Dockerhub before the 3.8.0 release.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <kr...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi Manikumar and Luke.
> > > > > Thanks for the questions.
> > > > >
> > > > > 1. No, the Docker inventory files and configurations will not be
> the
> > same
> > > > > for Open Source Software (OSS) Images and Docker Official Images
> > (DOI).
> > > > >
> > > > > For OSS images, the Dockerfile located in docker/jvm/dockerfile is
> > > > > utilized. This process is integrated with the existing release
> > pipeline
> > > > as
> > > > > outlined in KIP-975
> > > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > > > >,
> > > > > where the Kafka URL is provided as a build argument. This method
> > allows
> > > > for
> > > > > building, testing, and releasing OSS images dynamically. The OSS
> > images
> > > > > will continue to be released under the standard release process .
> > > > >
> > > > > In contrast, the release process for DOIs requires providing the
> > Docker
> > > > Hub
> > > > > team with a specific directory for each version release that
> > contains a
> > > > > standalone Dockerfile. These Dockerfiles are designed to be
> > > > > self-sufficient, hence require hardcoded values instead of relying
> on
> > > > build
> > > > > arguments. To accommodate this, in our proposed approach, a new
> > directory
> > > > > named docker_official_images has been created. This directory
> > contains
> > > > > version-specific directories, having Dockerfiles with hardcoded
> > > > > configurations for each release, acting as the source of truth for
> > DOI
> > > > > releases. The hardcoded dockerfiles will be created using the
> > > > > docker/jvm/dockerfile as a template. Thus, as part of post release
> we
> > > > will
> > > > > be creating a Dockerfile that will be reviewed by the Dockerhub
> > community
> > > > > and might need changes as per their review. This approach ensures
> > that
> > > > DOIs
> > > > > are built consistently and meet the specific requirements set by
> > Docker
> > > > Hub.
> > > > >
> > > > > 2. Yes Manikumar, transitioning the release of Docker Official
> Images
> > > > (DOI)
> > > > > to a post-release activity does address the concerns about
> > complicating
> > > > the
> > > > > release process. Initially, we considered incorporating DOI release
> > > > > directly into Kafka's release workflow. However, this approach
> > > > > significantly increased the RMs workload due to the addition of
> > numerous
> > > > > steps, complicating the process. By designating the DOI release as
> a
> > > > > post-release task, we maintain the original release process. This
> > > > > adjustment allows for the DOI release to be done after the main
> > release.
> > > > We
> > > > > have revised the KIP to reflect that DOI releases will now occur
> > after
> > > > the
> > > > > main release phase. Please review the updated document and provide
> > any
> > > > > feedback you might have.
> > > > >
> > > > > Thanks,
> > > > > Krish.
> > > > >
> > > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <sh...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Krishna,
> > > > > >
> > > > > > I also have the same question as Manikumar raised:
> > > > > > 1. Will the Docker inventory files/etc are the same for OSS Image
> > and
> > > > > > Docker Official Images?
> > > > > > If no, then why not? Could we make them identical so that we
> don't
> > > > have to
> > > > > > build 2 images for each release?
> > > > > >
> > > > > > Thank you.
> > > > > > Luke
> > > > > >
> > > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> > manikumar.reddy@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Krishna,
> > > > > > >
> > > > > > > Thanks for the KIP.
> > > > > > >
> > > > > > > I think Docker Official Images will be beneficial to the Kafka
> > > > community.
> > > > > > > Few queries below.
> > > > > > >
> > > > > > > 1. Will the Docker inventory files/etc are the same for OSS
> > Image and
> > > > > > > Docker Official Images
> > > > > > > 2. I am a bit worried about the new steps to the release
> process.
> > > > Maybe
> > > > > > we
> > > > > > > should consider Docker Official Images release as Post-Release
> > > > activity.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> > krishvora01@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Hector,
> > > > > > > >
> > > > > > > > Thanks for reaching out. This KIP builds on top of KIP-975
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > > > > >
> > > > > > > > and
> > > > > > > > aims to introduce a JVM-based Docker Official Image (DOI
> > > > > > > > <https://docs.docker.com/trusted-content/official-images/>)
> > for
> > > > Apache
> > > > > > > > Kafka that will be visible under Docker Official Images
> > > > > > > > <https://hub.docker.com/search?image_filter=official&q=>.
> Once
> > > > > > > implemented
> > > > > > > > for Apache Kafka, for each release, there will be one more
> > > > JVM-based
> > > > > > > Docker
> > > > > > > > image available to users.
> > > > > > > >
> > > > > > > > Currently, we already have an OSS sponsored image, which was
> > > > introduced
> > > > > > > via
> > > > > > > > KIP-975 (apache/kafka <
> > https://hub.docker.com/r/apache/kafka/tags
> > > > >)
> > > > > > > which
> > > > > > > > comes under The Apache Software Foundation <
> > > > > > > > https://hub.docker.com/u/apache> in
> > > > > > > > Docker Hub. The new Docker Image is the Docker Official Image
> > > > (DOI),
> > > > > > > which
> > > > > > > > will be built and maintained by Docker Community.
> > > > > > > >
> > > > > > > > For example, for a release version like 3.8.0 we will have
> two
> > JVM
> > > > > > based
> > > > > > > > docker images:-
> > > > > > > >
> > > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > > > > > >    - kafka:3.8.0 (Docker Official image)
> > > > > > > >
> > > > > > > >
> > > > > > > > I have added the same in the KIP too for everyone's
> reference.
> > > > > > > > Thanks,
> > > > > > > > Krish.
> > > > > > > >
> > > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino (BLOOMBERG/
> > 919
> > > > 3RD
> > > > > > A) <
> > > > > > > > hgeraldino@bloomberg.net> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > What is the difference between this KIP and KIP-975: Docker
> > > > Image for
> > > > > > > > > Apache Kafka?
> > > > > > > > >
> > > > > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07
> UTC-4:00To:
> > > > > > > > > dev@kafka.apache.org
> > > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image for
> Apache
> > > > Kafka
> > > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > I would like to start the discussion on
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > > > > > age+for+Apache+Kafka
> > > > > > > > >  .
> > > > > > > > >
> > > > > > > > > This KIP aims to introduce JVM based Docker Official Image
> > (DOI)
> > > > for
> > > > > > > > Apache
> > > > > > > > > Kafka.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Krish.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Vedarth Sharma <ve...@gmail.com>]

Hey folks,

Thanks a lot for reviewing the KIP and providing feedback.
The discussion thread seems resolved and KIP has been updated accordingly.
We will be starting the voting thread for this KIP in the next few days.
Please take a look at the KIP and let us know if any further discussion
is needed.

Thanks and regards,
Vedarth

On Fri, Apr 19, 2024 at 1:33 PM Manikumar <ma...@gmail.com> wrote:

> Thanks Krish. KIP looks good to me.
>
> On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <kr...@gmail.com> wrote:
> >
> > Hi Manikumar,
> >
> > Thanks for the comments.
> >
> > Maybe as part of the release process, RM can create a JIRA for this
> > > task. This can be taken by RM or any comitter or any contributor (with
> > > some help from commiters to run "Docker Image Preparation via GitHub
> > > Actions:"
> >
> > This sounds like a good idea. This step would be beneficial. By creating
> a
> > JIRA ticket, it will also serve as a reminder to complete the
> post-release
> > steps for the Docker official images. Have updated the KIP with this
> step.
> >
> > Is this using GitHub Actions workflow? or manual testing?
> >
> > This will be done by a Github Actions workflow, which will test the
> static
> > Docker Official Image assets for a specific release version.
> >
> > Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > > official images repository (or) can it be done by any contributor.
> >
> > I believe that it can be done by any contributor (ref: This link
> > <https://docs.docker.com/trusted-content/official-images/contributing/>
> > quotes "*Anyone can provide feedback, contribute code, suggest process
> > changes, or even propose a new Official Image.*")
> >
> > Also I was thinking, once the KIP gets voted, we should try to release
> > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> > > validate the process and allow us to fix any changes suggested by
> > > Dockerhub before the 3.8.0 release.
> >
> > This sounds like a great idea. This KIP proposes release of DOI as a
> > post-release process, which can be done anytime post release. Since 3.7.0
> > is already released, we can perform these steps for that release too. By
> > the time the KIP gets implemented, if 3.7.1 is released, we could do
> these
> > steps for 3.7.1, instead of 3.7.0. This would allow us to make changes to
> > the Dockerfiles and other assets based on feedback from Docker Hub before
> > the release of version 3.8.0.
> >
> > Thanks,
> > Krish.
> >
> > On Mon, Apr 15, 2024 at 12:59 PM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi Krish,
> > >
> > > Thanks for the updated KIP. a few comments below.
> > >
> > > > "These actions can be carried out by the RM or any contributor post
> the
> > > release process."
> > > Maybe as part of the release process, RM can create a JIRA for this
> > > task. This can be taken by RM or any comitter or any contributor (with
> > > some help from commiters to run "Docker Image Preparation via GitHub
> > > Actions:"
> > >
> > > > "Perform Docker build tests to ensure image integrity"
> > > Is this using GitHub Actions workflow? or manual testing?
> > >
> > > > "The RM will manually raise the final PR to Docker Hub’s official
> images
> > > repository using the contents of the generated file"
> > >  Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > > official images repository (or) can it be done by any contributor.
> > >
> > > Also I was thinking, once the KIP gets voted, we should try to release
> > > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> > > validate the process and allow us to fix any changes suggested by
> > > Dockerhub before the 3.8.0 release.
> > >
> > >
> > > Thanks,
> > >
> > > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <kr...@gmail.com>
> wrote:
> > > >
> > > > Hi Manikumar and Luke.
> > > > Thanks for the questions.
> > > >
> > > > 1. No, the Docker inventory files and configurations will not be the
> same
> > > > for Open Source Software (OSS) Images and Docker Official Images
> (DOI).
> > > >
> > > > For OSS images, the Dockerfile located in docker/jvm/dockerfile is
> > > > utilized. This process is integrated with the existing release
> pipeline
> > > as
> > > > outlined in KIP-975
> > > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > > >,
> > > > where the Kafka URL is provided as a build argument. This method
> allows
> > > for
> > > > building, testing, and releasing OSS images dynamically. The OSS
> images
> > > > will continue to be released under the standard release process .
> > > >
> > > > In contrast, the release process for DOIs requires providing the
> Docker
> > > Hub
> > > > team with a specific directory for each version release that
> contains a
> > > > standalone Dockerfile. These Dockerfiles are designed to be
> > > > self-sufficient, hence require hardcoded values instead of relying on
> > > build
> > > > arguments. To accommodate this, in our proposed approach, a new
> directory
> > > > named docker_official_images has been created. This directory
> contains
> > > > version-specific directories, having Dockerfiles with hardcoded
> > > > configurations for each release, acting as the source of truth for
> DOI
> > > > releases. The hardcoded dockerfiles will be created using the
> > > > docker/jvm/dockerfile as a template. Thus, as part of post release we
> > > will
> > > > be creating a Dockerfile that will be reviewed by the Dockerhub
> community
> > > > and might need changes as per their review. This approach ensures
> that
> > > DOIs
> > > > are built consistently and meet the specific requirements set by
> Docker
> > > Hub.
> > > >
> > > > 2. Yes Manikumar, transitioning the release of Docker Official Images
> > > (DOI)
> > > > to a post-release activity does address the concerns about
> complicating
> > > the
> > > > release process. Initially, we considered incorporating DOI release
> > > > directly into Kafka's release workflow. However, this approach
> > > > significantly increased the RMs workload due to the addition of
> numerous
> > > > steps, complicating the process. By designating the DOI release as a
> > > > post-release task, we maintain the original release process. This
> > > > adjustment allows for the DOI release to be done after the main
> release.
> > > We
> > > > have revised the KIP to reflect that DOI releases will now occur
> after
> > > the
> > > > main release phase. Please review the updated document and provide
> any
> > > > feedback you might have.
> > > >
> > > > Thanks,
> > > > Krish.
> > > >
> > > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <sh...@gmail.com> wrote:
> > > >
> > > > > Hi Krishna,
> > > > >
> > > > > I also have the same question as Manikumar raised:
> > > > > 1. Will the Docker inventory files/etc are the same for OSS Image
> and
> > > > > Docker Official Images?
> > > > > If no, then why not? Could we make them identical so that we don't
> > > have to
> > > > > build 2 images for each release?
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <
> manikumar.reddy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Krishna,
> > > > > >
> > > > > > Thanks for the KIP.
> > > > > >
> > > > > > I think Docker Official Images will be beneficial to the Kafka
> > > community.
> > > > > > Few queries below.
> > > > > >
> > > > > > 1. Will the Docker inventory files/etc are the same for OSS
> Image and
> > > > > > Docker Official Images
> > > > > > 2. I am a bit worried about the new steps to the release process.
> > > Maybe
> > > > > we
> > > > > > should consider Docker Official Images release as Post-Release
> > > activity.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <
> krishvora01@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Hector,
> > > > > > >
> > > > > > > Thanks for reaching out. This KIP builds on top of KIP-975
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > > > >
> > > > > > > and
> > > > > > > aims to introduce a JVM-based Docker Official Image (DOI
> > > > > > > <https://docs.docker.com/trusted-content/official-images/>)
> for
> > > Apache
> > > > > > > Kafka that will be visible under Docker Official Images
> > > > > > > <https://hub.docker.com/search?image_filter=official&q=>. Once
> > > > > > implemented
> > > > > > > for Apache Kafka, for each release, there will be one more
> > > JVM-based
> > > > > > Docker
> > > > > > > image available to users.
> > > > > > >
> > > > > > > Currently, we already have an OSS sponsored image, which was
> > > introduced
> > > > > > via
> > > > > > > KIP-975 (apache/kafka <
> https://hub.docker.com/r/apache/kafka/tags
> > > >)
> > > > > > which
> > > > > > > comes under The Apache Software Foundation <
> > > > > > > https://hub.docker.com/u/apache> in
> > > > > > > Docker Hub. The new Docker Image is the Docker Official Image
> > > (DOI),
> > > > > > which
> > > > > > > will be built and maintained by Docker Community.
> > > > > > >
> > > > > > > For example, for a release version like 3.8.0 we will have two
> JVM
> > > > > based
> > > > > > > docker images:-
> > > > > > >
> > > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > > > > >    - kafka:3.8.0 (Docker Official image)
> > > > > > >
> > > > > > >
> > > > > > > I have added the same in the KIP too for everyone's reference.
> > > > > > > Thanks,
> > > > > > > Krish.
> > > > > > >
> > > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino (BLOOMBERG/
> 919
> > > 3RD
> > > > > A) <
> > > > > > > hgeraldino@bloomberg.net> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > What is the difference between this KIP and KIP-975: Docker
> > > Image for
> > > > > > > > Apache Kafka?
> > > > > > > >
> > > > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:
> > > > > > > > dev@kafka.apache.org
> > > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache
> > > Kafka
> > > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I would like to start the discussion on
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > > > > age+for+Apache+Kafka
> > > > > > > >  .
> > > > > > > >
> > > > > > > > This KIP aims to introduce JVM based Docker Official Image
> (DOI)
> > > for
> > > > > > > Apache
> > > > > > > > Kafka.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Krish.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Manikumar <ma...@gmail.com>]

Thanks Krish. KIP looks good to me.

On Wed, Apr 17, 2024 at 1:38 PM Krish Vora <kr...@gmail.com> wrote:
>
> Hi Manikumar,
>
> Thanks for the comments.
>
> Maybe as part of the release process, RM can create a JIRA for this
> > task. This can be taken by RM or any comitter or any contributor (with
> > some help from commiters to run "Docker Image Preparation via GitHub
> > Actions:"
>
> This sounds like a good idea. This step would be beneficial. By creating a
> JIRA ticket, it will also serve as a reminder to complete the post-release
> steps for the Docker official images. Have updated the KIP with this step.
>
> Is this using GitHub Actions workflow? or manual testing?
>
> This will be done by a Github Actions workflow, which will test the static
> Docker Official Image assets for a specific release version.
>
> Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > official images repository (or) can it be done by any contributor.
>
> I believe that it can be done by any contributor (ref: This link
> <https://docs.docker.com/trusted-content/official-images/contributing/>
> quotes "*Anyone can provide feedback, contribute code, suggest process
> changes, or even propose a new Official Image.*")
>
> Also I was thinking, once the KIP gets voted, we should try to release
> > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> > validate the process and allow us to fix any changes suggested by
> > Dockerhub before the 3.8.0 release.
>
> This sounds like a great idea. This KIP proposes release of DOI as a
> post-release process, which can be done anytime post release. Since 3.7.0
> is already released, we can perform these steps for that release too. By
> the time the KIP gets implemented, if 3.7.1 is released, we could do these
> steps for 3.7.1, instead of 3.7.0. This would allow us to make changes to
> the Dockerfiles and other assets based on feedback from Docker Hub before
> the release of version 3.8.0.
>
> Thanks,
> Krish.
>
> On Mon, Apr 15, 2024 at 12:59 PM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi Krish,
> >
> > Thanks for the updated KIP. a few comments below.
> >
> > > "These actions can be carried out by the RM or any contributor post the
> > release process."
> > Maybe as part of the release process, RM can create a JIRA for this
> > task. This can be taken by RM or any comitter or any contributor (with
> > some help from commiters to run "Docker Image Preparation via GitHub
> > Actions:"
> >
> > > "Perform Docker build tests to ensure image integrity"
> > Is this using GitHub Actions workflow? or manual testing?
> >
> > > "The RM will manually raise the final PR to Docker Hub’s official images
> > repository using the contents of the generated file"
> >  Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> > official images repository (or) can it be done by any contributor.
> >
> > Also I was thinking, once the KIP gets voted, we should try to release
> > kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> > validate the process and allow us to fix any changes suggested by
> > Dockerhub before the 3.8.0 release.
> >
> >
> > Thanks,
> >
> > On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <kr...@gmail.com> wrote:
> > >
> > > Hi Manikumar and Luke.
> > > Thanks for the questions.
> > >
> > > 1. No, the Docker inventory files and configurations will not be the same
> > > for Open Source Software (OSS) Images and Docker Official Images (DOI).
> > >
> > > For OSS images, the Dockerfile located in docker/jvm/dockerfile is
> > > utilized. This process is integrated with the existing release pipeline
> > as
> > > outlined in KIP-975
> > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> > >,
> > > where the Kafka URL is provided as a build argument. This method allows
> > for
> > > building, testing, and releasing OSS images dynamically. The OSS images
> > > will continue to be released under the standard release process .
> > >
> > > In contrast, the release process for DOIs requires providing the Docker
> > Hub
> > > team with a specific directory for each version release that contains a
> > > standalone Dockerfile. These Dockerfiles are designed to be
> > > self-sufficient, hence require hardcoded values instead of relying on
> > build
> > > arguments. To accommodate this, in our proposed approach, a new directory
> > > named docker_official_images has been created. This directory contains
> > > version-specific directories, having Dockerfiles with hardcoded
> > > configurations for each release, acting as the source of truth for DOI
> > > releases. The hardcoded dockerfiles will be created using the
> > > docker/jvm/dockerfile as a template. Thus, as part of post release we
> > will
> > > be creating a Dockerfile that will be reviewed by the Dockerhub community
> > > and might need changes as per their review. This approach ensures that
> > DOIs
> > > are built consistently and meet the specific requirements set by Docker
> > Hub.
> > >
> > > 2. Yes Manikumar, transitioning the release of Docker Official Images
> > (DOI)
> > > to a post-release activity does address the concerns about complicating
> > the
> > > release process. Initially, we considered incorporating DOI release
> > > directly into Kafka's release workflow. However, this approach
> > > significantly increased the RMs workload due to the addition of numerous
> > > steps, complicating the process. By designating the DOI release as a
> > > post-release task, we maintain the original release process. This
> > > adjustment allows for the DOI release to be done after the main release.
> > We
> > > have revised the KIP to reflect that DOI releases will now occur after
> > the
> > > main release phase. Please review the updated document and provide any
> > > feedback you might have.
> > >
> > > Thanks,
> > > Krish.
> > >
> > > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > Hi Krishna,
> > > >
> > > > I also have the same question as Manikumar raised:
> > > > 1. Will the Docker inventory files/etc are the same for OSS Image and
> > > > Docker Official Images?
> > > > If no, then why not? Could we make them identical so that we don't
> > have to
> > > > build 2 images for each release?
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <ma...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Krishna,
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > I think Docker Official Images will be beneficial to the Kafka
> > community.
> > > > > Few queries below.
> > > > >
> > > > > 1. Will the Docker inventory files/etc are the same for OSS Image and
> > > > > Docker Official Images
> > > > > 2. I am a bit worried about the new steps to the release process.
> > Maybe
> > > > we
> > > > > should consider Docker Official Images release as Post-Release
> > activity.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <kr...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi Hector,
> > > > > >
> > > > > > Thanks for reaching out. This KIP builds on top of KIP-975
> > > > > > <
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > > >
> > > > > > and
> > > > > > aims to introduce a JVM-based Docker Official Image (DOI
> > > > > > <https://docs.docker.com/trusted-content/official-images/>) for
> > Apache
> > > > > > Kafka that will be visible under Docker Official Images
> > > > > > <https://hub.docker.com/search?image_filter=official&q=>. Once
> > > > > implemented
> > > > > > for Apache Kafka, for each release, there will be one more
> > JVM-based
> > > > > Docker
> > > > > > image available to users.
> > > > > >
> > > > > > Currently, we already have an OSS sponsored image, which was
> > introduced
> > > > > via
> > > > > > KIP-975 (apache/kafka <https://hub.docker.com/r/apache/kafka/tags
> > >)
> > > > > which
> > > > > > comes under The Apache Software Foundation <
> > > > > > https://hub.docker.com/u/apache> in
> > > > > > Docker Hub. The new Docker Image is the Docker Official Image
> > (DOI),
> > > > > which
> > > > > > will be built and maintained by Docker Community.
> > > > > >
> > > > > > For example, for a release version like 3.8.0 we will have two JVM
> > > > based
> > > > > > docker images:-
> > > > > >
> > > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > > > >    - kafka:3.8.0 (Docker Official image)
> > > > > >
> > > > > >
> > > > > > I have added the same in the KIP too for everyone's reference.
> > > > > > Thanks,
> > > > > > Krish.
> > > > > >
> > > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino (BLOOMBERG/ 919
> > 3RD
> > > > A) <
> > > > > > hgeraldino@bloomberg.net> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > What is the difference between this KIP and KIP-975: Docker
> > Image for
> > > > > > > Apache Kafka?
> > > > > > >
> > > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:
> > > > > > > dev@kafka.apache.org
> > > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache
> > Kafka
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I would like to start the discussion on
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > > > age+for+Apache+Kafka
> > > > > > >  .
> > > > > > >
> > > > > > > This KIP aims to introduce JVM based Docker Official Image (DOI)
> > for
> > > > > > Apache
> > > > > > > Kafka.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Krish.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Krish Vora <kr...@gmail.com>]

Hi Manikumar,

Thanks for the comments.

Maybe as part of the release process, RM can create a JIRA for this
> task. This can be taken by RM or any comitter or any contributor (with
> some help from commiters to run "Docker Image Preparation via GitHub
> Actions:"

This sounds like a good idea. This step would be beneficial. By creating a
JIRA ticket, it will also serve as a reminder to complete the post-release
steps for the Docker official images. Have updated the KIP with this step.

Is this using GitHub Actions workflow? or manual testing?

This will be done by a Github Actions workflow, which will test the static
Docker Official Image assets for a specific release version.

Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> official images repository (or) can it be done by any contributor.

I believe that it can be done by any contributor (ref: This link
<https://docs.docker.com/trusted-content/official-images/contributing/>
quotes "*Anyone can provide feedback, contribute code, suggest process
changes, or even propose a new Official Image.*")

Also I was thinking, once the KIP gets voted, we should try to release
> kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> validate the process and allow us to fix any changes suggested by
> Dockerhub before the 3.8.0 release.

This sounds like a great idea. This KIP proposes release of DOI as a
post-release process, which can be done anytime post release. Since 3.7.0
is already released, we can perform these steps for that release too. By
the time the KIP gets implemented, if 3.7.1 is released, we could do these
steps for 3.7.1, instead of 3.7.0. This would allow us to make changes to
the Dockerfiles and other assets based on feedback from Docker Hub before
the release of version 3.8.0.

Thanks,
Krish.

On Mon, Apr 15, 2024 at 12:59 PM Manikumar <ma...@gmail.com>
wrote:

> Hi Krish,
>
> Thanks for the updated KIP. a few comments below.
>
> > "These actions can be carried out by the RM or any contributor post the
> release process."
> Maybe as part of the release process, RM can create a JIRA for this
> task. This can be taken by RM or any comitter or any contributor (with
> some help from commiters to run "Docker Image Preparation via GitHub
> Actions:"
>
> > "Perform Docker build tests to ensure image integrity"
> Is this using GitHub Actions workflow? or manual testing?
>
> > "The RM will manually raise the final PR to Docker Hub’s official images
> repository using the contents of the generated file"
>  Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
> official images repository (or) can it be done by any contributor.
>
> Also I was thinking, once the KIP gets voted, we should try to release
> kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
> validate the process and allow us to fix any changes suggested by
> Dockerhub before the 3.8.0 release.
>
>
> Thanks,
>
> On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <kr...@gmail.com> wrote:
> >
> > Hi Manikumar and Luke.
> > Thanks for the questions.
> >
> > 1. No, the Docker inventory files and configurations will not be the same
> > for Open Source Software (OSS) Images and Docker Official Images (DOI).
> >
> > For OSS images, the Dockerfile located in docker/jvm/dockerfile is
> > utilized. This process is integrated with the existing release pipeline
> as
> > outlined in KIP-975
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status
> >,
> > where the Kafka URL is provided as a build argument. This method allows
> for
> > building, testing, and releasing OSS images dynamically. The OSS images
> > will continue to be released under the standard release process .
> >
> > In contrast, the release process for DOIs requires providing the Docker
> Hub
> > team with a specific directory for each version release that contains a
> > standalone Dockerfile. These Dockerfiles are designed to be
> > self-sufficient, hence require hardcoded values instead of relying on
> build
> > arguments. To accommodate this, in our proposed approach, a new directory
> > named docker_official_images has been created. This directory contains
> > version-specific directories, having Dockerfiles with hardcoded
> > configurations for each release, acting as the source of truth for DOI
> > releases. The hardcoded dockerfiles will be created using the
> > docker/jvm/dockerfile as a template. Thus, as part of post release we
> will
> > be creating a Dockerfile that will be reviewed by the Dockerhub community
> > and might need changes as per their review. This approach ensures that
> DOIs
> > are built consistently and meet the specific requirements set by Docker
> Hub.
> >
> > 2. Yes Manikumar, transitioning the release of Docker Official Images
> (DOI)
> > to a post-release activity does address the concerns about complicating
> the
> > release process. Initially, we considered incorporating DOI release
> > directly into Kafka's release workflow. However, this approach
> > significantly increased the RMs workload due to the addition of numerous
> > steps, complicating the process. By designating the DOI release as a
> > post-release task, we maintain the original release process. This
> > adjustment allows for the DOI release to be done after the main release.
> We
> > have revised the KIP to reflect that DOI releases will now occur after
> the
> > main release phase. Please review the updated document and provide any
> > feedback you might have.
> >
> > Thanks,
> > Krish.
> >
> > On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <sh...@gmail.com> wrote:
> >
> > > Hi Krishna,
> > >
> > > I also have the same question as Manikumar raised:
> > > 1. Will the Docker inventory files/etc are the same for OSS Image and
> > > Docker Official Images?
> > > If no, then why not? Could we make them identical so that we don't
> have to
> > > build 2 images for each release?
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi Krishna,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > I think Docker Official Images will be beneficial to the Kafka
> community.
> > > > Few queries below.
> > > >
> > > > 1. Will the Docker inventory files/etc are the same for OSS Image and
> > > > Docker Official Images
> > > > 2. I am a bit worried about the new steps to the release process.
> Maybe
> > > we
> > > > should consider Docker Official Images release as Post-Release
> activity.
> > > >
> > > > Thanks,
> > > >
> > > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <kr...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Hector,
> > > > >
> > > > > Thanks for reaching out. This KIP builds on top of KIP-975
> > > > > <
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > > >
> > > > > and
> > > > > aims to introduce a JVM-based Docker Official Image (DOI
> > > > > <https://docs.docker.com/trusted-content/official-images/>) for
> Apache
> > > > > Kafka that will be visible under Docker Official Images
> > > > > <https://hub.docker.com/search?image_filter=official&q=>. Once
> > > > implemented
> > > > > for Apache Kafka, for each release, there will be one more
> JVM-based
> > > > Docker
> > > > > image available to users.
> > > > >
> > > > > Currently, we already have an OSS sponsored image, which was
> introduced
> > > > via
> > > > > KIP-975 (apache/kafka <https://hub.docker.com/r/apache/kafka/tags
> >)
> > > > which
> > > > > comes under The Apache Software Foundation <
> > > > > https://hub.docker.com/u/apache> in
> > > > > Docker Hub. The new Docker Image is the Docker Official Image
> (DOI),
> > > > which
> > > > > will be built and maintained by Docker Community.
> > > > >
> > > > > For example, for a release version like 3.8.0 we will have two JVM
> > > based
> > > > > docker images:-
> > > > >
> > > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > > >    - kafka:3.8.0 (Docker Official image)
> > > > >
> > > > >
> > > > > I have added the same in the KIP too for everyone's reference.
> > > > > Thanks,
> > > > > Krish.
> > > > >
> > > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino (BLOOMBERG/ 919
> 3RD
> > > A) <
> > > > > hgeraldino@bloomberg.net> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > What is the difference between this KIP and KIP-975: Docker
> Image for
> > > > > > Apache Kafka?
> > > > > >
> > > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:
> > > > > > dev@kafka.apache.org
> > > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache
> Kafka
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I would like to start the discussion on
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > > age+for+Apache+Kafka
> > > > > >  .
> > > > > >
> > > > > > This KIP aims to introduce JVM based Docker Official Image (DOI)
> for
> > > > > Apache
> > > > > > Kafka.
> > > > > >
> > > > > > Regards,
> > > > > > Krish.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Manikumar <ma...@gmail.com>]

Hi Krish,

Thanks for the updated KIP. a few comments below.

> "These actions can be carried out by the RM or any contributor post the release process."
Maybe as part of the release process, RM can create a JIRA for this
task. This can be taken by RM or any comitter or any contributor (with
some help from commiters to run "Docker Image Preparation via GitHub
Actions:"

> "Perform Docker build tests to ensure image integrity"
Is this using GitHub Actions workflow? or manual testing?

> "The RM will manually raise the final PR to Docker Hub’s official images repository using the contents of the generated file"
 Is it mandatory for RM/comitters to raise the PR to Docker Hub’s
official images repository (or) can it be done by any contributor.

Also I was thinking, once the KIP gets voted, we should try to release
kafka:3.7.0 (or 3.7.1) Docker Official image. This will help us to
validate the process and allow us to fix any changes suggested by
Dockerhub before the 3.8.0 release.


Thanks,

On Mon, Apr 8, 2024 at 2:33 PM Krish Vora <kr...@gmail.com> wrote:
>
> Hi Manikumar and Luke.
> Thanks for the questions.
>
> 1. No, the Docker inventory files and configurations will not be the same
> for Open Source Software (OSS) Images and Docker Official Images (DOI).
>
> For OSS images, the Dockerfile located in docker/jvm/dockerfile is
> utilized. This process is integrated with the existing release pipeline as
> outlined in KIP-975
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status>,
> where the Kafka URL is provided as a build argument. This method allows for
> building, testing, and releasing OSS images dynamically. The OSS images
> will continue to be released under the standard release process .
>
> In contrast, the release process for DOIs requires providing the Docker Hub
> team with a specific directory for each version release that contains a
> standalone Dockerfile. These Dockerfiles are designed to be
> self-sufficient, hence require hardcoded values instead of relying on build
> arguments. To accommodate this, in our proposed approach, a new directory
> named docker_official_images has been created. This directory contains
> version-specific directories, having Dockerfiles with hardcoded
> configurations for each release, acting as the source of truth for DOI
> releases. The hardcoded dockerfiles will be created using the
> docker/jvm/dockerfile as a template. Thus, as part of post release we will
> be creating a Dockerfile that will be reviewed by the Dockerhub community
> and might need changes as per their review. This approach ensures that DOIs
> are built consistently and meet the specific requirements set by Docker Hub.
>
> 2. Yes Manikumar, transitioning the release of Docker Official Images (DOI)
> to a post-release activity does address the concerns about complicating the
> release process. Initially, we considered incorporating DOI release
> directly into Kafka's release workflow. However, this approach
> significantly increased the RMs workload due to the addition of numerous
> steps, complicating the process. By designating the DOI release as a
> post-release task, we maintain the original release process. This
> adjustment allows for the DOI release to be done after the main release. We
> have revised the KIP to reflect that DOI releases will now occur after the
> main release phase. Please review the updated document and provide any
> feedback you might have.
>
> Thanks,
> Krish.
>
> On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Krishna,
> >
> > I also have the same question as Manikumar raised:
> > 1. Will the Docker inventory files/etc are the same for OSS Image and
> > Docker Official Images?
> > If no, then why not? Could we make them identical so that we don't have to
> > build 2 images for each release?
> >
> > Thank you.
> > Luke
> >
> > On Wed, Apr 3, 2024 at 12:41 AM Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Hi Krishna,
> > >
> > > Thanks for the KIP.
> > >
> > > I think Docker Official Images will be beneficial to the Kafka community.
> > > Few queries below.
> > >
> > > 1. Will the Docker inventory files/etc are the same for OSS Image and
> > > Docker Official Images
> > > 2. I am a bit worried about the new steps to the release process. Maybe
> > we
> > > should consider Docker Official Images release as Post-Release activity.
> > >
> > > Thanks,
> > >
> > > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <kr...@gmail.com>
> > wrote:
> > >
> > > > Hi Hector,
> > > >
> > > > Thanks for reaching out. This KIP builds on top of KIP-975
> > > > <
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > >
> > > > and
> > > > aims to introduce a JVM-based Docker Official Image (DOI
> > > > <https://docs.docker.com/trusted-content/official-images/>) for Apache
> > > > Kafka that will be visible under Docker Official Images
> > > > <https://hub.docker.com/search?image_filter=official&q=>. Once
> > > implemented
> > > > for Apache Kafka, for each release, there will be one more JVM-based
> > > Docker
> > > > image available to users.
> > > >
> > > > Currently, we already have an OSS sponsored image, which was introduced
> > > via
> > > > KIP-975 (apache/kafka <https://hub.docker.com/r/apache/kafka/tags>)
> > > which
> > > > comes under The Apache Software Foundation <
> > > > https://hub.docker.com/u/apache> in
> > > > Docker Hub. The new Docker Image is the Docker Official Image (DOI),
> > > which
> > > > will be built and maintained by Docker Community.
> > > >
> > > > For example, for a release version like 3.8.0 we will have two JVM
> > based
> > > > docker images:-
> > > >
> > > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > > >    - kafka:3.8.0 (Docker Official image)
> > > >
> > > >
> > > > I have added the same in the KIP too for everyone's reference.
> > > > Thanks,
> > > > Krish.
> > > >
> > > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino (BLOOMBERG/ 919 3RD
> > A) <
> > > > hgeraldino@bloomberg.net> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > What is the difference between this KIP and KIP-975: Docker Image for
> > > > > Apache Kafka?
> > > > >
> > > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:
> > > > > dev@kafka.apache.org
> > > > > Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I would like to start the discussion on
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > > age+for+Apache+Kafka
> > > > >  .
> > > > >
> > > > > This KIP aims to introduce JVM based Docker Official Image (DOI) for
> > > > Apache
> > > > > Kafka.
> > > > >
> > > > > Regards,
> > > > > Krish.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Krish Vora <kr...@gmail.com>]

Hi Manikumar and Luke.
Thanks for the questions.

1. No, the Docker inventory files and configurations will not be the same
for Open Source Software (OSS) Images and Docker Official Images (DOI).

For OSS images, the Dockerfile located in docker/jvm/dockerfile is
utilized. This process is integrated with the existing release pipeline as
outlined in KIP-975
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-Status>,
where the Kafka URL is provided as a build argument. This method allows for
building, testing, and releasing OSS images dynamically. The OSS images
will continue to be released under the standard release process .

In contrast, the release process for DOIs requires providing the Docker Hub
team with a specific directory for each version release that contains a
standalone Dockerfile. These Dockerfiles are designed to be
self-sufficient, hence require hardcoded values instead of relying on build
arguments. To accommodate this, in our proposed approach, a new directory
named docker_official_images has been created. This directory contains
version-specific directories, having Dockerfiles with hardcoded
configurations for each release, acting as the source of truth for DOI
releases. The hardcoded dockerfiles will be created using the
docker/jvm/dockerfile as a template. Thus, as part of post release we will
be creating a Dockerfile that will be reviewed by the Dockerhub community
and might need changes as per their review. This approach ensures that DOIs
are built consistently and meet the specific requirements set by Docker Hub.

2. Yes Manikumar, transitioning the release of Docker Official Images (DOI)
to a post-release activity does address the concerns about complicating the
release process. Initially, we considered incorporating DOI release
directly into Kafka's release workflow. However, this approach
significantly increased the RMs workload due to the addition of numerous
steps, complicating the process. By designating the DOI release as a
post-release task, we maintain the original release process. This
adjustment allows for the DOI release to be done after the main release. We
have revised the KIP to reflect that DOI releases will now occur after the
main release phase. Please review the updated document and provide any
feedback you might have.

Thanks,
Krish.

On Wed, Apr 3, 2024 at 3:35 PM Luke Chen <sh...@gmail.com> wrote:

> Hi Krishna,
>
> I also have the same question as Manikumar raised:
> 1. Will the Docker inventory files/etc are the same for OSS Image and
> Docker Official Images?
> If no, then why not? Could we make them identical so that we don't have to
> build 2 images for each release?
>
> Thank you.
> Luke
>
> On Wed, Apr 3, 2024 at 12:41 AM Manikumar <ma...@gmail.com>
> wrote:
>
> > Hi Krishna,
> >
> > Thanks for the KIP.
> >
> > I think Docker Official Images will be beneficial to the Kafka community.
> > Few queries below.
> >
> > 1. Will the Docker inventory files/etc are the same for OSS Image and
> > Docker Official Images
> > 2. I am a bit worried about the new steps to the release process. Maybe
> we
> > should consider Docker Official Images release as Post-Release activity.
> >
> > Thanks,
> >
> > On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <kr...@gmail.com>
> wrote:
> >
> > > Hi Hector,
> > >
> > > Thanks for reaching out. This KIP builds on top of KIP-975
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > >
> > > and
> > > aims to introduce a JVM-based Docker Official Image (DOI
> > > <https://docs.docker.com/trusted-content/official-images/>) for Apache
> > > Kafka that will be visible under Docker Official Images
> > > <https://hub.docker.com/search?image_filter=official&q=>. Once
> > implemented
> > > for Apache Kafka, for each release, there will be one more JVM-based
> > Docker
> > > image available to users.
> > >
> > > Currently, we already have an OSS sponsored image, which was introduced
> > via
> > > KIP-975 (apache/kafka <https://hub.docker.com/r/apache/kafka/tags>)
> > which
> > > comes under The Apache Software Foundation <
> > > https://hub.docker.com/u/apache> in
> > > Docker Hub. The new Docker Image is the Docker Official Image (DOI),
> > which
> > > will be built and maintained by Docker Community.
> > >
> > > For example, for a release version like 3.8.0 we will have two JVM
> based
> > > docker images:-
> > >
> > >    - apache/kafka:3.8.0 (OSS sponsored image)
> > >    - kafka:3.8.0 (Docker Official image)
> > >
> > >
> > > I have added the same in the KIP too for everyone's reference.
> > > Thanks,
> > > Krish.
> > >
> > > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino (BLOOMBERG/ 919 3RD
> A) <
> > > hgeraldino@bloomberg.net> wrote:
> > >
> > > > Hi,
> > > >
> > > > What is the difference between this KIP and KIP-975: Docker Image for
> > > > Apache Kafka?
> > > >
> > > > From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:
> > > > dev@kafka.apache.org
> > > > Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka
> > > >
> > > > Hi everyone,
> > > >
> > > > I would like to start the discussion on
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > > age+for+Apache+Kafka
> > > >  .
> > > >
> > > > This KIP aims to introduce JVM based Docker Official Image (DOI) for
> > > Apache
> > > > Kafka.
> > > >
> > > > Regards,
> > > > Krish.
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Luke Chen <sh...@gmail.com>]

Hi Krishna,

I also have the same question as Manikumar raised:
1. Will the Docker inventory files/etc are the same for OSS Image and
Docker Official Images?
If no, then why not? Could we make them identical so that we don't have to
build 2 images for each release?

Thank you.
Luke

On Wed, Apr 3, 2024 at 12:41 AM Manikumar <ma...@gmail.com> wrote:

> Hi Krishna,
>
> Thanks for the KIP.
>
> I think Docker Official Images will be beneficial to the Kafka community.
> Few queries below.
>
> 1. Will the Docker inventory files/etc are the same for OSS Image and
> Docker Official Images
> 2. I am a bit worried about the new steps to the release process. Maybe we
> should consider Docker Official Images release as Post-Release activity.
>
> Thanks,
>
> On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <kr...@gmail.com> wrote:
>
> > Hi Hector,
> >
> > Thanks for reaching out. This KIP builds on top of KIP-975
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > >
> > and
> > aims to introduce a JVM-based Docker Official Image (DOI
> > <https://docs.docker.com/trusted-content/official-images/>) for Apache
> > Kafka that will be visible under Docker Official Images
> > <https://hub.docker.com/search?image_filter=official&q=>. Once
> implemented
> > for Apache Kafka, for each release, there will be one more JVM-based
> Docker
> > image available to users.
> >
> > Currently, we already have an OSS sponsored image, which was introduced
> via
> > KIP-975 (apache/kafka <https://hub.docker.com/r/apache/kafka/tags>)
> which
> > comes under The Apache Software Foundation <
> > https://hub.docker.com/u/apache> in
> > Docker Hub. The new Docker Image is the Docker Official Image (DOI),
> which
> > will be built and maintained by Docker Community.
> >
> > For example, for a release version like 3.8.0 we will have two JVM based
> > docker images:-
> >
> >    - apache/kafka:3.8.0 (OSS sponsored image)
> >    - kafka:3.8.0 (Docker Official image)
> >
> >
> > I have added the same in the KIP too for everyone's reference.
> > Thanks,
> > Krish.
> >
> > On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> > hgeraldino@bloomberg.net> wrote:
> >
> > > Hi,
> > >
> > > What is the difference between this KIP and KIP-975: Docker Image for
> > > Apache Kafka?
> > >
> > > From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:
> > > dev@kafka.apache.org
> > > Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka
> > >
> > > Hi everyone,
> > >
> > > I would like to start the discussion on
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > > age+for+Apache+Kafka
> > >  .
> > >
> > > This KIP aims to introduce JVM based Docker Official Image (DOI) for
> > Apache
> > > Kafka.
> > >
> > > Regards,
> > > Krish.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Manikumar <ma...@gmail.com>]

Hi Krishna,

Thanks for the KIP.

I think Docker Official Images will be beneficial to the Kafka community.
Few queries below.

1. Will the Docker inventory files/etc are the same for OSS Image and
Docker Official Images
2. I am a bit worried about the new steps to the release process. Maybe we
should consider Docker Official Images release as Post-Release activity.

Thanks,

On Fri, Mar 22, 2024 at 3:29 PM Krish Vora <kr...@gmail.com> wrote:

> Hi Hector,
>
> Thanks for reaching out. This KIP builds on top of KIP-975
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> >
> and
> aims to introduce a JVM-based Docker Official Image (DOI
> <https://docs.docker.com/trusted-content/official-images/>) for Apache
> Kafka that will be visible under Docker Official Images
> <https://hub.docker.com/search?image_filter=official&q=>. Once implemented
> for Apache Kafka, for each release, there will be one more JVM-based Docker
> image available to users.
>
> Currently, we already have an OSS sponsored image, which was introduced via
> KIP-975 (apache/kafka <https://hub.docker.com/r/apache/kafka/tags>) which
> comes under The Apache Software Foundation <
> https://hub.docker.com/u/apache> in
> Docker Hub. The new Docker Image is the Docker Official Image (DOI), which
> will be built and maintained by Docker Community.
>
> For example, for a release version like 3.8.0 we will have two JVM based
> docker images:-
>
>    - apache/kafka:3.8.0 (OSS sponsored image)
>    - kafka:3.8.0 (Docker Official image)
>
>
> I have added the same in the KIP too for everyone's reference.
> Thanks,
> Krish.
>
> On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> hgeraldino@bloomberg.net> wrote:
>
> > Hi,
> >
> > What is the difference between this KIP and KIP-975: Docker Image for
> > Apache Kafka?
> >
> > From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:
> > dev@kafka.apache.org
> > Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka
> >
> > Hi everyone,
> >
> > I would like to start the discussion on
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> > age+for+Apache+Kafka
> >  .
> >
> > This KIP aims to introduce JVM based Docker Official Image (DOI) for
> Apache
> > Kafka.
> >
> > Regards,
> > Krish.
> >
> >
> >
>

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

[Krish Vora <kr...@gmail.com>]

Hi Hector,

Thanks for reaching out. This KIP builds on top of KIP-975
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka>
and
aims to introduce a JVM-based Docker Official Image (DOI
<https://docs.docker.com/trusted-content/official-images/>) for Apache
Kafka that will be visible under Docker Official Images
<https://hub.docker.com/search?image_filter=official&q=>. Once implemented
for Apache Kafka, for each release, there will be one more JVM-based Docker
image available to users.

Currently, we already have an OSS sponsored image, which was introduced via
KIP-975 (apache/kafka <https://hub.docker.com/r/apache/kafka/tags>) which
comes under The Apache Software Foundation <https://hub.docker.com/u/apache> in
Docker Hub. The new Docker Image is the Docker Official Image (DOI), which
will be built and maintained by Docker Community.

For example, for a release version like 3.8.0 we will have two JVM based
docker images:-

   - apache/kafka:3.8.0 (OSS sponsored image)
   - kafka:3.8.0 (Docker Official image)


I have added the same in the KIP too for everyone's reference.
Thanks,
Krish.

On Fri, Mar 22, 2024 at 2:50 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgeraldino@bloomberg.net> wrote:

> Hi,
>
> What is the difference between this KIP and KIP-975: Docker Image for
> Apache Kafka?
>
> From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:
> dev@kafka.apache.org
> Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka
>
> Hi everyone,
>
> I would like to start the discussion on
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
> age+for+Apache+Kafka
>  .
>
> This KIP aims to introduce JVM based Docker Official Image (DOI) for Apache
> Kafka.
>
> Regards,
> Krish.
>
>
>

Re:[DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

["Hector Geraldino (BLOOMBERG/ 919 3RD A)" <hg...@bloomberg.net>]

Hi,

What is the difference between this KIP and KIP-975: Docker Image for Apache Kafka? 

From: dev@kafka.apache.org At: 03/21/24 07:30:07 UTC-4:00To:  dev@kafka.apache.org
Subject: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

Hi everyone,

I would like to start the discussion on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Im
age+for+Apache+Kafka
 .

This KIP aims to introduce JVM based Docker Official Image (DOI) for Apache
Kafka.

Regards,
Krish.