You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Asaf Mesika <as...@gmail.com> on 2023/06/01 08:51:43 UTC
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
I understand that you can't build a release process due to Apache
foundation rules, that makes it mandatory to release something you built on
your own machine.
On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli <eo...@gmail.com> wrote:
> Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall
> <mm...@apache.org> ha scritto:
> >
> > The best solution is to make the build repeatable. IIUC, that will let
> > us build all of the artifacts using CI instead of personal machines,
> > which removes this class of errors.
> >
> > That being said, I don't know how much effort it would be to achieve.
> >
> > Thanks,
> > Michael
> >
> > On Tue, May 30, 2023 at 2:24 AM Zike Yang <zi...@apache.org> wrote:
> > >
> > > Hi, Enrico
> > >
> > > > When we ran the VOTE and we provided the docker images, were they
> > > already broken ?
> > >
> > > Actually, they are not broken unless we use the new features of Pulsar
> 3.0.0.
>
> It is not Pulsar 3 but Pulsar 2.11.
> So for users this is a problem
>
> If we are able to prepare the docker image from the RELEASED tarbal
> and connectors then I am fine with it.
> If you have to rebuild from the sources I strongly believe that we
> cannot do it without a proper release process
>
> Enrico
>
>
>
> > >
> > > I think we need something like verification test scripts, to verify
> > > the release candidate. For example, we use the image provided in the
> > > RC to run the integration tests. And we need to make sure that we have
> > > tested the newly added feature for the RC docker image.
> > >
> > > What do you think?
> > >
> > > BR,
> > > Zike Yang
> > >
> > > On Tue, May 30, 2023 at 3:14 PM Zike Yang <zi...@apache.org> wrote:
> > > >
> > > > Hi, Asaf
> > > >
> > > > > How do you suggest we prevent it from happening next time?
> > > >
> > > > I have pushed a PR to fix it:
> https://github.com/apache/pulsar/pull/20435
> > > > This PR specifies the correct image name for `pulsar` image to build
> pulsar-all.
> > > >
> > > > Note that, in the release of Pulsar 3.0, we build the docker image by
> > > > executing the following command instead of the `docker/build.sh`:
> > > > ```
> > > > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
> \
> > > > -DskipTests \
> > > > -Pdocker -Pdocker-push \
> > > > -Ddocker.platforms=linux/amd64,linux/arm64 \
> > > > -Ddocker.organization=snzkyang \
> > > > -pl docker/pulsar,docker/pulsar-all
> > > > ```
> > > > I think to take it a step further, we could fix these
> scripts(build.sh
> > > > and publish.sh) and use the shell scripts to build the image.
> > > >
> > > > I have verified the PR, and it works well. Please see more detail in
> > > > the PR description.
> > > >
> > > > Thanks,
> > > > Zike Yang
> > > >
> > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
> > > > >
> > > > > I am really worried about the process.
> > > > >
> > > > > When we ran the VOTE and we provided the docker images, were they
> > > > > already broken ?
> > > > >
> > > > > In any case we cannot overwrite those images, they have been cached
> > > > > all over the world now.
> > > > >
> > > > > It is safer to cut a new 3.0.1 release and run a VOTE.
> > > > >
> > > > > Maybe we can remove the old images, forever
> > > > >
> > > > > Enrico
> > > > >
> > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika
> > > > > <as...@gmail.com> ha scritto:
> > > > > >
> > > > > > Good catch!
> > > > > >
> > > > > > How do you suggest we prevent it from happening next time?
> > > > > >
> > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang <zi...@apache.org>
> wrote:
> > > > > >
> > > > > > > Hi, all
> > > > > > >
> > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image.
> The
> > > > > > > pulsar library included in `pulsar-all:3.0.0` is the version of
> > > > > > > 2.11.0:
> > > > > > >
> > > > > > > ```
> > > > > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep
> pulsar-broker
> > > > > > >
> > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar
> > > > > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar
> > > > > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar
> > > > > > > ```
> > > > > > >
> > > > > > > The root cause is that we use `apachepulsar/pulsar:latest` to
> build
> > > > > > > the `pulsar-all` image. But at the time of building Pulsar
> 3.0.0,
> > > > > > > `apachepulsar/pulsar:latest` was pointing to version 2.11.0.
> > > > > > >
> > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0
> of
> > > > > > > Pulsar but with 3.0.0 connectors and offloaders.
> > > > > > >
> > > > > > > Please see more detail in this issue:
> > > > > > > https://github.com/apache/pulsar/issues/20420
> > > > > > >
> > > > > > > I have rebuilt the `pulsar-all:3.0.0` image:
> > > > > > >
> > > > > > >
> https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore
> > > > > > >
> > > > > > > Please help and verify it. And check if there are any other
> problems
> > > > > > > with the image.
> > > > > > >
> > > > > > > I'm going to publish the image to the `apachepulsar`
> organization to
> > > > > > > replace the old one. But before we do that, do we need a Vote
> or other
> > > > > > > ways to reach a consensus? Is there any problem if we replace
> the old
> > > > > > > image?
> > > > > > >
> > > > > > > Besides, I will also fix the docker build script to avoid
> similar issues.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Zike Yang
> > > > > > >
>
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
Posted by tison <wa...@gmail.com>.
I agree with Zike that the root cause of this specific case is that 3.0
changes the docker images artifacts and we don't update the process.
Before we correct the process to ensure that human can follow, automation
only automatically make mistakes.
tison <wa...@gmail.com>于2023年6月2日 周五00:53写道：
> The ASF only counts the source releases the official releases. All
> binaries including docker images should be for convenience only.
>
> Even the image needs to go through a vote, you can of course build and
> stage the image with any method in hand. The problem is where we can share
> a build environment - Since that building pulsar-all is a bit resource
> consuming, it can requires extra platform resource.
>
> Zixuan Liu <no...@gmail.com>于2023年6月1日 周四23:47写道：
>
>> > The best solution is to make the build repeatable. IIUC, that will let
>> us build all of the artifacts using CI instead of personal machines,
>> which removes this class of errors.
>>
>> I agree! But I don't know if Apache rules allow this operation, and if
>> so, I recommend using the CI to publish our project.
>>
>> Thanks,
>> Zixuan
>>
>>
>> Zike Yang <zi...@apache.org> 于2023年6月1日周四 18:45写道：
>> >
>> > > The best solution is to make the build repeatable. IIUC, that will let
>> > us build all of the artifacts using CI instead of personal machines,
>> > which removes this class of errors.
>> >
>> > I don't think this issue is related to where the docker image was
>> > built. It has nothing to do with caching and nothing to do with the
>> > machine on which it is built. It's just because the build uses the
>> > `latest` tag, which points to an older version.
>> > I'm + 0.5 for this solution. I am currently wondering how much reward
>> > it will bring.
>> >
>> > > If we are able to prepare the docker image from the RELEASED tarbal
>> > and connectors then I am fine with it.
>> >
>> > We currently build our images using compiled tarball rather than
>> > rebuilding from sources. You can check the code here:
>> >
>> https://github.com/apache/pulsar/blob/5e6e6cebcdbeec32ed49729f658f2d5cd0d98347/docker/pulsar/Dockerfile#L25-L26
>> >
>> > BR,
>> > Zike Yang
>> >
>> > On Thu, Jun 1, 2023 at 4:52 PM Asaf Mesika <as...@gmail.com>
>> wrote:
>> > >
>> > > I understand that you can't build a release process due to Apache
>> > > foundation rules, that makes it mandatory to release something you
>> built on
>> > > your own machine.
>> > >
>> > >
>> > > On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli <eo...@gmail.com>
>> wrote:
>> > >
>> > > > Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall
>> > > > <mm...@apache.org> ha scritto:
>> > > > >
>> > > > > The best solution is to make the build repeatable. IIUC, that
>> will let
>> > > > > us build all of the artifacts using CI instead of personal
>> machines,
>> > > > > which removes this class of errors.
>> > > > >
>> > > > > That being said, I don't know how much effort it would be to
>> achieve.
>> > > > >
>> > > > > Thanks,
>> > > > > Michael
>> > > > >
>> > > > > On Tue, May 30, 2023 at 2:24 AM Zike Yang <zi...@apache.org>
>> wrote:
>> > > > > >
>> > > > > > Hi, Enrico
>> > > > > >
>> > > > > > > When we ran the VOTE and we provided the docker images, were
>> they
>> > > > > > already broken ?
>> > > > > >
>> > > > > > Actually, they are not broken unless we use the new features of
>> Pulsar
>> > > > 3.0.0.
>> > > >
>> > > > It is not Pulsar 3 but Pulsar 2.11.
>> > > > So for users this is a problem
>> > > >
>> > > > If we are able to prepare the docker image from the RELEASED tarbal
>> > > > and connectors then I am fine with it.
>> > > > If you have to rebuild from the sources I strongly believe that we
>> > > > cannot do it without a proper release process
>> > > >
>> > > > Enrico
>> > > >
>> > > >
>> > > >
>> > > > > >
>> > > > > > I think we need something like verification test scripts, to
>> verify
>> > > > > > the release candidate. For example, we use the image provided
>> in the
>> > > > > > RC to run the integration tests. And we need to make sure that
>> we have
>> > > > > > tested the newly added feature for the RC docker image.
>> > > > > >
>> > > > > > What do you think?
>> > > > > >
>> > > > > > BR,
>> > > > > > Zike Yang
>> > > > > >
>> > > > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang <zi...@apache.org>
>> wrote:
>> > > > > > >
>> > > > > > > Hi, Asaf
>> > > > > > >
>> > > > > > > > How do you suggest we prevent it from happening next time?
>> > > > > > >
>> > > > > > > I have pushed a PR to fix it:
>> > > > https://github.com/apache/pulsar/pull/20435
>> > > > > > > This PR specifies the correct image name for `pulsar` image
>> to build
>> > > > pulsar-all.
>> > > > > > >
>> > > > > > > Note that, in the release of Pulsar 3.0, we build the docker
>> image by
>> > > > > > > executing the following command instead of the
>> `docker/build.sh`:
>> > > > > > > ```
>> > > > > > > mvn install -DUBUNTU_MIRROR=
>> http://azure.archive.ubuntu.com/ubuntu/
>> > > > \
>> > > > > > > -DskipTests \
>> > > > > > > -Pdocker -Pdocker-push \
>> > > > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \
>> > > > > > > -Ddocker.organization=snzkyang \
>> > > > > > > -pl docker/pulsar,docker/pulsar-all
>> > > > > > > ```
>> > > > > > > I think to take it a step further, we could fix these
>> > > > scripts(build.sh
>> > > > > > > and publish.sh) and use the shell scripts to build the image.
>> > > > > > >
>> > > > > > > I have verified the PR, and it works well. Please see more
>> detail in
>> > > > > > > the PR description.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Zike Yang
>> > > > > > >
>> > > > > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli <
>> eolivelli@gmail.com>
>> > > > wrote:
>> > > > > > > >
>> > > > > > > > I am really worried about the process.
>> > > > > > > >
>> > > > > > > > When we ran the VOTE and we provided the docker images,
>> were they
>> > > > > > > > already broken ?
>> > > > > > > >
>> > > > > > > > In any case we cannot overwrite those images, they have
>> been cached
>> > > > > > > > all over the world now.
>> > > > > > > >
>> > > > > > > > It is safer to cut a new 3.0.1 release and run a VOTE.
>> > > > > > > >
>> > > > > > > > Maybe we can remove the old images, forever
>> > > > > > > >
>> > > > > > > > Enrico
>> > > > > > > >
>> > > > > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika
>> > > > > > > > <as...@gmail.com> ha scritto:
>> > > > > > > > >
>> > > > > > > > > Good catch!
>> > > > > > > > >
>> > > > > > > > > How do you suggest we prevent it from happening next time?
>> > > > > > > > >
>> > > > > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang <
>> zike@apache.org>
>> > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi, all
>> > > > > > > > > >
>> > > > > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0`
>> image.
>> > > > The
>> > > > > > > > > > pulsar library included in `pulsar-all:3.0.0` is the
>> version of
>> > > > > > > > > > 2.11.0:
>> > > > > > > > > >
>> > > > > > > > > > ```
>> > > > > > > > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep
>> > > > pulsar-broker
>> > > > > > > > > >
>> > > > > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar
>> > > > > > > > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar
>> > > > > > > > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar
>> > > > > > > > > > ```
>> > > > > > > > > >
>> > > > > > > > > > The root cause is that we use
>> `apachepulsar/pulsar:latest` to
>> > > > build
>> > > > > > > > > > the `pulsar-all` image. But at the time of building
>> Pulsar
>> > > > 3.0.0,
>> > > > > > > > > > `apachepulsar/pulsar:latest` was pointing to version
>> 2.11.0.
>> > > > > > > > > >
>> > > > > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version
>> 2.11.0
>> > > > of
>> > > > > > > > > > Pulsar but with 3.0.0 connectors and offloaders.
>> > > > > > > > > >
>> > > > > > > > > > Please see more detail in this issue:
>> > > > > > > > > > https://github.com/apache/pulsar/issues/20420
>> > > > > > > > > >
>> > > > > > > > > > I have rebuilt the `pulsar-all:3.0.0` image:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > >
>> https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore
>> > > > > > > > > >
>> > > > > > > > > > Please help and verify it. And check if there are any
>> other
>> > > > problems
>> > > > > > > > > > with the image.
>> > > > > > > > > >
>> > > > > > > > > > I'm going to publish the image to the `apachepulsar`
>> > > > organization to
>> > > > > > > > > > replace the old one. But before we do that, do we need
>> a Vote
>> > > > or other
>> > > > > > > > > > ways to reach a consensus? Is there any problem if we
>> replace
>> > > > the old
>> > > > > > > > > > image?
>> > > > > > > > > >
>> > > > > > > > > > Besides, I will also fix the docker build script to
>> avoid
>> > > > similar issues.
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Zike Yang
>> > > > > > > > > >
>> > > >
>>
> --
> Best,
> tison.
>
--
Best,
tison.
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
Posted by tison <wa...@gmail.com>.
The ASF only counts the source releases the official releases. All binaries
including docker images should be for convenience only.
Even the image needs to go through a vote, you can of course build and
stage the image with any method in hand. The problem is where we can share
a build environment - Since that building pulsar-all is a bit resource
consuming, it can requires extra platform resource.
Zixuan Liu <no...@gmail.com>于2023年6月1日 周四23:47写道：
> > The best solution is to make the build repeatable. IIUC, that will let
> us build all of the artifacts using CI instead of personal machines,
> which removes this class of errors.
>
> I agree! But I don't know if Apache rules allow this operation, and if
> so, I recommend using the CI to publish our project.
>
> Thanks,
> Zixuan
>
>
> Zike Yang <zi...@apache.org> 于2023年6月1日周四 18:45写道：
> >
> > > The best solution is to make the build repeatable. IIUC, that will let
> > us build all of the artifacts using CI instead of personal machines,
> > which removes this class of errors.
> >
> > I don't think this issue is related to where the docker image was
> > built. It has nothing to do with caching and nothing to do with the
> > machine on which it is built. It's just because the build uses the
> > `latest` tag, which points to an older version.
> > I'm + 0.5 for this solution. I am currently wondering how much reward
> > it will bring.
> >
> > > If we are able to prepare the docker image from the RELEASED tarbal
> > and connectors then I am fine with it.
> >
> > We currently build our images using compiled tarball rather than
> > rebuilding from sources. You can check the code here:
> >
> https://github.com/apache/pulsar/blob/5e6e6cebcdbeec32ed49729f658f2d5cd0d98347/docker/pulsar/Dockerfile#L25-L26
> >
> > BR,
> > Zike Yang
> >
> > On Thu, Jun 1, 2023 at 4:52 PM Asaf Mesika <as...@gmail.com>
> wrote:
> > >
> > > I understand that you can't build a release process due to Apache
> > > foundation rules, that makes it mandatory to release something you
> built on
> > > your own machine.
> > >
> > >
> > > On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
> > >
> > > > Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall
> > > > <mm...@apache.org> ha scritto:
> > > > >
> > > > > The best solution is to make the build repeatable. IIUC, that will
> let
> > > > > us build all of the artifacts using CI instead of personal
> machines,
> > > > > which removes this class of errors.
> > > > >
> > > > > That being said, I don't know how much effort it would be to
> achieve.
> > > > >
> > > > > Thanks,
> > > > > Michael
> > > > >
> > > > > On Tue, May 30, 2023 at 2:24 AM Zike Yang <zi...@apache.org> wrote:
> > > > > >
> > > > > > Hi, Enrico
> > > > > >
> > > > > > > When we ran the VOTE and we provided the docker images, were
> they
> > > > > > already broken ?
> > > > > >
> > > > > > Actually, they are not broken unless we use the new features of
> Pulsar
> > > > 3.0.0.
> > > >
> > > > It is not Pulsar 3 but Pulsar 2.11.
> > > > So for users this is a problem
> > > >
> > > > If we are able to prepare the docker image from the RELEASED tarbal
> > > > and connectors then I am fine with it.
> > > > If you have to rebuild from the sources I strongly believe that we
> > > > cannot do it without a proper release process
> > > >
> > > > Enrico
> > > >
> > > >
> > > >
> > > > > >
> > > > > > I think we need something like verification test scripts, to
> verify
> > > > > > the release candidate. For example, we use the image provided in
> the
> > > > > > RC to run the integration tests. And we need to make sure that
> we have
> > > > > > tested the newly added feature for the RC docker image.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > BR,
> > > > > > Zike Yang
> > > > > >
> > > > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang <zi...@apache.org>
> wrote:
> > > > > > >
> > > > > > > Hi, Asaf
> > > > > > >
> > > > > > > > How do you suggest we prevent it from happening next time?
> > > > > > >
> > > > > > > I have pushed a PR to fix it:
> > > > https://github.com/apache/pulsar/pull/20435
> > > > > > > This PR specifies the correct image name for `pulsar` image to
> build
> > > > pulsar-all.
> > > > > > >
> > > > > > > Note that, in the release of Pulsar 3.0, we build the docker
> image by
> > > > > > > executing the following command instead of the
> `docker/build.sh`:
> > > > > > > ```
> > > > > > > mvn install -DUBUNTU_MIRROR=
> http://azure.archive.ubuntu.com/ubuntu/
> > > > \
> > > > > > > -DskipTests \
> > > > > > > -Pdocker -Pdocker-push \
> > > > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \
> > > > > > > -Ddocker.organization=snzkyang \
> > > > > > > -pl docker/pulsar,docker/pulsar-all
> > > > > > > ```
> > > > > > > I think to take it a step further, we could fix these
> > > > scripts(build.sh
> > > > > > > and publish.sh) and use the shell scripts to build the image.
> > > > > > >
> > > > > > > I have verified the PR, and it works well. Please see more
> detail in
> > > > > > > the PR description.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Zike Yang
> > > > > > >
> > > > > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli <
> eolivelli@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > I am really worried about the process.
> > > > > > > >
> > > > > > > > When we ran the VOTE and we provided the docker images, were
> they
> > > > > > > > already broken ?
> > > > > > > >
> > > > > > > > In any case we cannot overwrite those images, they have been
> cached
> > > > > > > > all over the world now.
> > > > > > > >
> > > > > > > > It is safer to cut a new 3.0.1 release and run a VOTE.
> > > > > > > >
> > > > > > > > Maybe we can remove the old images, forever
> > > > > > > >
> > > > > > > > Enrico
> > > > > > > >
> > > > > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika
> > > > > > > > <as...@gmail.com> ha scritto:
> > > > > > > > >
> > > > > > > > > Good catch!
> > > > > > > > >
> > > > > > > > > How do you suggest we prevent it from happening next time?
> > > > > > > > >
> > > > > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang <zike@apache.org
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, all
> > > > > > > > > >
> > > > > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0`
> image.
> > > > The
> > > > > > > > > > pulsar library included in `pulsar-all:3.0.0` is the
> version of
> > > > > > > > > > 2.11.0:
> > > > > > > > > >
> > > > > > > > > > ```
> > > > > > > > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep
> > > > pulsar-broker
> > > > > > > > > >
> > > > > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar
> > > > > > > > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar
> > > > > > > > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar
> > > > > > > > > > ```
> > > > > > > > > >
> > > > > > > > > > The root cause is that we use
> `apachepulsar/pulsar:latest` to
> > > > build
> > > > > > > > > > the `pulsar-all` image. But at the time of building
> Pulsar
> > > > 3.0.0,
> > > > > > > > > > `apachepulsar/pulsar:latest` was pointing to version
> 2.11.0.
> > > > > > > > > >
> > > > > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version
> 2.11.0
> > > > of
> > > > > > > > > > Pulsar but with 3.0.0 connectors and offloaders.
> > > > > > > > > >
> > > > > > > > > > Please see more detail in this issue:
> > > > > > > > > > https://github.com/apache/pulsar/issues/20420
> > > > > > > > > >
> > > > > > > > > > I have rebuilt the `pulsar-all:3.0.0` image:
> > > > > > > > > >
> > > > > > > > > >
> > > >
> https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore
> > > > > > > > > >
> > > > > > > > > > Please help and verify it. And check if there are any
> other
> > > > problems
> > > > > > > > > > with the image.
> > > > > > > > > >
> > > > > > > > > > I'm going to publish the image to the `apachepulsar`
> > > > organization to
> > > > > > > > > > replace the old one. But before we do that, do we need a
> Vote
> > > > or other
> > > > > > > > > > ways to reach a consensus? Is there any problem if we
> replace
> > > > the old
> > > > > > > > > > image?
> > > > > > > > > >
> > > > > > > > > > Besides, I will also fix the docker build script to avoid
> > > > similar issues.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Zike Yang
> > > > > > > > > >
> > > >
>
--
Best,
tison.
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
Posted by Zixuan Liu <no...@gmail.com>.
> The best solution is to make the build repeatable. IIUC, that will let
us build all of the artifacts using CI instead of personal machines,
which removes this class of errors.
I agree! But I don't know if Apache rules allow this operation, and if
so, I recommend using the CI to publish our project.
Thanks,
Zixuan
Zike Yang <zi...@apache.org> 于2023年6月1日周四 18:45写道：
>
> > The best solution is to make the build repeatable. IIUC, that will let
> us build all of the artifacts using CI instead of personal machines,
> which removes this class of errors.
>
> I don't think this issue is related to where the docker image was
> built. It has nothing to do with caching and nothing to do with the
> machine on which it is built. It's just because the build uses the
> `latest` tag, which points to an older version.
> I'm + 0.5 for this solution. I am currently wondering how much reward
> it will bring.
>
> > If we are able to prepare the docker image from the RELEASED tarbal
> and connectors then I am fine with it.
>
> We currently build our images using compiled tarball rather than
> rebuilding from sources. You can check the code here:
> https://github.com/apache/pulsar/blob/5e6e6cebcdbeec32ed49729f658f2d5cd0d98347/docker/pulsar/Dockerfile#L25-L26
>
> BR,
> Zike Yang
>
> On Thu, Jun 1, 2023 at 4:52 PM Asaf Mesika <as...@gmail.com> wrote:
> >
> > I understand that you can't build a release process due to Apache
> > foundation rules, that makes it mandatory to release something you built on
> > your own machine.
> >
> >
> > On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli <eo...@gmail.com> wrote:
> >
> > > Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall
> > > <mm...@apache.org> ha scritto:
> > > >
> > > > The best solution is to make the build repeatable. IIUC, that will let
> > > > us build all of the artifacts using CI instead of personal machines,
> > > > which removes this class of errors.
> > > >
> > > > That being said, I don't know how much effort it would be to achieve.
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > On Tue, May 30, 2023 at 2:24 AM Zike Yang <zi...@apache.org> wrote:
> > > > >
> > > > > Hi, Enrico
> > > > >
> > > > > > When we ran the VOTE and we provided the docker images, were they
> > > > > already broken ?
> > > > >
> > > > > Actually, they are not broken unless we use the new features of Pulsar
> > > 3.0.0.
> > >
> > > It is not Pulsar 3 but Pulsar 2.11.
> > > So for users this is a problem
> > >
> > > If we are able to prepare the docker image from the RELEASED tarbal
> > > and connectors then I am fine with it.
> > > If you have to rebuild from the sources I strongly believe that we
> > > cannot do it without a proper release process
> > >
> > > Enrico
> > >
> > >
> > >
> > > > >
> > > > > I think we need something like verification test scripts, to verify
> > > > > the release candidate. For example, we use the image provided in the
> > > > > RC to run the integration tests. And we need to make sure that we have
> > > > > tested the newly added feature for the RC docker image.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > BR,
> > > > > Zike Yang
> > > > >
> > > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang <zi...@apache.org> wrote:
> > > > > >
> > > > > > Hi, Asaf
> > > > > >
> > > > > > > How do you suggest we prevent it from happening next time?
> > > > > >
> > > > > > I have pushed a PR to fix it:
> > > https://github.com/apache/pulsar/pull/20435
> > > > > > This PR specifies the correct image name for `pulsar` image to build
> > > pulsar-all.
> > > > > >
> > > > > > Note that, in the release of Pulsar 3.0, we build the docker image by
> > > > > > executing the following command instead of the `docker/build.sh`:
> > > > > > ```
> > > > > > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
> > > \
> > > > > > -DskipTests \
> > > > > > -Pdocker -Pdocker-push \
> > > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \
> > > > > > -Ddocker.organization=snzkyang \
> > > > > > -pl docker/pulsar,docker/pulsar-all
> > > > > > ```
> > > > > > I think to take it a step further, we could fix these
> > > scripts(build.sh
> > > > > > and publish.sh) and use the shell scripts to build the image.
> > > > > >
> > > > > > I have verified the PR, and it works well. Please see more detail in
> > > > > > the PR description.
> > > > > >
> > > > > > Thanks,
> > > > > > Zike Yang
> > > > > >
> > > > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > I am really worried about the process.
> > > > > > >
> > > > > > > When we ran the VOTE and we provided the docker images, were they
> > > > > > > already broken ?
> > > > > > >
> > > > > > > In any case we cannot overwrite those images, they have been cached
> > > > > > > all over the world now.
> > > > > > >
> > > > > > > It is safer to cut a new 3.0.1 release and run a VOTE.
> > > > > > >
> > > > > > > Maybe we can remove the old images, forever
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika
> > > > > > > <as...@gmail.com> ha scritto:
> > > > > > > >
> > > > > > > > Good catch!
> > > > > > > >
> > > > > > > > How do you suggest we prevent it from happening next time?
> > > > > > > >
> > > > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang <zi...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi, all
> > > > > > > > >
> > > > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image.
> > > The
> > > > > > > > > pulsar library included in `pulsar-all:3.0.0` is the version of
> > > > > > > > > 2.11.0:
> > > > > > > > >
> > > > > > > > > ```
> > > > > > > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep
> > > pulsar-broker
> > > > > > > > >
> > > > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar
> > > > > > > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar
> > > > > > > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar
> > > > > > > > > ```
> > > > > > > > >
> > > > > > > > > The root cause is that we use `apachepulsar/pulsar:latest` to
> > > build
> > > > > > > > > the `pulsar-all` image. But at the time of building Pulsar
> > > 3.0.0,
> > > > > > > > > `apachepulsar/pulsar:latest` was pointing to version 2.11.0.
> > > > > > > > >
> > > > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0
> > > of
> > > > > > > > > Pulsar but with 3.0.0 connectors and offloaders.
> > > > > > > > >
> > > > > > > > > Please see more detail in this issue:
> > > > > > > > > https://github.com/apache/pulsar/issues/20420
> > > > > > > > >
> > > > > > > > > I have rebuilt the `pulsar-all:3.0.0` image:
> > > > > > > > >
> > > > > > > > >
> > > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore
> > > > > > > > >
> > > > > > > > > Please help and verify it. And check if there are any other
> > > problems
> > > > > > > > > with the image.
> > > > > > > > >
> > > > > > > > > I'm going to publish the image to the `apachepulsar`
> > > organization to
> > > > > > > > > replace the old one. But before we do that, do we need a Vote
> > > or other
> > > > > > > > > ways to reach a consensus? Is there any problem if we replace
> > > the old
> > > > > > > > > image?
> > > > > > > > >
> > > > > > > > > Besides, I will also fix the docker build script to avoid
> > > similar issues.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Zike Yang
> > > > > > > > >
> > >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
Posted by Zike Yang <zi...@apache.org>.
> The best solution is to make the build repeatable. IIUC, that will let
us build all of the artifacts using CI instead of personal machines,
which removes this class of errors.
I don't think this issue is related to where the docker image was
built. It has nothing to do with caching and nothing to do with the
machine on which it is built. It's just because the build uses the
`latest` tag, which points to an older version.
I'm + 0.5 for this solution. I am currently wondering how much reward
it will bring.
> If we are able to prepare the docker image from the RELEASED tarbal
and connectors then I am fine with it.
We currently build our images using compiled tarball rather than
rebuilding from sources. You can check the code here:
https://github.com/apache/pulsar/blob/5e6e6cebcdbeec32ed49729f658f2d5cd0d98347/docker/pulsar/Dockerfile#L25-L26
BR,
Zike Yang
On Thu, Jun 1, 2023 at 4:52 PM Asaf Mesika <as...@gmail.com> wrote:
>
> I understand that you can't build a release process due to Apache
> foundation rules, that makes it mandatory to release something you built on
> your own machine.
>
>
> On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli <eo...@gmail.com> wrote:
>
> > Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall
> > <mm...@apache.org> ha scritto:
> > >
> > > The best solution is to make the build repeatable. IIUC, that will let
> > > us build all of the artifacts using CI instead of personal machines,
> > > which removes this class of errors.
> > >
> > > That being said, I don't know how much effort it would be to achieve.
> > >
> > > Thanks,
> > > Michael
> > >
> > > On Tue, May 30, 2023 at 2:24 AM Zike Yang <zi...@apache.org> wrote:
> > > >
> > > > Hi, Enrico
> > > >
> > > > > When we ran the VOTE and we provided the docker images, were they
> > > > already broken ?
> > > >
> > > > Actually, they are not broken unless we use the new features of Pulsar
> > 3.0.0.
> >
> > It is not Pulsar 3 but Pulsar 2.11.
> > So for users this is a problem
> >
> > If we are able to prepare the docker image from the RELEASED tarbal
> > and connectors then I am fine with it.
> > If you have to rebuild from the sources I strongly believe that we
> > cannot do it without a proper release process
> >
> > Enrico
> >
> >
> >
> > > >
> > > > I think we need something like verification test scripts, to verify
> > > > the release candidate. For example, we use the image provided in the
> > > > RC to run the integration tests. And we need to make sure that we have
> > > > tested the newly added feature for the RC docker image.
> > > >
> > > > What do you think?
> > > >
> > > > BR,
> > > > Zike Yang
> > > >
> > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang <zi...@apache.org> wrote:
> > > > >
> > > > > Hi, Asaf
> > > > >
> > > > > > How do you suggest we prevent it from happening next time?
> > > > >
> > > > > I have pushed a PR to fix it:
> > https://github.com/apache/pulsar/pull/20435
> > > > > This PR specifies the correct image name for `pulsar` image to build
> > pulsar-all.
> > > > >
> > > > > Note that, in the release of Pulsar 3.0, we build the docker image by
> > > > > executing the following command instead of the `docker/build.sh`:
> > > > > ```
> > > > > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
> > \
> > > > > -DskipTests \
> > > > > -Pdocker -Pdocker-push \
> > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \
> > > > > -Ddocker.organization=snzkyang \
> > > > > -pl docker/pulsar,docker/pulsar-all
> > > > > ```
> > > > > I think to take it a step further, we could fix these
> > scripts(build.sh
> > > > > and publish.sh) and use the shell scripts to build the image.
> > > > >
> > > > > I have verified the PR, and it works well. Please see more detail in
> > > > > the PR description.
> > > > >
> > > > > Thanks,
> > > > > Zike Yang
> > > > >
> > > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> > > > > >
> > > > > > I am really worried about the process.
> > > > > >
> > > > > > When we ran the VOTE and we provided the docker images, were they
> > > > > > already broken ?
> > > > > >
> > > > > > In any case we cannot overwrite those images, they have been cached
> > > > > > all over the world now.
> > > > > >
> > > > > > It is safer to cut a new 3.0.1 release and run a VOTE.
> > > > > >
> > > > > > Maybe we can remove the old images, forever
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika
> > > > > > <as...@gmail.com> ha scritto:
> > > > > > >
> > > > > > > Good catch!
> > > > > > >
> > > > > > > How do you suggest we prevent it from happening next time?
> > > > > > >
> > > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang <zi...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > > Hi, all
> > > > > > > >
> > > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image.
> > The
> > > > > > > > pulsar library included in `pulsar-all:3.0.0` is the version of
> > > > > > > > 2.11.0:
> > > > > > > >
> > > > > > > > ```
> > > > > > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep
> > pulsar-broker
> > > > > > > >
> > > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar
> > > > > > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar
> > > > > > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar
> > > > > > > > ```
> > > > > > > >
> > > > > > > > The root cause is that we use `apachepulsar/pulsar:latest` to
> > build
> > > > > > > > the `pulsar-all` image. But at the time of building Pulsar
> > 3.0.0,
> > > > > > > > `apachepulsar/pulsar:latest` was pointing to version 2.11.0.
> > > > > > > >
> > > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0
> > of
> > > > > > > > Pulsar but with 3.0.0 connectors and offloaders.
> > > > > > > >
> > > > > > > > Please see more detail in this issue:
> > > > > > > > https://github.com/apache/pulsar/issues/20420
> > > > > > > >
> > > > > > > > I have rebuilt the `pulsar-all:3.0.0` image:
> > > > > > > >
> > > > > > > >
> > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore
> > > > > > > >
> > > > > > > > Please help and verify it. And check if there are any other
> > problems
> > > > > > > > with the image.
> > > > > > > >
> > > > > > > > I'm going to publish the image to the `apachepulsar`
> > organization to
> > > > > > > > replace the old one. But before we do that, do we need a Vote
> > or other
> > > > > > > > ways to reach a consensus? Is there any problem if we replace
> > the old
> > > > > > > > image?
> > > > > > > >
> > > > > > > > Besides, I will also fix the docker build script to avoid
> > similar issues.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Zike Yang
> > > > > > > >
> >