You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "nodece (via GitHub)" <gi...@apache.org> on 2023/05/29 15:32:28 UTC
[GitHub] [pulsar] nodece opened a new pull request, #20433: [improve][build] Improve the publishing of docker image
nodece opened a new pull request, #20433:
URL: https://github.com/apache/pulsar/pull/20433
### Motivation
Right now the `docker/publish.sh` script always publishes the latest image, this causes the following issues:
1. Pulsar version issue: https://github.com/apache/pulsar/issues/19544
2. The `apachepulsar/pulsar-all` image depends on the `apachepulsar/pulsar:latest` image, sometimes when building the `pulsar-all` image, which uses the old `apachepulsar/pulsar:latest` image: https://github.com/apache/pulsar/issues/20420
### Modifications
- Add a prompt to confirm the publishing of the latest images in the `docker/publish.sh` script.
- Use `apachepulsar/pulsar:${PULSAR_VERSION}` instead of `apachepulsar/pulsar:latest` in the `docker/pulsar-all/Dockerfile`, I expect to be able to use the correct pulsar image.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567850276
@nicoloboschi PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567393426
> * Use `apachepulsar/pulsar:${PULSAR_VERSION}` instead of `apachepulsar/pulsar:latest` in the `docker/pulsar-all/Dockerfile`, I expect to be able to use the correct pulsar image.
I support this change, but the usage of `apachepulsar/pulsar:latest` in `docker/pulsar-all/Dockerfile` wasn't the root cause that caused Pulsar 3.0.0 pulsar-all docker images to use the 2.11 image.
The default build process should have built `apachepulsar/pulsar:latest` locally before the pulsar-all image.
Perhaps the build dependency isn't working as expected?
https://github.com/apache/pulsar/blob/bdd1bf1fcabdfc52527d07a8f8a288930e2cb7dd/docker/pulsar-all/pom.xml#L71-L86
We might have to dig deeper.
A possible way to prevent using an old image would be to use a per-build invocation tag instead of using `project.version` alone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece closed pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece closed pull request #20433: [improve][build] Improve the publishing of docker image
URL: https://github.com/apache/pulsar/pull/20433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cbornet commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "cbornet (via GitHub)" <gi...@apache.org>.
cbornet commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567565172
Note that I didn't use the `publish.sh` script to build the image.
I used the following command to have cross-compilation for ARM:
mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ \
-DskipTests \
-Pmain,docker -Pdocker-push \
-Ddocker.platforms=linux/amd64,linux/arm64 \
-Ddocker.organization=cbornet \
-pl docker/pulsar,docker/pulsar-all
We should update the script to use this command and see why it didn't pick the correct image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567690437
@RobertIndie Good catch! BTW, we still need to update the `publish.sh` script, or use `docker-push` instead of that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1568101537
I think we need to handle the two points :
1. Use the correct image as the base image in the `pulsar-all/Dockerfile` and `tests/docker-images/latest-version-image/Dockerfile`
3. Cleanup the build and publish scripts, using maven command instead shell script
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1573110509
Move to #20462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567673100
Hi @cbornet, it looks your command is correct, I need to reproduce your case.
If you still have build logs on your computer, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567397084
Using the Git SHA as part of the tag and ensuring that the repository is clean before building would be a useful improvement over the changes already in this PR. The problem with project.version is that it's not unique. It's better than `latest`, but doesn't prevent the mistake from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567781900
> Hi @lhotari, thank you for your explanation!
>
> > Perhaps the build dependency isn't working as expected?
> > It seems that there is no locally built image used.
>
> I agreed that use Git SHA, which is unique, but we can also use the project.version when building the apachepulsar/puslar-all:3.0.0 image, and we use the apachepulsar/pulsar:3.0.0 as the base image, which will does not cause version confusion unless you provide the wrong version number.
The version number doesn't address the problem since the version number doesn't uniquely identify the build product (artifact). The project.version is not much better than `latest` in this sense, it won't fix the issue.
@nodece Your PR would be a good solution by simply continuing the work and making it use a tag that is for example a combination of the project.version and the git sha. I hope you could consider this.
If there's confusion, I could help by pushing the changes to this PR. It's not many extra changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567919161
> Note that I didn't use the `publish.sh` script to build the image. I used the following command to have cross-compilation for ARM:
>
> ```
> mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ \
> -DskipTests \
> -Pmain,docker -Pdocker-push \
> -Ddocker.platforms=linux/amd64,linux/arm64 \
> -Ddocker.organization=cbornet \
> -pl docker/pulsar,docker/pulsar-all
> ```
>
> We should update the script to use this command and see why it didn't pick the correct image.
@cbornet Would you mind getting the process description in https://pulsar.apache.org/contribute/release-process/#stage-docker-images up-to-date with this information? /cc @tisonkun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece closed pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece closed pull request #20433: [improve][build] Improve the publishing of docker image
URL: https://github.com/apache/pulsar/pull/20433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567669029
Hi @lhotari, thank you for your explanation!
> Perhaps the build dependency isn't working as expected?
It seems that there is no locally built image used.
I agreed that use Git SHA, which is unique, but we can also use the project.version when building the apachepulsar/puslar-all:3.0.0 image, and we use the apachepulsar/pulsar:3.0.0 as the base image, which will does not cause version confusion unless you provide the wrong version number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567842850
I guess we would need to backport the solution to all maintenance branches so that maintenance release docker images don't get mixed up. The script shouldn't attempt to push the `latest` tag when the script is run in maintenance branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567917634
> I am afraid that the `publish.sh` in this PR doesn't build the arm64 image.
@RobertIndie @cbornet @tisonkun Is https://pulsar.apache.org/contribute/release-process/#stage-docker-images up-to-date with this information?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] RobertIndie commented on pull request #20433: [improve][build] Improve the publishing of docker image
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
RobertIndie commented on PR #20433:
URL: https://github.com/apache/pulsar/pull/20433#issuecomment-1567687981
Hi, I don't think this is the root cause. The root cause is here: https://github.com/apache/pulsar/issues/20420#issuecomment-1566820505
I have pushed a PR to address the root causeļ¼ https://github.com/apache/pulsar/pull/20435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org