You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/13 03:07:23 UTC
[GitHub] [airflow] huxuan opened a new issue #19569: The latest docker image is not the "latest"
huxuan opened a new issue #19569:
URL: https://github.com/apache/airflow/issues/19569
### Apache Airflow version
2.2.1 (latest released)
### Operating System
N/A
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### What happened
The image tags with `latest` and `latest-python3.X` is either release two months ago or even 5 months ago.
https://hub.docker.com/r/apache/airflow/tags?name=latest
### What you expected to happen
According to the documentation here [1], seems it should be aligned with the latest stable version.
BTW, I am willing to submit a PR, but might need some hints how we manage the docker image tags.
[1] https://airflow.apache.org/docs/docker-stack/index.html
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huxuan edited a comment on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
huxuan edited a comment on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-970057918
> I recommend using 'versioned' images though :). You always have to run migration when you switch so this should be a conscious effort rather than accidental upgrade.
Gotcha, thanks for the notes! And thanks for everyone make this happen (so faster than I expected).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huxuan commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
huxuan commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-970057918
> I recommend using 'versioned' images though :). You always have to run migration when you switch so this should be a conscious effort rather than accidental upgrade.
Gotcha, thanks for the notes! And thanks for everyone make this happen (so fast than I expected).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huxuan commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
huxuan commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-967864273
@potiuk Really appreciate for the quick response. I am new to airflow community, this makes me more confident about promoting airflow to my team. Just wondering the release of airflow is still done manually? Though I know airflow is quite a complex (and mature) project, I was thought the release process should be done by something like Github Actions. Please feel free to point out if I misunderstand anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-967801638
Ah - actually it WAS there but was not followed (even by mysefl) so I have to see how we can make it more difficult to skip:
> If this is the newest image released, push the latest image as well.
``` shellscript
docker tag "apache/airflow:${VERSION}" "apache/airflow:latest"
docker push "apache/airflow:latest"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-967801638
Ah - actually it WAS there but was not followed (even by mysefl) so I have to see how we can make it more difficult to skip:
```
If this is the newest image released, push the latest image as well.
```shell script
docker tag "apache/airflow:${VERSION}" "apache/airflow:latest"
docker push "apache/airflow:latest"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huxuan commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
huxuan commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-967883458
@mik-laj Thanks for the quick and clear explanation! That makes sense. Looking forward to have more communication with airflow community!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-967769215
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-967872440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-969972852
And credit for tonight upload goes to @kaxil actually ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-967872440
> Though I know airflow is quite a complex (and mature) project, I was thought the release process should be done by something like Github Actions.
Yes. This applies to all Apache releases.
> # MUST RELEASES BE BUILT ON HARDWARE OWNED AND CONTROLLED BY THE COMMITTER?
> Strictly speaking, releases must be verified on hardware owned and controlled by the committer. That means hardware the committer has physical possession and control of and exclusively full administrative/superuser access to. That's because only such hardware is qualified to hold a PGP private key, and the release should be verified on the machine the private key lives on or on a machine as trusted as that.
>
> Practically speaking, when a release consists of anything beyond an archive (e.g., tarball or zip file) of a source control tag, the only practical way to validate that archive is to build it locally; manually inspecting generated files (especially binary files) is not feasible. So, basically, "Yes".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #19569:
URL: https://github.com/apache/airflow/issues/19569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huxuan commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
huxuan commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-969724588
Thanks @potiuk for your work. I also notice that the docker hub is already updated. That is great since we are going to kick off the official deploy process in my team today!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-967800512
Good. Spot. I fixed it now in the repo. I will update the process of releasing to include refresh of the latest images as it was missing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-968055704
But all the steps of releasing including building packages. Building images etc. are actually done in GitHub actions basically after every pull request. We even publish CI mages automatically (you can see main and branch images in ghcr.io published fully automatically. Even more we automatically bump dependencies of airflow during the CI build after every successful merge build so that latest compatible releases (we have > 500 of them) are used automatically including all security fixes.
And the same scripts are used when we release manually - in this case we simply did not run the scripts :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-969972101
I recommend using 'versioned' images though :). You always have to run migration when you switch so this should be a conscious effort rather than accidental upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-968055180
Yep. Releasing via GitHub actions is far too vulnerable to injecting malicious code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] huxuan commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
huxuan commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-968060873
@potiuk That just seems to be the good and bad for both manual and automatic processing. Anyway, thanks for the detailed clarification!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19569: The latest docker image is not the "latest"
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19569:
URL: https://github.com/apache/airflow/issues/19569#issuecomment-968093692
Yeah - the solution is really to semi-automate in a smart way. The fix I came up with in #19573 improves it in the way that it detects whethere we are (semi-automatically) releasing RC or "final image" and then asks a question if the "latest" tag shoudl be applied (latest should not be applied if we release a patch-level fix in an old branch - in case we ever do so).
This way we should have far less chance of missing it.
For fully automated releases, there is also always the question - who monitors and fixes problems if the "fully automated CI process" fails. Since this process of release is run every few weeks only (we do not fully release the X.Y.Z image more often than that), manual overlook over the release process is actually better than fully-automated release (as long as the semi-automation is complete).
Our semi-automated process does a lot of tests on the image and you also visually see the results of it - you can see which providers were installed, you can see if there are no suspicious warnings there, you can see that the tests were actually performed (for example we test if all the required and important packages are importable and whether important "aifllow" commands actually work in the image.
While "full automation" of that is possible, if the automation breaks and shows false-positive, having such manual inspection is really a "sanity check". So we have all the scripts tested that they return "success" with every PR but then every release a human eye looks at what's beeing released. I think this is really the best of both automated/manual worlds (as long you have process that make it possible you miss important steps - as we had). With improved semi-automation, I tihnk it's really how it should be (and as good as it gets).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org