You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/04 11:09:03 UTC
[GitHub] [airflow] sfc-gh-mkmak opened a new pull request, #24825: Dockerfile centos
sfc-gh-mkmak opened a new pull request, #24825:
URL: https://github.com/apache/airflow/pull/24825
Hello @mik-laj and @potiuk
I would like to contribute to airflow and I am sending preliminary changes for dockerfile for centos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
mik-laj commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1174423386
Honestly, it wasn't our intention to merge this change, and it certainly wasn't as it was. We created this PR because you were interested to see what our image looked like and we thought you would be interested to build official support based on it. I am currently too busy with other initiatives, so feel free to take this code and develop it yourself if you want.
As for the action plan, I agree with it, but I would probably start by building this image with a minimal set of dependencies and testing it on CI. And then, as further improvements, added support for other features, such as Kerberos. Testing all dependencies is possible, but I'm afraid it can be difficult for a non-full-time contributor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1173825804
This is really cool.
Before I get into details of the image, I think what we need here is some kind of testing to make sure the Dockerfile works and that it can be used - connected with a documentation on how it can be used. Eventually - when such code is contributed, we need to keep it "working" and our "users" need to know that they can use it and how.
We have a number of such "checks" and verifications in our repo for our "debian" base image and while it is certainly not necessary to get to the level of details and tests we have there, there needs to be some way that:
a) tests are run in the CI so that we are sure the neither CentOS nor any of the dependenies broke the image building process
b) users will discover they can use that image - either by building their own custom image or extend our image that we publish in dockerhub
c) users should know how they can customize to their needs (for example choose different airflow version, different python vesion, different centos base image (maybe). This might be just a doc or possibly support in the form of `--build-args` (same approach we have in our debian dockerfile.
d) we would publish the image in our DockerHub account
Not all of that needs to be impemented all at once in single PR but I think we should agree here what is the "target" we want to achieve (in follow-up PR) and what "level of a) b) c) d)" above should be in the first PR
My first thought for this PR:
a) we should at least build the image in CI. using latest released airflow version.
b) documentation should be added to https://airflow.apache.org/docs/docker-stack/index.html to let users know we give them a possibility to build CentoOS-based image using our Dockerfile.centos (and explain them how). Ideally examples of building it should be connected to tests ans should be included in our docs - we are currently doing that here - all our examples here are embedded from our scripts, and those scripts are actually executed in CI https://airflow.apache.org/docs/docker-stack/build.html#adding-new-apt-package . It does not have to be yet as comprehensive as our debian images, it could be one or two examples, but they should be there and guide the users.
c) I think adding build-args to support airflow version and python version should be good to start with - then the examples from b) could run a few combinations of buidls (and actually build them on CI). We should run it for all python versions that we support (3.7 - 3.10) or - if there are any reasons some of those are not supported we should mention why
d) I think that woudl be nice to have, but this can be done way later - when we will publish the "buildable" centos image we can also make it automatically published in Airflow Repo. This would likely require some discussion and likely lazy-consensus on Airlfow devlist so I am happy to make it happen.
Once (or during) a) b) c) are adressed - I can make a detailed review then. Seeing the CI builds that pass the builds would be great mark that it is ready to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1174437799
I converted it to Draft then. Maybe indeed there will be time to implement it at some poin in time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
mik-laj commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1173694210
@potiuk As we discussed earlier, here's a CentOS-based Dockerfile image that Snowflake uses. Hopefully, this will allow us to prepare official support for CentOS in the community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1207236417
Hey @mik-laj @sfc-gh-mkmak - I looked a bit closer to that one, and I have a concern. It looks like the base container image that you used for the image is reatehr old. In Airflow, we strive for releasing our images based on the latest and greatest (i.e. with all known fixed security issues) released by the Python Software Foundation: https://hub.docker.com/_/python?tab=tags
For example, the latest version of 3.7-3.10 debian images has been pushed 2 days ago (and our CI system will automatically refresh our base images we publish to use the latest version in ~ 1 day.
The centos base python image you used `centos/python-38-centos7:20210726-fad62e9` is ratehr old in comparision and unfortunately it looks like:
1) It's been updated last time > 1 year ago; https://hub.docker.com/r/centos/python-38-centos7
2) There are no 3.9/3.10 Python images at all released by centos organisation
I am a little concerned with using those (and I am a little concerned you are not concerned :) ). It does not only miss the latest security fixes, but also the Python 3.8 version there is rather old there were likely 6 or 8 patchlevel releases there bringin bugfixes to the 3.8 line
Do you have any thoughts/ideass/concerns about an up-to-date base for such a centos image?
I even looked at the "official centos image" and even that seems to be very out-dated (6-12 months) - which in the world of Security/IT and especially Supply Chain attacks is an eternity.
I am afraid we would not be able to put our "trust" in such rarely released images - especially that our users are deeply concerned about security and we had many requests and questions about up-todatednesss and handingl some known and published CVEs in the images.
Can you think about a good/reliable/updated source for an up-todated centOS based images we could use as a base? Aren't you concerned about it in Snowflake BTW?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sfc-gh-mkmak commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
sfc-gh-mkmak commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1209528138
Hello @potiuk - I understand your point. Now I pushed changes in which I upgraded base image to centos9 - `eurolinux/centos-stream-9:centos-stream-9-9.0.8` which was build 7 days ago. In Snowflake we have custom image which we use and upgrade frequently. It is not the same image like centos7. Could you take a look on that image and let me know if it is enough. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1279607856
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1356124608
Reopening. I might have some time to look back at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1227135369
sorry for not following up yet, live intervened a bit :) but I wil get to it, I promise :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sfc-gh-mkmak commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
sfc-gh-mkmak commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1231222171
Hello @potiuk thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #24825: Dockerfile centos
URL: https://github.com/apache/airflow/pull/24825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1173687799
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24825: Dockerfile centos
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1221565281
Just merged the change that allows to work on this one. I will take a closer look shortly. Never heard of eurolinux before and I was kinda surprised it was based in Poland :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #24825: Dockerfile centos
Posted by github-actions.
github-actions[bot] commented on PR #24825:
URL: https://github.com/apache/airflow/pull/24825#issuecomment-1411251315
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #24825: Dockerfile centos
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #24825: Dockerfile centos
URL: https://github.com/apache/airflow/pull/24825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org