You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "arjunanan6 (via GitHub)" <gi...@apache.org> on 2023/02/10 15:19:22 UTC
[GitHub] [airflow] arjunanan6 opened a new pull request, #29465: Use newer setuptools v67.2.0
arjunanan6 opened a new pull request, #29465:
URL: https://github.com/apache/airflow/pull/29465
Issue #29428 describes a potential CVE in setuptools v63.4.3 that is currently used, and says that any version before 65.5.1 could have the vulnerability.
Switching to the newest release of setuptools, v67.2.0 for this. A local test with breeze didn't indicate any issues after this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #29465: Use newer setuptools v67.2.0
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #29465:
URL: https://github.com/apache/airflow/pull/29465#issuecomment-1425957355
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29465: Use newer setuptools v67.2.0
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29465:
URL: https://github.com/apache/airflow/pull/29465#issuecomment-1426663405
Running tests. Let's see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on pull request #29465: Use newer setuptools v67.2.0
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on PR #29465:
URL: https://github.com/apache/airflow/pull/29465#issuecomment-1603952880
I am facing issues with this for local virtualenv setup for editable installs. Not sure what we can do here, but for now pinning it locally to 63.4.3 for getting my local virtualenv up.
There's also a comment we have leftover just above the change in this PR which mentions about the issue regarding editable installs.
Since, the description mentions fix for a potential vulnerability, I guess we have to live it with like this until setuptools fixes the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #29465: Use newer setuptools v67.2.0
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #29465:
URL: https://github.com/apache/airflow/pull/29465#issuecomment-1426808607
Awesome work, congrats on your first merged pull request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #29465: Use newer setuptools v67.2.0
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #29465:
URL: https://github.com/apache/airflow/pull/29465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arjunanan6 commented on pull request #29465: Use newer setuptools v67.2.0
Posted by "arjunanan6 (via GitHub)" <gi...@apache.org>.
arjunanan6 commented on PR #29465:
URL: https://github.com/apache/airflow/pull/29465#issuecomment-1426700468
@potiuk Seems like all tests pass except one for test_internal_api_command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29465: Use newer setuptools v67.2.0
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29465:
URL: https://github.com/apache/airflow/pull/29465#issuecomment-1604017790
We have an issue about it https://github.com/apache/airflow/issues/30764 already and yeah. I think it needs a bit closer look. I think it might be quite tricky to solve - our setup.py is pretty, well complex, so for now I kept on ignoring it and I think we will need to solve it possibly by generally modernizing out package build tool configuration - part of it has been done in #31378 . What makes it difficult is that we want to keep the possibilities of:
1) Being able to run `pip install -e .` and have all "airfow + providers" in one editable environment
2) Similarly - have the environment in Breeze where both airflow and providers are immediately "editable" - i.e. installed from sources so that you do not have to reinstall provider package every time you edit the source code of it.
3) But also have an option to only install airflow, without the providers and install the providers from `pypi` (for CI and testing and constraint generation).
So far this has been achieved by `INSTALL_PROVIDERS_FROM_SOURCES` env variable that determines the approach taken (and in breeze for example `INSTALL_PROVIDERS_FROM_SOURCES` is set to `true`. There is also quite complex setup.py code that has been handling that, but in recent releases of `pip` and `setuptools` some of the `hacks` we were using to achieve that got subtly brokent and we need to figure out another approach.
It might also be that we will only be able to do it "well" when we switch to a "proper" structure for provider's code, following https://github.com/apache/airflow/pull/28291 (scripts) and https://github.com/apache/airflow/pull/28292 (POC migration) - but then solving the above a little contradicting requirements might be tricky.
I've been following what's going on in Python PyPa and maybe we should switch to [flit](https://github.com/pypa/flit) or [hatch](https://github.com/pypa/hatch) - both are part of `PyPa` and both have great maintainers who are working on them. I believe that in hatch (listening to recent podcast of hatch creator: https://talkpython.fm/episodes/show/408/hatch-a-modern-python-workflow there is an idea or maybe even an early implementation already for something similar to our case where you have multiple projects in monorepo that should be edit-installable together.
And my preference would be rather than invent something on our own, tocontribute some work to hatch to make it capable of doing what we want to do. I think this is a great opportunity if our goals and ideas are aligned, because we could provide a nice testing ground for such case for hatch.
@uranusjr - I think you are the best person here to have a say and recommendations what we should do and likely you even have some good relation with hatch maintainer and we could jointly do something together and maybe even help to set the standards for the Python packaging worlds ? We could setup a small team around it and work on both - contributing things to hatch (if the direction would be aligned) and applying it to Airflow at the same time.
Maybe we should brainstorm a little here or in #30764 on what we should do and how we should approach it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on pull request #29465: Use newer setuptools v67.2.0
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on PR #29465:
URL: https://github.com/apache/airflow/pull/29465#issuecomment-1604044088
I have created a quick PR https://github.com/apache/airflow/pull/32090 to update the comment a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arjunanan6 commented on pull request #29465: Use newer setuptools v67.2.0
Posted by "arjunanan6 (via GitHub)" <gi...@apache.org>.
arjunanan6 commented on PR #29465:
URL: https://github.com/apache/airflow/pull/29465#issuecomment-1425964652
@potiuk @eladkal change is ready for review whenever you have time too look at it. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org