You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/03 15:43:39 UTC
[GitHub] [airflow] ashb opened a new pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
ashb opened a new pull request #12786:
URL: https://github.com/apache/airflow/pull/12786
One less thing to change, and one less pre-commit step needed :)
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738765276
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738114224
Cool, I'll update those use cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738350135
> > scripts/ci/libraries/_initialization.sh - it reads version from airflow/version.py, it should from setup.py
>
> We could use `python setup.py --version`, but do we have enough of a python environment wherever that is run? I feel we should, all it needs is setuptools.
>
> Pushed a fixup with that -- lets see what happens.
As commented - i'd prefer base it on having POSIX compliant is simply.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#discussion_r535678204
##########
File path: scripts/ci/libraries/_initialization.sh
##########
@@ -134,8 +134,8 @@ function initialization::initialize_base_variables() {
# otherwise it will use files/airflow-breeze-config/init.sh
export INIT_SCRIPT_FILE=${INIT_SCRIPT_FILE:=""}
- # Read airflow version from the version.py
- AIRFLOW_VERSION=$(grep version "${AIRFLOW_SOURCES}/airflow/version.py" | awk '{print $3}' | sed "s/['+]//g")
+ # Read airflow version from the setup.py.
+ AIRFLOW_VERSION=$(python "${AIRFLOW_SOURCES}/setup.py" --version)
Review comment:
I'd prefer not to run Python here if possible. While this is a reasonable expectation that Python is installed in the hos, there are still some cases where people will have python2 by default for example and i would rather keep it bash only. There is another case in the future where we will use those scripts in code spaces environment and vs code to run docker-only environment and i would not like to have to depend on Python there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#discussion_r535348096
##########
File path: airflow/version.py
##########
@@ -17,4 +17,13 @@
# under the License.
#
-version = '2.0.0b3'
+__all__ = ['version']
+
+try:
+ from importlib import metadata
+except ImportError:
+ import importlib_metadata as metadata
+
+version = metadata.version('apache-airflow')
+
+del metadata
Review comment:
(This line isn't strictly needed, it was just so that the only symbol in this module was `version`. For no god reason really)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#discussion_r535678716
##########
File path: scripts/ci/libraries/_initialization.sh
##########
@@ -134,8 +134,8 @@ function initialization::initialize_base_variables() {
# otherwise it will use files/airflow-breeze-config/init.sh
export INIT_SCRIPT_FILE=${INIT_SCRIPT_FILE:=""}
- # Read airflow version from the version.py
- AIRFLOW_VERSION=$(grep version "${AIRFLOW_SOURCES}/airflow/version.py" | awk '{print $3}' | sed "s/['+]//g")
+ # Read airflow version from the setup.py.
+ AIRFLOW_VERSION=$(python "${AIRFLOW_SOURCES}/setup.py" --version)
Review comment:
Sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738657993
> I'm not sure, but Google tried to use a similar trick and it causes problems.
> [googleapis/google-api-python-client#876](https://github.com/googleapis/google-api-python-client/issues/876)
That seems specific to using Pyinstaller
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738109274
We should fix it in few other places as well:
* scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh -> it can be removed, it was needed because setup.py imported it before
* Dockerfile.ci -> ditto
* STATIC_CODE_CHECKS.rst has reference to removed pre-commit rule
* dev/README.md needs to be updated
* dev/README_RELEASE_AIRLFOW.md as well
* dev/provider_packages/enter_breeze_provider_package_tests.sh also can be removed as it was needed only for setup.py
* scripts/ci/libraries/_initialization.sh - it reads version from airflow/version.py, it should from setup.py, also it uses airfflow/version.py in calculation of md5sum of files for which changes trigger the build (includes comment about the mechanism)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738421389
I'm not sure, but Google tried to use a similar trick and it causes problems.
https://github.com/googleapis/google-api-python-client/issues/876
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#discussion_r536046402
##########
File path: scripts/ci/libraries/_initialization.sh
##########
@@ -134,8 +134,8 @@ function initialization::initialize_base_variables() {
# otherwise it will use files/airflow-breeze-config/init.sh
export INIT_SCRIPT_FILE=${INIT_SCRIPT_FILE:=""}
- # Read airflow version from the version.py
- AIRFLOW_VERSION=$(grep version "${AIRFLOW_SOURCES}/airflow/version.py" | awk '{print $3}' | sed "s/['+]//g")
+ # Read airflow version from the setup.py.
+ AIRFLOW_VERSION=$(python "${AIRFLOW_SOURCES}/setup.py" --version)
Review comment:
Of course that all untill we rewrite breeze in python - which might change the whole approach: #12282
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#discussion_r535679509
##########
File path: scripts/ci/libraries/_initialization.sh
##########
@@ -134,8 +134,8 @@ function initialization::initialize_base_variables() {
# otherwise it will use files/airflow-breeze-config/init.sh
export INIT_SCRIPT_FILE=${INIT_SCRIPT_FILE:=""}
- # Read airflow version from the version.py
- AIRFLOW_VERSION=$(grep version "${AIRFLOW_SOURCES}/airflow/version.py" | awk '{print $3}' | sed "s/['+]//g")
+ # Read airflow version from the setup.py.
+ AIRFLOW_VERSION=$(python "${AIRFLOW_SOURCES}/setup.py" --version)
Review comment:
> # scripts/ci/libraries/_initialization.sh: line 138: python: command not found
Failed anyway :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738304448
> scripts/ci/libraries/_initialization.sh - it reads version from airflow/version.py, it should from setup.py
We could use `python setup.py --version`, but do we have enough of a python environment wherever that is run? I feel we should, all it needs is setuptools.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738304448
> scripts/ci/libraries/_initialization.sh - it reads version from airflow/version.py, it should from setup.py
We could use `python setup.py --version`, but do we have enough of a python environment wherever that is run? I feel we should, all it needs is setuptools.
Pushed a fixup with that -- lets see what happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738350135
> > scripts/ci/libraries/_initialization.sh - it reads version from airflow/version.py, it should from setup.py
>
> We could use `python setup.py --version`, but do we have enough of a python environment wherever that is run? I feel we should, all it needs is setuptools.
>
> Pushed a fixup with that -- lets see what happens.
As commented - i'd prefer base it on having POSIX compliant OS simply.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738742950
> > I'm not sure, but Google tried to use a similar trick and it causes problems.
> > [googleapis/google-api-python-client#876](https://github.com/googleapis/google-api-python-client/issues/876)
>
> That seems specific to using Pyinstaller
Yeah. I would not worry about it. At worst, it might only cause problems in the dev environment. If we find some, we can always catch any exceptions and make a workaround for dev-only env.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
ashb merged pull request #12786:
URL: https://github.com/apache/airflow/pull/12786
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12786: Get airflow version from importlib.metadata rather than hard-coding
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12786:
URL: https://github.com/apache/airflow/pull/12786#issuecomment-738109274
We should fix it in few other places:*
* scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh -> it can be removed, it was needed because setup.py imported it before
* Dockerfile.ci -> ditto
* STATIC_CODE_CHECKS.rst has reference to removed pre-commit rule
* dev/README.md needs to be updated
* dev/README_RELEASE_AIRLFOW.md as well
* dev/provider_packages/enter_breeze_provider_package_tests.sh also can be removed as it was needed only for setup.py
* scripts/ci/libraries/_initialization.sh - it reads version from airflow/version.py, it should from setup.py, also it uses airfflow/version.py in calculation of md5sum of files for which changes trigger the build (includes comment about the mechanism)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org