You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by GitBox <gi...@apache.org> on 2019/12/21 17:32:13 UTC
[GitHub] [libcloud] Kami opened a new pull request #1393: Fix wheel METADATA
and ensure conditional dependencies are handled correctly
Kami opened a new pull request #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393
In a prevision version we added some more conditional dependencies - ``typing`` and ``enum34`` packages which are needed for Python versions older than 3.4.0.
Those versions were dynamically included as part of ``install_requirements`` so the actual ``Reqires-Dist`` metadata entry which is part of the wheel file was not correct and depended on the version of Python which was used to generate the wheel.
## Proposed Solution
This pull request fixes that by utilizing PEP 508 (https://www.python.org/dev/peps/pep-0508/#environment-markers) environment markers notation.
In addition to that, we also use ``extra_requires`` fallback notation for users who install Libcloud using old versions of pip and setuptools (reference: https://hynek.me/articles/conditional-python-dependencies/).
Keep in mind that this change now requires a Libcloud committer which is making a release to generate a wheel using a recent version of setuptools for the METADATA entries to be correct.
I updated ``setup.py`` to fail on ``bdist_wheel`` command if an older version is used. I will also document this under committer docs.
I tested it locally using various Python versions and confirmed it works correctly.
The universal wheel metadata now looks like this:
```bash
...
Requires-Dist: requests (>=2.5.0)
Requires-Dist: backports.ssl-match-hostname ; python_version < "2.7.9"
Requires-Dist: typing ; python_version < "3.4.0"
Requires-Dist: enum34 ; python_version < "3.4.0"
...
```
## Note on supporting Python <= 3.4.0
In the very near future, we plan to drop support for Python <= 3.4.0. At that point, this change won't be needed anymore and we will be able to simplify ``setup.py``.
The reason why I made this change is so we can make another release in the ``v2.x`` release series which contains the correct wheel metadata entries. This will be the last release which will support Python <= 3.4.0.
Resolves #1392.
Thanks to @arterrey for reporting this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1393: Fix wheel METADATA and
ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393#issuecomment-568201071
Ignore my "no tests" comment above.
I decided to improve the existing installation tests (c2d23d5dfb6b4abc7ed5c7008e39fae1e014a2a9) and add new ones which test wheel installation method (6ad129fcda51d91489ab979345939cda199a23de).
This means we now have automated tests which test installing the package using ``python setup.py install`` method and directly installing it from a wheel and verify that the correct dependencies are installed.
Those tests won't catch 100% of the issues (there are still some edge cases when using different Python version to build and install the wheel), but it's much better than nothing (and they would likely have caught and original issue reported by @arterrey in #1392).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] codecov-io commented on issue #1393: Fix wheel METADATA
and ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393#issuecomment-568201916
# [Codecov](https://codecov.io/gh/apache/libcloud/pull/1393?src=pr&el=h1) Report
> Merging [#1393](https://codecov.io/gh/apache/libcloud/pull/1393?src=pr&el=desc) into [trunk](https://codecov.io/gh/apache/libcloud/commit/9ce7d847c14f8fbbbff33e3fe465a9aa7f93f702?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/libcloud/pull/1393?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## trunk #1393 +/- ##
========================================
- Coverage 86.3% 86.3% -0.01%
========================================
Files 372 372
Lines 76545 76545
Branches 7009 7009
========================================
- Hits 66062 66060 -2
- Misses 7682 7683 +1
- Partials 2801 2802 +1
```
| [Impacted Files](https://codecov.io/gh/apache/libcloud/pull/1393?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [libcloud/test/compute/test\_upcloud.py](https://codecov.io/gh/apache/libcloud/pull/1393/diff?src=pr&el=tree#diff-bGliY2xvdWQvdGVzdC9jb21wdXRlL3Rlc3RfdXBjbG91ZC5weQ==) | `90.06% <0%> (-1.33%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1393?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1393?src=pr&el=footer). Last update [9ce7d84...165151d](https://codecov.io/gh/apache/libcloud/pull/1393?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1393: Fix wheel METADATA and
ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393#issuecomment-568390938
@arterrey Great, thanks for confirming.
I will work on 2.8.0 release this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1393: Fix wheel METADATA and
ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393#issuecomment-568199951
@tonybaloney @erjohnso @vdloo Just a heads up^
> Keep in mind that this change now requires a Libcloud committer which is making a release to generate a wheel using a recent version of setuptools for the METADATA entries to be correct.
>
> I updated setup.py to fail on bdist_wheel command if an older version is used. I will also document this under committer docs.
I don't think it's likely that one of you will do another release in the ``v2.x`` series before we drop support for Python < 3.4.0, but in case you do, you need to use latest version of ``setuptools`` to ensure generated wheel contains correct metadata.
And on that note - I plan to do v2.8.0 once @arterrey confirms this fix is working. This will likely be the last major / minor release in the v2.x release series before dropping support for Python < 3.4.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1393: Fix wheel METADATA and
ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393#issuecomment-568199665
@arterrey I believe this should fix the issue you reported in #1392.
I had quite some trouble testing it using ``buildout`` since latest version of buildout wheel plugin is broken and doesn't work with the latest version of ``wheel`` packages (so I couldn't test it using the locally build wheel from this branch).
In the end, I got a bit creative and confirmed it's working by building an egg from this branch locally and using this ``buildout.cfg``:
```ini
[buildout]
parts = libcloud
[libcloud]
recipe = zc.recipe.egg
eggs = apache-libcloud
index =
find-links = .
interpreter = py
```
```bash
rm -rf eggs/*
python setup.py bdist_egg
cp dist/apache_libcloud-2.7.0-py3.6.egg eggs/
```
I confirmed it now doesn't try to download typing and enum34 packages when using Python >= 3.4.0.
Having said that, I would also appreciate if you could confirm the fix as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami edited a comment on issue #1393: Fix wheel METADATA
and ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
Kami edited a comment on issue #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393#issuecomment-568201071
Ignore my "no tests" comment above.
I decided to improve the existing installation tests (c2d23d5dfb6b4abc7ed5c7008e39fae1e014a2a9) and add new ones which test wheel installation method (6ad129fcda51d91489ab979345939cda199a23de, 165151dcc2787c7ec9975d728cc977338b5aad99).
This means we now have automated tests which test installing the package using ``python setup.py install`` method and directly installing it from a wheel and verify that the correct dependencies are installed.
Those tests won't catch 100% of the issues (there are still some edge cases when using different Python version to build and install the wheel), but it's much better than nothing (and they would likely have caught and original issue reported by @arterrey in #1392).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami commented on issue #1393: Fix wheel METADATA and
ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
Kami commented on issue #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393#issuecomment-568471837
@arterrey EDIT: From looking at the output you provided again, it looks like it used an egg from an old release (2.6.0) which you likely had cached locally.
To confirm the fix, you will probably need to delete rest of the libcloud eggs from ``eggs/`` directory, run ``python setup.py bdist_egg`` copy egg from ``dist/`` to ``eggs/`` and then run the buildout command (at least that's how I tested it).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami edited a comment on issue #1393: Fix wheel METADATA
and ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
Kami edited a comment on issue #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393#issuecomment-568199665
@arterrey I believe this should fix the issue you reported in #1392.
I had quite some trouble testing it using ``buildout`` since latest version of buildout wheel plugin is broken and doesn't work with the latest version of ``wheel`` packages (so I couldn't test it using the locally build wheel from this branch).
In the end, I got a bit creative and confirmed it's working by building an egg from this branch locally and using this ``buildout.cfg`` (both using Python 2.7 and Python 3):
```ini
[buildout]
parts = libcloud
[libcloud]
recipe = zc.recipe.egg
eggs = apache-libcloud
index =
find-links = .
```
```bash
rm -rf eggs/*
python setup.py bdist_egg
# For Python 3 venv
cp dist/apache_libcloud-2.7.0-py3.6.egg eggs/
# For Python 2 venv
cp dist/apache_libcloud-2.7.0-py2.7.egg eggs/
```
I confirmed it now doesn't try to download typing and enum34 packages when using Python >= 3.4.0 (and it tries to download them when using Python 2.7).
Having said that, I would also appreciate if you could confirm the fix as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] arterrey commented on issue #1393: Fix wheel METADATA
and ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
arterrey commented on issue #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393#issuecomment-568322190
Yes this is working now. Your awesome @Kami , thank you for looking into and fixing this so quickly.
---
`bdist_wheel` the repo and placed in test dir
used the following `buildout.cfg`:
```
[buildout]
parts = libcloud
[libcloud]
recipe = zc.recipe.egg
eggs = apache-libcloud
index =
find-links = .
interpreter = py
```
```
$ python3 -venv .
$ bin/pip install -U pip setuptools zc.buildout
...
$ bin/buildout
Creating directory '/home/adam/tmp/libcloud/test/eggs'.
Creating directory '/home/adam/tmp/libcloud/test/parts'.
Creating directory '/home/adam/tmp/libcloud/test/develop-eggs'.
Getting distribution for 'zc.recipe.egg>=2.0.6'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
Got zc.recipe.egg 2.0.7.
Installing libcloud.
Not found: /apache-libcloud/
Not found: /apache-libcloud/
Getting distribution for 'apache-libcloud'.
Got apache-libcloud 2.6.0.
Not found: /requests/
Not found: /requests/
Getting distribution for 'requests>=2.5.0'.
Got requests 2.22.0.
Not found: /urllib3/
Not found: /urllib3/
Getting distribution for 'urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1'.
Got urllib3 1.25.7.
Not found: /idna/
Not found: /idna/
Getting distribution for 'idna<2.9,>=2.5'.
Got idna 2.8.
Not found: /chardet/
Not found: /chardet/
Getting distribution for 'chardet<3.1.0,>=3.0.2'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
warning: no files found matching 'requirements.txt'
zip_safe flag not set; analyzing archive contents...
Got chardet 3.0.4.
Not found: /certifi/
Not found: /certifi/
Getting distribution for 'certifi>=2017.4.17'.
Got certifi 2019.9.11.
Generated interpreter '/home/adam/tmp/libcloud/test/bin/py'.
$ bin/py
>>> import enum
>>> enum.__file__
'/usr/lib64/python3.7/enum.py'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [libcloud] Kami merged pull request #1393: Fix wheel METADATA and
ensure conditional dependencies are handled correctly
Posted by GitBox <gi...@apache.org>.
Kami merged pull request #1393: Fix wheel METADATA and ensure conditional dependencies are handled correctly
URL: https://github.com/apache/libcloud/pull/1393
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services