You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "giftig (via GitHub)" <gi...@apache.org> on 2023/04/21 15:03:11 UTC

[GitHub] [superset] giftig opened a new issue, #23774: Dependency clash between tox and elasticsearch-dbapi

giftig opened a new issue, #23774:
URL: https://github.com/apache/superset/issues/23774

   When attempting to compile the dependencies for the project with the elasticsearch variant included, `pip-compile-multi` points out a dependency clash between `elasticsearch-dbapi==0.2.9` (from `.[elasticsearch]`) and `tox` (from `integration.in`) and requests version-locking a dependency to resolve it.
   
   ```
   Locking requirements/cip.in to requirements/cip.txt. References: ['requirements/base.in', 'requirements/development.in']
   Package packaging was resolved to different versions in different environments: 21.3 and 23.1
   Traceback (most recent call last):
     File "/usr/local/lib/python3.8/site-packages/pipcompilemulti/cli_v1.py", line 26, in cli
       recompile()
     File "/usr/local/lib/python3.8/site-packages/pipcompilemulti/actions.py", line 31, in recompile
       compile_topologically(env_confs, deduplicator)
     File "/usr/local/lib/python3.8/site-packages/pipcompilemulti/actions.py", line 38, in compile_topologically
       if env.maybe_create_lockfile():
     File "/usr/local/lib/python3.8/site-packages/pipcompilemulti/environment.py", line 51, in maybe_create_lockfile
       self.create_lockfile()
     File "/usr/local/lib/python3.8/site-packages/pipcompilemulti/environment.py", line 72, in create_lockfile
       self.fix_lockfile()
     File "/usr/local/lib/python3.8/site-packages/pipcompilemulti/environment.py", line 134, in fix_lockfile
       sections = [
     File "/usr/local/lib/python3.8/site-packages/pipcompilemulti/environment.py", line 135, in <listcomp>
       self.fix_pin(section)
     File "/usr/local/lib/python3.8/site-packages/pipcompilemulti/environment.py", line 211, in fix_pin
       raise RuntimeError(
   RuntimeError: Please add constraints for the package version listed above
   ```
   
   #### How to reproduce the bug
   
   Add a `requirements/custom.in` with content:
   
   ```
   -r development.in
   -e file:.[elasticsearch]
   ```
   and run `pip-compile-multi`
   
   ### Expected results
   
   Dependencies to compile correctly
   
   ### Actual results
   
   The above conflict. Full trees for the two packages:
   
   ```
   name                              summary
   --------------------------------  -----------------------------------------------------------------------------
   elasticsearch-dbapi==0.2.9        A DBAPI and SQLAlchemy dialect for Elasticsearch
   ├── elasticsearch<7.14,>7         Python client for Elasticsearch
   │   ├── certifi                   Python package for providing Mozilla's CA Bundle.
   │   └── urllib3<2,>=1.21.1        HTTP library with thread-safe connection pooling, file post, and more.
   ├── packaging<22.0,>=21.0         Core utilities for Python packages
   │   └── pyparsing!=3.0.5,>=2.0.2  pyparsing module - Classes and methods to define and execute parsing grammars
   └── sqlalchemy                    Database Abstraction Library
       ├── greenlet!=0.4.17          Lightweight in-process concurrent programming
       └── typing-extensions>=4.2.0  Backported and Experimental Type Hints for Python 3.7+
   
   name                          summary
   ----------------------------  --------------------------------------------------------------------------------------------------
   tox                           tox is a generic virtualenv management and test command line tool
   ├── cachetools>=5.3           Extensible memoizing collections and decorators
   ├── chardet>=5.1              Universal encoding detector for Python 3
   ├── colorama>=0.4.6           Cross-platform colored terminal text.
   ├── filelock>=3.11            A platform independent file lock.
   ├── packaging>=23             Core utilities for Python packages
   ├── platformdirs>=3.2         A small Python package for determining appropriate platform-specific dirs, e.g. a "user data dir".
   ├── pluggy>=1                 plugin and hook calling mechanisms for python
   ├── pyproject-api>=1.5.1      API to interact with the python pyproject.toml based projects
   │   ├── packaging>=23         Core utilities for Python packages
   │   └── tomli>=2.0.1          A lil' TOML parser
   ├── tomli>=2.0.1              A lil' TOML parser
   └── virtualenv>=20.21         Virtual Python Environment builder
       ├── distlib<1,>=0.3.6     Distribution utilities
       ├── filelock<4,>=3.11     A platform independent file lock.
       └── platformdirs<4,>=3.2  A small Python package for determining appropriate platform-specific dirs, e.g. a "user data dir".
   ```
   
   ### Environment
   
   (please complete the following information):
   
   - superset version: `2.1`
   - python version: `3.10.5`
   
   ### Checklist
   
   Make sure to follow these steps before submitting your issue - thank you!
   
   - [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
   - [x] I have reproduced the issue with at least the latest released version of superset.
   - [x] I have checked the issue tracker for the same issue and I haven't found one similar.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] giftig commented on issue #23774: Dependency clash between tox and elasticsearch-dbapi

Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on issue #23774:
URL: https://github.com/apache/superset/issues/23774#issuecomment-1517996476

   Interestingly `integration.txt` was updated in Feb and tox `4.4.x` was definitely out by then, so maybe my assumptions about how you're generating your dependencies are flawed. My understanding is the `.txt` files are only overridden by running `pip-compile-multi` (I'm not very familiar with this tool) , and running that in the current state should have updated that dependency in the `.txt` file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] giftig commented on issue #23774: Dependency clash between tox and elasticsearch-dbapi

Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on issue #23774:
URL: https://github.com/apache/superset/issues/23774#issuecomment-1517976182

   it looks like the problem is that there are no restrictions on `tox` version, and the latest version now requires `packaging >= 23`, whereas the compiled `integration.txt` shows
   
   ```
   tox==3.25.1
       # via -r integration.in
   ```
   which only requires `>= 14`:
   
   ```
   tox==3.25.1                                                                                     tox is a generic virtualenv management and test command line tool
   ├── filelock>=3.0.0                                                                             A platform independent file lock.
   ├── packaging>=14                                                                               Core utilities for Python packages
   ├── pluggy>=0.12.0                                                                              plugin and hook calling mechanisms for python
   ├── py>=1.4.17                                                                                  library with cross-python path, ini-parsing, io, code, log facilities
   ├── six>=1.14.0                                                                                 Python 2 and 3 compatibility utilities
   ├── toml>=0.9.4                                                                                 Python Library for Tom's Obvious, Minimal Language
   └── virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0  Virtual Python Environment builder
       ├── distlib<1,>=0.3.6                                                                       Distribution utilities
       ├── filelock<4,>=3.11                                                                       A platform independent file lock.
       └── platformdirs<4,>=3.2                                                                    A small Python package for determining appropriate platform-specific dirs, e.g. a "user data dir".
   ```
   So version-locking this to `tox>=3.25,<4` would probably do the trick and I can raise a PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] Dependency clash between tox and elasticsearch-dbapi [superset]

Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on issue #23774:
URL: https://github.com/apache/superset/issues/23774#issuecomment-1965492488

   @giftig Checking in to see if there's anything that we'd call a bug here, or if there's anything you'd like to tackle on this front. I was otherwise tempted to close it as stale 😅


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] giftig commented on issue #23774: Dependency clash between tox and elasticsearch-dbapi

Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on issue #23774:
URL: https://github.com/apache/superset/issues/23774#issuecomment-1518029247

   I see from your looking through the docs that you're using `--no-upgrade` when invoking `pip-compile-multi`, which I left out. That would explain the above behaviour. I think it's probably still worth version-locking `tox` in `integration.in` to avoid the dependency clash with elasticsearch when looking to upgrade dependencies, but for the moment it doesn't look like this is necessarily a problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] Dependency clash between tox and elasticsearch-dbapi [superset]

Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on issue #23774:
URL: https://github.com/apache/superset/issues/23774#issuecomment-1966286404

   I think versions have moved on enough that this specific issue is no longer relevant and I'll close it. We still do have some issues with dependency management I think, but that's a bit broader.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [I] Dependency clash between tox and elasticsearch-dbapi [superset]

Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig closed issue #23774: Dependency clash between tox and elasticsearch-dbapi
URL: https://github.com/apache/superset/issues/23774


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org