You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/07/29 07:17:41 UTC
[GitHub] [airflow] NBardelot opened a new issue #10038: Using >= or =~ in dependency management is a very bad practice
NBardelot opened a new issue #10038:
URL: https://github.com/apache/airflow/issues/10038
From an ops point of view, packaging Airflow with about 150 dependencies not strictly tied to a version is a bad practice. In `setup.py` there are about 20 dependencies using `=~` versions and about 130 dependencies using `>= ` versions. This means there is no guarantee that two instances of Airflow run the same verions of python libraries.
Being forced to manually compute dozens of dependency versions and force them in requirements.txt is not a viable solution.
Airflow should freeze its dependency tree when a new stable version comes out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10038: Using >= or =~ in dependency management is a very bad practice
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10038:
URL: https://github.com/apache/airflow/issues/10038#issuecomment-665568153
Just two more things if you are using the image.
You can take a look at the IMAGES.rst where the case about constraints and how to use them is very well explained with examples.
Also you can watch my talk from the Airflow Summit where I am explaining how the image works, how it is layered and how to build your own image. The part about constraints is not very detailed but you can watch all the other aspects such as adding your own dependencies - including build and runtime dependencies : https://youtu.be/wDr3Y7q2XoI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #10038: Using >= or =~ in dependency management is a very bad practice
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #10038:
URL: https://github.com/apache/airflow/issues/10038#issuecomment-665128754
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #10038: Using >= or =~ in dependency management is a very bad practice
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #10038:
URL: https://github.com/apache/airflow/issues/10038
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] NBardelot commented on issue #10038: Using >= or =~ in dependency management is a very bad practice
Posted by GitBox <gi...@apache.org>.
NBardelot commented on issue #10038:
URL: https://github.com/apache/airflow/issues/10038#issuecomment-665510347
Thank you @kaxil and @potiuk for such insightful and speedy answers.
The issue I think is that in my case we do not install Airflow from the command line using pypi, but we use a Docker image.
I see that you changed the constraints behaviour in the Dockerfile 9 days ago @potiuk, I did not see this commit in the version I currently use.
https://github.com/apache/airflow/commit/de9eaeb434747897a192ef31815fbdd519e29c4d
With this commit I no longer have an issue from a Docker perspective.
Again, thank you both for your help and for the nice work :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10038: Using >= or =~ in dependency management is a very bad practice
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10038:
URL: https://github.com/apache/airflow/issues/10038#issuecomment-671089649
Is there anything left to do in this ticket? We have docs about it: https://airflow.readthedocs.io/en/latest/installation.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10038: Using >= or =~ in dependency management is a very bad practice
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10038:
URL: https://github.com/apache/airflow/issues/10038#issuecomment-665131906
Hello Noel,
Indeed the problem you describe is very true for Applications but it is
quite the opposite for libraries. Unfortunately Airflow is a bit of both,
that's why we had to find a bit different solution - keeping the setup.py
dependencies as open as possible but allowing Airflow as application to be
installed with frozen set of dependencies. Those are frozen both - for
master (we automatically test and re-freeze them continuously after testing
during our CI process) but also we freeze the fixed set of dependencies
whenever we release a stable version of Airflow (and you can very easily
use either the latest or the 1.10.x frozen version at any time by providing
the right `--constraint` URL.
You can take a look at our 'Installimg from PyPI' chapter of README.md -
https://github.com/apache/airflow#installing-from-pypi where it is
explained how to install Airflow in repeatable way (using constraint files
we maintain automatically with all the dependencies frozen). Also
INSTALL.txt contains the right instructions (note that they changed a bit
in the upcoming version but you can find the corresponding instructions in
1.10.10 and 1.10.11 tags of the same files - for 1.10.10 and 1.10.12. the
current set of instructions will be valid as of 1.10.12)
For detailed instructions and explanation why we had gone that route, you
can see
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies
(also the following chapter 'Pinned constraint files' where we explained
how we addressed it). As mentioned - it's the second iteration of approach
to fix it. The first one in 1.10.10 and 1.10.11 has proven to be too
'heavy' for maintenance by contributors, so we improved it, simplified it,
and we fully automated the process of dependency freezing last week (and it
will be released in 1.10.12).
I hope this addresses your concerns.
Please read those links I sent and let us know if it does or does not
address your concerns (and let
Is know please why - if not). Also if you have any concrete proposals how
we can improve the current setup without breaking our 'both library and
app' requirement - we are happy to hear it. Whether it is the documentation
or the process improvement proposals.
I am particularly interested in how we can surface information about it
better. It seems that despite all our efforts to explain it in INSTALL.txt,
README.md and CONTRIBUTING.rst - it's not enough as people sometimes do not
find it easily, so I would love to hear some suggestions and proposals how
we could improve it - especially from person not familiar with the process
and whole reasoning.
The most welcome is a PR improving it that we can review and merge :).
J
wt., 28 lip 2020, 16:57 użytkownik Noël Bardelot <no...@github.com>
napisał:
> From an ops point of view, packaging Airflow with about 150 dependencies
> not strictly tied to a version is a bad practice. In setup.py there are
> about 20 dependencies using =~ versions and about 130 dependencies using
> >= versions. This means there is no guarantee that two instances of
> Airflow run the same verions of python libraries.
>
> Being forced to manually compute dozens of dependency versions and force
> them in requirements.txt is not a viable solution.
>
> Airflow should freeze its dependency tree when a new stable version comes
> out.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/10038>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAERMI3VCH6PZ5ASZEYDYV3R53RMNANCNFSM4PKTPFRQ>
> f
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #10038: Using >= or =~ in dependency management is a very bad practice
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #10038:
URL: https://github.com/apache/airflow/issues/10038#issuecomment-665128754
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org