You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/12 07:42:30 UTC
[GitHub] [airflow] potiuk opened a new pull request #16935: Adds warning about using dynamic installatio of packages
potiuk opened a new pull request #16935:
URL: https://github.com/apache/airflow/pull/16935
While we are supporting installing packages dynamically in our
helm chart and docker compose while testing, this method is
inherently insecure in production environments (it opens up for
an attack where removing dependency of a dependency migh bring
the Airflow deployment down).
Added explanation about it and explicit warning against this.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #16935: Adds warning about using dynamic installation of packages
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #16935:
URL: https://github.com/apache/airflow/pull/16935#issuecomment-878207603
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #16935: Adds warning about using dynamic installation of packages
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #16935:
URL: https://github.com/apache/airflow/pull/16935#discussion_r668062860
##########
File path: docs/docker-stack/entrypoint.rst
##########
@@ -262,11 +262,28 @@ and Admin role. They also forward local port ``8080`` to the webserver port and
Installing additional requirements
..................................
+.. warning:: Installing requirements this way is a very convenient method of running Airflow, very useful for
+ testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in
+ production environment. We have deliberately chose to make it a development/test dependency and we print
+ a warning, whenever it is used. There is an inherent security-related issue with using this method in
+ production. Installing the requirements this way can happen at literally any time - when your containers
+ get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen
+ literally any time. This opens you up to a serious vulnerability where your production environment
+ might be brought down by a single dependency being removed from PyPI - or even dependency of your
Review comment:
Libraries are not removed very often, but are updated very often, and the new version may not be compatible, thus destroying the environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #16935: Adds warning about using dynamic installation of packages
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #16935:
URL: https://github.com/apache/airflow/pull/16935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #16935: Adds warning about using dynamic installation of packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #16935:
URL: https://github.com/apache/airflow/pull/16935#discussion_r668168272
##########
File path: docs/docker-stack/entrypoint.rst
##########
@@ -262,11 +262,28 @@ and Admin role. They also forward local port ``8080`` to the webserver port and
Installing additional requirements
..................................
+.. warning:: Installing requirements this way is a very convenient method of running Airflow, very useful for
+ testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in
+ production environment. We have deliberately chose to make it a development/test dependency and we print
+ a warning, whenever it is used. There is an inherent security-related issue with using this method in
+ production. Installing the requirements this way can happen at literally any time - when your containers
+ get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen
+ literally any time. This opens you up to a serious vulnerability where your production environment
+ might be brought down by a single dependency being removed from PyPI - or even dependency of your
Review comment:
Yeah. But you can protect from that by specifying `==version` (which makes an easy counter-argument here like "always specify all dependencies and make them with `==`").
However. there is no way to protect even if you specify `==version` and someone removes that version. Which already happened to Airflow at least once that I rememeber. And this is exactly what happened with leftpad.
I really think that it's the "removal" of dependency which is far more disastrous and the REAL reason why we should never use this variable in production.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16935: Adds warning about using dynamic installatio of packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16935:
URL: https://github.com/apache/airflow/pull/16935#issuecomment-878052739
Hey @thesuperzapper -> while explaining this on Slack, I realised that using dynamically installing packages is even more dangerours and disruptive for production services. We are going to add explicit warning in our docker stack documentation explaining it and I'd recommend you the same in your chart.
It's quite irresponsible pattern actually, trading setup convenience with long-term security and stability of your production instance. A lot of poeple do not realise that (I only realised full extent of it when I thought a bit about it and looked at the past cases that actually happened that would have brought down production installations of Airflow if this pattern has been used).
I think it would be good if we educate our users (I am going to explicitly mention this parameter and why it is dangerous in our talk this week with @kaxil ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org