You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2021/07/12 20:11:03 UTC
[airflow-site] branch
release-warning-about-dynamic-package-installing-in-docs created (now
9784683)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch release-warning-about-dynamic-package-installing-in-docs
in repository https://gitbox.apache.org/repos/asf/airflow-site.git.
at 9784683 Publish warning about installing packages dynamically
This branch includes the following new commits:
new 9784683 Publish warning about installing packages dynamically
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[airflow-site] 01/01: Publish warning about installing packages
dynamically
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch release-warning-about-dynamic-package-installing-in-docs
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
commit 9784683becd9efdc828556ade3f70904c51541b8
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jul 12 22:09:44 2021 +0200
Publish warning about installing packages dynamically
---
.../docker-stack/_sources/entrypoint.rst.txt | 21 ++++++++++++++++++--
docs-archive/docker-stack/build.html | 4 ++--
docs-archive/docker-stack/entrypoint.html | 23 ++++++++++++++++++++--
docs-archive/docker-stack/searchindex.js | 2 +-
4 files changed, 43 insertions(+), 7 deletions(-)
diff --git a/docs-archive/docker-stack/_sources/entrypoint.rst.txt b/docs-archive/docker-stack/_sources/entrypoint.rst.txt
index c386a67..9438314 100644
--- a/docs-archive/docker-stack/_sources/entrypoint.rst.txt
+++ b/docs-archive/docker-stack/_sources/entrypoint.rst.txt
@@ -262,11 +262,28 @@ and Admin role. They also forward local port ``8080`` to the webserver port and
Installing additional requirements
..................................
+.. warning:: Installing requirements this way is a very convenient method of running Airflow, very useful for
+ testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in
+ production environment. We have deliberately chose to make it a development/test dependency and we print
+ a warning, whenever it is used. There is an inherent security-related issue with using this method in
+ production. Installing the requirements this way can happen at literally any time - when your containers
+ get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen
+ literally any time. This opens you up to a serious vulnerability where your production environment
+ might be brought down by a single dependency being removed from PyPI - or even dependency of your
+ dependency. This means that you put your production service availability in hands of 3rd-party developers.
+ At any time, any moment including weekends and holidays those 3rd party developers might bring your
+ production Airflow instance down, without you even knowing it. This is a serious vulnerability that
+ is similar to the infamous
+ `leftpad <https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/>`_
+ problem. You can fully protect against this case by building your own, immutable custom image, where the
+ dependencies are baked in. You have been warned.
+
Installing additional requirements can be done by specifying ``_PIP_ADDITIONAL_REQUIREMENTS`` variable.
The variable should contain a list of requirements that should be installed additionally when entering
the containers. Note that this option slows down starting of Airflow as every time any container starts
-it must install new packages. Therefore this option should only be used for testing. When testing is
-finished, you should create your custom image with dependencies baked in.
+it must install new packages and it opens up huge potential security vulnerability when used in production
+(see below). Therefore this option should only be used for testing. When testing is finished,
+you should create your custom image with dependencies baked in.
Example:
diff --git a/docs-archive/docker-stack/build.html b/docs-archive/docker-stack/build.html
index 27cf1a4..4d165cb 100644
--- a/docs-archive/docker-stack/build.html
+++ b/docs-archive/docker-stack/build.html
@@ -1283,7 +1283,7 @@ to provide this library from you repository if you want to build Airflow image i
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt <span class="o">||</span> <span class="nb">true</span>
curl -Lo <span class="s2">"docker-context-files/constraints-3.7.txt"</span> <span class="se">\</span>
- https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt
+ https://raw.githubusercontent.com/apache/airflow/constraints-2.1.0/constraints-3.7.txt
<span class="c1"># For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.</span>
pip install <span class="nv">pip</span><span class="o">==</span><span class="m">20</span>.2.4
@@ -1323,7 +1323,7 @@ to the below:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker build . <span class="se">\</span>
--build-arg <span class="nv">PYTHON_BASE_IMAGE</span><span class="o">=</span><span class="s2">"python:3.7-slim-buster"</span> <span class="se">\</span>
--build-arg <span class="nv">AIRFLOW_INSTALLATION_METHOD</span><span class="o">=</span><span class="s2">"apache-airflow"</span> <span class="se">\</span>
- --build-arg <span class="nv">AIRFLOW_VERSION</span><span class="o">=</span><span class="s2">"2.0.2"</span> <span class="se">\</span>
+ --build-arg <span class="nv">AIRFLOW_VERSION</span><span class="o">=</span><span class="s2">"2.1.0"</span> <span class="se">\</span>
--build-arg <span class="nv">INSTALL_MYSQL_CLIENT</span><span class="o">=</span><span class="s2">"false"</span> <span class="se">\</span>
--build-arg <span class="nv">AIRFLOW_PRE_CACHED_PIP_PACKAGES</span><span class="o">=</span><span class="s2">"false"</span> <span class="se">\</span>
--build-arg <span class="nv">INSTALL_FROM_DOCKER_CONTEXT_FILES</span><span class="o">=</span><span class="s2">"true"</span> <span class="se">\</span>
diff --git a/docs-archive/docker-stack/entrypoint.html b/docs-archive/docker-stack/entrypoint.html
index 771dda6..0d5f3c3 100644
--- a/docs-archive/docker-stack/entrypoint.html
+++ b/docs-archive/docker-stack/entrypoint.html
@@ -805,11 +805,30 @@ and Admin role. They also forward local port <code class="docutils literal notra
</div>
<div class="section" id="installing-additional-requirements">
<h3>Installing additional requirements<a class="headerlink" href="#installing-additional-requirements" title="Permalink to this headline">ΒΆ</a></h3>
+<div class="admonition warning">
+<p class="admonition-title">Warning</p>
+<p>Installing requirements this way is a very convenient method of running Airflow, very useful for
+testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in
+production environment. We have deliberately chose to make it a development/test dependency and we print
+a warning, whenever it is used. There is an inherent security-related issue with using this method in
+production. Installing the requirements this way can happen at literally any time - when your containers
+get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen
+literally any time. This opens you up to a serious vulnerability where your production environment
+might be brought down by a single dependency being removed from PyPI - or even dependency of your
+dependency. This means that you put your production service availability in hands of 3rd-party developers.
+At any time, any moment including weekends and holidays those 3rd party developers might bring your
+production Airflow instance down, without you even knowing it. This is a serious vulnerability that
+is similar to the infamous
+<a class="reference external" href="https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/">leftpad</a>
+problem. You can fully protect against this case by building your own, immutable custom image, where the
+dependencies are baked in. You have been warned.</p>
+</div>
<p>Installing additional requirements can be done by specifying <code class="docutils literal notranslate"><span class="pre">_PIP_ADDITIONAL_REQUIREMENTS</span></code> variable.
The variable should contain a list of requirements that should be installed additionally when entering
the containers. Note that this option slows down starting of Airflow as every time any container starts
-it must install new packages. Therefore this option should only be used for testing. When testing is
-finished, you should create your custom image with dependencies baked in.</p>
+it must install new packages and it opens up huge potential security vulnerability when used in production
+(see below). Therefore this option should only be used for testing. When testing is finished,
+you should create your custom image with dependencies baked in.</p>
<p>Example:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run -it -p <span class="m">8080</span>:8080 <span class="se">\</span>
--env <span class="s2">"_PIP_ADDITIONAL_REQUIREMENTS=lxml==4.6.3 charset-normalizer==1.4.1"</span> <span class="se">\</span>
diff --git a/docs-archive/docker-stack/searchindex.js b/docs-archive/docker-stack/searchindex.js
index 4ef34de..454396f 100644
--- a/docs-archive/docker-stack/searchindex.js
+++ b/docs-archive/docker-stack/searchindex.js
@@ -1 +1 @@
-Search.setIndex({docnames:["build","build-arg-ref","entrypoint","index","recipes"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":2,"sphinx.domains.rst":2,"sphinx.domains.std":1,"sphinx.ext.intersphinx":1,"sphinx.ext.viewcode":1,sphinx:56},filenames:["build.rst","build-arg-ref.rst","entrypoint.rst","index.rst","recipes.rs [...]
\ No newline at end of file
+Search.setIndex({docnames:["build","build-arg-ref","entrypoint","index","recipes"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":2,"sphinx.domains.rst":2,"sphinx.domains.std":1,"sphinx.ext.intersphinx":1,"sphinx.ext.viewcode":1,sphinx:56},filenames:["build.rst","build-arg-ref.rst","entrypoint.rst","index.rst","recipes.rs [...]
\ No newline at end of file