You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2021/07/12 20:31:41 UTC
[airflow-site] branch main updated: Publish warning about
installing packages dynamically (#450)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/main by this push:
new 2c887ee Publish warning about installing packages dynamically (#450)
2c887ee is described below
commit 2c887ee6fbd22e1e58feb71e386487d92688392e
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Mon Jul 12 22:31:35 2021 +0200
Publish warning about installing packages dynamically (#450)
---
.../docker-stack/_sources/entrypoint.rst.txt | 21 ++++++++++++++++++--
docs-archive/docker-stack/build.html | 4 ++--
docs-archive/docker-stack/entrypoint.html | 23 ++++++++++++++++++++--
docs-archive/docker-stack/searchindex.js | 2 +-
4 files changed, 43 insertions(+), 7 deletions(-)
diff --git a/docs-archive/docker-stack/_sources/entrypoint.rst.txt b/docs-archive/docker-stack/_sources/entrypoint.rst.txt
index c386a67..9438314 100644
--- a/docs-archive/docker-stack/_sources/entrypoint.rst.txt
+++ b/docs-archive/docker-stack/_sources/entrypoint.rst.txt
@@ -262,11 +262,28 @@ and Admin role. They also forward local port ``8080`` to the webserver port and
Installing additional requirements
..................................
+.. warning:: Installing requirements this way is a very convenient method of running Airflow, very useful for
+ testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in
+ production environment. We have deliberately chose to make it a development/test dependency and we print
+ a warning, whenever it is used. There is an inherent security-related issue with using this method in
+ production. Installing the requirements this way can happen at literally any time - when your containers
+ get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen
+ literally any time. This opens you up to a serious vulnerability where your production environment
+ might be brought down by a single dependency being removed from PyPI - or even dependency of your
+ dependency. This means that you put your production service availability in hands of 3rd-party developers.
+ At any time, any moment including weekends and holidays those 3rd party developers might bring your
+ production Airflow instance down, without you even knowing it. This is a serious vulnerability that
+ is similar to the infamous
+ `leftpad <https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/>`_
+ problem. You can fully protect against this case by building your own, immutable custom image, where the
+ dependencies are baked in. You have been warned.
+
Installing additional requirements can be done by specifying ``_PIP_ADDITIONAL_REQUIREMENTS`` variable.
The variable should contain a list of requirements that should be installed additionally when entering
the containers. Note that this option slows down starting of Airflow as every time any container starts
-it must install new packages. Therefore this option should only be used for testing. When testing is
-finished, you should create your custom image with dependencies baked in.
+it must install new packages and it opens up huge potential security vulnerability when used in production
+(see below). Therefore this option should only be used for testing. When testing is finished,
+you should create your custom image with dependencies baked in.
Example:
diff --git a/docs-archive/docker-stack/build.html b/docs-archive/docker-stack/build.html
index 27cf1a4..4d165cb 100644
--- a/docs-archive/docker-stack/build.html
+++ b/docs-archive/docker-stack/build.html
@@ -1283,7 +1283,7 @@ to provide this library from you repository if you want to build Airflow image i
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt <span class="o">||</span> <span class="nb">true</span>
curl -Lo <span class="s2">"docker-context-files/constraints-3.7.txt"</span> <span class="se">\</span>
- https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt
+ https://raw.githubusercontent.com/apache/airflow/constraints-2.1.0/constraints-3.7.txt
<span class="c1"># For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.</span>
pip install <span class="nv">pip</span><span class="o">==</span><span class="m">20</span>.2.4
@@ -1323,7 +1323,7 @@ to the below:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker build . <span class="se">\</span>
--build-arg <span class="nv">PYTHON_BASE_IMAGE</span><span class="o">=</span><span class="s2">"python:3.7-slim-buster"</span> <span class="se">\</span>
--build-arg <span class="nv">AIRFLOW_INSTALLATION_METHOD</span><span class="o">=</span><span class="s2">"apache-airflow"</span> <span class="se">\</span>
- --build-arg <span class="nv">AIRFLOW_VERSION</span><span class="o">=</span><span class="s2">"2.0.2"</span> <span class="se">\</span>
+ --build-arg <span class="nv">AIRFLOW_VERSION</span><span class="o">=</span><span class="s2">"2.1.0"</span> <span class="se">\</span>
--build-arg <span class="nv">INSTALL_MYSQL_CLIENT</span><span class="o">=</span><span class="s2">"false"</span> <span class="se">\</span>
--build-arg <span class="nv">AIRFLOW_PRE_CACHED_PIP_PACKAGES</span><span class="o">=</span><span class="s2">"false"</span> <span class="se">\</span>
--build-arg <span class="nv">INSTALL_FROM_DOCKER_CONTEXT_FILES</span><span class="o">=</span><span class="s2">"true"</span> <span class="se">\</span>
diff --git a/docs-archive/docker-stack/entrypoint.html b/docs-archive/docker-stack/entrypoint.html
index 771dda6..0d5f3c3 100644
--- a/docs-archive/docker-stack/entrypoint.html
+++ b/docs-archive/docker-stack/entrypoint.html
@@ -805,11 +805,30 @@ and Admin role. They also forward local port <code class="docutils literal notra
</div>
<div class="section" id="installing-additional-requirements">
<h3>Installing additional requirements<a class="headerlink" href="#installing-additional-requirements" title="Permalink to this headline">ΒΆ</a></h3>
+<div class="admonition warning">
+<p class="admonition-title">Warning</p>
+<p>Installing requirements this way is a very convenient method of running Airflow, very useful for
+testing and debugging. However, do not be tricked by its convenience. You should never, ever use it in
+production environment. We have deliberately chose to make it a development/test dependency and we print
+a warning, whenever it is used. There is an inherent security-related issue with using this method in
+production. Installing the requirements this way can happen at literally any time - when your containers
+get restarted, when your machines in K8S cluster get restarted. In a K8S Cluster those events can happen
+literally any time. This opens you up to a serious vulnerability where your production environment
+might be brought down by a single dependency being removed from PyPI - or even dependency of your
+dependency. This means that you put your production service availability in hands of 3rd-party developers.
+At any time, any moment including weekends and holidays those 3rd party developers might bring your
+production Airflow instance down, without you even knowing it. This is a serious vulnerability that
+is similar to the infamous
+<a class="reference external" href="https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/">leftpad</a>
+problem. You can fully protect against this case by building your own, immutable custom image, where the
+dependencies are baked in. You have been warned.</p>
+</div>
<p>Installing additional requirements can be done by specifying <code class="docutils literal notranslate"><span class="pre">_PIP_ADDITIONAL_REQUIREMENTS</span></code> variable.
The variable should contain a list of requirements that should be installed additionally when entering
the containers. Note that this option slows down starting of Airflow as every time any container starts
-it must install new packages. Therefore this option should only be used for testing. When testing is
-finished, you should create your custom image with dependencies baked in.</p>
+it must install new packages and it opens up huge potential security vulnerability when used in production
+(see below). Therefore this option should only be used for testing. When testing is finished,
+you should create your custom image with dependencies baked in.</p>
<p>Example:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run -it -p <span class="m">8080</span>:8080 <span class="se">\</span>
--env <span class="s2">"_PIP_ADDITIONAL_REQUIREMENTS=lxml==4.6.3 charset-normalizer==1.4.1"</span> <span class="se">\</span>
diff --git a/docs-archive/docker-stack/searchindex.js b/docs-archive/docker-stack/searchindex.js
index 4ef34de..454396f 100644
--- a/docs-archive/docker-stack/searchindex.js
+++ b/docs-archive/docker-stack/searchindex.js
@@ -1 +1 @@
-Search.setIndex({docnames:["build","build-arg-ref","entrypoint","index","recipes"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":2,"sphinx.domains.rst":2,"sphinx.domains.std":1,"sphinx.ext.intersphinx":1,"sphinx.ext.viewcode":1,sphinx:56},filenames:["build.rst","build-arg-ref.rst","entrypoint.rst","index.rst","recipes.rs [...]
\ No newline at end of file
+Search.setIndex({docnames:["build","build-arg-ref","entrypoint","index","recipes"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":2,"sphinx.domains.rst":2,"sphinx.domains.std":1,"sphinx.ext.intersphinx":1,"sphinx.ext.viewcode":1,sphinx:56},filenames:["build.rst","build-arg-ref.rst","entrypoint.rst","index.rst","recipes.rs [...]
\ No newline at end of file