You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2021/03/03 11:49:50 UTC
[airflow] branch v2-0-test updated (478c726 -> 6941434)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v2-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
discard 478c726 By default PIP will install all packages in .local folder (#14125)
new 6941434 By default PIP will install all packages in .local folder (#14125)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (478c726)
\
N -- N -- N refs/heads/v2-0-test (6941434)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
airflow/kubernetes/refresh_config.py | 1 +
airflow/models/dagbag.py | 2 +-
airflow/providers/amazon/aws/log/cloudwatch_task_handler.py | 1 +
airflow/providers/amazon/aws/log/s3_task_handler.py | 1 +
airflow/providers/google/cloud/operators/functions.py | 1 +
airflow/providers/microsoft/azure/log/wasb_task_handler.py | 2 ++
airflow/serialization/serialized_objects.py | 1 +
airflow/stats.py | 1 +
airflow/www/views.py | 1 +
9 files changed, 10 insertions(+), 1 deletion(-)
[airflow] 01/01: By default PIP will install all packages in .local
folder (#14125)
Posted by po...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v2-0-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6941434064784d063627ca57686f5857e33fbf0f
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Feb 21 21:09:13 2021 +0100
By default PIP will install all packages in .local folder (#14125)
In order to optimize the Docker image, we use the ~/.local
folder copied from build imge (this gives huge optimisations
regarding the docker image size). So far we instructed the users
to add --user flag manually when installing any packages when they
extend the images, however this has proven to be problematic as
users rarely read the whole documentation and simply try what they
know.
This PR attempts to fix it. `PIP_USER` variable is set to `true`
in the final image, which means that the installation by default
will use ~/.local folder as target. This can be disabled by
unsetting the variable or setting it to `false`.
Also since pylint version has been released to 2.7.0, it fixes
a few pylint versions so that we can update to the latest constraints.
(cherry picked from commit ca35bd7f7f6bc2fb4f2afd7762114ce262c61941)
---
Dockerfile | 3 +++
airflow/kubernetes/refresh_config.py | 1 +
airflow/models/dagbag.py | 2 +-
airflow/providers/amazon/aws/log/cloudwatch_task_handler.py | 1 +
airflow/providers/amazon/aws/log/s3_task_handler.py | 1 +
airflow/providers/google/cloud/operators/functions.py | 1 +
airflow/providers/microsoft/azure/log/wasb_task_handler.py | 2 ++
airflow/serialization/serialized_objects.py | 1 +
airflow/stats.py | 1 +
airflow/www/views.py | 1 +
docs/apache-airflow/production-deployment.rst | 3 +++
11 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/Dockerfile b/Dockerfile
index a2ffd2f..04ac076 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -524,6 +524,9 @@ LABEL org.apache.airflow.distro="debian" \
org.opencontainers.image.title="Production Airflow Image" \
org.opencontainers.image.description="Installed Apache Airflow"
+# By default PIP will install everything in ~/.local
+ARG PIP_USER="true"
+ENV PIP_USER=${PIP_USER}
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]
CMD ["--help"]
diff --git a/airflow/kubernetes/refresh_config.py b/airflow/kubernetes/refresh_config.py
index 9067cb1..0004cac 100644
--- a/airflow/kubernetes/refresh_config.py
+++ b/airflow/kubernetes/refresh_config.py
@@ -68,6 +68,7 @@ class RefreshKubeConfigLoader(KubeConfigLoader):
return True
except Exception as e: # pylint: disable=W0703
logging.error(str(e))
+ return None
def refresh_api_key(self, client_configuration):
"""Refresh API key if expired"""
diff --git a/airflow/models/dagbag.py b/airflow/models/dagbag.py
index 4a96f4f..9ddf013 100644
--- a/airflow/models/dagbag.py
+++ b/airflow/models/dagbag.py
@@ -330,7 +330,7 @@ class DagBag(LoggingMixin):
if not might_contain_dag(zip_info.filename, safe_mode, current_zip_file):
# todo: create ignore list
# Don't want to spam user with skip messages
- if not self.has_logged or True:
+ if not self.has_logged:
self.has_logged = True
self.log.info(
"File %s:%s assumed to contain no DAGs. Skipping.", filepath, zip_info.filename
diff --git a/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py b/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
index c233039..49beca0 100644
--- a/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
+++ b/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
@@ -63,6 +63,7 @@ class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
'the Cloudwatch logs connection exists.',
remote_conn_id,
)
+ return None
def _render_filename(self, ti, try_number):
# Replace unsupported log group name characters
diff --git a/airflow/providers/amazon/aws/log/s3_task_handler.py b/airflow/providers/amazon/aws/log/s3_task_handler.py
index 58d5eca..5a5e01b 100644
--- a/airflow/providers/amazon/aws/log/s3_task_handler.py
+++ b/airflow/providers/amazon/aws/log/s3_task_handler.py
@@ -54,6 +54,7 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
'the S3 connection exists.',
remote_conn_id,
)
+ return None
def set_context(self, ti):
super().set_context(ti)
diff --git a/airflow/providers/google/cloud/operators/functions.py b/airflow/providers/google/cloud/operators/functions.py
index c2ea87d..4517484 100644
--- a/airflow/providers/google/cloud/operators/functions.py
+++ b/airflow/providers/google/cloud/operators/functions.py
@@ -397,6 +397,7 @@ class CloudFunctionDeleteFunctionOperator(BaseOperator):
status = e.resp.status
if status == 404:
self.log.info('The function does not exist in this project')
+ return None
else:
self.log.error('An error occurred. Exiting.')
raise e
diff --git a/airflow/providers/microsoft/azure/log/wasb_task_handler.py b/airflow/providers/microsoft/azure/log/wasb_task_handler.py
index 3f12135..a17b7c7 100644
--- a/airflow/providers/microsoft/azure/log/wasb_task_handler.py
+++ b/airflow/providers/microsoft/azure/log/wasb_task_handler.py
@@ -66,6 +66,7 @@ class WasbTaskHandler(FileTaskHandler, LoggingMixin):
'the Wasb connection exists.',
remote_conn_id,
)
+ return None
def set_context(self, ti) -> None:
super().set_context(ti)
@@ -159,6 +160,7 @@ class WasbTaskHandler(FileTaskHandler, LoggingMixin):
# return error if needed
if return_error:
return msg
+ return None
def wasb_write(self, log: str, remote_log_location: str, append: bool = True) -> None:
"""
diff --git a/airflow/serialization/serialized_objects.py b/airflow/serialization/serialized_objects.py
index 38df10a..a5e9f15 100644
--- a/airflow/serialization/serialized_objects.py
+++ b/airflow/serialization/serialized_objects.py
@@ -290,6 +290,7 @@ class BaseSerialization:
elif type_ == DAT.SET:
return {cls._deserialize(v) for v in var}
elif type_ == DAT.TUPLE:
+ # pylint: disable=consider-using-generator
return tuple([cls._deserialize(v) for v in var])
else:
raise TypeError(f'Invalid type {type_!s} in deserialization.')
diff --git a/airflow/stats.py b/airflow/stats.py
index 2797afd..7472647 100644
--- a/airflow/stats.py
+++ b/airflow/stats.py
@@ -243,6 +243,7 @@ class AllowListValidator:
def __init__(self, allow_list=None):
if allow_list:
+ # pylint: disable=consider-using-generator
self.allow_list = tuple([item.strip().lower() for item in allow_list.split(',')])
else:
self.allow_list = None
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 4560197..b7c5372 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -3641,6 +3641,7 @@ class TaskInstanceModelView(AirflowModelView):
return redirect(self.get_redirect())
except Exception: # noqa pylint: disable=broad-except
flash('Failed to clear task instances', 'error')
+ return None
@provide_session
def set_task_instance_state(self, tis, target_state, session=None):
diff --git a/docs/apache-airflow/production-deployment.rst b/docs/apache-airflow/production-deployment.rst
index bea4f42..042b655 100644
--- a/docs/apache-airflow/production-deployment.rst
+++ b/docs/apache-airflow/production-deployment.rst
@@ -193,6 +193,9 @@ You should be aware, about a few things:
FROM apache/airflow:2.0.0
RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add
+* As of 2.0.1 image the ``--user`` flag is turned on by default by setting ``PIP_USER`` environment variable
+ to ``true``. This can be disabled by un-setting the variable or by setting it to ``false``.
+
* If your apt, or PyPI dependencies require some of the build-essentials, then your best choice is
to follow the "Customize the image" route. However it requires to checkout sources of Apache Airflow,