[airflow] branch master updated: Disables provider's manager warning for source-installed prod image. (#13729)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
     new f74da50  Disables provider's manager warning for source-installed prod image. (#13729)
f74da50 is described below

commit f74da5025d9f635bc49e631883fc1537cd16b620
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Mon Jan 18 09:00:32 2021 +0100

    Disables provider's manager warning for source-installed prod image. (#13729)
    
    When production image is built for development purpose, by default
    it installs all providers from sources, but not all dependencies
    are installed for all providers. Many providers require more
    dependencies and when you try to import those packages via
    provider's manager, they fail to import and print warnings.
    
    Those warnings are now turned into debug messages, in case
    AIRFLOW_INSTALLATION_METHOD=".", which is set when
    production image is built locally from sources. This is helpful
    especially when you use locally build production image to
    run K8S tests - otherwise the logs are flooded with
    warnings.
    
    This problem does not happe in CI, because there by default
    production image is built from locally prepared packages
    and it does not contain sources from providers that are not
    installed via packages.
---
 Dockerfile                   |  5 +++++
 airflow/providers_manager.py | 27 +++++++++++++++++++++------
 2 files changed, 26 insertions(+), 6 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index 496fa20..813b001 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -464,6 +464,11 @@ EXPOSE 8080
 
 USER ${AIRFLOW_UID}
 
+# Having the variable in final image allows to disable providers manager warnings when
+# production image is prepared from sources rather than from package
+ARG AIRFLOW_INSTALLATION_METHOD="apache-airflow"
+ENV AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD}
+
 ARG BUILD_ID
 ENV BUILD_ID=${BUILD_ID}
 ARG COMMIT_SHA
diff --git a/airflow/providers_manager.py b/airflow/providers_manager.py
index 02bf5e0..d3b7ffb 100644
--- a/airflow/providers_manager.py
+++ b/airflow/providers_manager.py
@@ -289,12 +289,27 @@ class ProvidersManager:
                     self._add_customized_fields(provider_package, hook_class, field_behaviours)
 
         except Exception as e:  # noqa pylint: disable=broad-except
-            log.warning(
-                "Exception when importing '%s' from '%s' package: %s",
-                hook_class_name,
-                provider_package,
-                e,
-            )
+            if os.environ.get("AIRFLOW_INSTALLATION_METHOD") != ".":
+                # print providers manager warning when airflow is not installed from sources in the
+                # production image
+                log.warning(
+                    "Exception when importing '%s' from '%s' package: %s",
+                    hook_class_name,
+                    provider_package,
+                    e,
+                )
+            else:
+                # This is special case - when airflow is installed from sources in production
+                # image, AIRFLOW_INSTALLATION_METHOD is set to ".". In this case we know that there
+                # Will be some warnings generated by ProviderManager, when it tries to import providers
+                # With missing dependencies, therefore we are turning such warnings into debug message
+                # so that we do not pollute logs.
+                log.debug(
+                    "Exception when importing '%s' from '%s' package: %s",
+                    hook_class_name,
+                    provider_package,
+                    e,
+                )
             return
 
         conn_type: str = self._get_attr(hook_class, 'conn_type')