[GitHub] [airflow] dstandish commented on a diff in pull request #23560: Add advanced secrets backend configurations

[GitBox <gi...@apache.org>]

dstandish commented on code in PR #23560:
URL: https://github.com/apache/airflow/pull/23560#discussion_r900491214


##########
airflow/configuration.py:
##########
@@ -1487,47 +1500,118 @@ def set(*args, **kwargs) -> None:
     conf.set(*args, **kwargs)
 
 
-def ensure_secrets_loaded() -> List[BaseSecretsBackend]:
-    """
-    Ensure that all secrets backends are loaded.
-    If the secrets_backend_list contains only 2 default backends, reload it.
-    """
-    # Check if the secrets_backend_list contains only 2 default backends
-    if len(secrets_backend_list) == 2:
-        return initialize_secrets_backends()
-    return secrets_backend_list
+class DefaultSecretsBackend(UserList):
+    """List Container which use for store default secrets backends."""
+
+
+@dataclass(frozen=True)
+class SecretsBackendConfig:
+    """Secrets Backend Config dataclass helper."""
+
+    backend: str
+    backend_kwargs: Dict[str, Any] = field(default_factory=dict)
+
+    @classmethod
+    def from_dict(cls, d: Dict[str, Any]) -> 'SecretsBackendConfig':
+        """
+        Read Secret Backend Config from dictionary
+
+        Ignores all unexpected keywords
+        """
+        return cls(d['backend'], d.get('backend_kwargs', {}))
 
+    @classmethod
+    def from_config(cls) -> Optional['SecretsBackendConfig']:
+        """Try to get ``SecretsBackendConfig`` from airflow config [secrets] section"""
+        secrets_backend = conf.get(section='secrets', key='backend', fallback=None)
 
-def get_custom_secret_backend() -> Optional[BaseSecretsBackend]:
-    """Get Secret Backend if defined in airflow.cfg"""
-    secrets_backend_cls = conf.getimport(section='secrets', key='backend')
+        if not secrets_backend:
+            return None
 
-    if secrets_backend_cls:
         try:
-            backends: Any = conf.get(section='secrets', key='backend_kwargs', fallback='{}')
-            alternative_secrets_config_dict = json.loads(backends)
+            secrets_backend_kwargs: Any = conf.get(section='secrets', key='backend_kwargs', fallback='{}')
+            secrets_backend_kwargs = json.loads(secrets_backend_kwargs)
         except JSONDecodeError:
-            alternative_secrets_config_dict = {}
+            secrets_backend_kwargs = {}
+
+        return cls(secrets_backend, secrets_backend_kwargs)
+
+    def initialize(self) -> BaseSecretsBackend:
+        """Initialize Secrets Backend."""
+        return import_string(self.backend)(**self.backend_kwargs)
+
+
+class UniqueSecretsBackendsConfigs(UserList):

Review Comment:
   do we really need this? if user provides duplicate backends, can we just let it be however they've configured it?



##########
airflow/config_templates/default_airflow.cfg:
##########
@@ -422,6 +422,13 @@ backend =
 # ``{{"connections_prefix": "/airflow/connections", "profile_name": "default"}}``
 backend_kwargs =
 
+# Advanced secrets backend configuration, allow to user configure more than one secret backend,

Review Comment:
   same comment as elsewhere; better have just one way then "a way" and "an advanced way"



##########
airflow/configuration.py:
##########
@@ -1487,47 +1500,118 @@ def set(*args, **kwargs) -> None:
     conf.set(*args, **kwargs)
 
 
-def ensure_secrets_loaded() -> List[BaseSecretsBackend]:
-    """
-    Ensure that all secrets backends are loaded.
-    If the secrets_backend_list contains only 2 default backends, reload it.
-    """
-    # Check if the secrets_backend_list contains only 2 default backends
-    if len(secrets_backend_list) == 2:
-        return initialize_secrets_backends()
-    return secrets_backend_list
+class DefaultSecretsBackend(UserList):
+    """List Container which use for store default secrets backends."""
+
+
+@dataclass(frozen=True)
+class SecretsBackendConfig:

Review Comment:
   if we provide only one way of configuring, and we "upgrade" legacy config for backward compatibility, maybe we don't need this either?



##########
docs/apache-airflow/security/secrets/secrets-backend/index.rst:
##########
@@ -101,8 +143,8 @@ Roll your own secrets backend
 A secrets backend is a subclass of :py:class:`airflow.secrets.BaseSecretsBackend` and must implement either
 :py:meth:`~airflow.secrets.BaseSecretsBackend.get_connection` or :py:meth:`~airflow.secrets.BaseSecretsBackend.get_conn_value`.
 
-After writing your backend class, provide the fully qualified class name in the ``backend`` key in the ``[secrets]``
-section of ``airflow.cfg``.
+After writing your backend class, provide the fully qualified class name in the ``backend`` key or provide

Review Comment:
   rather than supporting two ways of configuration, I think it would be better to just support one way and deprecate the old way.  if you look at AirflowConfigParser, you'll find things concerning "deprecated_values" and "upgraded_values"
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org