You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/28 10:34:30 UTC
[GitHub] [airflow] potiuk opened a new pull request #19859: Add a short chapter focusing on adapting secret format for connections
potiuk opened a new pull request #19859:
URL: https://github.com/apache/airflow/pull/19859
As a result of discussion in #19857, I propose to add this short
chapter to respond to anticipated need of organisations to keep the
connections in format that is not Airflow-exclusive. I think it would
be good to explicitly state what is the Airflow approach in this case
(i.e. either using existing capabilities of secret backends when
they are there - for example in AWS - or rolling your own backend,
possibly by extending the community provided ones if the flexibility
is not implemented by the community provided backend.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19859: Add a short chapter focusing on adapting secret format for connections
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19859:
URL: https://github.com/apache/airflow/pull/19859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a change in pull request #19859: Add a short chapter focusing on adapting secret format for connections
Posted by GitBox <gi...@apache.org>.
dstandish commented on a change in pull request #19859:
URL: https://github.com/apache/airflow/pull/19859#discussion_r758595096
##########
File path: docs/apache-airflow/security/secrets/secrets-backend/index.rst
##########
@@ -111,3 +111,19 @@ See :ref:`Configuration <secrets_backend_configuration>` for more details, and :
If you are rolling your own secrets backend, you don't strictly need to use airflow's URI format. But
doing so makes it easier to switch between environment variables, the metastore, and your secrets backend.
+
+Adapt to non-Airflow compatible secret formats for connections
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The default implementation of Secret backend requires to use Airflow-specific format of storing
+secrets for connections. Currently most community provided implementations require the connections to
+be stored as URIs (with the possibility of adding more friendly formats in the future)
+:doc:`apache-airflow-providers:core-extensions/secrets-backends`. However in some cases it might be
+better solution for your organization to keep the credentials (passwords/tokens etc) in other formats -
+for example when you want the same credentials to be used across multiple clients or when you want to
+use built-in mechanism of rotating the credentials that do not work well with the Airflow-specific format.
+Some of the secret backends provided by the community (for example
+:py:class:`~airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend` have
+mechanism where you can configure the retrieval in a flexible way, but others might not provide that much of
+flexibility. In this case you should simply roll your own secret backend as described in the previous chapter,
+possibly extending existing secret backend and adapt it to the scheme used by your organization.
Review comment:
```suggestion
The default implementation of Secret backend requires use of an Airflow-specific format of storing
secrets for connections. Currently most community provided implementations require the connections to
be stored as URIs (with the possibility of adding more friendly formats in the future)
:doc:`apache-airflow-providers:core-extensions/secrets-backends`. However some organizations may prefer
to keep the credentials (passwords/tokens etc) in other formats --
for example when you want the same credentials to be used across multiple clients, or when you want to
use built-in mechanism of rotating the credentials that do not work well with the Airflow-specific format.
In this case you will need to roll your own secret backend as described in the previous chapter,
possibly extending existing secret backend and adapt it to the scheme used by your organization.
```
Mainly I have suggested removing the `Some of the secret backends provided by the community...` paragraph because it's a bit unclear and its presence makes it a little ambiguous which case is referred to in `In this case...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19859: Add a short chapter focusing on adapting secret format for connections
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19859:
URL: https://github.com/apache/airflow/pull/19859#issuecomment-981865369
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org