You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/21 09:04:23 UTC
[GitHub] [airflow] hrishik opened a new issue #21008: AWS Systems Manager Parameter store support for Airflow AWS Provider package
hrishik opened a new issue #21008:
URL: https://github.com/apache/airflow/issues/21008
### Description
Airflow DAGs when used in AWS context, the parameter sharing across different subsystems becomes tedious job. There is a need to create a central parameter handling repository that can be accessed by any service provider (Such as AWS Service) outside of Airflow DAG.
### Use case/motivation
One important part of DAG based orchestration is sharing of metadata and configuration across different service providers.
Existing alternative of using XCom feature of Airflow has its own limitations in terms of security, encryption, extensibility etc.,
AWS Systems Manager parameter store provides the parameter sharing mechanism, and it is directly supported by all AWS services including lambda, glue, cloud-formation etc. It is an important addition to Airflow to develop the operator for AWS Systems Manager parameter store to standardize information sharing across different services and Airflow DAG seamlessly.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21008: AWS Systems Manager Parameter store support for Airflow AWS Provider package
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21008:
URL: https://github.com/apache/airflow/issues/21008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21008: AWS Systems Manager Parameter store support for Airflow AWS Provider package
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21008:
URL: https://github.com/apache/airflow/issues/21008#issuecomment-1018351371
The SystemsManagerParameterStoreBackend is already available in Airflow as custom Secrets manager:
https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/secrets/systems_manager/index.html#airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hrishik edited a comment on issue #21008: AWS Systems Manager Parameter store support for Airflow AWS Provider package
Posted by GitBox <gi...@apache.org>.
hrishik edited a comment on issue #21008:
URL: https://github.com/apache/airflow/issues/21008#issuecomment-1018397257
> The SystemsManagerParameterStoreBackend is already available in Airflow as custom Secrets manager:
>
> https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/secrets/systems_manager/index.html#airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
The existing operator only provides a read for specific actions i.e. connection URI, Config, and custom variable pertaining to airflow information. I am talking about using SSM parameter store as metadata storage to share configuration/metadata across different subsystems, as well as airflow. This means all actions including setting and getting parameters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21008: AWS Systems Manager Parameter store support for Airflow AWS Provider package
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21008:
URL: https://github.com/apache/airflow/issues/21008#issuecomment-1018412316
I don't thik Airlflow will never have "AWS" or "Google" or any other "specific" configuration built in in other way that current "customization options". Airflow is "Cloud Agnostic".
It provides features to "talk" to external systems (via Providers: Hoops, Operators, Sensors) and airflow configuration capabilities can be extended precisely via Custom Secrets Backend.
If you also want to use SystemsManagerParameter for sharing Inter-tasks metadata - feel free to add and contribute custom XCom Backend using https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html#custom-backends in the Amazon Provider. But that does not require a separate issue or agreement, just contribute it to the AWS provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hrishik commented on issue #21008: AWS Systems Manager Parameter store support for Airflow AWS Provider package
Posted by GitBox <gi...@apache.org>.
hrishik commented on issue #21008:
URL: https://github.com/apache/airflow/issues/21008#issuecomment-1018397257
> The SystemsManagerParameterStoreBackend is already available in Airflow as custom Secrets manager:
>
> https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/secrets/systems_manager/index.html#airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
The existing operator only provides a read for specific actions i.e. connection URI, Config, and custom variable pertaining to airflow information. I am talking about using SSM parameter store as metadata storage to share configuration/metadata across different subsystems, as well as airflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21008: AWS Systems Manager Parameter store support for Airflow AWS Provider package
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21008:
URL: https://github.com/apache/airflow/issues/21008#issuecomment-1018312576
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org