You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/24 19:29:18 UTC
[GitHub] [airflow] hsrocks opened a new pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
hsrocks opened a new pull request #21073:
URL: https://github.com/apache/airflow/pull/21073
**related: #21008 and #21012**
This change is to implement the operators and hooks for AWS Session manager parameter store to get the value of single parameter , multiple parameters by path, and to add new parameter and its value to parameter store
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1060027119
hey @hsrocks - sorry for the delay, I think you should take a look how Hashicorp Vault is implemented. It has very similar story. And what we came up with is:
Internal Client that is reused by both Hook and Secret Manager:
* https://github.com/apache/airflow/blob/main/airflow/providers/hashicorp/_internal_client/vault_client.py
Hook (which is a thin wrapper over the client):
* https://github.com/apache/airflow/blob/main/airflow/providers/hashicorp/hooks/vault.py
Secrets Manager (which is a thin wrapper over the client):
* https://github.com/apache/airflow/blob/main/airflow/providers/hashicorp/secrets/vault.py
Duplication is minimized (and we've already implemented some features in client that were automatically available to both SecretManager and Hook (see the history of "vault_client.py" - proving that this pattern is really good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsrocks closed pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
hsrocks closed pull request #21073:
URL: https://github.com/apache/airflow/pull/21073
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1020464575
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsrocks commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
hsrocks commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1060691109
> hey @hsrocks - sorry for the delay, I think you should take a look how Hashicorp Vault is implemented. It has very similar story. And what we came up with is:
>
> Internal Client that is reused by both Hook and Secret Manager:
>
> * https://github.com/apache/airflow/blob/main/airflow/providers/hashicorp/_internal_client/vault_client.py
>
> Hook (which is a thin wrapper over the client):
>
> * https://github.com/apache/airflow/blob/main/airflow/providers/hashicorp/hooks/vault.py
>
> Secrets Manager (which is a thin wrapper over the client):
>
> * https://github.com/apache/airflow/blob/main/airflow/providers/hashicorp/secrets/vault.py
>
> Duplication is minimized (and we've already implemented some features in client that were automatically available to both SecretManager and Hook (see the history of "vault_client.py" - proving that this pattern is really good.
Thanks for the Awesome pointers @potiuk ! Appreciate the level of details. Will go through them and implement this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1039184870
Would you please rebase the change to latest main (note the new "Update with rebase" feature in the UI - this way you can do it very easily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsrocks commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
hsrocks commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1044777473
> I would be curious to learn how these hooks and operators are related to the already existing logic available in Systems Manager Backend in Amazon Provider package: `airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend`
>
> I wonder if we should leverage these hooks in the Backend class now? it seems like some code is repeated in these 2?
>
> CC: @potiuk @o-nikolas
Thanks for review @dbarrundiag !! While I am not sure on the reason but I think they want to ensure segregation between hooks/operators and secrets module. But if thats not true the we can use secrets_manager hook in this class as well: airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dbarrundiag commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
dbarrundiag commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1044692338
I would be curious to learn how these hooks and operators are related to the already existing logic available in Systems Manager Backend in Amazon Provider package: `airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend`
I wonder if we should leverage these hooks in the Backend class now? it seems like some code is repeated in these 2?
CC: @potiuk @o-nikolas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#discussion_r800100710
##########
File path: airflow/providers/amazon/aws/hooks/ssm_parameter_store.py
##########
@@ -0,0 +1,107 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class SSMParameterStoreHook(AwsBaseHook):
+ """
+ Interact with Amazon SSM Service.
+
+ Additional arguments (such as ``aws_conn_id``) may be specified and
+ are passed down to the underlying AwsBaseHook.
+
+ .. see also::
+ :class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(client_type='ssm', *args, **kwargs)
+
+ def get_parameter(self, parameter_name: str, with_decryption: bool):
+ """
+ Retrieve parameter value from AWS Systems Manager Parameter Store.
+ Reflecting format it stored in the AWS Systems Manager Parameter Store
+ :param parameter_name: name of the parameter
+ :param with_decryption: Return decrypted values for secure string parameters
+ :return: str with the value of the Systems Manager Parameter
+ :rtype: str
+ """
+ ssm_client = self.get_conn()
+ try:
+ get_parameter_response = ssm_client.get_parameter(
+ Name=parameter_name, WithDecryption=with_decryption
+ )
+ if 'Parameter' in get_parameter_response:
+ self.log.info("Returning value of the parameter")
+ return get_parameter_response['Parameter']['Value']
+ except ssm_client.exceptions.ParameterNotFound:
+ self.log.error('Parameter Does Not Exist')
+ raise
+ except Exception as general_error:
+ self.log.error("Failed to list SSM Parameter, error: %s", general_error)
+ raise
+
+ def get_parameters_by_path(self, path: str, **parameter_kwargs):
+ """
+ Retrieve parameter value from AWS Systems Manager Parameter Store.
+ Reflecting format it stored in the AWS Systems Manager Parameter Store
+ :param path: hierarchy for the parameters
+ :param parameter_kwargs: Keyword args that define the configurations used to retrieve the parameter
+ :return: dict with the value of the Systems Manager Parameter
+ :rtype: dict
+ """
+ ssm_client = self.get_conn()
+ try:
+ get_parameter_by_path_response = ssm_client.get_parameters_by_path(
+ Path=path,
+ **parameter_kwargs,
+ )
+ if 'Parameters' in get_parameter_by_path_response:
+ self.log.info("Returning dictionary with the list of parameters")
+ return get_parameter_by_path_response
+ except Exception as general_error:
+ self.log.error("Failed to list SSM Parameter, error: %s", general_error)
Review comment:
Line 59 and Line 81 has the same error message, maybe something different to distinguish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsrocks commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
hsrocks commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1052315802
I went through the code to analyse the changes and need input.
The secret class uses profile_name also as an optional to load credentials. but SSMParameterStoreHook uses AWSBaseHook which don't accept profile_name as an argument or part of any config(please feel free to correct me if I am mistaken but I cant find it as part of AWSBaseHook class) Because it uses aws_conn_id param which can get connection from profile, aws credential etc (https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html) . So how the config will be loaded for SystemsManagerParameterStoreBackend if we switch to SSMParameterStoreHook ? Will it be with aws_conn_id? Then will it impact loaded of configurations and connection object ?
Also, I am not completely sure on this, but how the secrets are loaded vs Hooks are loaded /evaluated will there be any impact in sense of variable loading when we switch secret classes to use Hook?
Please help me with above information . Thanks!
CC: @potiuk @o-nikolas @dbarrundiag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1039185523
Also fixing static checks alongside the rebase would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsrocks commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
hsrocks commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1042573777
> Also fixing static checks alongside the rebase would be great.
Will do . Can see static check failure is not related to SSM changes but will relook it again and try to fix it. Thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1020464575
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsrocks commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
hsrocks commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1030144797
Updated the operators and hooks for SSM as per latest code changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsrocks commented on a change in pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
hsrocks commented on a change in pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#discussion_r800138933
##########
File path: airflow/providers/amazon/aws/hooks/ssm_parameter_store.py
##########
@@ -0,0 +1,107 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class SSMParameterStoreHook(AwsBaseHook):
+ """
+ Interact with Amazon SSM Service.
+
+ Additional arguments (such as ``aws_conn_id``) may be specified and
+ are passed down to the underlying AwsBaseHook.
+
+ .. see also::
+ :class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(client_type='ssm', *args, **kwargs)
+
+ def get_parameter(self, parameter_name: str, with_decryption: bool):
+ """
+ Retrieve parameter value from AWS Systems Manager Parameter Store.
+ Reflecting format it stored in the AWS Systems Manager Parameter Store
+ :param parameter_name: name of the parameter
+ :param with_decryption: Return decrypted values for secure string parameters
+ :return: str with the value of the Systems Manager Parameter
+ :rtype: str
+ """
+ ssm_client = self.get_conn()
+ try:
+ get_parameter_response = ssm_client.get_parameter(
+ Name=parameter_name, WithDecryption=with_decryption
+ )
+ if 'Parameter' in get_parameter_response:
+ self.log.info("Returning value of the parameter")
+ return get_parameter_response['Parameter']['Value']
+ except ssm_client.exceptions.ParameterNotFound:
+ self.log.error('Parameter Does Not Exist')
+ raise
+ except Exception as general_error:
+ self.log.error("Failed to list SSM Parameter, error: %s", general_error)
+ raise
+
+ def get_parameters_by_path(self, path: str, **parameter_kwargs):
+ """
+ Retrieve parameter value from AWS Systems Manager Parameter Store.
Review comment:
Thanks @subkanthi . Fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsrocks commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
hsrocks commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1050533091
@potiuk @o-nikolas Can you please review the request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#discussion_r800100710
##########
File path: airflow/providers/amazon/aws/hooks/ssm_parameter_store.py
##########
@@ -0,0 +1,107 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class SSMParameterStoreHook(AwsBaseHook):
+ """
+ Interact with Amazon SSM Service.
+
+ Additional arguments (such as ``aws_conn_id``) may be specified and
+ are passed down to the underlying AwsBaseHook.
+
+ .. see also::
+ :class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(client_type='ssm', *args, **kwargs)
+
+ def get_parameter(self, parameter_name: str, with_decryption: bool):
+ """
+ Retrieve parameter value from AWS Systems Manager Parameter Store.
+ Reflecting format it stored in the AWS Systems Manager Parameter Store
+ :param parameter_name: name of the parameter
+ :param with_decryption: Return decrypted values for secure string parameters
+ :return: str with the value of the Systems Manager Parameter
+ :rtype: str
+ """
+ ssm_client = self.get_conn()
+ try:
+ get_parameter_response = ssm_client.get_parameter(
+ Name=parameter_name, WithDecryption=with_decryption
+ )
+ if 'Parameter' in get_parameter_response:
+ self.log.info("Returning value of the parameter")
+ return get_parameter_response['Parameter']['Value']
+ except ssm_client.exceptions.ParameterNotFound:
+ self.log.error('Parameter Does Not Exist')
+ raise
+ except Exception as general_error:
+ self.log.error("Failed to list SSM Parameter, error: %s", general_error)
+ raise
+
+ def get_parameters_by_path(self, path: str, **parameter_kwargs):
+ """
+ Retrieve parameter value from AWS Systems Manager Parameter Store.
+ Reflecting format it stored in the AWS Systems Manager Parameter Store
+ :param path: hierarchy for the parameters
+ :param parameter_kwargs: Keyword args that define the configurations used to retrieve the parameter
+ :return: dict with the value of the Systems Manager Parameter
+ :rtype: dict
+ """
+ ssm_client = self.get_conn()
+ try:
+ get_parameter_by_path_response = ssm_client.get_parameters_by_path(
+ Path=path,
+ **parameter_kwargs,
+ )
+ if 'Parameters' in get_parameter_by_path_response:
+ self.log.info("Returning dictionary with the list of parameters")
+ return get_parameter_by_path_response
+ except Exception as general_error:
+ self.log.error("Failed to list SSM Parameter, error: %s", general_error)
Review comment:
Line 59 and Line 81 has the same error message, maybe something different to distinguish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
subkanthi commented on a change in pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#discussion_r800100429
##########
File path: airflow/providers/amazon/aws/hooks/ssm_parameter_store.py
##########
@@ -0,0 +1,107 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class SSMParameterStoreHook(AwsBaseHook):
+ """
+ Interact with Amazon SSM Service.
+
+ Additional arguments (such as ``aws_conn_id``) may be specified and
+ are passed down to the underlying AwsBaseHook.
+
+ .. see also::
+ :class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(client_type='ssm', *args, **kwargs)
+
+ def get_parameter(self, parameter_name: str, with_decryption: bool):
+ """
+ Retrieve parameter value from AWS Systems Manager Parameter Store.
+ Reflecting format it stored in the AWS Systems Manager Parameter Store
+ :param parameter_name: name of the parameter
+ :param with_decryption: Return decrypted values for secure string parameters
+ :return: str with the value of the Systems Manager Parameter
+ :rtype: str
+ """
+ ssm_client = self.get_conn()
+ try:
+ get_parameter_response = ssm_client.get_parameter(
+ Name=parameter_name, WithDecryption=with_decryption
+ )
+ if 'Parameter' in get_parameter_response:
+ self.log.info("Returning value of the parameter")
+ return get_parameter_response['Parameter']['Value']
+ except ssm_client.exceptions.ParameterNotFound:
+ self.log.error('Parameter Does Not Exist')
+ raise
+ except Exception as general_error:
+ self.log.error("Failed to list SSM Parameter, error: %s", general_error)
+ raise
+
+ def get_parameters_by_path(self, path: str, **parameter_kwargs):
+ """
+ Retrieve parameter value from AWS Systems Manager Parameter Store.
Review comment:
This looks like a copy/paste error, should this be retrieve parameter value by path maybe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #21073: Feature : Implement a AWS Session manager Parameter Store Operators and hooks
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on pull request #21073:
URL: https://github.com/apache/airflow/pull/21073#issuecomment-1051288667
> @potiuk @o-nikolas Can you please review the request
Hey folks,
Sorry, it's been on my ToDo list to dive into this PR, but I just haven't had the time. But at a high level, re: the earlier discussion, I see no reason on our end why their should be any duplicated code. If we have a new hook that provides the same functionality needed by the manager backend, we should reuse that code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org