You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/07 06:36:50 UTC
[GitHub] [airflow] raphaelauv opened a new issue #18789: Google Cloud Secret Manager Backend fail on variable with a non allow character
raphaelauv opened a new issue #18789:
URL: https://github.com/apache/airflow/issues/18789
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
5.1.0
### Apache Airflow version
2.1.2
### Operating System
GCP Container-Optimized OS
### Deployment
Composer
### Deployment details
_No response_
### What happened
When a variable have a non allow character like a `.` the secret manager is going to fail even if the variable is inside the airflow database itelf ( registred in the the airflow variable )
### What you expected to happen
The secret manager should catch the error and say that it can't find it
```
[2021-10-07 06:22:09,330] {secret_manager_client.py:93} ERROR - Google Cloud API Call Error (PermissionDenied): No access for Secret ID example-variables-prefix-toto.tata.
```
and then look for the airflow variable
### How to reproduce
[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
backend_kwargs = {"connections_prefix": "example-connections-prefix", "variables_prefix": "example-variables-prefix"}
"{{ var.value.get('toto.tata') }}"
throw this error :
```
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1158, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1296, in _prepare_and_execute_task_with_callbacks
self.render_templates(context=context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1796, in render_templates
self.task.render_template_fields(context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 992, in render_template_fields
self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1005, in _do_render_template_fields
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1042, in render_template
return jinja_env.from_string(content).render(**context)
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/sandbox.py", line 462, in call
return __context.call(__obj, *args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1682, in get
return Variable.get(item, default_var=default_var)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/variable.py", line 135, in get
var_val = Variable.get_variable_from_secrets(key=key)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/variable.py", line 204, in get_variable_from_secrets
var_val = secrets_backend.get_variable(key=key)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/secrets/secret_manager.py", line 154, in get_variable
return self._get_secret(self.variables_prefix, key)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/secrets/secret_manager.py", line 178, in _get_secret
return self.client.get_secret(secret_id=secret_id, project_id=self.project_id)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/_internal_client/secret_manager_client.py", line 86, in get_secret
response = self.client.access_secret_version(name)
File "/opt/python3.8/lib/python3.8/site-packages/google/cloud/secretmanager_v1/gapic/secret_manager_service_client.py", line 967, in access_secret_version
return self._inner_api_calls["access_secret_version"](
File "/opt/python3.8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
return retry_target(
File "/opt/python3.8/lib/python3.8/site-packages/google/api_core/retry.py", line 189, in retry_target
return target()
File "/opt/python3.8/lib/python3.8/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 69, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 The provided Secret Version ID [projects/XXXXXXXXXXX/secrets/example-variables-prefix-toto.tata/versions/latest] does not match the expected format [projects/*/secrets/*/versions/*]
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #18789: Google Cloud Secret Manager Backend fail on variable with a non allow character
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #18789:
URL: https://github.com/apache/airflow/issues/18789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org