You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/19 08:31:45 UTC
[GitHub] [airflow] narcotis opened a new issue, #25812: Remote logging for stackdriver error
narcotis opened a new issue, #25812:
URL: https://github.com/apache/airflow/issues/25812
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-google==8.3.0
### Apache Airflow version
airflow==2.3.2-python3.10
### Operating System
Debian GNU/Linux 11
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
I'm using Official Airflow Helm chart on GKE, with nothing customized.
All of the ENVs (configs) are based on Official Documents.
### What happened
I want to use remote logging with stackdrivertaskhandler (cloud logging), but it's not working.
These are my configs set
```
- name: "AIRFLOW__PROVIDERS_GOOGLE__VERBOSE_LOGGING"
value: "False"
// AIRFLOW CONFIG with extra_vars didn't work on remote_logging, so I deleted.
- name: "AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT"
value: 'google-cloud-platform://'
- name: "AIRFLOW__LOGGING__REMOTE_LOGGING"
value: "True"
// the Official documents says to write like `stackdriver://lab-flow-logs`, but by the code, we need to set like `stackdriver:///lab-flow-logs`
// https://github.com/apache/airflow/blob/providers-google/8.3.0/airflow/config_templates/airflow_local_settings.py#L248
- name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER"
value: "stackdriver:///lab-flow-logs"
// to set project_id with ADC. There's no part to set project_id on remote_logging to stackdriver.
- name: "GOOGLE_CLOUD_PROJECT"
value: "lab-flow-12345"
```
Other configs or codes are not customized.
On cloud logging, I got logs produced, and logName was unset as I wanted.
```
{
"textPayload": "[2022-08-19 06:02:48,830] {taskinstance.py:1159} INFO - Dependencies all met for <TaskInstance: kube_client_test.kube_api_test manual__2022-08-19T06:02:29.499371+00:00 [queued]>",
"insertId": "1pw1b3qf49uql4",
"resource": {
"type": "global",
"labels": {
"project_id": "lab-flow-12345"
}
},
"timestamp": "2022-08-19T06:02:48.830833Z",
"severity": "INFO",
"labels": {
"task_id": "kube_api_test",
"execution_date": "2022-08-19T06:02:29.499371+00:00",
"dag_id": "kube_client_test",
"python_logger": "airflow.task",
"try_number": "1"
},
"logName": "projects/lab-flow-12345/logs/task",
"receiveTimestamp": "2022-08-19T06:02:49.165762959Z"
}
```
The logName is set to `task` which is the log handler name. (perhaps)
https://github.com/apache/airflow/blob/providers-google/8.3.0/airflow/config_templates/airflow_local_settings.py#L249-L256
And remote logs are also not showing from the webserver UI.
I already set all the roles needed on the serviceaccount.
```
Python version: 3.10.5
Airflow version: 2.3.2
Node: lab-flow-helm-webserver-7d6b95b4fb-zc47p
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/airflow/.local/lib/python3.10/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "Permission 'logging.views.access' denied on resource (or it may not exist)."
debug_error_string = "{"created":"@1660897100.187647048","description":"Error received from peer ipv4:142.250.76.138:443","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Permission 'logging.views.access' denied on resource (or it may not exist).","grpc_status":7}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/airflow/.local/lib/python3.10/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/airflow/.local/lib/python3.10/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/airflow/.local/lib/python3.10/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/airflow/.local/lib/python3.10/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/airflow/.local/lib/python3.10/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/www/auth.py", line 43, in decorated
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/www/decorators.py", line 80, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/www/views.py", line 1474, in get_logs_with_metadata
logs, metadata = task_log_reader.read_log_chunks(ti, try_number, metadata)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/log/log_reader.py", line 57, in read_log_chunks
logs, metadatas = self.log_handler.read(ti, try_number, metadata=metadata)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/cloud/log/stackdriver_task_handler.py", line 200, in read
messages, end_of_log, next_page_token = self._read_logs(log_filter, next_page_token, all_pages)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/cloud/log/stackdriver_task_handler.py", line 259, in _read_logs
new_messages, next_page_token = self._read_single_logs_page(
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/cloud/log/stackdriver_task_handler.py", line 297, in _read_single_logs_page
response = self._logging_service_client.list_log_entries(request=request)
File "/home/airflow/.local/lib/python3.10/site-packages/google/cloud/logging_v2/services/logging_service_v2/client.py", line 848, in list_log_entries
response = rpc(
File "/home/airflow/.local/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
return wrapped_func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
return retry_target(
File "/home/airflow/.local/lib/python3.10/site-packages/google/api_core/retry.py", line 190, in retry_target
return target()
File "/home/airflow/.local/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.PermissionDenied: 403 Permission 'logging.views.access' denied on resource (or it may not exist). [reason: "IAM_PERMISSION_DENIED"
domain: "iam.googleapis.com"
metadata {
key: "permission"
value: "logging.views.access"
}
]
```
The above log is from airflow UI.
### What you think should happen instead
As I think, the logName should be set properly through the remote logging base log config.
(should be like `stackdriver://logs-name`, not `stackdriver:///logs-name')
Also, `AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT` ENV should be working with remote logging logics.
At least, I think the documentation for airflow providers google should be more detail.
Of course, setting the appropriate logName for cloud logging is most important. (showing on the airflow UI of course)
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #25812: Remote logging error for stackdriver
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25812: Remote logging error for stackdriver
URL: https://github.com/apache/airflow/issues/25812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25812: Remote logging for stackdriver error
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25812:
URL: https://github.com/apache/airflow/issues/25812#issuecomment-1220400581
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org