You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/19 18:29:07 UTC
[GitHub] [airflow] yehoshuadimarsky opened a new issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
yehoshuadimarsky opened a new issue #20408:
URL: https://github.com/apache/airflow/issues/20408
### Official Helm Chart version
1.3.0 (latest released)
### Apache Airflow version
2.2.1
### Kubernetes Version
1.19.14-gke.1900
### Helm Chart configuration
```yaml
executor: "KubernetesExecutor"
postgresql:
enabled: false
pgbouncer:
enabled: true
flower:
enabled: false
config:
core:
load_examples: 'False'
load_default_connections: 'False'
webserver:
expose_config: 'False'
logging:
remote_logging: 'True'
remote_log_conn_id: "gcs-conn-dev"
remote_base_log_folder: "gs://[REDACTED]/airflow_logs"
cleanup:
enabled: true
dags:
gitSync:
enabled: true
repo: ssh://git@github.com/[REDACTED]/[REDACTED].git
branch: airflow-dev
rev: HEAD
depth: 1
subPath: "airflow/dags"
sshKeySecret: airflow-git-ssh-secret
knownHosts: |
github.com ssh-rsa [REDACTED]==
```
### Docker Image customisations
Dockerfile
```Dockerfile
FROM apache/airflow:2.2.1-python3.9
SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
USER root
RUN apt-get update \
&& apt-get upgrade \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
USER airflow
COPY airflow/requirements.txt .
RUN pip install --upgrade --no-cache-dir -r requirements.txt && rm requirements.txt
```
requirements.txt
```requirements.txt
apache-airflow-providers-google==6.0
apache-airflow-providers-cncf-kubernetes==2.0
pandas==1.3
quandl==3.6.1
```
### What happened
I have simple tasks running with simple logs, sent to GCS via remote logging. But they don't show up in GCS.
### What you expected to happen
I expect logs to be visible but they are not. I cannot set up a PMV in my k8s cluster, so instead I chose to use remote logging to persist the logs in GCS. I have verified that the permissions are correct and even tested it out. But whenever any task runs, no logs appear in GCS. So of course, when I click "Log" in the task afterwards, I get this, because the worker pod is already deleted, and GCS never got the logs shipped to it.
```
*** Unable to read remote log from gs://[REDACTED]/airflow_logs/logging_test/list_gcp_bucket_objects_in_dev/2021-12-19T17:44:14.483757+00:00/1.log
*** 404 GET https://storage.googleapis.com/download/storage/v1/b/[REDACTED]/o/airflow_logs%2Flogging_test%2Flist_gcp_bucket_objects_in_dev%2F2021-12-19T17%3A44%3A14.483757%2B00%3A00%2F1.log?alt=media: No such object: [REDACTED]/airflow_logs/logging_test/list_gcp_bucket_objects_in_dev/2021-12-19T17:44:14.483757+00:00/1.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
*** Trying to get logs (last 100 lines) from worker pod loggingtestlistgcpbucketobjectsindev.6c24f1cc7ffe45a88c54afedeb ***
*** Unable to fetch logs from worker pod loggingtestlistgcpbucketobjectsindev.6c24f1cc7ffe45a88c54afedeb ***
(404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'de670d54-6f87-4ce5-90f8-a6c161d70fe2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Sun, 19 Dec 2021 18:16:45 GMT', 'Content-Length': '294'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \\"loggingtestlistgcpbucketobjectsindev.6c24f1cc7ffe45a88c54afedeb\\" not found","reason":"NotFound","details":{"name":"loggingtestlistgcpbucketobjectsindev.6c24f1cc7ffe45a88c54afedeb","kind":"pods"},"code":404}\n'
```
### How to reproduce
You will need a GCP account which I obviously cannot provide to reproduce.
Set up the Helm chart with the `values.yaml` override I provided.
Then run this Dag, I even added a long `time.sleep(30)` in case it needs some time to ship the logs before the pod is killed, but this didn't work.
```python
import logging
import time
from airflow.models.dag import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.google.cloud.hooks.gcs import GCSHook
from airflow.utils import dates
BUCKET ="some-redacted-gcs-bucket"
def ping_gcp(gcp_conn_id, bucket):
logging.debug("this is a test debug log")
logging.info("starting logging task")
logging.warning("this is a test warning log")
hook = GCSHook(gcp_conn_id=gcp_conn_id)
objs = hook.list(bucket)
logging.info(f"Here are the objects in GCS bucket: {objs}")
time.sleep(30)
default_args = {
"owner": "Barton Avenue",
"depends_on_past": False,
"email_on_failure": False,
"email_on_retry": False,
"retries": 1,
"start_date": dates.days_ago(1),
}
dag = DAG("logging_test", default_args=default_args)
task = PythonOperator(
dag=dag,
task_id="list_gcp_bucket_objects_in_dev",
python_callable=ping_gcp,
op_kwargs=dict(gcp_conn_id="gcs-conn-dev", bucket=BUCKET),
)
```
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yehoshuadimarsky edited a comment on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
yehoshuadimarsky edited a comment on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-997484801
I see. The docs are confusing in that regard then, I thought the `remote_log_conn_id` is used for both reading and writing - for all operations for remote logging.
So maybe I can broaden my question then - why are there 2 different connection IDs, `remote_log_conn_id` and `google_key_path`? Maybe the GCP provider package should change the remote logging to use `remote_log_conn_id` for both read and write?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yehoshuadimarsky commented on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
yehoshuadimarsky commented on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-999252229
Got it. Sorry to keep harping on this but I still can't get it to work. I verified that Workload Identity works correctly with the connected K8s service account, but when I try to get Airflow to use that specific K8s service account by specifying the pod template as you suggested, it seems to ignore it and not use that K8s service account.
Google docs state that the K8s pod must use the service account that is connected to Workload Identity - see [here](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#gcloud) step 4.
This part of `values.yaml` states
https://github.com/apache/airflow/blob/6dfc939833fd3dc477b3971d965ba142d3b8bd77/chart/values.yaml#L1301-L1305
So I copied the contents of that file ([here](https://github.com/apache/airflow/blob/helm-chart/1.3.0/chart/files/pod-template-file.kubernetes-helm-yaml)) as the value of `podTemplate` and just changed the service account from this
https://github.com/apache/airflow/blob/6dfc939833fd3dc477b3971d965ba142d3b8bd77/chart/files/pod-template-file.kubernetes-helm-yaml#L96
to the actual K8s service account that is used in Workload Identity. And... **doesn't work**. No logs written. The `Rendered K8s Pod Spec` in the Airflow UI for that task looks like this - note, `serviceAccountName` is MISSING:
```yaml
metadata:
annotations:
dag_id: logging_test
execution_date: '2021-12-22T02:39:29.525436+00:00'
task_id: list_gcp_bucket_objects_in_dev
try_number: '1'
labels:
airflow-worker: worker-config
airflow_version: 2.2.1
dag_id: logging_test
execution_date: 2021-12-22T02_39_29.525436_plus_00_00
kubernetes_executor: 'True'
task_id: list_gcp_bucket_objects_in_dev
try_number: '1'
name: loggingtestlistgcpbucketobjectsindev.c939facaa5a947e2b10ead18da287a1c
namespace: airflow
spec:
containers:
- args:
- airflow
- tasks
- run
- logging_test
- list_gcp_bucket_objects_in_dev
- manual__2021-12-22T02:39:29.525436+00:00
- --subdir
- DAGS_FOLDER/log_test_dag.py
env:
- name: AIRFLOW_IS_K8S_EXECUTOR_POD
value: 'True'
image: [REDACTED]
name: base
```
I thought that maybe Airflow didn't sync, so I deleted and fully recreated the Helm chart app. Didn't work.
**Why won't Airflow use the specified service account in the pod template file?**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yehoshuadimarsky commented on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
yehoshuadimarsky commented on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-997484801
I see. The docs are confusing in that regard then, I thought the `remote_log_conn_id` is used for both reading and writing - for all operations for remote logging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #20408:
URL: https://github.com/apache/airflow/issues/20408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-997792362
> I see. The docs are confusing in that regard then, I thought the `remote_log_conn_id` is used for both reading and writing - for all operations for remote logging.
I heartily invite you to make a PR that will make it clearer there. Airflow is created by > 1800 contributors - most of them contributing code and docs voluntarily and this is an easy way to become one of the contributors and "give back" for the software you get for free. This is super easy - you just use "Suggest a change on this page" button at the bottom-right of the doc page and you will get a possibility to submit changes to the dodcumentation using GitHub UI as a new PR. It's even easier to make a PR than to create an issue about it in fact. And you as the user are one of the best to know what and how to describe it in the way that other users will find useful.
> So maybe I can broaden my question then - why are there 2 different connection IDs, `remote_log_conn_id` and `google_key_path`? Maybe the GCP provider package should change the remote logging to use `remote_log_conn_id` for both read and write?
I belive (I have not implemented it) this is because of security. The conn_id is there and (currently) available to anyone who has access to the DB (including the webserver to read the logs). However writing should be exclusively available to the tasks "workload". Write access to the bucket is generally more "precious" and for example workload identity and using metadata server in GCP gives much better security, because the access to write is only granted to that particular machine executing the workload and only for the time of executing the task. Even if such "authtentication" leaks, it is useless outside of this single task execution. On the other hand the "conn_id" details are static and leaking it could give anyone unlimited and uncontrolled access to write any data to the bucket. Which could lead to a number of security issues (such as log injection vulnerabilities etc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-997792362
> I see. The docs are confusing in that regard then, I thought the `remote_log_conn_id` is used for both reading and writing - for all operations for remote logging.
I heartily invite you to make a PR that will make it clearer there. Airflow is created by > 1800 contributors - most of them contributing code and docs voluntarily and this is an easy way to become one of the contributors and "give back" for the software you get for free. This is super easy - you just use "Suggest a change on this page" button at the bottom-right of the doc page and you will get a possibility to submit changes to the dodcumentation using GitHub UI. It's even easier to make a PR than to create an issue about it in fact. And you as the user are one of the best to know what and how to describe it in the way that other users will find useful.
> So maybe I can broaden my question then - why are there 2 different connection IDs, `remote_log_conn_id` and `google_key_path`? Maybe the GCP provider package should change the remote logging to use `remote_log_conn_id` for both read and write?
I belive (I have not implemented it) this is because of security. The conn_id is there and (currently) available to anyone who has access to the DB (including the webserver to read the logs). However writing should be exclusively available to the tasks "workload". Write access to the bucket is generally more "precious" and for example workload identity and using metadata server in GCP gives much better security, because the access to write is only granted to that particular machine executing the workload and only for the time of executing the task. Even if such "authtentication" leaks, it is useless outside of this single task execution. On the other hand the "conn_id" details are static and leaking it could give anyone unlimited and uncontrolled access to write any data to the bucket. Which could lead to a number of security issues (such as log injection vulnerabilities etc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yehoshuadimarsky edited a comment on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
yehoshuadimarsky edited a comment on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-999252229
Got it. Thank you, that worked! I had a whole comment written up asking why the pod template wasn't using the Workload Identity service account, but it turned out that I was just a little impatient and it ended up working correctly after some time. Thank you for all your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-998630997
> I followed all the steps in the Google docs for Workload Identity but am still having issues. I see that there are many K8s service accounts created by default by the Airflow Helm chart, but for Workload Identity you need to specify a single K8s service account to connect to a GCP service account. Which K8s service account does Airflow use? Are you able to shed more light on the role of all of the service accounts used/created by the Helm chart?
It's not what "Helm Chart" does by default. If you are using K8S executor, then you can (and even should) define pod template https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html#configuration and it will create you the PODs in the way and configuration that you specify. What you need to do is to create your POD template with all the necessary configuration for K8S.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yehoshuadimarsky commented on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
yehoshuadimarsky commented on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-998437846
> > I see. The docs are confusing in that regard then, I thought the `remote_log_conn_id` is used for both reading and writing - for all operations for remote logging.
>
> I heartily invite you to make a PR that will make it clearer there. Airflow is created by > 1800 contributors - most of them contributing code and docs voluntarily and this is an easy way to become one of the contributors and "give back" for the software you get for free. This is super easy - you just use "Suggest a change on this page" button at the bottom-right of the doc page and you will get a possibility to submit changes to the dodcumentation using GitHub UI as a new PR. It's even easier to make a PR than to create an issue about it in fact. And you as the user are one of the best to know what and how to describe it in the way that other users will find useful.
Thank you! I will definitely try to submit a PR to change the docs, thank you for making it easy and inviting.
>
> > So maybe I can broaden my question then - why are there 2 different connection IDs, `remote_log_conn_id` and `google_key_path`? Maybe the GCP provider package should change the remote logging to use `remote_log_conn_id` for both read and write?
>
> I belive (I have not implemented it) this is because of security. The conn_id is there and (currently) available to anyone who has access to the DB (including the webserver to read the logs). However writing should be exclusively available to the tasks "workload". Write access to the bucket is generally more "precious" and for example workload identity and using metadata server in GCP gives much better security, because the access to write is only granted to that particular machine executing the workload and only for the time of executing the task. Even if such "authtentication" leaks, it is useless outside of this single task execution. On the other hand the "conn_id" details are static and leaking it could give anyone unlimited and uncontrolled access to write any data to the bucket. Which could lead to a number of security issues (such as log injection vulnerabilities etc).
This makes sense, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-997438879
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-997792362
> I see. The docs are confusing in that regard then, I thought the `remote_log_conn_id` is used for both reading and writing - for all operations for remote logging.
I heartily invite you to make a PR that will make it clearer there. Airflow is created by > 1800 contributors - most of them contributing code and docs voluntarily and this is an easy way to become one of the contributors and "give back" for the free software you get for free. This is super easy - you just use "Suggest a change on this page" button at the bottom-right of the doc page and you will get a possibility to submit changes to the dodcumentation using GitHub UI. It's even easier to make a PR than to create an issue about it in fact. And you as the user are one of the best to know what and how to describe it in the way that other users will find useful.
> So maybe I can broaden my question then - why are there 2 different connection IDs, `remote_log_conn_id` and `google_key_path`? Maybe the GCP provider package should change the remote logging to use `remote_log_conn_id` for both read and write?
I belive (I have not implemented it) this is because of security. The conn_id is there and (currently) available to anyone who has access to the DB (including the webserver to read the logs). However writing should be exclusively available to the tasks "workload". Write access to the bucket is generally more "precious" and for example workload identity and using metadata server in GCP gives much better security, because the access to write is only granted to that particular machine executing the workload and only for the time of executing the task. Even if such "authtentication" leaks, it is useless outside of this single task execution. On the other hand the "conn_id" details are static and leaking it could give anyone unlimited and uncontrolled access to write any data to the bucket. Which could lead to a number of security issues (such as log injection vulnerabilities etc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yehoshuadimarsky edited a comment on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
yehoshuadimarsky edited a comment on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-999252229
Got it. Sorry to keep harping on this but I still can't get it to work. I verified that Workload Identity works correctly with the connected K8s service account, but when I try to get Airflow to use that specific K8s service account by specifying the pod template as you suggested, it seems to ignore it and not use that K8s service account.
Google docs state that the K8s pod must use the service account that is connected to Workload Identity - see [here](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#gcloud) step 4.
This part of `values.yaml` states
https://github.com/apache/airflow/blob/6dfc939833fd3dc477b3971d965ba142d3b8bd77/chart/values.yaml#L1301-L1305
So I copied the contents of that file ([here](https://github.com/apache/airflow/blob/helm-chart/1.3.0/chart/files/pod-template-file.kubernetes-helm-yaml)) as the value of `podTemplate` and just changed the service account from this
https://github.com/apache/airflow/blob/6dfc939833fd3dc477b3971d965ba142d3b8bd77/chart/files/pod-template-file.kubernetes-helm-yaml#L96
to the actual K8s service account that is used in Workload Identity. And... **doesn't work**. No logs written. The `Rendered K8s Pod Spec` in the Airflow UI for that task looks like this - note, `serviceAccountName` is MISSING (it should be under `spec`):
```yaml
metadata:
annotations:
dag_id: logging_test
execution_date: '2021-12-22T02:39:29.525436+00:00'
task_id: list_gcp_bucket_objects_in_dev
try_number: '1'
labels:
airflow-worker: worker-config
airflow_version: 2.2.1
dag_id: logging_test
execution_date: 2021-12-22T02_39_29.525436_plus_00_00
kubernetes_executor: 'True'
task_id: list_gcp_bucket_objects_in_dev
try_number: '1'
name: loggingtestlistgcpbucketobjectsindev.c939facaa5a947e2b10ead18da287a1c
namespace: airflow
spec:
containers:
- args:
- airflow
- tasks
- run
- logging_test
- list_gcp_bucket_objects_in_dev
- manual__2021-12-22T02:39:29.525436+00:00
- --subdir
- DAGS_FOLDER/log_test_dag.py
env:
- name: AIRFLOW_IS_K8S_EXECUTOR_POD
value: 'True'
image: [REDACTED]
name: base
```
I thought that maybe Airflow didn't sync, so I deleted and fully recreated the Helm chart app. Didn't work.
**Why won't Airflow use the specified service account in the pod template file?**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yehoshuadimarsky edited a comment on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
yehoshuadimarsky edited a comment on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-997484801
I see. The docs are confusing in that regard then, I thought the `remote_log_conn_id` is used for both reading and writing - for all operations for remote logging. So maybe I can broaden my question then - why are there 2 different connection IDs, `remote_log_conn_id` and `google_key_path`? Maybe the GCP provider package should change the remote logging to use `remote_log_conn_id` for both read and write?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-997455534
I believe the problem is with understanding the logging authorization:
See: https://airflow.apache.org/docs/apache-airflow-providers-google/stable/logging/gcs.html
``By default Application Default Credentials are used to obtain credentials. You can also set google_key_path option in [logging] section, if you want to use your own service account``
As I read that - the ``remote_log_conn_id`` is used only to read the logs. but in order to write the logs, the PODs executed in K8S executor need to have "Application Default Credentials" with "write" access to the storage (you can also provide `google_key_path` parameter but you'd have to make sure you have the key available in the POD and is generallly less secure). Since your PODs are likely running in GKE you should have ideallly workload identity configured, so that your PODs can have write access to your bucket: https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.
Note that the "gcs-conn-dev" in this case only needs "read" access to the bucket.
I am closint it for now, assuming that your problem will be solved with this advice, please test it and comment here with your results and we can reopen it if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yehoshuadimarsky commented on issue #20408: Logs Are Not Written When Using Remote Logging with K8s Executor
Posted by GitBox <gi...@apache.org>.
yehoshuadimarsky commented on issue #20408:
URL: https://github.com/apache/airflow/issues/20408#issuecomment-998437091
> Since your PODs are likely running in GKE you should have ideallly workload identity configured, so that your PODs can have write access to your bucket: https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.
I followed all the steps in the Google docs for Workload Identity but am still having issues. I see that there are many K8s service accounts created by default by the Airflow Helm chart, but for Workload Identity you need to specify a single K8s service account to connect to a GCP service account. Which K8s service account does Airflow use? Are you able to shed more light on the role of all of the service accounts used/created by the Helm chart?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org