You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/29 19:19:30 UTC
[GitHub] [airflow] snjypl commented on pull request #28161: AIP-51 - Executor Coupling in Logging
snjypl commented on PR #28161:
URL: https://github.com/apache/airflow/pull/28161#issuecomment-1367534800
> Fantastic unit tests! Thanks for adding those.
>
> Just left one other small request for a doc string.
>
> Also, are you able to do any UAT/manual testing to ensure things are working as expected as compared to before this change?
@o-nikolas i did some manual testing with different executors. it is working as expected.
in main there is an issue with `LocalKubernetesExecutor`. scheduler is not serving the logs for local task. https://github.com/apache/airflow/pull/28638 . i tested with a patched version, and it is working fine.
`celery executor`
```
airflow-worker-0.airflow-worker.airflow.svc.cluster.local
*** Local log file does not exist: /opt/airflow/logs/dag_id=example_bash_operator/run_id=manual__2022-12-29T19:06:34.783246+00:00/task_id=runme_0/attempt=1.log
*** Failed to fetch log from executor. Falling back to fetching log from worker.
*** Fetching from: http://airflow-worker-0.airflow-worker.airflow.svc.cluster.local:8793/log/dag_id=example_bash_operator/run_id=manual__2022-12-29T19:06:34.783246+00:00/task_id=runme_0/attempt=1.log
[2022-12-29, 19:06:37 UTC] {taskinstance.py:1084} INFO - Dependencies all met for <TaskInstance: example_bash_operator.runme_0 manual__2022-12-29T19:06:34.783246+00:00 [queued]>
[2022-12-29, 19:06:37 UTC] {taskinstance.py:1084} INFO - Dependencies all met for <TaskInstance: example_bash_operator.runme_0 manual__2022-12-29T19:06:34.783246+00:00 [queued]>
[2022-12-29, 19:06:37 UTC] {taskinstance.py:1282} INFO -
```
`LocalKubernetesExecutor ` local task
```
airflow-scheduler-0.airflow-scheduler.airflow.svc.cluster.local
*** Local log file does not exist: /opt/airflow/logs/dag_id=example_local_kubernetes_executor/run_id=manual__2022-12-29T19:11:14.004608+00:00/task_id=task_with_local_executor/attempt=1.log
*** Failed to fetch log from executor. Falling back to fetching log from worker.
*** Fetching from: http://airflow-scheduler-0.airflow-scheduler.airflow.svc.cluster.local:8793/log/dag_id=example_local_kubernetes_executor/run_id=manual__2022-12-29T19:11:14.004608+00:00/task_id=task_with_local_executor/attempt=1.log
[2022-12-29, 19:11:15 UTC] {taskinstance.py:1084} INFO - Dependencies all met for <TaskInstance: example_local_kubernetes_executor.task_with_local_executor manual__2022-12-29T19:11:14.004608+00:00 [queued]>
```
`LocalKubernetesExecutor ` k8s task
```
example-local-kubernetes-executor-task-with-kubernetes-q2qf6dpa
*** Local log file does not exist: /opt/airflow/logs/dag_id=example_local_kubernetes_executor/run_id=manual__2022-12-29T19:11:14.004608+00:00/task_id=task_with_kubernetes_executor/attempt=1.log
*** Trying to get logs (last 100 lines) from worker pod example-local-kubernetes-executor-task-with-kubernetes-q2qf6dpa ***
/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py:925 DeprecationWarning: The namespace option in [kubernetes] has been moved to the namespace option in [kubernetes_executor] - the old setting has been used, but please update your config.
[2022-12-29, 19:11:30 UTC] {dagbag.py:538} INFO - Filling up the DagBag from /opt/airflow/dags/example_local_kubernetes_executor.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org