You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pdebelak (via GitHub)" <gi...@apache.org> on 2023/02/06 20:33:22 UTC
[GitHub] [airflow] pdebelak opened a new issue, #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
pdebelak opened a new issue, #29393:
URL: https://github.com/apache/airflow/issues/29393
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==7.1.0
### Apache Airflow version
2.5.1
### Operating System
Ubuntu 18.04
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
When looking at logs in the UI for a running task when using remote s3 logging, the logs for the task are only uploaded to s3 after the task has completed. The the `S3TaskHandler` falls back to the local logs stored on the worker in that case (by falling back to the `FileTaskHandler` behavior) and prepends the line `*** Falling back to local log` to those logs.
This is mostly fine, but for the new log streaming behavior, this means that `*** Falling back to local log` is returned from `/get_logs_with_metadata` on each call, even if there are no new logs.
### What you think should happen instead
I'd expect the falling back message only to be included in calls with no `line_pos` in the metadata or with a `line_pos` of `0`.
### How to reproduce
Start a task with `logging.remote_logging` set to `True` and `logging.remote_base_log_folder` set to `s3://something` and watch the logs while the task is running. You'll see `*** Falling back to local log` printed every few seconds.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1440780450
Hi y'all, thanks for the issue reports :)
I did not spend nearly as much time live testing the legacy logs view as I did with the grid logs view. Is there some reason y'all prefer the legacy logs view? Just curious.
I will try and look at this but honestly, I have a _lot_ going on right now and if one of you who have looked into the issue can contribute a fix that would be really welcome. I would be delighted to review any PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pdebelak commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "pdebelak (via GitHub)" <gi...@apache.org>.
pdebelak commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1419765071
There's some comments in `S3TaskHandler._read` about airflow 2.6 that seem to indicate that this fallback will work differently starting in airflow 2.6, but I'm not too sure what is changing in that version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccwillia commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "ccwillia (via GitHub)" <gi...@apache.org>.
ccwillia commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1424495926
I am also seeing the `*** Falling back to local log` message get printed every few seconds in version 2.5 while using s3 logging. Has anyone found a way to prevent this from being printed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish closed issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
URL: https://github.com/apache/airflow/issues/29393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pdebelak commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "pdebelak (via GitHub)" <gi...@apache.org>.
pdebelak commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1438517024
It seems like in the code as currently written since `FileTaskHandler` implements `_read_remote_logs` this function won't be called at all (see https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/log/s3_task_handler.py#L123-L125). But in the current airflow release `FileTaskHandler` doesn't have that method which is why this is called. As a result, I'm not even sure it's possible to write a test for this behavior without mocking out an old version of the superclass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kobethuwis commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "kobethuwis (via GitHub)" <gi...@apache.org>.
kobethuwis commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1438402879
Due to this issue, I tried understanding the way Airflow is (remotely) logging tasks. I am using S3 for remote log storage, but would like to access live logs for running tasks without having to use a persistent volume.
> In the Airflow UI, remote logs take precedence over local logs when remote logging is enabled. If remote logs can not be found or accessed, local logs will be displayed. Note that logs are only sent to remote storage once a task is complete (including failure). In other words, remote logs for running tasks are unavailable (but local logs are available).
However, the error log tells us that on the pod's level log retrieval is not possible, `*** Log file does not exist: /opt/airflow/logs/.../attempt=1.log *** `
This is because I am using the CeleryWorker in my deployment and is elaborated in the documentation [here](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html#serving-logs-from-workers)
> Airflow automatically starts an HTTP server to serve the logs ... The server is running on the port specified by worker_log_server_port option in [logging] section. By default, it is 8793 ...
This behaviour is confirmed by the second part of the error, where we are trying to fetch the from the server: `Fetching from: http://airflow-worker-46838b34-rq7r8:8793/log/.../attempt=1.log ` I can thus reduce my issue to not being able to fetch the logs from the server.
Using a shell inside the webserver I tried resolving/pinging `http://airflow-worker-46838b34-rq7r8:879`, without result. We're actually looking at a 'simple' DNS resolving error, which led me to this [solution
](https://stackoverflow.com/questions/62905221/dns-for-kubernetes-pods-and-airflow-worker-logs).
The suggested resolution works! This way I'm able to fall back to live local logs served by the worker for running tasks.
Specify the following in your helm chart:
```
extraEnv: |
- name: AIRFLOW__CORE__HOSTNAME_CALLABLE
value: 'airflow.utils.net:get_host_ip_address'
```
To be clear, this does not resolve @pdebelak 's original issue but did enable me to view live local logs for running tasks while still storing the finished tasks logs remotely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] PodbiAtGS commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "PodbiAtGS (via GitHub)" <gi...@apache.org>.
PodbiAtGS commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1436044382
Same here. Any ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kobethuwis commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "kobethuwis (via GitHub)" <gi...@apache.org>.
kobethuwis commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1431253928
Also experiencing this issue 😐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] goodseog commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "goodseog (via GitHub)" <gi...@apache.org>.
goodseog commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1436162264
Same here :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1437595099
cc: @dstandish - maybe you can comment on how it changes in 2.6 (and whether the problem is going to be fixed there).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kobethuwis commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "kobethuwis (via GitHub)" <gi...@apache.org>.
kobethuwis commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1512914779
> Due to this issue, I tried understanding the way Airflow is (remotely) logging tasks. I am using S3 for remote log storage, but would like to access live logs for running tasks without having to use a persistent volume.
>
> > In the Airflow UI, remote logs take precedence over local logs when remote logging is enabled. If remote logs can not be found or accessed, local logs will be displayed. Note that logs are only sent to remote storage once a task is complete (including failure). In other words, remote logs for running tasks are unavailable (but local logs are available).
>
> However, the error log tells us that on the pod's level log retrieval is not possible, `*** Log file does not exist: /opt/airflow/logs/.../attempt=1.log *** `
>
> This is because I am using the CeleryWorker in my deployment and is elaborated in the documentation [here](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html#serving-logs-from-workers)
>
> > Airflow automatically starts an HTTP server to serve the logs ... The server is running on the port specified by worker_log_server_port option in [logging] section. By default, it is 8793 ...
>
> This behaviour is confirmed by the second part of the error, where we are trying to fetch the from the server: `Fetching from: http://airflow-worker-46838b34-rq7r8:8793/log/.../attempt=1.log ` I can thus reduce my issue to not being able to fetch the logs from the server.
>
> Using a shell inside the webserver I tried resolving/pinging `http://airflow-worker-46838b34-rq7r8:879`, without result. We're actually looking at a 'simple' DNS resolving error, which led me to this [solution. ](https://stackoverflow.com/questions/62905221/dns-for-kubernetes-pods-and-airflow-worker-logs)
>
> The suggested resolution works! This way I'm able to fall back to live local logs served by the worker for running tasks.
>
> Specify the following in your helm chart:
>
> ```
> extraEnv: |
> - name: AIRFLOW__CORE__HOSTNAME_CALLABLE
> value: 'airflow.utils.net:get_host_ip_address'
> ```
>
> To be clear, this does not resolve @pdebelak 's original issue but did enable me to view live local logs for running tasks while still storing the finished tasks logs remotely.
Update, due to some deprecation in newer versions in Airflow I'm now using `'airflow.utils.net.get_host_ip_address'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pdebelak commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "pdebelak (via GitHub)" <gi...@apache.org>.
pdebelak commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1440784148
@dstandish
> Is there some reason y'all prefer the legacy logs view?
I did not know there was a different place to see logs. I see now what you are talking about. I'll try out the new log view and see how I like it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #29393: S3TaskHandler continuously returns "*** Falling back to local log" even if log_pos is provided when log not in s3
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on issue #29393:
URL: https://github.com/apache/airflow/issues/29393#issuecomment-1440788093
@pdebelak sounds good
in any case, your comment seems reasonable:
> I'd expect the falling back message only to be included in calls with no log_pos in the metadata or with a log_pos of 0.
You wanna try it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org