You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "walter9388 (via GitHub)" <gi...@apache.org> on 2025/01/10 14:46:58 UTC

[I] Web UI task logging with CloudWatch has a 60 second time delay [airflow]

walter9388 opened a new issue, #45554:
URL: https://github.com/apache/airflow/issues/45554

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.10.1
   
   ### What happened?
   
   When using CloudWatch logging there seems to be a 60 second time delay between the logging output updating. Please see the video below and observe:
   1. Initially Airflow can't find the remote logs (as there are none).
   2. Airflow detects local logs.
   3. Nothing happens in the UI for 60 seconds.
   4. After 60 seconds logging appears, and the top of the printout states it is from CloudWatch logs.
   5. It then periodically updates the logs every 60 seconds after until the task is completed.
   
   _Please skip ahead in the video as most of it is static!_
   
   https://github.com/user-attachments/assets/d3abd82b-793a-475a-9488-746d640573c7
   
   As a second minor point, you can also see that grouping now longer works with the logs read from CloudWatch. However, this doesn't concern me as much.
   
   ### What you think should happen instead?
   
   I'm not sure if I have configured something incorrectly, but I expected the same behaviour as local logging, i.e. tailing of the log file.
   
   I struggled to find the default behaviour documented, but what I expect to happen was that Airflow would use the local logs if they were available and only use the remote logs if no local logs were found.
   I found this logic in previous [documentation (<2.0)](https://airflow.apache.org/docs/apache-airflow/1.10.8/howto/write-logs.html), although this may now be outdated:
   > In the Airflow Web UI, remote logs take precedence over local logs when remote logging is enabled. If remote logs can not be found or accessed, local logs will be displayed. Note that logs are only sent to remote storage once a task is complete (including failure); In other words, remote logs for running tasks are unavailable (but local logs are available).
   
   Can you confirm that this is the expected behaviour and what is in the video above is a bug?
   
   Alternatively, I can see in the browser that a request is made every second to update the logs, and I can confirm that the logs are only being written to CloudWatch every 60 seconds or when the task is complete. 
   Is this expected behaviour? or should logs be written to cloudwatch at a higher rate?
   
   If the behaviour in the video is actually what is expected, I would like to suggest one of the following options as we need a <60 second refresh window in our logging setup:
   1. A configuration variable to use local logging first if available (e.g. `local_logging_prefer = True`).
   2. A configuration variable for the update frequency of the logging when using remote logging (e.g. `remote_logging_refresh_period = 60`).
   
   Let me know your thoughts.
   
   ### How to reproduce
   
   The remote logging config was copied from [here](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/logging/cloud-watch-task-handlers.html):
   ```
   [logging]
   # Airflow can store logs remotely in AWS Cloudwatch. Users must supply a log group
   # ARN (starting with 'cloudwatch://...') and an Airflow connection
   # id that provides write and read access to the log location.
   remote_logging = True
   remote_base_log_folder = cloudwatch://arn:aws:logs:<region name>:<account id>:log-group:<group name>
   remote_log_conn_id = MyCloudwatchConn
   ```
   
   The demo DAG used in the video above prints to logging every 10 seconds and is as follows:
   ```python
   import logging
   from datetime import datetime
   from time import sleep
   
   from airflow.models import DAG
   from airflow.operators.python import task
   
   with DAG(
       dag_id="dev__cloudwatch_logging_testing",
       start_date=datetime(2024, 1, 1),
       schedule=None,
   ):
   
       @task
       def task1():
           sleeptime = 10
           for i in range(0, 300, sleeptime):
               logging.info(i)
               sleep(sleeptime)
   
       task1()
   ```
   
   ### Operating System
   
   NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal
   
   ### Versions of Apache Airflow Providers
   
   ```
   apache-airflow==2.10.1
   apache-airflow-providers-amazon==8.28.0
   apache-airflow-providers-celery==3.8.1
   apache-airflow-providers-cncf-kubernetes==8.4.1
   apache-airflow-providers-common-compat==1.2.0
   apache-airflow-providers-common-io==1.4.0
   apache-airflow-providers-common-sql==1.16.0
   apache-airflow-providers-docker==3.13.0
   apache-airflow-providers-elasticsearch==5.5.0
   apache-airflow-providers-fab==1.3.0
   apache-airflow-providers-ftp==3.11.0
   apache-airflow-providers-google==10.22.0
   apache-airflow-providers-grpc==3.6.0
   apache-airflow-providers-hashicorp==3.8.0
   apache-airflow-providers-http==4.13.0
   apache-airflow-providers-imap==3.7.0
   apache-airflow-providers-microsoft-azure==10.4.0
   apache-airflow-providers-mysql==5.7.0
   apache-airflow-providers-odbc==4.7.0
   apache-airflow-providers-openlineage==1.11.0
   apache-airflow-providers-postgres==5.12.0
   apache-airflow-providers-redis==3.8.0
   apache-airflow-providers-sendgrid==3.6.0
   apache-airflow-providers-sftp==4.11.0
   apache-airflow-providers-slack==8.9.0
   apache-airflow-providers-smtp==1.8.0
   apache-airflow-providers-snowflake==5.7.0
   apache-airflow-providers-sqlite==3.9.0
   apache-airflow-providers-ssh==3.13.1
   ```
   
   ### Deployment
   
   Other Docker-based deployment
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Web UI task logging with CloudWatch has a 60 second time delay [airflow]

Posted by "anasatzemoso (via GitHub)" <gi...@apache.org>.
anasatzemoso commented on issue #45554:
URL: https://github.com/apache/airflow/issues/45554#issuecomment-3232712661

   Hi @walter9388,  just following up on this one.
   
   From my testing, the CloudWatch logging delay is significantly improved in Airflow 3.x (~20–30s) compared to 2.10.1 (~60s). So this appears to be already resolved in newer versions.
   
   The open question is:
   Do we want to bring this improvement to the 2.x, or should we assume it will stay as-is and users need to upgrade to 3.x for better CloudWatch logging?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Web UI task logging with CloudWatch has a 60 second time delay [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #45554: Web UI task logging with CloudWatch has a 60 second time delay
URL: https://github.com/apache/airflow/issues/45554


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Web UI task logging with CloudWatch has a 60 second time delay [airflow]

Posted by "walter9388 (via GitHub)" <gi...@apache.org>.
walter9388 commented on issue #45554:
URL: https://github.com/apache/airflow/issues/45554#issuecomment-3269510482

   Hi @anasatzemoso, thanks for looking into this! Glad to see there is an improvement in Airflow 3.x, do you know where the improvement has come from? 
   At my company I think it is going to be a while before we migrate to 3.x, so I would still like to understand if improvements can be made in 2.x on this matter.
   All of this seems slightly arbitrary to me. I feel like whether it's 20 seconds or 60 seconds, there must be some sort of polling going on internally for the logs, and it would be great to understand if we can either improve the latency to CloudWatch logs in general or parameterize the polling interval to CloudWatch logs somewhere if there is a reason it has been set at 30s, 60s etc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Web UI task logging with CloudWatch has a 60 second time delay [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #45554:
URL: https://github.com/apache/airflow/issues/45554#issuecomment-4172893836

   Closing as resolved in Airflow 3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Web UI task logging with CloudWatch has a 60 second time delay [airflow]

Posted by "anasatzemoso (via GitHub)" <gi...@apache.org>.
anasatzemoso commented on issue #45554:
URL: https://github.com/apache/airflow/issues/45554#issuecomment-3182309182

   <img width="3032" height="1488" alt="Image" src="https://github.com/user-attachments/assets/fdb08517-1dc0-434c-946a-1b5e8a04acfa" />
   
   I tested this in Airflow 3.x and can confirm the behavior has improved.  Logs are now printed and displayed in the UI with only a ~20–30 second delay when using CloudWatch logging. This is a significant improvement over v2.10.1, where the delay was consistently ~60 seconds.
   
   So the issue seems to be resolved in v3. The question now is whether we want to fix this improvement to Airflow 2.x


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Web UI task logging with CloudWatch has a 60 second time delay [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #45554:
URL: https://github.com/apache/airflow/issues/45554#issuecomment-2582874707

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Web UI task logging with CloudWatch has a 60 second time delay [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45554:
URL: https://github.com/apache/airflow/issues/45554#issuecomment-3272039394

   Just a watch-out. As a community we decided that Airflow 2 is still going to be supported - only major/serious issues and critical security issues will be solved. Airlfow 3 logging is vastly different than Airlfow 2 (and we are still cranking out some issues and 3.1 will have the next step of the refactoring and using structlog under the hood), so it's highly unlikely anyone from the maintainer team will spend their time on it and also if such change would be risky/substantial, we won't even likely approve and merge such change.
   
   Moving to Airflow 3 is generally recommended - we are planning Airflow 3.1 soon with much more improved parity comparing to Airlfow 2 (especially in UI) so you might want to consider including migration in your plans.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org