You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rushabh-lokhande (via GitHub)" <gi...@apache.org> on 2023/07/27 21:01:46 UTC
[GitHub] [airflow] rushabh-lokhande opened a new issue, #32897: Enhance Airflow Logs API to fetch logs from Amazon Cloudwatch with time range
rushabh-lokhande opened a new issue, #32897:
URL: https://github.com/apache/airflow/issues/32897
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
MWAA Version: 2.4.3
Airflow Version: 2.4.3
Airflow Logs currently do not fetch logs from Cloudwatch without time range, so when the cloudwatch logs are large and CloudWatch log streams are OLD, the airflow UI cannot display logs with error message:
*** Reading remote log from Cloudwatch log_group: airflow-cdp-airflow243-XXXX-Task log_stream: dag_id=<DAG_NAME>/run_id=scheduled__2023-07-27T07_25_00+00_00/task_id=<TASK_ID>/attempt=1.log.
Could not read remote logs from log_group: airflow-cdp-airflow243-XXXXXX-Task log_group: airflow-cdp-airflow243-XXXX-Task log_stream: dag_id=<DAG_NAME>/run_id=scheduled__2023-07-27T07_25_00+00_00/task_id=<TASK_ID>/attempt=1.log
The Airflow API need to pass start and end timestamps to GetLogEvents API from Amazon CloudWatch to resolve this error and it also improves performance of fetching logs.
This is critical issue for customers when they would like to fetch logs to investigate failed pipelines form few days to weeks old
### What you think should happen instead
The Airflow API need to pass start and end timestamps to GetLogEvents API from Amazon CloudWatch to resolve this error.
This should also improve performance of fetching logs.
### How to reproduce
This issue is intermittent and happens mostly on FAILD tasks.
1. Log onto Amazon MWAA Service
2. Open Airflow UI
3. Select DAG
4. Select the Failed Tasks
5. Select Logs
You should see error message like below in the logs:
*** Reading remote log from Cloudwatch log_group: airflow-cdp-airflow243-XXXX-Task log_stream: dag_id=<DAG_NAME>/run_id=scheduled__2023-07-27T07_25_00+00_00/task_id=<TASK_ID>/attempt=1.log.
Could not read remote logs from log_group: airflow-cdp-airflow243-XXXXXX-Task log_group: airflow-cdp-airflow243-XXXX-Task log_stream: dag_id=<DAG_NAME>/run_id=scheduled__2023-07-27T07_25_00+00_00/task_id=<TASK_ID>/attempt=1.log
### Operating System
Running with Amazon MWAA
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.3.1
apache-airflow==2.4.3
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas closed issue #32897: Enhance Airflow Logs API to fetch logs from Amazon Cloudwatch with time range
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas closed issue #32897: Enhance Airflow Logs API to fetch logs from Amazon Cloudwatch with time range
URL: https://github.com/apache/airflow/issues/32897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ivica-k commented on issue #32897: Enhance Airflow Logs API to fetch logs from Amazon Cloudwatch with time range
Posted by "ivica-k (via GitHub)" <gi...@apache.org>.
ivica-k commented on issue #32897:
URL: https://github.com/apache/airflow/issues/32897#issuecomment-1657117217
Hey @rushabh-lokhande, in your opinion, where would the values for start time and end time come from? I'm guessing from the start and end time of the DAG itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rlokhande1982 commented on issue #32897: Enhance Airflow Logs API to fetch logs from Amazon Cloudwatch with time range
Posted by "rlokhande1982 (via GitHub)" <gi...@apache.org>.
rlokhande1982 commented on issue #32897:
URL: https://github.com/apache/airflow/issues/32897#issuecomment-1658605400
> Hey @rushabh-lokhande, in your opinion, where would the values for start time and end time come from? I'm guessing from the start and end time of the DAG itself?
Yes @ivica-k we will pick up start time and end time with UTC conversion if needed, as ClopudWatch translates the dates to UTC. Also, even if we send start_date and end_date instead of start_timestamps and end_timestamps, that should narrow down Amazon Cloudwatch logs window and should improve performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #32897: Enhance Airflow Logs API to fetch logs from Amazon Cloudwatch with time range
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #32897:
URL: https://github.com/apache/airflow/issues/32897#issuecomment-1656935902
@rushabh-lokhande would you like to work on fixing this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rushabh-lokhande commented on issue #32897: Enhance Airflow Logs API to fetch logs from Amazon Cloudwatch with time range
Posted by "rushabh-lokhande (via GitHub)" <gi...@apache.org>.
rushabh-lokhande commented on issue #32897:
URL: https://github.com/apache/airflow/issues/32897#issuecomment-1670318844
This Issue is linked to performance enhancement of https://github.com/apache/airflow/pull/20814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shubham22 commented on issue #32897: Enhance Airflow Logs API to fetch logs from Amazon Cloudwatch with time range
Posted by "shubham22 (via GitHub)" <gi...@apache.org>.
shubham22 commented on issue #32897:
URL: https://github.com/apache/airflow/issues/32897#issuecomment-1671854327
fyi @vincbeck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #32897: Enhance Airflow Logs API to fetch logs from Amazon Cloudwatch with time range
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #32897:
URL: https://github.com/apache/airflow/issues/32897#issuecomment-1654580594
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org