You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "akumar1-godaddy (via GitHub)" <gi...@apache.org> on 2023/03/06 18:03:57 UTC
[GitHub] [airflow] akumar1-godaddy opened a new issue, #29939: AWS EMR Operators: Add Log URI in task logs to speed up debugging
akumar1-godaddy opened a new issue, #29939:
URL: https://github.com/apache/airflow/issues/29939
### Description
Airflow is widely used to launch, interact and submit jobs on AWS EMR Clusters. Existing EMR operators do not provide links to the EMR logs (Job Flow/Step logs), as a result in case of failures the users need to switch to EMR Console or go to AWS S3 console to locate the logs for EMR Jobs and Steps using the job_flow_id available in the EMR Operators and in Xcom.
It will be really convenient and help with debugging if the EMR log links are present in Operator Task logs, it will obviate the need to switch to AWS S3 or AWS EMR consoles from Airflow and lookup the logs using job_flow_ids. It will be a nice improvement for the developer experience.
LogUri for Cluster is available in [DescribeCluster](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr/client/describe_cluster.html)
LogFile path for Steps in case of failure is available in [ListSteps](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr/client/list_steps.html)
### Use case/motivation
Ability to go to EMR logs directly from Airflow EMR Task logs.
### Related issues
N/A
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akumar1-godaddy commented on issue #29939: AWS EMR Operators: Add Log URI in task logs to speed up debugging
Posted by "akumar1-godaddy (via GitHub)" <gi...@apache.org>.
akumar1-godaddy commented on issue #29939:
URL: https://github.com/apache/airflow/issues/29939#issuecomment-1467176111
Thanks @potiuk for the inputs and reference links.
I checked the EMR operators, as of now the EMR Operators only have links to the EMR Cluster Summary(EMR console) and the EMR Sensors have links to the S3 logs folder containing logs for nodes, steps and container.
I think adding the log uri to the operators would be helpful wherever applicable e.g. for failed steps it would be useful to have loguri for `EmrAddStepsOperator`.
Will work on the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas closed issue #29939: AWS EMR Operators: Add Log URI in task logs to speed up debugging
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas closed issue #29939: AWS EMR Operators: Add Log URI in task logs to speed up debugging
URL: https://github.com/apache/airflow/issues/29939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shubham22 commented on issue #29939: AWS EMR Operators: Add Log URI in task logs to speed up debugging
Posted by "shubham22 (via GitHub)" <gi...@apache.org>.
shubham22 commented on issue #29939:
URL: https://github.com/apache/airflow/issues/29939#issuecomment-1467197710
+1 to suggestion from @akumar1-godaddy; having additional extra link should only improve the UX.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29939: AWS EMR Operators: Add Log URI in task logs to speed up debugging
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29939:
URL: https://github.com/apache/airflow/issues/29939#issuecomment-1456665613
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29939: AWS EMR Operators: Add Log URI in task logs to speed up debugging
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29939:
URL: https://github.com/apache/airflow/issues/29939#issuecomment-1458918996
Isn't that done already via https://airflow.apache.org/docs/apache-airflow/stable/howto/define_extra_link.html ? There is EMR link already in the latest provider https://github.com/apache/airflow/pull/28180 - maybe you are not using the latest provider simply? Or maybe it needs to be used in other operators?
Can you please @akumar1-godaddy
a) close the issue if it is already solved in the latest provider or
b) simply add PR to add them (following the one I copied)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org