You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/03 04:35:45 UTC
[GitHub] [airflow] virendhar-aws opened a new issue, #23437: Logs for EmrStepSensor
virendhar-aws opened a new issue, #23437:
URL: https://github.com/apache/airflow/issues/23437
### Description
Add feature to EmrStepSensor to bring back the spark task url & logs after task execution
### Use case/motivation
After starting an EMR step task using EmrAddStepsOperator we generally have an EmrStepSensor to track the status of the step. The job ID is available for the sensor and is being poked at regular interval.
```
[2022-04-26, 22:07:43 UTC] {base_aws.py:100} INFO - Retrieving region_name from Connection.extra_config['region_name']
[2022-04-26, 22:07:44 UTC] {emr.py:316} INFO - Poking step s-123ABC123ABC on cluster j-123ABC123ABC
[2022-04-26, 22:07:44 UTC] {emr.py:74} INFO - Job flow currently PENDING
[2022-04-26, 22:08:44 UTC] {emr.py:316} INFO - Poking step s-123ABC123ABC on cluster j-123ABC123ABC
[2022-04-26, 22:08:44 UTC] {emr.py:74} INFO - Job flow currently PENDING
[2022-04-26, 22:09:44 UTC] {emr.py:316} INFO - Poking step s-123ABC123ABC on cluster j-123ABC123ABC
[2022-04-26, 22:09:44 UTC] {emr.py:74} INFO - Job flow currently COMPLETED
[2022-04-26, 22:09:44 UTC] {base.py:251} INFO - Success criteria met. Exiting.
[2022-04-26, 22:09:44 UTC] {taskinstance.py:1288} INFO - Marking task as SUCCESS. dag_id=datapipeline_sample, task_id=calculate_pi_watch_step, execution_date=20220426T220739, start_date=20220426T220743, end_date=20220426T220944
```
After the task is completed the status is displayed. If the user wants to review the logs of the task, it is a multistep process to get hold of the job logs from EMR cluster.
It will be a great addition to add the log url and possibly relay the logs to Airflow EmrStepSensor post completion of the task. This will be very handy when there are failures of many tasks and will make it a great user experience.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #23437:
URL: https://github.com/apache/airflow/issues/23437#issuecomment-1281568086
Unassigning Phani so that someone else can pick this up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #23437:
URL: https://github.com/apache/airflow/issues/23437#issuecomment-1256902330
@phanikumv are you still working on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shubham22 commented on issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
shubham22 commented on issue #23437:
URL: https://github.com/apache/airflow/issues/23437#issuecomment-1303004097
@syedahsn is going to pick this up. Can this be assigned to him?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #23437: Logs for EmrStepSensor
URL: https://github.com/apache/airflow/issues/23437
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #23437:
URL: https://github.com/apache/airflow/issues/23437#issuecomment-1303011521
We can assign only to users who commented on the issue (github limitation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] syedahsn commented on issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
syedahsn commented on issue #23437:
URL: https://github.com/apache/airflow/issues/23437#issuecomment-1303226366
Can I be assigned this issue? Thanks @eladkal :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23437:
URL: https://github.com/apache/airflow/issues/23437#issuecomment-1120492208
It's more of a feature for ExtraLink https://airflow.apache.org/docs/apache-airflow/stable/howto/define_extra_link.html rather than putting the link to the log (or maybe both). Anyhow marked ait as good first issue for someone who would like to pick it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shubham22 commented on issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
shubham22 commented on issue #23437:
URL: https://github.com/apache/airflow/issues/23437#issuecomment-1303013183
Oh, TIL! @syedahsn you know what to do : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] phanikumv commented on issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
phanikumv commented on issue #23437:
URL: https://github.com/apache/airflow/issues/23437#issuecomment-1160641023
Hi @potiuk is someone working on this or can I pick this up for development ?
cc @kaxil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #23437: Logs for EmrStepSensor
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #23437:
URL: https://github.com/apache/airflow/issues/23437#issuecomment-1160727817
Assigned it to you @phanikumv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org