You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/20 16:42:15 UTC
[GitHub] [airflow] ak-arun opened a new issue, #23832: AWS Glue Hook to implement more verbos logging
ak-arun opened a new issue, #23832:
URL: https://github.com/apache/airflow/issues/23832
### Description
Currently, when submitting AWS Glue jobs using Airflow, we are seeing the below in the log :-
**Polling for AWS Glue Job <job_name> current run state with status <job_run_state>**. This request is to enhance this by providing more detailed information about the Glue Job, show the same continuous logs from Glue/Cloudwatch into the Airflow logs. This avoids airflow users having to log onto Glue logs to be able to see detailed information on their Jobs. Also, AWS admins can limit the users access to just airflow and need not open up access to glue console and cloudwatch just for seeing the logs. Airflow could then be used as the single pane of glass on job orchestration and job health management.
### Use case/motivation
This will provide a single place to track the job status rather than having to hop screens. Airflow could then be used as the single pane of glass on job orchestration and job health management.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23832: AWS Glue Hook to implement enhanced logging
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23832:
URL: https://github.com/apache/airflow/issues/23832#issuecomment-1133884677
@ak-arun - maybe you would like to add Pull Request fixing it ? Seems ike an easy thing to do that would not require deep knowledge of Airlow itself and you could test it locally with your setup ? Can I assign it to you? This is the easiest and fastest way to get it implemented :). Otherwise I mark it as "good-first-issue".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dark-Knight11 commented on issue #23832: AWS Glue Hook to implement enhanced logging
Posted by GitBox <gi...@apache.org>.
Dark-Knight11 commented on issue #23832:
URL: https://github.com/apache/airflow/issues/23832#issuecomment-1137808156
Ok, Thanks. I'll look into it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ak-arun commented on issue #23832: AWS Glue Hook to implement enhanced logging
Posted by GitBox <gi...@apache.org>.
ak-arun commented on issue #23832:
URL: https://github.com/apache/airflow/issues/23832#issuecomment-1274369187
Hi Andrey,
Point taken. I am thinking if we just publish the logs when the job state
changes to Error/Failed.
Also, instead of mining all executor logs + the driver, I think we can read
the 2 recently added job insights streams -
https://docs.aws.amazon.com/glue/latest/dg/monitor-job-insights.html
For sure, this is a new feature in Glue and older jobs may not have these
streams enabled. We can mention it works only if job-insights are
enabled?-Thoughts ?
--
Thanks & Regards
*Arun A K*
On Wed, May 25, 2022 at 1:10 PM Andrey Anshin ***@***.***>
wrote:
> @Dark-Knight11 <https://github.com/Dark-Knight11> I would like to warn
> about side effect of fetch AWS Glue Logs into Airflow Tasks logs (already
> mentioned in the almost the same issue #23900 (comment)
> <https://github.com/apache/airflow/issues/23900#issuecomment-1137357273>).
>
> If your Glue Job uses 40 DPU that mean it spawn minimum 1 driver and 39
> workers/executors and all of them will create errors and output logs, as
> result
>
> 1. You need to find all of CloudWatch Logs Prefix started with job_id
> in correct log group, by default Glue uses
> - /aws-glue/jobs/output - for output logs
> - /aws-glue/jobs/error - for error logs
> - /aws-glue/jobs/logs-v2 - (optional) for continuous logging
> 2. You need to be sure that you fetch from all prefixes and this
> prefixes doesn't created in the same time
> 3. Be sure that fetch logging processes/threads do not use all
> CPU/Memory/IO of Airflow Worker
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/23832#issuecomment-1137572804>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEVRXFOQEOW4BUUI2YU5FHLVLZNHPANCNFSM5WP75PDA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #23832: AWS Glue Hook to implement enhanced logging
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #23832: AWS Glue Hook to implement enhanced logging
URL: https://github.com/apache/airflow/issues/23832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #23832: AWS Glue Hook to implement enhanced logging
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #23832:
URL: https://github.com/apache/airflow/issues/23832#issuecomment-1137572804
@Dark-Knight11 I would like to warn about side effect of fetch AWS Glue Logs into Airflow Tasks logs (already mentioned in the almost the same issue https://github.com/apache/airflow/issues/23900#issuecomment-1137357273).
If your Glue Job uses 40 DPU that mean it spawn minimum 1 driver and 39 workers/executors and all of them will create errors and output logs, as result
1. You need to find all of CloudWatch Logs Prefix started with `job_id` in correct log group, by default Glue uses
- `/aws-glue/jobs/output` - for output logs
- `/aws-glue/jobs/error` - for error logs
- `/aws-glue/jobs/logs-v2` - (optional) for continuous logging
2. You need to be sure that you fetch from all prefixes and this prefixes doesn't created in the same time
3. Be sure that fetch logging processes/threads do not use all CPU/Memory/IO of Airflow Worker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #23832: AWS Glue Hook to implement enhanced logging
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on issue #23832:
URL: https://github.com/apache/airflow/issues/23832#issuecomment-1189333129
Closed by https://github.com/apache/airflow/pull/25142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Dark-Knight11 commented on issue #23832: AWS Glue Hook to implement enhanced logging
Posted by GitBox <gi...@apache.org>.
Dark-Knight11 commented on issue #23832:
URL: https://github.com/apache/airflow/issues/23832#issuecomment-1137421187
Hi, I would like to work on this feature. Can you please guide me a little on how to proceed with it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org