[jira] [Created] (AIRFLOW-5027) Generalized CloudWatch log grabbing for ECS and SageMaker operators

["Steven Reitsma (JIRA)" <ji...@apache.org>]

Steven Reitsma created AIRFLOW-5027:
---------------------------------------

             Summary: Generalized CloudWatch log grabbing for ECS and SageMaker operators
                 Key: AIRFLOW-5027
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5027
             Project: Apache Airflow
          Issue Type: Improvement
          Components: aws, contrib, logging
    Affects Versions: 1.10.3, 1.10.4
            Reporter: Steven Reitsma
            Assignee: Steven Reitsma


It's very useful to have the CloudWatch task logs of an ECS/SageMaker job available within Airflow. Currently, this is not possible for ECS tasks.

The SageMaker operator starts a job on AWS SageMaker. It already supports grabbing the CloudWatch logs of the (finished) job to the Airflow instance.

The ECS operator is very similar in that it starts a job on AWS ECS. It doesn't, however, support grabbing CloudWatch logs at the end of a job.

I've separated the existing log grabbing of the SageMaker operator to a hook and both the SageMaker and ECS operators now call that hook at the end of a job.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)