[GitHub] [airflow] codenamestif edited a comment on issue #16753: Realtime ECS logging

[GitBox <gi...@apache.org>]

codenamestif edited a comment on issue #16753:
URL: https://github.com/apache/airflow/issues/16753#issuecomment-872581258


   @potiuk i have checked the example and it looks like it's a bit different. From my understanding that task handler is used for logging for the operator. In the example writing of the logs can be sync/async depends on provided transport type. Reading is always synchronous. But anyway, in ECSOperator we run a separated ECS task with a separated log stream. And what we need is to read logs from ECS task log stream and write them to ECSOperator logs.
   
   That's the code from ECSOperator:
   ```
           self._wait_for_task_ended()
   
           self._check_success_task()
   ```
   
   `self._wait_for_task_ended()` is blocking. And we read logs of the ECS task in `self._check_success_task()`. So my question is how to take the part that reads the logs from `self._check_success_task()` and have it running in parallel with ` self._wait_for_task_ended()`
   
   The only idea I have is not to use standard boto3 waiter, but have some custom wait implementation that pulls the status of the task and also pulls the logs in the same iteration. But that doesn't sound good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org