[PR] GlueJobOperator: add option to wait for cleanup before returning job status [airflow]

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal opened a new pull request, #43688:
URL: https://github.com/apache/airflow/pull/43688

   This changes to solve a bug around the case of concurrency=1. In that scenario Glue returns final state before actually cleaning up the resources. This leads to a problem we can have more than 1 concurrent job in Glue.
   To avoid this and according to AWS recommendation adding the option to wait (5-10) seconds before returning the job status 
   
   Not sure how to test this one (if we need testing for this at all?).
   It just adds sleep before returning the final status
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GlueJobOperator: add option to wait for cleanup before returning job status [airflow]

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal merged PR #43688:
URL: https://github.com/apache/airflow/pull/43688


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org