You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/19 16:59:56 UTC

[GitHub] [airflow] Taragolis commented on a diff in pull request #28464: Add waiter config params to emr.add_job_flow_steps

Taragolis commented on code in PR #28464:
URL: https://github.com/apache/airflow/pull/28464#discussion_r1052434511


##########
airflow/providers/amazon/aws/hooks/emr.py:
##########
@@ -126,14 +126,21 @@ def create_job_flow(self, job_flow_overrides: dict[str, Any]) -> dict[str, Any]:
         return response
 
     def add_job_flow_steps(
-        self, job_flow_id: str, steps: list[dict] | str | None = None, wait_for_completion: bool = False
+        self,
+        job_flow_id: str,
+        steps: list[dict] | str | None = None,
+        wait_for_completion: bool = False,
+        waiter_delay: int = 5,
+        waiter_max_attempts: int = 100,

Review Comment:
   Hmm.. what do you think if by default `waiter_delay` and `waiter_max_attempts` set to `None`  and use in waiter config only if this values not None?
   
   In this case default values from `botocore` waiters definition file will use: delay=30, max_attempts=60.
   
   Something like that
   
   ```python
   waiter.wait(
       ClusterId=job_flow_id,
       StepId=step_id,
       WaiterConfig=trim_none_values(
           {
               "Delay": waiter_delay,
               "MaxAttempts": waiter_max_attempts,
           }
       ),
   )
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org