[GitHub] [airflow] potiuk commented on a diff in pull request #27286: System Test for EMR (AIP-47)

[GitBox <gi...@apache.org>]

potiuk commented on code in PR #27286:
URL: https://github.com/apache/airflow/pull/27286#discussion_r1014856445


##########
airflow/providers/amazon/aws/operators/emr.py:
##########
@@ -113,11 +118,21 @@ def execute(self, context: Context) -> list[str]:
 
         response = emr.add_job_flow_steps(JobFlowId=job_flow_id, Steps=steps)
 
-        if not response["ResponseMetadata"]["HTTPStatusCode"] == 200:
+        if response["ResponseMetadata"]["HTTPStatusCode"] != 200:
             raise AirflowException(f"Adding steps failed: {response}")
-        else:
-            self.log.info("Steps %s added to JobFlow", response["StepIds"])
-            return response["StepIds"]
+
+        self.log.info("Steps %s added to JobFlow", response["StepIds"])
+        if self.wait_for_completion:
+            for step_id in response["StepIds"]:
+                boto3.client("emr").get_waiter("step_complete").wait(
+                    ClusterId=self.job_flow_id,
+                    StepId=step_id,
+                    WaiterConfig={
+                        "Delay": 5,
+                        "MaxAttempts": 100,
+                    },
+                )
+        return response["StepIds"]

Review Comment:
   Yep. It would be better. Agree.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org