[GitHub] [airflow] TobKed commented on a change in pull request #13478: Google Dataflow Flex Operator to handle no Job Type

[GitBox <gi...@apache.org>]

TobKed commented on a change in pull request #13478:
URL: https://github.com/apache/airflow/pull/13478#discussion_r551897070



##########
File path: airflow/providers/google/cloud/hooks/dataflow.py
##########
@@ -377,29 +377,32 @@ def _check_dataflow_job_state(self, job) -> bool:
         :rtype: bool
         :raise: Exception
         """
-        if self._wait_until_finished is None:
-            wait_for_running = job['type'] == DataflowJobType.JOB_TYPE_STREAMING

Review comment:
       what do you think about just using `wait_for_running = job.get('type', '') == DataflowJobType.JOB_TYPE_STREAMING` here?
   
   I am worried that skipping rest of the logic to wait for the job type may hang it forever.
   I can imagine situation where flex template job may be cancelled before state appear in payload.
   
   Another case is when job type is not necessary to return True:
   ```
               elif job['currentState'] in DataflowJobStatus.AWAITING_STATES:
                   return self._wait_until_finished is False
   ```
   
   cc @mik-laj 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org