[GitHub] [airflow] mik-laj commented on a change in pull request #6386: [AIRFLOW-5716][part of AIRFLOW-5697][depends on AIRFLOW-5711] Simplify DataflowJobsController logic

[GitBox <gi...@apache.org>]

mik-laj commented on a change in pull request #6386: [AIRFLOW-5716][part of AIRFLOW-5697][depends on AIRFLOW-5711] Simplify DataflowJobsController logic
URL: https://github.com/apache/airflow/pull/6386#discussion_r344175358
 
 

 ##########
 File path: airflow/gcp/hooks/dataflow.py
 ##########
 @@ -334,15 +357,16 @@ def _start_dataflow(
         name: str,
         command_prefix: List[str],
         label_formatter: Callable[[Dict], List[str]],
-        multiple_jobs: bool = False
+        project_id: str,
+        multiple_jobs: bool = False,
     ) -> None:
         variables = self._set_variables(variables)
-        cmd = command_prefix + self._build_cmd(variables, label_formatter)
+        cmd = command_prefix + self._build_cmd(variables, label_formatter, project_id)
         runner = _DataflowRunner(cmd)
         job_id = runner.wait_for_done()
 
 Review comment:
   We can't split this one method into two methods because a process is being run that supervisor the task.  Unfortunately, this is a limitation of Apache Beam, which does not have the option of forcing external supervision.  In any case, we must wait until the Apache Beam system process is completed to be sure of completing the job.
   
   This operator can also be used to initiate streaming jobs, but we lack the operator to stop the task if we want to handle your process fully.
   https://github.com/apache/airflow/blob/master/tests/gcp/hooks/test_dataflow.py#L428-L458

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services