[GitHub] [airflow] RachaelDS commented on a change in pull request #8954: Wait for pipeline state in Data Fusion operators

[GitBox <gi...@apache.org>]

RachaelDS commented on a change in pull request #8954:
URL: https://github.com/apache/airflow/pull/8954#discussion_r437672204



##########
File path: airflow/providers/google/cloud/hooks/datafusion.py
##########
@@ -386,15 +480,29 @@ def start_pipeline(
             pipeline_name,
             "workflows",
             "DataPipelineWorkflow",
-            "start"
+            "start",
         )
+        runtime_args = runtime_args or {}
+        # Unfortunately making the start call to CDAP does not return a run_id to poll for state.

Review comment:
       You can avoid using the faux run Id by making a call to the batch start pipeline endpoint - the run id will be returned in this case. For example:
   TYPE: POST
   URL: 'https://xxx.datafusion.googleusercontent.com/api/v3/namespaces/default/start'
   BODY: 
   [{"appId": "app_id", "programType": "workflow", "programId": "DataPipelineWorkflow","runtimeargs": {}}]
   
   Batch start pipeline endpoint info:
   https://docs.cdap.io/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#H3293 (documentation does not currently reflect that the run Id is returned)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org