You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/09 17:09:22 UTC

[GitHub] [airflow] jaketf commented on pull request #8954: Wait for pipeline state in Data Fusion operators

jaketf commented on pull request #8954:
URL: https://github.com/apache/airflow/pull/8954#issuecomment-641453114


   Thanks for following up @turbaszek.
   *tl;dr* I think we should merge this PR as it fixes the immediate issue. We can file a lower priority issue to handle streaming pipelines in the future. This can be an additional kwarg that accepts a streaming flag and uses a different paths for polling. 
   
   I've updated the threads. I agree I think we should keep this PR small and focused on patching the existing operator for starting data fusion batch pipelines.
   
   In general I think batch is more used than streaming and spark is more used than MR.
   In batch both MR and spark can be polled at the .../DataPipelineWorkflow/runs/run_id endpoint.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org