[GitHub] [airflow] kaxil commented on pull request #26156: Add deferrable big query operators and sensors

[GitBox <gi...@apache.org>]

kaxil commented on PR #26156:
URL: https://github.com/apache/airflow/pull/26156#issuecomment-1239325550

   > Can we merge the operators with existing one and add parameter to execute it in the deferrable mode? We did this already in the Cloud Composer and Dataproc operators. In my opinion we should have one standard when we want to add deferrable operators. Most of the logic in new operators are the same like we have in non-deferrable operators, this will be nightmare to maintain in the future because if we will have bug in one of it is high probality that we have it also in the other one.
   > 
   > Also in the non-deferrable operators we have links already to the resources inside GCP platform, so it will be good to have them also in the deferrable operators.
   
   We don't need to merge them into one. We can inheirt from the parent operator and re-use them when needed:
   
   1. Merging them makes it difficult for Sensors where we already have a "poke" and "reschedule" mode. Does that mean we add a "new mode" for "deferrable" for sensors but for operators we just have "deferrable" as kwarg?
   2. Not all Async operators will be drop-in replacements, some of them will have limitations as compared to sync counterparts, for e.g `SimpleHTTPOperator` supports a callback whereas with Async operators can't support that
   3. It makes tracking the adoption of async operator a little trickier.
   4. Debugging becomes difficult


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org