[jira] [Created] (AIRFLOW-5567) Improved primitive for building Operators that benefit from reschedule mode

["Jacob Ferriero (Jira)" <ji...@apache.org>]

Jacob Ferriero created AIRFLOW-5567:
---------------------------------------

             Summary: Improved primitive for building Operators that benefit from reschedule mode
                 Key: AIRFLOW-5567
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5567
             Project: Apache Airflow
          Issue Type: Improvement
          Components: models, operators
    Affects Versions: 1.10.5
            Reporter: Jacob Ferriero
            Assignee: Jacob Ferriero


Often times airflow operators (derived from BaseOperator) kick-off a long running tasks and then waits / polls, blocking a worker slot until the long running task completes. This can be problematic in environments with many long running tasks.

BaseSensorOperator was improved by implementing `reschedule` mode to solve the similar issue with long running sensors blocking a worker to poll for a long time.

This issue is to track how we could provide a primitive that would make it easy to develop operators for long running tasks that reschedule a `poll` operation rather than blocking in their `execute` method.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)