[jira] [Created] (AIRFLOW-6388) SparkSubmitOperator polling should not 'consume' a slot

["t oo (Jira)" <ji...@apache.org>]

t oo created AIRFLOW-6388:
-----------------------------

             Summary: SparkSubmitOperator polling should not 'consume' a slot
                 Key: AIRFLOW-6388
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6388
             Project: Apache Airflow
          Issue Type: Improvement
          Components: dependencies, scheduler
    Affects Versions: 1.10.3
            Reporter: t oo


My Dag has tasks from 12 different types of operators. One of the operators is the dummyoperator (which is meant to do 'nothing') but it can't be run during busy times as the '{{parallelism}}, {{dag_concurrency}}, {{max_active_dag_runs_per_dag}}, {{non_pooled_task_slot_count' }}limits have been met (so it is stuck in scheduled state). I would like a new config flag (dont_block_dummy=True) with the ability for dummyOperator tasks to always get run even if the parallelism.etc limits are met. Without this feature, the only workaround for this is to make a huge parallelism limit (above now) and then give pools to all the other operators in my dag. But my idea is that dummyOperator should not have limits as it is not a resource hog.

 
h4. Task Instance Details
h5. Dependencies Blocking Task From Getting Scheduled
||Dependency||Reason||
|Unknown|All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless:
- The scheduler is down or under heavy load
- The following configuration values may be limiting the number of queueable processes: {{parallelism}}, {{dag_concurrency}}, {{max_active_dag_runs_per_dag}}, {{non_pooled_task_slot_count}}
 
If this task instance does not start soon please contact your Airflow administrator for assistance.|



--
This message was sent by Atlassian Jira
(v8.3.4#803005)