You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Arunodoy18 (via GitHub)" <gi...@apache.org> on 2026/01/04 14:15:58 UTC

[PR] Fix/multiple executors queue routing [airflow]

Arunodoy18 opened a new pull request, #60102:
URL: https://github.com/apache/airflow/pull/60102

   Problem Summary
   
   In Apache Airflow 3.0.6+, when configuring multiple executors concurrently using a comma-separated executor list (for example CeleryExecutor,KubernetesExecutor), task routing based on the queue parameter does not work as expected.
   
   Observed Behavior
   
   All tasks are executed by the first (default) executor
   
   Tasks with queue='kubernetes' still run on:
   
   executor=CeleryExecutor(parallelism=32)
   
   
   No Kubernetes pods are created
   
   Expected Behavior
   
   Tasks without an explicit queue → CeleryExecutor
   
   Tasks with queue='kubernetes' → KubernetesExecutor
   
   This behavior differs from Airflow 2.x hybrid executors (e.g. CeleryKubernetesExecutor), which correctly supported queue-based routing via kubernetes_queue.
   
   Root Cause
   
   The scheduler’s executor selection logic in:
   
   _try_to_load_executor()
   
   
   only checks:
   
   Explicit ti.executor assignments
   
   It does not consider:
   
   The task’s queue parameter
   
   Executor-specific queue mappings (e.g. Kubernetes queues)
   
   As a result, when multiple executors are configured, all tasks fall back to the default executor, regardless of their queue.
   
   Solution Overview
   1️⃣ Enhanced Executor Selection Logic
   
   If ti.executor is explicitly set → use it (unchanged behavior)
   
   Otherwise:
   
   Match the task’s queue against executor-specific queue configurations
   
   Route the task to the matching executor before falling back to the default executor
   
   This preserves backward compatibility and aligns behavior with documented expectations.
   
   2️⃣ KubernetesExecutor Queue Configuration
   
   Introduced support for kubernetes_queue in the [[kubernetes_executor]] config section
   
   Default value:
   
   kubernetes_queue = ["kubernetes"]
   
   
   This mirrors the queue-based routing model used in Airflow 2.x hybrid executors.
   
   3️⃣ Configuration Template & Documentation Updates
   
   Added kubernetes_queue to:
   
   Configuration templates
   
   Provider documentation
   
   Ensures discoverability and correct usage
   
   4️⃣ Comprehensive Test Coverage
   
   Added tests to verify:
   
   ✅ Tasks with queue='kubernetes' route to KubernetesExecutor
   
   ✅ Tasks with other queues use the default CeleryExecutor
   
   ✅ Explicit executor assignments continue to work
   
   ✅ Multi-executor behavior remains unchanged
   
   ✅ No regressions or breaking changes
   
   Verification Results
   
   ✅ Kubernetes-queued tasks create Kubernetes pods
   
   ✅ Celery tasks continue to execute via CeleryExecutor
   
   ✅ Explicit executor overrides remain respected
   
   ✅ Backward compatible and safe for existing deployments
   
   Migration Impact
   Before (Airflow 3.0.6+)
   
   Queue-based routing ignored
   
   All tasks run on default executor
   
   After (Airflow 3.0+ with this fix)
   
   Queue-based routing works as documented
   
   Behavior consistent with Airflow 2.x hybrid executors
   
   No configuration changes required for existing users
   
   Conclusion
   
   This fix restores expected and documented behavior for multi-executor deployments in Airflow 3.x, enabling reliable queue-based task routing while maintaining backward compatibility and minimal risk.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix/multiple executors queue routing [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #60102:
URL: https://github.com/apache/airflow/pull/60102#issuecomment-3711702550

   @Arunodoy18 I am going to close your PRs -- Please review and test your changes with correct PR description. Using LLMs without those increase maintenance burdens and CI run time.
   
   Feel free to recreate focussed PRs following those guidelines.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix/multiple executors queue routing [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed pull request #60102: Fix/multiple executors queue routing
URL: https://github.com/apache/airflow/pull/60102


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org