[I] Multiple dag-processors with partial bundle configs deactivate each other's bundles [airflow]

["sortega (via GitHub)" <gi...@apache.org>]

sortega opened a new issue, #69963:
URL: https://github.com/apache/airflow/issues/69963

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.2.2 (also present on `main`/3.3.0; the affected code is unchanged)
   
   ### What happened and how to reproduce it?
   
   **Summary**
   
   When multiple `dag-processor` processes are each configured with a *partial* bundle configuration — i.e. each processor's `[dag_processor] dag_bundle_config_list` lists only the bundle(s) it owns — the processors repeatedly deactivate each other's bundles and the deployment never converges. This is a natural way to shard DAG parsing across processors (one bundle per processor), and it deadlocks.
   
   **Root cause**
   
   `DagBundlesManager.sync_bundles_to_db()` loads *all* bundles stored in the DB, pops the ones present in *this process's* config, and then unconditionally marks every remaining stored bundle inactive:
   
   ```python
   for name, bundle in stored.items():
       bundle.active = False
       bundle.teams = []
       self.log.warning("DAG bundle %s is no longer found in config and has been disabled", name)
       session.execute(delete(ParseImportError).where(ParseImportError.bundle_name == name))
   ```
   
   The method implicitly assumes the calling process sees the *complete* bundle configuration. When each processor only sees its own bundle, every processor treats all other processors' bundles as "no longer in config" and disables them. Processor A disables B's bundle; B then disables A's; on the next cycle they flip again. The log line
   
   ```
   DAG bundle <name> is no longer found in config and has been disabled
   ```
   
   is emitted continuously for bundles that are, in fact, actively configured on another processor.
   
   **Steps to reproduce**
   
   1. Configure two `dag-processor` processes.
   2. Give processor A `dag_bundle_config_list = [bundle-a]` and processor B `dag_bundle_config_list = [bundle-b]` (disjoint, per-processor configs).
   3. Let both run `sync_bundles_to_db()` (happens on each parse loop startup).
   4. Observe the `dag_bundle` table: `bundle-a.active` and `bundle-b.active` flip between `True`/`False` on every cycle, and the "no longer found in config and has been disabled" warning is logged for a bundle that is actively configured elsewhere.
   
   Minimal reproduction of the flip-flop (two managers with disjoint config, no error path involved):
   
   ```
   AFTER B syncs:       [('bundle-a', False), ('bundle-b', True)]   # B disabled A
   AFTER A syncs again: [('bundle-a', True),  ('bundle-b', False)]  # A disabled B
   ```
   
   ### What you think should happen instead?
   
   A dag-processor that only owns a subset of the bundle configuration must not deactivate bundles it does not know about. Deactivation of genuinely-removed bundles should only happen when the process sees the full configuration.
   
   Proposed fix: add a `deactivate_missing` flag to `sync_bundles_to_db()` (default `True`, preserving current single-processor behavior), and have `DagFileProcessorManager.sync_bundles()` pass `deactivate_missing=False` when the processor was started with a bundle filter (`bundle_names_to_parse`, i.e. `--bundle-name`). A PR implementing this follows.
   
   Related: #69698 addresses a different over-eager-deactivation path in the same method (transient errors during bundle template extraction). This issue is about partial per-processor configuration, not error handling.
   
   ### Operating System
   
   Linux (Kubernetes; also reproduced locally on macOS)
   
   ### Versions of Apache Airflow Providers
   
   N/A (airflow-core)
   
   ### Deployment
   
   Other 3rd-party Helm chart
   
   ### Deployment details
   
   Multiple `dag-processor` deployments, each with its own `AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST` containing only the bundle that processor is responsible for, and started with `--bundle-name` to restrict parsing.
   
   ### Anything else?
   
   The affected code path (`sync_bundles_to_db`) is identical on `main` (3.3.0/3.4.0).
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Multiple dag-processors with partial bundle configs deactivate each other's bundles [airflow]

["hussein-awala (via GitHub)" <gi...@apache.org>]

hussein-awala closed issue #69963: Multiple dag-processors with partial bundle configs deactivate each other's bundles
URL: https://github.com/apache/airflow/issues/69963


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org