You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "uplsh580 (via GitHub)" <gi...@apache.org> on 2026/01/16 15:45:16 UTC
[PR] Improve dag_bundle_config_list Configuration in Airflow Helm Chart [airflow]
uplsh580 opened a new pull request, #60645:
URL: https://github.com/apache/airflow/pull/60645
# Overview
Related Issue: https://github.com/apache/airflow/issues/60630
Improve `dag_bundle_config_list` Configuration in Airflow Helm Chart
This PR improves the `dag_bundle_config_list` configuration in the Airflow Helm Chart by allowing users to define DAG bundle configurations in a structured YAML format instead of requiring a JSON string directly in the values file.
# Changes
- Added a new `dag_bundles` section in `values.yaml` that allows users to define DAG bundles in a structured YAML format
- Created a Helm template helper function `dag_bundles` that automatically converts the YAML structure to the required JSON string format
- Set default value for `dag_bundles` with a local DAG bundle configuration
- Updated `config.dag_processor.dag_bundle_config_list` to use the helper function
# Benefits
1. **Better Readability**: Long JSON strings in YAML files are difficult to read and maintain. The new structured format is much more readable.
2. **Less Error-Prone**: Manual JSON string construction is prone to syntax errors. The YAML format reduces the chance of errors.
3. **Better IDE Support**: IDEs can provide proper syntax highlighting, validation, and autocomplete for YAML structures.
4. **Easier Maintenance**: Adding, removing, or modifying bundle configurations is now straightforward.
# Usage
Users can now define DAG bundles like this:
```yaml
dag_bundles:
- name: dags-folder
classpath: "airflow.dag_processing.bundles.local.LocalDagBundle"
kwargs: {}
- name: bundle1
classpath: "airflow.providers.git.bundles.git.GitDagBundle"
kwargs:
git_conn_id: "GITHUB__repo1"
subdir: "dags"
tracking_ref: "main"
refresh_interval: 60
- name: bundle2
classpath: "airflow.providers.git.bundles.git.GitDagBundle"
kwargs:
git_conn_id: "GITHUB__repo2"
subdir: "dags"
tracking_ref: "develop"
refresh_interval: 120
```
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [X] Yes (Cursor)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve dag_bundle_config_list Configuration in Airflow Helm Chart [airflow]
Posted by "uplsh580 (via GitHub)" <gi...@apache.org>.
uplsh580 commented on PR #60645:
URL: https://github.com/apache/airflow/pull/60645#issuecomment-3760694326
## Validation
<img width="928" height="237" alt="스크린샷 2026-01-17 오전 12 36 27" src="https://github.com/user-attachments/assets/175f4ee7-0576-45f7-bdb5-0994ca3fe3ac" />
## Limitation
There is a limitation that multi-line json in `airflow.cfg` is not supported.
<img width="925" height="170" alt="스크린샷 2026-01-17 오전 12 35 56" src="https://github.com/user-attachments/assets/4afb7702-20ce-453f-917e-909eab024091" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve dag_bundle_config_list Configuration in Airflow Helm Chart [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #60645:
URL: https://github.com/apache/airflow/pull/60645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve dag_bundle_config_list Configuration in Airflow Helm Chart [airflow]
Posted by "uplsh580 (via GitHub)" <gi...@apache.org>.
uplsh580 commented on PR #60645:
URL: https://github.com/apache/airflow/pull/60645#issuecomment-3764669823
> Some tests are missing - can you add some please?
Thank you for the feedback. I've updated the PR with the following changes:
- Added unit tests for `dagProcessor.dagBundleConfigList`.
- Marked the setting as deprecated in the comments to avoid overwhelming users with multiple parallel options, as suggested.
- Standardized variable names to align with the style of other values in the Helm chart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve dag_bundle_config_list Configuration in Airflow Helm Chart [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #60645:
URL: https://github.com/apache/airflow/pull/60645#discussion_r2701686103
##########
chart/values.yaml:
##########
@@ -2267,6 +2267,34 @@ triggerer:
# Airflow Dag Processor Config
dagProcessor:
enabled: ~
+
+ # DAG Bundle Configuration
+ # Define DAG bundles in a structured YAML format. This will be automatically
+ # converted to JSON string format for config.dag_processor.dag_bundle_config_list.
+ dag_bundles:
Review Comment:
THanks for putting this below dagProcessor - exactly this I would have recommended after reading the realted issue (where it was a top-level setting).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve dag_bundle_config_list Configuration in Airflow Helm Chart [airflow]
Posted by "uplsh580 (via GitHub)" <gi...@apache.org>.
uplsh580 commented on PR #60645:
URL: https://github.com/apache/airflow/pull/60645#issuecomment-3764946497
> One small nit suggestion:
> we’ve agreed to refer to DAG as Dag going forward.
> It would be great if you could update the occurrences accordingly :)
>
> [VOTE] results as of https://lists.apache.org/thread/0651j4vdgzmlhgndmotvznlg97kyh328
Great catch, thank you for sharing the discussion link. I've gone through the PR and adjusted `DAG` to `Dag` to stay consistent with the community's decision. Thanks for your guidance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve dag_bundle_config_list Configuration in Airflow Helm Chart [airflow]
Posted by "uplsh580 (via GitHub)" <gi...@apache.org>.
uplsh580 commented on code in PR #60645:
URL: https://github.com/apache/airflow/pull/60645#discussion_r2701968024
##########
chart/values.yaml:
##########
@@ -2267,6 +2267,34 @@ triggerer:
# Airflow Dag Processor Config
dagProcessor:
enabled: ~
+
+ # DAG Bundle Configuration
+ # Define DAG bundles in a structured YAML format. This will be automatically
+ # converted to JSON string format for config.dag_processor.dag_bundle_config_list.
+ dag_bundles:
Review Comment:
Thank you. I also concluded that this was the better location during the development process. I'm glad our opinions align on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve dag_bundle_config_list Configuration in Airflow Helm Chart [airflow]
Posted by "uplsh580 (via GitHub)" <gi...@apache.org>.
uplsh580 commented on code in PR #60645:
URL: https://github.com/apache/airflow/pull/60645#discussion_r2699052554
##########
chart/values.yaml:
##########
@@ -3268,6 +3296,11 @@ config:
statsd_host: '{{ printf "%s-statsd" (include "airflow.fullname" .) }}'
# `run_duration` included for Airflow 1.10 backward compatibility; removed in 2.0.
run_duration: 41460
+ dag_processor:
+ # DAG bundle configuration list in JSON string format.
+ # This is automatically generated from .Values.dag_bundles using the dag_bundles helper function.
+ # You can also provide a custom JSON string directly to override the generated value.
+ dag_bundle_config_list: '{{ include "dag_bundles" . }}'
Review Comment:
Users can override this value to ignore the `dag_bundles` configuration, maintaining backward compatibility with the approach of directly providing a JSON string in config.dag_processor.dag_bundle_config_list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve dag_bundle_config_list Configuration in Airflow Helm Chart [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #60645:
URL: https://github.com/apache/airflow/pull/60645#discussion_r2701688854
##########
chart/values.yaml:
##########
@@ -3268,6 +3296,11 @@ config:
statsd_host: '{{ printf "%s-statsd" (include "airflow.fullname" .) }}'
# `run_duration` included for Airflow 1.10 backward compatibility; removed in 2.0.
run_duration: 41460
+ dag_processor:
+ # DAG bundle configuration list in JSON string format.
+ # This is automatically generated from .Values.dagProcessor.dag_bundles using the dag_bundles helper function.
+ # You can also provide a custom JSON string directly to override the generated value.
+ dag_bundle_config_list: '{{ include "dag_bundles" . }}'
Review Comment:
I'd actually rather mark this as deprecated as multiple parallel settings might be overwhelming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org