You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "goncalo-m-c (via GitHub)" <gi...@apache.org> on 2025/09/18 12:25:02 UTC

[PR] Customize statsd dagprocessor labels [airflow]

goncalo-m-c opened a new pull request, #55832:
URL: https://github.com/apache/airflow/pull/55832

   TL;DR: Add support in Helm chart for custom labels in redis, statsd and dagProcessor objects
   
   ### Description
   I would like to be able to customize the labels defined for Airflow Kubernetes resources so I can comply with my company guidelines and be able to track all objects in the same way.
   
   This PR adds the ability to specify custom labels for all Airflow objects and pods defined in the Helm chart. Labels can be set globally through `.Values.labels` and component-specifically through `<component>.labels`. These labels are merged, with component-specific labels taking precedence.
   
   ### Changes
   - Added labels property to redis, statsd and dagProcessor components.
   - Updated values.schema.json to include schema definitions for label fields
   - Added documentation in `chart/docs/customizing-labels-for-pods.rst` explaining the labeling system
   
   ### Example Usage
   
   ```yaml
   # Global labels for all objects and pods
   labels:
     environment: production
   
   # Component-specific labels
   scheduler:
     labels:
       role: scheduler
   
   workers:
     labels:
       role: worker
   
   webserver:
     labels:
       role: ui
   ```
   
   ### Checklist
   
   - [x] Description above provides context of the change
   - [x] Added schema definitions for new configuration options
   - [x] Documented new values in values.yaml in docs/
   - [x] Used `mustMerge` to properly handle label merging
   - [x] No breaking changes introduced
   
   ### Additional Notes
   Deployments are currently only labeled with global labels and I don't know if this is done for a specific reason. If possible, I would also like to implement custom labels for Deployments.
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3474585637

   Oh, still a minor fix is needed as the CLI output of the breeze tooling needs a bit of updates. Can you run `prek run -a update-breeze-cmd-output` to get fixes and commit the files to the branch of the PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "goncalo-m-c (via GitHub)" <gi...@apache.org>.
goncalo-m-c commented on code in PR #55832:
URL: https://github.com/apache/airflow/pull/55832#discussion_r2435195538


##########
chart/templates/statsd/statsd-serviceaccount.yaml:
##########
@@ -32,8 +32,8 @@ metadata:
     release: {{ .Release.Name }}
     chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
     heritage: {{ .Release.Service }}
-    {{- with .Values.labels }}
-      {{- toYaml . | nindent 4 }}
+    {{- if or (.Values.labels) (.Values.statsd.labels) }}

Review Comment:
   There is no functional difference, I used it because it's used in other manifests in this chart. But the official Helm docs consistently show no parenthesis (`or value1 value2`) so I assume that is the recommended way. Let's just follow that. Parenthesis seem to make more sense in cases where we nest functions are used, like `deepEqual (list 1 2 3) (list 1 2 3)` (example [here](https://helm.sh/docs/chart_template_guide/function_list/#deepequal)). 



##########
chart/docs/customizing-labels.rst:
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+Customizing Labels for Pods
+---------------------------
+
+The Helm chart allows you to customize labels for your Airflow objects. You can set global labels that apply to all objects and pods defined in the chart, as well as component-specific labels for individual Airflow components.
+
+Global Labels
+~~~~~~~~~~~~~
+
+Global labels can be set using the ``labels`` parameter in your values file. These labels will be applied to all Airflow objects and pods defined in the chart:
+
+.. code-block:: yaml
+
+    labels:
+      environment: production
+
+Component-Specific Labels
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can also set specific labels for individual Airflow components. These labels will be merged with the global labels, with component-specific labels taking precedence.
+
+For example, to add specific labels to different components:
+
+.. code-block:: yaml
+
+    # Global labels applied to all pods
+    labels:
+      environment: production
+
+    # Scheduler specific labels
+    scheduler:
+      labels:
+        role: scheduler
+
+    # Worker specific labels
+    workers:
+      labels:
+        role: worker
+
+    # Webserver specific labels
+    webserver:
+      labels:
+        role: ui
+
+.. note::
+    Component-specific labels take precedence over global labels when there are conflicts. This allows you to override global labels for specific components when needed.

Review Comment:
   Makes sense, I'll clarify that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3482537665

   Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3464535387

   CI fails now in some static tests. Can you run pre-commit checks which we implemented with prek? (There is also a development quick start guide) --> https://github.com/apache/airflow/blob/main/contributing-docs/03a_contributors_quick_start_beginners.rst
   That will fix the static checks or tell you what need to be changed to make CI happy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3307173179

   Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
   - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
   - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
   - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
   Apache Airflow is a community-driven project and together we are making it better 🚀.
   In case of doubts contact the developers at:
   Mailing List: dev@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "goncalo-m-c (via GitHub)" <gi...@apache.org>.
goncalo-m-c commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3479503964

   > Oh, still a minor fix is needed as the CLI output of the breeze tooling needs a bit of updates. Can you run `prek run -a update-breeze-cmd-output` to get fixes and commit the files to the branch of the PR
   
   Updated ✅ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "EMNs (via GitHub)" <gi...@apache.org>.
EMNs commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3311570143

   Looks good! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on code in PR #55832:
URL: https://github.com/apache/airflow/pull/55832#discussion_r2437471735


##########
chart/templates/statsd/statsd-serviceaccount.yaml:
##########
@@ -32,8 +32,8 @@ metadata:
     release: {{ .Release.Name }}
     chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
     heritage: {{ .Release.Service }}
-    {{- with .Values.labels }}
-      {{- toYaml . | nindent 4 }}
+    {{- if or (.Values.labels) (.Values.statsd.labels) }}

Review Comment:
   In functions or nested conditionals - yes. Regarding the occurrence in the chart, it is not consistent. In some places we have them, and in some we don't.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on code in PR #55832:
URL: https://github.com/apache/airflow/pull/55832#discussion_r2434139841


##########
chart/templates/statsd/statsd-serviceaccount.yaml:
##########
@@ -32,8 +32,8 @@ metadata:
     release: {{ .Release.Name }}
     chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
     heritage: {{ .Release.Service }}
-    {{- with .Values.labels }}
-      {{- toYaml . | nindent 4 }}
+    {{- if or (.Values.labels) (.Values.statsd.labels) }}

Review Comment:
   I saw in couple of places that people write:
   ```
   or (value1) (value2)
   ```
   but I think the below will work also:
   ```
   or value1 value2
   ```
   Is there any difference between these two? I tried to find something, but without any success.



##########
chart/docs/customizing-labels.rst:
##########
@@ -0,0 +1,62 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+Customizing Labels for Pods
+---------------------------
+
+The Helm chart allows you to customize labels for your Airflow objects. You can set global labels that apply to all objects and pods defined in the chart, as well as component-specific labels for individual Airflow components.
+
+Global Labels
+~~~~~~~~~~~~~
+
+Global labels can be set using the ``labels`` parameter in your values file. These labels will be applied to all Airflow objects and pods defined in the chart:
+
+.. code-block:: yaml
+
+    labels:
+      environment: production
+
+Component-Specific Labels
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can also set specific labels for individual Airflow components. These labels will be merged with the global labels, with component-specific labels taking precedence.
+
+For example, to add specific labels to different components:
+
+.. code-block:: yaml
+
+    # Global labels applied to all pods
+    labels:
+      environment: production
+
+    # Scheduler specific labels
+    scheduler:
+      labels:
+        role: scheduler
+
+    # Worker specific labels
+    workers:
+      labels:
+        role: worker
+
+    # Webserver specific labels
+    webserver:
+      labels:
+        role: ui
+
+.. note::
+    Component-specific labels take precedence over global labels when there are conflicts. This allows you to override global labels for specific components when needed.

Review Comment:
   The first paragraph under the `Component-Specific Labels` section is, in a way, duplicating that information. Maybe we should combine these two to not repeat information and potentially miss one more place to change?



##########
chart/values.yaml:
##########
@@ -2174,6 +2174,9 @@ dagProcessor:
     securityContexts:
       container: {}
 
+  # Labels specific to dag processor objects and pods
+  labels: {}
+  # Environment variables to add to dag processor container

Review Comment:
   ```suggestion
     # Labels specific to dag processor objects
     labels: {}
   
     # Environment variables to add to dag processor container
   ```
   I think that `pods` are also `objects`, so the latter should not be needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #55832:
URL: https://github.com/apache/airflow/pull/55832


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "goncalo-m-c (via GitHub)" <gi...@apache.org>.
goncalo-m-c commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3410353679

   @Miretpl btw, I still think that the whole label configuration is a bit inconsistent. For example, at the moment we can customize labels for Pods created under a Deployment ([here](https://github.com/apache/airflow/blob/main/chart/templates/webserver/webserver-deployment.yaml#L87-L89)), but we cannot customize the Deployment labels themselves ([here](https://github.com/apache/airflow/blob/main/chart/templates/webserver/webserver-deployment.yaml#L43-L45)). 
   
   I didn't add such changes in this PR to keep consistency with the other resources, but I will open a proposal for that separately.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "goncalo-m-c (via GitHub)" <gi...@apache.org>.
goncalo-m-c commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3460858799

   > Thanks for the PR as well as also the documentation added.
   > 
   > Just a small nit - can you add a small test case as well to ensure the function is not degrading with some changes applied?
   
   Thanks for reviewing @jscheffl, can you clarify your request? I've just added new unit tests to reflect the changes I am proposing. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "goncalo-m-c (via GitHub)" <gi...@apache.org>.
goncalo-m-c commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3410100881

   > Nice change!
   > 
   > Could you add the test cases which would valided the changed behaviour of the labels?
   
   Thanks for reviewing! I've added 3 test cases in the PR description


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3412864999

   @goncalo-m-c yeah, there are a lot of like unfinished things in the chart or inconsistencies. I agree that the helm chart needs a little more love from the whole community.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "goncalo-m-c (via GitHub)" <gi...@apache.org>.
goncalo-m-c commented on code in PR #55832:
URL: https://github.com/apache/airflow/pull/55832#discussion_r2435123847


##########
chart/values.yaml:
##########
@@ -2174,6 +2174,9 @@ dagProcessor:
     securityContexts:
       container: {}
 
+  # Labels specific to dag processor objects and pods
+  labels: {}
+  # Environment variables to add to dag processor container

Review Comment:
   Good point, fixed 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Customize statsd dagprocessor labels [airflow]

Posted by "goncalo-m-c (via GitHub)" <gi...@apache.org>.
goncalo-m-c commented on PR #55832:
URL: https://github.com/apache/airflow/pull/55832#issuecomment-3468171492

   > CI fails now in some static tests. Can you run pre-commit checks which we implemented with prek? (There is also a development quick start guide) --> https://github.com/apache/airflow/blob/main/contributing-docs/03a_contributors_quick_start_beginners.rst That will fix the static checks or tell you what need to be changed to make CI happy.
   
   My bad, I did not enable those. However, I cannot get all the checks to work properly both locally and on a clean Codespace. What I was able to do was to run only the hooks that were failing, commit the changes, and they are passing now.
   ```
   # prek end-of-file-fixer ruff-format update-breeze-cmd-output --directory helm-tests
   Make sure that there is an empty line at the end.........................Passed
   Run 'ruff format'........................................................Passed
   Update breeze docs...................................(no files to check)Skipped
   ```
   
   Changes are in a [separate commit](https://github.com/apache/airflow/pull/55832/commits/ecb44e9c40032e96a881691720581b0ffc8ce898)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org