You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "stephen-bracken (via GitHub)" <gi...@apache.org> on 2026/07/17 15:46:11 UTC
[PR] Add subdomain to worker pods in Deployments [airflow]
stephen-bracken opened a new pull request, #70026:
URL: https://github.com/apache/airflow/pull/70026
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
Add subdomain to worker pods in Deployments
---
When not using a StatefulSet, setting the `spec.subdomain` property on the pod to the service name will return the fully qualified service address for the pod when calling `airflow.utils.net.getfqdn`, allowing the UI to resolve the worker address and retrieve the logs.
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
- [X] No
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on code in PR #70026:
URL: https://github.com/apache/airflow/pull/70026#discussion_r4041431109
##########
chart/tests/helm_tests/airflow_core/test_worker.py:
##########
@@ -1552,6 +1552,16 @@ def test_overwrite_extra_ports(self):
{"name": "test-extra-port", "containerPort": 10}
]
+ def test_worker_subdomain_matches_service(self):
Review Comment:
I see that I wasn't precise in my comment, but the result is what I meant. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #70026:
URL: https://github.com/apache/airflow/pull/70026#issuecomment-5690156736
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "stephen-bracken (via GitHub)" <gi...@apache.org>.
stephen-bracken commented on PR #70026:
URL: https://github.com/apache/airflow/pull/70026#issuecomment-5057592039
> I assume that the goal of this PR is to make the set of core.hostname_callable=airflow.utils.net.get_host_ip_address not needed (correct me if I'm wrong).
Partially yes. In order to make the calls work through the service we would need to make a new method that strips the pod name from the FQDN returned by `airflow.utils.net.getfqdn`, e.g.
```python
def get_host_service_name():
"""Service-aware hostname callable.
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#a-aaaa-records-1
"""
from airflow.utils.net import getfqdn
fqdn = getfqdn()
return ".".join(fqdn.split(".")[1:])
```
Without the subdomain this method does not work, as the pod's FQDN will only return the pod name without the subdomain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "stephen-bracken (via GitHub)" <gi...@apache.org>.
stephen-bracken commented on code in PR #70026:
URL: https://github.com/apache/airflow/pull/70026#discussion_r3672728482
##########
chart/tests/helm_tests/airflow_core/test_worker.py:
##########
@@ -1552,6 +1552,16 @@ def test_overwrite_extra_ports(self):
{"name": "test-extra-port", "containerPort": 10}
]
+ def test_worker_subdomain_matches_service(self):
Review Comment:
If persistence is disabled then the StatefulSet will be used, which sets `spec.serviceName` instead of `spec.template.spec.subdomain`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on PR #70026:
URL: https://github.com/apache/airflow/pull/70026#issuecomment-5051612949
I assume that the goal of this PR is to make the set of `core.hostname_callable=airflow.utils.net.get_host_ip_address` not needed (correct me if I'm wrong). I checked the change locally, and it doesn't work. Could you provide the configuration and how you tested it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl merged PR #70026:
URL: https://github.com/apache/airflow/pull/70026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #70026:
URL: https://github.com/apache/airflow/pull/70026#issuecomment-5721233522
### Backport failed to create: chart/v1-2x-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/35274723767'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>chart/v1-2x-test</td>
<td><a href="https://github.com/apache/airflow/commit/e03cfa2f925495be3f8c5b751505aaaa29a58df4"><img src='https://img.shields.io/badge/Commit-e03cfa2-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker e03cfa2 chart/v1-2x-test
```
This should apply the commit to the chart/v1-2x-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on code in PR #70026:
URL: https://github.com/apache/airflow/pull/70026#discussion_r3659887192
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -123,6 +123,11 @@ spec:
{{- tpl (toYaml $podAnnotations) . | nindent 8 }}
{{- end }}
spec:
+ {{- if not $persistence }}
+ # Fix TaskInstance hostname service resolution for Deployments
+ # https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-hostname-and-subdomain-field
Review Comment:
```suggestion
```
This is Kubernetes-specific knowledge that the reader should already have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "stephen-bracken (via GitHub)" <gi...@apache.org>.
stephen-bracken commented on PR #70026:
URL: https://github.com/apache/airflow/pull/70026#issuecomment-5057665688
I think additional changes will be needed to make `getfqdn` work with worker Deployments, but this is a required first step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add subdomain to worker pods in Deployments [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on code in PR #70026:
URL: https://github.com/apache/airflow/pull/70026#discussion_r3669273514
##########
chart/tests/helm_tests/airflow_core/test_worker.py:
##########
@@ -1552,6 +1552,16 @@ def test_overwrite_extra_ports(self):
{"name": "test-extra-port", "containerPort": 10}
]
+ def test_worker_subdomain_matches_service(self):
Review Comment:
Small nit: I would parameterise this test by persistence to make sure that the value is the same for the default StatefulSet behaviour and the Deployment one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org