You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Lee-W (via GitHub)" <gi...@apache.org> on 2026/07/22 09:39:37 UTC
[PR] Fix conn-fields check crash for nested provider packages [airflow]
Lee-W opened a new pull request, #70224:
URL: https://github.com/apache/airflow/pull/70224
check_provider_yaml_files.py mapped any changed Python file to "<first-path-segment>/provider.yaml", which breaks for namespace packages (apache, atlassian, common, cncf, dbt, microsoft) that nest an extra directory level, e.g. a change under apache/beam/... resolved to the nonexistent apache/provider.yaml instead of apache/beam/provider.yaml. Reuse the existing KNOWN_SECOND_LEVEL_PATHS list from common_prek_utils to resolve the correct nesting depth.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on PR #70224:
URL: https://github.com/apache/airflow/pull/70224#issuecomment-5046940042
> jira fix is up. let's see how CI goes (even though it won't really run without #70230)
Great, once this one is merged, I'll rebase into my PR and when mine is also green, will merge it. Thanks for the fix and approval @Lee-W !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on code in PR #70224:
URL: https://github.com/apache/airflow/pull/70224#discussion_r3629681298
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -46,7 +47,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
All paths are relative to the ``providers/`` directory, as supplied by
prek. The first path segment is the provider package name
- (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``).
+ (e.g. ``samba/src/airflow/...`` -> ``samba/provider.yaml``), except for
Review Comment:
This still crashes for IBM MQ, since providers/ibm/ is a namespace dir. I think we will need to add `ibm` in `common_prek_utils.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on code in PR #70224:
URL: https://github.com/apache/airflow/pull/70224#discussion_r3629690613
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -46,7 +47,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
All paths are relative to the ``providers/`` directory, as supplied by
prek. The first path segment is the provider package name
- (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``).
+ (e.g. ``samba/src/airflow/...`` -> ``samba/provider.yaml``), except for
Review Comment:
we should add ibm in `KNOWN_SECOND_LEVEL_PATHS = ["apache", "atlassian", "common", "cncf", "dbt", "ibm", "microsoft"]` within `common_prek_utils.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #70224:
URL: https://github.com/apache/airflow/pull/70224#issuecomment-5044880172
@dabla did you request changes by mistake?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on PR #70224:
URL: https://github.com/apache/airflow/pull/70224#issuecomment-5045207379
I created a [PR](https://github.com/apache/airflow/pull/70230) which fixes the selective checks issues, so that when a change is done to the static checks of the providers but no providers where changed, it will still run the static checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #70224:
URL: https://github.com/apache/airflow/pull/70224#discussion_r3634844763
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -59,7 +63,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
# e.g. samba/src/airflow/providers/samba/hooks/samba.py
parts = p.parts
if parts:
- result.add(f"{parts[0]}/provider.yaml")
+ if parts[0] in KNOWN_SECOND_LEVEL_PATHS and len(parts) > 1:
Review Comment:
will take a deeper look. since this one is blocking the CI, I'll merge it for now and work on that as a follow up. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #70224:
URL: https://github.com/apache/airflow/pull/70224#discussion_r3629548249
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -46,7 +47,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
All paths are relative to the ``providers/`` directory, as supplied by
prek. The first path segment is the provider package name
- (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``).
+ (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``), except for
Review Comment:
```suggestion
(e.g. ``samba/src/airflow/...`` -> ``samba/provider.yaml``), except for
```
Same small nit as @amoghrajesh suggested, looking good otherwise, thanks for the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #70224:
URL: https://github.com/apache/airflow/pull/70224#discussion_r3629597369
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -46,7 +47,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
All paths are relative to the ``providers/`` directory, as supplied by
prek. The first path segment is the provider package name
- (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``).
+ (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``), except for
+ namespace packages in ``KNOWN_SECOND_LEVEL_PATHS`` (e.g. ``apache``,
+ ``common``), which nest an extra level (e.g.
+ ``apache/beam/src/airflow/...`` → ``apache/beam/provider.yaml``).
Review Comment:
fixed. thanks!
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -46,7 +47,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
All paths are relative to the ``providers/`` directory, as supplied by
prek. The first path segment is the provider package name
- (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``).
+ (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``), except for
Review Comment:
fixed. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on PR #70224:
URL: https://github.com/apache/airflow/pull/70224#issuecomment-5044905373
> @dabla did you request changes by mistake?
Yes sorry for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #70224:
URL: https://github.com/apache/airflow/pull/70224#issuecomment-5046660698
jira fix is up. let's see how CI goes (even though it won't really run without https://github.com/apache/airflow/pull/70230)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #70224:
URL: https://github.com/apache/airflow/pull/70224#discussion_r3630746418
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -46,7 +47,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
All paths are relative to the ``providers/`` directory, as supplied by
prek. The first path segment is the provider package name
- (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``).
+ (e.g. ``samba/src/airflow/...`` -> ``samba/provider.yaml``), except for
Review Comment:
updated. with another jira issue fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #70224:
URL: https://github.com/apache/airflow/pull/70224#issuecomment-5053087622
### Backport failed to create: v3-3-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/29969764856'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/commit/c578603205b73aedb6fde559adc2a72fabdafb51"><img src='https://img.shields.io/badge/Commit-c578603-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker c578603 v3-3-test
```
This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #70224:
URL: https://github.com/apache/airflow/pull/70224#issuecomment-5044904099
yep, all fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #70224:
URL: https://github.com/apache/airflow/pull/70224#discussion_r3629319567
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -46,7 +47,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
All paths are relative to the ``providers/`` directory, as supplied by
prek. The first path segment is the provider package name
- (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``).
+ (e.g. ``samba/src/airflow/...`` → ``samba/provider.yaml``), except for
+ namespace packages in ``KNOWN_SECOND_LEVEL_PATHS`` (e.g. ``apache``,
+ ``common``), which nest an extra level (e.g.
+ ``apache/beam/src/airflow/...`` → ``apache/beam/provider.yaml``).
Review Comment:
```suggestion
``apache/beam/src/airflow/...`` -> ``apache/beam/provider.yaml``).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on code in PR #70224:
URL: https://github.com/apache/airflow/pull/70224#discussion_r3630890884
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -59,7 +63,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
# e.g. samba/src/airflow/providers/samba/hooks/samba.py
parts = p.parts
if parts:
- result.add(f"{parts[0]}/provider.yaml")
+ if parts[0] in KNOWN_SECOND_LEVEL_PATHS and len(parts) > 1:
Review Comment:
`common_prek_utils.get_provider_base_dir_from_path()` already has that shape, and using the actual provider layout as the source of truth would make this work for `ibm/mq` and any future nested provider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #70224:
URL: https://github.com/apache/airflow/pull/70224#issuecomment-5045623333
need another fix related to jira. will update it soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W merged PR #70224:
URL: https://github.com/apache/airflow/pull/70224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix conn-fields check crash for nested provider packages [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #70224:
URL: https://github.com/apache/airflow/pull/70224#discussion_r3636187259
##########
scripts/ci/prek/check_provider_yaml_files.py:
##########
@@ -59,7 +63,10 @@ def _resolve_provider_yaml_files(raw_files: list[str]) -> list[str]:
# e.g. samba/src/airflow/providers/samba/hooks/samba.py
parts = p.parts
if parts:
- result.add(f"{parts[0]}/provider.yaml")
+ if parts[0] in KNOWN_SECOND_LEVEL_PATHS and len(parts) > 1:
Review Comment:
https://github.com/apache/airflow/pull/70261
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org