You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/01/16 18:44:50 UTC

[I] Remove all dev dependencies before provider yaml check [airflow]

potiuk opened a new issue, #60662:
URL: https://github.com/apache/airflow/issues/60662

   Currently `scripts/in_container/run_provider_yaml_files_check.py` walks through all provider "source" files and does some checks (like whether classes mentioned in provider.yaml files as hooks are actually hooks.
   
   It is run inside breeze image as result of this pre-comit hook
   
   ```
   - id: check-provider-yaml-valid
     name: Validate provider.yaml files
     entry: ../scripts/ci/prek/check_provider_yaml_files.py
     language: python
     files: ^.*/provider\.yaml$
     exclude: ^.*/.venv/.*$
     require_serial: true
   ```
   
   We should make sure that when the script is run inside breeze CI docker files we run `uv sync --no-dev`. Ideally we should do it separately for each provider before re-importing each provider, but that might take a long time and require spawning new interpreters for each provider.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Remove all dev dependencies before provider yaml check [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk commented on issue #60662:
URL: https://github.com/apache/airflow/issues/60662#issuecomment-3761373988

   Feel free - just make sure that to understand it- there are number of moving parts there - such as calling CI image from whithin a `prek hook` and handlng single files versus "all files" there
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Remove all dev dependencies before provider yaml check [airflow]

Posted by "GauravFrontend (via GitHub)" <gi...@apache.org>.

GauravFrontend commented on issue #60662:
URL: https://github.com/apache/airflow/issues/60662#issuecomment-3761348332

   I will start working on a PR for this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Remove all dev dependencies before provider yaml check [airflow]

Posted by "Bhuvan-08 (via GitHub)" <gi...@apache.org>.

Bhuvan-08 commented on issue #60662:
URL: https://github.com/apache/airflow/issues/60662#issuecomment-4025532528

   Hi! I'd like to take a look at this issue.
   I noticed that there was a previous attempt in PR #60728 that introduces `uv sync --no-dev` before running the provider YAML checks.
   I'll review the existing implementation and CI scripts to see if there are remaining improvements needed or if the approach can be simplified.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Remove all dev dependencies before provider yaml check [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk closed issue #60662: Remove all dev dependencies before provider yaml check
URL: https://github.com/apache/airflow/issues/60662


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Remove all dev dependencies before provider yaml check [airflow]

Posted by "Fury0508 (via GitHub)" <gi...@apache.org>.

Fury0508 commented on issue #60662:
URL: https://github.com/apache/airflow/issues/60662#issuecomment-3764277199

   Hi! I've implemented a fix for this in PR https://github.com/apache/airflow/pull/60728.
   
   The Problem
   
   The script imports jsonpath_ng at the top level, but it's a dev dependency. Simply adding uv sync --no-dev would cause an ImportError when the script tries to use it.
   
   My Solution
   
   Added a sync_dependencies_without_dev() function that runs:
      uv sync --no-dev --all-packages --no-python-downloads --no-managed-python
   This strips dev dependencies to create an environment closer to production, helping detect unhandled optional cross-provider dependencies.
   
   Converted the jsonpath_ng import to a lazy import inside check_providers_are_mentioned_in_issue_template() to avoid the ImportError.
   This approach ensures the validation runs without dev dependencies while still allowing the script to function when it needs jsonpath_ng for that specific check.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Remove all dev dependencies before provider yaml check [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.

shahar1 commented on issue #60662:
URL: https://github.com/apache/airflow/issues/60662#issuecomment-3877071783

   Somehow I missed this PR and created another to solve the same issue (#61713), which fails the pre-commit even before running the tests.
   I think that it would be nice to have both to ensure that we're covered E2E.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Remove all dev dependencies before provider yaml check [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk commented on issue #60662:
URL: https://github.com/apache/airflow/issues/60662#issuecomment-3986413604

   @GauravFrontend We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
   
   This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
   
   We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org