You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/03/24 00:40:35 UTC

[PR] Add provider class registration checks to provider.yaml validation [airflow]

potiuk opened a new pull request, #64127:
URL: https://github.com/apache/airflow/pull/64127

   Add two new checks to `run_provider_yaml_files_check.py` that verify provider classes
   are properly registered in `provider.yaml`, closing the gap where existing checks only
   validated yaml→code (classes listed in YAML exist) but not code→yaml (classes in code
   are listed).
   
   ## New checks
   
   **1. Hook conn_type registration check**
   - Imports all hook modules, finds `BaseHook` subclasses that define `conn_type` directly
   - Verifies the connection type is registered in `connection-types`
   - Async hooks sharing `conn_type` with an already-registered sync hook are allowed
   - Skips abstract classes
   
   **2. Comprehensive class registration check**
   - Walks all provider source files (not just standard directories)
   - Finds concrete subclasses of `BaseHook`, `BaseOperator`, `BaseSensorOperator`,
     `BaseTrigger`, `BaseExecutor`, `BaseNotifier`
   - Verifies modules (or class paths for executors/notifications) are registered
   - Handles class aliases (e.g. `send_*_notification = FooNotifier`) via `id()` dedup
   - Handles `__init__.py` re-exports for executors
   - Skips `tests/`, `example_dags/`, `decorators/` directories
   - Prints every discovered class with OK/MISSING status for transparency
   
   **Verified**: 1,571 classes across all providers (936 operators, 307 hooks,
   155 triggers, 152 sensors, 13 notifiers, 8 executors).
   
   Also registers the missing `LocalKubernetesExecutor` in the `cncf.kubernetes`
   provider, found by the new check.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Claude Opus 4.6)
   
   Generated-by: Claude Code (Claude Opus 4.6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add provider class registration checks to provider.yaml validation [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #64127:
URL: https://github.com/apache/airflow/pull/64127#discussion_r3007082173


##########
scripts/in_container/run_provider_yaml_files_check.py:
##########
@@ -473,6 +473,233 @@ def check_hook_class_name_entries_in_connection_types(yaml_files: dict[str, dict
     return num_connection_types, num_errors
 
 
+@run_check("Checking that hook classes defining conn_type are registered in connection-types")
+def check_hook_classes_with_conn_type_are_registered(yaml_files: dict[str, dict]) -> tuple[int, int]:
+    """Find Hook subclasses that define conn_type but are not listed in connection-types."""
+    from airflow.sdk.bases.hook import BaseHook
+
+    num_checks = 0
+    num_errors = 0
+    for yaml_file_path, provider_data in yaml_files.items():
+        connection_types = provider_data.get("connection-types", [])
+        registered_hook_classes = {ct["hook-class-name"] for ct in connection_types}
+        # Collect conn_type values that are already covered by a registered hook class
+        registered_conn_types = {ct["connection-type"] for ct in connection_types}
+        hook_modules = [
+            mod for entry in provider_data.get("hooks", []) for mod in entry.get("python-modules", [])
+        ]
+        for module_name in hook_modules:
+            try:
+                with warnings.catch_warnings(record=True):
+                    module = importlib.import_module(module_name)
+            except (ImportError, AirflowOptionalProviderFeatureException):
+                continue  # Import errors are caught by other checks
+
+            for attr_name in dir(module):
+                if attr_name.startswith("_"):
+                    continue
+                obj = getattr(module, attr_name, None)
+                if not (inspect.isclass(obj) and issubclass(obj, BaseHook) and obj is not BaseHook):
+                    continue
+                # Only check classes defined in this module, not re-exported ones
+                if obj.__module__ != module_name:
+                    continue
+                # Skip abstract classes — they are base classes, not concrete hooks
+                if inspect.isabstract(obj):
+                    continue
+                num_checks += 1
+                # Check conn_type defined directly on the class, not inherited
+                if "conn_type" not in obj.__dict__:
+                    continue
+                conn_type = obj.__dict__["conn_type"]
+                if not conn_type:
+                    continue
+                full_class_name = f"{module_name}.{attr_name}"
+                if full_class_name in registered_hook_classes:
+                    continue
+                # If another hook class already registered the same conn_type, this is fine
+                # (e.g. async variants sharing conn_type with sync hooks)
+                if conn_type in registered_conn_types:
+                    continue
+                errors.append(
+                    f"Hook class `{full_class_name}` defines conn_type='{conn_type}' "
+                    f"but no hook class is registered for this connection type "
+                    f"in 'connection-types' in {yaml_file_path}.\n"
+                    f"[yellow]How to fix it[/]: Add an entry with "
+                    f"hook-class-name: {full_class_name} to the connection-types "
+                    f"section of {yaml_file_path}."
+                )
+                num_errors += 1
+    return num_checks, num_errors
+
+
+@run_check(
+    "Checking that all provider Hook/Operator/Sensor/Trigger/Executor/Notifier"
+    " classes are registered in provider.yaml"
+)
+def check_all_provider_classes_are_registered(yaml_files: dict[str, dict]) -> tuple[int, int]:
+    """
+    Walk all provider source files, find Hook/Operator/Sensor/Trigger/Executor/Notifier
+    subclasses, and verify they are registered in the appropriate provider.yaml section.
+
+    This catches classes placed in non-standard directories or modules that were missed
+    when updating provider.yaml.
+    """
+    from airflow.executors.base_executor import BaseExecutor
+    from airflow.models.baseoperator import BaseOperator
+    from airflow.sdk.bases.hook import BaseHook
+    from airflow.sdk.bases.notifier import BaseNotifier
+    from airflow.sensors.base import BaseSensorOperator
+    from airflow.triggers.base import BaseTrigger
+

Review Comment:
   Oh. Absolutely ... good point. Added :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add provider class registration checks to provider.yaml validation [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #64127:
URL: https://github.com/apache/airflow/pull/64127#issuecomment-4151639052

   > Nice improvement to guard the missing provider registration, thanks!
   > 
   > Features like assets or the CLI are not validated here as well, so perhaps we can handle them in a follow-up?
   
   Feel free :). Claude is your friend - all ther was entirely Claude generated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add provider class registration checks to provider.yaml validation [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #64127:
URL: https://github.com/apache/airflow/pull/64127#issuecomment-4158959154

   ### Backport failed to create: v3-1-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/23774128204'> Run details </a>
   
   Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
   the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
   
   In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
   
   <table>
       <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>❌</td>
           <td>v3-1-test</td>
           <td><a href="https://github.com/apache/airflow/commit/3dc0d9e7a772ba14e45aa6b857c435780314ed85"><img src='https://img.shields.io/badge/Commit-3dc0d9e-red' alt='Commit Link'></a></td>
       </tr>
   </table>
   
   You can attempt to backport this manually by running:
   
   ```bash
   cherry_picker 3dc0d9e v3-1-test
   ```
   
   This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
   the files that need manual conflict resolution.
   
   After you have resolved the conflicts, you can continue the backport process by running:
   
   ```bash
   cherry_picker --continue
   ```
   
   If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#how-to-backport-pr-with-cherry-picker-cli).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add provider class registration checks to provider.yaml validation [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #64127:
URL: https://github.com/apache/airflow/pull/64127


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add provider class registration checks to provider.yaml validation [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #64127:
URL: https://github.com/apache/airflow/pull/64127#discussion_r3004824799


##########
scripts/in_container/run_provider_yaml_files_check.py:
##########
@@ -473,6 +473,233 @@ def check_hook_class_name_entries_in_connection_types(yaml_files: dict[str, dict
     return num_connection_types, num_errors
 
 
+@run_check("Checking that hook classes defining conn_type are registered in connection-types")
+def check_hook_classes_with_conn_type_are_registered(yaml_files: dict[str, dict]) -> tuple[int, int]:
+    """Find Hook subclasses that define conn_type but are not listed in connection-types."""
+    from airflow.sdk.bases.hook import BaseHook
+
+    num_checks = 0
+    num_errors = 0
+    for yaml_file_path, provider_data in yaml_files.items():
+        connection_types = provider_data.get("connection-types", [])
+        registered_hook_classes = {ct["hook-class-name"] for ct in connection_types}
+        # Collect conn_type values that are already covered by a registered hook class
+        registered_conn_types = {ct["connection-type"] for ct in connection_types}
+        hook_modules = [
+            mod for entry in provider_data.get("hooks", []) for mod in entry.get("python-modules", [])
+        ]
+        for module_name in hook_modules:
+            try:
+                with warnings.catch_warnings(record=True):
+                    module = importlib.import_module(module_name)
+            except (ImportError, AirflowOptionalProviderFeatureException):
+                continue  # Import errors are caught by other checks
+
+            for attr_name in dir(module):
+                if attr_name.startswith("_"):
+                    continue
+                obj = getattr(module, attr_name, None)
+                if not (inspect.isclass(obj) and issubclass(obj, BaseHook) and obj is not BaseHook):
+                    continue
+                # Only check classes defined in this module, not re-exported ones
+                if obj.__module__ != module_name:
+                    continue
+                # Skip abstract classes — they are base classes, not concrete hooks
+                if inspect.isabstract(obj):
+                    continue
+                num_checks += 1
+                # Check conn_type defined directly on the class, not inherited
+                if "conn_type" not in obj.__dict__:
+                    continue
+                conn_type = obj.__dict__["conn_type"]
+                if not conn_type:
+                    continue
+                full_class_name = f"{module_name}.{attr_name}"
+                if full_class_name in registered_hook_classes:
+                    continue
+                # If another hook class already registered the same conn_type, this is fine
+                # (e.g. async variants sharing conn_type with sync hooks)
+                if conn_type in registered_conn_types:
+                    continue
+                errors.append(
+                    f"Hook class `{full_class_name}` defines conn_type='{conn_type}' "
+                    f"but no hook class is registered for this connection type "
+                    f"in 'connection-types' in {yaml_file_path}.\n"
+                    f"[yellow]How to fix it[/]: Add an entry with "
+                    f"hook-class-name: {full_class_name} to the connection-types "
+                    f"section of {yaml_file_path}."
+                )
+                num_errors += 1
+    return num_checks, num_errors
+
+
+@run_check(
+    "Checking that all provider Hook/Operator/Sensor/Trigger/Executor/Notifier"
+    " classes are registered in provider.yaml"
+)
+def check_all_provider_classes_are_registered(yaml_files: dict[str, dict]) -> tuple[int, int]:
+    """
+    Walk all provider source files, find Hook/Operator/Sensor/Trigger/Executor/Notifier
+    subclasses, and verify they are registered in the appropriate provider.yaml section.
+
+    This catches classes placed in non-standard directories or modules that were missed
+    when updating provider.yaml.
+    """
+    from airflow.executors.base_executor import BaseExecutor
+    from airflow.models.baseoperator import BaseOperator
+    from airflow.sdk.bases.hook import BaseHook
+    from airflow.sdk.bases.notifier import BaseNotifier
+    from airflow.sensors.base import BaseSensorOperator
+    from airflow.triggers.base import BaseTrigger
+

Review Comment:
   Do we need to add validation of provider subclasses such as ‎`BaseAuthManager`, ‎`BaseDagBundle`, ‎`BaseSecretsBackend`, or ‎`FileTaskHandler`?
   
   These are the features I found that are not covered here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org