[PR] Fix `RecursionError` in `common.compat` hook lineage `add_extra` polyfill [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil opened a new pull request, #68735:
URL: https://github.com/apache/airflow/pull/68735

   On Airflow < 3.2, `common.compat`'s `get_hook_lineage_collector()` polyfills `add_extra` by
   monkeypatching the `collected_assets` and `has_collected` **class** properties of the lineage
   collector. `_add_extra_polyfill` was not idempotent: the trigger gate (`_lacks_add_extra_method`)
   checks **instance-level** `_extra`, so a fresh collector instance of an already-patched class
   re-enters the polyfill and re-wraps the class property, capturing the previously installed wrapper
   as the "original". Each call stacks another layer onto the getter chain, so once enough collectors
   are created the next `collected_assets` / `has_collected` access exceeds the recursion limit and
   raises `RecursionError`.
   
   ## Where it shows up
   
   In production the global collector is a process singleton (the accessor is `@cache`-decorated) and
   the polyfill only runs on Airflow < 3.2, so it is applied exactly once -- no impact. The failure
   surfaces on the **Compat 3.1.x provider test matrix**, where a fresh `HookLineageCollector` is built
   per test: the class accumulates one wrapper per test until the lineage edge-case tests blow the
   recursion limit. This is a test-suite/robustness fix, not a production incident.
   
   ## Fix
   
   Make `_add_extra_polyfill` idempotent -- patch each collector class exactly once:
   
   - Skip re-patching when the class's own `__dict__` already carries a `_compat_extra_polyfilled`
     marker (checked on the exact class, not via inheritance, so a subclass that overrides these
     properties is still patched in its own right).
   - Set the marker only after all three patches (`collected_assets`, `has_collected`, `add_extra`)
     are installed, so a failure mid-patch leaves the class unmarked and retryable rather than
     half-patched.
   - Initialize the instance `_extra` / `_extra_counts` only when missing, so re-applying never clears
     a collector that already accumulated extras.
   
   Behavior is otherwise unchanged: every collector still gets `add_extra` and the extended
   `collected_assets` / `has_collected`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix `RecursionError` in `common.compat` hook lineage `add_extra` polyfill [airflow]

["kacpermuda (via GitHub)" <gi...@apache.org>]

kacpermuda commented on PR #68735:
URL: https://github.com/apache/airflow/pull/68735#issuecomment-4750322278

   Thanks Kaxil for fixing this, missed that for sure when writing tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix `RecursionError` in `common.compat` hook lineage `add_extra` polyfill [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil merged PR #68735:
URL: https://github.com/apache/airflow/pull/68735


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org