You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ramitkataria (via GitHub)" <gi...@apache.org> on 2026/08/07 19:31:34 UTC
[PR] Allow filtering hook-collected assets in OpenLineage emission policy [airflow]
ramitkataria opened a new pull request, #71312:
URL: https://github.com/apache/airflow/pull/71312
Hook-level lineage registers an output asset for every object a hook writes, so a task that writes many objects reports one asset per object. The existing hook_lineage control is all-or-nothing: suppressing the noisy assets also loses the ones worth tracking from the same hook, and the collector cannot tell the two apart once its assets are read as one set.
Filtering by asset URI or reporting hook class lets a Dag keep the lineage that matters while dropping the noise, and doing it where hook lineage is consumed covers every instrumented hook at once rather than one provider at a time.
Note this reduces what gets reported rather than what gets collected, so it does not recover assets already dropped by max_assets_per_collector.
closes: #63371
cc @SamWheating @kacpermuda @ashb @vincbeck @shnhdan @eladkal @mobuchowski
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow filtering hook-collected assets in OpenLineage emission policy [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5295525223
Hmm, I don't have a good answer now, will need to think about it a bit more. Going ooo for a week, but will definitely come back to it after I'm back.
I think we are ok with mixing hook as a scope, we can then disable something globally where we know it comes from hook, but also make it only per dag or task if needed, so all combinations make sense. `exclude_datasets` could also make sense:
- if we scope it to dag+ task - we drop all datasets from task
- if for task and dag+task+hook - we drop only hook level lineage
- if we scope for hook only - we drop them globally
Does that sound right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow filtering hook-collected assets in OpenLineage emission policy [airflow]
Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5344926277
Yes, that makes sense to me, and I think it covers everything this PR does: the chunked-upload case would be `{"scope": {"dag_id": ..., "task_id": ..., "hook": ".*S3Hook"}, "controls": {"exclude_datasets": [...]}}`, and `exclude_hook_lineage_hooks` becomes hook-only scope with `[".*"]`. I was wrong that hook can't be a scope: it can't join the up-front per-task resolution, but I could resolve dag/task/operator eagerly as now, carry any surviving hook-scoped rules into the collector read, and match each asset's `context` class there. Does that implementation shape sound reasonable, or did you have something else in mind?
I can reword the PR to this but I was wondering what your thoughts are on these before I do it:
1. Should hook scope be limited to per-asset controls, i.e. reject rules combining it with `emit` etc. during validation (like the existing operator + `emit_dag_events` check)? And would you want `hook_lineage: false` allowed at hook scope as a nicer spelling of "drop everything from this hook"?
2. What should `exclude_datasets` patterns match against? Hook assets have Airflow URIs pre-translation while operator lineage is OL datasets (namespace + name), so should I match the translated OL identity everywhere so one pattern behaves the same across sources?
3. Should it also filter manually annotated inlets/outlets, or only extractor and hook lineage?
4. For tier resolution, is replace-rather-than-merge (most specific matching rule wins per asset, so a task rule can narrow or clear a global list) the behaviour you'd expect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow filtering hook-collected assets in OpenLineage emission policy [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5226730365
@kacpermuda wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow excluding datasets from OpenLineage events by pattern and hook [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5831899445
> I was wondering: should a hook-scoped hook_lineage: false also suppress that hook's SQL per-query events? Right now only its datasets are dropped.
Good question, maybe we should have another key to be able to control these query events? I also have it on my todo list to allow control over whether or not external db calls are made by hooks / operators, I think this is also another control we'll be adding, so maybe this is the direction we can go to. The only downside I see now, is that we're starting to have a lot of combinations of scope + control that are valid, some that are invalid and it's getting hard to track. We might have to improve our docs on that and also improve logs to make sure we're very verbose on what's permitted and what's not.
I'll not be around for more than two weeks now to review this PR, but maybe @mobuchowski will find a moment 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow excluding datasets from OpenLineage events by pattern and hook [airflow]
Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-6295171620
Hey @kacpermuda, just a friendly reminder for once you're back from vacation :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow filtering hook-collected assets in OpenLineage emission policy [airflow]
Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5247210199
Thanks both! @kacpermuda @mobuchowski
Yes, I weighed a few alternatives, though I'm new to this code so please correct me where I've misread it:
- Producer-side gating in the hooks (the #63499 approach). ~32 call sites across 5 files, and can't go on core `BaseHook` without dropping airflow 2.11. Filtering at collector read covers every instrumented hook at once.
- Nesting: `{"hook_lineage": {"exclude_datasets": [...]}}`, so `hook_lineage` takes `bool | dict`. Probably the cleanest answer to control-vs-scope. I picked siblings thinking it kept the bool path clean, which was wrong: any non-bool control value widens `dict[str, bool]` in all 8 spots either way. Same cost, happy to switch.
- Hook-class only (Kacper's earlier hook-scope idea). Doesn't seem to solve the issue case: the chunk parts and the objects worth keeping come from the same `S3Hook`, so it drops both. Hence asset patterns.
- `match_mode` for these lists. Built it and then reverted it: two rules with different lists leave no single mode to record. So always regex, `match_mode` stays scope-only. This might be a good data point for Kacper's #5.
`hook` as a scope key: doesn't fit as far as I can tell. `_classify_task_rules` tiers on operator/dag_id/task_id only, one task event can span several hooks, and resolution returns one `EmissionPolicy`. Hook is only known per-asset, at collector read. Happy to be wrong here if you see a way. Note `{"dag_id": x, "task_id": y}` plus `exclude_hook_lineage_hooks` already gives the dag/task/hook combination you described.
`exclude_datasets` covering operator lineage: I agree and see two options:
- (a) `exclude_datasets` plus one hook dimension, keeps the chunked-upload case working
- (b) `exclude_datasets` only, hook granularity via `hook_lineage: false` at task scope
Agree on skipping preprocessors and extra match modes.
Also:
- these lists replace across tiers rather than merge, so a task rule can narrow or clear (`[]`) a global one. Merging would make a global rule unescapable locally. Worth confirming that's the behaviour you'd expect.
- filtering runs after `max_assets_per_collector`, so it cuts what's reported, not what's collected. Anything past the cap was never collected and no pattern brings it back. Changing that would be producer-side work, so a bigger change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow excluding datasets from OpenLineage events by pattern and hook [airflow]
Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5880200910
Thanks @kacpermuda, agreed: I'll leave the query events alone here so they can get their own control alongside the external-calls one. On the growing number of combinations, I've added a table of which controls each scope accepts to `emission_policy.rst`. Invalid combinations are already skipped with a warning listing what's allowed. Enjoy the time off!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow excluding datasets from OpenLineage events by pattern and hook [airflow]
Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5805141532
Thanks @kacpermuda! Reworked along those lines:
- `hook` is a scope key, combinable with any other scope. Hook-scoped rules only accept `exclude_datasets` or `hook_lineage` (`false` drops everything from that hook).
- `exclude_datasets` matches `<namespace>/<name>` across all lineage sources.
- The most specific rule's list replaces broader ones, with an INFO log. `locked` still only blocks authoring overrides.
I was wondering: should a hook-scoped `hook_lineage: false` also suppress that hook's SQL per-query events? Right now only its datasets are dropped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow filtering hook-collected assets in OpenLineage emission policy [airflow]
Posted by "ramitkataria (via GitHub)" <gi...@apache.org>.
ramitkataria commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5736193402
Bump @kacpermuda
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow filtering hook-collected assets in OpenLineage emission policy [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5763245255
Thanks for the ping, still not convinced I have all the answers, but let's try to move this forward:
> Should hook scope be limited to per-asset controls, i.e. reject rules combining it with emit etc. during validation (like the existing operator + emit_dag_events check)?
Yes, I think we can remove such combinations now, we can always allow them later. Let's stick to what makes the most sense now.
> And would you want hook_lineage: false allowed at hook scope as a nicer spelling of "drop everything from this hook"?
I think hook_lineage and emit would mean the same? We can probably allow it, but I think it's ok to also clearly mark that `emit` is the only valid way and reject the other one. Up to you.
> What should exclude_datasets patterns match against? Hook assets have Airflow URIs pre-translation while operator lineage is OL datasets (namespace + name), so should I match the translated OL identity everywhere so one pattern behaves the same across sources?
I think we can do namespace + name here? Some URIs do not get translated anyway, so this way we'll be dropping only those that survived the translation.
> Should it also filter manually annotated inlets/outlets, or only extractor and hook lineage?
Hmm, good question. I think if it's easy we can apply it to all to be consistent.
> For tier resolution, is replace-rather-than-merge (most specific matching rule wins per asset, so a task rule can narrow or clear a global list) the behaviour you'd expect?
I think it's fine, as long as we honor the `freeze` of the global rule and do not replace then. We need to clearly mark this behavior in the docs, and maybe also log some info when this happens, just for clearer debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow filtering hook-collected assets in OpenLineage emission policy [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5239160815
I like the idea of widening the support of emission policy, that's for sure. Few thoughts:
1. Not sure what should be a scope and what can be a control from this PR. Hook class may be the new scope (where we already have exact and regex modes), not control. If mixed with dag_id, or dag_id and task_id, it can provide the granularity that is needed (disable hook level lineage inside dag or just for specific task but just from specific hook).
> not narrowed down to hook-level lineage, but operator lineage too
3. Not sure if there is a need to drop only some datasets (regex based) only when they come from hook level lineage, and not if they come from operator methods. We do have some [filters](https://openlineage.io/docs/client/python/configuration#filters) on python client level, but they are looking purely at job name now and are dropping events. If we wanted to, we can extend them to drop datasets that do not match some conditions - that would be a wide dataset filter. If we need to differentiate between hll and operator datasets, we might need to do it in controls, but then again, we might just use hook as scope and then exclude_datasets as control, that can be used both on operator and hook level? Thinking out loud here, not sure how to make it less confusing for people.
> going even forward with it, maybe it could be just generalizable "preprocessor" for this data? This could take a form of generalized filters (python function) that are being passed context and lineage and return lineage output.
4. I like the idea of generic preprocessors, but I know that people rarely want to write them and we already have a transform transport that can be used to achieve whatever custom logic they have, so I don't think we need to provide more ways of customizing it. If you need more customization than the emission_policy provides, use transform transport and write your own transformer - would be my advice.
> more options than just regex
5. Not sure if this is needed now, for scope we have exact and regex, and for more custom stuff you can always use transform transport I think.
@ramitkataria , have you considered some alternatives when it comes to what is control and what is scope here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow filtering hook-collected assets in OpenLineage emission policy [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5226729729
In general I like this approach, but maybe this should be configurable more broadly:
- not narrowed down to hook-level lineage, but operator lineage too
- more options than just regex
- going even forward with it, maybe it could be just generalizable "preprocessor" for this data?
This could take a form of generalized filters (python function) that are being passed context and lineage and return lineage output.
We could have ones that are build in, and one of those could be "filter hook level lineage using regex"; but we could pass more using Airflow plugins if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow excluding datasets from OpenLineage events by pattern and hook [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck merged PR #71312:
URL: https://github.com/apache/airflow/pull/71312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org