You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Maciej Obuchowski <mo...@apache.org> on 2024/03/05 19:01:42 UTC

AIP-62 Getting Lineage from Hook Instrumentation

Hey,
I'd like to announce a new proposal. It proposes instrumenting Airflow
Hooks and Object Storage to collect dataset updates automatically, to allow
gathering lineage from PythonOperator and custom operators. Let me know
what you think about this.

The scope of this proposal was initially larger, and I left most of it as
potential future work - I would love to get some comments about it, even if
it won't be done soon.

Thanks,
Maciej

Re: AIP-62 Getting Lineage from Hook Instrumentation

Posted by Maciej Obuchowski <mo...@apache.org>.

And of course I've forgot to attach link, here it is:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-62+Getting+Lineage+from+Hook+Instrumentation

wt., 5 mar 2024 o 20:01 Maciej Obuchowski <mo...@apache.org>
napisał(a):

> Hey,
> I'd like to announce a new proposal. It proposes instrumenting Airflow
> Hooks and Object Storage to collect dataset updates automatically, to allow
> gathering lineage from PythonOperator and custom operators. Let me know
> what you think about this.
>
> The scope of this proposal was initially larger, and I left most of it as
> potential future work - I would love to get some comments about it, even if
> it won't be done soon.
>
> Thanks,
> Maciej
>
>