You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "uranusjr (via GitHub)" <gi...@apache.org> on 2025/03/26 07:48:41 UTC
[PR] Move lineage mechanism to sdk [airflow]
uranusjr opened a new pull request, #48379:
URL: https://github.com/apache/airflow/pull/48379
The LineageBackend base class is moved to sdk.definitions. Runtime mechanism in airflow.lineage now lives in sdk.execution_time and also called in the task runner.
Lineage hook and collectors are left in Core. You can make a case they can be in the sdk, but it's not necessarily so I'm keeping the work simple for now.
I left a compatibility layer in Core, but will submit a PR later to remove it and rewrite related usages.
A part of #45424. Close #47771.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move lineage mechanism to sdk [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #48379:
URL: https://github.com/apache/airflow/pull/48379#discussion_r2013558498
##########
airflow-core/docs/administration-and-deployment/lineage.rst:
##########
@@ -71,11 +71,6 @@ for the downstream task.
In the example DAG task ``run_this`` (``task_id=run_me_first``) is a BashOperator that takes 3 inlets: ``CAT1``, ``CAT2``, ``CAT3``, that are
generated from a list. Note that ``data_interval_start`` is a templated field and will be rendered when the task is running.
-.. note:: Behind the scenes Airflow prepares the lineage metadata as part of the ``pre_execute`` method of a task. When the task
- has finished execution ``post_execute`` is called and lineage metadata is pushed into XCOM. Thus if you are creating
- your own operators that override this method make sure to decorate your method with ``prepare_lineage`` and ``apply_lineage``
- respectively.
Review Comment:
The hooks are now unconditionally called in the sdk, so the decorators are no longer needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move lineage mechanism to sdk [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr closed pull request #48379: Move lineage mechanism to sdk
URL: https://github.com/apache/airflow/pull/48379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move lineage mechanism to sdk [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #48379:
URL: https://github.com/apache/airflow/pull/48379#issuecomment-2755428847
Going to remove `airflow.lineage` entirely instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org