You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/22 12:01:25 UTC
[GitHub] [airflow] pgzmnk commented on issue #17984: Add possibility to LineageBackend to be notified of task instance execution start and failure
pgzmnk commented on issue #17984:
URL: https://github.com/apache/airflow/issues/17984#issuecomment-924862638
A few thoughts:
- This is a great idea. It'd make OpenLineage an easier choice since it'd be easy to implement for multiple DAGs at once. It'd also be easier to maintain as Airflow evolves.
- Proposition 2 sounds more aligned with the existing state. Let's ensure it's backward compatible. Projects such as [DataHub subclass LineageBackend](https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub_provider/lineage/datahub.py#L38).
- Otherwise, for proposition 3 it might make sense to decorate `on_failure_callback` (and maybe `on_retry_callback` ?) to keep the method available.
On a further downstream note: What differences do you foresee for the lineage 'payload' for successful/started/failed tasks in terms of the OpenLineage schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org