[GitHub] [airflow] julienledem commented on issue #17984: Add possibility to LineageBackend to be notified of task instance execution start and failure

[GitBox <gi...@apache.org>]

julienledem commented on issue #17984:
URL: https://github.com/apache/airflow/issues/17984#issuecomment-926986837


   Just adding my two cents to what @mobuchowski is saying as one of the OpenLineage contributors.
   
   I'll start by describing how OpenLineage defines data lineage as this is where opinions might diverge. (my goal being to help make a decision on the discussion above)
   
   OpenLineage tracks lineage in real time as a task runs. 
   - An event is sent when the task starts
   - Another event is sent when the task completes with a status (success, failure, ...)
   
   This gives information in real-time that the data is being modified (or not). In terms of lineage it is important to track data transformations that should have happened as much as transformation that did happen.
   
   Tracking a start event let us know the difference between "it is in progress" and "it's not even started". And tracking a failed events tells us "it's not happening".
   
   It is also important to track failed tasks because, if it is a best practice for tasks to be atomic, a lot of tasks are not.
    - A failed task might have mutated data and it is important to know
    - Failed jobs still have read data
   
   This lineage is really at the task instance level which allows tracking how lineage changes over time and how code changes (for example) impact it.
   That's why the metadata related to that particular task instance is collected:
    - what was the schema of the inputs and outputs at the time it ran
    - what was the version of the code
    - etc
   
   I agree that this is a wider definition than just tracking the current dependencies between tables. It is more granular and it is a superset of what the current Lineage Backend does.
   
   Adding a new contract would work but I would think that over time it would subsume the current Lineage Backend as it would provide more functionality for a goal that is very similar and possibly that would be confusing.
   
   My opinion is that expanding that contract makes sense but we (OpenLineage) are happy to adapt to what the community thinks is best.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org