[jira] [Closed] (GSOC-294) Enhance lineage support in Beam

["Danny McCormick (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GSOC-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Danny McCormick closed GSOC-294.
--------------------------------
    Resolution: Won't Fix

> Enhance lineage support in Beam
> -------------------------------
>
>                 Key: GSOC-294
>                 URL: https://issues.apache.org/jira/browse/GSOC-294
>             Project: Comdev GSOC
>          Issue Type: Task
>            Reporter: Danny McCormick
>            Priority: Major
>              Labels: Beam, gsoc, gsoc2025, mentor
>
> Apache Beam provides a powerful way to define data processing pipelines. However, it is increasingly important for users to be able to track how their data is moving through Beam so that they can make informed choices on how they manage their data at the source or sink of their pipeline. To solve for this, we have recently introduced data lineage in Beam - [https://en.wikipedia.org/wiki/Data_lineage] - to this point support is still relatively limited though.
>  
> For this project, the focus would be on adding broader lineage support to Beam. This could include:
>  
>  * adding column level lineage to more transforms
>  * adding direct runner support for lineage graphs ([https://github.com/apache/beam/issues/33980)]
>  * Integrating Beam with Open Lineage ([https://github.com/apache/beam/issues/33981])



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: gsoc-unsubscribe@community.apache.org
For additional commands, e-mail: gsoc-help@community.apache.org