You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "rohitsinha54 (via GitHub)" <gi...@apache.org> on 2025/02/13 16:09:31 UTC
[I] [Feature Request]: Add support for Lineage Graphs in Beam's direct runner through job graph traversal [beam]
rohitsinha54 opened a new issue, #33980:
URL: https://github.com/apache/beam/issues/33980
### What would you like to happen?
Apache Beam from 2.63 onwards officially have capability to track [data lineage](https://en.wikipedia.org/wiki/Data_lineage) from various IOs.
This is done by the IOs emitting the lineage information as metrics individually.
Once the lineage information is collected from IOs in the pipeline they need to be linked to each other based on the IOs of the pipeline which are connected in the graph. Dataflow backend already does this by:
1. Traversing the job graph to identify all the connected paths (Note: Beam job graph can have more than one connected components)
2. Identify the sources in these paths (a middle node can be source too).
3. Forming pairs of sources and sinks based on the above connected node.
Once this lineage information is linked this can be served by:
1. Simple API which given a source/sink connected to each other show the data lineage.
or
2. Sending this information to some Lineage graph visualization tool.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support for Lineage Graphs in Beam's direct runner through job graph traversal [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33980:
URL: https://github.com/apache/beam/issues/33980#issuecomment-3183778531
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support for Lineage Graphs in Beam's direct runner through job graph traversal [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33980:
URL: https://github.com/apache/beam/issues/33980#issuecomment-2657087332
Label cannot be managed because it does not exist in the repo. Please check your spelling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support for Lineage Graphs in Beam's direct runner through job graph traversal [beam]
Posted by "Suidispy543 (via GitHub)" <gi...@apache.org>.
Suidispy543 commented on issue #33980:
URL: https://github.com/apache/beam/issues/33980#issuecomment-2657427724
Okay, looks interesting, good idea. I have understood the main aim, but can you tell me the final outcome and how's it increasing efficiency, so that it will bring clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support for Lineage Graphs in Beam's direct runner through job graph traversal [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33980:
URL: https://github.com/apache/beam/issues/33980#issuecomment-3069439967
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support for Lineage Graphs in Beam's direct runner through job graph traversal [beam]
Posted by "rohitsinha54 (via GitHub)" <gi...@apache.org>.
rohitsinha54 commented on issue #33980:
URL: https://github.com/apache/beam/issues/33980#issuecomment-2657085940
.set-labels gsoc, gsoc2025, lineage, infra, python, java, runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support for Lineage Graphs in Beam's direct runner through job graph traversal [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33980: [Feature Request]: Add support for Lineage Graphs in Beam's direct runner through job graph traversal
URL: https://github.com/apache/beam/issues/33980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org