You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/05 16:47:58 UTC
[GitHub] [airflow] JPonte opened a new issue #14106: Lineage Backend removed for no reason
JPonte opened a new issue #14106:
URL: https://github.com/apache/airflow/issues/14106
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
The possibility to add a lineage backend was removed in https://github.com/apache/airflow/pull/6564 but was never reintroduced. Now that this code is in 2.0, the lineage information is only in the xcoms and the only way to get it is through an experimental API that isn't very practical either.
**Use case / motivation**
A custom callback at the time lineage gets pushed is enough to send the lineage information to whatever lineage backend the user has.
**Are you willing to submit a PR?**
I'd be willing to make a PR recovering the LineageBackend and add changes if needed, unless there is a different plan for lineage from the maintainers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14106: Lineage Backend removed for no reason
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14106:
URL: https://github.com/apache/airflow/issues/14106#issuecomment-774186142
I think only @bolkedebruin so far was really pursuing the lineage, but I am sure if you would like to re-add it we can jointly review it and get it in 2.1, I think it is important to add it back in a form that will be good. Maybe also others would like to chime in?
If it is simple and you can prepare a proposed PR, I think the fastest way will be to prepare it and send a message to the devlist inviting people to discuss on it? I am not very into the lineage topic, but I know it is is important enough to deserve a bit of discussion and maybe others who already use Amundsen and other tools might chime in and iterate with you on the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JPonte commented on issue #14106: Lineage Backend removed for no reason
Posted by GitBox <gi...@apache.org>.
JPonte commented on issue #14106:
URL: https://github.com/apache/airflow/issues/14106#issuecomment-775339459
@kaxil Yes, I was pushing the lineage data to a custom lineage backend and it was surprising that it disappeared on 2.0. I can make a PR reintroducing it. I don't use Apache Atlas so I won't bring the AtlasBackend, only the base one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14106: Lineage Backend removed for no reason
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14106:
URL: https://github.com/apache/airflow/issues/14106#issuecomment-774150467
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14106: Lineage Backend removed for no reason
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14106:
URL: https://github.com/apache/airflow/issues/14106#issuecomment-775331466
Have you been using Lineage backend with 1.10.x ?
We want to make sure that it works as expected. Definitely an important topic but if we add/change something it should work :) @JPonte
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek closed issue #14106: Lineage Backend removed for no reason
Posted by GitBox <gi...@apache.org>.
turbaszek closed issue #14106:
URL: https://github.com/apache/airflow/issues/14106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org