You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jason810496 (via GitHub)" <gi...@apache.org> on 2024/12/28 07:33:47 UTC

[PR] [WIP] Remove old lineage stuff [airflow]

jason810496 opened a new pull request, #45260:
URL: https://github.com/apache/airflow/pull/45260

   
   closes: #44983 
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2588811851

   Hi @jason810496, I just found the PR we discussed yesterday. https://github.com/apache/airflow/pull/44720 I think this one would be useful for this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #45260:
URL: https://github.com/apache/airflow/pull/45260#discussion_r1929034039


##########
providers/src/airflow/providers/openlineage/extractors/manager.py:
##########
@@ -36,8 +36,8 @@
 if TYPE_CHECKING:
     from openlineage.client.event_v2 import Dataset
 
-    from airflow.lineage.entities import Table
     from airflow.models import Operator
+    from airflow.providers.common.compat.lineage.entities import Table

Review Comment:
   Ah - 1.3.0 is the latest released one, everything seems good right now then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #45260:
URL: https://github.com/apache/airflow/pull/45260#discussion_r1926845365


##########
providers/src/airflow/providers/openlineage/extractors/manager.py:
##########
@@ -36,8 +36,8 @@
 if TYPE_CHECKING:
     from openlineage.client.event_v2 import Dataset
 
-    from airflow.lineage.entities import Table
     from airflow.models import Operator
+    from airflow.providers.common.compat.lineage.entities import Table

Review Comment:
   Hi @mobuchowski, is there any update for this comment ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2567337940

   > 2.9.3 or 2.3.9. If it's 2.9.3, then we'll need to fix it as providers will still support airflow 2 for some time
   
   Sorry for the typo, I mean 2.9.3 and 2.10.4 . Thanks for reply, if then I will fix the test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2589178774

   > One small question: Does this PR need to be backported to 2.10?
   
   No since you only moved the classes. The providers are always released from main, and it’s OK for those classes to be available in two places in 2.10.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2567323698

   > The CI failure is caused by a flaky K8s test (#45145) and a breaking change in the compatibility tests for providers 2.3.9 and 2.3.10.  
   > 
   > 
   > 
   > Should we fix the tests to pass the compatibility checks, or is it acceptable to ignore the compatibility tests since all tests for 3.0 have passed ?
   
   2.9.3 or 2.3.9. If it's 2.9.3, then we'll need to fix it as providers will still support airflow 2 for some time 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Remove old lineage stuff [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2564284850

   
   When attempting to remove the lineage logic from the core module, I noticed that it causes failures in tests related to **OpenLineage listener capturing hook-level lineage** (https://github.com/apache/airflow/pull/41482).  
   
   For example, removing `apply_lineage` and `prepare_lineage` from `BaseOperator`:  
   https://github.com/apache/airflow/blob/0efd9e6a2fa8bde0f6c14e88951b44badca063a2/airflow/models/baseoperator.py#L705-L739  
   
   Results in the following test failure:  
   ```
   FAILED providers/tests/openlineage/extractors/test_manager.py::test_extractor_manager_gets_data_from_pythonoperator - assert 0 == 1
    +  where 0 = len([])
    +    where [] = HookLineage(inputs=[], outputs=[]).outputs
   ```
   https://github.com/apache/airflow/blob/0efd9e6a2fa8bde0f6c14e88951b44badca063a2/providers/tests/openlineage/extractors/test_manager.py#L301  
   
   It seems OpenLineage is still coupled with the lineage module and might need to be moved to `compact.lineage` for now (or to the OpenLineage module in a future PR ).  
   
   After some experimentation, implementing an `on_load` callback in the `OpenLineageProviderPlugin` to monkey-patch the core module at runtime prevents the OpenLineage test failures, even with the lineage module removed from the core. 
   > Based on https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html#interface
   
   I’m not sure if this is a suitable long-term solution for maintaining OpenLineage compatibility while cleaning up the lineage module ?
   
   cc @Lee-W @uranusjr  
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Remove old lineage stuff [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2567315195

   The CI failure is caused by a flaky K8s test (#45145) and a breaking change in the compatibility tests for providers 2.3.9 and 2.3.10.  
   
   Should we fix the tests to pass the compatibility checks, or is it acceptable to ignore the compatibility tests since all tests for 3.0 have passed?  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #45260:
URL: https://github.com/apache/airflow/pull/45260#discussion_r1914164535


##########
providers/src/airflow/providers/common/compat/__init__.py:
##########
@@ -29,7 +29,7 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.3.0"
+__version__ = "1.3.1"

Review Comment:
   I'm not sure we need to bump it here. @eladkal, I thought you are going to bump this when releasing. Or is there anything we missed? Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2570319488

   Fixed: I only moved `airflow.lineage.entities` to `airflow.providers.common.compat.lineage.entities`.  
   The CI failure is unrelated, as no relevant frontend tests failed.  
   
   One small question: Does this PR need to be backported to 2.10 ?  
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2567341419

   We should not remove the entire lineage mechanism. The issue is only meant to remove the old lineage constructs in `airflow.lineage.entities`. The rest of the mechanism integrates modern Airflow assets with OpenLineage, and they should be kept.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W merged PR #45260:
URL: https://github.com/apache/airflow/pull/45260


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2612970122

   CI failure is due to https://github.com/apache/airflow/issues/45683.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #45260:
URL: https://github.com/apache/airflow/pull/45260#discussion_r1928442661


##########
providers/src/airflow/providers/openlineage/extractors/manager.py:
##########
@@ -36,8 +36,8 @@
 if TYPE_CHECKING:
     from openlineage.client.event_v2 import Dataset
 
-    from airflow.lineage.entities import Table
     from airflow.models import Operator
+    from airflow.providers.common.compat.lineage.entities import Table

Review Comment:
   @uranusjr those classes haven't changed in a while, the compat classes will work too.
   
   Only thing necessary IMO is to bump the required version of `apache-airflow-providers-common-compat` in OpenLineage provider's provider.yaml - those would work on the old versions,



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #45260:
URL: https://github.com/apache/airflow/pull/45260#discussion_r1914216963


##########
providers/src/airflow/providers/common/compat/__init__.py:
##########
@@ -29,7 +29,7 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.3.0"
+__version__ = "1.3.1"

Review Comment:
   I just noticed that it was bumped to `1.4.0` 5 days ago after rebasing to the latest main. Does that mean we don't need to bump the version in this PR anymore, right?  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2567710194

   @jason810496  I rebased it -> we found and issue with @jscheffl with the new caching scheme - fixed in https://github.com/apache/airflow/pull/45347 that would run "main" version of the tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 closed pull request #45260: Remove old lineage stuff
URL: https://github.com/apache/airflow/pull/45260


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #45260:
URL: https://github.com/apache/airflow/pull/45260#discussion_r1905227703


##########
providers/src/airflow/providers/openlineage/extractors/manager.py:
##########
@@ -36,8 +36,8 @@
 if TYPE_CHECKING:
     from openlineage.client.event_v2 import Dataset
 
-    from airflow.lineage.entities import Table
     from airflow.models import Operator
+    from airflow.providers.common.compat.lineage.entities import Table

Review Comment:
   Please bump the minimum version of providers common.compat requirement in `provider.yaml` in this case. This would raise exception for previous version of common.compat dependency.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #45260:
URL: https://github.com/apache/airflow/pull/45260#issuecomment-2567404322

   Just as TP said. Removing [airflow/lineage/hook.py](https://github.com/apache/airflow/pull/45260/files#diff-da82ad12f9efa71b6de517375641691e1b83a69f66eb79fa4887b197b15c4147) is wrong


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #45260:
URL: https://github.com/apache/airflow/pull/45260#discussion_r1928971892


##########
providers/src/airflow/providers/openlineage/extractors/manager.py:
##########
@@ -36,8 +36,8 @@
 if TYPE_CHECKING:
     from openlineage.client.event_v2 import Dataset
 
-    from airflow.lineage.entities import Table
     from airflow.models import Operator
+    from airflow.providers.common.compat.lineage.entities import Table

Review Comment:
   Hi @uranusjr, I found that https://github.com/apache/airflow/pull/45242 already bumped the the required version of `apache-airflow-providers-common-compat` in OpenLineage provider's provider.yaml. I think we don't need to bump here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #45260:
URL: https://github.com/apache/airflow/pull/45260#discussion_r1914344863


##########
providers/src/airflow/providers/openlineage/extractors/manager.py:
##########
@@ -36,8 +36,8 @@
 if TYPE_CHECKING:
     from openlineage.client.event_v2 import Dataset
 
-    from airflow.lineage.entities import Table
     from airflow.models import Operator
+    from airflow.providers.common.compat.lineage.entities import Table

Review Comment:
   @mobuchowski Should we support both import classes for a while so `airflow.lineage.entities` don’t break in old Airflow versions when the user upgrade the OL provider?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove old lineage stuff [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #45260:
URL: https://github.com/apache/airflow/pull/45260#discussion_r1914341031


##########
providers/src/airflow/providers/common/compat/__init__.py:
##########
@@ -29,7 +29,7 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.3.0"
+__version__ = "1.3.1"

Review Comment:
   Yes I think so



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org