You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/04/14 12:00:00 UTC
[PR] Add cross dependencies between airflow-core and task-sdk [airflow]
potiuk opened a new pull request, #49221:
URL: https://github.com/apache/airflow/pull/49221
We have not managed fully unentangle dependencies between airlfow-core and task-sdk and for now they are cross-dependent - both airflow-core uses task-sdk and task-sdk uses airflow core. We are aiming to fix this for Airflow 3.1.
We are planning to relax dependencies between airflow-core and task-sdk in Airflow 3.1, but for now we are closely tying the versions because we are directly using imports between those packages and we have no backwards-compatibility checks implemented to guard if there are incompatible changes introduced.
The mechanism to prepare RC packages has been updated to make sure that rcN version of apache-airflow, apache-airflow-core, apache-airflow-task-sdk always refer to the same rc versions of each other - which means that all three packages should be released always together until we loosen relation between the packages.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042202874
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
Just to make a better decision, here is some more data.
The problem with open-ended here is that we have no mechanisms currently to take care about backwards compatibiliy - we simply do not know if the current task-sdk wil be compatible with future `airflow-core` or the other way round. Since the separation is not finished and the mechanism for making sure backwards compatibility is not here, this is very, very, prone to errors. We saw that multiple times even with simple things like common.sql and other providers, and I think at least for 3.0 and the way how task-sdk uses airflow-core and the other way round, they are still very tightly coupled to each other.
IF the only interface was api and cadwyn that would be fine - but now, effectively we have those imports as "internal-apis" between task-sdk and airflow-core:
* airflow.configuration.*
* airflow.utils.*
* airflow.typing_compat
* airflow.exceptions.*
* airflow.serialization.*
* airflow.task.priority_strategy
* airflow.models.xom_args
* airlfow.models.taskinstancekeys
* airflow.models.asset
* airflow.timteables.
* airflow.decorators
* airflow.cli
* airflow.ti_deps
* airflow.sensors.base
* airflow.providers.standard.* -> which is likely something we should remove
* airflow.settings
* airfow.secrets
* airflow.dag_processing
* airflow.listeners
* airflow.models.dagbag
* airflow.io.*
* airflow.logging_config
There are also many impots from airflow-core -> task-sdk.:
* airflow.sdk.definitions.*
* airflow.sdk.types.*
* airflow.sdk.execution_time.*
* airflow.sdk.DAG
* airflow.sdk.dag
* airflow.sdk.task
* airflow.sdk.BaseOperator
* airflow.sdk.Variable
* airlfow.sdk.Connection
* airflow.sdk.BaseOperatorLink
* airflow.sdk.TaskGroup
* airflow.sdk.ObjectStoragePath
* airflow.sdk.Context
* airflow.sdk.log
* airflow.sdk.io.*
* airflow.sdk.api.*
Those are quick scans and I likely missed something. And I believe except Cadwyn being API versioning for API communication, we have no mechanisms currently to maintain the compatibility.
As i understand, airflow-core-> task-sdk relation is more "intended" to be "public API and we intend to keep TaskSDK compliant - also because of providers that will have to be able to upgrade task-sdk independently), still we - I think - do not have any compatibility mechanisms and any changes in task-sdk might potentialy break something in airflow-core.
The other way round though is not really intended ->task-sdk long term is not supposed to import any of airflow-core classes, so here the brittleness is even more problematic - it's quite likely even task-sdk 1.0.1 will break Airflow 3.0.0 and that Airflow 3.0.1 will break task-sdk 1.0.0. It will be quite difficult to make sure those two work together without back-compatibilty checks.
I'd be personally for open-ending the dependency only after we get rid of the task-sdk-> airflow imports and have some back-compatibility checks or prevention built-in (which shoudl be done together in 3.1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042214991
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
And of course we need to discuss how to decouple those - there are many thing that are "common" between task-sdk and "airflow-core" (as we can see - the reason why some of those are used is because we have a common ways to handle things like logging, serialization, settings, secrets etc. so we will need to figure out what to do there. But that is quite secondary concern after Airflow 3.0 is out IMHO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042324403
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
I think we can also figure out some back-compatibility testing similar to provider's back-compatibility testing that might give us some test-harness for accidental changes. We can add it after the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042202874
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
Just to make a better decision, here is some more data.
The problem with open-ended here is that we have no mechanisms currently to take care about backwards compatibiliy - we simply do not know if the current task-sdk wil be compatible with future `airflow-core` or the other way round. Since the separation is not finished and the mechanism for making sure backwards compatibility is not here, this is very, very, prone to errors. We saw that multiple times even with simple things like common.sql and other providers, and I think at least for 3.0 and the way how task-sdk uses airflow-core and the other way round, they are still very tightly coupled to each other.
IF the only interface was api and cadwyn that would be fine - but now, effectively we have those imports as "internal-apis" between task-sdk and airflow-core:
* airflow.configuration.*
* airflow.utils.*
* airflow.typing_compat
* airflow.exceptions.*
* airflow.serialization.*
* airflow.task.priority_strategy
* airflow.models.xom_args
* airlfow.models.taskinstancekeys
* airflow.modes.asset
* airflow.timteables.
* airflow.decorators
* airflow.cli
* airflow.ti_deps
* airflow.sensors.base
* airflow.providers.standard.* -> which is likely something we should remove
* airflow.settings
* airfow.secrets
* airflow.dag_processing
* airflow.listeners
* airflow.models.dagbag
* airflow.io.*
* airflow.logging_config
There are also many impots from airflow-core -> task-sdk.:
* airflow.sdk.definitions.*
* airflow.sdk.types.*
* airflow.sdk.execution_time.*
* airflow.sdk.DAG
* airflow.sdk.dag
* airflow.sdk.task
* airflow.sdk.BaseOperator
* airflow.sdk.Variable
* airlfow.sdk.Connection
* airflow.sdk.BaseOperatorLink
* airflow.sdk.TaskGroup
* airflow.sdk.ObjectStoragePath
* airflow.sdk.Context
* airflow.sdk.log
* sitgloe.sdk.io.*
* airflow.sdk.api.*
Those are quick scans and I likely missed something. And I believe except Cadwyn being API versioning for API communication, we have no mechanisms currently to maintain the compatibility.
As i understand, airflow-core-> task-sdk relation is more "intended" to be "public API and we intend to keep TaskSDK compliant - also because of providers that will have to be able to upgrade task-sdk independently), still we - I think - do not have any compatibility mechanisms and any changes in task-sdk might potentialy break something in airflow-core.
The other way round though is not really intended ->task-sdk long term is not supposed to import any of airflow-core classes, so here the brittleness is even more problematic - it's quite likely even task-sdk 1.0.1 will break Airflow 3.0.0 and that Airflow 3.0.1 will break task-sdk 1.0.0. It will be quite difficult to make sure those two work together without back-compatibilty checks.
I'd be personally for open-ending the dependency only after we get rid of the task-sdk-> airflow imports and have some back-compatibility checks or prevention built-in (which shoudl be done together in 3.1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042325895
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
Updated,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042246145
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
> Yeah, `apache-airflow-task-sdk>=1.0.0, <1.1` is fine too.
How are we going to maintain compatibility ^ see above @kaxil ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042250792
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
Not planning to break anything in SDK 1.0.x patch version. Just bug fixing, hence `<1.1` -- i.e. allow all patch version of 1.0.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042334563
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
Yes thank you for pointing out the code that we have spent the past 5 months living in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042333316
##########
task-sdk/pyproject.toml:
##########
@@ -46,6 +46,7 @@ classifiers = [
]
dependencies = [
+ "apache-airflow-core<=3.1.0,>=3.0.0",
Review Comment:
@kaxil @ashb -> do you also feel this one should be as well? I guess the "not plnning breaking changes" applies here as well - just wanted to make sure :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042336943
##########
task-sdk/pyproject.toml:
##########
@@ -46,6 +46,7 @@ classifiers = [
]
dependencies = [
+ "apache-airflow-core<=3.1.0,>=3.0.0",
Review Comment:
Yup, this one is good too.
FInish this split is the main thing Kaxil and I will be working on for 3.1/Task-SDK 1.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #49221:
URL: https://github.com/apache/airflow/pull/49221#issuecomment-2801589305
I had to also make sure to build task-sdk and include it as "chicken-egg" case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042333316
##########
task-sdk/pyproject.toml:
##########
@@ -46,6 +46,7 @@ classifiers = [
]
dependencies = [
+ "apache-airflow-core<=3.1.0,>=3.0.0",
Review Comment:
@kaxil @ashb -> do you also feel this one should be as well? I guess the "not planning breaking changes" applies here as well - just wanted to make sure :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042202874
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
Just to make a better decision, here is some more data.
The problem with open-ended here is that we have no mechanisms currently to take care about backwards compatibiliy - we simply do not know if the current task-sdk wil be compatible with current `airflow-core` or the other way round. Since the separation is not finished and the mechanism for making sure backwards compatibility is not here, this is very, very, prone to errors. We saw that multiple times even with simple things like common.sql and other providers, and I think at least for 3.0 and the way how task-sdk uses airflow-core and the other way round, they are still very tightly coupled to each other.
IF the only interface was api and cadwyn that would be fine - but now, effectively we have those imports as "internal-apis" between task-sdk and airflow-core:
* airflow.configuration.*
* airflow.utils.*
* airflow.typing_compat
* airflow.exceptions.*
* airflow.serialization.*
* airflow.task.priority_strategy
* airflow.models.xom_args
* airlfow.models.taskinstancekeys
* airflow.modes.asset
* airflow.timteables.
* airflow.decorators
* airflow.cli
* airflow.ti_deps
* airflow.sensors.base
* airflow.providers.standard.* -> which is likely something we should remove
* airflow.settings
* airfow.secrets
* airflow.dag_processing
* airflow.listeners
* airflow.models.dagbag
* airflow.io.*
* airflow.logging_config
There are also many impots from airflow-core -> task-sdk.:
* airflow.sdk.definitions.*
* airflow.sdk.types.*
* airflow.sdk.execution_time.*
* airflow.sdk.DAG
* airflow.sdk.dag
* airflow.sdk.task
* airflow.sdk.BaseOperator
* airflow.sdk.Variable
* airlfow.sdk.Connection
* airflow.sdk.BaseOperatorLink
* airflow.sdk.TaskGroup
* airflow.sdk.ObjectStoragePath
* airflow.sdk.Context
* airflow.sdk.log
* sitgloe.sdk.io.*
* airflow.sdk.api.*
Those are quick scans and I likely missed something. And I believe except Cadwyn being API versioning for API communication, we have no mechanisms currently to maintain the compatibility.
As i understand, airflow-core-> task-sdk relation is more "intended" to be "public API and we intend to keep TaskSDK compliant - also because of providers that will have to be able to upgrade task-sdk independently), still we - I think - do not have any compatibility mechanisms and any changes in task-sdk might potentialy break something in airflow-core.
The other way round though is not really intended ->task-sdk long term is not supposed to import any of airflow-core classes, so here the brittleness is even more problematic - it's quite likely even task-sdk 1.0.1 will break Airflow 3.0.0 and that Airflow 3.0.1 will break task-sdk 1.0.0. It will be quite difficult to make sure those two work together without back-compatibilty checks.
I'd be personally for open-ending the dependency only after we get rid of the task-sdk-> airflow imports and have some back-compatibility checks or prevention built-in (which shoudl be done together in 3.1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042124217
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
Yeah, `apache-airflow-task-sdk>=1.0.0, <1.1` is fine too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042246145
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
> Yeah, `apache-airflow-task-sdk>=1.0.0, <1.1` is fine too.
How are we going to maintain compatibility ^ see above @kaxil @ashb ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #49221:
URL: https://github.com/apache/airflow/pull/49221
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042202874
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
Just to make a better decision, here is some more data.
The problem with open-ended here is that we have no mechanisms currently to take care about backwards compatibiliy - we simply do not know if the current task-sdk wil be compatible with future `airflow-core` or the other way round. Since the separation is not finished and the mechanism for making sure backwards compatibility is not here, this is very, very, prone to errors. We saw that multiple times even with simple things like common.sql and other providers, and I think at least for 3.0 and the way how task-sdk uses airflow-core and the other way round, they are still very tightly coupled to each other.
IF the only interface was api and cadwyn that would be fine - but now, effectively we have those imports as "internal-apis" between task-sdk and airflow-core:
* airflow.configuration.*
* airflow.utils.*
* airflow.typing_compat
* airflow.exceptions.*
* airflow.serialization.*
* airflow.task.priority_strategy
* airflow.models.xom_args
* airlfow.models.taskinstancekeys
* airflow.models.asset
* airflow.timteables.
* airflow.decorators
* airflow.cli
* airflow.ti_deps
* airflow.sensors.base
* airflow.providers.standard.* -> which is likely something we should remove
* airflow.settings
* airfow.secrets
* airflow.dag_processing
* airflow.listeners
* airflow.models.dagbag
* airflow.io.*
* airflow.logging_config
There are also many impots from airflow-core -> task-sdk.:
* airflow.sdk.definitions.*
* airflow.sdk.types.*
* airflow.sdk.execution_time.*
* airflow.sdk.DAG
* airflow.sdk.dag
* airflow.sdk.task
* airflow.sdk.BaseOperator
* airflow.sdk.Variable
* airlfow.sdk.Connection
* airflow.sdk.BaseOperatorLink
* airflow.sdk.TaskGroup
* airflow.sdk.ObjectStoragePath
* airflow.sdk.Context
* airflow.sdk.log
* sitgloe.sdk.io.*
* airflow.sdk.api.*
Those are quick scans and I likely missed something. And I believe except Cadwyn being API versioning for API communication, we have no mechanisms currently to maintain the compatibility.
As i understand, airflow-core-> task-sdk relation is more "intended" to be "public API and we intend to keep TaskSDK compliant - also because of providers that will have to be able to upgrade task-sdk independently), still we - I think - do not have any compatibility mechanisms and any changes in task-sdk might potentialy break something in airflow-core.
The other way round though is not really intended ->task-sdk long term is not supposed to import any of airflow-core classes, so here the brittleness is even more problematic - it's quite likely even task-sdk 1.0.1 will break Airflow 3.0.0 and that Airflow 3.0.1 will break task-sdk 1.0.0. It will be quite difficult to make sure those two work together without back-compatibilty checks.
I'd be personally for open-ending the dependency only after we get rid of the task-sdk-> airflow imports and have some back-compatibility checks or prevention built-in (which shoudl be done together in 3.1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042318573
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
Sure if you are not afraid unintentional breaking changes - up to you, you are releae manager :D.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #49221:
URL: https://github.com/apache/airflow/pull/49221#issuecomment-2801534542
Tested also with `rc` providers:
## apache-airflow:
```
Metadata-Version: 2.4
Name: apache-airflow
Version: 3.0.0rc1
.....
Requires-Python: <3.13,~=3.9
Requires-Dist: apache-airflow-core==3.0.0.rc1
Requires-Dist: apache-airflow-task-sdk==1.0.0.rc1
```
## apache-airflow-core:
Metadata-Version: 2.4
Name: apache-airflow-core
Version: 3.0.0rc1
...
Requires-Python: <3.13,~=3.9
Requires-Dist: a2wsgi>=1.10.8
Requires-Dist: aiosqlite>=0.20.0
Requires-Dist: alembic<2.0,>=1.13.1
Requires-Dist: apache-airflow-providers-common-compat>=1.6.0.rc1
Requires-Dist: apache-airflow-providers-common-io>=1.5.2.rc1
Requires-Dist: apache-airflow-providers-common-sql>=1.24.1.rc1
Requires-Dist: apache-airflow-providers-smtp>=2.0.1.rc1
Requires-Dist: apache-airflow-providers-standard>=0.2.0.rc1
Requires-Dist: apache-airflow-task-sdk==1.0.0.rc1
```
## apache-airflow-task-sdk
```
Metadata-Version: 2.4
Name: apache-airflow-task-sdk
Version: 1.0.0rc1
....
Requires-Python: <3.13,>=3.9
Requires-Dist: aiologic>=0.14.0
Requires-Dist: apache-airflow-core==3.0.0.rc1
Requires-Dist: attrs!=25.2.0,>=24.2.0
Requires-Dist: fsspec>=2023.10.0
```
@ashb - answering your question from Slack - cicular dependencies in Python Packaging are perfectly fine (as long as they can be resolved. With `fixing` them all now we make sure its' cool and we shoudl just make sure we keep those `==` updated for both `task-sdk` and `apache-airflow` - I guess in 3.0 branch we will have to either remember about bumping them together until 3.1 or automate it a bit more temporarily until 3.1 cc: @kaxil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add cross dependencies between airflow-core and task-sdk [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #49221:
URL: https://github.com/apache/airflow/pull/49221#discussion_r2042121462
##########
airflow-core/pyproject.toml:
##########
@@ -136,6 +136,7 @@ dependencies = [
# Does not work with it Tracked in https://github.com/fsspec/universal_pathlib/issues/276
"universal-pathlib>=0.2.2,!=0.2.4",
"uuid6>=2024.7.10",
+ "apache-airflow-task-sdk==1.0.0",
Review Comment:
What do you think about this @kaxil? (Other option would to be more open-ended here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org