You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2025/03/04 08:09:13 UTC
[PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
amoghrajesh opened a new pull request, #47339:
URL: https://github.com/apache/airflow/pull/47339
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
closes: #45481
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #47339:
URL: https://github.com/apache/airflow/pull/47339#issuecomment-2731846432
@eladkal yep that case is covered as well as that logic to handle threshold is handled within the custom xcom backend. No part of that has been tweaked at all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1988893787
##########
task-sdk/src/airflow/sdk/definitions/baseoperatorlink.py:
##########
@@ -38,6 +37,8 @@ class XComOperatorLink(LoggingMixin):
xcom_key: str
def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey) -> str:
+ from airflow.sdk.execution_time.xcom import BaseXCom
Review Comment:
Handled in dc8294832fe41ea7f9b4ad4628f7db841b9db063
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #47339:
URL: https://github.com/apache/airflow/pull/47339#issuecomment-2731842779
You mentioned:
> Situation 1: Using the database for xcoms (no xcom backend)
> Situation 2: Using a custom xcom backend.
There is also situation 3 that mix both using `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD`
https://airflow.apache.org/docs/apache-airflow-providers-common-io/stable/configurations-ref.html#xcom-objectstorage-threshold
Are we covered in this case as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998350137
##########
airflow/models/__init__.py:
##########
@@ -107,7 +107,7 @@ def __getattr__(name):
"TaskReschedule": "airflow.models.taskreschedule",
"Trigger": "airflow.models.trigger",
"Variable": "airflow.models.variable",
- "XCom": "airflow.models.xcom",
+ "XComModel": "airflow.models.xcom",
Review Comment:
Not add -- we don't need XComModel in here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987143842
##########
task-sdk/src/airflow/sdk/execution_time/__init__.py:
##########
Review Comment:
I'm not sure we need the lazy import here do we? Does anything do `from airflow.sdk.execution_time import BaseXCom`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987141257
##########
providers/common/io/src/airflow/providers/common/io/xcom/backend.py:
##########
@@ -149,7 +145,7 @@ def serialize_value( # type: ignore[override]
return BaseXCom.serialize_value(str(p))
@staticmethod
- def deserialize_value(result: XCom) -> Any:
+ def deserialize_value(result) -> Any:
Review Comment:
Is there no typing we can usefully have anymore here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987150750
##########
task-sdk/src/airflow/sdk/definitions/baseoperatorlink.py:
##########
@@ -38,6 +37,8 @@ class XComOperatorLink(LoggingMixin):
xcom_key: str
def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey) -> str:
+ from airflow.sdk.execution_time.xcom import BaseXCom
Review Comment:
And then change L52 to `XCom.get()` I think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991182422
##########
airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -221,6 +225,7 @@ def ti_run(
# TODO: Add variables and connections that are needed (and has perms) for the task
variables=[],
connections=[],
+ xcom_keys_to_clear=xcom_keys,
Review Comment:
Why are we sending the keys to be clearer to the client?
Shouldn't we be just deleting those keys in the server here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998073615
##########
airflow/api_fastapi/execution_api/routes/xcoms.py:
##########
@@ -224,16 +225,39 @@ def set_xcom(
# (the mapped task would fail in a moment as it can't be expanded anyway.)
# We use `BaseXCom.set` to set XComs directly to the database, bypassing the XCom Backend.
+ from airflow.models.dagrun import DagRun
+
+ if not run_id:
+ raise HTTPException(status.HTTP_404_NOT_FOUND, f"Run with ID: `{run_id}` was not found")
+
+ dag_run_id = session.query(DagRun.id).filter_by(dag_id=dag_id, run_id=run_id).scalar()
+ if dag_run_id is None:
+ raise HTTPException(status.HTTP_404_NOT_FOUND, f"DAG run not found on DAG {dag_id} with ID {run_id}")
+
+ # Remove duplicate XComs and insert a new one.
+ session.execute(
+ delete(XComModel).where(
+ XComModel.key == key,
+ XComModel.run_id == run_id,
+ XComModel.task_id == task_id,
+ XComModel.dag_id == dag_id,
+ XComModel.map_index == map_index,
+ )
+ )
+
try:
- BaseXCom.set(
+ # We expect serialised value from the caller - sdk, do not serialise in here
+ new = cast(Any, XComModel)( # Work around Mypy complaining model not defining '__init__'.
Review Comment:
Correct! We do not need it anymore as there's no magic here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998074842
##########
airflow/models/xcom.py:
##########
@@ -31,29 +30,27 @@
Integer,
PrimaryKeyConstraint,
String,
- delete,
select,
text,
)
from sqlalchemy.dialects import postgresql
from sqlalchemy.ext.associationproxy import association_proxy
-from sqlalchemy.orm import Query, reconstructor, relationship
+from sqlalchemy.orm import Query, relationship
-from airflow.configuration import conf
+import airflow.sdk.execution_time.xcom
Review Comment:
Not relevant anymore, changed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987137943
##########
airflow/models/xcom.py:
##########
@@ -105,26 +99,54 @@ class BaseXCom(TaskInstanceDependencies, LoggingMixin):
dag_run = relationship(
"DagRun",
- primaryjoin="BaseXCom.dag_run_id == foreign(DagRun.id)",
+ primaryjoin="XComModel.dag_run_id == foreign(DagRun.id)",
uselist=False,
lazy="joined",
passive_deletes="all",
)
logical_date = association_proxy("dag_run", "logical_date")
- @reconstructor
- def init_on_load(self):
+ @classmethod
+ @provide_session
+ def clear(
+ cls,
+ *,
+ dag_id: str,
+ task_id: str,
+ run_id: str,
+ map_index: int | None = None,
+ session: Session = NEW_SESSION,
+ ) -> None:
"""
- Execute after the instance has been loaded from the DB or otherwise reconstituted; called by the ORM.
+ Clear all XCom data from the database for the given task instance.
Review Comment:
```suggestion
Clear all XCom data from the database for the given task instance.
.. note:: This **will not** purge any data from a custom XCom backend.
```
just to be explicit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991123885
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -578,6 +541,19 @@ def run(
ti: RuntimeTaskInstance, log: Logger
) -> tuple[IntermediateTIState | TerminalTIState, ToSupervisor | None, BaseException | None]:
"""Run the task in this process."""
+ # First, clear the xcom data
+ if ti._ti_context_from_server and ti._ti_context_from_server.xcom_keys_to_clear:
+ keys_to_delete = ti._ti_context_from_server.xcom_keys_to_clear
Review Comment:
Yeah it is wrong indeed. I will move it right before getting the context and i like your nit better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1995120194
##########
airflow/api_fastapi/execution_api/routes/xcoms.py:
##########
@@ -224,16 +225,39 @@ def set_xcom(
# (the mapped task would fail in a moment as it can't be expanded anyway.)
# We use `BaseXCom.set` to set XComs directly to the database, bypassing the XCom Backend.
+ from airflow.models.dagrun import DagRun
+
+ if not run_id:
+ raise HTTPException(status.HTTP_404_NOT_FOUND, f"Run with ID: `{run_id}` was not found")
+
+ dag_run_id = session.query(DagRun.id).filter_by(dag_id=dag_id, run_id=run_id).scalar()
+ if dag_run_id is None:
+ raise HTTPException(status.HTTP_404_NOT_FOUND, f"DAG run not found on DAG {dag_id} with ID {run_id}")
+
+ # Remove duplicate XComs and insert a new one.
+ session.execute(
+ delete(XComModel).where(
+ XComModel.key == key,
+ XComModel.run_id == run_id,
+ XComModel.task_id == task_id,
+ XComModel.dag_id == dag_id,
+ XComModel.map_index == map_index,
+ )
+ )
+
try:
- BaseXCom.set(
+ # We expect serialised value from the caller - sdk, do not serialise in here
+ new = cast(Any, XComModel)( # Work around Mypy complaining model not defining '__init__'.
Review Comment:
The issue is I think to do with what mypy sees as the constructor args.
However I have a feeling this isn't needed anymore as XComModel isn't magic like the way XCom/BaseXCom was
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #47339:
URL: https://github.com/apache/airflow/pull/47339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed pull request #47339: AIP-72: Handle Custom XCom Backend on Task SDK
URL: https://github.com/apache/airflow/pull/47339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #47339:
URL: https://github.com/apache/airflow/pull/47339#issuecomment-2728666074
Did a final round of regression tests with a few example dags both with Xcom DB backend and custom XCom backend (described in PR desc)
DAG1:
```
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
def push_to_xcom(**kwargs):
value = ["This is a long message"] * 10
# value = ("key", "value")
return value
#
# def push_to_xcom2(**kwargs):
# value = ["Hello, XCom2!"]
# return value
def pull_from_xcom(**kwargs):
ti = kwargs['ti']
xcom_value1 = ti.xcom_pull(task_ids=["push_xcom_task"])
print("Pulled value", xcom_value1)
topush = xcom_value1 + ["modified"] * 10
print("Pushing value", topush)
# xcom_value2 = ti.xcom_pull(task_ids=["push_xcom_task2"])
return topush
with DAG(
'xcom_example',
schedule=None,
catchup=False,
) as dag:
push_xcom_task = PythonOperator(
task_id='push_xcom_task',
python_callable=push_to_xcom,
)
# push_xcom_task2 = PythonOperator(
# task_id='push_xcom_task2',
# python_callable=push_to_xcom2,
# )
pull_xcom_task = PythonOperator(
task_id='pull_xcom_task',
python_callable=pull_from_xcom,
)
push_xcom_task >> pull_xcom_task
```
DAG2:
```
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
def push_to_xcom(**kwargs):
value = ("Hello", "XCom!")
return value
with DAG(
'xcom_tuple_return',
schedule=None,
catchup=False,
) as dag:
push_xcom_task = PythonOperator(
task_id='tuple_task',
python_callable=push_to_xcom,
)
```
DAG3:
```
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
from datetime import datetime
# Task to push an XCom
def push_xcom(**kwargs):
kwargs['ti'].xcom_push(key="example_key", value="example_value")
# Task to pull an XCom
def pull_xcom(**kwargs):
pulled_value = kwargs['ti'].xcom_pull(task_ids="push_task", key="example_key")
print(f"Pulled XCom Value: {pulled_value}")
# Define the DAG
with DAG(
dag_id="simple_xcom_pull_example",
default_args={"owner": "airflow"},
schedule=None,
start_date=datetime(2023, 1, 1),
catchup=False,
) as dag:
# Task 1: Push XCom
push_task = PythonOperator(
task_id="push_task",
python_callable=push_xcom,
)
# Task 2: Pull XCom
pull_task = PythonOperator(
task_id="pull_task",
python_callable=pull_xcom,
)
# Define task dependencies
push_task >> pull_task
```
DAG4:
```
"""Example DAG demonstrating the usage of XComs."""
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator
from datetime import datetime, timedelta
dag = DAG(
"example_xcom",
start_date=datetime(2023, 11, 28),
default_args={"owner": "airflow"},
schedule="@daily",
catchup=False,
tags=["core"],
)
value_1 = [1, 2, 3]
value_2 = {"a": "b"}
def push(**kwargs):
"""Pushes an XCom without a specific target"""
kwargs["ti"].xcom_push(key="value from pusher 1", value=value_1)
def push_by_returning(**kwargs):
"""Pushes an XCom without a specific target, just by returning it"""
return value_2
def puller(**kwargs):
"""Pull all previously pushed XComs and check if the pushed values match the pulled values."""
ti = kwargs["ti"]
# get value_1
pulled_value_1 = ti.xcom_pull(key=None, task_ids="push")
if pulled_value_1 != value_1:
raise ValueError(f"The two values differ {pulled_value_1} and {value_1}")
# get value_2
pulled_value_2 = ti.xcom_pull(task_ids="push_by_returning")
if pulled_value_2 != value_2:
raise ValueError(f"The two values differ {pulled_value_2} and {value_2}")
# get both value_1 and value_2
pulled_value_1, pulled_value_2 = ti.xcom_pull(
key=None, task_ids=["push", "push_by_returning"]
)
print(f"pulled_value_1 is {pulled_value_1}")
print(f"pulled_value_2 is {pulled_value_2}")
if pulled_value_1 != value_1:
raise ValueError(f"The two values differ {pulled_value_1} and {value_1}")
if pulled_value_2 != value_2:
raise ValueError(f"The two values differ {pulled_value_2} and {value_2}")
push1 = PythonOperator(
task_id="push",
dag=dag,
python_callable=push,
depends_on_past=True,
)
push2 = PythonOperator(
task_id="push_by_returning",
dag=dag,
python_callable=push_by_returning,
)
pull = PythonOperator(
task_id="puller",
dag=dag,
python_callable=puller,
)
pull << [push1, push2]
```
These works as expected, both with the custom xcom backend as well as with the XCom db backend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998075561
##########
providers/google/src/airflow/providers/google/cloud/links/base.py:
##########
@@ -28,7 +27,9 @@
if AIRFLOW_V_3_0_PLUS:
from airflow.sdk import BaseOperatorLink
+ from airflow.sdk.execution_time.xcom import XCom
else:
+ from airflow.models import XCom # type: ignore[no-redef]
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987147249
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -578,6 +541,19 @@ def run(
ti: RuntimeTaskInstance, log: Logger
) -> tuple[IntermediateTIState | TerminalTIState, ToSupervisor | None, BaseException | None]:
"""Run the task in this process."""
+ # First, clear the xcom data
+ if ti._ti_context_from_server and ti._ti_context_from_server.xcom_keys_to_clear:
+ keys_to_delete = ti._ti_context_from_server.xcom_keys_to_clear
Review Comment:
Nit? (Not sure which is better actually)
```suggestion
if ti._ti_context_from_server and (keys_to_delete := ti._ti_context_from_server.xcom_keys_to_clear):
```
However putthingb this above the imports and setup feels wrong. This should be move to inside the try block I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987133534
##########
airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -186,18 +186,18 @@ def ti_run(
if not dr:
raise ValueError(f"DagRun with dag_id={ti.dag_id} and run_id={ti.run_id} not found.")
- # Clear XCom data for the task instance since we are certain it is executing
+ # Send the XCom data for clearance for the task instance since we are certain it is executing at this point.
# However, do not clear it for deferral
if not ti.next_method:
map_index = None if ti.map_index < 0 else ti.map_index
log.info("Clearing xcom data for task id: %s", ti_id_str)
- XCom.clear(
- dag_id=ti.dag_id,
- task_id=ti.task_id,
- run_id=ti.run_id,
- map_index=map_index,
- session=session,
+ query = session.query(XComModel.key).filter_by(
+ dag_id=ti.dag_id, task_id=ti.task_id, run_id=ti.run_id
)
+ if map_index is not None:
+ query = query.filter_by(map_index=map_index)
Review Comment:
Is this not `XComModel.get_many()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991193157
##########
airflow/models/xcom.py:
##########
@@ -31,29 +30,27 @@
Integer,
PrimaryKeyConstraint,
String,
- delete,
select,
text,
)
from sqlalchemy.dialects import postgresql
from sqlalchemy.ext.associationproxy import association_proxy
-from sqlalchemy.orm import Query, reconstructor, relationship
+from sqlalchemy.orm import Query, relationship
-from airflow.configuration import conf
+import airflow.sdk.execution_time.xcom
Review Comment:
```suggestion
from airflow.sdk.execution_time.xcom import resolve_xcom_backend
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991199458
##########
providers/google/src/airflow/providers/google/cloud/links/base.py:
##########
@@ -28,7 +27,9 @@
if AIRFLOW_V_3_0_PLUS:
from airflow.sdk import BaseOperatorLink
+ from airflow.sdk.execution_time.xcom import XCom
else:
+ from airflow.models import XCom # type: ignore[no-redef]
Review Comment:
```suggestion
from airflow.models.xcom import XCom # type: ignore[no-redef]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987131318
##########
airflow/api_fastapi/execution_api/datamodels/taskinstance.py:
##########
@@ -267,6 +267,9 @@ class TIRunContext(BaseModel):
Can either be a "decorated" dict, or a string encrypted with the shared Fernet key.
"""
+ """List of Xcom keys that need to be cleared."""
+ xcom_keys_to_clear: Annotated[list[str], Field(default_factory=list)]
Review Comment:
It's odd, but doc comments for a variable goes after the var
```suggestion
xcom_keys_to_clear: Annotated[list[str], Field(default_factory=list)]
"""List of Xcom keys that need to be cleared."""
```
##########
airflow/api_fastapi/execution_api/datamodels/taskinstance.py:
##########
@@ -267,6 +267,9 @@ class TIRunContext(BaseModel):
Can either be a "decorated" dict, or a string encrypted with the shared Fernet key.
"""
+ """List of Xcom keys that need to be cleared."""
+ xcom_keys_to_clear: Annotated[list[str], Field(default_factory=list)]
Review Comment:
It's odd, but doc comments for a variable goes after the var
```suggestion
xcom_keys_to_clear: Annotated[list[str], Field(default_factory=list)]
"""List of Xcom keys that need to be cleared on the worker."""
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987119161
##########
task_sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -357,6 +369,15 @@ def _xcom_push(ti: RuntimeTaskInstance, key: str, value: Any, mapped_length: int
# consumers
from airflow.serialization.serde import serialize
+ if XCom:
+ XCom.set(
+ key=key,
+ value=value,
+ dag_id=ti.dag_id,
+ task_id=ti.task_id,
+ run_id=ti.run_id,
+ )
+ return
Review Comment:
I reworked this part, resolving this comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987129862
##########
airflow/api_fastapi/core_api/routes/public/xcom.py:
##########
@@ -67,14 +68,17 @@ def get_xcom_entry(
raise HTTPException(
status.HTTP_400_BAD_REQUEST, "XCom deserialization is disabled in configuration."
)
- query = select(XCom, XCom.value)
+ query = select(XComModel, XComModel.value)
Review Comment:
Why select the whole model and the individual column at the same time? That seems a slightly odd way of doing things
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987142437
##########
task-sdk/src/airflow/sdk/definitions/baseoperatorlink.py:
##########
@@ -38,6 +37,8 @@ class XComOperatorLink(LoggingMixin):
xcom_key: str
def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey) -> str:
+ from airflow.sdk.execution_time.xcom import BaseXCom
Review Comment:
We shouldn't be using this as "BaseXCom" really -- that feels like an odd interface. I think the we should do the "export it as XCom" that we did before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #47339:
URL: https://github.com/apache/airflow/pull/47339#issuecomment-2726927093
Reopening to run tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1995060546
##########
airflow/models/xcom.py:
##########
@@ -490,44 +312,21 @@ class LazyXComSelectSequence(LazySelectSequence[Any]):
@staticmethod
def _rebuild_select(stmt: TextClause) -> Select:
- return select(XCom.value).from_statement(stmt)
+ return select(XComModel.value).from_statement(stmt)
@staticmethod
def _process_row(row: Row) -> Any:
- return XCom.deserialize_value(row)
-
-
-def _get_function_params(function) -> list[str]:
- """
- Return the list of variables names of a function.
-
- :param function: The function to inspect
- """
- parameters = inspect.signature(function).parameters
- bound_arguments = [
- name for name, p in parameters.items() if p.kind not in (p.VAR_POSITIONAL, p.VAR_KEYWORD)
- ]
- return bound_arguments
-
-
-def resolve_xcom_backend() -> type[BaseXCom]:
- """
- Resolve custom XCom class.
-
- Confirm that custom XCom class extends the BaseXCom.
- Compare the function signature of the custom XCom serialize_value to the base XCom serialize_value.
- """
- clazz = conf.getimport("core", "xcom_backend", fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
- if not clazz:
- return BaseXCom
- if not issubclass(clazz, BaseXCom):
- raise TypeError(
- f"Your custom XCom class `{clazz.__name__}` is not a subclass of `{BaseXCom.__name__}`."
- )
- return clazz
+ return XComModel.deserialize_value(row)
if TYPE_CHECKING:
- XCom = BaseXCom # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+ XCom = XComModel
else:
- XCom = resolve_xcom_backend()
+ XCom = airflow.sdk.execution_time.xcom.resolve_xcom_backend()
+
+
+# 1. Implement DELETE API -- run endpoint should return keys and those should eb called by worker. ✅
+# 2. (Do not clear during xcom during /run) ✅
+# 3. Wipe out BaseXcom in models ✅
+# 4. Purge after getting the ctx then delete request (XCOM.delete also)
+# 5. Lazy import from task sdk: __get_attr__ ✅
Review Comment:
This has been handled now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998994475
##########
airflow/models/__init__.py:
##########
@@ -107,7 +107,7 @@ def __getattr__(name):
"TaskReschedule": "airflow.models.taskreschedule",
"Trigger": "airflow.models.trigger",
"Variable": "airflow.models.variable",
- "XCom": "airflow.models.xcom",
+ "XComModel": "airflow.models.xcom",
Review Comment:
Made that change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991120936
##########
airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -186,18 +186,18 @@ def ti_run(
if not dr:
raise ValueError(f"DagRun with dag_id={ti.dag_id} and run_id={ti.run_id} not found.")
- # Clear XCom data for the task instance since we are certain it is executing
+ # Send the XCom data for clearance for the task instance since we are certain it is executing at this point.
Review Comment:
Yep explained.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1988888559
##########
airflow/models/xcom.py:
##########
@@ -490,44 +312,21 @@ class LazyXComSelectSequence(LazySelectSequence[Any]):
@staticmethod
def _rebuild_select(stmt: TextClause) -> Select:
- return select(XCom.value).from_statement(stmt)
+ return select(XComModel.value).from_statement(stmt)
@staticmethod
def _process_row(row: Row) -> Any:
- return XCom.deserialize_value(row)
-
-
-def _get_function_params(function) -> list[str]:
- """
- Return the list of variables names of a function.
-
- :param function: The function to inspect
- """
- parameters = inspect.signature(function).parameters
- bound_arguments = [
- name for name, p in parameters.items() if p.kind not in (p.VAR_POSITIONAL, p.VAR_KEYWORD)
- ]
- return bound_arguments
-
-
-def resolve_xcom_backend() -> type[BaseXCom]:
- """
- Resolve custom XCom class.
-
- Confirm that custom XCom class extends the BaseXCom.
- Compare the function signature of the custom XCom serialize_value to the base XCom serialize_value.
- """
- clazz = conf.getimport("core", "xcom_backend", fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
- if not clazz:
- return BaseXCom
- if not issubclass(clazz, BaseXCom):
- raise TypeError(
- f"Your custom XCom class `{clazz.__name__}` is not a subclass of `{BaseXCom.__name__}`."
- )
- return clazz
+ return XComModel.deserialize_value(row)
if TYPE_CHECKING:
- XCom = BaseXCom # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+ XCom = XComModel
else:
- XCom = resolve_xcom_backend()
+ XCom = airflow.sdk.execution_time.xcom.resolve_xcom_backend()
+
+
+# 1. Implement DELETE API -- run endpoint should return keys and those should eb called by worker. ✅
+# 2. (Do not clear during xcom during /run) ✅
+# 3. Wipe out BaseXcom in models ✅
+# 4. Purge after getting the ctx then delete request (XCOM.delete also)
+# 5. Lazy import from task sdk: __get_attr__ ✅
Review Comment:
Its already there between L325-326
```
if TYPE_CHECKING:
XCom = XComModel
else:
XCom = airflow.sdk.execution_time.xcom.resolve_xcom_backend()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1995124177
##########
airflow/models/skipmixin.py:
##########
@@ -124,12 +124,12 @@ def skip(
session.commit()
if task_id is not None:
- from airflow.models.xcom import XCom
+ from airflow.models.xcom import XComModel
Review Comment:
```suggestion
from airflow.sdk import XCom
```
##########
airflow/models/skipmixin.py:
##########
@@ -124,12 +124,12 @@ def skip(
session.commit()
if task_id is not None:
- from airflow.models.xcom import XCom
+ from airflow.models.xcom import XComModel
if map_index is None:
map_index = -1
- XCom.set(
+ XComModel.set(
Review Comment:
```suggestion
XCom.set(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1995116759
##########
airflow/api_fastapi/execution_api/routes/xcoms.py:
##########
@@ -224,16 +225,39 @@ def set_xcom(
# (the mapped task would fail in a moment as it can't be expanded anyway.)
# We use `BaseXCom.set` to set XComs directly to the database, bypassing the XCom Backend.
Review Comment:
```suggestion
```
since we don't anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #47339:
URL: https://github.com/apache/airflow/pull/47339#issuecomment-2710629508
Let the CI fixing war commence! ⚔️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1988898824
##########
airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -186,18 +186,18 @@ def ti_run(
if not dr:
raise ValueError(f"DagRun with dag_id={ti.dag_id} and run_id={ti.run_id} not found.")
- # Clear XCom data for the task instance since we are certain it is executing
+ # Send the XCom data for clearance for the task instance since we are certain it is executing at this point.
# However, do not clear it for deferral
if not ti.next_method:
map_index = None if ti.map_index < 0 else ti.map_index
log.info("Clearing xcom data for task id: %s", ti_id_str)
Review Comment:
Agreed, removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1988891137
##########
task-sdk/src/airflow/sdk/definitions/baseoperatorlink.py:
##########
@@ -38,6 +37,8 @@ class XComOperatorLink(LoggingMixin):
xcom_key: str
def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey) -> str:
+ from airflow.sdk.execution_time.xcom import BaseXCom
Review Comment:
Makes sense. Its get_one itself though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987134116
##########
airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -186,18 +186,18 @@ def ti_run(
if not dr:
raise ValueError(f"DagRun with dag_id={ti.dag_id} and run_id={ti.run_id} not found.")
- # Clear XCom data for the task instance since we are certain it is executing
+ # Send the XCom data for clearance for the task instance since we are certain it is executing at this point.
# However, do not clear it for deferral
if not ti.next_method:
map_index = None if ti.map_index < 0 else ti.map_index
log.info("Clearing xcom data for task id: %s", ti_id_str)
Review Comment:
This log isn't accurate anymore - maybe not worth keeping since we aren't doing anything destructive/of note here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987132946
##########
airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -186,18 +186,18 @@ def ti_run(
if not dr:
raise ValueError(f"DagRun with dag_id={ti.dag_id} and run_id={ti.run_id} not found.")
- # Clear XCom data for the task instance since we are certain it is executing
+ # Send the XCom data for clearance for the task instance since we are certain it is executing at this point.
Review Comment:
We should mention here why we can't do this on the server, namely we'll need to issue a purge in the custom XCom backend, and that the server may not have access or permission to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998071726
##########
providers/common/io/src/airflow/providers/common/io/xcom/backend.py:
##########
@@ -149,7 +145,7 @@ def serialize_value( # type: ignore[override]
return BaseXCom.serialize_value(str(p))
@staticmethod
- def deserialize_value(result: XCom) -> Any:
+ def deserialize_value(result) -> Any:
Review Comment:
Slightly difficult to do that right now, I will revisit it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998077093
##########
airflow/api_fastapi/execution_api/routes/xcoms.py:
##########
@@ -224,16 +225,39 @@ def set_xcom(
# (the mapped task would fail in a moment as it can't be expanded anyway.)
# We use `BaseXCom.set` to set XComs directly to the database, bypassing the XCom Backend.
Review Comment:
Yeah good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #47339:
URL: https://github.com/apache/airflow/pull/47339#issuecomment-2729915597
Huh, finally got a green run here. I am going to merge this since it has already gone too huge, will take up any issues in follow ups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1978927162
##########
task_sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -357,6 +369,15 @@ def _xcom_push(ti: RuntimeTaskInstance, key: str, value: Any, mapped_length: int
# consumers
from airflow.serialization.serde import serialize
+ if XCom:
+ XCom.set(
+ key=key,
+ value=value,
+ dag_id=ti.dag_id,
+ task_id=ti.task_id,
+ run_id=ti.run_id,
+ )
+ return
Review Comment:
Right now its calling it out of models, I think it is high time we move the classes out of models/xcoms.py into execution_time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987138614
##########
airflow/models/xcom.py:
##########
@@ -161,27 +183,6 @@ def set(
if dag_run_id is None:
raise ValueError(f"DAG run not found on DAG {dag_id!r} with ID {run_id!r}")
- # Seamlessly resolve LazySelectSequence to a list. This intends to work
- # as a "lazy list" to avoid pulling a ton of XComs unnecessarily, but if
- # it's pushed into XCom, the user should be aware of the performance
- # implications, and this avoids leaking the implementation detail.
- if isinstance(value, LazySelectSequence):
Review Comment:
We can probably delete that class now too, as I think this was the last usage of it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987150281
##########
task-sdk/src/airflow/sdk/definitions/baseoperatorlink.py:
##########
@@ -38,6 +37,8 @@ class XComOperatorLink(LoggingMixin):
xcom_key: str
def get_link(self, operator: BaseOperator, *, ti_key: TaskInstanceKey) -> str:
+ from airflow.sdk.execution_time.xcom import BaseXCom
Review Comment:
Yeah, there's an `XCom` exported from that
```suggestion
from airflow.sdk.execution_time.xcom import XCom
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1995122816
##########
airflow/models/__init__.py:
##########
@@ -107,7 +107,7 @@ def __getattr__(name):
"TaskReschedule": "airflow.models.taskreschedule",
"Trigger": "airflow.models.trigger",
"Variable": "airflow.models.variable",
- "XCom": "airflow.models.xcom",
+ "XComModel": "airflow.models.xcom",
Review Comment:
This is to support `from airflow.models importXCom` so ths should be
```suggestion
"XCom": "airflow.sdk.execution_time.xcom",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1995123296
##########
airflow/models/__init__.py:
##########
@@ -135,6 +135,6 @@ def __getattr__(name):
from airflow.models.taskreschedule import TaskReschedule
from airflow.models.trigger import Trigger
from airflow.models.variable import Variable
- from airflow.models.xcom import XCom
+ from airflow.models.xcom import XComModel
Review Comment:
ditto
```suggestion
from airflow.sdk.execution_time.xcom import XCom
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1988904646
##########
airflow/models/xcom.py:
##########
@@ -105,26 +99,54 @@ class BaseXCom(TaskInstanceDependencies, LoggingMixin):
dag_run = relationship(
"DagRun",
- primaryjoin="BaseXCom.dag_run_id == foreign(DagRun.id)",
+ primaryjoin="XComModel.dag_run_id == foreign(DagRun.id)",
uselist=False,
lazy="joined",
passive_deletes="all",
)
logical_date = association_proxy("dag_run", "logical_date")
- @reconstructor
- def init_on_load(self):
+ @classmethod
+ @provide_session
+ def clear(
+ cls,
+ *,
+ dag_id: str,
+ task_id: str,
+ run_id: str,
+ map_index: int | None = None,
+ session: Session = NEW_SESSION,
+ ) -> None:
"""
- Execute after the instance has been loaded from the DB or otherwise reconstituted; called by the ORM.
+ Clear all XCom data from the database for the given task instance.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1988897873
##########
airflow/models/xcom.py:
##########
@@ -161,27 +183,6 @@ def set(
if dag_run_id is None:
raise ValueError(f"DAG run not found on DAG {dag_id!r} with ID {run_id!r}")
- # Seamlessly resolve LazySelectSequence to a list. This intends to work
- # as a "lazy list" to avoid pulling a ton of XComs unnecessarily, but if
- # it's pushed into XCom, the user should be aware of the performance
- # implications, and this avoids leaking the implementation detail.
- if isinstance(value, LazySelectSequence):
Review Comment:
Its actually used in task instance models xcom_push and xcom_pull. Hence I didnt wipe it right away
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987140204
##########
airflow/models/xcom.py:
##########
@@ -490,44 +312,21 @@ class LazyXComSelectSequence(LazySelectSequence[Any]):
@staticmethod
def _rebuild_select(stmt: TextClause) -> Select:
- return select(XCom.value).from_statement(stmt)
+ return select(XComModel.value).from_statement(stmt)
@staticmethod
def _process_row(row: Row) -> Any:
- return XCom.deserialize_value(row)
-
-
-def _get_function_params(function) -> list[str]:
- """
- Return the list of variables names of a function.
-
- :param function: The function to inspect
- """
- parameters = inspect.signature(function).parameters
- bound_arguments = [
- name for name, p in parameters.items() if p.kind not in (p.VAR_POSITIONAL, p.VAR_KEYWORD)
- ]
- return bound_arguments
-
-
-def resolve_xcom_backend() -> type[BaseXCom]:
- """
- Resolve custom XCom class.
-
- Confirm that custom XCom class extends the BaseXCom.
- Compare the function signature of the custom XCom serialize_value to the base XCom serialize_value.
- """
- clazz = conf.getimport("core", "xcom_backend", fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
- if not clazz:
- return BaseXCom
- if not issubclass(clazz, BaseXCom):
- raise TypeError(
- f"Your custom XCom class `{clazz.__name__}` is not a subclass of `{BaseXCom.__name__}`."
- )
- return clazz
+ return XComModel.deserialize_value(row)
if TYPE_CHECKING:
- XCom = BaseXCom # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+ XCom = XComModel
else:
- XCom = resolve_xcom_backend()
+ XCom = airflow.sdk.execution_time.xcom.resolve_xcom_backend()
+
+
+# 1. Implement DELETE API -- run endpoint should return keys and those should eb called by worker. ✅
+# 2. (Do not clear during xcom during /run) ✅
+# 3. Wipe out BaseXcom in models ✅
+# 4. Purge after getting the ctx then delete request (XCOM.delete also)
+# 5. Lazy import from task sdk: __get_attr__ ✅
Review Comment:
You left in your todo list, but I don't think point 5 is done yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987151222
##########
airflow/models/xcom.py:
##########
@@ -490,44 +312,21 @@ class LazyXComSelectSequence(LazySelectSequence[Any]):
@staticmethod
def _rebuild_select(stmt: TextClause) -> Select:
- return select(XCom.value).from_statement(stmt)
+ return select(XComModel.value).from_statement(stmt)
@staticmethod
def _process_row(row: Row) -> Any:
- return XCom.deserialize_value(row)
-
-
-def _get_function_params(function) -> list[str]:
- """
- Return the list of variables names of a function.
-
- :param function: The function to inspect
- """
- parameters = inspect.signature(function).parameters
- bound_arguments = [
- name for name, p in parameters.items() if p.kind not in (p.VAR_POSITIONAL, p.VAR_KEYWORD)
- ]
- return bound_arguments
-
-
-def resolve_xcom_backend() -> type[BaseXCom]:
- """
- Resolve custom XCom class.
-
- Confirm that custom XCom class extends the BaseXCom.
- Compare the function signature of the custom XCom serialize_value to the base XCom serialize_value.
- """
- clazz = conf.getimport("core", "xcom_backend", fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
- if not clazz:
- return BaseXCom
- if not issubclass(clazz, BaseXCom):
- raise TypeError(
- f"Your custom XCom class `{clazz.__name__}` is not a subclass of `{BaseXCom.__name__}`."
- )
- return clazz
+ return XComModel.deserialize_value(row)
if TYPE_CHECKING:
- XCom = BaseXCom # Hack to avoid Mypy "Variable 'XCom' is not valid as a type".
+ XCom = XComModel
else:
- XCom = resolve_xcom_backend()
+ XCom = airflow.sdk.execution_time.xcom.resolve_xcom_backend()
+
+
+# 1. Implement DELETE API -- run endpoint should return keys and those should eb called by worker. ✅
+# 2. (Do not clear during xcom during /run) ✅
+# 3. Wipe out BaseXcom in models ✅
+# 4. Purge after getting the ctx then delete request (XCOM.delete also)
+# 5. Lazy import from task sdk: __get_attr__ ✅
Review Comment:
Yeah, the lazy import should be in this module as a back-compat shim I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998075962
##########
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py:
##########
@@ -57,6 +56,11 @@
from tests_common.test_utils import db
from tests_common.test_utils.version_compat import AIRFLOW_V_3_0_PLUS
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.xcom import XCom
+else:
+ from airflow.models import XCom # type: ignore[no-redef]
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998077819
##########
airflow/models/__init__.py:
##########
@@ -107,7 +107,7 @@ def __getattr__(name):
"TaskReschedule": "airflow.models.taskreschedule",
"Trigger": "airflow.models.trigger",
"Variable": "airflow.models.variable",
- "XCom": "airflow.models.xcom",
+ "XComModel": "airflow.models.xcom",
Review Comment:
Yep added it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998079819
##########
airflow/models/skipmixin.py:
##########
@@ -124,12 +124,12 @@ def skip(
session.commit()
if task_id is not None:
- from airflow.models.xcom import XCom
+ from airflow.models.xcom import XComModel
Review Comment:
Not relevant anymore, the file is gone
##########
airflow/models/skipmixin.py:
##########
@@ -124,12 +124,12 @@ def skip(
session.commit()
if task_id is not None:
- from airflow.models.xcom import XCom
+ from airflow.models.xcom import XComModel
if map_index is None:
map_index = -1
- XCom.set(
+ XComModel.set(
Review Comment:
Same for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1978927162
##########
task_sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -357,6 +369,15 @@ def _xcom_push(ti: RuntimeTaskInstance, key: str, value: Any, mapped_length: int
# consumers
from airflow.serialization.serde import serialize
+ if XCom:
+ XCom.set(
+ key=key,
+ value=value,
+ dag_id=ti.dag_id,
+ task_id=ti.task_id,
+ run_id=ti.run_id,
+ )
+ return
Review Comment:
Right now its calling it out of models (set), I think it is high time we move the classes out of models/xcoms.py into execution_time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991182422
##########
airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -221,6 +225,7 @@ def ti_run(
# TODO: Add variables and connections that are needed (and has perms) for the task
variables=[],
connections=[],
+ xcom_keys_to_clear=xcom_keys,
Review Comment:
Why are we sending the keys to be clearer to the client?
Shouldn't we be just deleting those keys in the server here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991110713
##########
airflow/api_fastapi/execution_api/datamodels/taskinstance.py:
##########
@@ -267,6 +267,9 @@ class TIRunContext(BaseModel):
Can either be a "decorated" dict, or a string encrypted with the shared Fernet key.
"""
+ """List of Xcom keys that need to be cleared."""
+ xcom_keys_to_clear: Annotated[list[str], Field(default_factory=list)]
Review Comment:
Ah! I thought it was a preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1995113962
##########
airflow/api_fastapi/execution_api/datamodels/taskinstance.py:
##########
@@ -267,6 +267,9 @@ class TIRunContext(BaseModel):
Can either be a "decorated" dict, or a string encrypted with the shared Fernet key.
"""
+ xcom_keys_to_clear: Annotated[list[str], Field(default_factory=list)]
+ """List of Xcom keys that need to be cleared."""
Review Comment:
```suggestion
"""List of Xcom keys that need to be cleared and purged on by the worker."""
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1995060855
##########
task-sdk/src/airflow/sdk/execution_time/__init__.py:
##########
Review Comment:
Nope, we dotn need it, instead moved it to where its needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991189781
##########
airflow/api_fastapi/execution_api/routes/xcoms.py:
##########
@@ -224,16 +225,39 @@ def set_xcom(
# (the mapped task would fail in a moment as it can't be expanded anyway.)
# We use `BaseXCom.set` to set XComs directly to the database, bypassing the XCom Backend.
+ from airflow.models.dagrun import DagRun
+
+ if not run_id:
+ raise HTTPException(status.HTTP_404_NOT_FOUND, f"Run with ID: `{run_id}` was not found")
+
+ dag_run_id = session.query(DagRun.id).filter_by(dag_id=dag_id, run_id=run_id).scalar()
+ if dag_run_id is None:
+ raise HTTPException(status.HTTP_404_NOT_FOUND, f"DAG run not found on DAG {dag_id} with ID {run_id}")
+
+ # Remove duplicate XComs and insert a new one.
+ session.execute(
+ delete(XComModel).where(
+ XComModel.key == key,
+ XComModel.run_id == run_id,
+ XComModel.task_id == task_id,
+ XComModel.dag_id == dag_id,
+ XComModel.map_index == map_index,
+ )
+ )
+
try:
- BaseXCom.set(
+ # We expect serialised value from the caller - sdk, do not serialise in here
+ new = cast(Any, XComModel)( # Work around Mypy complaining model not defining '__init__'.
Review Comment:
```suggestion
new = XComModel(...)
new = cast(Any, XComModel) # Work around Mypy complaining model not defining '__init__'.
```
nit: I find that clearer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991197915
##########
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py:
##########
@@ -57,6 +56,11 @@
from tests_common.test_utils import db
from tests_common.test_utils.version_compat import AIRFLOW_V_3_0_PLUS
+if AIRFLOW_V_3_0_PLUS:
+ from airflow.sdk.execution_time.xcom import XCom
+else:
+ from airflow.models import XCom # type: ignore[no-redef]
Review Comment:
```suggestion
from airflow.models.xcom import XCom # type: ignore[no-redef]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998076398
##########
airflow/api_fastapi/execution_api/datamodels/taskinstance.py:
##########
@@ -267,6 +267,9 @@ class TIRunContext(BaseModel):
Can either be a "decorated" dict, or a string encrypted with the shared Fernet key.
"""
+ xcom_keys_to_clear: Annotated[list[str], Field(default_factory=list)]
+ """List of Xcom keys that need to be cleared."""
Review Comment:
Cool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1998078782
##########
airflow/models/__init__.py:
##########
@@ -135,6 +135,6 @@ def __getattr__(name):
from airflow.models.taskreschedule import TaskReschedule
from airflow.models.trigger import Trigger
from airflow.models.variable import Variable
- from airflow.models.xcom import XCom
+ from airflow.models.xcom import XComModel
Review Comment:
Added it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #47339:
URL: https://github.com/apache/airflow/pull/47339#issuecomment-2729916255
Thanks @ashb and @pierrejeambrun for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1991127607
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -578,6 +541,19 @@ def run(
ti: RuntimeTaskInstance, log: Logger
) -> tuple[IntermediateTIState | TerminalTIState, ToSupervisor | None, BaseException | None]:
"""Run the task in this process."""
+ # First, clear the xcom data
+ if ti._ti_context_from_server and ti._ti_context_from_server.xcom_keys_to_clear:
+ keys_to_delete = ti._ti_context_from_server.xcom_keys_to_clear
Review Comment:
Handled in 4b183084fc2979a4cb63bd60b74989fefc2e61cd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1978911862
##########
task_sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -357,6 +369,15 @@ def _xcom_push(ti: RuntimeTaskInstance, key: str, value: Any, mapped_length: int
# consumers
from airflow.serialization.serde import serialize
+ if XCom:
+ XCom.set(
+ key=key,
+ value=value,
+ dag_id=ti.dag_id,
+ task_id=ti.task_id,
+ run_id=ti.run_id,
+ )
+ return
Review Comment:
How are xcom backends meant to store their reference (I.e. an s3 path) without db access?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Handle Custom XCom Backend on Task SDK [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #47339:
URL: https://github.com/apache/airflow/pull/47339#discussion_r1987134116
##########
airflow/api_fastapi/execution_api/routes/task_instances.py:
##########
@@ -186,18 +186,18 @@ def ti_run(
if not dr:
raise ValueError(f"DagRun with dag_id={ti.dag_id} and run_id={ti.run_id} not found.")
- # Clear XCom data for the task instance since we are certain it is executing
+ # Send the XCom data for clearance for the task instance since we are certain it is executing at this point.
# However, do not clear it for deferral
if not ti.next_method:
map_index = None if ti.map_index < 0 else ti.map_index
log.info("Clearing xcom data for task id: %s", ti_id_str)
Review Comment:
This log isn't accurate anymore - maybe not worth keeping since we aren't doing anything destructive/of note here.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org