You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vatsrahul1001 (via GitHub)" <gi...@apache.org> on 2025/02/06 11:01:07 UTC
[I] Issues while reading Xcom values [airflow]
vatsrahul1001 opened a new issue, #46513:
URL: https://github.com/apache/airflow/issues/46513
### Apache Airflow version
3.0.0a1
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I see the below issue while reading Xcom
1. `ti.xcom_pull(key=key)` used to work without task_id in AF2, however failing in AF3
**AF2**
<img width="1353" alt="Image" src="https://github.com/user-attachments/assets/9dc08a59-3b83-4bbf-a71c-c3b5c779f025" />
**AF3**
<img width="1462" alt="Image" src="https://github.com/user-attachments/assets/d7ea1fdd-5b07-4ac8-87d3-63fa728def86" />
1. xcom values are showing in ""
**AF3**
**AF2**
<img width="1048" alt="Image" src="https://github.com/user-attachments/assets/3e764713-2d9b-4d72-b6f5-719ecde95ef0" />
### What you think should happen instead?
There is be no deviations in retriving xcom from AF2
### How to reproduce
1. Execute below DAG
2. Verify Xcom values are with qoutes.
3. `val = ti.xcom_pull(key=key)` in **wait_pull** method was able to get the xcom value with only using key, however, in AF3 is not able to see . See xcom value of seperate.y task
```
from airflow.decorators import dag, task, task_group
from airflow.models.baseoperator import chain, cross_downstream
from time import sleep
from pendulum import today
def wait_pull(**kwargs):
ti = kwargs["ti"]
key = ti.task_id.split(".")[-2]
val = None
while val is None:
val = ti.xcom_pull(key=key)
if not val:
print("not yet")
sleep(5)
print(f"got {val}")
return val
def push(val, **kwargs):
ti = kwargs["ti"]
key = ti.task_id.split(".")[-2]
ti.xcom_push(key=key, value=val)
@task
def x(**kwargs):
push("X", **kwargs)
return "x"
@task
def y(a, **kwargs):
b = wait_pull(**kwargs)
push(f"{b}Y", **kwargs)
return f"{a}y"
@task
def z(a, **kwargs):
b = wait_pull(**kwargs)
push(f"{b}Z", **kwargs)
return f"{a}z"
@task
def get(**kwargs):
return wait_pull(**kwargs)
@task
def cat(a, b):
if not a:
return b
if not b:
return a
return a + b
@task
def assert_one_of(to):
@task
def approved(val):
print(val)
assert int(val) in to
return approved
@task_group()
def separate():
a = x()
y(a)
c = z(a)
d = get()
return cat(c, d)
@task_group()
def chained_one_way():
a = x()
b = y(a)
c = z(a)
d = get()
chain(a, b, c, d)
return cat(c, d)
@task_group()
def chained_another_way():
a = x()
b = y(a)
c = z(a)
d = get()
b >> d # to remove race condition
chain(a, c, b, d)
return cat(c, d)
@task_group()
def crossed_one_way():
a = x()
b = y(a)
c = z(a)
d = get()
c >> d # to remove race condition
cross_downstream([a, b], [c, d])
return cat(c, d)
@task_group()
def crossed_another_way():
a = x()
b = y(a)
c = z(a)
d = get()
cross_downstream([a, c], [b, d])
return cat(c, d)
@task
def assert_equal(a, b):
print(f"{a} == {b}")
assert a == b
@dag(
schedule=None,
start_date=today('UTC').add(days=-1),
default_args={"owner": "airflow"},
max_active_runs=1,
catchup=False,
tags=["core"],
)
def cross_chain_structure_hash():
"""
The patterns below aren't supposed to be meaningful, it's enough that a
change in the underlying functionality will fail this test and attract
human attention.
"""
assert_equal("xzX", separate())
assert_equal("xzXYZ", chained_one_way())
assert_equal("xzXZY", chained_another_way())
assert_equal("xzXYZ", crossed_one_way())
assert_equal("xzXZ", crossed_another_way())
the_dag = cross_chain_structure_hash()
```
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2695229964
@vatsrahul1001 I think there were 2 problems in this issue:
1. We doubly serialised and stored in Db, leading to the UI showing quotes "xcoms"
2. The issue with `ti.xcom_pull`.
1 was solved by the PR I worked on. Can we instead create a new issue for 2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2813194482
To clarify on the folks looking at this issue: The Airflow 3 behavior is correct.
Here is why:
In Airflow 2, `xcom_pull()` allowed pulling by just a key without specifying `task_ids`, even though the underlying DB model defines `task_id` as part of the XCom primary key.
https://github.com/apache/airflow/blob/2.10.5/airflow/models/xcom.py#L82-L85
```python
dag_run_id = Column(Integer(), nullable=False, primary_key=True)
task_id = Column(String(ID_LEN, **COLLATION_ARGS), nullable=False, primary_key=True)
map_index = Column(Integer, primary_key=True, nullable=False, server_default=text("-1"))
key = Column(String(512, **COLLATION_ARGS), nullable=False, primary_key=True)
```
This leads to ambiguous and unstable behavior — for example, if two tasks push XComs with the same key (`config_blob`), `xcom_pull(key="config_blob")` would return whichever happened to come first in the query, making the result unpredictable and order-dependent. Airflow 3 addresses this inconsistency by requiring task_ids when pulling by key to reflect the task-scoped nature of XComs as enforced by the schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv closed issue #46513: Issues while reading Xcom values
URL: https://github.com/apache/airflow/issues/46513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2697981107
@vatsrahul1001 @amoghrajesh
In https://github.com/apache/airflow/pull/46728
I have a small fix related to None in `task_ids`, I can extract it.
For this issue or maybe for a separate one, if it's more reasonable to separate the two problems mentioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2768436558
@amoghrajesh as per @insomnes ti.xcom_pull(key=key) without taskid wouldn't be possible inAF3 right? We can close this issue in that case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed issue #46513: Issues while reading Xcom values
URL: https://github.com/apache/airflow/issues/46513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "atul-astronomer (via GitHub)" <gi...@apache.org>.
atul-astronomer commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2743196518
Still getting error, checking on main latest
"XCom with key='separate' map_index=-1 not found for task 'separate.z' in DAG run 'manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e' of 'cross_chain_structure_hash'"}} key=separate map_index=-1 run_id=manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e status_code=404 task_id=separate.z
2025-03-21 12:12:58.743651 [warning ] Server error [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "XCom with key='separate' map_index=-1 not found for task 'separate.y' in DAG run 'manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e' of 'cross_chain_structure_hash'"}}
2025-03-21 12:12:58.743803 [error ] XCom not found [airflow.sdk.api.client] dag_id=cross_chain_structure_hash detail={'detail': {'reason': 'not_found', 'message': "XCom with key='separate' map_index=-1 not found for task 'separate.y' in DAG run 'manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e' of 'cross_chain_structure_hash'"}} key=separate map_index=-1 run_id=manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e status_code=404 task_id=separate.y
2025-03-21 12:12:58.743855 [warning ] Server error [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "XCom with key='chained_one_way' map_index=-1 not found for task 'chained_one_way.y' in DAG run 'manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e' of 'cross_chain_structure_hash'"}}
2025-03-21 12:12:58.743940 [warning ] Server error [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "XCom with key='crossed_one_way' map_index=-1 not found for task 'crossed_one_way.y' in DAG run 'manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e' of 'cross_chain_structure_hash'"}}
2025-03-21 12:12:58.744077 [error ] XCom not found [airflow.sdk.api.client] dag_id=cross_chain_structure_hash detail={'detail': {'reason': 'not_found', 'message': "XCom with key='chained_one_way' map_index=-1 not found for task 'chained_one_way.y' in DAG run 'manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e' of 'cross_chain_structure_hash'"}} key=chained_one_way map_index=-1 run_id=manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e status_code=404 task_id=chained_one_way.y
2025-03-21 12:12:58.744099 [error ] XCom not found [airflow.sdk.api.client] dag_id=cross_chain_structure_hash detail={'detail': {'reason': 'not_found', 'message': "XCom with key='crossed_one_way' map_index=-1 not found for task 'crossed_one_way.y' in DAG run 'manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e' of 'cross_chain_structure_hash'"}} key=crossed_one_way map_index=-1 run_id=manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e status_code=404 task_id=crossed_one_way.y
2025-03-21 12:12:58.745186 [warning ] Server error [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "XCom with key='chained_another_way' map_index=-1 not found for task 'chained_another_way.z' in DAG run 'manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e' of 'cross_chain_structure_hash'"}}
2025-03-21 12:12:58.745329 [error ] XCom not found [airflow.sdk.api.client] dag_id=cross_chain_structure_hash detail={'detail': {'reason': 'not_found', 'message': "XCom with key='chained_another_way' map_index=-1 not found for task 'chained_another_way.z' in DAG run 'manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e' of 'cross_chain_structure_hash'"}} key=chained_another_way map_index=-1 run_id=manual__2025-03-21T12:12:05.852885+00:00_EwhhU50e status_code=4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "atul-astronomer (via GitHub)" <gi...@apache.org>.
atul-astronomer commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2812026867
Any update here?
@amoghrajesh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2694792510
@amoghrajesh` ti.xcom_pull(key=key) used to work without task_id in AF2, however failing in AF3` still persists
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2701769911
Created new PR for it:
https://github.com/apache/airflow/pull/47407
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2658488318
@amoghrajesh I think, I detected and fixed no task_id provided case when I was expanding the tests for multiple map_indexes in:
https://github.com/apache/airflow/pull/46728
But I would be happy to discard it in favor of your PR if you have your own plans on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2812059851
@vatsrahul1001 / @atul-astronomer / @phanikumv / @insomnes apologies for the late reply. This issue got lost in the sea of different issues.
As i read this conversation, i do not see a real need to even support such a case when task_id is not provided and as per current logic, i do not think it is doable as of now too. We should revisit this if there is a complaint of someone from the community as i anticipate this to be a non issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2788369681
@amoghrajesh can you please comment on the above note from @vatsrahul1001 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2639655214
I will take a look at this tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2662248990
Just merged https://github.com/apache/airflow/pull/46719, @vatsrahul1001 would you be able to test this again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2812074981
I am sorry that my patch led to such a misunderstanding. I should have dug deeper into the subject before suggesting my solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "atul-astronomer (via GitHub)" <gi...@apache.org>.
atul-astronomer commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2812076119
Thanks @amoghrajesh for the update, we can close the issue. I don't have the permissions.
@phanikumv @vatsrahul1001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "insomnes (via GitHub)" <gi...@apache.org>.
insomnes commented on issue #46513:
URL: https://github.com/apache/airflow/issues/46513#issuecomment-2743587778
I think I understand now. I based my fix suggestion based on the previous code:
```python
if task_ids is None:
# default to the current task if not provided
task_ids = self.task_id
```
But it's not what this DAG is expecting. It expects no filter for task_id in case of `None` value and creation of `LazyXComSelectSequence` from database select. That is the old XCOM behavior.
This kind of request is not supported through execution time API for now, because there are only handlers for specific task ids.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Issues while reading Xcom values [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed issue #46513: Issues while reading Xcom values
URL: https://github.com/apache/airflow/issues/46513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org