You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "sdhilip200 (via GitHub)" <gi...@apache.org> on 2026/06/21 02:27:31 UTC
[PR] Fix XCom get_value outside task runner [airflow]
sdhilip200 opened a new pull request, #68789:
URL: https://github.com/apache/airflow/pull/68789
When `BaseXCom.get_value()` is called outside a task-runner context, such as from plugin extra links resolved by the API server, `BaseXCom.get_one()` attempted to import `SUPERVISOR_COMMS` and raised `ImportError` before it could read the stored XCom value.\n\nThis changes sync `BaseXCom.get_one()` to keep using supervisor comms when they are available, and to fall back to reading the matching XCom row from the metadata database when no supervisor comms are present. The fallback uses `XComModel.deserialize_value()` so DB-backed values are decoded consistently with the model path.\n\ncloses: #59093\n\nTests:\n- `uv run pytest airflow-core/tests/unit/models/test_xcom.py::TestXComGet::test_basexcom_get_value_reads_from_db_without_supervisor_comms`\n- `uv run pytest airflow-core/tests/unit/models/test_xcom.py::TestXComGet`\n- `uv run pytest task-sdk/tests/task_sdk/bases/test_xcom.py`\n- `uv run pytest airflow-core/tests/unit/models/test_xcom.py`\n- `uv run ruff check task-sdk/src/airf
low/sdk/bases/xcom.py airflow-core/tests/unit/models/test_xcom.py`\n- `uv run ruff format --check task-sdk/src/airflow/sdk/bases/xcom.py airflow-core/tests/unit/models/test_xcom.py`\n\n---\n\n##### Was generative AI tooling used to co-author this PR?\n\n- [X] Yes: OpenAI Codex\n\nGenerated-by: OpenAI Codex following the guidelines at https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-4764148546
This gives `sdk.XComModel.get_value()` different semantics between Airflow 2 and 3. On Airflow 2 the compat fallback resolves through `XCom.get_value()` and the configured XCom backend, but this Airflow 3 helper always calls `XComModel.deserialize_value()`. With object-storage/custom XCom backends, extra-link code can receive the stored reference/path instead of the actual value. Please either route deserialization through the configured backend for this server-side helper, or make the raw-DB semantics explicit and add a regression test with `XComObjectStorageBackend`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "sdhilip200 (via GitHub)" <gi...@apache.org>.
sdhilip200 commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-4808525525
Updated in 990456d to add the `# use next version` marker for `common.io`'s `common-compat` dependency.
This addresses the `Build info` selective-check failure:
```text
common.compat provider changed but the following providers don't have '# use next version'
- common.io (providers/common/io/pyproject.toml)
```
Verified locally with:
```bash
uv run breeze ci selective-check --commit-ref HEAD --pr-labels "['area:task-sdk']" --github-event-name pull_request --github-repository apache/airflow --github-actor sdhilip200
```
The local check now reports:
```text
common-compat-changed-without-next-version=false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-5942944184
Does this need new API on `XComModel`? On #59093 the problem was using the Task SDK `XCom` in API-server code, and the existing `XComModel.get_many(...)` covers reading XComs there. `get_one` also calls `deserialize_value` on the result, which `operatorlink.py` deliberately avoids in the same API-server context because it "could instantiate arbitrary classes from untrusted XCom data". Exporting the server ORM model through `common.compat.sdk` mixes server and task imports in one module. Is there a case `get_many` doesn't cover? Converting to draft until there is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-5533888989
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "sdhilip200 (via GitHub)" <gi...@apache.org>.
sdhilip200 commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-4918807623
Updated in 7ac3b146 to make the XComObjectStorageBackend regression run only when the installed Airflow exposes the new server-side XComModel helpers added by this PR. This keeps the object-storage backend regression active for the current implementation while preventing provider-compat jobs for released Airflow 3 wheels from failing on missing get_value or set(..., serialize=False) support. Verified locally with uv run pytest --rootdir=providers/common/io providers/common/io/tests/unit/common/io/xcom/test_backend.py -k xcommodel_get_value_uses_configured_backend. Also verified uv run ruff check providers/common/io/tests/unit/common/io/xcom/test_backend.py, uv run ruff format --check providers/common/io/tests/unit/common/io/xcom/test_backend.py, git diff --check, and uv run scripts/ci/prek/update_providers_dependencies.py with no generated-file diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-4760672887
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "sdhilip200 (via GitHub)" <gi...@apache.org>.
sdhilip200 commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-4985866583
Updated in d8b1907 to fix the static-check failure from check-airflow-v-imports-in-tests by importing AIRFLOW_V_3_0_PLUS from tests_common.test_utils.version_compat at module scope in the common.compat SDK test. Verified locally: python3 -m py_compile providers/common/compat/tests/unit/common/compat/test_sdk.py; uv run ruff check providers/common/compat/tests/unit/common/compat/test_sdk.py; uv run ruff format --check providers/common/compat/tests/unit/common/compat/test_sdk.py; uv run pytest --rootdir=providers/common/compat providers/common/compat/tests/unit/common/compat/test_sdk.py; git diff --check. I also inspected the provider-distribution compat failures in the same run; they fail while collecting providers/microsoft/azure/tests/unit/microsoft/azure/hooks/test_data_factory.py because FactoryListResponse is missing from azure.mgmt.datafactory.models, which is outside this PR's changed files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "justinpakzad (via GitHub)" <gi...@apache.org>.
justinpakzad commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-4773146445
> Updated the PR in [cee3476](https://github.com/apache/airflow/commit/cee34764a366b170f5dd415fff4aa68532d1c896) to avoid DB access from task-sdk. The revised patch restores BaseXCom to the supervisor-comms-only path, adds server-side XComModel.get_one()/get_value() helpers, exposes XComModel from the common compat SDK for plugin extra links, and adds /links route coverage for a plugin-registered extra link reading XCom through XComModel.
The new changes avoid the task-sdk db access issue, but I'm not sure exposing XComModel through the compat sdk is the correct approach either as it's mixing server and task imports in the same module. That being said, I'm not an expert here so I will defer to a maintainer.
Just curious, did you take a look at the proposed solution on the issue? It avoids the need for any of these changes by having the plugin use XComModel and create_session directly from core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "sdhilip200 (via GitHub)" <gi...@apache.org>.
sdhilip200 commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-4763419825
Updated the PR in cee3476 to avoid DB access from task-sdk. The revised patch restores BaseXCom to the supervisor-comms-only path, adds server-side XComModel.get_one()/get_value() helpers, exposes XComModel from the common compat SDK for plugin extra links, and adds /links route coverage for a plugin-registered extra link reading XCom through XComModel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "justinpakzad (via GitHub)" <gi...@apache.org>.
justinpakzad commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-4762664586
Hi, I don't think this is the right approach. The task-sdk should not be accessing the database directly. I would take a look at the docs here for a bit more detail on database access restrictions:
https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html#database-access-restrictions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix XCom get_value outside task runner [airflow]
Posted by "sdhilip200 (via GitHub)" <gi...@apache.org>.
sdhilip200 commented on PR #68789:
URL: https://github.com/apache/airflow/pull/68789#issuecomment-4773090144
Updated in 5a6f988.
`XComModel.get_one()` / `get_value()` now keeps the DB lookup server-side in `airflow-core`, but routes deserialization through the configured custom XCom backend when one is configured. The default `BaseXCom` path still uses `XComModel.deserialize_value()` so existing JSON rows keep their current behavior.
I also added `XComObjectStorageBackend` regression coverage showing an object-storage-backed row returns the original value instead of the stored file path.
Verified locally on the rebased branch:
- `uv run pytest --rootdir=providers/common/io providers/common/io/tests/unit/common/io/xcom/test_backend.py::TestXComObjectStorageBackend`
- `uv run pytest airflow-core/tests/unit/models/test_xcom.py airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_extra_links.py task-sdk/tests/task_sdk/bases/test_xcom.py`
- `uv run pytest --rootdir=providers/common/compat providers/common/compat/tests/unit/common/compat/test_sdk.py`
- `uv run ruff check airflow-core/src/airflow/models/xcom.py airflow-core/tests/unit/models/test_xcom.py airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_extra_links.py providers/common/compat/src/airflow/providers/common/compat/sdk.py providers/common/compat/tests/unit/common/compat/test_sdk.py providers/common/io/tests/unit/common/io/xcom/test_backend.py`
- `uv run ruff format --check airflow-core/src/airflow/models/xcom.py airflow-core/tests/unit/models/test_xcom.py airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_extra_links.py providers/common/compat/src/airflow/providers/common/compat/sdk.py providers/common/compat/tests/unit/common/compat/test_sdk.py providers/common/io/tests/unit/common/io/xcom/test_backend.py`
- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org