You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Alwaysgaurav1 (via GitHub)" <gi...@apache.org> on 2026/07/06 05:50:10 UTC
[PR] feat(task-sdk): Move XCom serialization and deserialization to Task SDK [airflow]
Alwaysgaurav1 opened a new pull request, #69432:
URL: https://github.com/apache/airflow/pull/69432
### PR Title
feat(task-sdk): Move XCom serialization and deserialization to Task SDK
### PR Description
#### What does this PR do?
This PR moves the responsibility of XCom serialization and deserialization from the Airflow API Server to the language-specific client-side **Task SDK** as part of **AIP-72 (Task Execution Interface)**.
Under this new architecture:
1. The **Task SDK** client-side (Python/Go/Java) handles serializing native Python objects directly into raw JSON-formatted string literals before sending them to the API Server via `SetXCom`.
2. The **API Server** now treats XCom values as opaque JSON string literals, saving them directly to the database without parsing or re-serializing them.
3. Upon retrieval, the **API Server** returns the raw JSON string. The **Task SDK** parses and deserializes it back to native objects.
4. For backward compatibility, the API Server will automatically convert legacy database objects (stored as dicts/lists/primitives) into JSON strings if they are not already stored as JSON string literals.
#### Why is it needed?
- **Decoupled Languages**: Decouples language-specific serialization logic from the API Server, allowing non-Python SDKs (Go, Java, TypeScript) to use the same standardized JSON-formatted string contract.
- **Performance**: Eliminates "triple serialization" overhead (Task SDK serialization -> API Server `XCom.serialize` -> SQLAlchemy JSON column type serialization).
- **Bug Fix**: Resolves a bug where traditional components pulling XCom values set by Task Execution tasks retrieved raw serialized dictionary structures (e.g., `{"__classname__": "builtins.tuple", ...}`) instead of fully deserialized Python objects (e.g., tuples). By storing them as JSON string literals, `XComModel.deserialize_value` can now parse and deserialize them correctly.
#### Changes Made
- **Task SDK Client**:
- `BaseXCom.serialize_value`: Modified to fully JSON-serialize values using `json.dumps(serialize(value))`.
- `BaseXCom.deserialize_value`: Modified to load raw JSON string values before calling the `deserialize` utility.
- `_generated.py`: Updated datamodel types (`XComResponse`, `XComSequenceIndexResponse`, `XComSequenceSliceResponse`) to use `str | None` for values.
- **Airflow Core API**:
- `api_fastapi/execution_api/datamodels/xcom.py`: Updated types to use `str | None`.
- `api_fastapi/execution_api/routes/xcoms.py`: Updated GET and POST routes to support JSON-serialized string handling and preserve backward compatibility for legacy non-string records.
- `test_xcoms.py` & `test_xcom.py`: Updated tests to assert correct serialization outputs and verify database-to-native object roundtrip validation.
#### Related Issue
- Closes #45231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(task-sdk): Move XCom serialization and deserialization to Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed pull request #69432: feat(task-sdk): Move XCom serialization and deserialization to Task SDK
URL: https://github.com/apache/airflow/pull/69432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(task-sdk): Move XCom serialization and deserialization to Task SDK [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #69432:
URL: https://github.com/apache/airflow/pull/69432#issuecomment-4924040777
Please review code thoroughly before raising them @Alwaysgaurav1, you are raising a PR referencing a closed issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org