You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/09/08 15:16:32 UTC

[PR] Read XCom without object reconstruction during dependency evaluation [airflow]

potiuk opened a new pull request, #72718:
URL: https://github.com/apache/airflow/pull/72718

The scheduler evaluates task dependencies in-process, and two of those deps read XCom:

- `NotPreviouslySkippedDep` reads `skipmixin_key` for every parent that inherits from `SkipMixin` — any Branch or ShortCircuit operator, so this is a common path.
- Both it and `TriggerRuleDep` read `past_depends_met` when `wait_for_past_depends_before_skipping` applies.

Both went through `TaskInstance.xcom_pull` → `XComModel.deserialize_value`, which is `json.loads(value, cls=XComDecoder)`. `XComDecoder.object_hook` calls the serialization module with `full=True`, so a nested dict carrying `__classname__` gets imported and instantiated.

The values these deps actually consume are primitives — a mapping of lists of task ids, and a boolean. Nothing here needs an object rebuilt, and the scheduler is the wrong process to rebuild one in: the [security model](https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html) reserves it for code the Deployment Manager installed.

### Change

- `XComModel.deserialize_value_primitives` — plain `json.loads`, no `object_hook`. A `__classname__` envelope comes back as an inert `dict`.
- `LazyXComPrimitivesSelectSequence` alongside `LazyXComSelectSequence`, for the lazy path.
- `full: bool = True` threaded through `TaskInstance.xcom_pull`; `full=False` at the four dep call sites.

All three of `xcom_pull`'s return paths are covered — the direct one and **both** lazy-sequence ones. The lazy path is what a mapped task instance takes, so covering only the direct return would have left the hole open exactly there.

`NotPreviouslySkippedDep` also now checks the skipmixin payload is a mapping before testing membership. It reads whatever the parent task wrote, and a scalar there previously raised `TypeError` inside dependency evaluation instead of being ignored.

Task code pulling another task's return value is unchanged and still receives the original Python object. Only the scheduler-side dep reads are narrowed.

### Tests

Two regression tests, both verified to fail on unpatched sources and pass with the change:

- a serde envelope stored under `skipmixin_key` is read as an inert mapping rather than instantiated;
- a scalar under the same key is ignored rather than raising.

Both write through `XComModel.set(..., serialize=False)` with a pre-serialized string, which is what the Execution API does with a caller-supplied value — `serialize_value` would have rejected the payload, so testing through that door would have proved nothing.

Suites run: `test_not_previously_skipped_dep.py` (8), `test_trigger_rule_dep.py` + `models/test_xcom.py` (202), `test_taskinstance.py -k xcom` (19). All pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_015DKYM5PoVgKC5JqxLso8rn

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Read XCom without object reconstruction during dependency evaluation [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk commented on PR #72718:
URL: https://github.com/apache/airflow/pull/72718#issuecomment-5855524235

   @ashb @XD-DENG — gentle reminder: this needs a review to make the 3.4.0 cut on 2026-10-19. CI is green and it's ready from my side.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Read XCom without object reconstruction during dependency evaluation [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk merged PR #72718:
URL: https://github.com/apache/airflow/pull/72718


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Read XCom without object reconstruction during dependency evaluation [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk commented on PR #72718:
URL: https://github.com/apache/airflow/pull/72718#issuecomment-6207489805

   @ashb @kaxil Could you take a look at this when you have a moment? It's targeted at 3.4.0 and needs a review to make the cut on 2026-10-19. CI is green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Read XCom without object reconstruction during dependency evaluation [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #72718:
URL: https://github.com/apache/airflow/pull/72718#issuecomment-6270340937

   ### Backport successfully created: v3-4-test
   
   Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
   the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
   
   In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
   
   <table>
           <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>✅</td>
           <td>v3-4-test</td>
           <td><a href="https://github.com/apache/airflow/pull/75117"><img src="https://img.shields.io/badge/PR-75117-blue" alt="PR Link"></a></td>
       </tr>
   </table>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org