[PR] [v3-4-test] Read XCom without object reconstruction during dependency evaluation (#72718) [airflow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] opened a new pull request, #75117:
URL: https://github.com/apache/airflow/pull/75117

   * Read XCom without object reconstruction during dependency evaluation
   
   The scheduler evaluates task dependencies in-process, and two of those deps read
   XCom: NotPreviouslySkippedDep reads skipmixin_key for every parent that inherits
   from SkipMixin, and both it and TriggerRuleDep read past_depends_met when
   wait_for_past_depends_before_skipping applies.
   
   Both went through TaskInstance.xcom_pull -> XComModel.deserialize_value, which is
   json.loads(value, cls=XComDecoder). XComDecoder's object_hook calls the Airflow
   serialization module with full=True, so any nested dict carrying __classname__ is
   imported and instantiated. The values these deps actually consume are primitives:
   a mapping of lists of task ids, and a boolean. Nothing here needs an object
   rebuilt, so the scheduler can read these values as plain JSON.
   
   Add XComModel.deserialize_value_primitives, a plain json.loads with no
   object_hook, and LazyXComPrimitivesSelectSequence alongside it. Thread full=True
   through TaskInstance.xcom_pull and pass full=False at the four dep call sites.
   All three of xcom_pull's return paths are covered -- the direct one, and both
   lazy-sequence ones, which is the path a mapped task instance takes. The skip
   decisions of SkipMixin tasks in the same mapped task group, which
   NotPreviouslySkippedDep reads with one query per pass, are decoded the same way.
   
   NotPreviouslySkippedDep also now checks that the skipmixin payload is a mapping
   before testing membership in it. It reads whatever the parent task wrote, so a
   scalar there previously raised TypeError inside dependency evaluation rather than
   being ignored. Because the payload is now plain JSON, a set or tuple of task ids
   arrives as its serde wrapper; its items are read from the wrapper without
   constructing anything, and a decision whose task ids are not a list of strings is
   ignored the same way.
   
   Task code pulling another task's return value is unchanged and still gets the
   original Python object; only the dependency-evaluation reads are narrowed.
   
   Generated-by: Claude Opus 5
   
   * Treat only a literal True as past-depends-met during dependency evaluation
   
   Generated-by: Claude Opus 5 following the guidelines at https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions
   
   * Ignore skip decisions with a non-string classname
   (cherry picked from commit a7ac9379cf9d0a443319dd740849c63fff95359c)
   
   Co-authored-by: Jarek Potiuk <ja...@potiuk.com>
   Generated-by: Claude Opus 5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-4-test] Read XCom without object reconstruction during dependency evaluation (#72718) [airflow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #75117:
URL: https://github.com/apache/airflow/pull/75117#issuecomment-6272385758

   Hi maintainer, this PR was merged without a milestone set.
   We've automatically set the milestone to **[Airflow 3.4.1](https://github.com/apache/airflow/milestone/154)** based on: merged to version branch
   If this milestone is not correct, please update it to the appropriate milestone.
   
   > This comment was generated by [Milestone Tag Assistant](https://github.com/apache/airflow/blob/main/.github/workflows/milestone-tag-assistant.yml).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-4-test] Read XCom without object reconstruction during dependency evaluation (#72718) [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk merged PR #75117:
URL: https://github.com/apache/airflow/pull/75117


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org