[GitHub] [airflow] uranusjr commented on pull request #24838: Foundation for list-of-dicts task mapping

[GitBox <gi...@apache.org>]

uranusjr commented on PR #24838:
URL: https://github.com/apache/airflow/pull/24838#issuecomment-1177284127

   Some notes explaining what happened here and my plan moving this forward. The main difference between `expand(**kwargs)` and `expand_kwargs(xcom_arg)` is that we have no way to know what arguments will be passed until runtime, when the mapped task is actually being executed. This means we need very different behaviour on validation and calculating how many tis we need in the scheduler.
   
   To account for this difference in behaviour, I created one additional abstraction under `MappedOperator`. It is named `MappedKwargs` in this PR (I’m thinking about better names since that name implies a dict). This `MappedKwargs` type encapsulates logic between cross-product and zip behaviour.
   
   Another thing that needs to change is the order of operations we render templates in a mapped operator. Right now in main, we unmap first, and then resolve XCom references on the unmapped operator. But that won’t work with `expand_kwargs` since we _can’t_ unmap without accessing the rendered value, so I switched the behaviour around, rendering all XCom references first and create the unmapped operator directly with those rendered values. This is more difficult to implement than it sounds.
   
   I plan to continue implementing `expand_kwargs()` on this PR to ensure my refactorings all work (and modify them if not), and then split the result into separate PRs to submit so the changes are easier to review. This PR (and this comment) will be kept for reference if the reviewers need some broad context to understand why a particular PR needs to be done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org