You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/06 17:15:48 UTC
[GitHub] [airflow] uranusjr opened a new pull request, #22792: Serialize mapped operator expansion kwargs logic
uranusjr opened a new pull request, #22792:
URL: https://github.com/apache/airflow/pull/22792
This is needed because we need to be able to access the correct expansion kwargs from the serialized mapped operator in the scheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #22792: Serialize mapped operator expansion kwargs logic
Posted by GitBox <gi...@apache.org>.
ashb merged PR #22792:
URL: https://github.com/apache/airflow/pull/22792
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22792: Serialize mapped operator expansion kwargs logic
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #22792:
URL: https://github.com/apache/airflow/pull/22792#discussion_r844306246
##########
airflow/decorators/base.py:
##########
@@ -357,6 +357,10 @@ def expand(self, **map_kwargs: "Mappable") -> XComArg:
multiple_outputs=self.multiple_outputs,
python_callable=self.function,
mapped_op_kwargs=map_kwargs,
+ # Different from classic operators, kwargs passed to a taskflow
+ # task's expand() contribute to the op_kwargs operator argument, not
+ # the operator arguments themselves, and should expand against it.
+ expansion_kwargs_attr="mapped_op_kwargs",
Review Comment:
Oh no, not and have it be de-serialized (easily)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22792: Serialize mapped operator expansion kwargs logic
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22792:
URL: https://github.com/apache/airflow/pull/22792#issuecomment-1090594197
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22792: Serialize mapped operator expansion kwargs logic
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #22792:
URL: https://github.com/apache/airflow/pull/22792#discussion_r844305754
##########
airflow/decorators/base.py:
##########
@@ -357,6 +357,10 @@ def expand(self, **map_kwargs: "Mappable") -> XComArg:
multiple_outputs=self.multiple_outputs,
python_callable=self.function,
mapped_op_kwargs=map_kwargs,
+ # Different from classic operators, kwargs passed to a taskflow
+ # task's expand() contribute to the op_kwargs operator argument, not
+ # the operator arguments themselves, and should expand against it.
+ expansion_kwargs_attr="mapped_op_kwargs",
Review Comment:
nit: Couldn't this be a class variable rather than an attribute?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22792: Serialize mapped operator expansion kwargs logic
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #22792:
URL: https://github.com/apache/airflow/pull/22792#issuecomment-1090705562
I'll add a test that covers/fails without this in https://github.com/apache/airflow/pull/22679 so merging now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org