You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2019/10/29 06:17:20 UTC
[GitHub] [airflow] dstandish commented on issue #6370: AIRFLOW-5701: Don't
clear xcom explicitly before execution
dstandish commented on issue #6370: AIRFLOW-5701: Don't clear xcom explicitly before execution
URL: https://github.com/apache/airflow/pull/6370#issuecomment-547273710
What motivated this PR was the need to persist state through reschedules, in the service of creating an async operator. I think we can narrow the scope of this change to meet this need without causing these undesired consequences.
Why not _only_ not clear at start when the task is resuming after a reschedule. There is a state `up_for_reschedule` which seems promising but it appears to be quite shortlived, i.e. it does not last until the task resumes.
Perhaps an async operator, or a sensor in `reschedule` mode should take on a "hibernate"-like state when waiting for reschedule time. Something not occupying worker slot, but still considered part of the same task instance and same attempt, so that idempotence is still respected. When merely being reanimated following a hibernate, xcom is not purged; when retrying after a failure, xcom is purged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services