You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/11 14:49:06 UTC
[GitHub] [airflow] sahil-b-shah opened a new issue #10883: Improve ExternalTaskSensor execution date pattern
sahil-b-shah opened a new issue #10883:
URL: https://github.com/apache/airflow/issues/10883
Currently in the `ExternalTaskSensor` there are two ways two set the execution date of a DAG/task the sensor is waiting on:
- Provide the `execution_delta` which simply adds/subtracts a time value to/from the `execution_date` of the sensor itself
- Provide the `execution_date_fn` which is a function that takes in just the `execution_date` and returns a manipulated one.
I believe that the correct way to go about handling the `execution_date` for the `ExternalTaskSensor` should be analogous to the way we handle it for other operators, such as the `ExternalTaskMarker`, or more relevantly, the `TriggerDagRunOperator`, which is an operator usually used in conjunction with the `ExternalTaskSensor`.
Specifically, we should remove the `execution_delta` parameter and replace it with a templated `execution_date` parameter. This not only covers the existing `execution_delta` use case through e.g. `{{ execution_date - macros.timedelta(days=1) }}`, but also provides more flexibility by enabling the benefits of templated parameters in general, such as pulling from xcoms, reading from DAG/operator params, using macros, etc. It also establishes the pattern that we see in the previously mentioned operators.
The proposed solution is also sufficient enough to remove the `execution_date_fn` because we can register custom macros to manipulate the execution date the same way as an `execution_date_fn` would. But I do realize we may want to keep it for backwards compatibility reasons.
First time contributor here so happy to implement/help with this to get some airflow OSS chops.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10883: Improve ExternalTaskSensor execution date pattern
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10883:
URL: https://github.com/apache/airflow/issues/10883#issuecomment-691140568
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10883: Improve ExternalTaskSensor execution date pattern
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10883:
URL: https://github.com/apache/airflow/issues/10883#issuecomment-691140568
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10883: Improve ExternalTaskSensor execution date pattern
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10883:
URL: https://github.com/apache/airflow/issues/10883#issuecomment-691140568
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10883: Improve ExternalTaskSensor execution date pattern
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10883:
URL: https://github.com/apache/airflow/issues/10883#issuecomment-691140568
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org