You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "cherrera20 (via GitHub)" <gi...@apache.org> on 2024/05/07 13:56:53 UTC
[I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
cherrera20 opened a new issue, #39463:
URL: https://github.com/apache/airflow/issues/39463
### Description
I'm proposing to add a new ExternalAPITaskSensor to the Apache Airflow project. This sensor leverages the Airflow REST API to monitor the status of an external DAG or task. It can be used to check the completion of another DAG or task before proceeding with the execution of the current task, providing better orchestration across DAGs.
### Use case/motivation
The new sensor provides the ability to monitor the status of external DAGs using the Airflow REST API, which is particularly useful in a distributed environment where multiple Airflow instances are running.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
Posted by "cherrera20 (via GitHub)" <gi...@apache.org>.
cherrera20 commented on issue #39463:
URL: https://github.com/apache/airflow/issues/39463#issuecomment-2101960623
OK thanks. Just to illustrate how the sensor would look like I leave you a code snippet of the hypothetical call to the sensor. If it makes sense to you we can move on :-)
`
wait_for_single_task = ExternalAPITaskSensor(
task_id="wait_for_single_task",
external_dag_id="secondary_dag",
external_task_id="task_a",
allowed_states=["success"],
failed_states=["failed"],
skipped_states=["skipped"],
http_conn_id="http_default",
execution_date="{{ execution_date }}",
deferrable=True
)
wait_for_multiple_tasks = ExternalAPITaskSensor(
task_id="wait_for_multiple_tasks",
external_dag_id="secondary_dag",
external_task_ids=["task_b", "task_c"],
allowed_states=["success"],
failed_states=["failed"],
skipped_states=["skipped"],
http_conn_id="http_default",
execution_date="{{ execution_date }}",
deferrable=True
)
wait_for_task_group = ExternalAPITaskSensor(
task_id="wait_for_task_group",
external_dag_id="secondary_dag",
external_task_group_id="group_1",
allowed_states=["success"],
failed_states=["failed"],
skipped_states=["skipped"],
http_conn_id="http_default",
execution_date="{{ execution_date }}",
deferrable=True
)
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39463:
URL: https://github.com/apache/airflow/issues/39463#issuecomment-2099403546
Could you use the [HTTP Sensor](https://airflow.apache.org/docs/apache-airflow-providers-http/stable/_api/airflow/providers/http/sensors/http/index.html)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
Posted by "cherrera20 (via GitHub)" <gi...@apache.org>.
cherrera20 commented on issue #39463:
URL: https://github.com/apache/airflow/issues/39463#issuecomment-2099995962
Hi there! Thanks for your response.
The idea behind the **ExternalAPITaskSensor** is to address specific needs that go beyond what the **HttpSensor** can offer. While the **HttpSensor** provides a basic mechanism to poll an API endpoint, it doesn't really grasp the concept of external tasks and DAGs in the Airflow world.
The **ExternalAPITaskSensor** would retain the context of external tasks and DAGs, providing a seamless way to monitor specific DAG runs or tasks in another Airflow instance via the API. It would leverage attributes like **external_dag_id**, **execution_date**, **allowed_states**, **skipped_states**, and **failed_states**, which are unique to external DAGs and tasks.
Here's why I think this sensor is necessary:
- **State Awareness and Exception Management**: It would handle different states (like success, skipped, and failed) and raise exceptions (e.g., **AirflowFailException**, **AirflowSkipException**) when a specific state is reached. This ensures that task dependencies are managed correctly.
- **Advanced Error Handling**: It would differentiate between transient network errors (retries with a threshold) and API-related issues, providing more robust error handling.
- **Deferrable** Execution: The sensor would be able to defer execution to a custom WorkflowTrigger.
In summary, while the **HttpSensor** is great for basic API polling, it lacks the features needed to monitor external tasks and DAGs properly. The **ExternalAPITaskSensor** would encapsulate this logic, making it a valuable tool for workflows that rely on task dependencies across multiple DAGs or external Airflow instances.
I hope this explanation makes the purpose clear. Let me know if you have any questions or suggestions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
Posted by "cherrera20 (via GitHub)" <gi...@apache.org>.
cherrera20 commented on issue #39463:
URL: https://github.com/apache/airflow/issues/39463#issuecomment-2226892939
Ok, I could work on it. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #39463:
URL: https://github.com/apache/airflow/issues/39463#issuecomment-2098473560
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on issue #39463:
URL: https://github.com/apache/airflow/issues/39463#issuecomment-2226808512
@cherrera20 Would you still like to work on this issue?
If you do, I'd suggest that you'll create a proof of concept as a PR, and send a message in the dev list.
Then, it could be decided by the community whether to classify it as a core provider or as an internal airflow provider as Taragolis suggested. To make it easier, for starters - just create it as a core sensor, and it could be changed later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39463:
URL: https://github.com/apache/airflow/issues/39463#issuecomment-2138446154
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39463:
URL: https://github.com/apache/airflow/issues/39463#issuecomment-2100642010
> State Awareness and Exception Management: It would handle different states (like success, skipped, and failed) and raise exceptions (e.g., AirflowFailException, AirflowSkipException) when a specific state is reached. This ensures that task dependencies are managed correctly.
Can't that be achieved with the `HTTPSensor`'s `response_check` parameter? This allows you to pass a function that evaluates the response to the `HTTPSensor` for some condition.
So, if you query the `taskInstances` [endpoint of the Airflow REST API ](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_task_instance), you can configure your `response_check` accordingly.
You could then use existing [TI states](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#task-instances) to determine how to respond to a given state.
> Advanced Error Handling: It would differentiate between transient network errors (retries with a threshold) and API-related issues, providing more robust error handling.
I believe this can also be done both with `HTTPSensor` parameters (for example `tcp_keep_alive_count `) and normal Airflow retries.
> Deferrable Execution: The sensor would be able to defer execution to a custom WorkflowTrigger.
The `HTTPSensor` already supports deferrable execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #39463:
URL: https://github.com/apache/airflow/issues/39463#issuecomment-2101749986
IMHO, technically it could be added as a separate Provider with access through [apache-airflow-client](https://pypi.org/project/apache-airflow-client/), but as every new provider it should comes comes thought [Accepting new community providers](https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers) process. So I would recommend to start a discussion in [Dev List](https://airflow.apache.org/community/)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org