You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/27 01:20:51 UTC
[GitHub] [airflow] pwachira opened a new issue #21139: Websocket trigger for long-lived deferable operators
pwachira opened a new issue #21139:
URL: https://github.com/apache/airflow/issues/21139
### Description
Create a trigger for deferrable operators that uses a websocket client to send a long-lived/long-running task request to an external remote websocket server that handles (or intiates the handling of) the requested task.
The Async trigger then triggers when the remote server replies
The remote websocket server will be out of the scope of airflow and not implemented here.
### Use case/motivation
A common use case is to run long lived tasks asynchronously when the duration of execution is unknown.
Http requests are not the best for long lived connections so a http trigger may not be optimal.
The currently provided timer trigger does not meet this requirement
A websocket request would be better suited for this (I would even go further and propose a messaging system(e.g kafka) is the best option in such a case - I might open a different feature request if this current feature is approved)
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21139: Websocket trigger for long-lived deferable operators
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21139:
URL: https://github.com/apache/airflow/issues/21139#issuecomment-1022756095
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21139: Websocket trigger for long-lived deferable operators
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21139:
URL: https://github.com/apache/airflow/issues/21139#issuecomment-1023092746
Assigned you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #21139: Websocket trigger for long-lived deferable operators
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21139:
URL: https://github.com/apache/airflow/issues/21139#issuecomment-1023096254
And messageing system triggers is the next best thing after that one too, especially if we could make them generic for multiple messaging systems. I am not sure if there are good proven libraries that could provide the right abstraction - I'd think something like Kombu (which we already use through Celery) might be a good candidate but I do not have enough deep experience to make the right call here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21139: Websocket trigger for long-lived deferable operators
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21139:
URL: https://github.com/apache/airflow/issues/21139#issuecomment-1023092454
By all means - please add it :). That's definitely something that will make Defferable Operators easier to implement !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #21139: Websocket trigger for long-lived deferable operators
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21139:
URL: https://github.com/apache/airflow/issues/21139#issuecomment-1023092454
By all means - please add it :). That's definitely something that will make many Defferable Operators easier to implement !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21139: Websocket trigger for long-lived deferable operators
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21139:
URL: https://github.com/apache/airflow/issues/21139#issuecomment-1023096254
And messageing system triggers is the next best thing after that one too, especially if we could make them generic for multiple messaging systems. I am not sure if there are good proven libraries that could provide the right abstraction - I'd think something lik Kombu (which we already use through Celery) might be a good candidate but I do not have enough deep experience to make the right call here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org