You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/05 10:47:25 UTC
[GitHub] [airflow] lromor opened a new issue #14622: REST API: dag status stream.
lromor opened a new issue #14622:
URL: https://github.com/apache/airflow/issues/14622
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
It would be really nice to provide a streaming endpoint so that clients can receive a realtime status update of a running dag instance.
**Use case / motivation**
from a rest client, instead of polling for updates or creating intermediate task triggers I want to know the status of a running dag.
-->
**Are you willing to submit a PR?**
It depends by how complex (in terms of time) would be a potential implementation and how much we will invest on airflow in the future.
**Related Issues**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vemikhaylov commented on issue #14622: REST API: dag status stream.
Posted by GitBox <gi...@apache.org>.
vemikhaylov commented on issue #14622:
URL: https://github.com/apache/airflow/issues/14622#issuecomment-791410590
Should it be like a websockets solution or a client can be notified with web-hooks? Or would be great to have the both?
I'll be happy to implement such feature if it's necessity's confirmed by maintainers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #14622: REST API: dag status stream.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14622:
URL: https://github.com/apache/airflow/issues/14622#issuecomment-817715425
What do you think about webhook support? https://github.com/apache/airflow/issues/15322 Will this meet your business requirements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lromor commented on issue #14622: REST API: dag status stream.
Posted by GitBox <gi...@apache.org>.
lromor commented on issue #14622:
URL: https://github.com/apache/airflow/issues/14622#issuecomment-791438079
Hi @vemikhaylov , thank you for message.
I don't think websockets are necessary. A standard http streamed response should be more than enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #14622: REST API: dag status stream.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #14622:
URL: https://github.com/apache/airflow/issues/14622#issuecomment-791786413
Airflow uses a database for communication, which means there is no efficient way to send messages from the scheduler to the webserver. If we wanted to do it right, we would have to have a queue where the scheduler / worker could publish information, and the webserver could respond to these messages and forward to the user. This is an interesting change, but would require a major architectural change. For now, we have to use HTTP/database pooling
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lromor edited a comment on issue #14622: REST API: dag status stream.
Posted by GitBox <gi...@apache.org>.
lromor edited a comment on issue #14622:
URL: https://github.com/apache/airflow/issues/14622#issuecomment-791438079
Hi @vemikhaylov , thank you for message!
I don't think websockets are necessary. A standard http streamed response should be more than enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lromor commented on issue #14622: REST API: dag status stream.
Posted by GitBox <gi...@apache.org>.
lromor commented on issue #14622:
URL: https://github.com/apache/airflow/issues/14622#issuecomment-817736789
Yes, would be fine for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #14622: REST API: dag status stream.
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #14622:
URL: https://github.com/apache/airflow/issues/14622#issuecomment-791505178
Yes, this would be nice, but doing it efficiently is a lot of work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vemikhaylov edited a comment on issue #14622: REST API: dag status stream.
Posted by GitBox <gi...@apache.org>.
vemikhaylov edited a comment on issue #14622:
URL: https://github.com/apache/airflow/issues/14622#issuecomment-791410590
Should it be like a websockets solution or a client can be notified with webhooks? Or would be great to have the both?
I'll be happy to implement such feature if it's necessity's confirmed by maintainers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14622: REST API: dag status stream.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14622:
URL: https://github.com/apache/airflow/issues/14622#issuecomment-791339661
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org