You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/17 22:04:37 UTC
[GitHub] [airflow] turbaszek opened a new issue #11623: Add celery stats to Airflow webui
turbaszek opened a new issue #11623:
URL: https://github.com/apache/airflow/issues/11623
**Description**
Show basic information about Celery workers in Airflow webui.
**Use case / motivation**
Currently, users using CeleryExecutor have to use Flower to access information about Celery workers. I would like to propose adding new view in Airflow webui that would show similar information.
This should be easily doable by using:
```py
from airflow.executors.celery_executor import app
inspect = self.app.control.inspect(timeout=0.01, destination=None)
result = partial(getattr(inspect, "stats"))()
```
Note: this is how Flower gets the information.
**Related Issues**
N/A
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #11623: Add celery stats to Airflow webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #11623:
URL: https://github.com/apache/airflow/issues/11623#issuecomment-711153240
We can start flower in the background and use its API to show data in Airflow. By more info I thought about showing Airflow-related information like, number of (failed, succeded, queued) tasks. I know users can use statsd and grafana, but having simple overview of the state without additional configuration would be helpful imho
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11623: Add celery stats to Airflow webui
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11623:
URL: https://github.com/apache/airflow/issues/11623#issuecomment-711150945
I think if those are easy to retrieve, and show, it might make sense, but If those can be seen in Flower, I agree with @ashb that there's no reason for duplication. I think however (if it is not done already) that flower should become a first-class-citizen of Airflow in this case and it should have at least some support for RBAC authentication integrated with the Airflow RBAC. This might mean either an iframe (BRRRR!) or some other kind of proxying to flower interface and RBAC permission to control access there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on issue #11623: Add celery stats to Airflow webui
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #11623:
URL: https://github.com/apache/airflow/issues/11623#issuecomment-711147531
<img width="2560" alt="Screenshot 2020-10-18 at 12 27 29" src="https://user-images.githubusercontent.com/9528307/96364918-70518500-113d-11eb-8e02-825f76a95969.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11623: Add celery stats to Airflow webui
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11623:
URL: https://github.com/apache/airflow/issues/11623#issuecomment-711134826
Might be useful - what info does that include?
Also there's so much more that flower does, and can do better because it uses Tornado, so we should draw a clear line in the code, and in the view say something like "for more details use flower"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #11623: Add celery stats to Airflow webui
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #11623:
URL: https://github.com/apache/airflow/issues/11623#issuecomment-711150945
I think if those are easy to retrieve, and show, it might make sense, but If those can be seen in Flower, I agree with @ashb that there's no reason for duplication. I think however (if it is not done already) that flower should become a first-class-citizen of Airflow in this case and it should have at least some support for RBAC authentication integrated with the Airflow RBAC. This might mean either an iframe (BRRRR!) or some other kind of proxying to flower interface and RBAC permission to control access there.
It would be nice to have to open only one firewall port (airflow UI) and one authentication/SSO integration to access flower interface.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11623: Add celery stats to Airflow webui
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11623:
URL: https://github.com/apache/airflow/issues/11623#issuecomment-711148788
Cool!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #11623: Add celery stats to Airflow webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #11623:
URL: https://github.com/apache/airflow/issues/11623#issuecomment-711147531
![Uploading Screenshot 2020-10-18 at 12.27.29.png…]()
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #11623: Add celery stats to Airflow webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #11623:
URL: https://github.com/apache/airflow/issues/11623#issuecomment-711085943
@ashb @dimberman @kaxil @mik-laj @potiuk WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #11623: Add celery stats to Airflow webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #11623:
URL: https://github.com/apache/airflow/issues/11623#issuecomment-711149677
I'm wondering if it would make sense to show more info, like number of tasks and so on. Maybe charts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org