You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/18 12:04:38 UTC
[GitHub] [airflow] turbaszek opened a new pull request #11631: Show Celery stats in webui
turbaszek opened a new pull request #11631:
URL: https://github.com/apache/airflow/pull/11631
Add simple view showing information about Celery workers.
Closes: #11623
<img width="2560" alt="Screenshot 2020-10-18 at 13 16 02" src="https://user-images.githubusercontent.com/9528307/96366874-b82ad900-114a-11eb-9b53-79ca1ff4c371.png">
I'm still facing some issue with the stability of getting stats (some missing keys and redis errors like here https://github.com/celery/kombu/issues/1063)
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-803212026
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507747037
##########
File path: airflow/www/extensions/init_views.py
##########
@@ -58,6 +58,7 @@ def init_appbuilder_views(app):
appbuilder.add_view(views.PoolModelView, "Pools", category="Admin")
appbuilder.add_view(views.VariableModelView, "Variables", category="Admin")
appbuilder.add_view(views.XComModelView, "XComs", category="Admin")
+ appbuilder.add_view(views.StatsView, "Stats", category="Admin")
Review comment:
Yeah, that's exactly what I was thinking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507601355
##########
File path: airflow/www/extensions/init_views.py
##########
@@ -58,6 +58,7 @@ def init_appbuilder_views(app):
appbuilder.add_view(views.PoolModelView, "Pools", category="Admin")
appbuilder.add_view(views.VariableModelView, "Variables", category="Admin")
appbuilder.add_view(views.XComModelView, "XComs", category="Admin")
+ appbuilder.add_view(views.StatsView, "Stats", category="Admin")
Review comment:
@ashb I added information about task instances and dag runs. When users use something else than celery executor then they will only see this simple info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] HaloKo4 commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
HaloKo4 commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-771767084
Rechecking about this. Is it expected to be included in 2.1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507808930
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
+ <h2>Airflow stats</h2>
+ <p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+ </p>
+ <h3>Task instances</h3>
+ <table id="tis-table" class="table table-bordered table-striped" style="width: 50%">
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.QUEUED | upper }}</th>
+ <th>{{ State.SCHEDULED | upper }}</th>
+ <th>{{ State.UP_FOR_RESCHEDULE | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ ti_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ ti_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ ti_info.get(State.SCHEDULED, 0) }}</td>
+ <td>{{ ti_info.get(State.UP_FOR_RESCHEDULE, 0) }}</td>
+ </tr>
+ </tbody>
+ </table>
+
+ <h3>Dag runs</h3>
+ <table id="drs-table" class="table table-bordered table-striped" style="width: 50%">
Review comment:
Trailing `;` in your suggestion - on purpose?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] HaloKo4 commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
HaloKo4 commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-735954942
@turbaszek this looks amazing! is this part of Airflow 2 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ryanahamilton commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
ryanahamilton commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507715127
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
+ <h2>Airflow stats</h2>
+ <p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+ </p>
+ <h3>Task instances</h3>
+ <table id="tis-table" class="table table-bordered table-striped" style="width: 50%">
Review comment:
```suggestion
<table class="table table-bordered table-striped" style="width: auto; min-width: 50%;">
```
The id isn't necessary
Probably not a big deal with this limited content, but setting the `width: 50%` can yield some undesirable effects with narrower screen-widths. This `width/min-width` combo will be a bit more flexible.
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
Review comment:
This top-level `<div>` is superfluous, since all of the contained children are block elements.
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
+ <h2>Airflow stats</h2>
+ <p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+ </p>
+ <h3>Task instances</h3>
+ <table id="tis-table" class="table table-bordered table-striped" style="width: 50%">
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.QUEUED | upper }}</th>
+ <th>{{ State.SCHEDULED | upper }}</th>
+ <th>{{ State.UP_FOR_RESCHEDULE | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ ti_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ ti_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ ti_info.get(State.SCHEDULED, 0) }}</td>
+ <td>{{ ti_info.get(State.UP_FOR_RESCHEDULE, 0) }}</td>
+ </tr>
+ </tbody>
+ </table>
+
+ <h3>Dag runs</h3>
+ <table id="drs-table" class="table table-bordered table-striped" style="width: 50%">
Review comment:
```suggestion
<table" class="table table-bordered table-striped" style="width: auto; min-width: 50%;">
```
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
+ <h2>Airflow stats</h2>
+ <p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+ </p>
+ <h3>Task instances</h3>
+ <table id="tis-table" class="table table-bordered table-striped" style="width: 50%">
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.QUEUED | upper }}</th>
+ <th>{{ State.SCHEDULED | upper }}</th>
+ <th>{{ State.UP_FOR_RESCHEDULE | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ ti_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ ti_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ ti_info.get(State.SCHEDULED, 0) }}</td>
+ <td>{{ ti_info.get(State.UP_FOR_RESCHEDULE, 0) }}</td>
+ </tr>
+ </tbody>
+ </table>
+
+ <h3>Dag runs</h3>
+ <table id="drs-table" class="table table-bordered table-striped" style="width: 50%">
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.SUCCESS | upper }}</th>
+ <th>{{ State.FAILED | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ dr_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ dr_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ dr_info.get(State.SCHEDULED, 0) }}</td>
+ </tr>
+ </tbody>
+ </table>
+
+ {% if celery %}
+ <h2>Celery workers</h2>
+ <p>This information is limited. For more insight use
+ <a href="https://airflow.apache.org/docs/stable/cli-ref.html#flower" target="_blank">Flower</a>.
+ </p>
+ <table id="workers-table" class="table table-bordered table-striped">
Review comment:
```suggestion
<table class="table table-bordered table-striped">
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-736865155
Yeah, let's only spend time on it after other high-priority items are done. And once those are done, I have no problems merging this in :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-711295026
[The Workflow run](https://github.com/apache/airflow/actions/runs/314002646) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-711295811
[The Workflow run](https://github.com/apache/airflow/actions/runs/314002901) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-711327038
[The Workflow run](https://github.com/apache/airflow/actions/runs/314041553) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kurtqq commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
kurtqq commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507186049
##########
File path: airflow/www/extensions/init_views.py
##########
@@ -58,6 +58,7 @@ def init_appbuilder_views(app):
appbuilder.add_view(views.PoolModelView, "Pools", category="Admin")
appbuilder.add_view(views.VariableModelView, "Variables", category="Admin")
appbuilder.add_view(views.XComModelView, "XComs", category="Admin")
+ appbuilder.add_view(views.StatsView, "Stats", category="Admin")
Review comment:
shouldn't be also a permission that can be granted to access this page for custom roles?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ryanahamilton commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
ryanahamilton commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507716514
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
+ <h2>Airflow stats</h2>
+ <p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+ </p>
+ <h3>Task instances</h3>
+ <table id="tis-table" class="table table-bordered table-striped" style="width: 50%">
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.QUEUED | upper }}</th>
+ <th>{{ State.SCHEDULED | upper }}</th>
+ <th>{{ State.UP_FOR_RESCHEDULE | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ ti_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ ti_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ ti_info.get(State.SCHEDULED, 0) }}</td>
+ <td>{{ ti_info.get(State.UP_FOR_RESCHEDULE, 0) }}</td>
+ </tr>
+ </tbody>
+ </table>
+
+ <h3>Dag runs</h3>
+ <table id="drs-table" class="table table-bordered table-striped" style="width: 50%">
Review comment:
```suggestion
<table class="table table-bordered table-striped" style="width: auto; min-width: 50%;">
```
The `id` isn't necessary
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
+ <h2>Airflow stats</h2>
+ <p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+ </p>
+ <h3>Task instances</h3>
+ <table id="tis-table" class="table table-bordered table-striped" style="width: 50%">
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.QUEUED | upper }}</th>
+ <th>{{ State.SCHEDULED | upper }}</th>
+ <th>{{ State.UP_FOR_RESCHEDULE | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ ti_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ ti_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ ti_info.get(State.SCHEDULED, 0) }}</td>
+ <td>{{ ti_info.get(State.UP_FOR_RESCHEDULE, 0) }}</td>
+ </tr>
+ </tbody>
+ </table>
+
+ <h3>Dag runs</h3>
+ <table id="drs-table" class="table table-bordered table-striped" style="width: 50%">
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.SUCCESS | upper }}</th>
+ <th>{{ State.FAILED | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ dr_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ dr_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ dr_info.get(State.SCHEDULED, 0) }}</td>
+ </tr>
+ </tbody>
+ </table>
+
+ {% if celery %}
+ <h2>Celery workers</h2>
+ <p>This information is limited. For more insight use
+ <a href="https://airflow.apache.org/docs/stable/cli-ref.html#flower" target="_blank">Flower</a>.
+ </p>
+ <table id="workers-table" class="table table-bordered table-striped">
Review comment:
```suggestion
<table class="table table-bordered table-striped">
```
The `id` isn't necessary
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-735974751
If nothing else will be blocked by this one - why not. Seems rather disconnected from everything else and it has a very low risk. However I think at most we should rebase it now and only include if we manage to properly review it (after implementing all the high priority stuff)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r508011760
##########
File path: airflow/www/views.py
##########
@@ -2331,6 +2336,45 @@ def task_instances(self):
return json.dumps(task_instances)
+class StatsView(AirflowBaseView):
+ """View to show statistics about Airflow"""
+
+ default_view = 'stats'
+
+ @expose('/stats')
+ @has_access
+ @provide_session
+ def stats(self, session=None):
+ """Shows Airflow stats"""
+ # TaskInstance summary
+ TI = TaskInstance
+ ti_info = dict(session.query(
+ TI.state, func.count(TI.execution_date.distinct())
+ ).filter(TI.state.notin_(list(State.finished))).group_by(TI.state).all())
+
+ # DagRuns summary
+ DR = DagRun
+ dr_info = dict(session.query(
+ DR.state, func.count(DR.execution_date.distinct())
+ ).group_by(DR.state).all())
+
+ # Celery workers info
+ worker_info = []
+ celery_executors = (ExecutorLoader.CELERY_EXECUTOR, ExecutorLoader.CELERY_KUBERNETES_EXECUTOR)
+ celery_executor = conf.get("core", "executor") in celery_executors
+ if celery_executor:
+ worker_info = celery_controller.get_information()
+
+ return self.render_template(
+ 'airflow/stats.html',
Review comment:
To be honest I'm not sure if I see value of adding more templates
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507664998
##########
File path: airflow/utils/celery.py
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import collections
+from functools import partial
+from typing import Dict, List, NamedTuple
+
+from celery import Celery
+
+from airflow.executors.celery_executor import create_app
Review comment:
If we do https://github.com/apache/airflow/pull/11631#discussion_r507568576 -- it could live somewhere in celery_executor.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-711286283
CC @ryanahamilton @olchas
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-735959919
> @turbaszek this looks amazing! is this part of Airflow 2 ?
I think we can do it, however it's not necessary. @kaxil @potiuk should we include this or should we limit the scope?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-711283586
> I'm still facing some issue with the stability of getting stats (some missing keys and redis errors like here celery/kombu#1063)
I think increasing timeout in inspect solved this issue 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r508137239
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,96 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+{% block page_title %}Airflow Executor Stats{% endblock %}
+
+{% block content %}
+<h2>Airflow Summary</h2>
+<p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+</p>
+<h3>Task instances</h3>
+ <table class="table table-bordered table-striped" style="width: auto; min-width: 50%";>
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.QUEUED | upper }}</th>
+ <th>{{ State.SCHEDULED | upper }}</th>
+ <th>{{ State.UP_FOR_RESCHEDULE | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ ti_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ ti_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ ti_info.get(State.SCHEDULED, 0) }}</td>
+ <td>{{ ti_info.get(State.UP_FOR_RESCHEDULE, 0) }}</td>
+ </tr>
+ </tbody>
+</table>
+
+<h3>Dag runs</h3>
+ <table class="table table-bordered table-striped" style="width: 50%">
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.SUCCESS | upper }}</th>
+ <th>{{ State.FAILED | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ dr_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ dr_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ dr_info.get(State.SCHEDULED, 0) }}</td>
+ </tr>
+ </tbody>
+</table>
+
+{% if celery %}
+<h2>Celery workers</h2>
+<p>This information is limited. For more insight use
+ <a href="https://airflow.apache.org/docs/stable/cli-ref.html#flower" target="_blank">Flower</a>.
Review comment:
Can you use [get_docs_link](https://github.com/apache/airflow/blob/master/airflow/utils/docs.py) function to generate link to fixed version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ryanahamilton commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
ryanahamilton commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507811382
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
+ <h2>Airflow stats</h2>
+ <p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+ </p>
+ <h3>Task instances</h3>
+ <table id="tis-table" class="table table-bordered table-striped" style="width: 50%">
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.QUEUED | upper }}</th>
+ <th>{{ State.SCHEDULED | upper }}</th>
+ <th>{{ State.UP_FOR_RESCHEDULE | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ ti_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ ti_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ ti_info.get(State.SCHEDULED, 0) }}</td>
+ <td>{{ ti_info.get(State.UP_FOR_RESCHEDULE, 0) }}</td>
+ </tr>
+ </tbody>
+ </table>
+
+ <h3>Dag runs</h3>
+ <table id="drs-table" class="table table-bordered table-striped" style="width: 50%">
Review comment:
I know it's not required for the last one, but I like to add them for consistency sake. (If this weren't inline CSS, the stylelint rules would require it.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ryanahamilton commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
ryanahamilton commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507715127
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
+ <h2>Airflow stats</h2>
+ <p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+ </p>
+ <h3>Task instances</h3>
+ <table id="tis-table" class="table table-bordered table-striped" style="width: 50%">
Review comment:
```suggestion
<table class="table table-bordered table-striped" style="width: auto; min-width: 50%;">
```
- The `id` isn't necessary
- Probably not a big deal with this limited content, but setting the `width: 50%` can yield some undesirable effects with narrower screen-widths. This `width/min-width` combo will be a bit more flexible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507686340
##########
File path: airflow/www/extensions/init_views.py
##########
@@ -58,6 +58,7 @@ def init_appbuilder_views(app):
appbuilder.add_view(views.PoolModelView, "Pools", category="Admin")
appbuilder.add_view(views.VariableModelView, "Variables", category="Admin")
appbuilder.add_view(views.XComModelView, "XComs", category="Admin")
+ appbuilder.add_view(views.StatsView, "Stats", category="Admin")
Review comment:
It would be great if we could also get information from Kubernetes. We can do it separately, but the method's interface should be generic enough to be useful for KubernetesExecutor as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-721445341
Can you please rebase your PR on latest Master since we have applied [Black](https://github.com/apache/airflow/commit/4e8f9cc8d02b29c325b8a5a76b4837671bdf5f68) and [PyUpgrade](https://github.com/apache/airflow/commit/8c42cf1b00c90f0d7f11b8a3a455381de8e003c5) on Master.
It will help if your squash your commits into single commit first so that there are less conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507601972
##########
File path: airflow/utils/celery.py
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import collections
+from functools import partial
+from typing import Dict, List, NamedTuple
+
+from celery import Celery
+
+from airflow.executors.celery_executor import create_app
Review comment:
Happy to do that, I was not sure where to put this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507186229
##########
File path: airflow/www/extensions/init_views.py
##########
@@ -58,6 +58,7 @@ def init_appbuilder_views(app):
appbuilder.add_view(views.PoolModelView, "Pools", category="Admin")
appbuilder.add_view(views.VariableModelView, "Variables", category="Admin")
appbuilder.add_view(views.XComModelView, "XComs", category="Admin")
+ appbuilder.add_view(views.StatsView, "Stats", category="Admin")
Review comment:
@kurtqq good catch, I will add
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] HaloKo4 commented on pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
HaloKo4 commented on pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#issuecomment-771767084
Rechecking about this. Is it expected to be included in 2.1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #11631:
URL: https://github.com/apache/airflow/pull/11631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507569020
##########
File path: airflow/www/views.py
##########
@@ -2331,6 +2336,45 @@ def task_instances(self):
return json.dumps(task_instances)
+class StatsView(AirflowBaseView):
+ """View to show statistics about Airflow"""
+
+ default_view = 'stats'
+
+ @expose('/stats')
+ @has_access
+ @provide_session
+ def stats(self, session=None):
+ """Shows Airflow stats"""
+ # TaskInstance summary
+ TI = TaskInstance
+ ti_info = dict(session.query(
+ TI.state, func.count(TI.execution_date.distinct())
+ ).filter(TI.state.notin_(list(State.finished))).group_by(TI.state).all())
+
+ # DagRuns summary
+ DR = DagRun
+ dr_info = dict(session.query(
+ DR.state, func.count(DR.execution_date.distinct())
+ ).group_by(DR.state).all())
+
+ # Celery workers info
+ worker_info = []
+ celery_executors = (ExecutorLoader.CELERY_EXECUTOR, ExecutorLoader.CELERY_KUBERNETES_EXECUTOR)
+ celery_executor = conf.get("core", "executor") in celery_executors
+ if celery_executor:
+ worker_info = celery_controller.get_information()
+
+ return self.render_template(
+ 'airflow/stats.html',
Review comment:
```suggestion
'airflow/celery_executor_stats.html',
```
##########
File path: airflow/www/views.py
##########
@@ -2331,6 +2336,45 @@ def task_instances(self):
return json.dumps(task_instances)
+class StatsView(AirflowBaseView):
+ """View to show statistics about Airflow"""
+
+ default_view = 'stats'
+
+ @expose('/stats')
+ @has_access
+ @provide_session
+ def stats(self, session=None):
+ """Shows Airflow stats"""
+ # TaskInstance summary
+ TI = TaskInstance
+ ti_info = dict(session.query(
+ TI.state, func.count(TI.execution_date.distinct())
+ ).filter(TI.state.notin_(list(State.finished))).group_by(TI.state).all())
+
+ # DagRuns summary
+ DR = DagRun
+ dr_info = dict(session.query(
+ DR.state, func.count(DR.execution_date.distinct())
+ ).group_by(DR.state).all())
+
+ # Celery workers info
+ worker_info = []
+ celery_executors = (ExecutorLoader.CELERY_EXECUTOR, ExecutorLoader.CELERY_KUBERNETES_EXECUTOR)
+ celery_executor = conf.get("core", "executor") in celery_executors
+ if celery_executor:
+ worker_info = celery_controller.get_information()
Review comment:
I think this should be a method on the executor -- this lets us add the capability for stats to other executors in the future.
##########
File path: airflow/utils/celery.py
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import collections
+from functools import partial
+from typing import Dict, List, NamedTuple
+
+from celery import Celery
+
+from airflow.executors.celery_executor import create_app
Review comment:
I think all this should go in to the celery_executor.py (either directly as methods, or in a nested class under CeleryExecutor)
My main reason is that anything under `utils/` is a code-smell to me - it quickly becomes a junkyard where we just place anything and everything :)
##########
File path: airflow/www/views.py
##########
@@ -2331,6 +2336,45 @@ def task_instances(self):
return json.dumps(task_instances)
+class StatsView(AirflowBaseView):
+ """View to show statistics about Airflow"""
+
+ default_view = 'stats'
+
+ @expose('/stats')
+ @has_access
+ @provide_session
+ def stats(self, session=None):
+ """Shows Airflow stats"""
+ # TaskInstance summary
+ TI = TaskInstance
+ ti_info = dict(session.query(
+ TI.state, func.count(TI.execution_date.distinct())
+ ).filter(TI.state.notin_(list(State.finished))).group_by(TI.state).all())
+
+ # DagRuns summary
+ DR = DagRun
+ dr_info = dict(session.query(
+ DR.state, func.count(DR.execution_date.distinct())
+ ).group_by(DR.state).all())
+
+ # Celery workers info
+ worker_info = []
+ celery_executors = (ExecutorLoader.CELERY_EXECUTOR, ExecutorLoader.CELERY_KUBERNETES_EXECUTOR)
+ celery_executor = conf.get("core", "executor") in celery_executors
+ if celery_executor:
+ worker_info = celery_controller.get_information()
+
+ return self.render_template(
+ 'airflow/stats.html',
Review comment:
Which could possibly be template which extends the common `stats.html` template.
##########
File path: airflow/www/extensions/init_views.py
##########
@@ -58,6 +58,7 @@ def init_appbuilder_views(app):
appbuilder.add_view(views.PoolModelView, "Pools", category="Admin")
appbuilder.add_view(views.VariableModelView, "Variables", category="Admin")
appbuilder.add_view(views.XComModelView, "XComs", category="Admin")
+ appbuilder.add_view(views.StatsView, "Stats", category="Admin")
Review comment:
```suggestion
appbuilder.add_view(views.StatsView, "Executor Stats", category="Admin")
```
##########
File path: airflow/utils/celery.py
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import collections
+from functools import partial
+from typing import Dict, List, NamedTuple
+
+from celery import Celery
+
+from airflow.executors.celery_executor import create_app
+
+
+class WorkerInfo(NamedTuple):
+ """Information about single Celery worker"""
+
+ id: str
+ active_task: int
+ total_task: int
+ uptime: str
+ max_concurrency: int
+ active_queues: List[str]
+
+ @classmethod
+ def from_dict(cls, worker_id: str, raw_info: dict):
+ """Creates WorkerInfo from dictionary"""
+ stats = raw_info.get("stats", {})
+ return cls(
+ id=worker_id,
+ active_task=len(raw_info.get("active", [])),
+ total_task=sum(stats.get("total", {}).values()),
+ max_concurrency=stats.get("pool", {}).get("max-concurrency", 0),
+ uptime=cls._ts_to_readable_format(stats.get("uptime", 0)),
+ active_queues=[q["name"] for q in raw_info.get("active_queues", [])],
+ )
+
+ @staticmethod
+ def _ts_to_readable_format(timestamp) -> str:
+ day = timestamp // (24 * 3600)
+ timestamp = timestamp % (24 * 3600)
+ hour = timestamp // 3600
+ timestamp %= 3600
+ minutes = timestamp // 60
+ timestamp %= 60
+ seconds = timestamp
+ return f"{day}d {hour}h {minutes}m {seconds}s"
+
+
+class CeleryControlHandler:
+ """Class to access Celery workers information using inspect"""
+
+ INSPECT_METHODS = ('stats', 'active_queues', 'active')
+ worker_cache: Dict[str, dict] = collections.defaultdict(dict)
+
+ @classmethod
+ def _update_workers(cls):
+ app: Celery = create_app()
+ inspect = app.control.inspect(timeout=0.5)
+ for method in cls.INSPECT_METHODS:
+ result = partial(getattr(inspect, method))()
Review comment:
What does the `partial` here do? Wouldn't `getattr(inspect, method)()` do the same?
##########
File path: airflow/www/extensions/init_views.py
##########
@@ -58,6 +58,7 @@ def init_appbuilder_views(app):
appbuilder.add_view(views.PoolModelView, "Pools", category="Admin")
appbuilder.add_view(views.VariableModelView, "Variables", category="Admin")
appbuilder.add_view(views.XComModelView, "XComs", category="Admin")
+ appbuilder.add_view(views.StatsView, "Stats", category="Admin")
Review comment:
Does it also make sense to show this if not on Celery/CeleryKube executor?
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
Review comment:
We should override the `page_title` too so we set the `<title>`.
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,97 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+
+{% block content %}
+ <div>
+ <h2>Airflow stats</h2>
Review comment:
```suggestion
<h2>State Summary</h2>
```
##########
File path: airflow/www/views.py
##########
@@ -84,6 +85,10 @@
FILTER_TAGS_COOKIE = 'tags_filter'
FILTER_STATUS_COOKIE = 'dag_status_filter'
+log = logging.getLogger(__name__)
Review comment:
Not used by anything?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r507600339
##########
File path: airflow/utils/celery.py
##########
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import collections
+from functools import partial
+from typing import Dict, List, NamedTuple
+
+from celery import Celery
+
+from airflow.executors.celery_executor import create_app
+
+
+class WorkerInfo(NamedTuple):
+ """Information about single Celery worker"""
+
+ id: str
+ active_task: int
+ total_task: int
+ uptime: str
+ max_concurrency: int
+ active_queues: List[str]
+
+ @classmethod
+ def from_dict(cls, worker_id: str, raw_info: dict):
+ """Creates WorkerInfo from dictionary"""
+ stats = raw_info.get("stats", {})
+ return cls(
+ id=worker_id,
+ active_task=len(raw_info.get("active", [])),
+ total_task=sum(stats.get("total", {}).values()),
+ max_concurrency=stats.get("pool", {}).get("max-concurrency", 0),
+ uptime=cls._ts_to_readable_format(stats.get("uptime", 0)),
+ active_queues=[q["name"] for q in raw_info.get("active_queues", [])],
+ )
+
+ @staticmethod
+ def _ts_to_readable_format(timestamp) -> str:
+ day = timestamp // (24 * 3600)
+ timestamp = timestamp % (24 * 3600)
+ hour = timestamp // 3600
+ timestamp %= 3600
+ minutes = timestamp // 60
+ timestamp %= 60
+ seconds = timestamp
+ return f"{day}d {hour}h {minutes}m {seconds}s"
+
+
+class CeleryControlHandler:
+ """Class to access Celery workers information using inspect"""
+
+ INSPECT_METHODS = ('stats', 'active_queues', 'active')
+ worker_cache: Dict[str, dict] = collections.defaultdict(dict)
+
+ @classmethod
+ def _update_workers(cls):
+ app: Celery = create_app()
+ inspect = app.control.inspect(timeout=0.5)
+ for method in cls.INSPECT_METHODS:
+ result = partial(getattr(inspect, method))()
Review comment:
I think it should but I took this cod from flower 😄
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ryanahamilton commented on a change in pull request #11631: Show Celery stats in webui
Posted by GitBox <gi...@apache.org>.
ryanahamilton commented on a change in pull request #11631:
URL: https://github.com/apache/airflow/pull/11631#discussion_r508471450
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,96 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+{% block page_title %}Airflow Executor Stats{% endblock %}
+
+{% block content %}
+<h2>Airflow Summary</h2>
+<p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+</p>
+<h3>Task instances</h3>
+ <table class="table table-bordered table-striped" style="width: auto; min-width: 50%";>
Review comment:
```suggestion
<table class="table table-bordered table-striped" style="width: auto; min-width: 50%;">
```
Typo
##########
File path: airflow/www/templates/airflow/stats.html
##########
@@ -0,0 +1,96 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends base_template %}
+{% block page_title %}Airflow Executor Stats{% endblock %}
+
+{% block content %}
+<h2>Airflow Summary</h2>
+<p>If you are interested in more detailed statistics please consider using
+ <a href="https://airflow.apache.org/docs/stable/metrics.html" target="_blank">statsd integration</a>.
+</p>
+<h3>Task instances</h3>
+ <table class="table table-bordered table-striped" style="width: auto; min-width: 50%";>
+ <thead>
+ <tr>
+ <th>{{ State.RUNNING | upper }}</th>
+ <th>{{ State.QUEUED | upper }}</th>
+ <th>{{ State.SCHEDULED | upper }}</th>
+ <th>{{ State.UP_FOR_RESCHEDULE | upper }}</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>{{ ti_info.get(State.RUNNING, 0) }}</td>
+ <td>{{ ti_info.get(State.QUEUED, 0) }}</td>
+ <td>{{ ti_info.get(State.SCHEDULED, 0) }}</td>
+ <td>{{ ti_info.get(State.UP_FOR_RESCHEDULE, 0) }}</td>
+ </tr>
+ </tbody>
+</table>
+
+<h3>Dag runs</h3>
+ <table class="table table-bordered table-striped" style="width: 50%">
Review comment:
Looks like this previous suggestion got lost:
```suggestion
<table class="table table-bordered table-striped" style="width: auto; min-width: 50%;">
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org