You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kibble.apache.org by hu...@apache.org on 2018/02/16 18:56:15 UTC
[kibble] branch master updated (ddcdb0a -> 0817d31)
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kibble.git.
from ddcdb0a Add Jenkins as a potential source type
new 227b6fe Add in a CI explorer widget
new 2fd2c89 Add preliminary CI status pages
new f590323 regenerate OpenAPI Yaml
new 20013d5 set up a widget for CI
new 0817d31 regen JS
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../{mail/timeseries-single.py => ci/queue.py} | 47 ++++--
.../{mail/timeseries-single.py => ci/status.py} | 43 ++++--
.../{mail/timeseries.py => ci/top-buildcount.py} | 94 ++++++------
.../{mail/timeseries.py => ci/top-buildtime.py} | 94 ++++++------
api/yaml/openapi.yaml | 160 +++++++++++++++++++++
api/yaml/widgets.yaml | 49 +++++++
ui/js/coffee/explorer.coffee | 56 ++++++++
ui/js/coffee/pageloader.coffee | 1 +
ui/js/coffee/widget.coffee | 1 +
ui/js/kibble.v1.js | 99 ++++++++++++-
10 files changed, 518 insertions(+), 126 deletions(-)
copy api/pages/{mail/timeseries-single.py => ci/queue.py} (79%)
copy api/pages/{mail/timeseries-single.py => ci/status.py} (81%)
copy api/pages/{mail/timeseries.py => ci/top-buildcount.py} (72%)
copy api/pages/{mail/timeseries.py => ci/top-buildtime.py} (72%)
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
[kibble] 03/05: regenerate OpenAPI Yaml
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kibble.git
commit f590323a3780f173bf70921149a53e54a1dcca88
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Fri Feb 16 19:55:48 2018 +0100
regenerate OpenAPI Yaml
---
api/yaml/openapi.yaml | 160 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 160 insertions(+)
diff --git a/api/yaml/openapi.yaml b/api/yaml/openapi.yaml
index 4e4a535..d06f1ca 100644
--- a/api/yaml/openapi.yaml
+++ b/api/yaml/openapi.yaml
@@ -745,6 +745,166 @@ paths:
security:
- cookieAuth: []
summary: Shows a quick trend summary of the past 6 months for a contributor
+ /api/ci/queue:
+ get:
+ responses:
+ '200':
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Timeseries'
+ description: 200 Response
+ default:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
+ description: unexpected error
+ security:
+ - cookieAuth: []
+ summary: Shows email sent over time
+ post:
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/defaultWidgetArgs'
+ responses:
+ '200':
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Timeseries'
+ description: 200 Response
+ default:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
+ description: unexpected error
+ security:
+ - cookieAuth: []
+ summary: Shows CI queue over time
+ /api/ci/status:
+ get:
+ responses:
+ '200':
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Timeseries'
+ description: 200 Response
+ default:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
+ description: unexpected error
+ security:
+ - cookieAuth: []
+ summary: Shows email sent over time
+ post:
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/defaultWidgetArgs'
+ responses:
+ '200':
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Timeseries'
+ description: 200 Response
+ default:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
+ description: unexpected error
+ security:
+ - cookieAuth: []
+ summary: Shows CI queue over time
+ /api/ci/top-buildcount:
+ get:
+ responses:
+ '200':
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Timeseries'
+ description: 200 Response
+ default:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
+ description: unexpected error
+ security:
+ - cookieAuth: []
+ summary: Shows top 25 repos by lines of code
+ post:
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/defaultWidgetArgs'
+ responses:
+ '200':
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Timeseries'
+ description: 200 Response
+ default:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
+ description: unexpected error
+ security:
+ - cookieAuth: []
+ summary: Shows top 25 jobs by total builds done. Essentially buildtime, tweaked
+ /api/ci/top-buildtime:
+ get:
+ responses:
+ '200':
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Timeseries'
+ description: 200 Response
+ default:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
+ description: unexpected error
+ security:
+ - cookieAuth: []
+ summary: Shows top 25 repos by lines of code
+ post:
+ requestBody:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/defaultWidgetArgs'
+ responses:
+ '200':
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Timeseries'
+ description: 200 Response
+ default:
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/Error'
+ description: unexpected error
+ security:
+ - cookieAuth: []
+ summary: Shows top 25 jobs by total build time spent
/api/code/changes:
get:
responses:
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
[kibble] 01/05: Add in a CI explorer widget
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kibble.git
commit 227b6fed05a7810aacf3ecf0d40987739c4213e4
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Fri Feb 16 19:55:21 2018 +0100
Add in a CI explorer widget
---
ui/js/coffee/explorer.coffee | 56 ++++++++++++++++++++++++++++++++++++++++++
ui/js/coffee/pageloader.coffee | 1 +
ui/js/coffee/widget.coffee | 1 +
3 files changed, 58 insertions(+)
diff --git a/ui/js/coffee/explorer.coffee b/ui/js/coffee/explorer.coffee
index 61f0411..b94ef28 100644
--- a/ui/js/coffee/explorer.coffee
+++ b/ui/js/coffee/explorer.coffee
@@ -383,6 +383,62 @@ imexplorer = (json, state) ->
, false)
$('select').chosen();
+
+ciexplorer = (json, state) ->
+
+ org = json.organisation
+ if json.tag
+ org.name += " (Filter: " + json.tag + ")"
+ h = document.createElement('h4')
+ h.appendChild(document.createTextNode("Exploring " + org.name + ":"))
+ state.widget.inject(h, true)
+ list = document.createElement('select')
+ state.widget.inject(list)
+ opt = document.createElement('option')
+ opt.value = ""
+ slen = 0
+ for item in json.sources
+ if item.type in ['jenkins','travis','buildbot']
+ slen++
+ opt.text = "All " + slen + " CI Services"
+ list.appendChild(opt)
+ json.sources.sort((a,b) ->
+ return if (a.sourceURL == b.sourceURL) then 0 else (if a.sourceURL > b.sourceURL then 1 else -1)
+ )
+ for item in json.sources
+ if item.type in ['jenkins','travis','buildbot']
+ opt = document.createElement('option')
+ opt.value = item.sourceID
+ ezURL = null
+ n = item.sourceURL.match(/^([a-z]+:\/\/.+?)\/([#\S+]+)$/i)
+ m = item.sourceURL.match(/^([a-z]+:\/\/.+?)\s(.+)$/i)
+ if n and n.length == 3
+ ezURL = "#{n[2]} - (#{n[1]})"
+ else if m and m.length == 3
+ ezURL = "#{m[2]} - (#{m[1]})"
+ opt.text = if ezURL then ezURL else item.sourceURL
+ if globArgs.source and globArgs.source == item.sourceID
+ opt.selected = 'selected'
+ list.appendChild(opt)
+
+ ID = Math.floor(Math.random() * 987654321).toString(16)
+ list.setAttribute('id', ID)
+ $("#"+ID).chosen().change(() ->
+ source = this.value
+
+ if source == ""
+ source = null
+ globArgs.source = source
+ updateWidgets('donut', null, { source: source })
+ updateWidgets('gauge', null, { source: source })
+ updateWidgets('line', null, { source: source })
+ updateWidgets('contacts', null, { source: source })
+ updateWidgets('top5', null, { source: source })
+ updateWidgets('factors', null, { source: source })
+ updateWidgets('trends', null, { source: source })
+ updateWidgets('relationship', null, { source: source })
+
+ )
multiviewexplorer = (json, state) ->
diff --git a/ui/js/coffee/pageloader.coffee b/ui/js/coffee/pageloader.coffee
index 9938b6b..56db581 100644
--- a/ui/js/coffee/pageloader.coffee
+++ b/ui/js/coffee/pageloader.coffee
@@ -80,6 +80,7 @@ setupPage = (json, state) ->
when 'logpicker' then widget.load(logexplorer)
when 'impicker' then widget.load(imexplorer)
when 'logpicker' then widget.load(logexplorer)
+ when 'cipicker' then widget.load(ciexplorer)
when 'widgetpicker' then widget.load(widgetexplorer)
when 'multiviewpicker' then widget.load(multiviewexplorer)
when 'donut' then widget.load(donut)
diff --git a/ui/js/coffee/widget.coffee b/ui/js/coffee/widget.coffee
index b8f672a..517171b 100644
--- a/ui/js/coffee/widget.coffee
+++ b/ui/js/coffee/widget.coffee
@@ -126,6 +126,7 @@ updateWidgets = (type, target, eargs) ->
when 'issuepicker' then widget.load(issueexplorer)
when 'viewpicker' then widget.load(viewexplorer)
when 'mailpicker' then widget.load(mailexplorer)
+ when 'cipicker' then widget.load(ciexplorer)
when 'logpicker' then widget.load(logexplorer)
when 'relationship' then widget.load(relationship)
when 'treemap' then widget.load(treemap)
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
[kibble] 04/05: set up a widget for CI
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kibble.git
commit 20013d500a881b9ee536e051f4158f7e97793728
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Fri Feb 16 19:55:58 2018 +0100
set up a widget for CI
---
api/yaml/widgets.yaml | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 49 insertions(+)
diff --git a/api/yaml/widgets.yaml b/api/yaml/widgets.yaml
index dedbbab..f9b9006 100644
--- a/api/yaml/widgets.yaml
+++ b/api/yaml/widgets.yaml
@@ -710,4 +710,53 @@ widgets:
source: "bio/newtimers"
name: "New contributors"
blocks: 12
+
+
+## CONTINUOUS INTEGRATION
+ ci:
+ title: "Continuous Integration"
+ rows:
+ -
+ name: "Date picker row"
+ children:
+ -
+ type: "datepicker"
+ blocks: 4
+ name: "Date picker"
+ -
+ type: "viewpicker"
+ blocks: 4
+ source: "views"
+ name: "Quick filter"
+ -
+ type: cipicker
+ blocks: 4
+ source: "sources"
+ name: "CI picker"
+ -
+ name: "CI queue"
+ children:
+ -
+ type: "line"
+ source: "ci/queue"
+ name: "CI Queue"
+ blocks: 6
+ -
+ type: "line"
+ source: "ci/status"
+ name: "CI Queue Status"
+ blocks: 6
+ -
+ name: "Busiest Jobs"
+ children:
+ -
+ type: "donut"
+ source: "ci/top-buildtime"
+ name: "Busiest jobs by build time (in days)"
+ blocks: 6
+ -
+ type: "donut"
+ source: "ci/top-buildcount"
+ name: "Busiest jobs by build count"
+ blocks: 6
\ No newline at end of file
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
[kibble] 05/05: regen JS
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kibble.git
commit 0817d31df2799b1e8c94193c5af30c57f204ee13
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Fri Feb 16 19:56:07 2018 +0100
regen JS
---
ui/js/kibble.v1.js | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 98 insertions(+), 1 deletion(-)
diff --git a/ui/js/kibble.v1.js b/ui/js/kibble.v1.js
index ec194a2..1720360 100644
--- a/ui/js/kibble.v1.js
+++ b/ui/js/kibble.v1.js
@@ -1,5 +1,5 @@
// Generated by CoffeeScript 1.10.0
-var API, APIVERSION, Chart, HTML, Row, Widget, aSourceTypes, accountCallback, addSourceType, addSources, addorguser, addsources, affiliate, affiliation, affiliationWizard, altemail, app, badModal, bio, chartOnclick, chartToSvg, chartWrapperButtons, charts_donutchart, charts_gaugechart, charts_linechart, charts_linechart_stacked, charts_linked, charts_radarchart, cog, comShow, comstat, copyCSS, currentSources, dataTable, datepicker, datepickers, defaultOrgChanged, deletesource, doResetPas [...]
+var API, APIVERSION, Chart, HTML, Row, Widget, aSourceTypes, accountCallback, addSourceType, addSources, addorguser, addsources, affiliate, affiliation, affiliationWizard, altemail, app, badModal, bio, chartOnclick, chartToSvg, chartWrapperButtons, charts_donutchart, charts_gaugechart, charts_linechart, charts_linechart_stacked, charts_linked, charts_radarchart, ciexplorer, cog, comShow, comstat, copyCSS, currentSources, dataTable, datepicker, datepickers, defaultOrgChanged, deletesource [...]
indexOf = [].indexOf || function(item) { for (var i = 0, l = this.length; i < l; i++) { if (i in this && this[i] === item) return i; } return -1; };
signup = function(form) {
@@ -2178,6 +2178,97 @@ imexplorer = function(json, state) {
return $('select').chosen();
};
+ciexplorer = function(json, state) {
+ var ID, ezURL, h, item, len, len1, list, m, n, opt, org, q, ref, ref1, ref2, ref3, slen, u;
+ org = json.organisation;
+ if (json.tag) {
+ org.name += " (Filter: " + json.tag + ")";
+ }
+ h = document.createElement('h4');
+ h.appendChild(document.createTextNode("Exploring " + org.name + ":"));
+ state.widget.inject(h, true);
+ list = document.createElement('select');
+ state.widget.inject(list);
+ opt = document.createElement('option');
+ opt.value = "";
+ slen = 0;
+ ref = json.sources;
+ for (q = 0, len = ref.length; q < len; q++) {
+ item = ref[q];
+ if ((ref1 = item.type) === 'jenkins' || ref1 === 'travis' || ref1 === 'buildbot') {
+ slen++;
+ }
+ }
+ opt.text = "All " + slen + " CI Services";
+ list.appendChild(opt);
+ json.sources.sort(function(a, b) {
+ if (a.sourceURL === b.sourceURL) {
+ return 0;
+ } else {
+ if (a.sourceURL > b.sourceURL) {
+ return 1;
+ } else {
+ return -1;
+ }
+ }
+ });
+ ref2 = json.sources;
+ for (u = 0, len1 = ref2.length; u < len1; u++) {
+ item = ref2[u];
+ if ((ref3 = item.type) === 'jenkins' || ref3 === 'travis' || ref3 === 'buildbot') {
+ opt = document.createElement('option');
+ opt.value = item.sourceID;
+ ezURL = null;
+ n = item.sourceURL.match(/^([a-z]+:\/\/.+?)\/([#\S+]+)$/i);
+ m = item.sourceURL.match(/^([a-z]+:\/\/.+?)\s(.+)$/i);
+ if (n && n.length === 3) {
+ ezURL = n[2] + " - (" + n[1] + ")";
+ } else if (m && m.length === 3) {
+ ezURL = m[2] + " - (" + m[1] + ")";
+ }
+ opt.text = ezURL ? ezURL : item.sourceURL;
+ if (globArgs.source && globArgs.source === item.sourceID) {
+ opt.selected = 'selected';
+ }
+ list.appendChild(opt);
+ }
+ }
+ ID = Math.floor(Math.random() * 987654321).toString(16);
+ list.setAttribute('id', ID);
+ return $("#" + ID).chosen().change(function() {
+ var source;
+ source = this.value;
+ if (source === "") {
+ source = null;
+ }
+ globArgs.source = source;
+ updateWidgets('donut', null, {
+ source: source
+ });
+ updateWidgets('gauge', null, {
+ source: source
+ });
+ updateWidgets('line', null, {
+ source: source
+ });
+ updateWidgets('contacts', null, {
+ source: source
+ });
+ updateWidgets('top5', null, {
+ source: source
+ });
+ updateWidgets('factors', null, {
+ source: source
+ });
+ updateWidgets('trends', null, {
+ source: source
+ });
+ return updateWidgets('relationship', null, {
+ source: source
+ });
+ });
+};
+
multiviewexplorer = function(json, state) {
var ID, h, item, k, list, opt, org, q, results, tName;
org = json.organisation;
@@ -3599,6 +3690,9 @@ setupPage = function(json, state) {
case 'logpicker':
results1.push(widget.load(logexplorer));
break;
+ case 'cipicker':
+ results1.push(widget.load(ciexplorer));
+ break;
case 'widgetpicker':
results1.push(widget.load(widgetexplorer));
break;
@@ -5051,6 +5145,9 @@ updateWidgets = function(type, target, eargs) {
case 'mailpicker':
results.push(widget.load(mailexplorer));
break;
+ case 'cipicker':
+ results.push(widget.load(ciexplorer));
+ break;
case 'logpicker':
results.push(widget.load(logexplorer));
break;
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.
[kibble] 02/05: Add preliminary CI status pages
Posted by hu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
humbedooh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kibble.git
commit 2fd2c89040320f8216864e3f21e09d20c5bbe159
Author: Daniel Gruno <hu...@apache.org>
AuthorDate: Fri Feb 16 19:55:40 2018 +0100
Add preliminary CI status pages
---
api/pages/ci/queue.py | 182 ++++++++++++++++++++++++++++++++++++++++
api/pages/ci/status.py | 180 ++++++++++++++++++++++++++++++++++++++++
api/pages/ci/top-buildcount.py | 183 +++++++++++++++++++++++++++++++++++++++++
api/pages/ci/top-buildtime.py | 183 +++++++++++++++++++++++++++++++++++++++++
4 files changed, 728 insertions(+)
diff --git a/api/pages/ci/queue.py b/api/pages/ci/queue.py
new file mode 100644
index 0000000..06d495d
--- /dev/null
+++ b/api/pages/ci/queue.py
@@ -0,0 +1,182 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+########################################################################
+# OPENAPI-URI: /api/ci/queue
+########################################################################
+# get:
+# responses:
+# '200':
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Timeseries'
+# description: 200 Response
+# default:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Error'
+# description: unexpected error
+# security:
+# - cookieAuth: []
+# summary: Shows email sent over time
+# post:
+# requestBody:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/defaultWidgetArgs'
+# responses:
+# '200':
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Timeseries'
+# description: 200 Response
+# default:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Error'
+# description: unexpected error
+# security:
+# - cookieAuth: []
+# summary: Shows CI queue over time
+#
+########################################################################
+
+
+
+"""
+This is the CI queue timeseries renderer for Kibble
+"""
+
+import json
+import time
+import hashlib
+
+def run(API, environ, indata, session):
+
+ # We need to be logged in for this!
+ if not session.user:
+ raise API.exception(403, "You must be logged in to use this API endpoint! %s")
+
+ now = time.time()
+
+ # First, fetch the view if we have such a thing enabled
+ viewList = []
+ if indata.get('view'):
+ viewList = session.getView(indata.get('view'))
+ if indata.get('subfilter'):
+ viewList = session.subFilter(indata.get('subfilter'), view = viewList)
+
+
+ dateTo = indata.get('to', int(time.time()))
+ dateFrom = indata.get('from', dateTo - (86400*30*6)) # Default to a 6 month span
+
+ interval = indata.get('interval', 'month')
+
+
+ ####################################################################
+ ####################################################################
+ dOrg = session.user['defaultOrganisation'] or "apache"
+ query = {
+ 'query': {
+ 'bool': {
+ 'must': [
+ {'range':
+ {
+ 'time': {
+ 'from': dateFrom,
+ 'to': dateTo
+ }
+ }
+ },
+ {
+ 'term': {
+ 'organisation': dOrg
+ }
+ }
+ ]
+ }
+ }
+ }
+ # Source-specific or view-specific??
+ if indata.get('source'):
+ query['query']['bool']['must'].append({'term': {'sourceID': indata.get('source')}})
+ elif viewList:
+ query['query']['bool']['must'].append({'terms': {'sourceID': viewList}})
+
+ # Get queue stats
+ query['aggs'] = {
+ 'timeseries': {
+ 'date_histogram': {
+ 'field': 'date',
+ 'interval': interval
+ },
+ 'aggs': {
+ 'size': {
+ 'avg': {
+ 'field': 'size'
+ }
+ },
+ 'blocked': {
+ 'avg': {
+ 'field': 'blocked'
+ }
+ },
+ 'stuck': {
+ 'avg': {
+ 'field': 'stuck'
+ }
+ },
+ 'wait': {
+ 'avg': {
+ 'field': 'avgwait'
+ }
+ }
+ }
+ }
+ }
+ res = session.DB.ES.search(
+ index=session.DB.dbname,
+ doc_type="ci_queue",
+ size = 0,
+ body = query
+ )
+
+ timeseries = []
+ for bucket in res['aggregations']['timeseries']['buckets']:
+ ts = int(bucket['key'] / 1000)
+ timeseries.append({
+ 'date': ts,
+ 'queue size': bucket['size']['value'],
+# 'builds blocked': bucket['blocked']['value'],
+# 'builds stuck': bucket['stuck']['value'],
+ 'average wait (hours)': int(bucket['wait']['value']/3600),
+ })
+
+ JSON_OUT = {
+ 'widgetType': {
+ 'chartType': 'line' # Recommendation for the UI
+ },
+ 'timeseries': timeseries,
+ 'interval': interval,
+ 'okay': True,
+ 'responseTime': time.time() - now
+ }
+ yield json.dumps(JSON_OUT)
diff --git a/api/pages/ci/status.py b/api/pages/ci/status.py
new file mode 100644
index 0000000..e3678a2
--- /dev/null
+++ b/api/pages/ci/status.py
@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+########################################################################
+# OPENAPI-URI: /api/ci/status
+########################################################################
+# get:
+# responses:
+# '200':
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Timeseries'
+# description: 200 Response
+# default:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Error'
+# description: unexpected error
+# security:
+# - cookieAuth: []
+# summary: Shows email sent over time
+# post:
+# requestBody:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/defaultWidgetArgs'
+# responses:
+# '200':
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Timeseries'
+# description: 200 Response
+# default:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Error'
+# description: unexpected error
+# security:
+# - cookieAuth: []
+# summary: Shows CI queue over time
+#
+########################################################################
+
+
+
+"""
+This is the CI queue status (blocked/stuck) timeseries renderer for Kibble
+"""
+
+import json
+import time
+import hashlib
+
+def run(API, environ, indata, session):
+
+ # We need to be logged in for this!
+ if not session.user:
+ raise API.exception(403, "You must be logged in to use this API endpoint! %s")
+
+ now = time.time()
+
+ # First, fetch the view if we have such a thing enabled
+ viewList = []
+ if indata.get('view'):
+ viewList = session.getView(indata.get('view'))
+ if indata.get('subfilter'):
+ viewList = session.subFilter(indata.get('subfilter'), view = viewList)
+
+
+ dateTo = indata.get('to', int(time.time()))
+ dateFrom = indata.get('from', dateTo - (86400*30*6)) # Default to a 6 month span
+
+ interval = indata.get('interval', 'month')
+
+
+ ####################################################################
+ ####################################################################
+ dOrg = session.user['defaultOrganisation'] or "apache"
+ query = {
+ 'query': {
+ 'bool': {
+ 'must': [
+ {'range':
+ {
+ 'time': {
+ 'from': dateFrom,
+ 'to': dateTo
+ }
+ }
+ },
+ {
+ 'term': {
+ 'organisation': dOrg
+ }
+ }
+ ]
+ }
+ }
+ }
+ # Source-specific or view-specific??
+ if indata.get('source'):
+ query['query']['bool']['must'].append({'term': {'sourceID': indata.get('source')}})
+ elif viewList:
+ query['query']['bool']['must'].append({'terms': {'sourceID': viewList}})
+
+ # Get queue stats
+ query['aggs'] = {
+ 'timeseries': {
+ 'date_histogram': {
+ 'field': 'date',
+ 'interval': interval
+ },
+ 'aggs': {
+ 'size': {
+ 'avg': {
+ 'field': 'size'
+ }
+ },
+ 'blocked': {
+ 'avg': {
+ 'field': 'blocked'
+ }
+ },
+ 'stuck': {
+ 'avg': {
+ 'field': 'stuck'
+ }
+ },
+ 'wait': {
+ 'avg': {
+ 'field': 'avgwait'
+ }
+ }
+ }
+ }
+ }
+ res = session.DB.ES.search(
+ index=session.DB.dbname,
+ doc_type="ci_queue",
+ size = 0,
+ body = query
+ )
+
+ timeseries = []
+ for bucket in res['aggregations']['timeseries']['buckets']:
+ ts = int(bucket['key'] / 1000)
+ timeseries.append({
+ 'date': ts,
+ 'builds blocked': bucket['blocked']['value'],
+ 'builds stuck': bucket['stuck']['value']
+ })
+
+ JSON_OUT = {
+ 'widgetType': {
+ 'chartType': 'bar' # Recommendation for the UI
+ },
+ 'timeseries': timeseries,
+ 'interval': interval,
+ 'okay': True,
+ 'responseTime': time.time() - now
+ }
+ yield json.dumps(JSON_OUT)
diff --git a/api/pages/ci/top-buildcount.py b/api/pages/ci/top-buildcount.py
new file mode 100644
index 0000000..31ef5c3
--- /dev/null
+++ b/api/pages/ci/top-buildcount.py
@@ -0,0 +1,183 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+########################################################################
+# OPENAPI-URI: /api/ci/top-buildcount
+########################################################################
+# get:
+# responses:
+# '200':
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Timeseries'
+# description: 200 Response
+# default:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Error'
+# description: unexpected error
+# security:
+# - cookieAuth: []
+# summary: Shows top 25 repos by lines of code
+# post:
+# requestBody:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/defaultWidgetArgs'
+# responses:
+# '200':
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Timeseries'
+# description: 200 Response
+# default:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Error'
+# description: unexpected error
+# security:
+# - cookieAuth: []
+# summary: Shows top 25 jobs by total builds done. Essentially buildtime, tweaked
+#
+########################################################################
+
+
+
+
+
+"""
+This is the TopN CI jobs by total build time renderer for Kibble
+"""
+
+import json
+import time
+import re
+
+def run(API, environ, indata, session):
+
+ # We need to be logged in for this!
+ if not session.user:
+ raise API.exception(403, "You must be logged in to use this API endpoint! %s")
+
+ now = time.time()
+
+ # First, fetch the view if we have such a thing enabled
+ viewList = []
+ if indata.get('view'):
+ viewList = session.getView(indata.get('view'))
+ if indata.get('subfilter'):
+ viewList = session.subFilter(indata.get('subfilter'), view = viewList)
+
+ dateTo = indata.get('to', int(time.time()))
+ dateFrom = indata.get('from', dateTo - (86400*30*6)) # Default to a 6 month span
+
+ ####################################################################
+ ####################################################################
+ dOrg = session.user['defaultOrganisation'] or "apache"
+ query = {
+ 'query': {
+ 'bool': {
+ 'must': [
+ {'range':
+ {
+ 'date': {
+ 'from': time.strftime("%Y/%m/%d %H:%M:%S", time.gmtime(dateFrom)),
+ 'to': time.strftime("%Y/%m/%d %H:%M:%S", time.gmtime(dateTo))
+ }
+ }
+ },
+ {
+ 'term': {
+ 'organisation': dOrg
+ }
+ }
+ ]
+ }
+ }
+ }
+ # Source-specific or view-specific??
+ if indata.get('source'):
+ query['query']['bool']['must'].append({'term': {'sourceID': indata.get('source')}})
+ elif viewList:
+ query['query']['bool']['must'].append({'terms': {'sourceID': viewList}})
+
+ query['aggs'] = {
+ 'by_job': {
+ 'terms': {
+ 'field': 'jobURL.keyword',
+ 'size': 5000,
+ },
+ 'aggs': {
+ 'duration': {
+ 'sum': {
+ 'field': 'duration'
+ }
+ },
+ 'ci': {
+ 'terms': {
+ 'field': 'ci.keyword',
+ 'size': 1
+ }
+ },
+ 'name': {
+ 'terms': {
+ 'field': 'job.keyword',
+ 'size': 1
+ }
+ }
+ }
+ }
+ }
+
+ res = session.DB.ES.search(
+ index=session.DB.dbname,
+ doc_type="ci_build",
+ size = 0,
+ body = query
+ )
+
+ jobs = []
+ for doc in res['aggregations']['by_job']['buckets']:
+ job = doc['key']
+ builds = doc['doc_count']
+ duration = doc['duration']['value']
+ ci = doc['ci']['buckets'][0]['key']
+ jobname = doc['name']['buckets'][0]['key']
+ jobs.append([builds, duration, jobname, ci])
+
+ topjobs = sorted(jobs, key = lambda x: int(x[0]), reverse = True)
+ top = topjobs[0:24]
+ if len(topjobs) > 25:
+ count = 0
+ for repo in topjobs[25:]:
+ count += repo[1]
+ top.append(["Other jobs", 1, count, '??'])
+
+ tophash = {}
+ for v in top:
+ tophash["%s (%s)" % (v[2], v[3])] = v[0]
+
+ JSON_OUT = {
+ 'counts': tophash,
+ 'okay': True,
+ 'responseTime': time.time() - now,
+ }
+ yield json.dumps(JSON_OUT)
diff --git a/api/pages/ci/top-buildtime.py b/api/pages/ci/top-buildtime.py
new file mode 100644
index 0000000..995bd2c
--- /dev/null
+++ b/api/pages/ci/top-buildtime.py
@@ -0,0 +1,183 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+########################################################################
+# OPENAPI-URI: /api/ci/top-buildtime
+########################################################################
+# get:
+# responses:
+# '200':
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Timeseries'
+# description: 200 Response
+# default:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Error'
+# description: unexpected error
+# security:
+# - cookieAuth: []
+# summary: Shows top 25 repos by lines of code
+# post:
+# requestBody:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/defaultWidgetArgs'
+# responses:
+# '200':
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Timeseries'
+# description: 200 Response
+# default:
+# content:
+# application/json:
+# schema:
+# $ref: '#/components/schemas/Error'
+# description: unexpected error
+# security:
+# - cookieAuth: []
+# summary: Shows top 25 jobs by total build time spent
+#
+########################################################################
+
+
+
+
+
+"""
+This is the TopN CI jobs by total build time renderer for Kibble
+"""
+
+import json
+import time
+import re
+
+def run(API, environ, indata, session):
+
+ # We need to be logged in for this!
+ if not session.user:
+ raise API.exception(403, "You must be logged in to use this API endpoint! %s")
+
+ now = time.time()
+
+ # First, fetch the view if we have such a thing enabled
+ viewList = []
+ if indata.get('view'):
+ viewList = session.getView(indata.get('view'))
+ if indata.get('subfilter'):
+ viewList = session.subFilter(indata.get('subfilter'), view = viewList)
+
+ dateTo = indata.get('to', int(time.time()))
+ dateFrom = indata.get('from', dateTo - (86400*30*6)) # Default to a 6 month span
+
+ ####################################################################
+ ####################################################################
+ dOrg = session.user['defaultOrganisation'] or "apache"
+ query = {
+ 'query': {
+ 'bool': {
+ 'must': [
+ {'range':
+ {
+ 'date': {
+ 'from': time.strftime("%Y/%m/%d %H:%M:%S", time.gmtime(dateFrom)),
+ 'to': time.strftime("%Y/%m/%d %H:%M:%S", time.gmtime(dateTo))
+ }
+ }
+ },
+ {
+ 'term': {
+ 'organisation': dOrg
+ }
+ }
+ ]
+ }
+ }
+ }
+ # Source-specific or view-specific??
+ if indata.get('source'):
+ query['query']['bool']['must'].append({'term': {'sourceID': indata.get('source')}})
+ elif viewList:
+ query['query']['bool']['must'].append({'terms': {'sourceID': viewList}})
+
+ query['aggs'] = {
+ 'by_job': {
+ 'terms': {
+ 'field': 'jobURL.keyword',
+ 'size': 5000,
+ },
+ 'aggs': {
+ 'duration': {
+ 'sum': {
+ 'field': 'duration'
+ }
+ },
+ 'ci': {
+ 'terms': {
+ 'field': 'ci.keyword',
+ 'size': 1
+ }
+ },
+ 'name': {
+ 'terms': {
+ 'field': 'job.keyword',
+ 'size': 1
+ }
+ }
+ }
+ }
+ }
+
+ res = session.DB.ES.search(
+ index=session.DB.dbname,
+ doc_type="ci_build",
+ size = 0,
+ body = query
+ )
+
+ jobs = []
+ for doc in res['aggregations']['by_job']['buckets']:
+ job = doc['key']
+ builds = doc['doc_count']
+ duration = doc['duration']['value']
+ ci = doc['ci']['buckets'][0]['key']
+ jobname = doc['name']['buckets'][0]['key']
+ jobs.append([builds, duration, jobname, ci])
+
+ topjobs = sorted(jobs, key = lambda x: int(x[1]), reverse = True)
+ top = topjobs[0:24]
+ if len(topjobs) > 25:
+ count = 0
+ for repo in topjobs[25:]:
+ count += repo[1]
+ top.append(["Other jobs", 1, count, '??'])
+
+ tophash = {}
+ for v in top:
+ tophash["%s (%s)" % (v[2], v[3])] = int((v[1]/86400))
+
+ JSON_OUT = {
+ 'counts': tophash,
+ 'okay': True,
+ 'responseTime': time.time() - now,
+ }
+ yield json.dumps(JSON_OUT)
--
To stop receiving notification emails like this one, please contact
humbedooh@apache.org.