You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2021/05/20 23:14:43 UTC
Change in asterixdb[master]: Added prometheus monitoring to asterix
From Ian Maxon <im...@uci.edu>:
Hello Krish Avvari,
I'd like you to do a code review. Please visit
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544
to review the following change.
Change subject: Added prometheus monitoring to asterix
......................................................................
Added prometheus monitoring to asterix
- Ansible playbooks to install, start, and stop the monitoring environment
- Metrics exporters to export jobs, storage, memory, and cpu info
- Grafana dashboards
Change-Id: Ib9aa81fa2e58954846218e3902562685c8cd09e1
---
M asterixdb/asterix-app/pom.xml
A asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/MetricsServlet.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
A asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/StorageExporter.java
M asterixdb/asterix-app/src/test/resources/cc.conf
A asterixdb/asterix-server/src/main/opt/ansible/bin/install_monitoring.sh
A asterixdb/asterix-server/src/main/opt/ansible/bin/start_monitoring.sh
A asterixdb/asterix-server/src/main/opt/ansible/bin/stop_monitoring.sh
A asterixdb/asterix-server/src/main/opt/ansible/conf/dashboards.yml
A asterixdb/asterix-server/src/main/opt/ansible/conf/datasources.yml
A asterixdb/asterix-server/src/main/opt/ansible/conf/defaults.ini
A asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_jobs_stats.json
A asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_memory_stats.json
A asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_storage_stats.json
A asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_system_stats.json
A asterixdb/asterix-server/src/main/opt/ansible/conf/instance/cc.conf
M asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml
M asterixdb/asterix-server/src/main/opt/ansible/conf/inventory
A asterixdb/asterix-server/src/main/opt/ansible/conf/monitoring_inventory
A asterixdb/asterix-server/src/main/opt/ansible/conf/prometheus.conf.j2
A asterixdb/asterix-server/src/main/opt/ansible/yaml/grafana.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_install.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_start.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_stop.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus_node_exporter.yml
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/JobsExporter.java
A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/MemoryExporter.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
33 files changed, 3,535 insertions(+), 9 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/44/11544/1
diff --git a/asterixdb/asterix-app/pom.xml b/asterixdb/asterix-app/pom.xml
index 34bab13..7830fbe 100644
--- a/asterixdb/asterix-app/pom.xml
+++ b/asterixdb/asterix-app/pom.xml
@@ -867,5 +867,15 @@
<artifactId>kite-data-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>io.prometheus</groupId>
+ <artifactId>simpleclient_hotspot</artifactId>
+ <version>0.9.0</version>
+ </dependency>
+ <dependency>
+ <groupId>io.prometheus</groupId>
+ <artifactId>simpleclient_httpserver</artifactId>
+ <version>0.9.0</version>
+ </dependency>
</dependencies>
</project>
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/MetricsServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/MetricsServlet.java
new file mode 100644
index 0000000..b7e32d1
--- /dev/null
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/MetricsServlet.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.asterix.api.http.server;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.Writer;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.hyracks.http.api.IServletRequest;
+import org.apache.hyracks.http.api.IServletResponse;
+import org.apache.hyracks.http.server.AbstractServlet;
+
+import io.netty.handler.codec.http.HttpHeaderNames;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+
+public class MetricsServlet extends AbstractServlet {
+ private CollectorRegistry registry;
+
+ public MetricsServlet(ConcurrentMap<String, Object> ctx, String... paths) {
+ this(CollectorRegistry.defaultRegistry, ctx, paths);
+ }
+
+ public MetricsServlet(CollectorRegistry registry, ConcurrentMap<String, Object> ctx, String... paths) {
+ super(ctx, paths);
+ this.registry = registry;
+ }
+
+ @Override
+ protected void get(IServletRequest request, IServletResponse response) throws IOException {
+ response.setHeader(HttpHeaderNames.CONTENT_TYPE, TextFormat.CONTENT_TYPE_004);
+
+ Writer writer = new BufferedWriter(response.writer());
+ try {
+ TextFormat.write004(writer, registry.filteredMetricFamilySamples(parse(request)));
+ response.setStatus(HttpResponseStatus.OK);
+ writer.flush();
+ } finally {
+ writer.close();
+ }
+ }
+
+ private Set<String> parse(IServletRequest req) {
+ List<String> includedParam = req.getParameterValues("name[]");
+ if (includedParam == null) {
+ return Collections.emptySet();
+ } else {
+ return new HashSet<String>(includedParam);
+ }
+ }
+
+ @Override
+ protected void post(IServletRequest req, IServletResponse resp) throws IOException {
+ get(req, resp);
+ }
+}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java
index 8f3deb1..8138932 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java
@@ -43,6 +43,7 @@
import org.apache.asterix.api.http.server.ClusterControllerDetailsApiServlet;
import org.apache.asterix.api.http.server.ConnectorApiServlet;
import org.apache.asterix.api.http.server.DiagnosticsApiServlet;
+import org.apache.asterix.api.http.server.MetricsServlet;
import org.apache.asterix.api.http.server.NodeControllerDetailsApiServlet;
import org.apache.asterix.api.http.server.QueryResultApiServlet;
import org.apache.asterix.api.http.server.QueryServiceServlet;
@@ -117,6 +118,8 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import io.prometheus.client.hotspot.DefaultExports;
+
public class CCApplication extends BaseCCApplication {
private static final Logger LOGGER = LogManager.getLogger();
@@ -242,6 +245,7 @@
webManager.add(setupWebServer(appCtx.getExternalProperties()));
webManager.add(setupJSONAPIServer(appCtx.getExternalProperties()));
webManager.add(setupQueryWebServer(appCtx.getExternalProperties()));
+ webManager.add(setupMetricsServer(appCtx.getExternalProperties()));
}
@Override
@@ -253,6 +257,18 @@
webManager.stop();
}
+ protected HttpServer setupMetricsServer(ExternalProperties externalProperties) throws Exception {
+ CCConfig ccConfig = ((ClusterControllerService) ccServiceCtx.getControllerService()).getConfig();
+ DefaultExports.initialize();
+ final HttpServerConfig config =
+ HttpServerConfigBuilder.custom().setMaxRequestSize(externalProperties.getMaxWebRequestSize()).build();
+ HttpServer webServer =
+ new HttpServer(webManager.getBosses(), webManager.getWorkers(), ccConfig.getMetricsPort(), config);
+ webServer.setAttribute(HYRACKS_CONNECTION_ATTR, hcc);
+ webServer.addServlet(new MetricsServlet(webServer.ctx(), "/*"));
+ return webServer;
+ }
+
protected HttpServer setupWebServer(ExternalProperties externalProperties) throws Exception {
final HttpServerConfig config =
HttpServerConfigBuilder.custom().setMaxRequestSize(externalProperties.getMaxWebRequestSize()).build();
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
index c148c92..1105289 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
@@ -18,6 +18,7 @@
*/
package org.apache.asterix.hyracks.bootstrap;
+import static io.prometheus.client.CollectorRegistry.defaultRegistry;
import static org.apache.asterix.api.http.server.ServletConstants.HYRACKS_CONNECTION_ATTR;
import static org.apache.asterix.api.http.server.ServletConstants.SYS_AUTH_HEADER;
import static org.apache.asterix.common.utils.Servlets.QUERY_RESULT;
@@ -39,6 +40,7 @@
import org.apache.asterix.algebra.base.ILangExtension;
import org.apache.asterix.api.http.server.BasicAuthServlet;
+import org.apache.asterix.api.http.server.MetricsServlet;
import org.apache.asterix.api.http.server.NCQueryServiceServlet;
import org.apache.asterix.api.http.server.NCUdfApiServlet;
import org.apache.asterix.api.http.server.NCUdfRecoveryServlet;
@@ -108,6 +110,8 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import io.prometheus.client.hotspot.DefaultExports;
+
public class NCApplication extends BaseNCApplication {
private static final Logger LOGGER = LogManager.getLogger();
protected NCExtensionManager ncExtensionManager;
@@ -183,6 +187,9 @@
LOGGER.info("Stores: " + PrintUtil.toString(metadataProperties.getStores()));
}
webManager = new WebManager();
+
+ defaultRegistry.register(new StorageExporter(
+ ((PersistentLocalResourceRepository) getApplicationContext().getLocalResourceRepository())));
performLocalCleanUp();
}
@@ -235,7 +242,21 @@
auth.getFirst(), auth.getSecond()));
apiServer.addServlet(new QueryStatusApiServlet(apiServer.ctx(), getApplicationContext(), QUERY_STATUS));
apiServer.addServlet(new QueryResultApiServlet(apiServer.ctx(), getApplicationContext(), QUERY_RESULT));
+
webManager.add(apiServer);
+ webManager.add(setupMetricsServer(getApplicationContext().getExternalProperties()));
+ }
+
+ protected HttpServer setupMetricsServer(ExternalProperties externalProperties) throws Exception {
+ NCConfig ncConfig = ((NodeControllerService) ncServiceCtx.getControllerService()).getConfiguration();
+ DefaultExports.initialize();
+ final HttpServerConfig config =
+ HttpServerConfigBuilder.custom().setMaxRequestSize(externalProperties.getMaxWebRequestSize()).build();
+ HttpServer webServer =
+ new HttpServer(webManager.getBosses(), webManager.getWorkers(), ncConfig.getMetricsPort(), config);
+ webServer.setAttribute(HYRACKS_CONNECTION_ATTR, getApplicationContext().getHcc());
+ webServer.addServlet(new MetricsServlet(webServer.ctx(), "/*"));
+ return webServer;
}
protected List<AsterixExtension> getExtensions() throws Exception {
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/StorageExporter.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/StorageExporter.java
new file mode 100644
index 0000000..456e5ff
--- /dev/null
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/StorageExporter.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.asterix.hyracks.bootstrap;
+
+import static com.fasterxml.jackson.databind.MapperFeature.SORT_PROPERTIES_ALPHABETICALLY;
+import static com.fasterxml.jackson.databind.SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.asterix.common.storage.ResourceStorageStats;
+import org.apache.asterix.transaction.management.resource.PersistentLocalResourceRepository;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hyracks.api.exceptions.HyracksDataException;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+
+import io.prometheus.client.Collector;
+import io.prometheus.client.Gauge;
+
+public class StorageExporter extends Collector {
+ private ArrayNode storageStats;
+ private PersistentLocalResourceRepository storageRepository;
+ protected static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ static {
+ OBJECT_MAPPER.enable(SerializationFeature.INDENT_OUTPUT);
+ OBJECT_MAPPER.configure(SORT_PROPERTIES_ALPHABETICALLY, true);
+ OBJECT_MAPPER.configure(ORDER_MAP_ENTRIES_BY_KEYS, true);
+ }
+
+ private static final Gauge storageBytesGauge = Gauge.build().name("asterix_storage_bytes")
+ .help("current size of storage").labelNames("partition", "dataverse", "dataset", "index").register();
+
+ public StorageExporter(PersistentLocalResourceRepository storageRepository) {
+ this.storageStats = OBJECT_MAPPER.createArrayNode();
+ this.storageRepository = storageRepository;
+ }
+
+ @Override
+ public List<MetricFamilySamples> collect() {
+ List<MetricFamilySamples> mfs = new ArrayList<>();
+
+ storageBytesGauge.clear();
+ updateStorageStats();
+ for (int i = 0; i < storageStats.size(); i++) {
+ JsonNode node = storageStats.get(i);
+ String index = node.get("index").toString();
+ String dataset = node.get("dataset").toString();
+ String partition = node.get("partition").toString();
+ Double nodeSize = node.get("totalSize").asDouble();
+ String[] tokens = StringUtils.split(node.get("path").toString(), File.separatorChar);
+ storageBytesGauge.labels(partition, tokens[2], dataset, index).set(nodeSize);
+ }
+
+ return mfs;
+ }
+
+ private void updateStorageStats() {
+ ArrayNode storageStats = OBJECT_MAPPER.createArrayNode();
+
+ try {
+ storageRepository.getStorageStats().stream().map(ResourceStorageStats::asJson).forEach(storageStats::add);
+ this.storageStats = storageStats;
+ } catch (HyracksDataException e) {
+ e.printStackTrace();
+ }
+ }
+}
diff --git a/asterixdb/asterix-app/src/test/resources/cc.conf b/asterixdb/asterix-app/src/test/resources/cc.conf
index 953284964..dd5743a 100644
--- a/asterixdb/asterix-app/src/test/resources/cc.conf
+++ b/asterixdb/asterix-app/src/test/resources/cc.conf
@@ -23,12 +23,13 @@
nc.api.port=19004
#jvm.args=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5006
-[nc/asterix_nc2]
-ncservice.port=9091
-txn.log.dir=target/tmp/asterix_nc2/txnlog
-core.dump.dir=target/tmp/asterix_nc2/coredump
-iodevices=target/tmp/asterix_nc2/iodevice1,../asterix-server/target/tmp/asterix_nc2/iodevice2
-nc.api.port=19005
+; [nc/asterix_nc2]
+; ncservice.port=9091
+; txn.log.dir=target/tmp/asterix_nc2/txnlog
+; core.dump.dir=target/tmp/asterix_nc2/coredump
+; iodevices=target/tmp/asterix_nc2/iodevice1,../asterix-server/target/tmp/asterix_nc2/iodevice2
+; nc.api.port=19005
+; metrics.port=19011
#jvm.args=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5007
[nc]
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/bin/install_monitoring.sh b/asterixdb/asterix-server/src/main/opt/ansible/bin/install_monitoring.sh
new file mode 100755
index 0000000..15e9d4d
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/bin/install_monitoring.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+# Gets the absolute path so that the script can work no matter where it is invoked.
+pushd `dirname $0` > /dev/null
+SCRIPT_PATH=`pwd -P`
+popd > /dev/null
+ANSB_PATH=`dirname "${SCRIPT_PATH}"`
+
+MONITORING_INVENTORY=$ANSB_PATH/conf/monitoring_inventory
+
+# Start installing binaries for asterix monitoring
+export ANSIBLE_HOST_KEY_CHECKING=false
+ansible-playbook -i $MONITORING_INVENTORY $ANSB_PATH/yaml/monitoring_install.yml
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/bin/start_monitoring.sh b/asterixdb/asterix-server/src/main/opt/ansible/bin/start_monitoring.sh
new file mode 100755
index 0000000..2e220e3
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/bin/start_monitoring.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+# Gets the absolute path so that the script can work no matter where it is invoked.
+pushd `dirname $0` > /dev/null
+SCRIPT_PATH=`pwd -P`
+popd > /dev/null
+ANSB_PATH=`dirname "${SCRIPT_PATH}"`
+
+MONITORING_INVENTORY=$ANSB_PATH/conf/monitoring_inventory
+INVENTORY=$ANSB_PATH/conf/inventory
+
+# Start the monitoring binaries
+export ANSIBLE_HOST_KEY_CHECKING=false
+ansible-playbook -i $INVENTORY -i $MONITORING_INVENTORY $ANSB_PATH/yaml/monitoring_start.yml
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/bin/stop_monitoring.sh b/asterixdb/asterix-server/src/main/opt/ansible/bin/stop_monitoring.sh
new file mode 100755
index 0000000..4e2a311
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/bin/stop_monitoring.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+# Gets the absolute path so that the script can work no matter where it is invoked.
+pushd `dirname $0` > /dev/null
+SCRIPT_PATH=`pwd -P`
+popd > /dev/null
+ANSB_PATH=`dirname "${SCRIPT_PATH}"`
+
+MONITORING_INVENTORY=$ANSB_PATH/conf/monitoring_inventory
+
+# Stop the monitoring services
+export ANSIBLE_HOST_KEY_CHECKING=false
+ansible-playbook -i $MONITORING_INVENTORY $ANSB_PATH/yaml/monitoring_stop.yml
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/dashboards.yml b/asterixdb/asterix-server/src/main/opt/ansible/conf/dashboards.yml
new file mode 100644
index 0000000..6eae093
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/dashboards.yml
@@ -0,0 +1,43 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+apiVersion: 1
+
+providers:
+ # <string> an unique provider name. Required
+ - name: 'a unique provider name'
+ # <int> Org id. Default to 1
+ orgId: 1
+ # <string> name of the dashboard folder.
+ folder: ''
+ # <string> folder UID. will be automatically generated if not specified
+ folderUid: ''
+ # <string> provider type. Default to 'file'
+ type: file
+ # <bool> disable dashboard deletion
+ disableDeletion: false
+ # <int> how often Grafana will scan for changed dashboards
+ updateIntervalSeconds: 10
+ # <bool> allow updating provisioned dashboards from the UI
+ allowUiUpdates: false
+ options:
+ # <string, required> path to dashboard files on disk. Required when using the 'file' type
+ path: grafana_dashboards
+ # <bool> use folder names from filesystem to create folders in Grafana
+ foldersFromFilesStructure: true
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/datasources.yml b/asterixdb/asterix-server/src/main/opt/ansible/conf/datasources.yml
new file mode 100644
index 0000000..3d66273
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/datasources.yml
@@ -0,0 +1,54 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+# # config file version
+apiVersion: 1
+
+# # list of datasources to insert/update depending
+# # on what's available in the database
+datasources:
+ # <string, required> name of the datasource. Required
+ - name: Prometheus
+ # <string, required> datasource type. Required
+ type: prometheus
+ # <string, required> access mode. direct or proxy. Required
+ access: proxy
+ # <int> org id. will default to orgId 1 if not specified
+ orgId: 1
+ # <string> url
+ url: http://localhost:8080
+ # <string> database password, if used
+ password:
+ # <string> database user, if used
+ user:
+ # <string> database name, if used
+ database:
+ # <bool> enable/disable basic auth
+ basicAuth:
+ # <string> basic auth username
+ basicAuthUser:
+ # <string> basic auth password
+ basicAuthPassword:
+ # <bool> enable/disable with credentials headers
+ withCredentials:
+ # <bool> mark as default datasource. Max one per org
+ isDefault:
+ version: 1
+ # <bool> allow users to edit datasources from the UI.
+ editable: false
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/defaults.ini b/asterixdb/asterix-server/src/main/opt/ansible/conf/defaults.ini
new file mode 100644
index 0000000..5a11a8b
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/defaults.ini
@@ -0,0 +1,876 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+##################### Grafana Configuration Defaults #####################
+#
+# Do not modify this file in grafana installs
+#
+
+# possible values : production, development
+app_mode = production
+
+# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
+instance_name = ${HOSTNAME}
+
+#################################### Paths ###############################
+[paths]
+# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
+data = data
+
+# Temporary files in `data` directory older than given duration will be removed
+temp_data_lifetime = 24h
+
+# Directory where grafana can store logs
+logs = data/log
+
+# Directory where grafana will automatically scan and look for plugins
+plugins = data/plugins
+
+# folder that contains provisioning config files that grafana will apply on startup and while running.
+provisioning = conf/provisioning
+
+#################################### Server ##############################
+[server]
+# Protocol (http, https, h2, socket)
+protocol = http
+
+# The ip address to bind to, empty will bind to all interfaces
+http_addr =
+
+# The http port to use
+http_port = 3000
+
+# The public facing domain name used to access grafana from a browser
+domain = localhost
+
+# Redirect to correct domain if host header does not match domain
+# Prevents DNS rebinding attacks
+enforce_domain = false
+
+# The full public facing url
+root_url = %(protocol)s://%(domain)s:%(http_port)s/
+
+# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
+serve_from_sub_path = false
+
+# Log web requests
+router_logging = false
+
+# the path relative working path
+static_root_path = public
+
+# enable gzip
+enable_gzip = false
+
+# https certs & key file
+cert_file =
+cert_key =
+
+# Unix socket path
+socket = /tmp/grafana.sock
+
+#################################### Database ############################
+[database]
+# You can configure the database connection by specifying type, host, name, user and password
+# as separate properties or as on string using the url property.
+
+# Either "mysql", "postgres" or "sqlite3", it's your choice
+type = sqlite3
+host = 127.0.0.1:3306
+name = grafana
+user = root
+# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
+password =
+# Use either URL or the previous fields to configure the database
+# Example: mysql://user:secret@host:port/database
+url =
+
+# Max idle conn setting default is 2
+max_idle_conn = 2
+
+# Max conn setting default is 0 (mean not set)
+max_open_conn =
+
+# Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours)
+conn_max_lifetime = 14400
+
+# Set to true to log the sql calls and execution times.
+log_queries =
+
+# For "postgres", use either "disable", "require" or "verify-full"
+# For "mysql", use either "true", "false", or "skip-verify".
+ssl_mode = disable
+
+ca_cert_path =
+client_key_path =
+client_cert_path =
+server_cert_name =
+
+# For "sqlite3" only, path relative to data_path setting
+path = grafana.db
+
+# For "sqlite3" only. cache mode setting used for connecting to the database
+cache_mode = private
+
+#################################### Cache server #############################
+[remote_cache]
+# Either "redis", "memcached" or "database" default is "database"
+type = database
+
+# cache connectionstring options
+# database: will use Grafana primary database.
+# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=0,ssl=false`. Only addr is required. ssl may be 'true', 'false', or 'insecure'.
+# memcache: 127.0.0.1:11211
+connstr =
+
+#################################### Data proxy ###########################
+[dataproxy]
+
+# This enables data proxy logging, default is false
+logging = false
+
+# How long the data proxy waits before timing out, default is 30 seconds.
+# This setting also applies to core backend HTTP data sources where query requests use an HTTP client with timeout set.
+timeout = 30
+
+# How many seconds the data proxy waits before sending a keepalive request.
+keep_alive_seconds = 30
+
+# How many seconds the data proxy waits for a successful TLS Handshake before timing out.
+tls_handshake_timeout_seconds = 10
+
+# How many seconds the data proxy will wait for a server's first response headers after
+# fully writing the request headers if the request has an "Expect: 100-continue"
+# header. A value of 0 will result in the body being sent immediately, without
+# waiting for the server to approve.
+expect_continue_timeout_seconds = 1
+
+# The maximum number of idle connections that Grafana will keep alive.
+max_idle_connections = 100
+
+# How many seconds the data proxy keeps an idle connection open before timing out.
+idle_conn_timeout_seconds = 90
+
+# If enabled and user is not anonymous, data proxy will add X-Grafana-User header with username into the request.
+send_user_header = false
+
+#################################### Analytics ###########################
+[analytics]
+# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
+# No ip addresses are being tracked, only simple counters to track
+# running instances, dashboard and error counts. It is very helpful to us.
+# Change this option to false to disable reporting.
+reporting_enabled = true
+
+# Set to false to disable all checks to https://grafana.com
+# for new versions (grafana itself and plugins), check is used
+# in some UI views to notify that grafana or plugin update exists
+# This option does not cause any auto updates, nor send any information
+# only a GET request to https://grafana.com to get latest versions
+check_for_updates = true
+
+# Google Analytics universal tracking code, only enabled if you specify an id here
+google_analytics_ua_id =
+
+# Google Tag Manager ID, only enabled if you specify an id here
+google_tag_manager_id =
+
+#################################### Security ############################
+[security]
+# disable creation of admin user on first start of grafana
+disable_initial_admin_creation = false
+
+# default admin user, created on startup
+admin_user = admin
+
+# default admin password, can be changed before first start of grafana, or in profile settings
+admin_password = admin
+
+# used for signing
+secret_key = SW2YcwTIb9zpOOhoPsMm
+
+# disable gravatar profile images
+disable_gravatar = false
+
+# data source proxy whitelist (ip_or_domain:port separated by spaces)
+data_source_proxy_whitelist =
+
+# disable protection against brute force login attempts
+disable_brute_force_login_protection = false
+
+# set to true if you host Grafana behind HTTPS. default is false.
+cookie_secure = false
+
+# set cookie SameSite attribute. defaults to `lax`. can be set to "lax", "strict", "none" and "disabled"
+cookie_samesite = lax
+
+# set to true if you want to allow browsers to render Grafana in a <frame>, <iframe>, <embed> or <object>. default is false.
+allow_embedding = false
+
+# Set to true if you want to enable http strict transport security (HSTS) response header.
+# This is only sent when HTTPS is enabled in this configuration.
+# HSTS tells browsers that the site should only be accessed using HTTPS.
+strict_transport_security = false
+
+# Sets how long a browser should cache HSTS. Only applied if strict_transport_security is enabled.
+strict_transport_security_max_age_seconds = 86400
+
+# Set to true if to enable HSTS preloading option. Only applied if strict_transport_security is enabled.
+strict_transport_security_preload = false
+
+# Set to true if to enable the HSTS includeSubDomains option. Only applied if strict_transport_security is enabled.
+strict_transport_security_subdomains = false
+
+# Set to true to enable the X-Content-Type-Options response header.
+# The X-Content-Type-Options response HTTP header is a marker used by the server to indicate that the MIME types advertised
+# in the Content-Type headers should not be changed and be followed.
+x_content_type_options = true
+
+# Set to true to enable the X-XSS-Protection header, which tells browsers to stop pages from loading
+# when they detect reflected cross-site scripting (XSS) attacks.
+x_xss_protection = true
+
+
+#################################### Snapshots ###########################
+[snapshots]
+# snapshot sharing options
+external_enabled = true
+external_snapshot_url = https://snapshots-origin.raintank.io
+external_snapshot_name = Publish to snapshot.raintank.io
+
+# Set to true to enable this Grafana instance act as an external snapshot server and allow unauthenticated requests for
+# creating and deleting snapshots.
+public_mode = false
+
+# remove expired snapshot
+snapshot_remove_expired = true
+
+#################################### Dashboards ##################
+
+[dashboards]
+# Number dashboard versions to keep (per dashboard). Default: 20, Minimum: 1
+versions_to_keep = 20
+
+# Minimum dashboard refresh interval. When set, this will restrict users to set the refresh interval of a dashboard lower than given interval. Per default this is 5 seconds.
+# The interval string is a possibly signed sequence of decimal numbers, followed by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
+min_refresh_interval = 5s
+
+# Path to the default home dashboard. If this value is empty, then Grafana uses StaticRootPath + "dashboards/home.json"
+default_home_dashboard_path = grafana_dashboards/asterix_system_stats.json
+
+#################################### Users ###############################
+[users]
+# disable user signup / registration
+allow_sign_up = false
+
+# Allow non admin users to create organizations
+allow_org_create = false
+
+# Set to true to automatically assign new users to the default organization (id 1)
+auto_assign_org = true
+
+# Set this value to automatically add new users to the provided organization (if auto_assign_org above is set to true)
+auto_assign_org_id = 1
+
+# Default role new users will be automatically assigned (if auto_assign_org above is set to true)
+auto_assign_org_role = Viewer
+
+# Require email validation before sign up completes
+verify_email_enabled = false
+
+# Background text for the user field on the login page
+login_hint = email or username
+password_hint = password
+
+# Default UI theme ("dark" or "light")
+default_theme = dark
+
+# External user management
+external_manage_link_url =
+external_manage_link_name =
+external_manage_info =
+
+# Viewers can edit/inspect dashboard settings in the browser. But not save the dashboard.
+viewers_can_edit = false
+
+# Editors can administrate dashboard, folders and teams they create
+editors_can_admin = false
+
+# The duration in time a user invitation remains valid before expiring. This setting should be expressed as a duration. Examples: 6h (hours), 2d (days), 1w (week). Default is 24h (24 hours). The minimum supported duration is 15m (15 minutes).
+user_invite_max_lifetime_duration = 24h
+
+[auth]
+# Login cookie name
+login_cookie_name = grafana_session
+
+# The maximum lifetime (duration) an authenticated user can be inactive before being required to login at next visit. Default is 7 days (7d). This setting should be expressed as a duration, e.g. 5m (minutes), 6h (hours), 10d (days), 2w (weeks), 1M (month). The lifetime resets at each successful token rotation (token_rotation_interval_minutes).
+login_maximum_inactive_lifetime_duration =
+
+# The maximum lifetime (duration) an authenticated user can be logged in since login time before being required to login. Default is 30 days (30d). This setting should be expressed as a duration, e.g. 5m (minutes), 6h (hours), 10d (days), 2w (weeks), 1M (month).
+login_maximum_lifetime_duration =
+
+# How often should auth tokens be rotated for authenticated users when being active. The default is each 10 minutes.
+token_rotation_interval_minutes = 10
+
+# Set to true to disable (hide) the login form, useful if you use OAuth
+disable_login_form = false
+
+# Set to true to disable the signout link in the side menu. useful if you use auth.proxy
+disable_signout_menu = false
+
+# URL to redirect the user to after sign out
+signout_redirect_url =
+
+# Set to true to attempt login with OAuth automatically, skipping the login screen.
+# This setting is ignored if multiple OAuth providers are configured.
+oauth_auto_login = false
+
+# OAuth state max age cookie duration in seconds. Defaults to 600 seconds.
+oauth_state_cookie_max_age = 600
+
+# limit of api_key seconds to live before expiration
+api_key_max_seconds_to_live = -1
+
+# Set to true to enable SigV4 authentication option for HTTP-based datasources
+sigv4_auth_enabled = false
+
+#################################### Anonymous Auth ######################
+[auth.anonymous]
+# enable anonymous access
+enabled = false
+
+# specify organization name that should be used for unauthenticated users
+org_name = Main Org.
+
+# specify role for unauthenticated users
+org_role = Viewer
+
+# mask the Grafana version number for unauthenticated users
+hide_version = false
+
+#################################### GitHub Auth #########################
+[auth.github]
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = user:email,read:org
+auth_url = https://github.com/login/oauth/authorize
+token_url = https://github.com/login/oauth/access_token
+api_url = https://api.github.com/user
+allowed_domains =
+team_ids =
+allowed_organizations =
+
+#################################### GitLab Auth #########################
+[auth.gitlab]
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = api
+auth_url = https://gitlab.com/oauth/authorize
+token_url = https://gitlab.com/oauth/token
+api_url = https://gitlab.com/api/v4
+allowed_domains =
+allowed_groups =
+
+#################################### Google Auth #########################
+[auth.google]
+enabled = false
+allow_sign_up = true
+client_id = some_client_id
+client_secret =
+scopes = https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
+auth_url = https://accounts.google.com/o/oauth2/auth
+token_url = https://accounts.google.com/o/oauth2/token
+api_url = https://www.googleapis.com/oauth2/v1/userinfo
+allowed_domains =
+hosted_domain =
+
+#################################### Grafana.com Auth ####################
+# legacy key names (so they work in env variables)
+[auth.grafananet]
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = user:email
+allowed_organizations =
+
+[auth.grafana_com]
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = user:email
+allowed_organizations =
+
+#################################### Azure AD OAuth #######################
+[auth.azuread]
+name = Azure AD
+enabled = false
+allow_sign_up = true
+client_id = some_client_id
+client_secret =
+scopes = openid email profile
+auth_url = https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/authorize
+token_url = https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token
+allowed_domains =
+allowed_groups =
+
+#################################### Okta OAuth #######################
+[auth.okta]
+name = Okta
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = openid profile email groups
+auth_url = https://<tenant-id>.okta.com/oauth2/v1/authorize
+token_url = https://<tenant-id>.okta.com/oauth2/v1/token
+api_url = https://<tenant-id>.okta.com/oauth2/v1/userinfo
+allowed_domains =
+allowed_groups =
+role_attribute_path =
+
+#################################### Generic OAuth #######################
+[auth.generic_oauth]
+name = OAuth
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = user:email
+email_attribute_name = email:primary
+email_attribute_path =
+login_attribute_path =
+role_attribute_path =
+id_token_attribute_name =
+auth_url =
+token_url =
+api_url =
+allowed_domains =
+team_ids =
+allowed_organizations =
+tls_skip_verify_insecure = false
+tls_client_cert =
+tls_client_key =
+tls_client_ca =
+
+#################################### Basic Auth ##########################
+[auth.basic]
+enabled = true
+
+#################################### Auth Proxy ##########################
+[auth.proxy]
+enabled = false
+header_name = X-WEBAUTH-USER
+header_property = username
+auto_sign_up = true
+# Deprecated, use sync_ttl instead
+ldap_sync_ttl = 60
+sync_ttl = 60
+whitelist =
+headers =
+enable_login_token = false
+
+#################################### Auth LDAP ###########################
+[auth.ldap]
+enabled = false
+config_file = /etc/grafana/ldap.toml
+allow_sign_up = true
+
+# LDAP backround sync (Enterprise only)
+# At 1 am every day
+sync_cron = "0 0 1 * * *"
+active_sync_enabled = true
+
+#################################### SMTP / Emailing #####################
+[smtp]
+enabled = false
+host = localhost:25
+user =
+# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
+password =
+cert_file =
+key_file =
+skip_verify = false
+from_address = admin@grafana.localhost
+from_name = Grafana
+ehlo_identity =
+startTLS_policy =
+
+[emails]
+welcome_email_on_sign_up = false
+templates_pattern = emails/*.html
+
+#################################### Logging ##########################
+[log]
+# Either "console", "file", "syslog". Default is console and file
+# Use space to separate multiple modes, e.g. "console file"
+mode = console file
+
+# Either "debug", "info", "warn", "error", "critical", default is "info"
+level = info
+
+# optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug
+filters =
+
+# For "console" mode only
+[log.console]
+level =
+
+# log line format, valid options are text, console and json
+format = console
+
+# For "file" mode only
+[log.file]
+level =
+
+# log line format, valid options are text, console and json
+format = text
+
+# This enables automated log rotate(switch of following options), default is true
+log_rotate = true
+
+# Max line number of single file, default is 1000000
+max_lines = 1000000
+
+# Max size shift of single file, default is 28 means 1 << 28, 256MB
+max_size_shift = 28
+
+# Segment log daily, default is true
+daily_rotate = true
+
+# Expired days of log file(delete after max days), default is 7
+max_days = 7
+
+[log.syslog]
+level =
+
+# log line format, valid options are text, console and json
+format = text
+
+# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
+network =
+address =
+
+# Syslog facility. user, daemon and local0 through local7 are valid.
+facility =
+
+# Syslog tag. By default, the process' argv[0] is used.
+tag =
+
+#################################### Usage Quotas ########################
+[quota]
+enabled = false
+
+#### set quotas to -1 to make unlimited. ####
+# limit number of users per Org.
+org_user = 10
+
+# limit number of dashboards per Org.
+org_dashboard = 100
+
+# limit number of data_sources per Org.
+org_data_source = 10
+
+# limit number of api_keys per Org.
+org_api_key = 10
+
+# limit number of orgs a user can create.
+user_org = 10
+
+# Global limit of users.
+global_user = -1
+
+# global limit of orgs.
+global_org = -1
+
+# global limit of dashboards
+global_dashboard = -1
+
+# global limit of api_keys
+global_api_key = -1
+
+# global limit on number of logged in users.
+global_session = -1
+
+#################################### Alerting ############################
+[alerting]
+# Disable alerting engine & UI features
+enabled = true
+# Makes it possible to turn off alert rule execution but alerting UI is visible
+execute_alerts = true
+
+# Default setting for new alert rules. Defaults to categorize error and timeouts as alerting. (alerting, keep_state)
+error_or_timeout = alerting
+
+# Default setting for how Grafana handles nodata or null values in alerting. (alerting, no_data, keep_state, ok)
+nodata_or_nullvalues = no_data
+
+# Alert notifications can include images, but rendering many images at the same time can overload the server
+# This limit will protect the server from render overloading and make sure notifications are sent out quickly
+concurrent_render_limit = 5
+
+# Default setting for alert calculation timeout. Default value is 30
+evaluation_timeout_seconds = 30
+
+# Default setting for alert notification timeout. Default value is 30
+notification_timeout_seconds = 30
+
+# Default setting for max attempts to sending alert notifications. Default value is 3
+max_attempts = 3
+
+# Makes it possible to enforce a minimal interval between evaluations, to reduce load on the backend
+min_interval_seconds = 1
+
+# Configures for how long alert annotations are stored. Default is 0, which keeps them forever.
+# This setting should be expressed as an duration. Ex 6h (hours), 10d (days), 2w (weeks), 1M (month).
+max_annotation_age =
+
+# Configures max number of alert annotations that Grafana stores. Default value is 0, which keeps all alert annotations.
+max_annotations_to_keep =
+
+#################################### Annotations #########################
+
+[annotations.dashboard]
+# Dashboard annotations means that annotations are associated with the dashboard they are created on.
+
+# Configures how long dashboard annotations are stored. Default is 0, which keeps them forever.
+# This setting should be expressed as a duration. Examples: 6h (hours), 10d (days), 2w (weeks), 1M (month).
+max_age =
+
+# Configures max number of dashboard annotations that Grafana stores. Default value is 0, which keeps all dashboard annotations.
+max_annotations_to_keep =
+
+[annotations.api]
+# API annotations means that the annotations have been created using the API without any
+# association with a dashboard.
+
+# Configures how long Grafana stores API annotations. Default is 0, which keeps them forever.
+# This setting should be expressed as a duration. Examples: 6h (hours), 10d (days), 2w (weeks), 1M (month).
+max_age =
+
+# Configures max number of API annotations that Grafana keeps. Default value is 0, which keeps all API annotations.
+max_annotations_to_keep =
+
+#################################### Explore #############################
+[explore]
+# Enable the Explore section
+enabled = true
+
+#################################### Internal Grafana Metrics ############
+# Metrics available at HTTP API Url /metrics
+[metrics]
+enabled = true
+interval_seconds = 10
+# Disable total stats (stat_totals_*) metrics to be generated
+disable_total_stats = false
+
+#If both are set, basic auth will be required for the metrics endpoint.
+basic_auth_username =
+basic_auth_password =
+
+# Metrics environment info adds dimensions to the `grafana_environment_info` metric, which
+# can expose more information about the Grafana instance.
+[metrics.environment_info]
+#exampleLabel1 = exampleValue1
+#exampleLabel2 = exampleValue2
+
+# Send internal Grafana metrics to graphite
+[metrics.graphite]
+# Enable by setting the address setting (ex localhost:2003)
+address =
+prefix = prod.grafana.%(instance_name)s.
+
+#################################### Grafana.com integration ##########################
+[grafana_net]
+url = https://grafana.com
+
+[grafana_com]
+url = https://grafana.com
+
+#################################### Distributed tracing ############
+[tracing.jaeger]
+# jaeger destination (ex localhost:6831)
+address =
+# tag that will always be included in when creating new spans. ex (tag1:value1,tag2:value2)
+always_included_tag =
+# Type specifies the type of the sampler: const, probabilistic, rateLimiting, or remote
+sampler_type = const
+# jaeger samplerconfig param
+# for "const" sampler, 0 or 1 for always false/true respectively
+# for "probabilistic" sampler, a probability between 0 and 1
+# for "rateLimiting" sampler, the number of spans per second
+# for "remote" sampler, param is the same as for "probabilistic"
+# and indicates the initial sampling rate before the actual one
+# is received from the mothership
+sampler_param = 1
+# Whether or not to use Zipkin span propagation (x-b3- HTTP headers).
+zipkin_propagation = false
+# Setting this to true disables shared RPC spans.
+# Not disabling is the most common setting when using Zipkin elsewhere in your infrastructure.
+disable_shared_zipkin_spans = false
+
+#################################### External Image Storage ##############
+[external_image_storage]
+# Used for uploading images to public servers so they can be included in slack/email messages.
+# You can choose between (s3, webdav, gcs, azure_blob, local)
+provider =
+
+[external_image_storage.s3]
+endpoint =
+path_style_access =
+bucket_url =
+bucket =
+region =
+path =
+access_key =
+secret_key =
+
+[external_image_storage.webdav]
+url =
+username =
+password =
+public_url =
+
+[external_image_storage.gcs]
+key_file =
+bucket =
+path =
+enable_signed_urls = false
+signed_url_expiration =
+
+[external_image_storage.azure_blob]
+account_name =
+account_key =
+container_name =
+
+[external_image_storage.local]
+# does not require any configuration
+
+[rendering]
+# Options to configure a remote HTTP image rendering service, e.g. using https://github.com/grafana/grafana-image-renderer.
+# URL to a remote HTTP image renderer service, e.g. http://localhost:8081/render, will enable Grafana to render panels and dashboards to PNG-images using HTTP requests to an external service.
+server_url =
+# If the remote HTTP image renderer service runs on a different server than the Grafana server you may have to configure this to a URL where Grafana is reachable, e.g. http://grafana.domain/.
+callback_url =
+# Concurrent render request limit affects when the /render HTTP endpoint is used. Rendering many images at the same time can overload the server,
+# which this setting can help protect against by only allowing a certain amount of concurrent requests.
+concurrent_render_request_limit = 30
+
+[panels]
+# here for to support old env variables, can remove after a few months
+enable_alpha = false
+disable_sanitize_html = false
+
+[plugins]
+enable_alpha = false
+app_tls_skip_verify_insecure = false
+# Enter a comma-separated list of plugin identifiers to identify plugins that are allowed to be loaded even if they lack a valid signature.
+allow_loading_unsigned_plugins =
+
+#################################### Grafana Image Renderer Plugin ##########################
+[plugin.grafana-image-renderer]
+# Instruct headless browser instance to use a default timezone when not provided by Grafana, e.g. when rendering panel image of alert.
+# See ICU’s metaZones.txt (https://cs.chromium.org/chromium/src/third_party/icu/source/data/misc/metaZones.txt) for a list of supported
+# timezone IDs. Fallbacks to TZ environment variable if not set.
+rendering_timezone =
+
+# Instruct headless browser instance to use a default language when not provided by Grafana, e.g. when rendering panel image of alert.
+# Please refer to the HTTP header Accept-Language to understand how to format this value, e.g. 'fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5'.
+rendering_language =
+
+# Instruct headless browser instance to use a default device scale factor when not provided by Grafana, e.g. when rendering panel image of alert.
+# Default is 1. Using a higher value will produce more detailed images (higher DPI), but will require more disk space to store an image.
+rendering_viewport_device_scale_factor =
+
+# Instruct headless browser instance whether to ignore HTTPS errors during navigation. Per default HTTPS errors are not ignored. Due to
+# the security risk it's not recommended to ignore HTTPS errors.
+rendering_ignore_https_errors =
+
+# Instruct headless browser instance whether to capture and log verbose information when rendering an image. Default is false and will
+# only capture and log error messages. When enabled, debug messages are captured and logged as well.
+# For the verbose information to be included in the Grafana server log you have to adjust the rendering log level to debug, configure
+# [log].filter = rendering:debug.
+rendering_verbose_logging =
+
+# Instruct headless browser instance whether to output its debug and error messages into running process of remote rendering service.
+# Default is false. This can be useful to enable (true) when troubleshooting.
+rendering_dumpio =
+
+# Additional arguments to pass to the headless browser instance. Default is --no-sandbox. The list of Chromium flags can be found
+# here (https://peter.sh/experiments/chromium-command-line-switches/). Multiple arguments is separated with comma-character.
+rendering_args =
+
+# You can configure the plugin to use a different browser binary instead of the pre-packaged version of Chromium.
+# Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not
+# compatible with the plugin.
+rendering_chrome_bin =
+
+# Instruct how headless browser instances are created. Default is 'default' and will create a new browser instance on each request.
+# Mode 'clustered' will make sure that only a maximum of browsers/incognito pages can execute concurrently.
+# Mode 'reusable' will have one browser instance and will create a new incognito page on each request.
+rendering_mode =
+
+# When rendering_mode = clustered you can instruct how many browsers or incognito pages can execute concurrently. Default is 'browser'
+# and will cluster using browser instances.
+# Mode 'context' will cluster using incognito pages.
+rendering_clustering_mode =
+# When rendering_mode = clustered you can define maximum number of browser instances/incognito pages that can execute concurrently..
+rendering_clustering_max_concurrency =
+
+# Limit the maximum viewport width, height and device scale factor that can be requested.
+rendering_viewport_max_width =
+rendering_viewport_max_height =
+rendering_viewport_max_device_scale_factor =
+
+# Change the listening host and port of the gRPC server. Default host is 127.0.0.1 and default port is 0 and will automatically assign
+# a port not in use.
+grpc_host =
+grpc_port =
+
+[enterprise]
+license_path =
+
+[feature_toggles]
+# enable features, separated by spaces
+enable =
+
+[date_formats]
+# For information on what formatting patterns that are supported https://momentjs.com/docs/#/displaying/
+
+# Default system date format used in time range picker and other places where full time is displayed
+full_date = YYYY-MM-DD HH:mm:ss
+
+# Used by graph and other places where we only show small intervals
+interval_second = HH:mm:ss
+interval_minute = HH:mm
+interval_hour = MM/DD HH:mm
+interval_day = MM/DD
+interval_month = YYYY-MM
+interval_year = YYYY
+
+# Experimental feature
+use_browser_locale = false
+
+# Default timezone for user preferences. Options are 'browser' for the browser local timezone or a timezone name from IANA Time Zone database, e.g. 'UTC' or 'Europe/Amsterdam' etc.
+default_timezone = browser
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_jobs_stats.json b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_jobs_stats.json
new file mode 100644
index 0000000..99c6c09
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_jobs_stats.json
@@ -0,0 +1,542 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "iteration": 1621011410751,
+ "links": [
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "title": "ClusterController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "title": "NodeController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "system"
+ ],
+ "type": "dashboards"
+ }
+ ],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the number of jobs that were completed in the specified time interval (30 seconds).",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "jobs"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 2,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "increase(asterix_job_total_count{instance=~\"$cluster\", status=~\"$status\"}[30s])",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, status={{status}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Job Completions in last 30 Seconds",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "jobs",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the count of jobs of running and pending jobs at the specific time.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "align": null,
+ "filterable": false
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green",
+ "value": null
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ },
+ "unit": "jobs"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 4,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "asterix_job_count{instance=~\"$cluster\", status=~\"$status\"}",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, status={{status}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Job Count for Running and Pending Jobs",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "jobs",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the increase in the amount of time jobs are taking in the time interval (which is 30 seconds). This metric is only for archived jobs as we need the starting and ending time.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 8
+ },
+ "hiddenSeries": false,
+ "id": 6,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "increase(asterix_job_total_milliseconds{instance=~\"$cluster\", status=~\"$status\"}[30s]) / 1000",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, status={{status}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Cumulative Job Time Increase",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "s",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the average amount of time each job is taking. If no jobs are running, it will show 0. This metric is only for archived jobs as we need the starting and ending times.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 8
+ },
+ "hiddenSeries": false,
+ "id": 8,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null as zero",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "(sum(rate(asterix_job_total_milliseconds{instance=~\"$cluster\", status=~\"$status\"}[30s])) by (status,instance)) /\n(sum(rate(asterix_job_total_count{instance=~\"$cluster\", status=~\"$status\"}[30s])) by (status, instance)) / 1000",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, status={{status}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Average Time Taken Per Job",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "s",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": "10s",
+ "schemaVersion": 26,
+ "style": "dark",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "templating": {
+ "list": [
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "localhost:19007"
+ ],
+ "value": [
+ "localhost:19007"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_job_total_count, instance)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "cluster",
+ "options": [],
+ "query": "label_values(asterix_job_total_count, instance)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_job_total_count, status)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "status",
+ "options": [],
+ "query": "label_values(asterix_job_total_count, status)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ }
+ ]
+ },
+ "time": {
+ "from": "now-15m",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "",
+ "title": "Asterix Job Stats",
+ "uid": "e7Z7loPMk",
+ "version": 1
+}
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_memory_stats.json b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_memory_stats.json
new file mode 100644
index 0000000..d48548c
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_memory_stats.json
@@ -0,0 +1,167 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 5,
+ "links": [
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "title": "ClusterController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "title": "NodeController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "system"
+ ],
+ "type": "dashboards"
+ }
+ ],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "decbytes"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 2,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "asterix_cluster_memory_bytes",
+ "interval": "",
+ "legendFormat": "",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Memory",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "decbytes",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": "10s",
+ "schemaVersion": 26,
+ "style": "dark",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-15m",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "",
+ "title": "Asterix Memory Stats",
+ "uid": "erI4XTEGz",
+ "version": 3
+}
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_storage_stats.json b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_storage_stats.json
new file mode 100644
index 0000000..2cbb63f
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_storage_stats.json
@@ -0,0 +1,488 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "iteration": 1618605669241,
+ "links": [
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "title": "ClusterController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "title": "NodeController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "system"
+ ],
+ "type": "dashboards"
+ }
+ ],
+ "panels": [
+ {
+ "datasource": "Prometheus",
+ "description": "This panel shows the number of bytes Asterix used in storage for each dataset, index, partition.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green",
+ "value": null
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ },
+ "unit": "decbytes"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 2,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "none",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "last"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "7.3.5",
+ "targets": [
+ {
+ "expr": "sum(asterix_storage_bytes{instance=~\"$node\", partition=~\"$partition\", dataverse=~\"$dataverse\", dataset=~\"\\\"$dataset\\\"\", index=~\"\\\"$index\\\"\"})",
+ "instant": false,
+ "interval": "",
+ "legendFormat": "instance={{instance}}, dataset={{dataset}}, index={{index}}, partition={{partition}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "timeFrom": null,
+ "timeShift": null,
+ "title": "Asterix Storage Size",
+ "type": "stat"
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the breakdown of storage for a given node, partition, dataverse, dataset, and index",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "decbytes"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 10
+ },
+ "hiddenSeries": false,
+ "id": 6,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "asterix_storage_bytes{instance=~\"$node\", partition=~\"$partition\", dataverse=~\"$dataverse\", dataset=~\"\\\"$dataset\\\"\", index=~\"\\\"$index\\\"\"}",
+ "interval": "",
+ "legendFormat": "node={{instance}}, partition={{partition}}, dataverse={{dataverse}}, dataset={{dataset}}, index={{index}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Storage Size Breakdown",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "$$hashKey": "object:113",
+ "format": "decbytes",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "$$hashKey": "object:114",
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the rate of increase of the storage sizes of the composite key of dataset, index, and partition. It uses a least squares regression of all the values from the last 30 seconds to calculate the slope",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "Bps"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 10
+ },
+ "hiddenSeries": false,
+ "id": 4,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "deriv(asterix_storage_bytes{instance=~\"$node\", partition=~\"$partition\", dataverse=~\"$dataverse\", dataset=~\"\\\"$dataset\\\"\", index=~\"\\\"$index\\\"\"}[5m])",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, dataset={{dataset}}, index={{index}}, partition={{partition}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Storage Size Rate of Increase/Decrease in the last 5 Minutes",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "Bps",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": "5m",
+ "schemaVersion": 26,
+ "style": "dark",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "templating": {
+ "list": [
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, instance)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "node",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, instance)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, partition)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "partition",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, partition)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, dataverse)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "dataverse",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, dataverse)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": "",
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, dataset)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "dataset",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, dataset)",
+ "refresh": 1,
+ "regex": "\"([^\"]*)\"",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, index)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "index",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, index)",
+ "refresh": 1,
+ "regex": "\"([^\"]*)\"",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ }
+ ]
+ },
+ "time": {
+ "from": "now-5m",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "",
+ "title": "Asterix Storage Stats",
+ "uid": "RSzD_oEMz",
+ "version": 1
+}
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_system_stats.json b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_system_stats.json
new file mode 100644
index 0000000..f8a3da1
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_system_stats.json
@@ -0,0 +1,479 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 2,
+ "links": [
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "title": "ClusterController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "title": "NodeController Dashboards",
+ "type": "dashboards"
+ }
+ ],
+ "panels": [
+ {
+ "collapsed": false,
+ "datasource": null,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 10,
+ "panels": [],
+ "title": "CPU Usage",
+ "type": "row"
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the Asterix cpu usage",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "percent"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 0,
+ "y": 1
+ },
+ "hiddenSeries": false,
+ "id": 2,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "rate(process_cpu_seconds_total[1m]) * 100",
+ "interval": "",
+ "legendFormat": "{{instance}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix CPU Usage Percentage",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "percent",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the system cpu usage (per cpu)",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "percent"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 1
+ },
+ "hiddenSeries": false,
+ "id": 6,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "(1-rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100",
+ "interval": "",
+ "legendFormat": "CPU {{cpu}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "System CPU Usage",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "percent",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the proportion of non idle cpu that Asterix is using",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "percent"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 10
+ },
+ "hiddenSeries": false,
+ "id": 8,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "(rate(process_cpu_seconds_total[1m]) * 100) / ignoring (instance, job) group_left avg(1-rate(node_cpu_seconds_total{mode=\"idle\"}[1m]))",
+ "interval": "",
+ "legendFormat": "{{instance}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Non Idle CPU Usage",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "percent",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "collapsed": false,
+ "datasource": null,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 18
+ },
+ "id": 4,
+ "panels": [],
+ "title": "File Descriptor Usage",
+ "type": "row"
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the Asterix file descriptor usage (number of file descriptors open / pre set file descriptor max set by Prometheus)",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "percent"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 14,
+ "w": 23,
+ "x": 0,
+ "y": 19
+ },
+ "hiddenSeries": false,
+ "id": 12,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "(process_open_fds / process_max_fds) * 100",
+ "interval": "",
+ "legendFormat": "{{instance}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "File Descriptor Usage",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "percent",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": "10s",
+ "schemaVersion": 26,
+ "style": "dark",
+ "tags": [
+ "asterix",
+ "system"
+ ],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-15m",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "",
+ "title": "Asterix System Stats",
+ "uid": "l0Rj8oEMk",
+ "version": 9
+}
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/instance/cc.conf b/asterixdb/asterix-server/src/main/opt/ansible/conf/instance/cc.conf
new file mode 100644
index 0000000..d9360bf
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/instance/cc.conf
@@ -0,0 +1,32 @@
+; Licensed to the Apache Software Foundation (ASF) under one
+; or more contributor license agreements. See the NOTICE file
+; distributed with this work for additional information
+; regarding copyright ownership. The ASF licenses this file
+; to you under the Apache License, Version 2.0 (the
+; "License"); you may not use this file except in compliance
+; with the License. You may obtain a copy of the License at
+;
+; http://www.apache.org/licenses/LICENSE-2.0
+;
+; Unless required by applicable law or agreed to in writing,
+; software distributed under the License is distributed on an
+; "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+; KIND, either express or implied. See the License for the
+; specific language governing permissions and limitations
+; under the License.
+
+[common]
+log.level=INFO
+log.dir=logs
+
+[nc]
+txn.log.dir=txnlog
+iodevices=iodevice
+command=asterixnc
+
+[nc/1]
+address=localhost
+
+[cc]
+address=localhost
+
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml b/asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml
index b5cd1ae..4eb42a4 100644
--- a/asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml
@@ -54,3 +54,25 @@
# The port for Yourkit profiling
profiler_port: 20002
+
+# The version of the node exporter
+node_exporter_version: 1.0.1
+
+# The version of the prometheus
+prometheus_version: 2.22.1
+
+# The version of the grafana
+grafana_version: 7.3.5
+
+# The os to download monitoring binaries
+host_os: darwin-amd64
+
+# ports for cc
+cc_ports:
+ localhost:
+ - "19007"
+
+# ports for nc
+nc_ports:
+ localhost:
+ - "19010"
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/inventory b/asterixdb/asterix-server/src/main/opt/ansible/conf/inventory
index 140ed43..d3d7079 100644
--- a/asterixdb/asterix-server/src/main/opt/ansible/conf/inventory
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/inventory
@@ -24,4 +24,4 @@
localhost
[ncs]
-localhost
+localhost
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/monitoring_inventory b/asterixdb/asterix-server/src/main/opt/ansible/conf/monitoring_inventory
new file mode 100644
index 0000000..354d6bf
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/monitoring_inventory
@@ -0,0 +1,25 @@
+; Licensed to the Apache Software Foundation (ASF) under one
+; or more contributor license agreements. See the NOTICE file
+; distributed with this work for additional information
+; regarding copyright ownership. The ASF licenses this file
+; to you under the Apache License, Version 2.0 (the
+; "License"); you may not use this file except in compliance
+; with the License. You may obtain a copy of the License at
+;
+; http://www.apache.org/licenses/LICENSE-2.0
+;
+; Unless required by applicable law or agreed to in writing,
+; software distributed under the License is distributed on an
+; "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+; KIND, either express or implied. See the License for the
+; specific language governing permissions and limitations
+; under the License.
+
+[prometheus]
+localhost
+
+[node_exporter]
+localhost
+
+[grafana]
+localhost
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/prometheus.conf.j2 b/asterixdb/asterix-server/src/main/opt/ansible/conf/prometheus.conf.j2
new file mode 100644
index 0000000..e5e549f
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/prometheus.conf.j2
@@ -0,0 +1,62 @@
+{#
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+#}
+
+global:
+ scrape_interval: 1s
+
+scrape_configs:
+ - job_name: 'prometheus'
+ scrape_interval: 2s
+ static_configs:
+ - targets: ['localhost:8080']
+
+ - job_name: 'node_exporter'
+ scrape_interval: 2s
+ static_configs:
+ {%- for host in groups['node_exporter'] %}
+
+ - targets: ['{{ host }}:9100']
+
+ {%- endfor %}
+
+ - job_name: 'cluster_controller'
+ scrape_interval: 5s
+ static_configs:
+ - targets:
+ {%- for host in groups['ncs'] %}
+ {%- for port in cc_ports[host] %}
+
+ - '{{ host }}:{{ port }}'
+
+ {%- endfor %}
+ {%- endfor %}
+
+ - job_name: 'node_controller'
+ scrape_interval: 5s
+ static_configs:
+ - targets:
+ {%- for host in groups['ncs'] %}
+ {%- for port in nc_ports[host] %}
+
+ - '{{ host }}:{{ port }}'
+
+ {%- endfor %}
+ {%- endfor %}
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/grafana.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/grafana.yml
new file mode 100644
index 0000000..58a8034
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/grafana.yml
@@ -0,0 +1,27 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- name: "Grafana server started"
+ shell: |
+ nohup grafana-{{ grafana_version }}/bin/grafana-server --config=conf/defaults.ini --homepath=grafana-{{ grafana_version }} >> "logs/grafana.log" 2>&1 &
+ sleep 1
+ args:
+ chdir: "{{ binarydir }}"
+ async: 10
+ poll: 0
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_install.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_install.yml
new file mode 100644
index 0000000..41f9f80
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_install.yml
@@ -0,0 +1,86 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- hosts: all
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+ - name: Ensure the binarydir exists
+ file:
+ path: "{{ binarydir }}"
+ state: directory
+
+ - name: Ensure the log directory exists
+ file:
+ path: "{{ binarydir }}/logs"
+ state: directory
+
+ - name: Ensure the conf directory exists
+ file:
+ path: "{{ binarydir }}/conf"
+ state: directory
+
+- hosts: node_exporter
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+
+ - name: Install prometheus node exporter
+ unarchive:
+ src: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.{{ host_os }}.tar.gz"
+ dest: "{{ binarydir }}"
+ remote_src: yes
+
+- hosts: prometheus
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+
+ - name: Install prometheus
+ unarchive:
+ src: "https://github.com/prometheus/prometheus/releases/download/v{{ prometheus_version }}/prometheus-{{ prometheus_version }}.{{ host_os }}.tar.gz"
+ dest: "{{ binarydir }}"
+ remote_src: yes
+
+- hosts: grafana
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+
+ - name: Install grafana
+ unarchive:
+ src: "https://dl.grafana.com/oss/release/grafana-{{ grafana_version }}.{{ host_os }}.tar.gz"
+ dest: "{{ binarydir }}"
+ remote_src: yes
+
+ - name: Copy grafana config file
+ copy:
+ src: ../conf/defaults.ini
+ dest: "{{ binarydir }}/conf"
+
+ - name: Copy grafana datasources provisioning
+ copy:
+ src: ../conf/datasources.yml
+ dest: "{{ binarydir }}/grafana-{{ grafana_version }}/conf/provisioning/datasources"
+
+ - name: Copy grafana dashboards provisioning
+ copy:
+ src: ../conf/dashboards.yml
+ dest: "{{ binarydir }}/grafana-{{ grafana_version }}/conf/provisioning/dashboards"
+
+ - name: Copy grafana dashboards
+ copy:
+ src: ../conf/grafana_dashboards
+ dest: "{{ binarydir }}"
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_start.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_start.yml
new file mode 100644
index 0000000..5d7b372
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_start.yml
@@ -0,0 +1,33 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- hosts: node_exporter
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+ - include: prometheus_node_exporter.yml
+
+- hosts: prometheus
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+ - include: prometheus.yml
+
+- hosts: grafana
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+ - include: grafana.yml
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_stop.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_stop.yml
new file mode 100644
index 0000000..1c181ad
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_stop.yml
@@ -0,0 +1,36 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- hosts: grafana
+ tasks:
+ - name: Stop grafana
+ shell: kill -9 `lsof -i :3000 | grep LISTEN | awk -F ' ' '{print $2}'`
+ failed_when: false
+
+- hosts: prometheus
+ tasks:
+ - name: Stop prometheus
+ shell: kill -9 `lsof -i :8080 | grep LISTEN | awk -F ' ' '{print $2}'`
+ failed_when: false
+
+- hosts: node_exporter
+ tasks:
+ - name: Stop node_exporter
+ shell: kill -9 `lsof -i :9100 | grep LISTEN | awk -F ' ' '{print $2}'`
+ failed_when: false
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus.yml
new file mode 100644
index 0000000..e83a96e
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus.yml
@@ -0,0 +1,32 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- name: config file
+ template:
+ src: ../conf/prometheus.conf.j2
+ dest: "{{ binarydir }}/conf"
+
+- name: Start prometheus service
+ shell: |
+ nohup "prometheus-{{ prometheus_version }}.darwin-amd64/prometheus" --config.file=conf/prometheus.conf.j2 --web.listen-address=:8080 >> logs/prometheus.log 2>&1 &
+ sleep 1
+ args:
+ chdir: "{{ binarydir }}"
+ async: 10
+ poll: 0
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus_node_exporter.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus_node_exporter.yml
new file mode 100644
index 0000000..4aa4a99
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus_node_exporter.yml
@@ -0,0 +1,27 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- name: Start node_exporter service
+ shell: |
+ nohup "node_exporter-{{ node_exporter_version }}.darwin-amd64/node_exporter" >> logs/node_exporter.log 2>&1 &
+ sleep 1
+ args:
+ chdir: "{{ binarydir }}"
+ async: 10
+ poll: 0
\ No newline at end of file
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml
index 0f1be96..8e2b5bf 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml
@@ -99,5 +99,10 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</dependency>
+ <dependency>
+ <groupId>io.prometheus</groupId>
+ <artifactId>simpleclient</artifactId>
+ <version>0.9.0</version>
+ </dependency>
</dependencies>
</project>
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
index f11e7ff..8d280a4 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
@@ -18,6 +18,8 @@
*/
package org.apache.hyracks.control.cc;
+import static io.prometheus.client.CollectorRegistry.defaultRegistry;
+
import java.io.File;
import java.io.FileReader;
import java.lang.reflect.Constructor;
@@ -239,7 +241,10 @@
workQueue.start();
connectNCs();
LOGGER.log(Level.INFO, "Started ClusterControllerService");
+ defaultRegistry.register(new JobsExporter(jobManager));
+ defaultRegistry.register(new MemoryExporter(resourceManager));
notifyApplication();
+
}
protected void startWebServers() throws Exception {
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/JobsExporter.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/JobsExporter.java
new file mode 100644
index 0000000..0e2baeb
--- /dev/null
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/JobsExporter.java
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hyracks.control.cc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+
+import org.apache.hyracks.api.job.JobStatus;
+import org.apache.hyracks.control.cc.job.IJobManager;
+import org.apache.hyracks.control.cc.job.JobRun;
+
+import io.prometheus.client.Collector;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+
+public class JobsExporter extends Collector {
+ private final IJobManager jobManager;
+ private HashMap<String, Double> jobsCount;
+ private HashSet<Long> jobsTerminatedIds;
+
+ private static final Counter jobsSecondsCounter = Counter.build().name("asterix_job_total_milliseconds")
+ .help("total milliseconds taken by jobs").labelNames("status").register();
+
+ private static final Counter jobsCountCounter = Counter.build().name("asterix_job_total_count")
+ .help("current total count of jobs").labelNames("status").register();
+
+ private static final Gauge jobsCountGauge =
+ Gauge.build().name("asterix_job_count").help("current count of jobs").labelNames("status").register();
+
+ public JobsExporter(IJobManager jobManager) {
+ this.jobManager = jobManager;
+ this.jobsCount = new HashMap<>();
+ this.jobsTerminatedIds = new HashSet<>();
+
+ // initialize counter to 0
+ for (JobStatus status : JobStatus.values()) {
+ if (status.equals(JobStatus.FAILURE) || status.equals(JobStatus.FAILURE_BEFORE_EXECUTION)
+ || status.equals(JobStatus.TERMINATED)) {
+ jobsSecondsCounter.labels(status.toString()).inc(0);
+ }
+ jobsCountCounter.labels(status.toString()).inc(0);
+ }
+ resetJobCount();
+ }
+
+ @Override
+ public List<MetricFamilySamples> collect() {
+ List<MetricFamilySamples> mfs = new ArrayList<>();
+
+ HashSet<Long> tempTerminatedIds = new HashSet<>();
+ for (JobRun job : jobManager.getArchivedJobs()) {
+ long jobId = job.getJobId().getId();
+ if (!jobsTerminatedIds.contains(jobId)) {
+ String status = job.getStatus().toString();
+ jobsSecondsCounter.labels(status).inc(job.getEndTime() - job.getStartTime());
+ jobsCountCounter.labels(status).inc();
+ }
+ tempTerminatedIds.add(jobId);
+ }
+ jobsTerminatedIds = tempTerminatedIds;
+
+ resetJobCount();
+ updateJobCount(jobManager.getRunningJobs());
+ updateJobCount(jobManager.getPendingJobs());
+ for (String jobElement : jobsCount.keySet()) {
+ jobsCountGauge.labels(jobElement).set(jobsCount.get(jobElement));
+ }
+
+ return mfs;
+ }
+
+ private void updateJobCount(Collection<JobRun> jobs) {
+ for (JobRun job : jobs) {
+ String jobStatus = job.getStatus().toString();
+ jobsCount.put(jobStatus, jobsCount.get(jobStatus) + 1);
+ }
+ }
+
+ private void resetJobCount() {
+ for (JobStatus status : JobStatus.values()) {
+ if (status.equals(JobStatus.RUNNING) || (status.equals(JobStatus.PENDING))) {
+ this.jobsCount.put(status.toString(), 0.0);
+ }
+ }
+ }
+}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/MemoryExporter.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/MemoryExporter.java
new file mode 100644
index 0000000..2e162eb
--- /dev/null
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/MemoryExporter.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hyracks.control.cc;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hyracks.control.cc.scheduler.IResourceManager;
+
+import io.prometheus.client.Collector;
+import io.prometheus.client.Gauge;
+
+public class MemoryExporter extends Collector {
+ private IResourceManager resourceManager;
+ private static final Gauge clusterMemorySizeBytes =
+ Gauge.build().name("asterix_cluster_memory_bytes").help("cluster aggregate bytes available").register();
+
+ public MemoryExporter(IResourceManager resourceManager) {
+ this.resourceManager = resourceManager;
+ }
+
+ @Override
+ public List<MetricFamilySamples> collect() {
+ List<MetricFamilySamples> mfs = new ArrayList<MetricFamilySamples>();
+
+ clusterMemorySizeBytes.set(resourceManager.getMaximumCapacity().getAggregatedMemoryByteSize()
+ - resourceManager.getCurrentCapacity().getAggregatedMemoryByteSize());
+
+ return mfs;
+ }
+}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
index d148145..fe04a9e 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
@@ -84,7 +84,8 @@
OptionTypes.STRING,
(Function<IApplicationConfig, String>) appConfig -> FileUtil
.joinPath(appConfig.getString(ControllerConfig.Option.DEFAULT_DIR), "passwd"),
- ControllerConfig.Option.DEFAULT_DIR.cmdline() + "/passwd");
+ ControllerConfig.Option.DEFAULT_DIR.cmdline() + "/passwd"),
+ METRICS_PORT(POSITIVE_INTEGER, 19007);
private final IOptionType parser;
private Object defaultValue;
@@ -206,6 +207,8 @@
return "The password to the provided key store";
case CREDENTIAL_FILE:
return "Path to HTTP basic credentials";
+ case METRICS_PORT:
+ return "Port the CC exports monitoring metrics to";
default:
throw new IllegalStateException("NYI: " + this);
}
@@ -481,4 +484,11 @@
return getAppConfig().getString(Option.CREDENTIAL_FILE);
}
+ public void setMetricsPort(int metricsPort) {
+ configManager.set(CCConfig.Option.METRICS_PORT, metricsPort);
+ }
+
+ public int getMetricsPort() {
+ return getAppConfig().getInt(CCConfig.Option.METRICS_PORT);
+ }
}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
index 01cb9bf..fa159a6 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
@@ -103,7 +103,8 @@
OptionTypes.STRING,
(Function<IApplicationConfig, String>) appConfig -> FileUtil
.joinPath(appConfig.getString(ControllerConfig.Option.DEFAULT_DIR), "passwd"),
- ControllerConfig.Option.DEFAULT_DIR.cmdline() + "/passwd");
+ ControllerConfig.Option.DEFAULT_DIR.cmdline() + "/passwd"),
+ METRICS_PORT(INTEGER, 19010);
private final IOptionType parser;
private final String defaultValueDescription;
@@ -250,6 +251,8 @@
return "List of environment variables to set when invoking the Python interpreter for Python UDFs. E.g. FOO=1";
case CREDENTIAL_FILE:
return "Path to HTTP basic credentials";
+ case METRICS_PORT:
+ return "Port the NC exports monitoring metrics to";
default:
throw new IllegalStateException("Not yet implemented: " + this);
}
@@ -625,4 +628,11 @@
return getAppConfig().getString(Option.CREDENTIAL_FILE);
}
+ public void setMetricsPort(int metricsPort) {
+ configManager.set(nodeId, Option.METRICS_PORT, metricsPort);
+ }
+
+ public int getMetricsPort() {
+ return appConfig.getInt(Option.METRICS_PORT);
+ }
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ib9aa81fa2e58954846218e3902562685c8cd09e1
Gerrit-Change-Number: 11544
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon <im...@uci.edu>
Gerrit-Reviewer: Krish Avvari <av...@gmail.com>
Gerrit-MessageType: newchange
Change in asterixdb[master]: Added prometheus monitoring to asterix
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544 )
Change subject: Added prometheus monitoring to asterix
......................................................................
Patch Set 1: Contrib-2
Analytics Compatibility Tests Failed
https://cbjenkins.page.link/wAWbHuQSSvzjXQbj8 : UNSTABLE
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ib9aa81fa2e58954846218e3902562685c8cd09e1
Gerrit-Change-Number: 11544
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon <im...@uci.edu>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Krish Avvari <av...@gmail.com>
Gerrit-Comment-Date: Fri, 21 May 2021 02:41:41 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment
Change in asterixdb[master]: Added prometheus monitoring to asterix
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Ian Maxon <im...@uci.edu>:
Hello Krish Avvari,
I'd like you to do a code review. Please visit
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544
to review the following change.
Change subject: Added prometheus monitoring to asterix
......................................................................
Added prometheus monitoring to asterix
- Ansible playbooks to install, start, and stop the monitoring environment
- Metrics exporters to export jobs, storage, memory, and cpu info
- Grafana dashboards
Change-Id: Ib9aa81fa2e58954846218e3902562685c8cd09e1
---
M asterixdb/asterix-app/pom.xml
A asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/MetricsServlet.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
A asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/StorageExporter.java
M asterixdb/asterix-app/src/test/resources/cc.conf
A asterixdb/asterix-server/src/main/opt/ansible/bin/install_monitoring.sh
A asterixdb/asterix-server/src/main/opt/ansible/bin/start_monitoring.sh
A asterixdb/asterix-server/src/main/opt/ansible/bin/stop_monitoring.sh
A asterixdb/asterix-server/src/main/opt/ansible/conf/dashboards.yml
A asterixdb/asterix-server/src/main/opt/ansible/conf/datasources.yml
A asterixdb/asterix-server/src/main/opt/ansible/conf/defaults.ini
A asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_jobs_stats.json
A asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_memory_stats.json
A asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_storage_stats.json
A asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_system_stats.json
A asterixdb/asterix-server/src/main/opt/ansible/conf/instance/cc.conf
M asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml
M asterixdb/asterix-server/src/main/opt/ansible/conf/inventory
A asterixdb/asterix-server/src/main/opt/ansible/conf/monitoring_inventory
A asterixdb/asterix-server/src/main/opt/ansible/conf/prometheus.conf.j2
A asterixdb/asterix-server/src/main/opt/ansible/yaml/grafana.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_install.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_start.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_stop.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus.yml
A asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus_node_exporter.yml
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/JobsExporter.java
A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/MemoryExporter.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
33 files changed, 3,535 insertions(+), 9 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/44/11544/1
diff --git a/asterixdb/asterix-app/pom.xml b/asterixdb/asterix-app/pom.xml
index 34bab13..7830fbe 100644
--- a/asterixdb/asterix-app/pom.xml
+++ b/asterixdb/asterix-app/pom.xml
@@ -867,5 +867,15 @@
<artifactId>kite-data-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>io.prometheus</groupId>
+ <artifactId>simpleclient_hotspot</artifactId>
+ <version>0.9.0</version>
+ </dependency>
+ <dependency>
+ <groupId>io.prometheus</groupId>
+ <artifactId>simpleclient_httpserver</artifactId>
+ <version>0.9.0</version>
+ </dependency>
</dependencies>
</project>
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/MetricsServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/MetricsServlet.java
new file mode 100644
index 0000000..b7e32d1
--- /dev/null
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/MetricsServlet.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.asterix.api.http.server;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.Writer;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentMap;
+
+import org.apache.hyracks.http.api.IServletRequest;
+import org.apache.hyracks.http.api.IServletResponse;
+import org.apache.hyracks.http.server.AbstractServlet;
+
+import io.netty.handler.codec.http.HttpHeaderNames;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+
+public class MetricsServlet extends AbstractServlet {
+ private CollectorRegistry registry;
+
+ public MetricsServlet(ConcurrentMap<String, Object> ctx, String... paths) {
+ this(CollectorRegistry.defaultRegistry, ctx, paths);
+ }
+
+ public MetricsServlet(CollectorRegistry registry, ConcurrentMap<String, Object> ctx, String... paths) {
+ super(ctx, paths);
+ this.registry = registry;
+ }
+
+ @Override
+ protected void get(IServletRequest request, IServletResponse response) throws IOException {
+ response.setHeader(HttpHeaderNames.CONTENT_TYPE, TextFormat.CONTENT_TYPE_004);
+
+ Writer writer = new BufferedWriter(response.writer());
+ try {
+ TextFormat.write004(writer, registry.filteredMetricFamilySamples(parse(request)));
+ response.setStatus(HttpResponseStatus.OK);
+ writer.flush();
+ } finally {
+ writer.close();
+ }
+ }
+
+ private Set<String> parse(IServletRequest req) {
+ List<String> includedParam = req.getParameterValues("name[]");
+ if (includedParam == null) {
+ return Collections.emptySet();
+ } else {
+ return new HashSet<String>(includedParam);
+ }
+ }
+
+ @Override
+ protected void post(IServletRequest req, IServletResponse resp) throws IOException {
+ get(req, resp);
+ }
+}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java
index 8f3deb1..8138932 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java
@@ -43,6 +43,7 @@
import org.apache.asterix.api.http.server.ClusterControllerDetailsApiServlet;
import org.apache.asterix.api.http.server.ConnectorApiServlet;
import org.apache.asterix.api.http.server.DiagnosticsApiServlet;
+import org.apache.asterix.api.http.server.MetricsServlet;
import org.apache.asterix.api.http.server.NodeControllerDetailsApiServlet;
import org.apache.asterix.api.http.server.QueryResultApiServlet;
import org.apache.asterix.api.http.server.QueryServiceServlet;
@@ -117,6 +118,8 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import io.prometheus.client.hotspot.DefaultExports;
+
public class CCApplication extends BaseCCApplication {
private static final Logger LOGGER = LogManager.getLogger();
@@ -242,6 +245,7 @@
webManager.add(setupWebServer(appCtx.getExternalProperties()));
webManager.add(setupJSONAPIServer(appCtx.getExternalProperties()));
webManager.add(setupQueryWebServer(appCtx.getExternalProperties()));
+ webManager.add(setupMetricsServer(appCtx.getExternalProperties()));
}
@Override
@@ -253,6 +257,18 @@
webManager.stop();
}
+ protected HttpServer setupMetricsServer(ExternalProperties externalProperties) throws Exception {
+ CCConfig ccConfig = ((ClusterControllerService) ccServiceCtx.getControllerService()).getConfig();
+ DefaultExports.initialize();
+ final HttpServerConfig config =
+ HttpServerConfigBuilder.custom().setMaxRequestSize(externalProperties.getMaxWebRequestSize()).build();
+ HttpServer webServer =
+ new HttpServer(webManager.getBosses(), webManager.getWorkers(), ccConfig.getMetricsPort(), config);
+ webServer.setAttribute(HYRACKS_CONNECTION_ATTR, hcc);
+ webServer.addServlet(new MetricsServlet(webServer.ctx(), "/*"));
+ return webServer;
+ }
+
protected HttpServer setupWebServer(ExternalProperties externalProperties) throws Exception {
final HttpServerConfig config =
HttpServerConfigBuilder.custom().setMaxRequestSize(externalProperties.getMaxWebRequestSize()).build();
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
index c148c92..1105289 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
@@ -18,6 +18,7 @@
*/
package org.apache.asterix.hyracks.bootstrap;
+import static io.prometheus.client.CollectorRegistry.defaultRegistry;
import static org.apache.asterix.api.http.server.ServletConstants.HYRACKS_CONNECTION_ATTR;
import static org.apache.asterix.api.http.server.ServletConstants.SYS_AUTH_HEADER;
import static org.apache.asterix.common.utils.Servlets.QUERY_RESULT;
@@ -39,6 +40,7 @@
import org.apache.asterix.algebra.base.ILangExtension;
import org.apache.asterix.api.http.server.BasicAuthServlet;
+import org.apache.asterix.api.http.server.MetricsServlet;
import org.apache.asterix.api.http.server.NCQueryServiceServlet;
import org.apache.asterix.api.http.server.NCUdfApiServlet;
import org.apache.asterix.api.http.server.NCUdfRecoveryServlet;
@@ -108,6 +110,8 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import io.prometheus.client.hotspot.DefaultExports;
+
public class NCApplication extends BaseNCApplication {
private static final Logger LOGGER = LogManager.getLogger();
protected NCExtensionManager ncExtensionManager;
@@ -183,6 +187,9 @@
LOGGER.info("Stores: " + PrintUtil.toString(metadataProperties.getStores()));
}
webManager = new WebManager();
+
+ defaultRegistry.register(new StorageExporter(
+ ((PersistentLocalResourceRepository) getApplicationContext().getLocalResourceRepository())));
performLocalCleanUp();
}
@@ -235,7 +242,21 @@
auth.getFirst(), auth.getSecond()));
apiServer.addServlet(new QueryStatusApiServlet(apiServer.ctx(), getApplicationContext(), QUERY_STATUS));
apiServer.addServlet(new QueryResultApiServlet(apiServer.ctx(), getApplicationContext(), QUERY_RESULT));
+
webManager.add(apiServer);
+ webManager.add(setupMetricsServer(getApplicationContext().getExternalProperties()));
+ }
+
+ protected HttpServer setupMetricsServer(ExternalProperties externalProperties) throws Exception {
+ NCConfig ncConfig = ((NodeControllerService) ncServiceCtx.getControllerService()).getConfiguration();
+ DefaultExports.initialize();
+ final HttpServerConfig config =
+ HttpServerConfigBuilder.custom().setMaxRequestSize(externalProperties.getMaxWebRequestSize()).build();
+ HttpServer webServer =
+ new HttpServer(webManager.getBosses(), webManager.getWorkers(), ncConfig.getMetricsPort(), config);
+ webServer.setAttribute(HYRACKS_CONNECTION_ATTR, getApplicationContext().getHcc());
+ webServer.addServlet(new MetricsServlet(webServer.ctx(), "/*"));
+ return webServer;
}
protected List<AsterixExtension> getExtensions() throws Exception {
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/StorageExporter.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/StorageExporter.java
new file mode 100644
index 0000000..456e5ff
--- /dev/null
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/StorageExporter.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.asterix.hyracks.bootstrap;
+
+import static com.fasterxml.jackson.databind.MapperFeature.SORT_PROPERTIES_ALPHABETICALLY;
+import static com.fasterxml.jackson.databind.SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.asterix.common.storage.ResourceStorageStats;
+import org.apache.asterix.transaction.management.resource.PersistentLocalResourceRepository;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hyracks.api.exceptions.HyracksDataException;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+
+import io.prometheus.client.Collector;
+import io.prometheus.client.Gauge;
+
+public class StorageExporter extends Collector {
+ private ArrayNode storageStats;
+ private PersistentLocalResourceRepository storageRepository;
+ protected static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+ static {
+ OBJECT_MAPPER.enable(SerializationFeature.INDENT_OUTPUT);
+ OBJECT_MAPPER.configure(SORT_PROPERTIES_ALPHABETICALLY, true);
+ OBJECT_MAPPER.configure(ORDER_MAP_ENTRIES_BY_KEYS, true);
+ }
+
+ private static final Gauge storageBytesGauge = Gauge.build().name("asterix_storage_bytes")
+ .help("current size of storage").labelNames("partition", "dataverse", "dataset", "index").register();
+
+ public StorageExporter(PersistentLocalResourceRepository storageRepository) {
+ this.storageStats = OBJECT_MAPPER.createArrayNode();
+ this.storageRepository = storageRepository;
+ }
+
+ @Override
+ public List<MetricFamilySamples> collect() {
+ List<MetricFamilySamples> mfs = new ArrayList<>();
+
+ storageBytesGauge.clear();
+ updateStorageStats();
+ for (int i = 0; i < storageStats.size(); i++) {
+ JsonNode node = storageStats.get(i);
+ String index = node.get("index").toString();
+ String dataset = node.get("dataset").toString();
+ String partition = node.get("partition").toString();
+ Double nodeSize = node.get("totalSize").asDouble();
+ String[] tokens = StringUtils.split(node.get("path").toString(), File.separatorChar);
+ storageBytesGauge.labels(partition, tokens[2], dataset, index).set(nodeSize);
+ }
+
+ return mfs;
+ }
+
+ private void updateStorageStats() {
+ ArrayNode storageStats = OBJECT_MAPPER.createArrayNode();
+
+ try {
+ storageRepository.getStorageStats().stream().map(ResourceStorageStats::asJson).forEach(storageStats::add);
+ this.storageStats = storageStats;
+ } catch (HyracksDataException e) {
+ e.printStackTrace();
+ }
+ }
+}
diff --git a/asterixdb/asterix-app/src/test/resources/cc.conf b/asterixdb/asterix-app/src/test/resources/cc.conf
index 953284964..dd5743a 100644
--- a/asterixdb/asterix-app/src/test/resources/cc.conf
+++ b/asterixdb/asterix-app/src/test/resources/cc.conf
@@ -23,12 +23,13 @@
nc.api.port=19004
#jvm.args=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5006
-[nc/asterix_nc2]
-ncservice.port=9091
-txn.log.dir=target/tmp/asterix_nc2/txnlog
-core.dump.dir=target/tmp/asterix_nc2/coredump
-iodevices=target/tmp/asterix_nc2/iodevice1,../asterix-server/target/tmp/asterix_nc2/iodevice2
-nc.api.port=19005
+; [nc/asterix_nc2]
+; ncservice.port=9091
+; txn.log.dir=target/tmp/asterix_nc2/txnlog
+; core.dump.dir=target/tmp/asterix_nc2/coredump
+; iodevices=target/tmp/asterix_nc2/iodevice1,../asterix-server/target/tmp/asterix_nc2/iodevice2
+; nc.api.port=19005
+; metrics.port=19011
#jvm.args=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5007
[nc]
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/bin/install_monitoring.sh b/asterixdb/asterix-server/src/main/opt/ansible/bin/install_monitoring.sh
new file mode 100755
index 0000000..15e9d4d
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/bin/install_monitoring.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+# Gets the absolute path so that the script can work no matter where it is invoked.
+pushd `dirname $0` > /dev/null
+SCRIPT_PATH=`pwd -P`
+popd > /dev/null
+ANSB_PATH=`dirname "${SCRIPT_PATH}"`
+
+MONITORING_INVENTORY=$ANSB_PATH/conf/monitoring_inventory
+
+# Start installing binaries for asterix monitoring
+export ANSIBLE_HOST_KEY_CHECKING=false
+ansible-playbook -i $MONITORING_INVENTORY $ANSB_PATH/yaml/monitoring_install.yml
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/bin/start_monitoring.sh b/asterixdb/asterix-server/src/main/opt/ansible/bin/start_monitoring.sh
new file mode 100755
index 0000000..2e220e3
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/bin/start_monitoring.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+# Gets the absolute path so that the script can work no matter where it is invoked.
+pushd `dirname $0` > /dev/null
+SCRIPT_PATH=`pwd -P`
+popd > /dev/null
+ANSB_PATH=`dirname "${SCRIPT_PATH}"`
+
+MONITORING_INVENTORY=$ANSB_PATH/conf/monitoring_inventory
+INVENTORY=$ANSB_PATH/conf/inventory
+
+# Start the monitoring binaries
+export ANSIBLE_HOST_KEY_CHECKING=false
+ansible-playbook -i $INVENTORY -i $MONITORING_INVENTORY $ANSB_PATH/yaml/monitoring_start.yml
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/bin/stop_monitoring.sh b/asterixdb/asterix-server/src/main/opt/ansible/bin/stop_monitoring.sh
new file mode 100755
index 0000000..4e2a311
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/bin/stop_monitoring.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+# Gets the absolute path so that the script can work no matter where it is invoked.
+pushd `dirname $0` > /dev/null
+SCRIPT_PATH=`pwd -P`
+popd > /dev/null
+ANSB_PATH=`dirname "${SCRIPT_PATH}"`
+
+MONITORING_INVENTORY=$ANSB_PATH/conf/monitoring_inventory
+
+# Stop the monitoring services
+export ANSIBLE_HOST_KEY_CHECKING=false
+ansible-playbook -i $MONITORING_INVENTORY $ANSB_PATH/yaml/monitoring_stop.yml
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/dashboards.yml b/asterixdb/asterix-server/src/main/opt/ansible/conf/dashboards.yml
new file mode 100644
index 0000000..6eae093
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/dashboards.yml
@@ -0,0 +1,43 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+apiVersion: 1
+
+providers:
+ # <string> an unique provider name. Required
+ - name: 'a unique provider name'
+ # <int> Org id. Default to 1
+ orgId: 1
+ # <string> name of the dashboard folder.
+ folder: ''
+ # <string> folder UID. will be automatically generated if not specified
+ folderUid: ''
+ # <string> provider type. Default to 'file'
+ type: file
+ # <bool> disable dashboard deletion
+ disableDeletion: false
+ # <int> how often Grafana will scan for changed dashboards
+ updateIntervalSeconds: 10
+ # <bool> allow updating provisioned dashboards from the UI
+ allowUiUpdates: false
+ options:
+ # <string, required> path to dashboard files on disk. Required when using the 'file' type
+ path: grafana_dashboards
+ # <bool> use folder names from filesystem to create folders in Grafana
+ foldersFromFilesStructure: true
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/datasources.yml b/asterixdb/asterix-server/src/main/opt/ansible/conf/datasources.yml
new file mode 100644
index 0000000..3d66273
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/datasources.yml
@@ -0,0 +1,54 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+# # config file version
+apiVersion: 1
+
+# # list of datasources to insert/update depending
+# # on what's available in the database
+datasources:
+ # <string, required> name of the datasource. Required
+ - name: Prometheus
+ # <string, required> datasource type. Required
+ type: prometheus
+ # <string, required> access mode. direct or proxy. Required
+ access: proxy
+ # <int> org id. will default to orgId 1 if not specified
+ orgId: 1
+ # <string> url
+ url: http://localhost:8080
+ # <string> database password, if used
+ password:
+ # <string> database user, if used
+ user:
+ # <string> database name, if used
+ database:
+ # <bool> enable/disable basic auth
+ basicAuth:
+ # <string> basic auth username
+ basicAuthUser:
+ # <string> basic auth password
+ basicAuthPassword:
+ # <bool> enable/disable with credentials headers
+ withCredentials:
+ # <bool> mark as default datasource. Max one per org
+ isDefault:
+ version: 1
+ # <bool> allow users to edit datasources from the UI.
+ editable: false
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/defaults.ini b/asterixdb/asterix-server/src/main/opt/ansible/conf/defaults.ini
new file mode 100644
index 0000000..5a11a8b
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/defaults.ini
@@ -0,0 +1,876 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+##################### Grafana Configuration Defaults #####################
+#
+# Do not modify this file in grafana installs
+#
+
+# possible values : production, development
+app_mode = production
+
+# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
+instance_name = ${HOSTNAME}
+
+#################################### Paths ###############################
+[paths]
+# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
+data = data
+
+# Temporary files in `data` directory older than given duration will be removed
+temp_data_lifetime = 24h
+
+# Directory where grafana can store logs
+logs = data/log
+
+# Directory where grafana will automatically scan and look for plugins
+plugins = data/plugins
+
+# folder that contains provisioning config files that grafana will apply on startup and while running.
+provisioning = conf/provisioning
+
+#################################### Server ##############################
+[server]
+# Protocol (http, https, h2, socket)
+protocol = http
+
+# The ip address to bind to, empty will bind to all interfaces
+http_addr =
+
+# The http port to use
+http_port = 3000
+
+# The public facing domain name used to access grafana from a browser
+domain = localhost
+
+# Redirect to correct domain if host header does not match domain
+# Prevents DNS rebinding attacks
+enforce_domain = false
+
+# The full public facing url
+root_url = %(protocol)s://%(domain)s:%(http_port)s/
+
+# Serve Grafana from subpath specified in `root_url` setting. By default it is set to `false` for compatibility reasons.
+serve_from_sub_path = false
+
+# Log web requests
+router_logging = false
+
+# the path relative working path
+static_root_path = public
+
+# enable gzip
+enable_gzip = false
+
+# https certs & key file
+cert_file =
+cert_key =
+
+# Unix socket path
+socket = /tmp/grafana.sock
+
+#################################### Database ############################
+[database]
+# You can configure the database connection by specifying type, host, name, user and password
+# as separate properties or as on string using the url property.
+
+# Either "mysql", "postgres" or "sqlite3", it's your choice
+type = sqlite3
+host = 127.0.0.1:3306
+name = grafana
+user = root
+# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
+password =
+# Use either URL or the previous fields to configure the database
+# Example: mysql://user:secret@host:port/database
+url =
+
+# Max idle conn setting default is 2
+max_idle_conn = 2
+
+# Max conn setting default is 0 (mean not set)
+max_open_conn =
+
+# Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours)
+conn_max_lifetime = 14400
+
+# Set to true to log the sql calls and execution times.
+log_queries =
+
+# For "postgres", use either "disable", "require" or "verify-full"
+# For "mysql", use either "true", "false", or "skip-verify".
+ssl_mode = disable
+
+ca_cert_path =
+client_key_path =
+client_cert_path =
+server_cert_name =
+
+# For "sqlite3" only, path relative to data_path setting
+path = grafana.db
+
+# For "sqlite3" only. cache mode setting used for connecting to the database
+cache_mode = private
+
+#################################### Cache server #############################
+[remote_cache]
+# Either "redis", "memcached" or "database" default is "database"
+type = database
+
+# cache connectionstring options
+# database: will use Grafana primary database.
+# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=0,ssl=false`. Only addr is required. ssl may be 'true', 'false', or 'insecure'.
+# memcache: 127.0.0.1:11211
+connstr =
+
+#################################### Data proxy ###########################
+[dataproxy]
+
+# This enables data proxy logging, default is false
+logging = false
+
+# How long the data proxy waits before timing out, default is 30 seconds.
+# This setting also applies to core backend HTTP data sources where query requests use an HTTP client with timeout set.
+timeout = 30
+
+# How many seconds the data proxy waits before sending a keepalive request.
+keep_alive_seconds = 30
+
+# How many seconds the data proxy waits for a successful TLS Handshake before timing out.
+tls_handshake_timeout_seconds = 10
+
+# How many seconds the data proxy will wait for a server's first response headers after
+# fully writing the request headers if the request has an "Expect: 100-continue"
+# header. A value of 0 will result in the body being sent immediately, without
+# waiting for the server to approve.
+expect_continue_timeout_seconds = 1
+
+# The maximum number of idle connections that Grafana will keep alive.
+max_idle_connections = 100
+
+# How many seconds the data proxy keeps an idle connection open before timing out.
+idle_conn_timeout_seconds = 90
+
+# If enabled and user is not anonymous, data proxy will add X-Grafana-User header with username into the request.
+send_user_header = false
+
+#################################### Analytics ###########################
+[analytics]
+# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
+# No ip addresses are being tracked, only simple counters to track
+# running instances, dashboard and error counts. It is very helpful to us.
+# Change this option to false to disable reporting.
+reporting_enabled = true
+
+# Set to false to disable all checks to https://grafana.com
+# for new versions (grafana itself and plugins), check is used
+# in some UI views to notify that grafana or plugin update exists
+# This option does not cause any auto updates, nor send any information
+# only a GET request to https://grafana.com to get latest versions
+check_for_updates = true
+
+# Google Analytics universal tracking code, only enabled if you specify an id here
+google_analytics_ua_id =
+
+# Google Tag Manager ID, only enabled if you specify an id here
+google_tag_manager_id =
+
+#################################### Security ############################
+[security]
+# disable creation of admin user on first start of grafana
+disable_initial_admin_creation = false
+
+# default admin user, created on startup
+admin_user = admin
+
+# default admin password, can be changed before first start of grafana, or in profile settings
+admin_password = admin
+
+# used for signing
+secret_key = SW2YcwTIb9zpOOhoPsMm
+
+# disable gravatar profile images
+disable_gravatar = false
+
+# data source proxy whitelist (ip_or_domain:port separated by spaces)
+data_source_proxy_whitelist =
+
+# disable protection against brute force login attempts
+disable_brute_force_login_protection = false
+
+# set to true if you host Grafana behind HTTPS. default is false.
+cookie_secure = false
+
+# set cookie SameSite attribute. defaults to `lax`. can be set to "lax", "strict", "none" and "disabled"
+cookie_samesite = lax
+
+# set to true if you want to allow browsers to render Grafana in a <frame>, <iframe>, <embed> or <object>. default is false.
+allow_embedding = false
+
+# Set to true if you want to enable http strict transport security (HSTS) response header.
+# This is only sent when HTTPS is enabled in this configuration.
+# HSTS tells browsers that the site should only be accessed using HTTPS.
+strict_transport_security = false
+
+# Sets how long a browser should cache HSTS. Only applied if strict_transport_security is enabled.
+strict_transport_security_max_age_seconds = 86400
+
+# Set to true if to enable HSTS preloading option. Only applied if strict_transport_security is enabled.
+strict_transport_security_preload = false
+
+# Set to true if to enable the HSTS includeSubDomains option. Only applied if strict_transport_security is enabled.
+strict_transport_security_subdomains = false
+
+# Set to true to enable the X-Content-Type-Options response header.
+# The X-Content-Type-Options response HTTP header is a marker used by the server to indicate that the MIME types advertised
+# in the Content-Type headers should not be changed and be followed.
+x_content_type_options = true
+
+# Set to true to enable the X-XSS-Protection header, which tells browsers to stop pages from loading
+# when they detect reflected cross-site scripting (XSS) attacks.
+x_xss_protection = true
+
+
+#################################### Snapshots ###########################
+[snapshots]
+# snapshot sharing options
+external_enabled = true
+external_snapshot_url = https://snapshots-origin.raintank.io
+external_snapshot_name = Publish to snapshot.raintank.io
+
+# Set to true to enable this Grafana instance act as an external snapshot server and allow unauthenticated requests for
+# creating and deleting snapshots.
+public_mode = false
+
+# remove expired snapshot
+snapshot_remove_expired = true
+
+#################################### Dashboards ##################
+
+[dashboards]
+# Number dashboard versions to keep (per dashboard). Default: 20, Minimum: 1
+versions_to_keep = 20
+
+# Minimum dashboard refresh interval. When set, this will restrict users to set the refresh interval of a dashboard lower than given interval. Per default this is 5 seconds.
+# The interval string is a possibly signed sequence of decimal numbers, followed by a unit suffix (ms, s, m, h, d), e.g. 30s or 1m.
+min_refresh_interval = 5s
+
+# Path to the default home dashboard. If this value is empty, then Grafana uses StaticRootPath + "dashboards/home.json"
+default_home_dashboard_path = grafana_dashboards/asterix_system_stats.json
+
+#################################### Users ###############################
+[users]
+# disable user signup / registration
+allow_sign_up = false
+
+# Allow non admin users to create organizations
+allow_org_create = false
+
+# Set to true to automatically assign new users to the default organization (id 1)
+auto_assign_org = true
+
+# Set this value to automatically add new users to the provided organization (if auto_assign_org above is set to true)
+auto_assign_org_id = 1
+
+# Default role new users will be automatically assigned (if auto_assign_org above is set to true)
+auto_assign_org_role = Viewer
+
+# Require email validation before sign up completes
+verify_email_enabled = false
+
+# Background text for the user field on the login page
+login_hint = email or username
+password_hint = password
+
+# Default UI theme ("dark" or "light")
+default_theme = dark
+
+# External user management
+external_manage_link_url =
+external_manage_link_name =
+external_manage_info =
+
+# Viewers can edit/inspect dashboard settings in the browser. But not save the dashboard.
+viewers_can_edit = false
+
+# Editors can administrate dashboard, folders and teams they create
+editors_can_admin = false
+
+# The duration in time a user invitation remains valid before expiring. This setting should be expressed as a duration. Examples: 6h (hours), 2d (days), 1w (week). Default is 24h (24 hours). The minimum supported duration is 15m (15 minutes).
+user_invite_max_lifetime_duration = 24h
+
+[auth]
+# Login cookie name
+login_cookie_name = grafana_session
+
+# The maximum lifetime (duration) an authenticated user can be inactive before being required to login at next visit. Default is 7 days (7d). This setting should be expressed as a duration, e.g. 5m (minutes), 6h (hours), 10d (days), 2w (weeks), 1M (month). The lifetime resets at each successful token rotation (token_rotation_interval_minutes).
+login_maximum_inactive_lifetime_duration =
+
+# The maximum lifetime (duration) an authenticated user can be logged in since login time before being required to login. Default is 30 days (30d). This setting should be expressed as a duration, e.g. 5m (minutes), 6h (hours), 10d (days), 2w (weeks), 1M (month).
+login_maximum_lifetime_duration =
+
+# How often should auth tokens be rotated for authenticated users when being active. The default is each 10 minutes.
+token_rotation_interval_minutes = 10
+
+# Set to true to disable (hide) the login form, useful if you use OAuth
+disable_login_form = false
+
+# Set to true to disable the signout link in the side menu. useful if you use auth.proxy
+disable_signout_menu = false
+
+# URL to redirect the user to after sign out
+signout_redirect_url =
+
+# Set to true to attempt login with OAuth automatically, skipping the login screen.
+# This setting is ignored if multiple OAuth providers are configured.
+oauth_auto_login = false
+
+# OAuth state max age cookie duration in seconds. Defaults to 600 seconds.
+oauth_state_cookie_max_age = 600
+
+# limit of api_key seconds to live before expiration
+api_key_max_seconds_to_live = -1
+
+# Set to true to enable SigV4 authentication option for HTTP-based datasources
+sigv4_auth_enabled = false
+
+#################################### Anonymous Auth ######################
+[auth.anonymous]
+# enable anonymous access
+enabled = false
+
+# specify organization name that should be used for unauthenticated users
+org_name = Main Org.
+
+# specify role for unauthenticated users
+org_role = Viewer
+
+# mask the Grafana version number for unauthenticated users
+hide_version = false
+
+#################################### GitHub Auth #########################
+[auth.github]
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = user:email,read:org
+auth_url = https://github.com/login/oauth/authorize
+token_url = https://github.com/login/oauth/access_token
+api_url = https://api.github.com/user
+allowed_domains =
+team_ids =
+allowed_organizations =
+
+#################################### GitLab Auth #########################
+[auth.gitlab]
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = api
+auth_url = https://gitlab.com/oauth/authorize
+token_url = https://gitlab.com/oauth/token
+api_url = https://gitlab.com/api/v4
+allowed_domains =
+allowed_groups =
+
+#################################### Google Auth #########################
+[auth.google]
+enabled = false
+allow_sign_up = true
+client_id = some_client_id
+client_secret =
+scopes = https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
+auth_url = https://accounts.google.com/o/oauth2/auth
+token_url = https://accounts.google.com/o/oauth2/token
+api_url = https://www.googleapis.com/oauth2/v1/userinfo
+allowed_domains =
+hosted_domain =
+
+#################################### Grafana.com Auth ####################
+# legacy key names (so they work in env variables)
+[auth.grafananet]
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = user:email
+allowed_organizations =
+
+[auth.grafana_com]
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = user:email
+allowed_organizations =
+
+#################################### Azure AD OAuth #######################
+[auth.azuread]
+name = Azure AD
+enabled = false
+allow_sign_up = true
+client_id = some_client_id
+client_secret =
+scopes = openid email profile
+auth_url = https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/authorize
+token_url = https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token
+allowed_domains =
+allowed_groups =
+
+#################################### Okta OAuth #######################
+[auth.okta]
+name = Okta
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = openid profile email groups
+auth_url = https://<tenant-id>.okta.com/oauth2/v1/authorize
+token_url = https://<tenant-id>.okta.com/oauth2/v1/token
+api_url = https://<tenant-id>.okta.com/oauth2/v1/userinfo
+allowed_domains =
+allowed_groups =
+role_attribute_path =
+
+#################################### Generic OAuth #######################
+[auth.generic_oauth]
+name = OAuth
+enabled = false
+allow_sign_up = true
+client_id = some_id
+client_secret =
+scopes = user:email
+email_attribute_name = email:primary
+email_attribute_path =
+login_attribute_path =
+role_attribute_path =
+id_token_attribute_name =
+auth_url =
+token_url =
+api_url =
+allowed_domains =
+team_ids =
+allowed_organizations =
+tls_skip_verify_insecure = false
+tls_client_cert =
+tls_client_key =
+tls_client_ca =
+
+#################################### Basic Auth ##########################
+[auth.basic]
+enabled = true
+
+#################################### Auth Proxy ##########################
+[auth.proxy]
+enabled = false
+header_name = X-WEBAUTH-USER
+header_property = username
+auto_sign_up = true
+# Deprecated, use sync_ttl instead
+ldap_sync_ttl = 60
+sync_ttl = 60
+whitelist =
+headers =
+enable_login_token = false
+
+#################################### Auth LDAP ###########################
+[auth.ldap]
+enabled = false
+config_file = /etc/grafana/ldap.toml
+allow_sign_up = true
+
+# LDAP backround sync (Enterprise only)
+# At 1 am every day
+sync_cron = "0 0 1 * * *"
+active_sync_enabled = true
+
+#################################### SMTP / Emailing #####################
+[smtp]
+enabled = false
+host = localhost:25
+user =
+# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
+password =
+cert_file =
+key_file =
+skip_verify = false
+from_address = admin@grafana.localhost
+from_name = Grafana
+ehlo_identity =
+startTLS_policy =
+
+[emails]
+welcome_email_on_sign_up = false
+templates_pattern = emails/*.html
+
+#################################### Logging ##########################
+[log]
+# Either "console", "file", "syslog". Default is console and file
+# Use space to separate multiple modes, e.g. "console file"
+mode = console file
+
+# Either "debug", "info", "warn", "error", "critical", default is "info"
+level = info
+
+# optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug
+filters =
+
+# For "console" mode only
+[log.console]
+level =
+
+# log line format, valid options are text, console and json
+format = console
+
+# For "file" mode only
+[log.file]
+level =
+
+# log line format, valid options are text, console and json
+format = text
+
+# This enables automated log rotate(switch of following options), default is true
+log_rotate = true
+
+# Max line number of single file, default is 1000000
+max_lines = 1000000
+
+# Max size shift of single file, default is 28 means 1 << 28, 256MB
+max_size_shift = 28
+
+# Segment log daily, default is true
+daily_rotate = true
+
+# Expired days of log file(delete after max days), default is 7
+max_days = 7
+
+[log.syslog]
+level =
+
+# log line format, valid options are text, console and json
+format = text
+
+# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
+network =
+address =
+
+# Syslog facility. user, daemon and local0 through local7 are valid.
+facility =
+
+# Syslog tag. By default, the process' argv[0] is used.
+tag =
+
+#################################### Usage Quotas ########################
+[quota]
+enabled = false
+
+#### set quotas to -1 to make unlimited. ####
+# limit number of users per Org.
+org_user = 10
+
+# limit number of dashboards per Org.
+org_dashboard = 100
+
+# limit number of data_sources per Org.
+org_data_source = 10
+
+# limit number of api_keys per Org.
+org_api_key = 10
+
+# limit number of orgs a user can create.
+user_org = 10
+
+# Global limit of users.
+global_user = -1
+
+# global limit of orgs.
+global_org = -1
+
+# global limit of dashboards
+global_dashboard = -1
+
+# global limit of api_keys
+global_api_key = -1
+
+# global limit on number of logged in users.
+global_session = -1
+
+#################################### Alerting ############################
+[alerting]
+# Disable alerting engine & UI features
+enabled = true
+# Makes it possible to turn off alert rule execution but alerting UI is visible
+execute_alerts = true
+
+# Default setting for new alert rules. Defaults to categorize error and timeouts as alerting. (alerting, keep_state)
+error_or_timeout = alerting
+
+# Default setting for how Grafana handles nodata or null values in alerting. (alerting, no_data, keep_state, ok)
+nodata_or_nullvalues = no_data
+
+# Alert notifications can include images, but rendering many images at the same time can overload the server
+# This limit will protect the server from render overloading and make sure notifications are sent out quickly
+concurrent_render_limit = 5
+
+# Default setting for alert calculation timeout. Default value is 30
+evaluation_timeout_seconds = 30
+
+# Default setting for alert notification timeout. Default value is 30
+notification_timeout_seconds = 30
+
+# Default setting for max attempts to sending alert notifications. Default value is 3
+max_attempts = 3
+
+# Makes it possible to enforce a minimal interval between evaluations, to reduce load on the backend
+min_interval_seconds = 1
+
+# Configures for how long alert annotations are stored. Default is 0, which keeps them forever.
+# This setting should be expressed as an duration. Ex 6h (hours), 10d (days), 2w (weeks), 1M (month).
+max_annotation_age =
+
+# Configures max number of alert annotations that Grafana stores. Default value is 0, which keeps all alert annotations.
+max_annotations_to_keep =
+
+#################################### Annotations #########################
+
+[annotations.dashboard]
+# Dashboard annotations means that annotations are associated with the dashboard they are created on.
+
+# Configures how long dashboard annotations are stored. Default is 0, which keeps them forever.
+# This setting should be expressed as a duration. Examples: 6h (hours), 10d (days), 2w (weeks), 1M (month).
+max_age =
+
+# Configures max number of dashboard annotations that Grafana stores. Default value is 0, which keeps all dashboard annotations.
+max_annotations_to_keep =
+
+[annotations.api]
+# API annotations means that the annotations have been created using the API without any
+# association with a dashboard.
+
+# Configures how long Grafana stores API annotations. Default is 0, which keeps them forever.
+# This setting should be expressed as a duration. Examples: 6h (hours), 10d (days), 2w (weeks), 1M (month).
+max_age =
+
+# Configures max number of API annotations that Grafana keeps. Default value is 0, which keeps all API annotations.
+max_annotations_to_keep =
+
+#################################### Explore #############################
+[explore]
+# Enable the Explore section
+enabled = true
+
+#################################### Internal Grafana Metrics ############
+# Metrics available at HTTP API Url /metrics
+[metrics]
+enabled = true
+interval_seconds = 10
+# Disable total stats (stat_totals_*) metrics to be generated
+disable_total_stats = false
+
+#If both are set, basic auth will be required for the metrics endpoint.
+basic_auth_username =
+basic_auth_password =
+
+# Metrics environment info adds dimensions to the `grafana_environment_info` metric, which
+# can expose more information about the Grafana instance.
+[metrics.environment_info]
+#exampleLabel1 = exampleValue1
+#exampleLabel2 = exampleValue2
+
+# Send internal Grafana metrics to graphite
+[metrics.graphite]
+# Enable by setting the address setting (ex localhost:2003)
+address =
+prefix = prod.grafana.%(instance_name)s.
+
+#################################### Grafana.com integration ##########################
+[grafana_net]
+url = https://grafana.com
+
+[grafana_com]
+url = https://grafana.com
+
+#################################### Distributed tracing ############
+[tracing.jaeger]
+# jaeger destination (ex localhost:6831)
+address =
+# tag that will always be included in when creating new spans. ex (tag1:value1,tag2:value2)
+always_included_tag =
+# Type specifies the type of the sampler: const, probabilistic, rateLimiting, or remote
+sampler_type = const
+# jaeger samplerconfig param
+# for "const" sampler, 0 or 1 for always false/true respectively
+# for "probabilistic" sampler, a probability between 0 and 1
+# for "rateLimiting" sampler, the number of spans per second
+# for "remote" sampler, param is the same as for "probabilistic"
+# and indicates the initial sampling rate before the actual one
+# is received from the mothership
+sampler_param = 1
+# Whether or not to use Zipkin span propagation (x-b3- HTTP headers).
+zipkin_propagation = false
+# Setting this to true disables shared RPC spans.
+# Not disabling is the most common setting when using Zipkin elsewhere in your infrastructure.
+disable_shared_zipkin_spans = false
+
+#################################### External Image Storage ##############
+[external_image_storage]
+# Used for uploading images to public servers so they can be included in slack/email messages.
+# You can choose between (s3, webdav, gcs, azure_blob, local)
+provider =
+
+[external_image_storage.s3]
+endpoint =
+path_style_access =
+bucket_url =
+bucket =
+region =
+path =
+access_key =
+secret_key =
+
+[external_image_storage.webdav]
+url =
+username =
+password =
+public_url =
+
+[external_image_storage.gcs]
+key_file =
+bucket =
+path =
+enable_signed_urls = false
+signed_url_expiration =
+
+[external_image_storage.azure_blob]
+account_name =
+account_key =
+container_name =
+
+[external_image_storage.local]
+# does not require any configuration
+
+[rendering]
+# Options to configure a remote HTTP image rendering service, e.g. using https://github.com/grafana/grafana-image-renderer.
+# URL to a remote HTTP image renderer service, e.g. http://localhost:8081/render, will enable Grafana to render panels and dashboards to PNG-images using HTTP requests to an external service.
+server_url =
+# If the remote HTTP image renderer service runs on a different server than the Grafana server you may have to configure this to a URL where Grafana is reachable, e.g. http://grafana.domain/.
+callback_url =
+# Concurrent render request limit affects when the /render HTTP endpoint is used. Rendering many images at the same time can overload the server,
+# which this setting can help protect against by only allowing a certain amount of concurrent requests.
+concurrent_render_request_limit = 30
+
+[panels]
+# here for to support old env variables, can remove after a few months
+enable_alpha = false
+disable_sanitize_html = false
+
+[plugins]
+enable_alpha = false
+app_tls_skip_verify_insecure = false
+# Enter a comma-separated list of plugin identifiers to identify plugins that are allowed to be loaded even if they lack a valid signature.
+allow_loading_unsigned_plugins =
+
+#################################### Grafana Image Renderer Plugin ##########################
+[plugin.grafana-image-renderer]
+# Instruct headless browser instance to use a default timezone when not provided by Grafana, e.g. when rendering panel image of alert.
+# See ICU’s metaZones.txt (https://cs.chromium.org/chromium/src/third_party/icu/source/data/misc/metaZones.txt) for a list of supported
+# timezone IDs. Fallbacks to TZ environment variable if not set.
+rendering_timezone =
+
+# Instruct headless browser instance to use a default language when not provided by Grafana, e.g. when rendering panel image of alert.
+# Please refer to the HTTP header Accept-Language to understand how to format this value, e.g. 'fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5'.
+rendering_language =
+
+# Instruct headless browser instance to use a default device scale factor when not provided by Grafana, e.g. when rendering panel image of alert.
+# Default is 1. Using a higher value will produce more detailed images (higher DPI), but will require more disk space to store an image.
+rendering_viewport_device_scale_factor =
+
+# Instruct headless browser instance whether to ignore HTTPS errors during navigation. Per default HTTPS errors are not ignored. Due to
+# the security risk it's not recommended to ignore HTTPS errors.
+rendering_ignore_https_errors =
+
+# Instruct headless browser instance whether to capture and log verbose information when rendering an image. Default is false and will
+# only capture and log error messages. When enabled, debug messages are captured and logged as well.
+# For the verbose information to be included in the Grafana server log you have to adjust the rendering log level to debug, configure
+# [log].filter = rendering:debug.
+rendering_verbose_logging =
+
+# Instruct headless browser instance whether to output its debug and error messages into running process of remote rendering service.
+# Default is false. This can be useful to enable (true) when troubleshooting.
+rendering_dumpio =
+
+# Additional arguments to pass to the headless browser instance. Default is --no-sandbox. The list of Chromium flags can be found
+# here (https://peter.sh/experiments/chromium-command-line-switches/). Multiple arguments is separated with comma-character.
+rendering_args =
+
+# You can configure the plugin to use a different browser binary instead of the pre-packaged version of Chromium.
+# Please note that this is not recommended, since you may encounter problems if the installed version of Chrome/Chromium is not
+# compatible with the plugin.
+rendering_chrome_bin =
+
+# Instruct how headless browser instances are created. Default is 'default' and will create a new browser instance on each request.
+# Mode 'clustered' will make sure that only a maximum of browsers/incognito pages can execute concurrently.
+# Mode 'reusable' will have one browser instance and will create a new incognito page on each request.
+rendering_mode =
+
+# When rendering_mode = clustered you can instruct how many browsers or incognito pages can execute concurrently. Default is 'browser'
+# and will cluster using browser instances.
+# Mode 'context' will cluster using incognito pages.
+rendering_clustering_mode =
+# When rendering_mode = clustered you can define maximum number of browser instances/incognito pages that can execute concurrently..
+rendering_clustering_max_concurrency =
+
+# Limit the maximum viewport width, height and device scale factor that can be requested.
+rendering_viewport_max_width =
+rendering_viewport_max_height =
+rendering_viewport_max_device_scale_factor =
+
+# Change the listening host and port of the gRPC server. Default host is 127.0.0.1 and default port is 0 and will automatically assign
+# a port not in use.
+grpc_host =
+grpc_port =
+
+[enterprise]
+license_path =
+
+[feature_toggles]
+# enable features, separated by spaces
+enable =
+
+[date_formats]
+# For information on what formatting patterns that are supported https://momentjs.com/docs/#/displaying/
+
+# Default system date format used in time range picker and other places where full time is displayed
+full_date = YYYY-MM-DD HH:mm:ss
+
+# Used by graph and other places where we only show small intervals
+interval_second = HH:mm:ss
+interval_minute = HH:mm
+interval_hour = MM/DD HH:mm
+interval_day = MM/DD
+interval_month = YYYY-MM
+interval_year = YYYY
+
+# Experimental feature
+use_browser_locale = false
+
+# Default timezone for user preferences. Options are 'browser' for the browser local timezone or a timezone name from IANA Time Zone database, e.g. 'UTC' or 'Europe/Amsterdam' etc.
+default_timezone = browser
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_jobs_stats.json b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_jobs_stats.json
new file mode 100644
index 0000000..99c6c09
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_jobs_stats.json
@@ -0,0 +1,542 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "iteration": 1621011410751,
+ "links": [
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "title": "ClusterController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "title": "NodeController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "system"
+ ],
+ "type": "dashboards"
+ }
+ ],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the number of jobs that were completed in the specified time interval (30 seconds).",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "jobs"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 2,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "increase(asterix_job_total_count{instance=~\"$cluster\", status=~\"$status\"}[30s])",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, status={{status}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Job Completions in last 30 Seconds",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "jobs",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the count of jobs of running and pending jobs at the specific time.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "align": null,
+ "filterable": false
+ },
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green",
+ "value": null
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ },
+ "unit": "jobs"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 4,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "asterix_job_count{instance=~\"$cluster\", status=~\"$status\"}",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, status={{status}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Job Count for Running and Pending Jobs",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "jobs",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the increase in the amount of time jobs are taking in the time interval (which is 30 seconds). This metric is only for archived jobs as we need the starting and ending time.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 8
+ },
+ "hiddenSeries": false,
+ "id": 6,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "increase(asterix_job_total_milliseconds{instance=~\"$cluster\", status=~\"$status\"}[30s]) / 1000",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, status={{status}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Cumulative Job Time Increase",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "s",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the average amount of time each job is taking. If no jobs are running, it will show 0. This metric is only for archived jobs as we need the starting and ending times.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 8
+ },
+ "hiddenSeries": false,
+ "id": 8,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null as zero",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "(sum(rate(asterix_job_total_milliseconds{instance=~\"$cluster\", status=~\"$status\"}[30s])) by (status,instance)) /\n(sum(rate(asterix_job_total_count{instance=~\"$cluster\", status=~\"$status\"}[30s])) by (status, instance)) / 1000",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, status={{status}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Average Time Taken Per Job",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "s",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": "10s",
+ "schemaVersion": 26,
+ "style": "dark",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "templating": {
+ "list": [
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "localhost:19007"
+ ],
+ "value": [
+ "localhost:19007"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_job_total_count, instance)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "cluster",
+ "options": [],
+ "query": "label_values(asterix_job_total_count, instance)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_job_total_count, status)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "status",
+ "options": [],
+ "query": "label_values(asterix_job_total_count, status)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ }
+ ]
+ },
+ "time": {
+ "from": "now-15m",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "",
+ "title": "Asterix Job Stats",
+ "uid": "e7Z7loPMk",
+ "version": 1
+}
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_memory_stats.json b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_memory_stats.json
new file mode 100644
index 0000000..d48548c
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_memory_stats.json
@@ -0,0 +1,167 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 5,
+ "links": [
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "title": "ClusterController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "title": "NodeController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "system"
+ ],
+ "type": "dashboards"
+ }
+ ],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "decbytes"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 2,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "asterix_cluster_memory_bytes",
+ "interval": "",
+ "legendFormat": "",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Memory",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "decbytes",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": "10s",
+ "schemaVersion": 26,
+ "style": "dark",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-15m",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "",
+ "title": "Asterix Memory Stats",
+ "uid": "erI4XTEGz",
+ "version": 3
+}
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_storage_stats.json b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_storage_stats.json
new file mode 100644
index 0000000..2cbb63f
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_storage_stats.json
@@ -0,0 +1,488 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "iteration": 1618605669241,
+ "links": [
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "title": "ClusterController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "title": "NodeController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "system"
+ ],
+ "type": "dashboards"
+ }
+ ],
+ "panels": [
+ {
+ "datasource": "Prometheus",
+ "description": "This panel shows the number of bytes Asterix used in storage for each dataset, index, partition.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "mappings": [],
+ "thresholds": {
+ "mode": "absolute",
+ "steps": [
+ {
+ "color": "green",
+ "value": null
+ },
+ {
+ "color": "red",
+ "value": 80
+ }
+ ]
+ },
+ "unit": "decbytes"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 2,
+ "options": {
+ "colorMode": "value",
+ "graphMode": "none",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "last"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto"
+ },
+ "pluginVersion": "7.3.5",
+ "targets": [
+ {
+ "expr": "sum(asterix_storage_bytes{instance=~\"$node\", partition=~\"$partition\", dataverse=~\"$dataverse\", dataset=~\"\\\"$dataset\\\"\", index=~\"\\\"$index\\\"\"})",
+ "instant": false,
+ "interval": "",
+ "legendFormat": "instance={{instance}}, dataset={{dataset}}, index={{index}}, partition={{partition}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "timeFrom": null,
+ "timeShift": null,
+ "title": "Asterix Storage Size",
+ "type": "stat"
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the breakdown of storage for a given node, partition, dataverse, dataset, and index",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "decbytes"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 10
+ },
+ "hiddenSeries": false,
+ "id": 6,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "asterix_storage_bytes{instance=~\"$node\", partition=~\"$partition\", dataverse=~\"$dataverse\", dataset=~\"\\\"$dataset\\\"\", index=~\"\\\"$index\\\"\"}",
+ "interval": "",
+ "legendFormat": "node={{instance}}, partition={{partition}}, dataverse={{dataverse}}, dataset={{dataset}}, index={{index}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Storage Size Breakdown",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "$$hashKey": "object:113",
+ "format": "decbytes",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "$$hashKey": "object:114",
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the rate of increase of the storage sizes of the composite key of dataset, index, and partition. It uses a least squares regression of all the values from the last 30 seconds to calculate the slope",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "Bps"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 10
+ },
+ "hiddenSeries": false,
+ "id": 4,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.5",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "deriv(asterix_storage_bytes{instance=~\"$node\", partition=~\"$partition\", dataverse=~\"$dataverse\", dataset=~\"\\\"$dataset\\\"\", index=~\"\\\"$index\\\"\"}[5m])",
+ "interval": "",
+ "legendFormat": "instance={{instance}}, dataset={{dataset}}, index={{index}}, partition={{partition}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Storage Size Rate of Increase/Decrease in the last 5 Minutes",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "Bps",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": "5m",
+ "schemaVersion": 26,
+ "style": "dark",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "templating": {
+ "list": [
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, instance)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "node",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, instance)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, partition)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "partition",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, partition)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, dataverse)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "dataverse",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, dataverse)",
+ "refresh": 1,
+ "regex": "",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": "",
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, dataset)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "dataset",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, dataset)",
+ "refresh": 1,
+ "regex": "\"([^\"]*)\"",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ },
+ {
+ "allValue": null,
+ "current": {
+ "selected": true,
+ "tags": [],
+ "text": [
+ "All"
+ ],
+ "value": [
+ "$__all"
+ ]
+ },
+ "datasource": "Prometheus",
+ "definition": "label_values(asterix_storage_bytes, index)",
+ "error": null,
+ "hide": 0,
+ "includeAll": true,
+ "label": null,
+ "multi": true,
+ "name": "index",
+ "options": [],
+ "query": "label_values(asterix_storage_bytes, index)",
+ "refresh": 1,
+ "regex": "\"([^\"]*)\"",
+ "skipUrlSync": false,
+ "sort": 0,
+ "tagValuesQuery": "",
+ "tags": [],
+ "tagsQuery": "",
+ "type": "query",
+ "useTags": false
+ }
+ ]
+ },
+ "time": {
+ "from": "now-5m",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "",
+ "title": "Asterix Storage Stats",
+ "uid": "RSzD_oEMz",
+ "version": 1
+}
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_system_stats.json b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_system_stats.json
new file mode 100644
index 0000000..f8a3da1
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/grafana_dashboards/asterix_system_stats.json
@@ -0,0 +1,479 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 2,
+ "links": [
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "cc"
+ ],
+ "title": "ClusterController Dashboards",
+ "type": "dashboards"
+ },
+ {
+ "asDropdown": true,
+ "icon": "external link",
+ "tags": [
+ "asterix",
+ "nc"
+ ],
+ "title": "NodeController Dashboards",
+ "type": "dashboards"
+ }
+ ],
+ "panels": [
+ {
+ "collapsed": false,
+ "datasource": null,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 10,
+ "panels": [],
+ "title": "CPU Usage",
+ "type": "row"
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the Asterix cpu usage",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "percent"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 0,
+ "y": 1
+ },
+ "hiddenSeries": false,
+ "id": 2,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "rate(process_cpu_seconds_total[1m]) * 100",
+ "interval": "",
+ "legendFormat": "{{instance}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix CPU Usage Percentage",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "percent",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the system cpu usage (per cpu)",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "percent"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 1
+ },
+ "hiddenSeries": false,
+ "id": 6,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "(1-rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100",
+ "interval": "",
+ "legendFormat": "CPU {{cpu}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "System CPU Usage",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "percent",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the proportion of non idle cpu that Asterix is using",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "percent"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 10
+ },
+ "hiddenSeries": false,
+ "id": 8,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "(rate(process_cpu_seconds_total[1m]) * 100) / ignoring (instance, job) group_left avg(1-rate(node_cpu_seconds_total{mode=\"idle\"}[1m]))",
+ "interval": "",
+ "legendFormat": "{{instance}}",
+ "queryType": "randomWalk",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Asterix Non Idle CPU Usage",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "percent",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "collapsed": false,
+ "datasource": null,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 18
+ },
+ "id": 4,
+ "panels": [],
+ "title": "File Descriptor Usage",
+ "type": "row"
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "Prometheus",
+ "description": "This panel shows the Asterix file descriptor usage (number of file descriptors open / pre set file descriptor max set by Prometheus)",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "percent"
+ },
+ "overrides": []
+ },
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 14,
+ "w": 23,
+ "x": 0,
+ "y": 19
+ },
+ "hiddenSeries": false,
+ "id": 12,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "nullPointMode": "null",
+ "options": {
+ "alertThreshold": true
+ },
+ "percentage": false,
+ "pluginVersion": "7.3.1",
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "expr": "(process_open_fds / process_max_fds) * 100",
+ "interval": "",
+ "legendFormat": "{{instance}}",
+ "refId": "A"
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "File Descriptor Usage",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "percent",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": "10s",
+ "schemaVersion": 26,
+ "style": "dark",
+ "tags": [
+ "asterix",
+ "system"
+ ],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-15m",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "",
+ "title": "Asterix System Stats",
+ "uid": "l0Rj8oEMk",
+ "version": 9
+}
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/instance/cc.conf b/asterixdb/asterix-server/src/main/opt/ansible/conf/instance/cc.conf
new file mode 100644
index 0000000..d9360bf
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/instance/cc.conf
@@ -0,0 +1,32 @@
+; Licensed to the Apache Software Foundation (ASF) under one
+; or more contributor license agreements. See the NOTICE file
+; distributed with this work for additional information
+; regarding copyright ownership. The ASF licenses this file
+; to you under the Apache License, Version 2.0 (the
+; "License"); you may not use this file except in compliance
+; with the License. You may obtain a copy of the License at
+;
+; http://www.apache.org/licenses/LICENSE-2.0
+;
+; Unless required by applicable law or agreed to in writing,
+; software distributed under the License is distributed on an
+; "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+; KIND, either express or implied. See the License for the
+; specific language governing permissions and limitations
+; under the License.
+
+[common]
+log.level=INFO
+log.dir=logs
+
+[nc]
+txn.log.dir=txnlog
+iodevices=iodevice
+command=asterixnc
+
+[nc/1]
+address=localhost
+
+[cc]
+address=localhost
+
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml b/asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml
index b5cd1ae..4eb42a4 100644
--- a/asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/instance_settings.yml
@@ -54,3 +54,25 @@
# The port for Yourkit profiling
profiler_port: 20002
+
+# The version of the node exporter
+node_exporter_version: 1.0.1
+
+# The version of the prometheus
+prometheus_version: 2.22.1
+
+# The version of the grafana
+grafana_version: 7.3.5
+
+# The os to download monitoring binaries
+host_os: darwin-amd64
+
+# ports for cc
+cc_ports:
+ localhost:
+ - "19007"
+
+# ports for nc
+nc_ports:
+ localhost:
+ - "19010"
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/inventory b/asterixdb/asterix-server/src/main/opt/ansible/conf/inventory
index 140ed43..d3d7079 100644
--- a/asterixdb/asterix-server/src/main/opt/ansible/conf/inventory
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/inventory
@@ -24,4 +24,4 @@
localhost
[ncs]
-localhost
+localhost
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/monitoring_inventory b/asterixdb/asterix-server/src/main/opt/ansible/conf/monitoring_inventory
new file mode 100644
index 0000000..354d6bf
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/monitoring_inventory
@@ -0,0 +1,25 @@
+; Licensed to the Apache Software Foundation (ASF) under one
+; or more contributor license agreements. See the NOTICE file
+; distributed with this work for additional information
+; regarding copyright ownership. The ASF licenses this file
+; to you under the Apache License, Version 2.0 (the
+; "License"); you may not use this file except in compliance
+; with the License. You may obtain a copy of the License at
+;
+; http://www.apache.org/licenses/LICENSE-2.0
+;
+; Unless required by applicable law or agreed to in writing,
+; software distributed under the License is distributed on an
+; "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+; KIND, either express or implied. See the License for the
+; specific language governing permissions and limitations
+; under the License.
+
+[prometheus]
+localhost
+
+[node_exporter]
+localhost
+
+[grafana]
+localhost
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/conf/prometheus.conf.j2 b/asterixdb/asterix-server/src/main/opt/ansible/conf/prometheus.conf.j2
new file mode 100644
index 0000000..e5e549f
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/conf/prometheus.conf.j2
@@ -0,0 +1,62 @@
+{#
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+#}
+
+global:
+ scrape_interval: 1s
+
+scrape_configs:
+ - job_name: 'prometheus'
+ scrape_interval: 2s
+ static_configs:
+ - targets: ['localhost:8080']
+
+ - job_name: 'node_exporter'
+ scrape_interval: 2s
+ static_configs:
+ {%- for host in groups['node_exporter'] %}
+
+ - targets: ['{{ host }}:9100']
+
+ {%- endfor %}
+
+ - job_name: 'cluster_controller'
+ scrape_interval: 5s
+ static_configs:
+ - targets:
+ {%- for host in groups['ncs'] %}
+ {%- for port in cc_ports[host] %}
+
+ - '{{ host }}:{{ port }}'
+
+ {%- endfor %}
+ {%- endfor %}
+
+ - job_name: 'node_controller'
+ scrape_interval: 5s
+ static_configs:
+ - targets:
+ {%- for host in groups['ncs'] %}
+ {%- for port in nc_ports[host] %}
+
+ - '{{ host }}:{{ port }}'
+
+ {%- endfor %}
+ {%- endfor %}
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/grafana.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/grafana.yml
new file mode 100644
index 0000000..58a8034
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/grafana.yml
@@ -0,0 +1,27 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- name: "Grafana server started"
+ shell: |
+ nohup grafana-{{ grafana_version }}/bin/grafana-server --config=conf/defaults.ini --homepath=grafana-{{ grafana_version }} >> "logs/grafana.log" 2>&1 &
+ sleep 1
+ args:
+ chdir: "{{ binarydir }}"
+ async: 10
+ poll: 0
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_install.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_install.yml
new file mode 100644
index 0000000..41f9f80
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_install.yml
@@ -0,0 +1,86 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- hosts: all
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+ - name: Ensure the binarydir exists
+ file:
+ path: "{{ binarydir }}"
+ state: directory
+
+ - name: Ensure the log directory exists
+ file:
+ path: "{{ binarydir }}/logs"
+ state: directory
+
+ - name: Ensure the conf directory exists
+ file:
+ path: "{{ binarydir }}/conf"
+ state: directory
+
+- hosts: node_exporter
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+
+ - name: Install prometheus node exporter
+ unarchive:
+ src: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.{{ host_os }}.tar.gz"
+ dest: "{{ binarydir }}"
+ remote_src: yes
+
+- hosts: prometheus
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+
+ - name: Install prometheus
+ unarchive:
+ src: "https://github.com/prometheus/prometheus/releases/download/v{{ prometheus_version }}/prometheus-{{ prometheus_version }}.{{ host_os }}.tar.gz"
+ dest: "{{ binarydir }}"
+ remote_src: yes
+
+- hosts: grafana
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+
+ - name: Install grafana
+ unarchive:
+ src: "https://dl.grafana.com/oss/release/grafana-{{ grafana_version }}.{{ host_os }}.tar.gz"
+ dest: "{{ binarydir }}"
+ remote_src: yes
+
+ - name: Copy grafana config file
+ copy:
+ src: ../conf/defaults.ini
+ dest: "{{ binarydir }}/conf"
+
+ - name: Copy grafana datasources provisioning
+ copy:
+ src: ../conf/datasources.yml
+ dest: "{{ binarydir }}/grafana-{{ grafana_version }}/conf/provisioning/datasources"
+
+ - name: Copy grafana dashboards provisioning
+ copy:
+ src: ../conf/dashboards.yml
+ dest: "{{ binarydir }}/grafana-{{ grafana_version }}/conf/provisioning/dashboards"
+
+ - name: Copy grafana dashboards
+ copy:
+ src: ../conf/grafana_dashboards
+ dest: "{{ binarydir }}"
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_start.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_start.yml
new file mode 100644
index 0000000..5d7b372
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_start.yml
@@ -0,0 +1,33 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- hosts: node_exporter
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+ - include: prometheus_node_exporter.yml
+
+- hosts: prometheus
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+ - include: prometheus.yml
+
+- hosts: grafana
+ tasks:
+ - include_vars: ../conf/instance_settings.yml
+ - include: grafana.yml
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_stop.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_stop.yml
new file mode 100644
index 0000000..1c181ad
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/monitoring_stop.yml
@@ -0,0 +1,36 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- hosts: grafana
+ tasks:
+ - name: Stop grafana
+ shell: kill -9 `lsof -i :3000 | grep LISTEN | awk -F ' ' '{print $2}'`
+ failed_when: false
+
+- hosts: prometheus
+ tasks:
+ - name: Stop prometheus
+ shell: kill -9 `lsof -i :8080 | grep LISTEN | awk -F ' ' '{print $2}'`
+ failed_when: false
+
+- hosts: node_exporter
+ tasks:
+ - name: Stop node_exporter
+ shell: kill -9 `lsof -i :9100 | grep LISTEN | awk -F ' ' '{print $2}'`
+ failed_when: false
\ No newline at end of file
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus.yml
new file mode 100644
index 0000000..e83a96e
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus.yml
@@ -0,0 +1,32 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- name: config file
+ template:
+ src: ../conf/prometheus.conf.j2
+ dest: "{{ binarydir }}/conf"
+
+- name: Start prometheus service
+ shell: |
+ nohup "prometheus-{{ prometheus_version }}.darwin-amd64/prometheus" --config.file=conf/prometheus.conf.j2 --web.listen-address=:8080 >> logs/prometheus.log 2>&1 &
+ sleep 1
+ args:
+ chdir: "{{ binarydir }}"
+ async: 10
+ poll: 0
diff --git a/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus_node_exporter.yml b/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus_node_exporter.yml
new file mode 100644
index 0000000..4aa4a99
--- /dev/null
+++ b/asterixdb/asterix-server/src/main/opt/ansible/yaml/prometheus_node_exporter.yml
@@ -0,0 +1,27 @@
+# ------------------------------------------------------------
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ------------------------------------------------------------
+
+- name: Start node_exporter service
+ shell: |
+ nohup "node_exporter-{{ node_exporter_version }}.darwin-amd64/node_exporter" >> logs/node_exporter.log 2>&1 &
+ sleep 1
+ args:
+ chdir: "{{ binarydir }}"
+ async: 10
+ poll: 0
\ No newline at end of file
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml
index 0f1be96..8e2b5bf 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml
@@ -99,5 +99,10 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</dependency>
+ <dependency>
+ <groupId>io.prometheus</groupId>
+ <artifactId>simpleclient</artifactId>
+ <version>0.9.0</version>
+ </dependency>
</dependencies>
</project>
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
index f11e7ff..8d280a4 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java
@@ -18,6 +18,8 @@
*/
package org.apache.hyracks.control.cc;
+import static io.prometheus.client.CollectorRegistry.defaultRegistry;
+
import java.io.File;
import java.io.FileReader;
import java.lang.reflect.Constructor;
@@ -239,7 +241,10 @@
workQueue.start();
connectNCs();
LOGGER.log(Level.INFO, "Started ClusterControllerService");
+ defaultRegistry.register(new JobsExporter(jobManager));
+ defaultRegistry.register(new MemoryExporter(resourceManager));
notifyApplication();
+
}
protected void startWebServers() throws Exception {
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/JobsExporter.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/JobsExporter.java
new file mode 100644
index 0000000..0e2baeb
--- /dev/null
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/JobsExporter.java
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hyracks.control.cc;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+
+import org.apache.hyracks.api.job.JobStatus;
+import org.apache.hyracks.control.cc.job.IJobManager;
+import org.apache.hyracks.control.cc.job.JobRun;
+
+import io.prometheus.client.Collector;
+import io.prometheus.client.Counter;
+import io.prometheus.client.Gauge;
+
+public class JobsExporter extends Collector {
+ private final IJobManager jobManager;
+ private HashMap<String, Double> jobsCount;
+ private HashSet<Long> jobsTerminatedIds;
+
+ private static final Counter jobsSecondsCounter = Counter.build().name("asterix_job_total_milliseconds")
+ .help("total milliseconds taken by jobs").labelNames("status").register();
+
+ private static final Counter jobsCountCounter = Counter.build().name("asterix_job_total_count")
+ .help("current total count of jobs").labelNames("status").register();
+
+ private static final Gauge jobsCountGauge =
+ Gauge.build().name("asterix_job_count").help("current count of jobs").labelNames("status").register();
+
+ public JobsExporter(IJobManager jobManager) {
+ this.jobManager = jobManager;
+ this.jobsCount = new HashMap<>();
+ this.jobsTerminatedIds = new HashSet<>();
+
+ // initialize counter to 0
+ for (JobStatus status : JobStatus.values()) {
+ if (status.equals(JobStatus.FAILURE) || status.equals(JobStatus.FAILURE_BEFORE_EXECUTION)
+ || status.equals(JobStatus.TERMINATED)) {
+ jobsSecondsCounter.labels(status.toString()).inc(0);
+ }
+ jobsCountCounter.labels(status.toString()).inc(0);
+ }
+ resetJobCount();
+ }
+
+ @Override
+ public List<MetricFamilySamples> collect() {
+ List<MetricFamilySamples> mfs = new ArrayList<>();
+
+ HashSet<Long> tempTerminatedIds = new HashSet<>();
+ for (JobRun job : jobManager.getArchivedJobs()) {
+ long jobId = job.getJobId().getId();
+ if (!jobsTerminatedIds.contains(jobId)) {
+ String status = job.getStatus().toString();
+ jobsSecondsCounter.labels(status).inc(job.getEndTime() - job.getStartTime());
+ jobsCountCounter.labels(status).inc();
+ }
+ tempTerminatedIds.add(jobId);
+ }
+ jobsTerminatedIds = tempTerminatedIds;
+
+ resetJobCount();
+ updateJobCount(jobManager.getRunningJobs());
+ updateJobCount(jobManager.getPendingJobs());
+ for (String jobElement : jobsCount.keySet()) {
+ jobsCountGauge.labels(jobElement).set(jobsCount.get(jobElement));
+ }
+
+ return mfs;
+ }
+
+ private void updateJobCount(Collection<JobRun> jobs) {
+ for (JobRun job : jobs) {
+ String jobStatus = job.getStatus().toString();
+ jobsCount.put(jobStatus, jobsCount.get(jobStatus) + 1);
+ }
+ }
+
+ private void resetJobCount() {
+ for (JobStatus status : JobStatus.values()) {
+ if (status.equals(JobStatus.RUNNING) || (status.equals(JobStatus.PENDING))) {
+ this.jobsCount.put(status.toString(), 0.0);
+ }
+ }
+ }
+}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/MemoryExporter.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/MemoryExporter.java
new file mode 100644
index 0000000..2e162eb
--- /dev/null
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/MemoryExporter.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hyracks.control.cc;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hyracks.control.cc.scheduler.IResourceManager;
+
+import io.prometheus.client.Collector;
+import io.prometheus.client.Gauge;
+
+public class MemoryExporter extends Collector {
+ private IResourceManager resourceManager;
+ private static final Gauge clusterMemorySizeBytes =
+ Gauge.build().name("asterix_cluster_memory_bytes").help("cluster aggregate bytes available").register();
+
+ public MemoryExporter(IResourceManager resourceManager) {
+ this.resourceManager = resourceManager;
+ }
+
+ @Override
+ public List<MetricFamilySamples> collect() {
+ List<MetricFamilySamples> mfs = new ArrayList<MetricFamilySamples>();
+
+ clusterMemorySizeBytes.set(resourceManager.getMaximumCapacity().getAggregatedMemoryByteSize()
+ - resourceManager.getCurrentCapacity().getAggregatedMemoryByteSize());
+
+ return mfs;
+ }
+}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
index d148145..fe04a9e 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/CCConfig.java
@@ -84,7 +84,8 @@
OptionTypes.STRING,
(Function<IApplicationConfig, String>) appConfig -> FileUtil
.joinPath(appConfig.getString(ControllerConfig.Option.DEFAULT_DIR), "passwd"),
- ControllerConfig.Option.DEFAULT_DIR.cmdline() + "/passwd");
+ ControllerConfig.Option.DEFAULT_DIR.cmdline() + "/passwd"),
+ METRICS_PORT(POSITIVE_INTEGER, 19007);
private final IOptionType parser;
private Object defaultValue;
@@ -206,6 +207,8 @@
return "The password to the provided key store";
case CREDENTIAL_FILE:
return "Path to HTTP basic credentials";
+ case METRICS_PORT:
+ return "Port the CC exports monitoring metrics to";
default:
throw new IllegalStateException("NYI: " + this);
}
@@ -481,4 +484,11 @@
return getAppConfig().getString(Option.CREDENTIAL_FILE);
}
+ public void setMetricsPort(int metricsPort) {
+ configManager.set(CCConfig.Option.METRICS_PORT, metricsPort);
+ }
+
+ public int getMetricsPort() {
+ return getAppConfig().getInt(CCConfig.Option.METRICS_PORT);
+ }
}
diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
index 01cb9bf..fa159a6 100644
--- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
+++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/controllers/NCConfig.java
@@ -103,7 +103,8 @@
OptionTypes.STRING,
(Function<IApplicationConfig, String>) appConfig -> FileUtil
.joinPath(appConfig.getString(ControllerConfig.Option.DEFAULT_DIR), "passwd"),
- ControllerConfig.Option.DEFAULT_DIR.cmdline() + "/passwd");
+ ControllerConfig.Option.DEFAULT_DIR.cmdline() + "/passwd"),
+ METRICS_PORT(INTEGER, 19010);
private final IOptionType parser;
private final String defaultValueDescription;
@@ -250,6 +251,8 @@
return "List of environment variables to set when invoking the Python interpreter for Python UDFs. E.g. FOO=1";
case CREDENTIAL_FILE:
return "Path to HTTP basic credentials";
+ case METRICS_PORT:
+ return "Port the NC exports monitoring metrics to";
default:
throw new IllegalStateException("Not yet implemented: " + this);
}
@@ -625,4 +628,11 @@
return getAppConfig().getString(Option.CREDENTIAL_FILE);
}
+ public void setMetricsPort(int metricsPort) {
+ configManager.set(nodeId, Option.METRICS_PORT, metricsPort);
+ }
+
+ public int getMetricsPort() {
+ return appConfig.getInt(Option.METRICS_PORT);
+ }
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ib9aa81fa2e58954846218e3902562685c8cd09e1
Gerrit-Change-Number: 11544
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon <im...@uci.edu>
Gerrit-Reviewer: Krish Avvari <av...@gmail.com>
Gerrit-MessageType: newchange
Change in asterixdb[master]: Added prometheus monitoring to asterix
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Jenkins <je...@fulliautomatix.ics.uci.edu>:
Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544 )
Change subject: Added prometheus monitoring to asterix
......................................................................
Patch Set 1: Integration-Tests-1
Integration Tests Failed
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/11998/ : UNSTABLE
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ib9aa81fa2e58954846218e3902562685c8cd09e1
Gerrit-Change-Number: 11544
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon <im...@uci.edu>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Krish Avvari <av...@gmail.com>
Gerrit-CC: Anon. E. Moose #1000171
Gerrit-Comment-Date: Fri, 21 May 2021 00:23:05 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment
Change in asterixdb[master]: Added prometheus monitoring to asterix
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
From Jenkins <je...@fulliautomatix.ics.uci.edu>:
Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544 )
Change subject: Added prometheus monitoring to asterix
......................................................................
Patch Set 1:
Integration Tests Failed
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/12028/ : UNSTABLE
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ib9aa81fa2e58954846218e3902562685c8cd09e1
Gerrit-Change-Number: 11544
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon <im...@uci.edu>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Krish Avvari <av...@gmail.com>
Gerrit-Comment-Date: Fri, 28 May 2021 21:35:09 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: No
Gerrit-MessageType: comment
Change in asterixdb[master]: Added prometheus monitoring to asterix
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544 )
Change subject: Added prometheus monitoring to asterix
......................................................................
Patch Set 1: -Contrib
Analytics Compatibility Compilation Successful
https://cbjenkins.page.link/FtF45QMQUgdoSGak6 : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ib9aa81fa2e58954846218e3902562685c8cd09e1
Gerrit-Change-Number: 11544
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon <im...@uci.edu>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Krish Avvari <av...@gmail.com>
Gerrit-Comment-Date: Mon, 24 May 2021 22:18:43 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment
Change in asterixdb[master]: Added prometheus monitoring to asterix
Posted by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>.
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544 )
Change subject: Added prometheus monitoring to asterix
......................................................................
Patch Set 1: Contrib-2
Analytics Compatibility Tests Failed
https://cbjenkins.page.link/1gdXG9hfaUsSGwMz8 : UNSTABLE
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/11544
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ib9aa81fa2e58954846218e3902562685c8cd09e1
Gerrit-Change-Number: 11544
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon <im...@uci.edu>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Krish Avvari <av...@gmail.com>
Gerrit-Comment-Date: Tue, 25 May 2021 01:40:04 +0000
Gerrit-HasComments: No
Gerrit-Has-Labels: Yes
Gerrit-MessageType: comment