You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/02/15 04:10:55 UTC
[GitHub] [ozone] k5342 opened a new pull request #3088: HDDS-6317: Export ReconTaskStatus as Prometheus metrics
k5342 opened a new pull request #3088:
URL: https://github.com/apache/ozone/pull/3088
## What changes were proposed in this pull request?
Add an ability to Recon to export a new metrics about timestamps from `ReconTaskStatus` table in Derby DB. Recon also has OzoneManagerSyncMetrics for OM snapshot related tasks but this metrics focuses entire Recon sub-tasks.
```
# TYPE recon_task_status_metrics_last_updated_seq_number gauge
recon_task_status_metrics_last_updated_seq_number{type="ContainerHealthTask",hostname="localhost"} 0
recon_task_status_metrics_last_updated_seq_number{type="ContainerKeyMapperTask",hostname="localhost"} 2
recon_task_status_metrics_last_updated_seq_number{type="FileSizeCountTask",hostname="localhost"} 2
recon_task_status_metrics_last_updated_seq_number{type="NSSummaryTask",hostname="localhost"} 2
recon_task_status_metrics_last_updated_seq_number{type="OmDeltaRequest",hostname="localhost"} 2
recon_task_status_metrics_last_updated_seq_number{type="OmSnapshotRequest",hostname="localhost"} 2
recon_task_status_metrics_last_updated_seq_number{type="PipelineSyncTask",hostname="localhost"} 0
recon_task_status_metrics_last_updated_seq_number{type="TableCountTask",hostname="localhost"} 2
# TYPE recon_task_status_metrics_last_updated_timestamp gauge
recon_task_status_metrics_last_updated_timestamp{type="ContainerHealthTask",hostname="localhost"} 1644897761573
recon_task_status_metrics_last_updated_timestamp{type="ContainerKeyMapperTask",hostname="localhost"} 1644896921928
recon_task_status_metrics_last_updated_timestamp{type="FileSizeCountTask",hostname="localhost"} 1644896921932
recon_task_status_metrics_last_updated_timestamp{type="NSSummaryTask",hostname="localhost"} 1644896921919
recon_task_status_metrics_last_updated_timestamp{type="OmDeltaRequest",hostname="localhost"} 1644897522075
recon_task_status_metrics_last_updated_timestamp{type="OmSnapshotRequest",hostname="localhost"} 1644896920719
recon_task_status_metrics_last_updated_timestamp{type="PipelineSyncTask",hostname="localhost"} 1644897762221
recon_task_status_metrics_last_updated_timestamp{type="TableCountTask",hostname="localhost"} 1644896921923
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6317
## How was this patch tested?
Ran on localhost && CI check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx merged pull request #3088: HDDS-6317: Export ReconTaskStatus as Prometheus metrics
Posted by GitBox <gi...@apache.org>.
avijayanhwx merged pull request #3088:
URL: https://github.com/apache/ozone/pull/3088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] k5342 commented on pull request #3088: HDDS-6317: Export ReconTaskStatus as Prometheus metrics
Posted by GitBox <gi...@apache.org>.
k5342 commented on pull request #3088:
URL: https://github.com/apache/ozone/pull/3088#issuecomment-1053304990
I've rebased to fix conflicts.
@avijayanhwx Could you help to review this PR?
IMO, we don't have to put a cache here. This is a trade-off between the DB load and time-resolution. I think N seconds polling is small enough load and acceptable.
From a security standpoint, I think the load is from internal metrics system only (means not to be published as external API because `/prom` is protected, malformed access or F5-attack can be ignored). So it is expected to be small as well and we can take time-resolution in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] k5342 commented on pull request #3088: HDDS-6317: Export ReconTaskStatus as Prometheus metrics
Posted by GitBox <gi...@apache.org>.
k5342 commented on pull request #3088:
URL: https://github.com/apache/ozone/pull/3088#issuecomment-1057606830
Thank you for the review! @avijayanhwx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] k5342 commented on a change in pull request #3088: HDDS-6317: Export ReconTaskStatus as Prometheus metrics
Posted by GitBox <gi...@apache.org>.
k5342 commented on a change in pull request #3088:
URL: https://github.com/apache/ozone/pull/3088#discussion_r806461221
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/ReconTaskStatusMetrics.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import com.google.inject.Inject;
+import com.google.inject.Singleton;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.MetricsTag;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+
+import java.util.List;
+
+/**
+ * Ship ReconTaskStatus table on persistent DB as a metrics.
+ */
+@Singleton
+@Metrics(about = "Recon Task Status Metrics", context = OzoneConsts.OZONE)
+public class ReconTaskStatusMetrics implements MetricsSource {
Review comment:
To group metrics as ReconTask, I created a new MetricsSource instead of `@Metric` annotation. I followed other MetricsSource class naming and context. Suggestion of class naming or metrics naming is welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] k5342 commented on a change in pull request #3088: HDDS-6317: Export ReconTaskStatus as Prometheus metrics
Posted by GitBox <gi...@apache.org>.
k5342 commented on a change in pull request #3088:
URL: https://github.com/apache/ozone/pull/3088#discussion_r806461221
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/ReconTaskStatusMetrics.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import com.google.inject.Inject;
+import com.google.inject.Singleton;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.MetricsTag;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+
+import java.util.List;
+
+/**
+ * Ship ReconTaskStatus table on persistent DB as a metrics.
+ */
+@Singleton
+@Metrics(about = "Recon Task Status Metrics", context = OzoneConsts.OZONE)
+public class ReconTaskStatusMetrics implements MetricsSource {
Review comment:
To group metrics as ReconTask, I created a new MetricsSource instead of `@Metric` annotation. I followed other MetricsSource class naming and context. Suggestion of class naming or metrics naming is welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] avijayanhwx commented on pull request #3088: HDDS-6317: Export ReconTaskStatus as Prometheus metrics
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #3088:
URL: https://github.com/apache/ozone/pull/3088#issuecomment-1057182449
Thank you for the contribution @k5342.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kuenishi commented on a change in pull request #3088: HDDS-6317: Export ReconTaskStatus as Prometheus metrics
Posted by GitBox <gi...@apache.org>.
kuenishi commented on a change in pull request #3088:
URL: https://github.com/apache/ozone/pull/3088#discussion_r807548194
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/ReconTaskStatusMetrics.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import com.google.inject.Inject;
+import com.google.inject.Singleton;
+import org.apache.hadoop.metrics2.MetricsCollector;
+import org.apache.hadoop.metrics2.MetricsInfo;
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.MetricsSource;
+import org.apache.hadoop.metrics2.MetricsTag;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.Interns;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+
+import java.util.List;
+
+/**
+ * Ship ReconTaskStatus table on persistent DB as a metrics.
+ */
+@Singleton
+@Metrics(about = "Recon Task Status Metrics", context = OzoneConsts.OZONE)
+public class ReconTaskStatusMetrics implements MetricsSource {
+
+ private static final String SOURCE_NAME =
+ ReconTaskStatusMetrics.class.getSimpleName();
+
+ @Inject
+ private ReconTaskStatusDao reconTaskStatusDao;
+
+ private static final MetricsInfo RECORD_INFO_LAST_UPDATED_TS =
+ Interns.info("lastUpdatedTimestamp",
+ "Last updated timestamp of corresponding Recon Task");
+
+ private static final MetricsInfo RECORD_INFO_LAST_UPDATED_SEQ =
+ Interns.info("lastUpdatedSeqNumber",
+ "Last updated sequence number of corresponding Recon Task");
+
+ public void register() {
+ DefaultMetricsSystem.instance()
+ .register(SOURCE_NAME, "Recon Task Metrics", this);
+ }
+
+ public void unregister() {
+ DefaultMetricsSystem.instance()
+ .unregisterSource(SOURCE_NAME);
+ }
+
+ public void getMetrics(MetricsCollector collector, boolean all) {
+ List<ReconTaskStatus> rows = reconTaskStatusDao.findAll();
+ rows.forEach((rts) -> {
+ MetricsRecordBuilder builder = collector.addRecord(SOURCE_NAME);
+ builder.add(
+ new MetricsTag(
+ Interns.info("type", "Recon Task type"),
+ rts.getTaskName()));
+ builder.addGauge(RECORD_INFO_LAST_UPDATED_TS,
+ rts.getLastUpdatedTimestamp());
+ builder.addGauge(RECORD_INFO_LAST_UPDATED_SEQ,
Review comment:
Given that it's sequence numbers, it should be monotonically increasing. Maybe it's better be counters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] k5342 commented on pull request #3088: HDDS-6317: Export ReconTaskStatus as Prometheus metrics
Posted by GitBox <gi...@apache.org>.
k5342 commented on pull request #3088:
URL: https://github.com/apache/ozone/pull/3088#issuecomment-1042491770
> if we have a metric system pulling this data every N seconds, would it be better to cache this info rather than do a SQL lookup everytime?
Yes, I can put a cache inside `getMetrics()` but MetricsSystem already has a similar feature [^1] so individual cache may be out of the way of MetricsSystem.
MetricsSystem defaults to expose metrics from sources every 10 seconds by `publishMetrics()` but I think `PrometheusServlet#doGet()` uses `publishMetricsNow()` [^2] and this disables the feature. Why is this used here? Could we switch to `publishMetrics()` instead?
If we switch to `publishMetrics()`:
- Pros) Simple. We don’t need to worry about individual cache on each metrics.
- Cons) The metric resolution would become 10 seconds (that is configurable. I guess), which may let us miss important events for users who use short scrape interval (but, how many users? – Prometheus defaults to scrape in 1 min). This change is given to PrometheusServlet, which affects every component e.g. OM/SCM/DN.
[^1]: https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java#L360-L397
[^2]: https://github.com/apache/ozone/blob/129325829ebdbea5359db2081d2f9f3850d5bb1b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/PrometheusServlet.java#L61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org