You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2021/11/03 12:45:15 UTC
[skywalking] branch opt created (now 797268b)
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a change to branch opt
in repository https://gitbox.apache.org/repos/asf/skywalking.git.
at 797268b Optimize metrics of minute dimensionality persistence. - The value of metrics, which has declaration of the default value and current value equals the default value logically, the whole row wouldn't be pushed into database.
This branch includes the following new commits:
new 797268b Optimize metrics of minute dimensionality persistence. - The value of metrics, which has declaration of the default value and current value equals the default value logically, the whole row wouldn't be pushed into database.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[skywalking] 01/01: Optimize metrics of minute dimensionality
persistence. - The value of metrics,
which has declaration of the default value and current value equals the
default value logically, the whole row wouldn't be pushed into database.
Posted by wu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch opt
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 797268b75b6185f44e1925975c3cc899d47ff065
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Wed Nov 3 20:42:14 2021 +0800
Optimize metrics of minute dimensionality persistence.
- The value of metrics, which has declaration of the default value and current value equals the default value logically, the whole row wouldn't be pushed into database.
---
CHANGES.md | 8 +++--
.../core/alarm/provider/RunningRuleTest.java | 30 ++++++++++++++++
.../analysis/meter/function/avg/AvgFunction.java | 10 ++++++
.../meter/function/latest/LatestFunction.java | 10 ++++++
.../analysis/meter/function/sum/SumFunction.java | 10 ++++++
.../server/core/analysis/metrics/ApdexMetrics.java | 10 ++++++
.../server/core/analysis/metrics/CPMMetrics.java | 10 ++++++
.../server/core/analysis/metrics/CountMetrics.java | 10 ++++++
.../core/analysis/metrics/DoubleAvgMetrics.java | 11 ++++++
.../core/analysis/metrics/HavingDefaultValue.java | 42 ++++++++++++++++++++++
.../core/analysis/metrics/LongAvgMetrics.java | 10 ++++++
.../core/analysis/metrics/MaxDoubleMetrics.java | 11 ++++++
.../core/analysis/metrics/MaxLongMetrics.java | 3 --
.../oap/server/core/analysis/metrics/Metrics.java | 2 +-
.../core/analysis/metrics/MinDoubleMetrics.java | 11 ++++++
.../core/analysis/metrics/MinLongMetrics.java | 10 ++++++
.../core/analysis/metrics/PercentMetrics.java | 10 ++++++
.../server/core/analysis/metrics/PxxMetrics.java | 10 ++++++
.../server/core/analysis/metrics/RateMetrics.java | 10 ++++++
.../server/core/analysis/metrics/SumMetrics.java | 10 ++++++
.../analysis/worker/MetricsPersistentWorker.java | 40 +++++++++++++++++----
21 files changed, 265 insertions(+), 13 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index cf6a9ae..a0de11e 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -43,6 +43,8 @@ Release Notes.
* Fix ElasticSearch implementation of `queryMetricsValues` and `readLabeledMetricsValues` doesn't fill default values
when no available data in the ElasticSearch server.
* Fix config yaml data type conversion bug when meets special character like !.
+* Optimize metrics of minute dimensionality persistence. The value of metrics, which has declaration of the default
+ value and current value equals the default value logically, the whole row wouldn't be pushed into database.
#### UI
@@ -62,9 +64,9 @@ Release Notes.
#### Documentation
* Enhance documents about the data report and query protocols.
-* Restructure documents about receivers and fetchers.
- 1. Remove general receiver and fetcher docs
- 2. Add more specific menu with docs to help users to find documents easier.
+* Restructure documents about receivers and fetchers.
+ 1. Remove general receiver and fetcher docs
+ 2. Add more specific menu with docs to help users to find documents easier.
All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/101?closed=1)
diff --git a/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRuleTest.java b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRuleTest.java
index 2802722..3ffcae2 100644
--- a/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRuleTest.java
+++ b/oap-server/server-alarm-plugin/src/test/java/org/apache/skywalking/oap/server/core/alarm/provider/RunningRuleTest.java
@@ -476,6 +476,16 @@ public class RunningRuleTest {
public int remoteHashCode() {
return 0;
}
+
+ @Override
+ public boolean haveDefault() {
+ return false;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return false;
+ }
}
private class MockMultipleValueMetrics extends Metrics implements MultiIntValuesHolder {
@@ -529,6 +539,16 @@ public class RunningRuleTest {
public RemoteData.Builder serialize() {
return null;
}
+
+ @Override
+ public boolean haveDefault() {
+ return false;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return false;
+ }
}
private class MockLabeledValueMetrics extends Metrics implements LabeledValueHolder {
@@ -576,6 +596,16 @@ public class RunningRuleTest {
public RemoteData.Builder serialize() {
return null;
}
+
+ @Override
+ public boolean haveDefault() {
+ return false;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return false;
+ }
}
private void assertLabeled(AlarmRule alarmRule) {
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/avg/AvgFunction.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/avg/AvgFunction.java
index dc2dec3..85f128d 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/avg/AvgFunction.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/avg/AvgFunction.java
@@ -213,4 +213,14 @@ public abstract class AvgFunction extends Metrics implements AcceptableValue<Lon
public int hashCode() {
return Objects.hash(entityId, getTimeBucket());
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/latest/LatestFunction.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/latest/LatestFunction.java
index 9e49426..29a14a6 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/latest/LatestFunction.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/latest/LatestFunction.java
@@ -182,4 +182,14 @@ public abstract class LatestFunction extends Metrics implements AcceptableValue<
public int hashCode() {
return Objects.hash(entityId, getTimeBucket());
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/sum/SumFunction.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/sum/SumFunction.java
index d229f6b..7436696 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/sum/SumFunction.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/sum/SumFunction.java
@@ -183,4 +183,14 @@ public abstract class SumFunction extends Metrics implements AcceptableValue<Lon
public int hashCode() {
return Objects.hash(getEntityId(), getTimeBucket());
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/ApdexMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/ApdexMetrics.java
index 14b4750..be1ff82 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/ApdexMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/ApdexMetrics.java
@@ -95,4 +95,14 @@ public abstract class ApdexMetrics extends Metrics implements IntValueHolder {
public int getValue() {
return value;
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/CPMMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/CPMMetrics.java
index bc0f02f..948c1f1 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/CPMMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/CPMMetrics.java
@@ -57,5 +57,15 @@ public abstract class CPMMetrics extends Metrics implements LongValueHolder {
public void calculate() {
this.value = total / getDurationInMinute();
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/CountMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/CountMetrics.java
index 9a22314..1de36db 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/CountMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/CountMetrics.java
@@ -51,4 +51,14 @@ public abstract class CountMetrics extends Metrics implements LongValueHolder {
@Override
public void calculate() {
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/DoubleAvgMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/DoubleAvgMetrics.java
index 9356c21..f5e7f759 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/DoubleAvgMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/DoubleAvgMetrics.java
@@ -64,4 +64,15 @@ public abstract class DoubleAvgMetrics extends Metrics implements DoubleValueHol
public final void calculate() {
this.value = this.summation / this.count;
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ // Value in the query stage will ignore decimal places after a decimal point
+ return Double.valueOf(value).longValue() == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/HavingDefaultValue.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/HavingDefaultValue.java
new file mode 100644
index 0000000..d446282
--- /dev/null
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/HavingDefaultValue.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.analysis.metrics;
+
+/**
+ * HavingDefaultValue interface defines a capability for the metric implementations, which has the declaration of the
+ * default value.
+ *
+ * For the declared metrics, the OAP server would skip the persistence of minute dimensionality metrics to reduce
+ * resource costs for the database, when the value of the metric is the default.
+ */
+public interface HavingDefaultValue {
+ /**
+ * @return true if the implementation of this metric has the definition of default value.
+ */
+ default boolean haveDefault() {
+ return false;
+ }
+
+ /**
+ * @return true when the latest value equals the default value.
+ */
+ default boolean isDefaultValue() {
+ return false;
+ }
+}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/LongAvgMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/LongAvgMetrics.java
index 661281c..ad0c12d 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/LongAvgMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/LongAvgMetrics.java
@@ -64,4 +64,14 @@ public abstract class LongAvgMetrics extends Metrics implements LongValueHolder
public final void calculate() {
this.value = this.summation / this.count;
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxDoubleMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxDoubleMetrics.java
index 5dcef23..c0d5336 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxDoubleMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxDoubleMetrics.java
@@ -52,4 +52,15 @@ public abstract class MaxDoubleMetrics extends Metrics implements DoubleValueHol
@Override
public void calculate() {
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ // Value in the query stage will ignore decimal places after a decimal point
+ return Double.valueOf(value).longValue() == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxLongMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxLongMetrics.java
index 9b70a47..0d3043f 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxLongMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxLongMetrics.java
@@ -25,9 +25,6 @@ import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.Metrics
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.SourceFrom;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
-/**
- *
- **/
@MetricsFunction(functionName = "max")
public abstract class MaxLongMetrics extends Metrics implements LongValueHolder {
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Metrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Metrics.java
index 1bbcd06..6763a47 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Metrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Metrics.java
@@ -33,7 +33,7 @@ import org.apache.skywalking.oap.server.core.storage.annotation.Column;
@EqualsAndHashCode(of = {
"timeBucket"
})
-public abstract class Metrics extends StreamData implements StorageData {
+public abstract class Metrics extends StreamData implements StorageData, HavingDefaultValue {
public static final String TIME_BUCKET = "time_bucket";
public static final String ENTITY_ID = "entity_id";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MinDoubleMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MinDoubleMetrics.java
index bdc45e6..3315b50 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MinDoubleMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MinDoubleMetrics.java
@@ -52,4 +52,15 @@ public abstract class MinDoubleMetrics extends Metrics implements DoubleValueHol
@Override
public void calculate() {
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ // Value in the query stage will ignore decimal places after a decimal point
+ return Double.valueOf(value).longValue() == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MinLongMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MinLongMetrics.java
index 88f7fab..bcdd15b 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MinLongMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MinLongMetrics.java
@@ -52,4 +52,14 @@ public abstract class MinLongMetrics extends Metrics implements LongValueHolder
@Override
public void calculate() {
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java
index aa6d32f..bf0e463 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java
@@ -69,4 +69,14 @@ public abstract class PercentMetrics extends Metrics implements IntValueHolder {
public int getValue() {
return percentage;
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return percentage == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PxxMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PxxMetrics.java
index a65ebce..65d773e 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PxxMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PxxMetrics.java
@@ -106,4 +106,14 @@ public abstract class PxxMetrics extends Metrics implements IntValueHolder {
}
}
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/RateMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/RateMetrics.java
index 830f1d7..479727d 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/RateMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/RateMetrics.java
@@ -73,4 +73,14 @@ public abstract class RateMetrics extends Metrics implements IntValueHolder {
public int getValue() {
return percentage;
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return percentage == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/SumMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/SumMetrics.java
index a2f3d61..5ad9962 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/SumMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/SumMetrics.java
@@ -51,4 +51,14 @@ public abstract class SumMetrics extends Metrics implements LongValueHolder {
@Override
public void calculate() {
}
+
+ @Override
+ public boolean haveDefault() {
+ return true;
+ }
+
+ @Override
+ public boolean isDefaultValue() {
+ return value == 0;
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/worker/MetricsPersistentWorker.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/worker/MetricsPersistentWorker.java
index 12c1206..07d14ef 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/worker/MetricsPersistentWorker.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/worker/MetricsPersistentWorker.java
@@ -28,10 +28,6 @@ import java.util.Optional;
import java.util.Properties;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
-import org.apache.skywalking.oap.server.library.datacarrier.DataCarrier;
-import org.apache.skywalking.oap.server.library.datacarrier.consumer.BulkConsumePool;
-import org.apache.skywalking.oap.server.library.datacarrier.consumer.ConsumerPoolFactory;
-import org.apache.skywalking.oap.server.library.datacarrier.consumer.IConsumer;
import org.apache.skywalking.oap.server.core.UnexpectedException;
import org.apache.skywalking.oap.server.core.analysis.data.MergableBufferedData;
import org.apache.skywalking.oap.server.core.analysis.data.ReadWriteSafeCache;
@@ -41,6 +37,10 @@ import org.apache.skywalking.oap.server.core.storage.IMetricsDAO;
import org.apache.skywalking.oap.server.core.storage.model.Model;
import org.apache.skywalking.oap.server.core.worker.AbstractWorker;
import org.apache.skywalking.oap.server.library.client.request.PrepareRequest;
+import org.apache.skywalking.oap.server.library.datacarrier.DataCarrier;
+import org.apache.skywalking.oap.server.library.datacarrier.consumer.BulkConsumePool;
+import org.apache.skywalking.oap.server.library.datacarrier.consumer.ConsumerPoolFactory;
+import org.apache.skywalking.oap.server.library.datacarrier.consumer.IConsumer;
import org.apache.skywalking.oap.server.library.module.ModuleDefineHolder;
import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
import org.apache.skywalking.oap.server.telemetry.api.CounterMetrics;
@@ -68,6 +68,7 @@ public class MetricsPersistentWorker extends PersistenceWorker<Metrics> {
private final boolean supportUpdate;
private long sessionTimeout;
private CounterMetrics aggregationCounter;
+ private CounterMetrics skippedMetricsCounter;
/**
* The counter for the round of persistent.
*/
@@ -82,6 +83,11 @@ public class MetricsPersistentWorker extends PersistenceWorker<Metrics> {
* @since 8.7.0 TTL settings from {@link org.apache.skywalking.oap.server.core.CoreModuleConfig#getMetricsDataTTL()}
*/
private int metricsDataTTL;
+ /**
+ * @since 8.9.0 The persistence of minute dimensionality metrics will be skipped if the value of the metric is the
+ * default.
+ */
+ private boolean skipDefaultValueMetric;
MetricsPersistentWorker(ModuleDefineHolder moduleDefineHolder, Model model, IMetricsDAO metricsDAO,
AbstractWorker<Metrics> nextAlarmWorker, AbstractWorker<ExportEvent> nextExportWorker,
@@ -100,6 +106,7 @@ public class MetricsPersistentWorker extends PersistenceWorker<Metrics> {
this.persistentCounter = 0;
this.persistentMod = 1;
this.metricsDataTTL = metricsDataTTL;
+ this.skipDefaultValueMetric = true;
String name = "METRICS_L2_AGGREGATION";
int size = BulkConsumePool.Creator.recommendMaxSize() / 8;
@@ -124,6 +131,11 @@ public class MetricsPersistentWorker extends PersistenceWorker<Metrics> {
new MetricsTag.Keys("metricName", "level", "dimensionality"),
new MetricsTag.Values(model.getName(), "2", model.getDownsampling().getName())
);
+ skippedMetricsCounter = metricsCreator.createCounter(
+ "metrics_persistence_skipped", "The number of metrics skipped in persistence due to be in default value",
+ new MetricsTag.Keys("metricName", "dimensionality"),
+ new MetricsTag.Values(model.getName(), model.getDownsampling().getName())
+ );
SESSION_TIMEOUT_OFFSITE_COUNTER++;
}
@@ -141,6 +153,12 @@ public class MetricsPersistentWorker extends PersistenceWorker<Metrics> {
null, null, null,
enableDatabaseSession, supportUpdate, storageSessionTimeout, metricsDataTTL
);
+
+ // Skipping default value mechanism only works for minute dimensionality.
+ // Metrics in hour and day dimensionalities would not apply this as duration is too long,
+ // applying this could make precision of hour/day metrics to be lost easily.
+ this.skipDefaultValueMetric = false;
+
// For a down-sampling metrics, we prolong the session timeout for 4 times, nearly 5 minutes.
// And add offset according to worker creation sequence, to avoid context clear overlap,
// eventually optimize load of IDs reading.
@@ -225,12 +243,22 @@ public class MetricsPersistentWorker extends PersistenceWorker<Metrics> {
continue;
}
cachedMetrics.calculate();
- prepareRequests.add(metricsDAO.prepareBatchUpdate(model, cachedMetrics));
+ if (skipDefaultValueMetric && cachedMetrics.haveDefault() && cachedMetrics.isDefaultValue()) {
+ // Skip metrics in default value
+ skippedMetricsCounter.inc();
+ } else {
+ prepareRequests.add(metricsDAO.prepareBatchUpdate(model, cachedMetrics));
+ }
nextWorker(cachedMetrics);
cachedMetrics.setLastUpdateTimestamp(timestamp);
} else {
metrics.calculate();
- prepareRequests.add(metricsDAO.prepareBatchInsert(model, metrics));
+ if (skipDefaultValueMetric && metrics.haveDefault() && metrics.isDefaultValue()) {
+ // Skip metrics in default value
+ skippedMetricsCounter.inc();
+ } else {
+ prepareRequests.add(metricsDAO.prepareBatchInsert(model, metrics));
+ }
nextWorker(metrics);
metrics.setLastUpdateTimestamp(timestamp);
}