You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2022/08/08 08:43:43 UTC
[skywalking] branch es-critical created (now 09cfedc9cb)
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a change to branch es-critical
in repository https://gitbox.apache.org/repos/asf/skywalking.git
at 09cfedc9cb [Critical] Fix a low performance risk of metrics persistent in the ElasticSearch storage implementation.
This branch includes the following new commits:
new 09cfedc9cb [Critical] Fix a low performance risk of metrics persistent in the ElasticSearch storage implementation.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[skywalking] 01/01: [Critical] Fix a low performance risk of metrics persistent in the ElasticSearch storage implementation.
Posted by wu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch es-critical
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 09cfedc9cb03dd633f138c563847b8bc14f6c99f
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Mon Aug 8 16:43:27 2022 +0800
[Critical] Fix a low performance risk of metrics persistent in the ElasticSearch storage implementation.
---
docs/en/changes/changes.md | 24 +++++++++++++++------
.../library/elasticsearch/bulk/BulkProcessor.java | 2 +-
.../so11y_oap/so11y-instance.json | 4 ++--
.../elasticsearch/base/BatchProcessEsDAO.java | 25 ++++++++++++++--------
4 files changed, 36 insertions(+), 19 deletions(-)
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 2e3e5494a7..d74ca4188a 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -2,6 +2,8 @@
#### Project
+* [Critical] Fix a low performance risk of metrics persistent in the ElasticSearch storage implementation. One single
+ metric could have to wait unnecessary 7~10s(System Env Variable `SW_STORAGE_ES_FLUSH_INTERVAL`).
* Upgrade Armeria to 1.16.0, Kubernetes Java client to 15.0.1.
#### OAP Server
@@ -21,18 +23,25 @@
* Remove legacy OAL `percentile` functions, `p99`, `p95`, `p90`, `p75`, `p50` func(s).
* Revert [#8066](https://github.com/apache/skywalking/pull/8066). Keep all metrics persistent even it is default value.
* Skip loading UI templates if folder is empty or doesn't exist.
-* Optimize ElasticSearch query performance by using `_mGet` and physical index name rather than alias in these scenarios, (a) Metrics aggregation (b) Zipkin query (c) Metrics query (d) Log query
+* Optimize ElasticSearch query performance by using `_mGet` and physical index name rather than alias in these
+ scenarios, (a) Metrics aggregation (b) Zipkin query (c) Metrics query (d) Log query
* Support the `NETWORK` type of eBPF Profiling task.
* Support `sumHistogram` in `MAL`.
-* [Breaking Change] Make the eBPF Profiling task support to the service instance level, index/table `ebpf_profiling_task` is required to be re-created when bump up from previous releases.
+* [Breaking Change] Make the eBPF Profiling task support to the service instance level,
+ index/table `ebpf_profiling_task` is required to be re-created when bump up from previous releases.
* Fix race condition in Banyandb storage
* Support `SUM_PER_MIN` downsampling in `MAL`.
* Support `sumHistogramPercentile` in `MAL`.
* Add `VIRTUAL_CACHE` to Layer, to fix conjectured Redis server, which icon can't show on the topology.
-* [Breaking Change] Elasticsearch storage merge all metrics/meter and records(without super datasets) indices into one physical index template `metrics-all` and `records-all` on the default setting.
- Provide system environment variable(`SW_STORAGE_ES_LOGIC_SHARDING`) to shard metrics/records indices into multi-physical indices as the previous versions(one index template per metric/meter aggregation function).
- In the current one index mode, users still could choose to adjust ElasticSearch's shard number(`SW_STORAGE_ES_INDEX_SHARDS_NUMBER`) to scale out.
-* [Breaking Change] Many columns of metrics and records model names are changed, The H2/Mysql/Tidb/Postgres storage users are required to remove all metrics-related and records-related tables for OAP to re-create or use a new database instance.
+* [Breaking Change] Elasticsearch storage merge all metrics/meter and records(without super datasets) indices into one
+ physical index template `metrics-all` and `records-all` on the default setting.
+ Provide system environment variable(`SW_STORAGE_ES_LOGIC_SHARDING`) to shard metrics/records indices into
+ multi-physical indices as the previous versions(one index template per metric/meter aggregation function).
+ In the current one index mode, users still could choose to adjust ElasticSearch's shard
+ number(`SW_STORAGE_ES_INDEX_SHARDS_NUMBER`) to scale out.
+* [Breaking Change] Many columns of metrics and records model names are changed, The H2/Mysql/Tidb/Postgres storage
+ users are required to remove all metrics-related and records-related tables for OAP to re-create or use a new database
+ instance.
* Fix Zipkin trace query the max size of spans.
* Add `tls` and `https` component IDs for Network Profiling.
@@ -54,7 +63,8 @@
* Dashboard: Add metrics association.
* Dashboard: Fix `FaaS-Root` document link and topology service relation dashboard link.
* Dashboard: Fix `Mesh-Instance` metric `Throughput`.
-* Dashboard: Fix `Mesh-Service-Relation` metric `Throughput` and `Proxy Sidecar Internal Latency in Nanoseconds (Client Response)`.
+* Dashboard: Fix `Mesh-Service-Relation` metric `Throughput`
+ and `Proxy Sidecar Internal Latency in Nanoseconds (Client Response)`.
* Dashboard: Fix `Mesh-Instance-Relation` metric `Throughput`.
* Enhance associations for the Event widget.
* Add event widgets in dashboard where applicable.
diff --git a/oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/bulk/BulkProcessor.java b/oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/bulk/BulkProcessor.java
index a5d23e0958..6841cb5fb6 100644
--- a/oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/bulk/BulkProcessor.java
+++ b/oap-server/server-library/library-elasticsearch-client/src/main/java/org/apache/skywalking/library/elasticsearch/bulk/BulkProcessor.java
@@ -101,7 +101,7 @@ public final class BulkProcessor {
}
}
- void flush() {
+ public void flush() {
if (requests.isEmpty()) {
return;
}
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/so11y_oap/so11y-instance.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/so11y_oap/so11y-instance.json
index 685b46be56..5465063283 100644
--- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/so11y_oap/so11y-instance.json
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/so11y_oap/so11y-instance.json
@@ -241,7 +241,7 @@
"i": "8",
"type": "Widget",
"widget": {
- "title": "Persistence Execution Latency (ms)"
+ "title": "Persistence Execution Latency Per Metric Type (ms)"
},
"graph": {
"type": "Line",
@@ -267,7 +267,7 @@
"i": "9",
"type": "Widget",
"widget": {
- "title": "Persistence Preparing Latency (ms)"
+ "title": "Persistence Preparing Latency Per Metric Type (ms)"
},
"graph": {
"type": "Line",
diff --git a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/BatchProcessEsDAO.java b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/BatchProcessEsDAO.java
index 39b6a81fc6..1a4aaa52ac 100644
--- a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/BatchProcessEsDAO.java
+++ b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/BatchProcessEsDAO.java
@@ -52,7 +52,8 @@ public class BatchProcessEsDAO extends EsDAO implements IBatchDAO {
if (bulkProcessor == null) {
synchronized (this) {
if (bulkProcessor == null) {
- this.bulkProcessor = getClient().createBulkProcessor(bulkActions, flushInterval, concurrentRequests);
+ this.bulkProcessor = getClient().createBulkProcessor(
+ bulkActions, flushInterval, concurrentRequests);
}
}
}
@@ -65,19 +66,25 @@ public class BatchProcessEsDAO extends EsDAO implements IBatchDAO {
if (bulkProcessor == null) {
synchronized (this) {
if (bulkProcessor == null) {
- this.bulkProcessor = getClient().createBulkProcessor(bulkActions, flushInterval, concurrentRequests);
+ this.bulkProcessor = getClient().createBulkProcessor(
+ bulkActions, flushInterval, concurrentRequests);
}
}
}
if (CollectionUtils.isNotEmpty(prepareRequests)) {
- return CompletableFuture.allOf(prepareRequests.stream().map(prepareRequest -> {
- if (prepareRequest instanceof InsertRequest) {
- return bulkProcessor.add(((IndexRequestWrapper) prepareRequest).getRequest());
- } else {
- return bulkProcessor.add(((UpdateRequestWrapper) prepareRequest).getRequest());
- }
- }).toArray(CompletableFuture[]::new));
+ try {
+ return CompletableFuture.allOf(prepareRequests.stream().map(prepareRequest -> {
+ if (prepareRequest instanceof InsertRequest) {
+ return bulkProcessor.add(((IndexRequestWrapper) prepareRequest).getRequest());
+ } else {
+ return bulkProcessor.add(((UpdateRequestWrapper) prepareRequest).getRequest());
+ }
+ }).toArray(CompletableFuture[]::new));
+ } finally {
+ // Flush forcedly due to this kind of metrics has been pushed into the bulk processor.
+ bulkProcessor.flush();
+ }
}
return CompletableFuture.completedFuture(null);
}