You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by ha...@apache.org on 2024/11/20 00:49:17 UTC
(skywalking) branch banyandb created (now b3d8fd3220)
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a change to branch banyandb
in repository https://gitbox.apache.org/repos/asf/skywalking.git
at b3d8fd3220 Introduce index mode and refactor banyandb settings
This branch includes the following new commits:
new b3d8fd3220 Introduce index mode and refactor banyandb settings
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(skywalking) 01/01: Introduce index mode and refactor banyandb settings
Posted by ha...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch banyandb
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit b3d8fd32201aba57c3da3b818bb07cf2e5edbfbe
Author: Gao Hongtao <ha...@gmail.com>
AuthorDate: Wed Nov 20 08:48:28 2024 +0800
Introduce index mode and refactor banyandb settings
Signed-off-by: Gao Hongtao <ha...@gmail.com>
---
docs/en/setup/backend/storages/banyandb.md | 212 ++++++++++++++++-----
oap-server-bom/pom.xml | 2 +-
.../manual/cache/TopNCacheReadCommand.java | 1 +
.../manual/cache/TopNCacheWriteCommand.java | 1 +
.../manual/database/TopNDatabaseStatement.java | 1 +
.../analysis/manual/endpoint/EndpointTraffic.java | 1 +
.../analysis/manual/instance/InstanceTraffic.java | 1 +
.../manual/networkalias/NetworkAddressAlias.java | 1 +
.../analysis/manual/process/ProcessTraffic.java | 1 +
.../EndpointRelationServerSideMetrics.java | 1 +
.../ServiceInstanceRelationClientSideMetrics.java | 1 +
.../ServiceInstanceRelationServerSideMetrics.java | 1 +
.../process/ProcessRelationClientSideMetrics.java | 1 +
.../process/ProcessRelationServerSideMetrics.java | 1 +
.../service/ServiceRelationClientSideMetrics.java | 1 +
.../service/ServiceRelationServerSideMetrics.java | 1 +
.../manual/searchtag/TagAutocompleteData.java | 1 +
.../analysis/manual/service/ServiceTraffic.java | 1 +
.../oap/server/core/analysis/metrics/Event.java | 1 +
.../core/analysis/metrics/PercentMetrics.java | 1 +
.../instance/InstanceHierarchyRelationTraffic.java | 1 +
.../service/ServiceHierarchyRelationTraffic.java | 1 +
.../ebpf/storage/EBPFProfilingScheduleRecord.java | 1 +
.../server/core/storage/annotation/BanyanDB.java | 11 ++
.../core/storage/model/BanyanDBModelExtension.java | 9 +
.../server/core/storage/model/StorageModels.java | 4 +
.../core/zipkin/ZipkinServiceRelationTraffic.java | 1 +
.../core/zipkin/ZipkinServiceSpanTraffic.java | 1 +
.../server/core/zipkin/ZipkinServiceTraffic.java | 1 +
.../src/main/resources/banyandb-min.yml | 10 +
.../server-starter/src/main/resources/banyandb.yml | 25 +++
.../storage/plugin/banyandb/MetadataRegistry.java | 6 +
32 files changed, 258 insertions(+), 45 deletions(-)
diff --git a/docs/en/setup/backend/storages/banyandb.md b/docs/en/setup/backend/storages/banyandb.md
index fab1a26b60..3d52598e20 100644
--- a/docs/en/setup/backend/storages/banyandb.md
+++ b/docs/en/setup/backend/storages/banyandb.md
@@ -1,70 +1,194 @@
-
## BanyanDB
-[BanyanDB](https://github.com/apache/skywalking-banyandb) is a dedicated storage implementation developed by the SkyWalking Team and the community.
-Activate BanyanDB as the storage, and set storage provider to **banyandb**.
-The OAP requires BanyanDB 0.7 server. From this version, BanyanDB provides general compatibility.
+[BanyanDB](https://github.com/apache/skywalking-banyandb) is a dedicated storage implementation developed by the SkyWalking Team and the community. Activate BanyanDB as the storage by setting the storage provider to **banyandb**.
+
+The OAP requires BanyanDB version **0.7** or later. From this version onwards, BanyanDB provides general compatibility.
+
+### Configuration
```yaml
storage:
banyandb:
# Targets is the list of BanyanDB servers, separated by commas.
- # Each target is a BanyanDB server in the format of `host:port`
- # If the BanyanDB is deployed as a standalone server, the target should be the IP address or domain name and port of the BanyanDB server.
- # If the BanyanDB is deployed in a cluster, the targets should be the IP address or domain name and port of the `liaison` nodes, separated by commas.
+ # Each target is a BanyanDB server in the format of `host:port`.
+ # If BanyanDB is deployed as a standalone server, the target should be the IP address or domain name and port of the BanyanDB server.
+ # If BanyanDB is deployed in a cluster, the targets should be the IP address or domain name and port of the `liaison` nodes, separated by commas.
targets: ${SW_STORAGE_BANYANDB_TARGETS:127.0.0.1:17912}
- # The max number of records in a bulk write request.
- # Bigger value can improve the write performance, but also increase the OAP and BanyanDB Server memory usage.
+
+ # The maximum number of records in a bulk write request.
+ # A larger value can improve write performance but also increases OAP and BanyanDB Server memory usage.
maxBulkSize: ${SW_STORAGE_BANYANDB_MAX_BULK_SIZE:10000}
+
# The minimum seconds between two bulk flushes.
# If the data in a bulk is less than maxBulkSize, the data will be flushed after this period.
- # If the data in a bulk is more than maxBulkSize, the data will be flushed immediately.
- # Bigger value can reduce the write pressure on BanyanDB Server, but also increase the latency of the data.
+ # If the data in a bulk exceeds maxBulkSize, the data will be flushed immediately.
+ # A larger value can reduce write pressure on BanyanDB Server but increase data latency.
flushInterval: ${SW_STORAGE_BANYANDB_FLUSH_INTERVAL:15}
- # The timeout seconds of a bulk flush.
+
+ # The timeout in seconds for a bulk flush.
flushTimeout: ${SW_STORAGE_BANYANDB_FLUSH_TIMEOUT:10}
- # The shard number of `measure` groups that store the metrics data.
- metricsShardsNumber: ${SW_STORAGE_BANYANDB_METRICS_SHARDS_NUMBER:1}
- # The shard number of `stream` groups that store the trace, log and profile data.
- recordShardsNumber: ${SW_STORAGE_BANYANDB_RECORD_SHARDS_NUMBER:1}
- # The multiplier of the number of shards of the super dataset.
- # Super dataset is a special dataset that stores the trace or log data that is too large to be stored in the normal dataset.
- # If the normal dataset has `n` shards, the super dataset will have `n * superDatasetShardsFactor` shards.
- # For example, supposing `recordShardsNumber` is 3, and `superDatasetShardsFactor` is 2,
- # `segment-default` is a normal dataset that has 3 shards, and `segment-minute` is a super dataset that has 6 shards.
- superDatasetShardsFactor: ${SW_STORAGE_BANYANDB_SUPERDATASET_SHARDS_FACTOR:2}
+
+ # The path to the BanyanDB group settings file.
+ groupSettingsFile: ${SW_STORAGE_BANYANDB_GROUP_SETTINGS_FILE:"banyandb.yml"}
+
# The number of threads that write data to BanyanDB concurrently.
- # Bigger value can improve the write performance, but also increase the OAP and BanyanDB Server CPU usage.
+ # A higher value can improve write performance but also increases CPU usage on both OAP and BanyanDB Server.
concurrentWriteThreads: ${SW_STORAGE_BANYANDB_CONCURRENT_WRITE_THREADS:15}
- # The maximum size of dataset when the OAP loads cache, such as network aliases.
+
+ # The maximum size of the dataset when the OAP loads cache, such as network aliases.
resultWindowMaxSize: ${SW_STORAGE_BANYANDB_QUERY_MAX_WINDOW_SIZE:10000}
+
# The maximum size of metadata per query.
metadataQueryMaxSize: ${SW_STORAGE_BANYANDB_QUERY_MAX_SIZE:10000}
- # The maximum size of trace segments per query.
+
+ # The maximum number of trace segments per query.
segmentQueryMaxSize: ${SW_STORAGE_BANYANDB_QUERY_SEGMENT_SIZE:200}
- # The max number of profile task query in a request.
+
+ # The maximum number of profile task queries in a request.
profileTaskQueryMaxSize: ${SW_STORAGE_BANYANDB_QUERY_PROFILE_TASK_SIZE:200}
- # The batch size of query profiling data.
+
+ # The batch size for querying profile data.
profileDataQueryBatchSize: ${SW_STORAGE_BANYANDB_QUERY_PROFILE_DATA_BATCH_SIZE:100}
- # Data is stored in BanyanDB in segments. A segment is a time range of data.
- # The segment interval is the time range of a segment.
- # The value should be less or equal to data TTL relevant settings.
- segmentIntervalDays: ${SW_STORAGE_BANYANDB_SEGMENT_INTERVAL_DAYS:1}
- # The super dataset segment interval is the time range of a segment in the super dataset.
- superDatasetSegmentIntervalDays: ${SW_STORAGE_BANYANDB_SUPER_DATASET_SEGMENT_INTERVAL_DAYS:1}
- # Specific groups settings.
- # For example, {"group1": {"blockIntervalHours": 4, "segmentIntervalDays": 1}}
- # Please refer to https://github.com/apache/skywalking-banyandb/blob/${BANYANDB_RELEASE}/docs/crud/group.md#create-operation
- # for group setting details.
- specificGroupSettings: ${SW_STORAGE_BANYANDB_SPECIFIC_GROUP_SETTINGS:""}
- # If the BanyanDB server is configured with TLS, config the TLS cert file path and open tls connection.
+
+ # If the BanyanDB server is configured with TLS, configure the TLS cert file path and enable TLS connection.
sslTrustCAPath: ${SW_STORAGE_BANYANDB_SSL_TRUST_CA_PATH:""}
```
-BanyanDB Server supports two installation modes: standalone and cluster. The standalone mode is suitable for small-scale deployments, while the cluster mode is suitable for large-scale deployments.
+### Installation Modes
+
+BanyanDB Server supports two installation modes:
+
+- **Standalone Mode**: Suitable for small-scale deployments.
+ - **Configuration**: `targets` is the IP address/hostname and port of the BanyanDB server.
+
+- **Cluster Mode**: Suitable for large-scale deployments.
+ - **Configuration**: `targets` is the IP address/hostname and port of the `liaison` nodes, separated by commas. `Liaison` nodes are the entry points of the BanyanDB cluster.
+
+### Group Settings
+
+BanyanDB supports **group settings** to configure storage groups, shards, segment intervals, and TTL (Time-To-Live). The group settings file is a YAML file required when using BanyanDB as the storage.
+
+#### Basic Group Settings
+
+- `shardNum`: Number of shards in the group. Shards are the basic units of data storage in BanyanDB. Data is distributed across shards based on the hash value of the series ID. Refer to the [BanyanDB Shard](https://skywalking.apache.org/docs/skywalking-banyandb/latest/concept/clustering/#52-data-sharding) documentation for more details.
+- `segmentIntervalDays`: Interval in days for creating a new segment. Segments are time-based, allowing efficient data retention and querying.
+- `ttlDays`: Time-to-live for the data in the group, in days. Data exceeding the TTL will be deleted.
+
+For more details on setting `segmentIntervalDays` and `ttlDays`, refer to the [BanyanDB Rotation](https://skywalking.apache.org/docs/skywalking-banyandb/latest/concept/rotation/) documentation.
-* Standalone mode: `targets` is the IP address/host name and port of the BanyanDB server.
-* Cluster mode: `targets` is the IP address/host name and port of the `liaison` nodes, separated by commas. `liaison` nodes are the entry points of the BanyanDB cluster.
+#### Example Group Settings File
+
+```yaml
+# "record" defines how and where to store record data.
+record:
+ # "superDataSet" is a special dataset designed to store trace or log data that is too large for normal datasets.
+ # Each super dataset will be a separate group in BanyanDB, following the settings defined in the "superDataSet" section.
+ superDataSet:
+ shardNum: 2 # Number of shards for the super dataset.
+ segmentIntervalDays: 1 # Number of days before creating a new segment.
+ ttlDays: 3 # Time-to-live (TTL) for the data, in days.
+
+ # The "dedicated" section defines settings for datasets that are either normal or super datasets.
+ #
+ # - **Normal Datasets**:
+ # - A new group will be created in BanyanDB, similar to super datasets.
+ # - Valid dataset names are defined by the code. You can find them by checking BanyanDB's schema.
+ # - Once OAP starts, it will automatically create the dataset schema in BanyanDB.
+ # - Normal datasets are grouped into a single group named "default".
+ # - You can [list all streams in the default group](https://skywalking.apache.org/docs/skywalking-banyandb/latest/interacting/bydbctl/schema/stream/#list-operation) to find the dataset names.
+ #
+ # - **Super Datasets**:
+ # - Dataset names are prefixed with "stream_".
+ # - You can [list all groups](https://skywalking.apache.org/docs/skywalking-banyandb/latest/interacting/bydbctl/schema/group/#list-operation) to find the dataset names.
+ dedicated:
+ segment:
+ name: segment # Optional. Name of the group.
+ shardNum: 3 # Number of shards for the "segment" dataset.
+ segmentIntervalDays: 1 # Number of days before creating a new segment.
+ ttlDays: 2 # TTL for the "segment" dataset, in days.
+ log:
+ shardNum: 3 # Number of shards for the "log" dataset.
+ segmentIntervalDays: 1 # Number of days before creating a new segment.
+ ttlDays: 2 # TTL for the "log" dataset, in days.
+
+ # The "default" section defines settings for datasets not specified in "superDataSet" or "dedicated".
+ # Each dataset will be grouped under a single group named "default".
+ default:
+ shardNum: 1 # Number of shards for the default dataset.
+ segmentIntervalDays: 1 # Number of days before creating a new segment.
+ ttlDays: 3 # TTL for the default dataset, in days.
+
+# "metrics" defines how and where to store metrics data.
+metrics:
+ # The "dimension" section organizes datasets based on the metrics' dimensions.
+ # The group name is derived from the dimension value. Valid dimensions are "day", "hour", and "minute".
+ # Non-"minute" dimensions are governed by the "core.downsampling" setting.
+ # For example, if "core.downsampling" is set to "hour", the "hour" dimension will be used, while "day" dimensions are ignored.
+ dimension:
+ day:
+ shardNum: 1 # Number of shards for the "day" dimension.
+ segmentIntervalDays: 1 # Number of days before creating a new segment.
+ ttlDays: 30 # TTL for the "day" dimension, in days.
+ hour:
+ shardNum: 1 # Number of shards for the "hour" dimension.
+ segmentIntervalDays: 1 # Number of days before creating a new segment.
+ ttlDays: 30 # TTL for the "hour" dimension, in days.
+ minute:
+ name: min # Optional. Name of the group.
+ shardNum: 2 # Number of shards for the "minute" dimension.
+ segmentIntervalDays: 1 # Number of days before creating a new segment.
+ ttlDays: 7 # TTL for the "minute" dimension, in days.
+
+ # The "default" section defines settings for metrics that do not fall under any specific dimension.
+ # Examples include service_traffic, ebpf_profiling_schedule_record, event, network_alias, etc.
+ default:
+ shardNum: 1 # Number of shards for the default metrics group.
+ segmentIntervalDays: 8 # Number of days before creating a new segment.
+ ttlDays: 7 # TTL for the default metrics group, in days.
+```
+
+#### Minimum Configuration
+
+The minimum configuration for the group settings file is as follows:
+
+```yaml
+record:
+ default:
+ shardNum: 1
+ segmentIntervalDays: 1
+ ttlDays: 3
+
+metrics:
+ default:
+ shardNum: 1
+ segmentIntervalDays: 8
+ ttlDays: 7
+```
+
+#### Aggregated Groups
+
+You can aggregate several datasets into one group. An aggregated group is a special group that contains data from multiple datasets. Data in the aggregated group is stored in the same shard and segment as the original dataset. This is useful for reducing the number of groups.
+
+To create an aggregated group, join multiple dataset names with the `|` character. For example, `browser_error_log|log` is an aggregated group containing data from the `browser_error_log` and `log` datasets. The `name` field is mandatory and specifies the name of the aggregated group.
+
+**Note:** Only the `record.dedicated` and `metrics.dimension` sections support aggregated groups.
+
+```yaml
+record:
+ dedicated:
+ browser_error_log|log:
+ name: log # Mandatory. Name of the aggregated group.
+ shardNum: 3 # Number of shards for the aggregated group.
+ segmentIntervalDays: 1 # Number of days before creating a new segment.
+ ttlDays: 2 # TTL for the aggregated group, in days.
+
+metrics:
+ dimension:
+ day|hour:
+ name: high_dim # Mandatory. Name of the aggregated group.
+ shardNum: 1 # Number of shards for the aggregated group.
+ segmentIntervalDays: 1 # Number of days before creating a new segment.
+ ttlDays: 30 # TTL for the aggregated group, in days.
+```
-For more details, please refer to the documents of [BanyanDB](https://skywalking.apache.org/docs/skywalking-banyandb/latest/readme/)
-and [BanyanDB Java Client](https://github.com/apache/skywalking-banyandb-java-client) subprojects.
+For more details, refer to the documentation of [BanyanDB](https://skywalking.apache.org/docs/skywalking-banyandb/latest/readme/) and the [BanyanDB Java Client](https://github.com/apache/skywalking-banyandb-java-client) subprojects.
diff --git a/oap-server-bom/pom.xml b/oap-server-bom/pom.xml
index 0a6a8e1ce1..2e4646bcb9 100644
--- a/oap-server-bom/pom.xml
+++ b/oap-server-bom/pom.xml
@@ -73,7 +73,7 @@
<httpcore.version>4.4.13</httpcore.version>
<httpasyncclient.version>4.1.5</httpasyncclient.version>
<commons-compress.version>1.21</commons-compress.version>
- <banyandb-java-client.version>0.7.0</banyandb-java-client.version>
+ <banyandb-java-client.version>0.8.0-SNAPSHOT</banyandb-java-client.version>
<kafka-clients.version>3.4.0</kafka-clients.version>
<spring-kafka-test.version>2.4.6.RELEASE</spring-kafka-test.version>
<consul.client.version>1.5.3</consul.client.version>
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/cache/TopNCacheReadCommand.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/cache/TopNCacheReadCommand.java
index d541b0dc7f..e466961cf4 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/cache/TopNCacheReadCommand.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/cache/TopNCacheReadCommand.java
@@ -38,6 +38,7 @@ import java.util.Objects;
*/
@Stream(name = TopNCacheReadCommand.INDEX_NAME, scopeId = DefaultScopeDefine.CACHE_SLOW_ACCESS, builder = TopNCacheReadCommand.Builder.class, processor = TopNStreamProcessor.class)
@BanyanDB.TimestampColumn(TopN.TIMESTAMP)
+@BanyanDB.IndexMode
public class TopNCacheReadCommand extends TopN {
public static final String INDEX_NAME = "top_n_cache_read_command";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/cache/TopNCacheWriteCommand.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/cache/TopNCacheWriteCommand.java
index f7a12cf8f9..e08e70b15e 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/cache/TopNCacheWriteCommand.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/cache/TopNCacheWriteCommand.java
@@ -38,6 +38,7 @@ import java.util.Objects;
*/
@Stream(name = TopNCacheWriteCommand.INDEX_NAME, scopeId = DefaultScopeDefine.CACHE_SLOW_ACCESS, builder = TopNCacheWriteCommand.Builder.class, processor = TopNStreamProcessor.class)
@BanyanDB.TimestampColumn(TopN.TIMESTAMP)
+@BanyanDB.IndexMode
public class TopNCacheWriteCommand extends TopN {
public static final String INDEX_NAME = "top_n_cache_write_command";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/database/TopNDatabaseStatement.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/database/TopNDatabaseStatement.java
index 5a3e71bbcd..3762e9f9c8 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/database/TopNDatabaseStatement.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/database/TopNDatabaseStatement.java
@@ -38,6 +38,7 @@ import java.util.Objects;
*/
@Stream(name = TopNDatabaseStatement.INDEX_NAME, scopeId = DefaultScopeDefine.DATABASE_SLOW_STATEMENT, builder = TopNDatabaseStatement.Builder.class, processor = TopNStreamProcessor.class)
@BanyanDB.TimestampColumn(TopN.TIMESTAMP)
+@BanyanDB.IndexMode
public class TopNDatabaseStatement extends TopN {
public static final String INDEX_NAME = "top_n_database_statement";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/endpoint/EndpointTraffic.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/endpoint/EndpointTraffic.java
index f44a1af6eb..27b00390c7 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/endpoint/EndpointTraffic.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/endpoint/EndpointTraffic.java
@@ -45,6 +45,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
"serviceId",
"name"
})
+@BanyanDB.IndexMode
public class EndpointTraffic extends Metrics {
public static final String INDEX_NAME = "endpoint_traffic";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java
index 24f3bb0d62..12815c0eb8 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java
@@ -48,6 +48,7 @@ import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.SE
"serviceId",
"name"
})
+@BanyanDB.IndexMode
public class InstanceTraffic extends Metrics {
public static final String INDEX_NAME = "instance_traffic";
public static final String SERVICE_ID = "service_id";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/networkalias/NetworkAddressAlias.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/networkalias/NetworkAddressAlias.java
index 4f0d3e4c68..8e8c489419 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/networkalias/NetworkAddressAlias.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/networkalias/NetworkAddressAlias.java
@@ -44,6 +44,7 @@ import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.NE
@EqualsAndHashCode(of = {
"address"
})
+@BanyanDB.IndexMode
public class NetworkAddressAlias extends Metrics {
public static final String INDEX_NAME = "network_address_alias";
public static final String ADDRESS = "address";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/process/ProcessTraffic.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/process/ProcessTraffic.java
index dcd74504f8..5a1d8589e4 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/process/ProcessTraffic.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/process/ProcessTraffic.java
@@ -51,6 +51,7 @@ import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.PR
"name",
})
@BanyanDB.StoreIDAsTag
+@BanyanDB.IndexMode
public class ProcessTraffic extends Metrics {
public static final String INDEX_NAME = "process_traffic";
public static final String SERVICE_ID = "service_id";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/endpoint/EndpointRelationServerSideMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/endpoint/EndpointRelationServerSideMetrics.java
index f300971527..c0740d42cd 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/endpoint/EndpointRelationServerSideMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/endpoint/EndpointRelationServerSideMetrics.java
@@ -40,6 +40,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
+@BanyanDB.IndexMode
public class EndpointRelationServerSideMetrics extends Metrics {
public static final String INDEX_NAME = "endpoint_relation_server_side";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationClientSideMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationClientSideMetrics.java
index 36585ef5f4..011a81e2c9 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationClientSideMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationClientSideMetrics.java
@@ -40,6 +40,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
+@BanyanDB.IndexMode
public class ServiceInstanceRelationClientSideMetrics extends Metrics {
public static final String INDEX_NAME = "service_instance_relation_client_side";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationServerSideMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationServerSideMetrics.java
index 147800be0a..c042fd6479 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationServerSideMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationServerSideMetrics.java
@@ -40,6 +40,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
+@BanyanDB.IndexMode
public class ServiceInstanceRelationServerSideMetrics extends Metrics {
public static final String INDEX_NAME = "service_instance_relation_server_side";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationClientSideMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationClientSideMetrics.java
index 1d15e65e68..283bffb4f8 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationClientSideMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationClientSideMetrics.java
@@ -42,6 +42,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
"entityId",
"component_id"
}, callSuper = true)
+@BanyanDB.IndexMode
public class ProcessRelationClientSideMetrics extends Metrics {
public static final String INDEX_NAME = "process_relation_client_side";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationServerSideMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationServerSideMetrics.java
index f544b0a030..95cba6e861 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationServerSideMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationServerSideMetrics.java
@@ -41,6 +41,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
+@BanyanDB.IndexMode
public class ProcessRelationServerSideMetrics extends Metrics {
public static final String INDEX_NAME = "process_relation_server_side";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationClientSideMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationClientSideMetrics.java
index 9911fd1fff..90dea51ff6 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationClientSideMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationClientSideMetrics.java
@@ -40,6 +40,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
+@BanyanDB.IndexMode
public class ServiceRelationClientSideMetrics extends Metrics {
public static final String INDEX_NAME = "service_relation_client_side";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationServerSideMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationServerSideMetrics.java
index a252d326d2..1ff930f6f3 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationServerSideMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationServerSideMetrics.java
@@ -42,6 +42,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"entityId"
}, callSuper = true)
+@BanyanDB.IndexMode
public class ServiceRelationServerSideMetrics extends Metrics {
public static final String INDEX_NAME = "service_relation_server_side";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/TagAutocompleteData.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/TagAutocompleteData.java
index 469b6eaac7..b31aac9ea9 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/TagAutocompleteData.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/searchtag/TagAutocompleteData.java
@@ -45,6 +45,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
"tagValue",
"tagType"
})
+@BanyanDB.IndexMode
public class TagAutocompleteData extends Metrics {
public static final String INDEX_NAME = "tag_autocomplete";
public static final String TAG_KEY = "tag_key";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java
index d1404b8809..65f60fe6fb 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java
@@ -48,6 +48,7 @@ import static org.apache.skywalking.oap.server.core.Const.DOUBLE_COLONS_SPLIT;
"name",
"layer"
})
+@BanyanDB.IndexMode
public class ServiceTraffic extends Metrics {
public static final String INDEX_NAME = "service_traffic";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Event.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Event.java
index c3598698ee..46445efab8 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Event.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/Event.java
@@ -48,6 +48,7 @@ import static org.apache.skywalking.oap.server.library.util.StringUtil.isNotBlan
callSuper = false,
of = "uuid"
)
+@BanyanDB.IndexMode
public class Event extends Metrics {
public static final String INDEX_NAME = "events";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java
index 94a5590142..8c0db02271 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java
@@ -45,6 +45,7 @@ public abstract class PercentMetrics extends Metrics implements IntValueHolder {
@Getter
@Setter
@Column(name = MATCH, storageOnly = true)
+ @BanyanDB.MeasureField
private long match;
@Entrance
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/instance/InstanceHierarchyRelationTraffic.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/instance/InstanceHierarchyRelationTraffic.java
index f8a3dfb005..a2b6743d1b 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/instance/InstanceHierarchyRelationTraffic.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/instance/InstanceHierarchyRelationTraffic.java
@@ -40,6 +40,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
builder = InstanceHierarchyRelationTraffic.Builder.class, processor = MetricsStreamProcessor.class)
@MetricsExtension(supportDownSampling = false, supportUpdate = false)
@EqualsAndHashCode(callSuper = false)
+@BanyanDB.IndexMode
public class InstanceHierarchyRelationTraffic extends Metrics {
public static final String INDEX_NAME = "instance_hierarchy_relation";
public static final String SERVICE_LAYER = "service_layer";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/service/ServiceHierarchyRelationTraffic.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/service/ServiceHierarchyRelationTraffic.java
index 057cf3c7fe..972b60ef2d 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/service/ServiceHierarchyRelationTraffic.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/service/ServiceHierarchyRelationTraffic.java
@@ -40,6 +40,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
builder = ServiceHierarchyRelationTraffic.Builder.class, processor = MetricsStreamProcessor.class)
@MetricsExtension(supportDownSampling = false, supportUpdate = false)
@EqualsAndHashCode(callSuper = false)
+@BanyanDB.IndexMode
public class ServiceHierarchyRelationTraffic extends Metrics {
public static final String INDEX_NAME = "service_hierarchy_relation";
public static final String SERVICE_ID = "service_id";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/ebpf/storage/EBPFProfilingScheduleRecord.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/ebpf/storage/EBPFProfilingScheduleRecord.java
index d6349e928e..bb81da34f6 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/ebpf/storage/EBPFProfilingScheduleRecord.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/profiling/ebpf/storage/EBPFProfilingScheduleRecord.java
@@ -50,6 +50,7 @@ import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.EB
"processId",
"startTime",
})
+@BanyanDB.IndexMode
public class EBPFProfilingScheduleRecord extends Metrics {
public static final String INDEX_NAME = "ebpf_profiling_schedule";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java
index a4c0b2e4e3..3e95ad11ef 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java
@@ -206,4 +206,15 @@ public @interface BanyanDB {
URL
}
}
+
+ /**
+ * IndexMode is used to indicate the index mode of the metric.
+ * All columns in the metric will be stored in the index exclusively.
+ *
+ * @since 10.2.0
+ */
+ @Target({ElementType.TYPE})
+ @Retention(RetentionPolicy.RUNTIME)
+ @interface IndexMode {
+ }
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/BanyanDBModelExtension.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/BanyanDBModelExtension.java
index ae9c7d657c..5e2601cb11 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/BanyanDBModelExtension.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/BanyanDBModelExtension.java
@@ -52,6 +52,15 @@ public class BanyanDBModelExtension {
@Setter
private TopN topN;
+ /**
+ * indexMode indicates whether a metric is in the index mode.
+ *
+ * @since 10.2.0
+ */
+ @Getter
+ @Setter
+ private boolean indexMode;
+
public static class TopN {
/**
* lru_size defines how many time_buckets are held in the memory.
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java
index eabcfdc350..6f2d78b4d3 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java
@@ -99,6 +99,10 @@ public class StorageModels implements IModelManager, ModelCreator, ModelManipula
banyanDBModelExtension.setStoreIDTag(true);
}
+ if (aClass.isAnnotationPresent(BanyanDB.IndexMode.class)) {
+ banyanDBModelExtension.setIndexMode(true);
+ }
+
// Set routing rules for ElasticSearch
elasticSearchModelExtension.setRouting(storage.getModelName(), modelColumns);
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceRelationTraffic.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceRelationTraffic.java
index 30aeb407d2..b91722d9f2 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceRelationTraffic.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceRelationTraffic.java
@@ -41,6 +41,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
"serviceName",
"remoteServiceName"
})
+@BanyanDB.IndexMode
public class ZipkinServiceRelationTraffic extends Metrics {
public static final String INDEX_NAME = "zipkin_service_relation_traffic";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceSpanTraffic.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceSpanTraffic.java
index 2681815a1e..f814b0ff72 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceSpanTraffic.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceSpanTraffic.java
@@ -42,6 +42,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
"serviceName",
"spanName"
})
+@BanyanDB.IndexMode
public class ZipkinServiceSpanTraffic extends Metrics {
public static final String INDEX_NAME = "zipkin_service_span_traffic";
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceTraffic.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceTraffic.java
index 338cb72fd7..847c68f853 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceTraffic.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinServiceTraffic.java
@@ -41,6 +41,7 @@ import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
@EqualsAndHashCode(of = {
"serviceName"
})
+@BanyanDB.IndexMode
public class ZipkinServiceTraffic extends Metrics {
public static final String INDEX_NAME = "zipkin_service_traffic";
diff --git a/oap-server/server-starter/src/main/resources/banyandb-min.yml b/oap-server/server-starter/src/main/resources/banyandb-min.yml
new file mode 100644
index 0000000000..c07994fbc9
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/banyandb-min.yml
@@ -0,0 +1,10 @@
+record:
+ default:
+ shardNum: 1
+ segmentIntervalDays: 1
+ ttlDays: 3
+metrics:
+ default:
+ shardNum: 1
+ segmentIntervalDays: 8
+ ttlDays: 7
diff --git a/oap-server/server-starter/src/main/resources/banyandb.yml b/oap-server/server-starter/src/main/resources/banyandb.yml
new file mode 100644
index 0000000000..7cfc9122aa
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/banyandb.yml
@@ -0,0 +1,25 @@
+record:
+ default:
+ shardNum: 1
+ segmentIntervalDays: 1
+ ttlDays: 3
+ superDataSet:
+ shardNum: 2
+ segmentIntervalDays: 1
+ ttlDays: 3
+metrics:
+ default:
+ shardNum: 1
+ segmentIntervalDays: 8
+ ttlDays: 7
+ dimension:
+ hour|day:
+ name: high_dem
+ shardNum: 1
+ segmentIntervalDays: 1
+ ttlDays: 30
+ minute:
+ name: minute
+ shardNum: 2
+ segmentIntervalDays: 1
+ ttlDays: 7
diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java
index 77e8b1263a..e30b0bcb74 100644
--- a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java
+++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java
@@ -177,6 +177,12 @@ public enum MetadataRegistry {
builder.setInterval(downSamplingDuration(model.getDownsampling()).format());
builder.setEntity(BanyandbDatabase.Entity.newBuilder().addAllTagNames(shardingColumns));
builder.addAllTagFamilies(tagFamilySpecs);
+ if (model.getBanyanDBModelExtension().isIndexMode()) {
+ builder.setIndexMode(true);
+ if (!tagsAndFields.fields.isEmpty()) {
+ throw new StorageException("index mode is enabled, but fields are defined");
+ }
+ }
// parse and set field
for (BanyandbDatabase.FieldSpec field : tagsAndFields.fields) {
builder.addFields(field);