This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push:
new a5b244817f Add `ai-pipeline` module and support HTTP URI formatting (#10884)
a5b244817f is described below
commit a5b244817f8410e1e1e2198d0c8f02d9cacf151a
Author: 吴晟 Wu Sheng <wu...@foxmail.com>
AuthorDate: Sat Jun 10 12:22:00 2023 +0800
Add `ai-pipeline` module and support HTTP URI formatting (#10884)
* Support HTTP URI formatting through ai-pipeline to do pattern recognition.
* Add new HTTP URI grouping engine with benchmark.
* [Breaking Change] Use the new HTTP URI grouping engine to replace the old regex based mechanism.
---
docs/en/changes/changes.md | 4 +
docs/en/setup/ai-pipeline/README.md | 22 +
.../setup/ai-pipeline/http-restful-uri-pattern.md | 39 ++
docs/en/setup/backend/configuration-vocabulary.md | 632 +++++++++++----------
docs/en/setup/backend/endpoint-grouping-rules.md | 7 +-
docs/menu.yml | 6 +
oap-server/ai-pipeline/pom.xml | 100 ++++
.../oap/server/ai/pipeline/AIPipelineConfig.java | 30 +
.../oap/server/ai/pipeline/AIPipelineModule.java | 37 ++
.../oap/server/ai/pipeline/AIPipelineProvider.java | 80 +++
.../services/HttpUriRecognitionService.java | 105 ++++
.../ai/pipeline/services/api/HttpUriPattern.java | 33 ++
.../pipeline/services/api/HttpUriRecognition.java | 57 ++
.../src/main/proto/ai_http_uri_recognition.proto | 63 ++
...walking.oap.server.library.module.ModuleDefine} | 18 +-
...lking.oap.server.library.module.ModuleProvider} | 18 +-
.../group/uri/RegexVSQuickMatchBenchmark.java | 166 ++++++
oap-server/pom.xml | 1 +
oap-server/server-core/pom.xml | 5 +
.../oap/server/core/CoreModuleConfig.java | 5 +
.../oap/server/core/CoreModuleProvider.java | 17 +-
.../oap/server/core/config/NamingControl.java | 4 +-
.../core/config/group/EndpointGroupingRule.java | 5 +-
.../config/group/EndpointGroupingRuleReader.java | 15 +-
.../core/config/group/EndpointNameGrouping.java | 153 ++++-
.../group/EndpointNameGroupingRuleWatcher.java | 3 +-
.../config/group/uri/quickmatch/PatternToken.java | 47 ++
.../config/group/uri/quickmatch/PatternTree.java | 105 ++++
.../quickmatch/QuickUriGroupingRule.java} | 33 +-
.../config/group/uri/quickmatch/StringToken.java | 59 ++
.../core/config/group/uri/quickmatch/VarToken.java | 57 ++
.../group/EndpointGroupingRuleReaderTest.java | 5 +-
.../group/EndpointNameGroupingRuleWatcherTest.java | 33 +-
.../EndpointGroupingRuleReader4OpenapiTest.java | 36 +-
...ndpointNameGroupingRule4OpenapiWatcherTest.java | 6 +-
.../group/uri/quickmatch/PatternTreeTest.java | 102 ++++
.../src/test/resources/endpoint-name-grouping.yml | 4 +-
.../oap/server/library/util/StringFormatGroup.java | 7 +
.../src/main/resources/application.yml | 8 +
.../src/main/resources/endpoint-name-grouping.yml | 6 +-
40 files changed, 1682 insertions(+), 451 deletions(-)
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 3d9550d5e4..c08048c920 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -53,6 +53,10 @@
* Deprecated record query protocol.
* Add component ID for go-redis.
* Add OpenSearch 2.8.0 to test case.
+* Add `ai-pipeline` module.
+* Support HTTP URI formatting through `ai-pipeline` to do pattern recognition.
+* Add new HTTP URI grouping engine with benchmark.
+* [Breaking Change] Use the new HTTP URI grouping engine to replace the old regex based mechanism.
#### UI
* Revert: cpm5d function. This feature is cancelled from backend.
diff --git a/docs/en/setup/ai-pipeline/README.md b/docs/en/setup/ai-pipeline/README.md
new file mode 100644
index 0000000000..0726cb9e62
--- /dev/null
+++ b/docs/en/setup/ai-pipeline/README.md
@@ -0,0 +1,22 @@
+# AI Pipeline
+
+**Warning, this module is still in the ALPHA stage. This is not stable.**
+
+Pattern Recognition, Machine Learning(ML) and Artificial Intelligence(AI) are common technology to identify patterns in data.
+This module provides a way to integrate these technologies in a standardized way about shipping the data from OAP kernel
+to 3rd party.
+
+From the industry practice, Pattern Recognition, Machine Learning(ML) and Artificial Intelligence(AI) are always overestimated,
+they are good at many things but have to run in a clear context.
+
+## How to enable
+
+ai-pipeline module is activated by default but only running until you set up `uriRecognitionServerAddr` correctly.
+
+```yaml
+ai-pipeline:
+ selector: ${SW_AI_PIPELINE:-}
+ default:
+ uriRecognitionServerAddr: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_ADDR:}
+ uriRecognitionServerPort: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_PORT:17128}
+```
\ No newline at end of file
diff --git a/docs/en/setup/ai-pipeline/http-restful-uri-pattern.md b/docs/en/setup/ai-pipeline/http-restful-uri-pattern.md
new file mode 100644
index 0000000000..2afd8bba22
--- /dev/null
+++ b/docs/en/setup/ai-pipeline/http-restful-uri-pattern.md
@@ -0,0 +1,39 @@
+# HTTP Restful URI recognition
+
+As introduced in the [Group Parameterized Endpoints](../backend/endpoint-grouping-rules.md) doc, HTTP Restful URIs are identified
+as endpoints. With some additional rules, we can identify the parameters in the URI and group the endpoints in case of annoying
+and huge size of endpoint candidates with low value of the metrics.
+
+In the ML/AI specific fields, decision trees or neural networks can be trained on labeled URI data to automatically
+recognize and classify different URI patterns, as well as many other ways.
+
+In this pipeline, OAP has the capabilities to cache the URI candidates with occurrence count,
+and push the data to 3rd party for further analysis. Then OAP would pull the analyzed results for
+processing the further telemetry traffic.
+
+## Set up OAP to connect remote URI recognition server
+`uriRecognitionServerAddr` and `uriRecognitionServerPort` are the configurations to set up the remote URI recognition server.
+
+The URI recognition server is a gRPC server, which is defined in [URIRecognition.proto](../../../../oap-server/ai-pipeline/src/main/proto/ai_http_uri_recognition.proto).
+
+```protobuf
+service HttpUriRecognitionService {
+ // Sync for the pattern recognition dictionary.
+ rpc fetchAllPatterns(HttpUriRecognitionSyncRequest) returns (HttpUriRecognitionResponse) {}
+ // Feed new raw data and matched patterns to the AI-server.
+ rpc feedRawData(HttpUriRecognitionRequest) returns (google.protobuf.Empty) {}
+}
+```
+
+- fetchAllPatterns service
+
+fetchAllPatterns is up and running in 1 minute period from every OAP to fetch all recognized patterns from the remote server.
+
+- feedRawData service
+
+feedRawData is running in 25-30 minutes period to push the raw data to the remote server for training.
+
+## Configurations
+
+- `core/maxHttpUrisNumberPerService` The max number of HTTP URIs per service for further URI pattern recognition.
+- No configuration to set periods of feedRawData and fetchAllPatterns services.
\ No newline at end of file
diff --git a/docs/en/setup/backend/configuration-vocabulary.md b/docs/en/setup/backend/configuration-vocabulary.md
index 35beba6b1c..5b56494b8e 100644
--- a/docs/en/setup/backend/configuration-vocabulary.md
+++ b/docs/en/setup/backend/configuration-vocabulary.md
@@ -2,321 +2,325 @@
The Configuration Vocabulary lists all available configurations provided by `application.yml`.
-| Module | Provider | Settings | Value(s) and Explanation [...]
-|-------------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| core | default | role | Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode OAP opens the service to the agents, then analyzes and aggregates the results, and forwards the results for distributed aggregation. Aggregator mode OAP receives data from Mixer and Receiver role OAP nodes, and perf [...]
-| - | - | restHost | Binding IP of RESTful services. Services include GraphQL query and HTTP data report. [...]
-| - | - | restPort | Binding port of RESTful services. [...]
-| - | - | restContextPath | Web context path of RESTful services. [...]
-| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
-| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
-| - | - | restAcceptQueueSize | ServerSocketChannel Backlog of RESTful services. [...]
-| - | - | httpMaxRequestHeaderSize | Maximum request header size accepted. [...]
-| - | - | gRPCHost | Binding IP of gRPC services, including gRPC data report and internal communication among OAP nodes. [...]
-| - | - | gRPCPort | Binding port of gRPC services. [...]
-| - | - | gRPCSslEnabled | Activates SSL for gRPC services. [...]
-| - | - | gRPCSslKeyPath | File path of gRPC SSL key. [...]
-| - | - | gRPCSslCertChainPath | File path of gRPC SSL cert chain. [...]
-| - | - | gRPCSslTrustedCAPath | File path of gRPC trusted CA. [...]
-| - | - | downsampling | Activated level of down sampling aggregation. [...]
-| - | - | enableDataKeeperExecutor | Controller of TTL scheduler. Once disabled, TTL wouldn't work. [...]
-| - | - | dataKeeperExecutePeriod | Execution period of TTL scheduler (in minutes). Execution doesn't mean deleting data. The storage provider (e.g. ElasticSearch storage) could override this. [...]
-| - | - | recordDataTTL | The lifecycle of record data (in days). Record data includes traces, top N sample records, and logs. Minimum value is 2. [...]
-| - | - | metricsDataTTL | The lifecycle of metrics data (in days), including metadata. We recommend setting metricsDataTTL >= recordDataTTL. Minimum value is 2. [...]
-| - | - | l1FlushPeriod | The period of L1 aggregation flush to L2 aggregation (in milliseconds). [...]
-| - | - | storageSessionTimeout | The threshold of session time (in milliseconds). Default value is 70000. [...]
-| - | - | persistentPeriod | The period of doing data persistence. Unit is second.Default value is 25s [...]
-| - | - | topNReportPeriod | The execution period (in minutes) of top N sampler, which saves sampled data into the storage. [...]
-| - | - | activeExtraModelColumns | Appends entity names (e.g. service names) into metrics storage entities. [...]
-| - | - | serviceNameMaxLength | Maximum length limit of service names. [...]
-| - | - | instanceNameMaxLength | Maximum length limit of service instance names. The maximum length of service + instance names should be less than 200. [...]
-| - | - | endpointNameMaxLength | Maximum length limit of endpoint names. The maximum length of service + endpoint names should be less than 240. [...]
-| - | - | searchableTracesTags | Defines a set of span tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
-| - | - | searchableLogsTags | Defines a set of log tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
-| - | - | searchableAlarmTags | Defines a set of alarm tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
-| - | - | autocompleteTagKeysQueryMaxSize | The max size of tags keys for autocomplete select. [...]
-| - | - | autocompleteTagValuesQueryMaxSize | The max size of tags values for autocomplete select. [...]
-| - | - | gRPCThreadPoolSize | Pool size of gRPC server. [...]
-| - | - | gRPCThreadPoolQueueSize | Queue size of gRPC server. [...]
-| - | - | maxConcurrentCallsPerConnection | The maximum number of concurrent calls permitted for each incoming connection. Defaults to no limit. [...]
-| - | - | maxMessageSize | Sets the maximum message size allowed to be received on the server. Empty means 4 MiB. [...]
-| - | - | remoteTimeout | Timeout for cluster internal communication (in seconds). [...]
-| - | - | maxSizeOfNetworkAddressAlias | The maximum size of network address detected in the system being monitored. [...]
-| - | - | maxPageSizeOfQueryProfileSnapshot | The maximum size for snapshot analysis in an OAP query. [...]
-| - | - | maxSizeOfAnalyzeProfileSnapshot | The maximum number of snapshots analyzed by the OAP. [...]
-| - | - | prepareThreads | The number of threads used to prepare metrics data to the storage. [...]
-| - | - | enableEndpointNameGroupingByOpenapi | Automatically groups endpoints by the given OpenAPI definitions. [...]
-| - | - | maxDurationOfQueryEBPFProfilingData | The maximum duration(in second) of query the eBPF profiling data from database. [...]
-| - | - | maxThreadCountOfQueryEBPFProfilingData | The maximum thread count of query the eBPF profiling data from database. [...]
-| cluster | standalone | - | Standalone is not suitable for running on a single node running. No configuration available. [...]
-| - | zookeeper | namespace | The namespace, represented by root path, isolates the configurations in Zookeeper. [...]
-| - | - | hostPort | Hosts and ports of Zookeeper Cluster. [...]
-| - | - | baseSleepTimeMs | The period of Zookeeper client between two retries (in milliseconds). [...]
-| - | - | maxRetries | The maximum retry time. [...]
-| - | - | enableACL | Opens ACL using `schema` and `expression`. [...]
-| - | - | schema | Schema for the authorization. [...]
-| - | - | expression | Expression for the authorization. [...]
-| - | - | internalComHost | The hostname registered in Zookeeper for the internal communication of OAP cluster. [...]
-| - | - | internalComPort | The port registered in Zookeeper for the internal communication of OAP cluster. [...]
-| - | kubernetes | namespace | Namespace deployed by SkyWalking in k8s. [...]
-| - | - | labelSelector | Labels used for filtering OAP deployment in k8s. [...]
-| - | - | uidEnvName | Environment variable name for reading uid. [...]
-| - | consul | serviceName | Service name for SkyWalking cluster. [...]
-| - | - | hostPort | Hosts and ports for Consul cluster. [...]
-| - | - | aclToken | ACL Token of Consul. Empty string means `without ALC token`. [...]
-| - | - | internalComHost | The hostname registered in Consul for internal communications of the OAP cluster. [...]
-| - | - | internalComPort | The port registered in Consul for internal communications of the OAP cluster. [...]
-| - | etcd | serviceName | Service name for SkyWalking cluster. [...]
-| - | - | endpoints | Hosts and ports for etcd cluster. [...]
-| - | - | namespace | Namespace for SkyWalking cluster. [...]
-| - | - | authentication | Indicates whether there is authentication. [...]
-| - | - | user | Etcd auth username. [...]
-| - | - | password | Etcd auth password. [...]
-| - | - | internalComHost | The hostname registered in etcd for internal communications of the OAP cluster. [...]
-| - | - | internalComPort | The port registered in etcd for internal communications of the OAP cluster. [...]
-| - | Nacos | serviceName | Service name for SkyWalking cluster. [...]
-| - | - | hostPort | Hosts and ports for Nacos cluster. [...]
-| - | - | namespace | Namespace used by SkyWalking node coordination. [...]
-| - | - | internalComHost | The hostname registered in Nacos for internal communications of the OAP cluster. [...]
-| - | - | internalComPort | The port registered in Nacos for internal communications of the OAP cluster. [...]
-| - | - | username | Nacos Auth username. [...]
-| - | - | password | Nacos Auth password. [...]
-| - | - | accessKey | Nacos Auth accessKey. [...]
-| - | - | secretKey | Nacos Auth secretKey. [...]
-| storage | elasticsearch | - | ElasticSearch (and OpenSearch) storage implementation. [...]
-| - | - | namespace | Prefix of indexes created and used by SkyWalking. [...]
-| - | - | clusterNodes | ElasticSearch cluster nodes for client connection. [...]
-| - | - | protocol | HTTP or HTTPs. [...]
-| - | - | connectTimeout | Connect timeout of ElasticSearch client (in milliseconds). [...]
-| - | - | socketTimeout | Socket timeout of ElasticSearch client (in milliseconds). [...]
-| - | - | responseTimeout | Response timeout of ElasticSearch client (in milliseconds), `0` disables the timeout. [...]
-| - | - | numHttpClientThread | The number of threads for the underlying HTTP client to perform socket I/O. If the value is <= 0, the number of available processors will be used. [...]
-| - | - | user | Username of ElasticSearch cluster. [...]
-| - | - | password | Password of ElasticSearch cluster. [...]
-| - | - | trustStorePath | Trust JKS file path. Only works when username and password are enabled. [...]
-| - | - | trustStorePass | Trust JKS file password. Only works when username and password are enabled. [...]
-| - | - | secretsManagementFile | Secrets management file in the properties format, including username and password, which are managed by a 3rd party tool. Capable of being updated them at runtime. [...]
-| - | - | dayStep | Represents the number of days in the one-minute/hour/day index. [...]
-| - | - | indexShardsNumber | Shard number of new indexes. [...]
-| - | - | indexReplicasNumber | Replicas number of new indexes. [...]
-| - | - | specificIndexSettings | Specify the settings for each index individually. If configured, this setting has the highest priority and overrides the generic settings. [...]
-| - | - | superDatasetDayStep | Represents the number of days in the super size dataset record index. Default value is the same as dayStep when the value is less than 0. [...]
-| - | - | superDatasetIndexShardsFactor | Super dataset is defined in the code (e.g. trace segments). This factor provides more shards for the super dataset: shards number = indexShardsNumber * superDatasetIndexShardsFactor. This factor also affects Zipkin and Jaeger traces. [...]
-| - | - | superDatasetIndexReplicasNumber | Represents the replicas number in the super size dataset record index. [...]
-| - | - | indexTemplateOrder | The order of index template. [...]
-| - | - | bulkActions | Async bulk size of the record data batch execution. [...]
-| - | - | batchOfBytes | A threshold to control the max body size of ElasticSearch Bulk flush. [...]
-| - | - | flushInterval | Period of flush (in seconds). Does not matter whether `bulkActions` is reached or not. [...]
-| - | - | concurrentRequests | The number of concurrent requests allowed to be executed. [...]
-| - | - | resultWindowMaxSize | The maximum size of dataset when the OAP loads cache, such as network aliases. [...]
-| - | - | metadataQueryMaxSize | The maximum size of metadata per query. [...]
-| - | - | scrollingBatchSize | The batch size of metadata per iteration when `metadataQueryMaxSize` or `resultWindowMaxSize` is too large to be retrieved in a single query. [...]
-| - | - | segmentQueryMaxSize | The maximum size of trace segments per query. [...]
-| - | - | profileTaskQueryMaxSize | The maximum size of profile task per query. [...]
-| - | - | profileDataQueryScrollBatchSize | The batch size of query profiling data. [...]
-| - | - | advanced | All settings of ElasticSearch index creation. The value should be in JSON format. [...]
-| - | - | logicSharding | Shard metrics and records indices into multi-physical indices, one index template per metric/meter aggregation function or record. [...]
-| - | h2 | - | H2 storage is designed for demonstration and running in short term (i.e. 1-2 hours) only. [...]
-| - | - | url | H2 connection URL. Defaults to H2 memory mode. [...]
-| - | - | user | Username of H2 database. [...]
-| - | - | password | Password of H2 database. [...]
-| - | - | metadataQueryMaxSize | The maximum size of metadata per query. [...]
-| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution [...]
-| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size [...]
-| - | mysql | - | MySQL Storage. The MySQL JDBC Driver is not in the dist. Please copy it into the oap-lib folder manually. [...]
-| - | - | properties | Hikari connection pool configurations. [...]
-| - | - | metadataQueryMaxSize | The maximum size of metadata per query. [...]
-| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution [...]
-| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size [...]
-| - | postgresql | - | PostgreSQL storage. [...]
-| - | - | properties | Hikari connection pool configurations. [...]
-| - | - | metadataQueryMaxSize | The maximum size of metadata per query. [...]
-| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution [...]
-| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size [...]
-| - | banyandb | - | BanyanDB storage. [...]
-| - | - | host | Host of the BanyanDB. [...]
-| - | - | port | Port of the BanyanDB. [...]
-| - | - | maxBulkSize | The maximum size of write entities in a single batch write call. [...]
-| - | - | flushInterval | Period of flush interval. In the timeunit of seconds. [...]
-| - | - | metricsShardsNumber | Shards Number for measure/metrics. [...]
-| - | - | recordShardsNumber | Shards Number for a normal record. [...]
-| - | - | superDatasetShardsFactor | Shards Factor for a super dataset record, i.e. Shard number of a super dataset is recordShardsNumber*superDatasetShardsFactor. [...]
-| - | - | concurrentWriteThreads | Concurrent consumer threads for batch writing. [...]
-| - | - | profileTaskQueryMaxSize | Max size of ProfileTask to be fetched. [...]
-| agent-analyzer | default | Agent Analyzer. | SW_AGENT_ANALYZER [...]
-| - | - | traceSamplingPolicySettingsFile | The sampling policy including `sampling rate` and `the threshold of trace segment latency` can be configured by the `traceSamplingPolicySettingsFile` file. [...]
-| - | - | slowDBAccessThreshold | The slow database access threshold (in milliseconds). [...]
-| - | - | forceSampleErrorSegment | When sampling mechanism is activated, this config samples the error status segment and ignores the sampling rate. [...]
-| - | - | segmentStatusAnalysisStrategy | Determines the final segment status from span status. Available values are `FROM_SPAN_STATUS` , `FROM_ENTRY_SPAN`, and `FROM_FIRST_SPAN`. `FROM_SPAN_STATUS` indicates that the segment status would be error if any span has an error status. `FROM_ENTRY_SPAN` means that the segme [...]
-| - | - | noUpstreamRealAddressAgents | Exit spans with the component in the list would not generate client-side instance relation metrics, since some tracing plugins (e.g. Nginx-LUA and Envoy) can't collect the real peer IP address. [...]
-| - | - | meterAnalyzerActiveFiles | Indicates which files could be instrumented and analyzed. Multiple files are split by ",". [...]
-| - | - | slowCacheWriteThreshold | The threshold of slow command which is used for writing operation (in milliseconds). [...]
-| - | - | slowCacheReadThreshold | The threshold of slow command which is used for reading (getting) operation (in milliseconds). [...]
-| receiver-sharing-server | default | Sharing server provides new gRPC and restful servers for data collection. Ana designates that servers in the core module are to be used for internal communication only. | - [...]
-| - | - | restHost | Binding IP of RESTful services. Services include GraphQL query and HTTP data report. [...]
-| - | - | restPort | Binding port of RESTful services. [...]
-| - | - | restContextPath | Web context path of RESTful services. [...]
-| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
-| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
-| - | - | restAcceptQueueSize | ServerSocketChannel backlog of RESTful services. [...]
-| - | - | httpMaxRequestHeaderSize | Maximum request header size accepted. [...]
-| - | - | gRPCHost | Binding IP of gRPC services. Services include gRPC data report and internal communication among OAP nodes. [...]
-| - | - | gRPCPort | Binding port of gRPC services. [...]
-| - | - | gRPCThreadPoolSize | Pool size of gRPC server. [...]
-| - | - | gRPCThreadPoolQueueSize | Queue size of gRPC server. [...]
-| - | - | gRPCSslEnabled | Activates SSL for gRPC services. [...]
-| - | - | gRPCSslKeyPath | File path of gRPC SSL key. [...]
-| - | - | gRPCSslCertChainPath | File path of gRPC SSL cert chain. [...]
-| - | - | maxConcurrentCallsPerConnection | The maximum number of concurrent calls permitted for each incoming connection. Defaults to no limit. [...]
-| - | - | authentication | The token text for authentication. Works for gRPC connection only. Once this is set, the client is required to use the same token. [...]
-| log-analyzer | default | Log Analyzer. | SW_LOG_ANALYZER [...]
-| - | - | lalFiles | The LAL configuration file names (without file extension) to be activated. Read [LAL](../../concepts-and-designs/lal.md) for more details. [...]
-| - | - | malFiles | The MAL configuration file names (without file extension) to be activated. Read [LAL](../../concepts-and-designs/lal.md) for more details. [...]
-| event-analyzer | default | Event Analyzer. | SW_EVENT_ANALYZER [...]
-| receiver-register | default | gRPC and HTTPRestful services that provide service, service instance and endpoint register. | - [...]
-| receiver-trace | default | gRPC and HTTPRestful services that accept SkyWalking format traces. | - [...]
-| receiver-jvm | default | gRPC services that accept JVM metrics data. | - [...]
-| receiver-clr | default | gRPC services that accept .Net CLR metrics data. | - [...]
-| receiver-profile | default | gRPC services that accept profile task status and snapshot reporter. | - [...]
-| receiver-zabbix | default | TCP receiver accepts Zabbix format metrics. | - [...]
-| - | - | port | Exported TCP port. Zabbix agent could connect and transport data. [...]
-| - | - | host | Binds to host. [...]
-| - | - | activeFiles | Enables config when agent request is received. [...]
-| service-mesh | default | gRPC services that accept data from inbound mesh probes. | - [...]
-| envoy-metric | default | Envoy `metrics_service` and `ALS(access log service)` are supported by this receiver. The OAL script supports all GAUGE type metrics. | - [...]
-| - | - | acceptMetricsService | Starts Envoy Metrics Service analysis. [...]
-| - | - | alsHTTPAnalysis | Starts Envoy HTTP Access Log Service analysis. Value = `k8s-mesh` means starting the analysis. [...]
-| - | - | alsTCPAnalysis | Starts Envoy TCP Access Log Service analysis. Value = `k8s-mesh` means starting the analysis. [...]
-| - | - | k8sServiceNameRule | `k8sServiceNameRule` allows you to customize the service name in ALS via Kubernetes metadata. The available variables are `pod` and `service`. E.g. you can use `${service.metadata.name}-${pod.metadata.labels.version}` to append the version number to the service name. Note that [...]
-| receiver-otel | default | A receiver for analyzing metrics data from OpenTelemetry. | - [...]
-| - | - | enabledHandlers | Enabled handlers for otel. [...]
-| - | - | enabledOtelMetricsRules | Enabled metric rules for OTLP handler. [...]
-| receiver-zipkin | default | A receiver for Zipkin traces. | - [...]
-| - | - | sampleRate | The sample rate precision is 1/10000, should be between 0 and 10000 [...]
-| - | - | searchableTracesTags | Defines a set of span tag keys which are searchable. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
-| - | - | enableHttpCollector | Enable Http Collector. [...]
-| - | - | restHost | Binding IP of RESTful services. [...]
-| - | - | restPort | Binding port of RESTful services. [...]
-| - | - | restContextPath | Web context path of RESTful services. [...]
-| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
-| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
-| - | - | restAcceptQueueSize | Maximum request header size accepted. [...]
-| - | - | enableKafkaCollector | Enable Kafka Collector. [...]
-| - | - | kafkaBootstrapServers | Kafka ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG. [...]
-| - | - | kafkaGroupId | Kafka ConsumerConfig.GROUP_ID_CONFIG. [...]
-| - | - | kafkaTopic | Kafka Topics. [...]
-| - | - | kafkaConsumerConfig | Kafka consumer config, JSON format as Properties. If it contains the same key with above, would override. [...]
-| - | - | kafkaConsumers | The number of consumers to create. [...]
-| - | - | kafkaHandlerThreadPoolSize | Pool size of Kafka message handler executor. [...]
-| - | - | kafkaHandlerThreadPoolQueueSize | Queue size of Kafka message handler executor. [...]
-| kafka-fetcher | default | Read SkyWalking's native metrics/logs/traces through Kafka server. | - [...]
-| - | - | bootstrapServers | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. [...]
-| - | - | namespace | Namespace aims to isolate multi OAP cluster when using the same Kafka cluster. If you set a namespace for Kafka fetcher, OAP will add a prefix to topic name. You should also set namespace in `agent.config`. The property is named `plugin.kafka.namespace`. [...]
-| - | - | groupId | A unique string that identifies the consumer group to which this consumer belongs. [...]
-| - | - | createTopicIfNotExist | If true, this creates Kafka topic (if it does not already exist). [...]
-| - | - | partitions | The number of partitions for the topic being created. [...]
-| - | - | consumers | The number of consumers to create. [...]
-| - | - | enableNativeProtoLog | Enables fetching and handling native proto log data. [...]
-| - | - | enableNativeJsonLog | Enables fetching and handling native json log data. [...]
-| - | - | replicationFactor | The replication factor for each partition in the topic being created. [...]
-| - | - | kafkaHandlerThreadPoolSize | Pool size of Kafka message handler executor. [...]
-| - | - | kafkaHandlerThreadPoolQueueSize | Queue size of Kafka message handler executor. [...]
-| - | - | topicNameOfMeters | Kafka topic name for meter system data. [...]
-| - | - | topicNameOfMetrics | Kafka topic name for JVM metrics data. [...]
-| - | - | topicNameOfProfiling | Kafka topic name for profiling data. [...]
-| - | - | topicNameOfTracingSegments | Kafka topic name for tracing data. [...]
-| - | - | topicNameOfManagements | Kafka topic name for service instance reporting and registration. [...]
-| - | - | topicNameOfLogs | Kafka topic name for native proto log data. [...]
-| - | - | topicNameOfJsonLogs | Kafka topic name for native json log data. [...]
-| receiver-browser | default | gRPC services that accept browser performance data and error log. | - [...]
-| - | - | sampleRate | Sampling rate for receiving trace. Precise to 1/10000. 10000 means sampling rate of 100% by default. [...]
-| query | graphql | - | GraphQL query implementation. [...]
-| - | - | enableLogTestTool | Enable the log testing API to test the LAL. **NOTE**: This API evaluates untrusted code on the OAP server. A malicious script can do significant damage (steal keys and secrets, remove files and directories, install malware, etc). As such, please enable this API only when you c [...]
-| - | - | maxQueryComplexity | Maximum complexity allowed for the GraphQL query that can be used to abort a query if the total number of data fields queried exceeds the defined threshold. [...]
-| - | - | enableUpdateUITemplate | Allow user add，disable and update UI template. [...]
-| - | - | enableOnDemandPodLog | Ondemand Pod log: fetch the Pod logs on users' demand, the logs are fetched and displayed in real time, and are not persisted in any kind. This is helpful when users want to do some experiments and monitor the logs and see what's happing inside the service. Note: if you print [...]
-| query-zipkin | default | - | This module is for Zipkin query API and support zipkin-lens UI [...]
-| - | - | restHost | Binding IP of RESTful services. [...]
-| - | - | restPort | Binding port of RESTful services. [...]
-| - | - | restContextPath | Web context path of RESTful services. [...]
-| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
-| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
-| - | - | restAcceptQueueSize | Maximum request header size accepted. [...]
-| - | - | lookback | Default look back for traces and autocompleteTags, 1 day in millis [...]
-| - | - | namesMaxAge | The Cache-Control max-age (seconds) for serviceNames, remoteServiceNames and spanNames [...]
-| - | - | uiQueryLimit | Default traces query max size [...]
-| - | - | uiDefaultLookback | Default look back on the UI for search traces, 15 minutes in millis [...]
-| promql | default | - | This module is for PromQL API. [...]
-| - | - | restHost | Binding IP of RESTful services. [...]
-| - | - | restPort | Binding port of RESTful services. [...]
-| - | - | restContextPath | Web context path of RESTful services. [...]
-| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
-| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
-| - | - | restAcceptQueueSize | Maximum request header size accepted. [...]
-| alarm | default | - | Read [alarm doc](backend-alarm.md) for more details. [...]
-| telemetry | - | - | Read [telemetry doc](backend-telemetry.md) for more details. [...]
-| - | none | - | No op implementation. [...]
-| - | prometheus | host | Binding host for Prometheus server fetching data. [...]
-| - | - | port | Binding port for Prometheus server fetching data. [...]
-| configuration | - | - | Read [dynamic configuration doc](dynamic-config.md) for more details. [...]
-| - | grpc | host | DCS server binding hostname. [...]
-| - | - | port | DCS server binding port. [...]
-| - | - | clusterName | Cluster name when reading the latest configuration from DSC server. [...]
-| - | - | period | The period of reading data from DSC server by the OAP (in seconds). [...]
-| - | apollo | apolloMeta | `apollo.meta` in Apollo. [...]
-| - | - | apolloCluster | `apollo.cluster` in Apollo. [...]
-| - | - | apolloEnv | `env` in Apollo. [...]
-| - | - | appId | `app.id` in Apollo. [...]
-| - | - | period | The period of data sync (in seconds). [...]
-| - | zookeeper | namespace | The namespace (represented by root path) that isolates the configurations in the Zookeeper. [...]
-| - | - | hostPort | Hosts and ports of Zookeeper Cluster. [...]
-| - | - | baseSleepTimeMs | The period of Zookeeper client between two retries (in milliseconds). [...]
-| - | - | maxRetries | The maximum retry time. [...]
-| - | - | period | The period of data sync (in seconds). [...]
-| - | etcd | endpoints | Hosts and ports for etcd cluster (separated by commas if multiple). [...]
-| - | - | namespace | Namespace for SkyWalking cluster. [...]
-| - | - | authentication | Indicates whether there is authentication. [...]
-| - | - | user | Etcd auth username. [...]
-| - | - | password | Etcd auth password. [...]
-| - | - | period | The period of data sync (in seconds). [...]
-| - | consul | hostPort | Hosts and ports for Consul cluster. [...]
-| - | - | aclToken | ACL Token of Consul. Empty string means `without ACL token`. [...]
-| - | - | period | The period of data sync (in seconds). [...]
-| - | k8s-configmap | namespace | Deployment namespace of the config map. [...]
-| - | - | labelSelector | Labels for locating configmap. [...]
-| - | - | period | The period of data sync (in seconds). [...]
-| - | nacos | serverAddr | Nacos Server Host. [...]
-| - | - | port | Nacos Server Port. [...]
-| - | - | group | Nacos Configuration namespace. [...]
-| - | - | period | The period of data sync (in seconds). [...]
-| - | - | username | Nacos Auth username. [...]
-| - | - | password | Nacos Auth password. [...]
-| - | - | accessKey | Nacos Auth accessKey. [...]
-| - | - | secretKey | Nacos Auth secretKey. [...]
-| exporter | default | enableGRPCMetrics | Enable gRPC metrics exporter. [...]
-| - | - | gRPCTargetHost | The host of target gRPC server for receiving export data [...]
-| - | - | gRPCTargetPort | The port of target gRPC server for receiving export data. [...]
-| - | - | enableKafkaTrace | Enable Kafka trace exporter. [...]
-| - | - | enableKafkaLog | Enable Kafka log exporter. [...]
+| Module | Provider | Settings | Value(s) and Explanation [...]
+|-------------------------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| core | default | role | Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode OAP opens the service to the agents, then analyzes and aggregates the results, and forwards the results for distributed aggregation. Aggregator mode OAP receives data from Mixer and Receiver role OAP nodes, and perf [...]
+| - | - | restHost | Binding IP of RESTful services. Services include GraphQL query and HTTP data report. [...]
+| - | - | restPort | Binding port of RESTful services. [...]
+| - | - | restContextPath | Web context path of RESTful services. [...]
+| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
+| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
+| - | - | restAcceptQueueSize | ServerSocketChannel Backlog of RESTful services. [...]
+| - | - | httpMaxRequestHeaderSize | Maximum request header size accepted. [...]
+| - | - | gRPCHost | Binding IP of gRPC services, including gRPC data report and internal communication among OAP nodes. [...]
+| - | - | gRPCPort | Binding port of gRPC services. [...]
+| - | - | gRPCSslEnabled | Activates SSL for gRPC services. [...]
+| - | - | gRPCSslKeyPath | File path of gRPC SSL key. [...]
+| - | - | gRPCSslCertChainPath | File path of gRPC SSL cert chain. [...]
+| - | - | gRPCSslTrustedCAPath | File path of gRPC trusted CA. [...]
+| - | - | downsampling | Activated level of down sampling aggregation. [...]
+| - | - | enableDataKeeperExecutor | Controller of TTL scheduler. Once disabled, TTL wouldn't work. [...]
+| - | - | dataKeeperExecutePeriod | Execution period of TTL scheduler (in minutes). Execution doesn't mean deleting data. The storage provider (e.g. ElasticSearch storage) could override this. [...]
+| - | - | recordDataTTL | The lifecycle of record data (in days). Record data includes traces, top N sample records, and logs. Minimum value is 2. [...]
+| - | - | metricsDataTTL | The lifecycle of metrics data (in days), including metadata. We recommend setting metricsDataTTL >= recordDataTTL. Minimum value is 2. [...]
+| - | - | l1FlushPeriod | The period of L1 aggregation flush to L2 aggregation (in milliseconds). [...]
+| - | - | storageSessionTimeout | The threshold of session time (in milliseconds). Default value is 70000. [...]
+| - | - | persistentPeriod | The period of doing data persistence. Unit is second.Default value is 25s [...]
+| - | - | topNReportPeriod | The execution period (in minutes) of top N sampler, which saves sampled data into the storage. [...]
+| - | - | activeExtraModelColumns | Appends entity names (e.g. service names) into metrics storage entities. [...]
+| - | - | serviceNameMaxLength | Maximum length limit of service names. [...]
+| - | - | instanceNameMaxLength | Maximum length limit of service instance names. The maximum length of service + instance names should be less than 200. [...]
+| - | - | endpointNameMaxLength | Maximum length limit of endpoint names. The maximum length of service + endpoint names should be less than 240. [...]
+| - | - | searchableTracesTags | Defines a set of span tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
+| - | - | searchableLogsTags | Defines a set of log tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
+| - | - | searchableAlarmTags | Defines a set of alarm tag keys which are searchable through GraphQL. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
+| - | - | autocompleteTagKeysQueryMaxSize | The max size of tags keys for autocomplete select. [...]
+| - | - | autocompleteTagValuesQueryMaxSize | The max size of tags values for autocomplete select. [...]
+| - | - | gRPCThreadPoolSize | Pool size of gRPC server. [...]
+| - | - | gRPCThreadPoolQueueSize | Queue size of gRPC server. [...]
+| - | - | maxConcurrentCallsPerConnection | The maximum number of concurrent calls permitted for each incoming connection. Defaults to no limit. [...]
+| - | - | maxMessageSize | Sets the maximum message size allowed to be received on the server. Empty means 4 MiB. [...]
+| - | - | remoteTimeout | Timeout for cluster internal communication (in seconds). [...]
+| - | - | maxSizeOfNetworkAddressAlias | The maximum size of network address detected in the system being monitored. [...]
+| - | - | maxPageSizeOfQueryProfileSnapshot | The maximum size for snapshot analysis in an OAP query. [...]
+| - | - | maxSizeOfAnalyzeProfileSnapshot | The maximum number of snapshots analyzed by the OAP. [...]
+| - | - | prepareThreads | The number of threads used to prepare metrics data to the storage. [...]
+| - | - | enableEndpointNameGroupingByOpenapi | Automatically groups endpoints by the given OpenAPI definitions. [...]
+| - | - | maxDurationOfQueryEBPFProfilingData | The maximum duration(in second) of query the eBPF profiling data from database. [...]
+| - | - | maxThreadCountOfQueryEBPFProfilingData | The maximum thread count of query the eBPF profiling data from database. [...]
+| cluster | standalone | - | Standalone is not suitable for running on a single node running. No configuration available. [...]
+| - | zookeeper | namespace | The namespace, represented by root path, isolates the configurations in Zookeeper. [...]
+| - | - | hostPort | Hosts and ports of Zookeeper Cluster. [...]
+| - | - | baseSleepTimeMs | The period of Zookeeper client between two retries (in milliseconds). [...]
+| - | - | maxRetries | The maximum retry time. [...]
+| - | - | enableACL | Opens ACL using `schema` and `expression`. [...]
+| - | - | schema | Schema for the authorization. [...]
+| - | - | expression | Expression for the authorization. [...]
+| - | - | internalComHost | The hostname registered in Zookeeper for the internal communication of OAP cluster. [...]
+| - | - | internalComPort | The port registered in Zookeeper for the internal communication of OAP cluster. [...]
+| - | kubernetes | namespace | Namespace deployed by SkyWalking in k8s. [...]
+| - | - | labelSelector | Labels used for filtering OAP deployment in k8s. [...]
+| - | - | uidEnvName | Environment variable name for reading uid. [...]
+| - | consul | serviceName | Service name for SkyWalking cluster. [...]
+| - | - | hostPort | Hosts and ports for Consul cluster. [...]
+| - | - | aclToken | ACL Token of Consul. Empty string means `without ALC token`. [...]
+| - | - | internalComHost | The hostname registered in Consul for internal communications of the OAP cluster. [...]
+| - | - | internalComPort | The port registered in Consul for internal communications of the OAP cluster. [...]
+| - | etcd | serviceName | Service name for SkyWalking cluster. [...]
+| - | - | endpoints | Hosts and ports for etcd cluster. [...]
+| - | - | namespace | Namespace for SkyWalking cluster. [...]
+| - | - | authentication | Indicates whether there is authentication. [...]
+| - | - | user | Etcd auth username. [...]
+| - | - | password | Etcd auth password. [...]
+| - | - | internalComHost | The hostname registered in etcd for internal communications of the OAP cluster. [...]
+| - | - | internalComPort | The port registered in etcd for internal communications of the OAP cluster. [...]
+| - | Nacos | serviceName | Service name for SkyWalking cluster. [...]
+| - | - | hostPort | Hosts and ports for Nacos cluster. [...]
+| - | - | namespace | Namespace used by SkyWalking node coordination. [...]
+| - | - | internalComHost | The hostname registered in Nacos for internal communications of the OAP cluster. [...]
+| - | - | internalComPort | The port registered in Nacos for internal communications of the OAP cluster. [...]
+| - | - | username | Nacos Auth username. [...]
+| - | - | password | Nacos Auth password. [...]
+| - | - | accessKey | Nacos Auth accessKey. [...]
+| - | - | secretKey | Nacos Auth secretKey. [...]
+| - | - | maxHttpUrisNumberPerService | The maximum number of HTTP URIs per service. [...]
+| storage | elasticsearch | - | ElasticSearch (and OpenSearch) storage implementation. [...]
+| - | - | namespace | Prefix of indexes created and used by SkyWalking. [...]
+| - | - | clusterNodes | ElasticSearch cluster nodes for client connection. [...]
+| - | - | protocol | HTTP or HTTPs. [...]
+| - | - | connectTimeout | Connect timeout of ElasticSearch client (in milliseconds). [...]
+| - | - | socketTimeout | Socket timeout of ElasticSearch client (in milliseconds). [...]
+| - | - | responseTimeout | Response timeout of ElasticSearch client (in milliseconds), `0` disables the timeout. [...]
+| - | - | numHttpClientThread | The number of threads for the underlying HTTP client to perform socket I/O. If the value is <= 0, the number of available processors will be used. [...]
+| - | - | user | Username of ElasticSearch cluster. [...]
+| - | - | password | Password of ElasticSearch cluster. [...]
+| - | - | trustStorePath | Trust JKS file path. Only works when username and password are enabled. [...]
+| - | - | trustStorePass | Trust JKS file password. Only works when username and password are enabled. [...]
+| - | - | secretsManagementFile | Secrets management file in the properties format, including username and password, which are managed by a 3rd party tool. Capable of being updated them at runtime. [...]
+| - | - | dayStep | Represents the number of days in the one-minute/hour/day index. [...]
+| - | - | indexShardsNumber | Shard number of new indexes. [...]
+| - | - | indexReplicasNumber | Replicas number of new indexes. [...]
+| - | - | specificIndexSettings | Specify the settings for each index individually. If configured, this setting has the highest priority and overrides the generic settings. [...]
+| - | - | superDatasetDayStep | Represents the number of days in the super size dataset record index. Default value is the same as dayStep when the value is less than 0. [...]
+| - | - | superDatasetIndexShardsFactor | Super dataset is defined in the code (e.g. trace segments). This factor provides more shards for the super dataset: shards number = indexShardsNumber * superDatasetIndexShardsFactor. This factor also affects Zipkin and Jaeger traces. [...]
+| - | - | superDatasetIndexReplicasNumber | Represents the replicas number in the super size dataset record index. [...]
+| - | - | indexTemplateOrder | The order of index template. [...]
+| - | - | bulkActions | Async bulk size of the record data batch execution. [...]
+| - | - | batchOfBytes | A threshold to control the max body size of ElasticSearch Bulk flush. [...]
+| - | - | flushInterval | Period of flush (in seconds). Does not matter whether `bulkActions` is reached or not. [...]
+| - | - | concurrentRequests | The number of concurrent requests allowed to be executed. [...]
+| - | - | resultWindowMaxSize | The maximum size of dataset when the OAP loads cache, such as network aliases. [...]
+| - | - | metadataQueryMaxSize | The maximum size of metadata per query. [...]
+| - | - | scrollingBatchSize | The batch size of metadata per iteration when `metadataQueryMaxSize` or `resultWindowMaxSize` is too large to be retrieved in a single query. [...]
+| - | - | segmentQueryMaxSize | The maximum size of trace segments per query. [...]
+| - | - | profileTaskQueryMaxSize | The maximum size of profile task per query. [...]
+| - | - | profileDataQueryScrollBatchSize | The batch size of query profiling data. [...]
+| - | - | advanced | All settings of ElasticSearch index creation. The value should be in JSON format. [...]
+| - | - | logicSharding | Shard metrics and records indices into multi-physical indices, one index template per metric/meter aggregation function or record. [...]
+| - | h2 | - | H2 storage is designed for demonstration and running in short term (i.e. 1-2 hours) only. [...]
+| - | - | url | H2 connection URL. Defaults to H2 memory mode. [...]
+| - | - | user | Username of H2 database. [...]
+| - | - | password | Password of H2 database. [...]
+| - | - | metadataQueryMaxSize | The maximum size of metadata per query. [...]
+| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution [...]
+| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size [...]
+| - | mysql | - | MySQL Storage. The MySQL JDBC Driver is not in the dist. Please copy it into the oap-lib folder manually. [...]
+| - | - | properties | Hikari connection pool configurations. [...]
+| - | - | metadataQueryMaxSize | The maximum size of metadata per query. [...]
+| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution [...]
+| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size [...]
+| - | postgresql | - | PostgreSQL storage. [...]
+| - | - | properties | Hikari connection pool configurations. [...]
+| - | - | metadataQueryMaxSize | The maximum size of metadata per query. [...]
+| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution [...]
+| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size [...]
+| - | banyandb | - | BanyanDB storage. [...]
+| - | - | host | Host of the BanyanDB. [...]
+| - | - | port | Port of the BanyanDB. [...]
+| - | - | maxBulkSize | The maximum size of write entities in a single batch write call. [...]
+| - | - | flushInterval | Period of flush interval. In the timeunit of seconds. [...]
+| - | - | metricsShardsNumber | Shards Number for measure/metrics. [...]
+| - | - | recordShardsNumber | Shards Number for a normal record. [...]
+| - | - | superDatasetShardsFactor | Shards Factor for a super dataset record, i.e. Shard number of a super dataset is recordShardsNumber*superDatasetShardsFactor. [...]
+| - | - | concurrentWriteThreads | Concurrent consumer threads for batch writing. [...]
+| - | - | profileTaskQueryMaxSize | Max size of ProfileTask to be fetched. [...]
+| agent-analyzer | default | Agent Analyzer. | SW_AGENT_ANALYZER [...]
+| - | - | traceSamplingPolicySettingsFile | The sampling policy including `sampling rate` and `the threshold of trace segment latency` can be configured by the `traceSamplingPolicySettingsFile` file. [...]
+| - | - | slowDBAccessThreshold | The slow database access threshold (in milliseconds). [...]
+| - | - | forceSampleErrorSegment | When sampling mechanism is activated, this config samples the error status segment and ignores the sampling rate. [...]
+| - | - | segmentStatusAnalysisStrategy | Determines the final segment status from span status. Available values are `FROM_SPAN_STATUS` , `FROM_ENTRY_SPAN`, and `FROM_FIRST_SPAN`. `FROM_SPAN_STATUS` indicates that the segment status would be error if any span has an error status. `FROM_ENTRY_SPAN` means that the segme [...]
+| - | - | noUpstreamRealAddressAgents | Exit spans with the component in the list would not generate client-side instance relation metrics, since some tracing plugins (e.g. Nginx-LUA and Envoy) can't collect the real peer IP address. [...]
+| - | - | meterAnalyzerActiveFiles | Indicates which files could be instrumented and analyzed. Multiple files are split by ",". [...]
+| - | - | slowCacheWriteThreshold | The threshold of slow command which is used for writing operation (in milliseconds). [...]
+| - | - | slowCacheReadThreshold | The threshold of slow command which is used for reading (getting) operation (in milliseconds). [...]
+| receiver-sharing-server | default | Sharing server provides new gRPC and restful servers for data collection. Ana designates that servers in the core module are to be used for internal communication only. | - [...]
+| - | - | restHost | Binding IP of RESTful services. Services include GraphQL query and HTTP data report. [...]
+| - | - | restPort | Binding port of RESTful services. [...]
+| - | - | restContextPath | Web context path of RESTful services. [...]
+| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
+| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
+| - | - | restAcceptQueueSize | ServerSocketChannel backlog of RESTful services. [...]
+| - | - | httpMaxRequestHeaderSize | Maximum request header size accepted. [...]
+| - | - | gRPCHost | Binding IP of gRPC services. Services include gRPC data report and internal communication among OAP nodes. [...]
+| - | - | gRPCPort | Binding port of gRPC services. [...]
+| - | - | gRPCThreadPoolSize | Pool size of gRPC server. [...]
+| - | - | gRPCThreadPoolQueueSize | Queue size of gRPC server. [...]
+| - | - | gRPCSslEnabled | Activates SSL for gRPC services. [...]
+| - | - | gRPCSslKeyPath | File path of gRPC SSL key. [...]
+| - | - | gRPCSslCertChainPath | File path of gRPC SSL cert chain. [...]
+| - | - | maxConcurrentCallsPerConnection | The maximum number of concurrent calls permitted for each incoming connection. Defaults to no limit. [...]
+| - | - | authentication | The token text for authentication. Works for gRPC connection only. Once this is set, the client is required to use the same token. [...]
+| log-analyzer | default | Log Analyzer. | SW_LOG_ANALYZER [...]
+| - | - | lalFiles | The LAL configuration file names (without file extension) to be activated. Read [LAL](../../concepts-and-designs/lal.md) for more details. [...]
+| - | - | malFiles | The MAL configuration file names (without file extension) to be activated. Read [LAL](../../concepts-and-designs/lal.md) for more details. [...]
+| event-analyzer | default | Event Analyzer. | SW_EVENT_ANALYZER [...]
+| receiver-register | default | gRPC and HTTPRestful services that provide service, service instance and endpoint register. | - [...]
+| receiver-trace | default | gRPC and HTTPRestful services that accept SkyWalking format traces. | - [...]
+| receiver-jvm | default | gRPC services that accept JVM metrics data. | - [...]
+| receiver-clr | default | gRPC services that accept .Net CLR metrics data. | - [...]
+| receiver-profile | default | gRPC services that accept profile task status and snapshot reporter. | - [...]
+| receiver-zabbix | default | TCP receiver accepts Zabbix format metrics. | - [...]
+| - | - | port | Exported TCP port. Zabbix agent could connect and transport data. [...]
+| - | - | host | Binds to host. [...]
+| - | - | activeFiles | Enables config when agent request is received. [...]
+| service-mesh | default | gRPC services that accept data from inbound mesh probes. | - [...]
+| envoy-metric | default | Envoy `metrics_service` and `ALS(access log service)` are supported by this receiver. The OAL script supports all GAUGE type metrics. | - [...]
+| - | - | acceptMetricsService | Starts Envoy Metrics Service analysis. [...]
+| - | - | alsHTTPAnalysis | Starts Envoy HTTP Access Log Service analysis. Value = `k8s-mesh` means starting the analysis. [...]
+| - | - | alsTCPAnalysis | Starts Envoy TCP Access Log Service analysis. Value = `k8s-mesh` means starting the analysis. [...]
+| - | - | k8sServiceNameRule | `k8sServiceNameRule` allows you to customize the service name in ALS via Kubernetes metadata. The available variables are `pod` and `service`. E.g. you can use `${service.metadata.name}-${pod.metadata.labels.version}` to append the version number to the service name. Note that [...]
+| receiver-otel | default | A receiver for analyzing metrics data from OpenTelemetry. | - [...]
+| - | - | enabledHandlers | Enabled handlers for otel. [...]
+| - | - | enabledOtelMetricsRules | Enabled metric rules for OTLP handler. [...]
+| receiver-zipkin | default | A receiver for Zipkin traces. | - [...]
+| - | - | sampleRate | The sample rate precision is 1/10000, should be between 0 and 10000 [...]
+| - | - | searchableTracesTags | Defines a set of span tag keys which are searchable. Multiple values are separated by commas. The max length of key=value should be less than 256 or will be dropped. [...]
+| - | - | enableHttpCollector | Enable Http Collector. [...]
+| - | - | restHost | Binding IP of RESTful services. [...]
+| - | - | restPort | Binding port of RESTful services. [...]
+| - | - | restContextPath | Web context path of RESTful services. [...]
+| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
+| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
+| - | - | restAcceptQueueSize | Maximum request header size accepted. [...]
+| - | - | enableKafkaCollector | Enable Kafka Collector. [...]
+| - | - | kafkaBootstrapServers | Kafka ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG. [...]
+| - | - | kafkaGroupId | Kafka ConsumerConfig.GROUP_ID_CONFIG. [...]
+| - | - | kafkaTopic | Kafka Topics. [...]
+| - | - | kafkaConsumerConfig | Kafka consumer config, JSON format as Properties. If it contains the same key with above, would override. [...]
+| - | - | kafkaConsumers | The number of consumers to create. [...]
+| - | - | kafkaHandlerThreadPoolSize | Pool size of Kafka message handler executor. [...]
+| - | - | kafkaHandlerThreadPoolQueueSize | Queue size of Kafka message handler executor. [...]
+| kafka-fetcher | default | Read SkyWalking's native metrics/logs/traces through Kafka server. | - [...]
+| - | - | bootstrapServers | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. [...]
+| - | - | namespace | Namespace aims to isolate multi OAP cluster when using the same Kafka cluster. If you set a namespace for Kafka fetcher, OAP will add a prefix to topic name. You should also set namespace in `agent.config`. The property is named `plugin.kafka.namespace`. [...]
+| - | - | groupId | A unique string that identifies the consumer group to which this consumer belongs. [...]
+| - | - | createTopicIfNotExist | If true, this creates Kafka topic (if it does not already exist). [...]
+| - | - | partitions | The number of partitions for the topic being created. [...]
+| - | - | consumers | The number of consumers to create. [...]
+| - | - | enableNativeProtoLog | Enables fetching and handling native proto log data. [...]
+| - | - | enableNativeJsonLog | Enables fetching and handling native json log data. [...]
+| - | - | replicationFactor | The replication factor for each partition in the topic being created. [...]
+| - | - | kafkaHandlerThreadPoolSize | Pool size of Kafka message handler executor. [...]
+| - | - | kafkaHandlerThreadPoolQueueSize | Queue size of Kafka message handler executor. [...]
+| - | - | topicNameOfMeters | Kafka topic name for meter system data. [...]
+| - | - | topicNameOfMetrics | Kafka topic name for JVM metrics data. [...]
+| - | - | topicNameOfProfiling | Kafka topic name for profiling data. [...]
+| - | - | topicNameOfTracingSegments | Kafka topic name for tracing data. [...]
+| - | - | topicNameOfManagements | Kafka topic name for service instance reporting and registration. [...]
+| - | - | topicNameOfLogs | Kafka topic name for native proto log data. [...]
+| - | - | topicNameOfJsonLogs | Kafka topic name for native json log data. [...]
+| receiver-browser | default | gRPC services that accept browser performance data and error log. | - [...]
+| - | - | sampleRate | Sampling rate for receiving trace. Precise to 1/10000. 10000 means sampling rate of 100% by default. [...]
+| query | graphql | - | GraphQL query implementation. [...]
+| - | - | enableLogTestTool | Enable the log testing API to test the LAL. **NOTE**: This API evaluates untrusted code on the OAP server. A malicious script can do significant damage (steal keys and secrets, remove files and directories, install malware, etc). As such, please enable this API only when you c [...]
+| - | - | maxQueryComplexity | Maximum complexity allowed for the GraphQL query that can be used to abort a query if the total number of data fields queried exceeds the defined threshold. [...]
+| - | - | enableUpdateUITemplate | Allow user add，disable and update UI template. [...]
+| - | - | enableOnDemandPodLog | Ondemand Pod log: fetch the Pod logs on users' demand, the logs are fetched and displayed in real time, and are not persisted in any kind. This is helpful when users want to do some experiments and monitor the logs and see what's happing inside the service. Note: if you print [...]
+| query-zipkin | default | - | This module is for Zipkin query API and support zipkin-lens UI [...]
+| - | - | restHost | Binding IP of RESTful services. [...]
+| - | - | restPort | Binding port of RESTful services. [...]
+| - | - | restContextPath | Web context path of RESTful services. [...]
+| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
+| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
+| - | - | restAcceptQueueSize | Maximum request header size accepted. [...]
+| - | - | lookback | Default look back for traces and autocompleteTags, 1 day in millis [...]
+| - | - | namesMaxAge | The Cache-Control max-age (seconds) for serviceNames, remoteServiceNames and spanNames [...]
+| - | - | uiQueryLimit | Default traces query max size [...]
+| - | - | uiDefaultLookback | Default look back on the UI for search traces, 15 minutes in millis [...]
+| promql | default | - | This module is for PromQL API. [...]
+| - | - | restHost | Binding IP of RESTful services. [...]
+| - | - | restPort | Binding port of RESTful services. [...]
+| - | - | restContextPath | Web context path of RESTful services. [...]
+| - | - | restMaxThreads | Maximum thread number of RESTful services. [...]
+| - | - | restIdleTimeOut | Connector idle timeout of RESTful services (in milliseconds). [...]
+| - | - | restAcceptQueueSize | Maximum request header size accepted. [...]
+| alarm | default | - | Read [alarm doc](backend-alarm.md) for more details. [...]
+| telemetry | - | - | Read [telemetry doc](backend-telemetry.md) for more details. [...]
+| - | none | - | No op implementation. [...]
+| - | prometheus | host | Binding host for Prometheus server fetching data. [...]
+| - | - | port | Binding port for Prometheus server fetching data. [...]
+| configuration | - | - | Read [dynamic configuration doc](dynamic-config.md) for more details. [...]
+| - | grpc | host | DCS server binding hostname. [...]
+| - | - | port | DCS server binding port. [...]
+| - | - | clusterName | Cluster name when reading the latest configuration from DSC server. [...]
+| - | - | period | The period of reading data from DSC server by the OAP (in seconds). [...]
+| - | apollo | apolloMeta | `apollo.meta` in Apollo. [...]
+| - | - | apolloCluster | `apollo.cluster` in Apollo. [...]
+| - | - | apolloEnv | `env` in Apollo. [...]
+| - | - | appId | `app.id` in Apollo. [...]
+| - | - | period | The period of data sync (in seconds). [...]
+| - | zookeeper | namespace | The namespace (represented by root path) that isolates the configurations in the Zookeeper. [...]
+| - | - | hostPort | Hosts and ports of Zookeeper Cluster. [...]
+| - | - | baseSleepTimeMs | The period of Zookeeper client between two retries (in milliseconds). [...]
+| - | - | maxRetries | The maximum retry time. [...]
+| - | - | period | The period of data sync (in seconds). [...]
+| - | etcd | endpoints | Hosts and ports for etcd cluster (separated by commas if multiple). [...]
+| - | - | namespace | Namespace for SkyWalking cluster. [...]
+| - | - | authentication | Indicates whether there is authentication. [...]
+| - | - | user | Etcd auth username. [...]
+| - | - | password | Etcd auth password. [...]
+| - | - | period | The period of data sync (in seconds). [...]
+| - | consul | hostPort | Hosts and ports for Consul cluster. [...]
+| - | - | aclToken | ACL Token of Consul. Empty string means `without ACL token`. [...]
+| - | - | period | The period of data sync (in seconds). [...]
+| - | k8s-configmap | namespace | Deployment namespace of the config map. [...]
+| - | - | labelSelector | Labels for locating configmap. [...]
+| - | - | period | The period of data sync (in seconds). [...]
+| - | nacos | serverAddr | Nacos Server Host. [...]
+| - | - | port | Nacos Server Port. [...]
+| - | - | group | Nacos Configuration namespace. [...]
+| - | - | period | The period of data sync (in seconds). [...]
+| - | - | username | Nacos Auth username. [...]
+| - | - | password | Nacos Auth password. [...]
+| - | - | accessKey | Nacos Auth accessKey. [...]
+| - | - | secretKey | Nacos Auth secretKey. [...]
+| exporter | default | enableGRPCMetrics | Enable gRPC metrics exporter. [...]
+| - | - | gRPCTargetHost | The host of target gRPC server for receiving export data [...]
+| - | - | gRPCTargetPort | The port of target gRPC server for receiving export data. [...]
+| - | - | enableKafkaTrace | Enable Kafka trace exporter. [...]
+| - | - | enableKafkaLog | Enable Kafka log exporter. [...]
| - | - | kafkaBootstrapServers | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. [...]
-| - | - | kafkaProducerConfig | Kafka producer config, JSON format as Properties. [...]
-| - | - | kafkaTopicTrace | Kafka topic name for trace. [...]
-| - | - | kafkaTopicLog | Kafka topic name for log. [...]
-| - | - | exportErrorStatusTraceOnly | Export error status trace segments through the Kafka channel. [...]
-| health-checker | default | checkIntervalSeconds | The period of checking OAP internal health status (in seconds). [...]
-| configuration-discovery | default | disableMessageDigest | If true, agent receives the latest configuration every time, even without making any changes. By default, OAP uses the SHA512 message digest mechanism to detect changes in configuration. [...]
-| receiver-event | default | gRPC services that handle events data. | - [...]
-| aws-firehose-receiver | default | host | Binding IP of HTTP server [...]
-| - | - | port | Binding port of HTTP server [...]
-| - | - | contextPath | Context path of HTTP server [...]
-| - | - | maxThreads | Max Thtread number of HTTP server [...]
-| - | - | idleTimeOut | Idle timeout of a connection for keep-alive. [...]
-| - | - | acceptQueueSize | Maximum allowed number of open connections [...]
-| - | - | maxRequestHeaderSize | Maximum length of all headers in an HTTP/1 response [...]
-| - | - | firehoseAccessKey | The AccessKey of AWS firhose [...]
-| - | - | enableTLS | Indicate if enable HTTPS for the server [...]
-| - | - | tlsKeyPath | TLS key path [...]
-| - | - | tlsCertChainPath | TLS certificate chain path [...]
+| - | - | kafkaProducerConfig | Kafka producer config, JSON format as Properties. [...]
+| - | - | kafkaTopicTrace | Kafka topic name for trace. [...]
+| - | - | kafkaTopicLog | Kafka topic name for log. [...]
+| - | - | exportErrorStatusTraceOnly | Export error status trace segments through the Kafka channel. [...]
+| health-checker | default | checkIntervalSeconds | The period of checking OAP internal health status (in seconds). [...]
+| configuration-discovery | default | disableMessageDigest | If true, agent receives the latest configuration every time, even without making any changes. By default, OAP uses the SHA512 message digest mechanism to detect changes in configuration. [...]
+| receiver-event | default | gRPC services that handle events data. | - [...]
+| aws-firehose-receiver | default | host | Binding IP of HTTP server [...]
+| - | - | port | Binding port of HTTP server [...]
+| - | - | contextPath | Context path of HTTP server [...]
+| - | - | maxThreads | Max Thtread number of HTTP server [...]
+| - | - | idleTimeOut | Idle timeout of a connection for keep-alive. [...]
+| - | - | acceptQueueSize | Maximum allowed number of open connections [...]
+| - | - | maxRequestHeaderSize | Maximum length of all headers in an HTTP/1 response [...]
+| - | - | firehoseAccessKey | The AccessKey of AWS firhose [...]
+| - | - | enableTLS | Indicate if enable HTTPS for the server [...]
+| - | - | tlsKeyPath | TLS key path [...]
+| - | - | tlsCertChainPath | TLS certificate chain path [...]
+| ai-pipeline | default | | [...]
+| - | - | uriRecognitionServerAddr | The address of the URI recognition server. [...]
+| - | - | uriRecognitionServerPort | The port of the URI recognition server. [...]
## Note
diff --git a/docs/en/setup/backend/endpoint-grouping-rules.md b/docs/en/setup/backend/endpoint-grouping-rules.md
index 6253dd4ba5..f8b8949637 100644
--- a/docs/en/setup/backend/endpoint-grouping-rules.md
+++ b/docs/en/setup/backend/endpoint-grouping-rules.md
@@ -275,8 +275,8 @@ Here are some use cases:
| Incoming Endpoint | Incoming Service | x-sw-service-name | x-sw-endpoint-name-match-rule | x-sw-endpoint-name-format | Matched | Grouping Result |
|-----|-----|-----|-----|-----|-----|-----|
| `GET:/products` | serviceB | default | default | default | true | `GET:/products` |
- | `GET:/products/123` | serviceB | default | default | default | true | `GET:/products{id}` |
| `GET:/products/asia/cn` | serviceB | default | default | default | true | `GET:/products/{region}/{country}` |
+ | `GET:/products/123` | serviceB | default | default | default | true | `GET:/products{id}` |
| `GET:/products/123/abc/efg` | serviceB | default | default | default | false | `GET:/products/123/abc/efg` |
| `<GET>:/products/123` | serviceB | default | default | default | false | `<GET>:/products/123`|
| `GET:/products/123` | serviceC | default | default | default | false | `GET:/products/123` |
@@ -302,7 +302,6 @@ grouping:
# Endpoint of the service would follow the following rules
- service-name: serviceA
rules:
- # Logic name when the regex expression matched.
- - endpoint-name: /prod/{id}
- regex: \/prod\/.+
+ # {var} represents any variable string in the URI.
+ - /prod/{var}
```
diff --git a/docs/menu.yml b/docs/menu.yml
index 8d1532820f..87b2bee79f 100644
--- a/docs/menu.yml
+++ b/docs/menu.yml
@@ -157,6 +157,12 @@ catalog:
path: "/en/setup/backend/exporter"
- name: "Dynamic Configuration"
path: "/en/setup/backend/dynamic-config"
+ - name: "AI Pipeline"
+ catalog:
+ - name: "Introduction"
+ path: "/en/setup/ai-pipeline/readme"
+ - name: "HTTP Restful URI recognition"
+ path: "/en/setup/ai-pipeline/http-restful-uri-pattern"
- name: "UI Setup"
catalog:
- name: "Native UI"
diff --git a/oap-server/ai-pipeline/pom.xml b/oap-server/ai-pipeline/pom.xml
new file mode 100644
index 0000000000..da3b73f0e0
--- /dev/null
+++ b/oap-server/ai-pipeline/pom.xml
@@ -0,0 +1,100 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>oap-server</artifactId>
+ <version>9.5.0-SNAPSHOT</version>
+ </parent>
+
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>ai-pipeline</artifactId>
+
+ <description>
+ AI pipeline module include pipelines connecting the OAP kernel and any 3rd party AI and/or algorithm
+ server. The kernel would call the pipeline in the specific context/scenario with well formatted telemetry data,
+ and expect the remote to respond with the result of the AI/algorithm as quick as possible, but in async way in
+ case of large size of data or lower performance behavior of the remote.
+ </description>
+
+ <dependencies>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-netty</artifactId>
+ <version>${grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-protobuf</artifactId>
+ <version>${grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>io.grpc</groupId>
+ <artifactId>grpc-stub</artifactId>
+ <version>${grpc.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-module</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-client</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.xolstice.maven.plugins</groupId>
+ <artifactId>protobuf-maven-plugin</artifactId>
+ <version>${protobuf-maven-plugin.version}</version>
+ <configuration>
+ <!--
+ The version of protoc must match protobuf-java. If you don't depend on
+ protobuf-java directly, you will be transitively depending on the
+ protobuf-java version that grpc depends on.
+ -->
+ <protocArtifact>
+ com.google.protobuf:protoc:${com.google.protobuf.protoc.version}:exe:${os.detected.classifier}
+ </protocArtifact>
+ <pluginId>grpc-java</pluginId>
+ <pluginArtifact>
+ io.grpc:protoc-gen-grpc-java:${protoc-gen-grpc-java.plugin.version}:exe:${os.detected.classifier}
+ </pluginArtifact>
+ </configuration>
+ <executions>
+ <execution>
+ <goals>
+ <goal>compile</goal>
+ <goal>compile-custom</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
\ No newline at end of file
diff --git a/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/AIPipelineConfig.java b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/AIPipelineConfig.java
new file mode 100644
index 0000000000..64714a77d4
--- /dev/null
+++ b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/AIPipelineConfig.java
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.ai.pipeline;
+
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+
+@Getter
+@Setter
+public class AIPipelineConfig extends ModuleConfig {
+ private String uriRecognitionServerAddr;
+ private int uriRecognitionServerPort = 17128;
+}
diff --git a/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/AIPipelineModule.java b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/AIPipelineModule.java
new file mode 100644
index 0000000000..0ffe9d975a
--- /dev/null
+++ b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/AIPipelineModule.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.ai.pipeline;
+
+import org.apache.skywalking.oap.server.ai.pipeline.services.api.HttpUriRecognition;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+
+public class AIPipelineModule extends ModuleDefine {
+ public static final String NAME = "ai-pipeline";
+
+ public AIPipelineModule() {
+ super(NAME);
+ }
+
+ @Override
+ public Class[] services() {
+ return new Class[]{
+ HttpUriRecognition.class
+ };
+ }
+}
diff --git a/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/AIPipelineProvider.java b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/AIPipelineProvider.java
new file mode 100644
index 0000000000..6bc924781c
--- /dev/null
+++ b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/AIPipelineProvider.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.ai.pipeline;
+
+import org.apache.skywalking.oap.server.ai.pipeline.services.HttpUriRecognitionService;
+import org.apache.skywalking.oap.server.ai.pipeline.services.api.HttpUriRecognition;
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+import org.apache.skywalking.oap.server.library.module.ModuleProvider;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
+
+public class AIPipelineProvider extends ModuleProvider {
+ private AIPipelineConfig aiPipelineConfig;
+
+ @Override
+ public String name() {
+ return "default";
+ }
+
+ @Override
+ public Class<? extends ModuleDefine> module() {
+ return AIPipelineModule.class;
+ }
+
+ @Override
+ public ConfigCreator<? extends ModuleConfig> newConfigCreator() {
+ return new ConfigCreator<AIPipelineConfig>() {
+ @Override
+ public Class<AIPipelineConfig> type() {
+ return AIPipelineConfig.class;
+ }
+
+ @Override
+ public void onInitialized(final AIPipelineConfig initialized) {
+ aiPipelineConfig = initialized;
+ }
+ };
+ }
+
+ @Override
+ public void prepare() throws ServiceNotProvidedException, ModuleStartException {
+ final HttpUriRecognitionService httpUriRecognitionService = new HttpUriRecognitionService(
+ aiPipelineConfig.getUriRecognitionServerAddr(),
+ aiPipelineConfig.getUriRecognitionServerPort()
+ );
+ this.registerServiceImplementation(HttpUriRecognition.class, httpUriRecognitionService);
+ }
+
+ @Override
+ public void start() throws ServiceNotProvidedException, ModuleStartException {
+
+ }
+
+ @Override
+ public void notifyAfterCompleted() throws ServiceNotProvidedException, ModuleStartException {
+
+ }
+
+ @Override
+ public String[] requiredModules() {
+ return new String[0];
+ }
+}
diff --git a/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/services/HttpUriRecognitionService.java b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/services/HttpUriRecognitionService.java
new file mode 100644
index 0000000000..66d5e5bfe8
--- /dev/null
+++ b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/services/HttpUriRecognitionService.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.ai.pipeline.services;
+
+import io.grpc.ManagedChannel;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.ai.pipeline.grpc.HttpRawUri;
+import org.apache.skywalking.oap.server.ai.pipeline.grpc.HttpUriRecognitionRequest;
+import org.apache.skywalking.oap.server.ai.pipeline.grpc.HttpUriRecognitionResponse;
+import org.apache.skywalking.oap.server.ai.pipeline.grpc.HttpUriRecognitionServiceGrpc;
+import org.apache.skywalking.oap.server.ai.pipeline.grpc.HttpUriRecognitionSyncRequest;
+import org.apache.skywalking.oap.server.ai.pipeline.services.api.HttpUriPattern;
+import org.apache.skywalking.oap.server.ai.pipeline.services.api.HttpUriRecognition;
+import org.apache.skywalking.oap.server.library.client.grpc.GRPCClient;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
+
+@Slf4j
+public class HttpUriRecognitionService implements HttpUriRecognition {
+ private HttpUriRecognitionServiceGrpc.HttpUriRecognitionServiceBlockingStub stub;
+ private String version = "NULL";
+
+ public HttpUriRecognitionService(String addr, int port) {
+ if (StringUtil.isEmpty(addr) || port <= 0) {
+ return;
+ }
+ GRPCClient client = new GRPCClient(addr, port);
+ client.connect();
+ ManagedChannel channel = client.getChannel();
+ stub = HttpUriRecognitionServiceGrpc.newBlockingStub(channel);
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return stub != null;
+ }
+
+ @Override
+ public List<HttpUriPattern> fetchAllPatterns(final String service) {
+ try {
+ if (stub == null) {
+ return null;
+ }
+ final HttpUriRecognitionResponse httpUriRecognitionResponse
+ = stub.withDeadlineAfter(30, TimeUnit.SECONDS)
+ .fetchAllPatterns(
+ HttpUriRecognitionSyncRequest.newBuilder()
+ .setService(service)
+ .setVersion(version)
+ .build()
+ );
+ final String newVersion = httpUriRecognitionResponse.getVersion();
+ if (version.equals(newVersion)) {
+ // Same version, nothing changed.
+ return null;
+ }
+ return httpUriRecognitionResponse.getPatternsList()
+ .stream()
+ .map(pattern -> new HttpUriPattern(pattern.getPattern()))
+ .collect(Collectors.toList());
+
+ } catch (Exception e) {
+ log.error("fetch all patterns failed from remote server.", e);
+ return null;
+ }
+ }
+
+ @Override
+ public void feedRawData(final String service, final List<HTTPUri> unrecognizedURIs) {
+ try {
+ if (stub == null) {
+ return;
+ }
+ final HttpUriRecognitionRequest.Builder builder = HttpUriRecognitionRequest.newBuilder();
+ builder.setService(service);
+ unrecognizedURIs.forEach(httpUri -> {
+ builder.getUnrecognizedUrisBuilderList().add(
+ HttpRawUri.newBuilder().setName(httpUri.getName()).setMatchedCounter(httpUri.getMatchedCounter())
+ );
+ });
+ stub.withDeadlineAfter(30, TimeUnit.SECONDS)
+ .feedRawData(builder.build());
+ } catch (Exception e) {
+ log.error("feed matched and unmatched URIs to the remote server.", e);
+ }
+ }
+}
diff --git a/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/services/api/HttpUriPattern.java b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/services/api/HttpUriPattern.java
new file mode 100644
index 0000000000..bb3f0a588d
--- /dev/null
+++ b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/services/api/HttpUriPattern.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.ai.pipeline.services.api;
+
+import lombok.Getter;
+import lombok.RequiredArgsConstructor;
+
+@RequiredArgsConstructor
+@Getter
+public class HttpUriPattern {
+ /**
+ * Quick match URI pattern. {var} represents a variable part in the URI.
+ *
+ * /product/{var} is the pattern for /product/1, /product/2
+ */
+ private final String pattern;
+}
diff --git a/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/services/api/HttpUriRecognition.java b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/services/api/HttpUriRecognition.java
new file mode 100644
index 0000000000..51d299ccb9
--- /dev/null
+++ b/oap-server/ai-pipeline/src/main/java/org/apache/skywalking/oap/server/ai/pipeline/services/api/HttpUriRecognition.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.ai.pipeline.services.api;
+
+import java.util.List;
+import lombok.Getter;
+import lombok.RequiredArgsConstructor;
+import org.apache.skywalking.oap.server.library.module.Service;
+
+/**
+ * HttpUriRecognition is a service to recognize the patterns of HTTP URIs
+ */
+public interface HttpUriRecognition extends Service {
+
+ /**
+ * @return true if the service is initialized and active.
+ */
+ boolean isInitialized();
+
+ /**
+ * Fetch all patterns of identified HTTP URIs
+ * @param service the name of the service
+ * @return the list of patterns of HTTP URIs
+ */
+ List<HttpUriPattern> fetchAllPatterns(String service);
+
+ /**
+ * Feed all data for the pattern recognition of HTTP URIs
+ *
+ * @param service the name of the service
+ * @param unrecognizedURIs the list of unrecognized URIs
+ */
+ void feedRawData(String service, List<HTTPUri> unrecognizedURIs);
+
+ @RequiredArgsConstructor
+ @Getter
+ class HTTPUri {
+ private final String name;
+ private final long matchedCounter;
+ }
+}
diff --git a/oap-server/ai-pipeline/src/main/proto/ai_http_uri_recognition.proto b/oap-server/ai-pipeline/src/main/proto/ai_http_uri_recognition.proto
new file mode 100644
index 0000000000..f5cb57eb30
--- /dev/null
+++ b/oap-server/ai-pipeline/src/main/proto/ai_http_uri_recognition.proto
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+import "google/protobuf/empty.proto";
+
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.oap.server.ai.pipeline.grpc";
+
+service HttpUriRecognitionService {
+ // Sync for the pattern recognition dictionary.
+ rpc fetchAllPatterns(HttpUriRecognitionSyncRequest) returns (HttpUriRecognitionResponse) {}
+ // Feed new raw data and matched patterns to the AI-server.
+ rpc feedRawData(HttpUriRecognitionRequest) returns (google.protobuf.Empty) {}
+}
+
+message HttpUriRecognitionSyncRequest {
+ string service = 1;
+ // The version of pattern dictionary at the OAP side.
+ string version = 2;
+}
+
+message HttpUriRecognitionRequest {
+ string service = 1;
+ repeated HttpRawUri unrecognizedUris = 2;
+}
+
+message HttpRawUri {
+ string name = 1;
+ int64 matchedCounter = 2;
+}
+
+message HttpUriRecognitionResponse {
+ repeated Pattern patterns = 1;
+ // The version of pattern dictionary at the AI-server side.
+ // If it is as same as HttpUriRecognitionSyncRequest#version,
+ // patterns could be empty as nothing should be updated.
+ string version = 2;
+}
+
+message Pattern {
+ // Quick match URI pattern. {var} represents a variable part in the URI.
+ // /product/{var} is the pattern for /product/1, /product/2
+ // GET:/product/{var}/detail is the pattern for GET:/product/1/detail, GET:/product/2/detail
+ string pattern = 2;
+}
\ No newline at end of file
diff --git a/oap-server/server-core/src/test/resources/endpoint-name-grouping.yml b/oap-server/ai-pipeline/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine
similarity index 54%
copy from oap-server/server-core/src/test/resources/endpoint-name-grouping.yml
copy to oap-server/ai-pipeline/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine
index 63bddcba2d..fcfbe3cc1c 100644
--- a/oap-server/server-core/src/test/resources/endpoint-name-grouping.yml
+++ b/oap-server/ai-pipeline/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine
@@ -1,3 +1,4 @@
+#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -12,18 +13,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+#
+#
-# Endpoint name grouping rules.
-# In most cased, endpoint name should be detected by agents or service mesh automatically, and aggregate the metrics based
-# on the name.
-# But, in some cases, application puts the parameter in the endpoint name, such as putting order id in the URI, like
-# /prod/ORDER123, /prod/ORDER456.
-# This grouping file provides the regex based definition capability to merge those endpoints into a group by better and
-# more meaningful aggregation metrics.
-
-grouping:
- # Endpoint of the service would follow the following rules
- - service-name: serviceA
- rules:
- - endpoint-name: /prod/{id}
- regex: \/prod\/.+
\ No newline at end of file
+org.apache.skywalking.oap.server.ai.pipeline.AIPipelineModule
diff --git a/oap-server/server-core/src/test/resources/endpoint-name-grouping.yml b/oap-server/ai-pipeline/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider
similarity index 54%
copy from oap-server/server-core/src/test/resources/endpoint-name-grouping.yml
copy to oap-server/ai-pipeline/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider
index 63bddcba2d..84a4ce71d6 100644
--- a/oap-server/server-core/src/test/resources/endpoint-name-grouping.yml
+++ b/oap-server/ai-pipeline/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider
@@ -1,3 +1,4 @@
+#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -12,18 +13,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+#
+#
-# Endpoint name grouping rules.
-# In most cased, endpoint name should be detected by agents or service mesh automatically, and aggregate the metrics based
-# on the name.
-# But, in some cases, application puts the parameter in the endpoint name, such as putting order id in the URI, like
-# /prod/ORDER123, /prod/ORDER456.
-# This grouping file provides the regex based definition capability to merge those endpoints into a group by better and
-# more meaningful aggregation metrics.
-
-grouping:
- # Endpoint of the service would follow the following rules
- - service-name: serviceA
- rules:
- - endpoint-name: /prod/{id}
- regex: \/prod\/.+
\ No newline at end of file
+org.apache.skywalking.oap.server.ai.pipeline.AIPipelineProvider
\ No newline at end of file
diff --git a/oap-server/microbench/src/main/java/org/apache/skywalking/oap/server/microbench/core/config/group/uri/RegexVSQuickMatchBenchmark.java b/oap-server/microbench/src/main/java/org/apache/skywalking/oap/server/microbench/core/config/group/uri/RegexVSQuickMatchBenchmark.java
new file mode 100644
index 0000000000..ce0165abeb
--- /dev/null
+++ b/oap-server/microbench/src/main/java/org/apache/skywalking/oap/server/microbench/core/config/group/uri/RegexVSQuickMatchBenchmark.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.microbench.core.config.group.uri;
+
+import org.apache.skywalking.oap.server.core.config.group.EndpointGroupingRule;
+import org.apache.skywalking.oap.server.core.config.group.uri.quickmatch.QuickUriGroupingRule;
+import org.apache.skywalking.oap.server.library.util.StringFormatGroup;
+import org.apache.skywalking.oap.server.microbench.base.AbstractMicrobenchmark;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+
+@Warmup(iterations = 1)
+@Measurement(iterations = 1)
+@Fork(1)
+@State(Scope.Thread)
+@BenchmarkMode({Mode.Throughput})
+@Threads(4)
+public class RegexVSQuickMatchBenchmark extends AbstractMicrobenchmark {
+
+ @State(Scope.Benchmark)
+ public static class RegexMatch {
+ private final EndpointGroupingRule rule = new EndpointGroupingRule();
+
+ public RegexMatch() {
+ rule.addRule("service1", "/products/{var}", "/products/.+");
+ rule.addRule("service1", "/products/{var}/detail", "/products/.+/detail");
+ rule.addRule("service1", "/sales/{var}/1", "/sales/.+/1");
+ rule.addRule("service1", "/sales/{var}/2", "/sales/.+/2");
+ rule.addRule("service1", "/sales/{var}/3", "/sales/.+/3");
+ rule.addRule("service1", "/sales/{var}/4", "/sales/.+/4");
+ rule.addRule("service1", "/sales/{var}/5", "/sales/.+/5");
+ rule.addRule("service1", "/sales/{var}/6", "/sales/.+/6");
+ rule.addRule("service1", "/sales/{var}/7", "/sales/.+/7");
+ rule.addRule("service1", "/sales/{var}/8", "/sales/.+/8");
+ rule.addRule("service1", "/sales/{var}/9", "/sales/.+/9");
+ rule.addRule("service1", "/sales/{var}/10", "/sales/.+/10");
+ rule.addRule("service1", "/sales/{var}/11", "/sales/.+/11");
+ rule.addRule("service1", "/employees/{var}/profile", "/employees/.+/profile");
+ }
+
+ public StringFormatGroup.FormatResult match(String serviceName, String endpointName) {
+ return rule.format(serviceName, endpointName);
+ }
+ }
+
+ @State(Scope.Benchmark)
+ public static class QuickMatch {
+ private final QuickUriGroupingRule rule = new QuickUriGroupingRule();
+
+ public QuickMatch() {
+ rule.addRule("service1", "/products/{var}");
+ rule.addRule("service1", "/products/{var}/detail");
+ rule.addRule("service1", "/sales/{var}/1");
+ rule.addRule("service1", "/sales/{var}/2");
+ rule.addRule("service1", "/sales/{var}/3");
+ rule.addRule("service1", "/sales/{var}/4");
+ rule.addRule("service1", "/sales/{var}/5");
+ rule.addRule("service1", "/sales/{var}/6");
+ rule.addRule("service1", "/sales/{var}/7");
+ rule.addRule("service1", "/sales/{var}/8");
+ rule.addRule("service1", "/sales/{var}/9");
+ rule.addRule("service1", "/sales/{var}/10");
+ rule.addRule("service1", "/sales/{var}/11");
+ rule.addRule("service1", "/employees/{var}/profile");
+ }
+
+ public StringFormatGroup.FormatResult match(String serviceName, String endpointName) {
+ return rule.format(serviceName, endpointName);
+ }
+ }
+
+ @Benchmark
+ public void matchFirstRegex(Blackhole bh, RegexVSQuickMatchBenchmark.RegexMatch formatClass) {
+ bh.consume(formatClass.match("service1", "/products/123"));
+ }
+
+ @Benchmark
+ public void matchFirstQuickUriGrouping(Blackhole bh, RegexVSQuickMatchBenchmark.QuickMatch formatClass) {
+ bh.consume(formatClass.match("service1", "/products/123"));
+ }
+
+ @Benchmark
+ public void notMatchRegex(Blackhole bh, RegexVSQuickMatchBenchmark.RegexMatch formatClass) {
+ bh.consume(formatClass.match("service1", "/employees/123"));
+ }
+
+ @Benchmark
+ public void notMatchQuickUriGrouping(Blackhole bh, RegexVSQuickMatchBenchmark.QuickMatch formatClass) {
+ bh.consume(formatClass.match("service1", "/employees/123"));
+ }
+}
+
+/**
+ * # JMH version: 1.25
+ * # VM version: JDK 11.0.18, OpenJDK 64-Bit Server VM, 11.0.18+10
+ * # VM invoker: /Users/wusheng/Library/Java/JavaVirtualMachines/temurin-11.0.18/Contents/Home/bin/java
+ * # VM options: -ea --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED -Didea.test.cyclic.buffer.size=1048576 -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=53714:/Applications/IntelliJ IDEA.app/Contents/bin -Dfile.encoding=UTF-8 -Xmx512m -Xms512m -XX:MaxDirectMemorySize=512m -XX:BiasedLockingStartupDelay=0 -Djmh.executor=CUSTOM -Djmh.executor.class=org.apache.skywalking.oap.server.microbench.base.AbstractMicrobenchmark$JmhThread [...]
+ * # Warmup: 1 iterations, 10 s each
+ * # Measurement: 1 iterations, 10 s each
+ * # Timeout: 10 min per iteration
+ * # Threads: 4 threads, will synchronize iterations
+ * # Benchmark mode: Throughput, ops/time
+ * # Benchmark: org.apache.skywalking.oap.server.microbench.core.config.group.uri.RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping
+ *
+ Benchmark Mode Cnt Score Error Units
+ RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping thrpt 21824009.131 ops/s
+ RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping:·gc.alloc.rate thrpt 5388.565 MB/sec
+ RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping:·gc.alloc.rate.norm thrpt 272.000 B/op
+ RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping:·gc.churn.G1_Eden_Space thrpt 5414.783 MB/sec
+ RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping:·gc.churn.G1_Eden_Space.norm thrpt 273.323 B/op
+ RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping:·gc.churn.G1_Old_Gen thrpt 0.019 MB/sec
+ RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping:·gc.churn.G1_Old_Gen.norm thrpt 0.001 B/op
+ RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping:·gc.count thrpt 186.000 counts
+ RegexVSQuickMatchBenchmark.matchFirstQuickUriGrouping:·gc.time thrpt 112.000 ms
+ RegexVSQuickMatchBenchmark.matchFirstRegex thrpt 52815735.190 ops/s
+ RegexVSQuickMatchBenchmark.matchFirstRegex:·gc.alloc.rate thrpt 7286.910 MB/sec
+ RegexVSQuickMatchBenchmark.matchFirstRegex:·gc.alloc.rate.norm thrpt 152.000 B/op
+ RegexVSQuickMatchBenchmark.matchFirstRegex:·gc.churn.G1_Eden_Space thrpt 7306.492 MB/sec
+ RegexVSQuickMatchBenchmark.matchFirstRegex:·gc.churn.G1_Eden_Space.norm thrpt 152.408 B/op
+ RegexVSQuickMatchBenchmark.matchFirstRegex:·gc.churn.G1_Old_Gen thrpt 0.026 MB/sec
+ RegexVSQuickMatchBenchmark.matchFirstRegex:·gc.churn.G1_Old_Gen.norm thrpt 0.001 B/op
+ RegexVSQuickMatchBenchmark.matchFirstRegex:·gc.count thrpt 251.000 counts
+ RegexVSQuickMatchBenchmark.matchFirstRegex:·gc.time thrpt 143.000 ms
+ RegexVSQuickMatchBenchmark.notMatchQuickUriGrouping thrpt 23265119.660 ops/s
+ RegexVSQuickMatchBenchmark.notMatchQuickUriGrouping:·gc.alloc.rate thrpt 5914.434 MB/sec
+ RegexVSQuickMatchBenchmark.notMatchQuickUriGrouping:·gc.alloc.rate.norm thrpt 280.000 B/op
+ RegexVSQuickMatchBenchmark.notMatchQuickUriGrouping:·gc.churn.G1_Eden_Space thrpt 5939.872 MB/sec
+ RegexVSQuickMatchBenchmark.notMatchQuickUriGrouping:·gc.churn.G1_Eden_Space.norm thrpt 281.204 B/op
+ RegexVSQuickMatchBenchmark.notMatchQuickUriGrouping:·gc.churn.G1_Old_Gen thrpt 0.025 MB/sec
+ RegexVSQuickMatchBenchmark.notMatchQuickUriGrouping:·gc.churn.G1_Old_Gen.norm thrpt 0.001 B/op
+ RegexVSQuickMatchBenchmark.notMatchQuickUriGrouping:·gc.count thrpt 204.000 counts
+ RegexVSQuickMatchBenchmark.notMatchQuickUriGrouping:·gc.time thrpt 108.000 ms
+ RegexVSQuickMatchBenchmark.notMatchRegex thrpt 10462244.647 ops/s
+ RegexVSQuickMatchBenchmark.notMatchRegex:·gc.alloc.rate thrpt 26817.565 MB/sec
+ RegexVSQuickMatchBenchmark.notMatchRegex:·gc.alloc.rate.norm thrpt 2824.000 B/op
+ RegexVSQuickMatchBenchmark.notMatchRegex:·gc.churn.G1_Eden_Space thrpt 26925.722 MB/sec
+ RegexVSQuickMatchBenchmark.notMatchRegex:·gc.churn.G1_Eden_Space.norm thrpt 2835.389 B/op
+ RegexVSQuickMatchBenchmark.notMatchRegex:·gc.churn.G1_Old_Gen thrpt 0.128 MB/sec
+ RegexVSQuickMatchBenchmark.notMatchRegex:·gc.churn.G1_Old_Gen.norm thrpt 0.013 B/op
+ RegexVSQuickMatchBenchmark.notMatchRegex:·gc.count thrpt 925.000 counts
+ RegexVSQuickMatchBenchmark.notMatchRegex:·gc.time thrpt 507.000 ms
+ */
\ No newline at end of file
diff --git a/oap-server/pom.xml b/oap-server/pom.xml
index 93bc812d5b..bd2d72bcf1 100755
--- a/oap-server/pom.xml
+++ b/oap-server/pom.xml
@@ -47,6 +47,7 @@
<module>server-fetcher-plugin</module>
<module>server-health-checker</module>
<module>microbench</module>
+ <module>ai-pipeline</module>
</modules>
<properties>
diff --git a/oap-server/server-core/pom.xml b/oap-server/server-core/pom.xml
index 30bb2ee8c5..769c76517d 100644
--- a/oap-server/server-core/pom.xml
+++ b/oap-server/server-core/pom.xml
@@ -97,6 +97,11 @@
<groupId>io.zipkin.zipkin2</groupId>
<artifactId>zipkin</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>ai-pipeline</artifactId>
+ <version>${project.version}</version>
+ </dependency>
</dependencies>
<build>
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleConfig.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleConfig.java
index de7081f81c..2f7f74fdb3 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleConfig.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleConfig.java
@@ -194,6 +194,11 @@ public class CoreModuleConfig extends ModuleConfig {
*/
private int httpMaxRequestHeaderSize = 8192;
+ /**
+ * The max number of HTTP URIs per service for further URI pattern recognition.
+ */
+ private int maxHttpUrisNumberPerService = 3000;
+
public CoreModuleConfig() {
this.downsampling = new ArrayList<>();
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java
index 47f9a3bdb3..0223213c75 100755
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java
@@ -20,6 +20,8 @@ package org.apache.skywalking.oap.server.core;
import java.io.FileNotFoundException;
import java.io.IOException;
+import org.apache.skywalking.oap.server.ai.pipeline.AIPipelineModule;
+import org.apache.skywalking.oap.server.ai.pipeline.services.api.HttpUriRecognition;
import org.apache.skywalking.oap.server.configuration.api.ConfigurationModule;
import org.apache.skywalking.oap.server.configuration.api.DynamicConfigurationService;
import org.apache.skywalking.oap.server.core.analysis.ApdexThresholdConfig;
@@ -127,6 +129,7 @@ public class CoreModuleProvider extends ModuleProvider {
private OALEngineLoaderService oalEngineLoaderService;
private LoggingConfigWatcher loggingConfigWatcher;
private EndpointNameGroupingRule4OpenapiWatcher endpointNameGroupingRule4OpenapiWatcher;
+ private EndpointNameGrouping endpointNameGrouping;
public CoreModuleProvider() {
super();
@@ -165,7 +168,7 @@ public class CoreModuleProvider extends ModuleProvider {
if (moduleConfig.isActiveExtraModelColumns()) {
DefaultScopeDefine.activeExtraModelColumns();
}
- EndpointNameGrouping endpointNameGrouping = new EndpointNameGrouping();
+ endpointNameGrouping = new EndpointNameGrouping();
final NamingControl namingControl = new NamingControl(
moduleConfig.getServiceNameMaxLength(),
moduleConfig.getInstanceNameMaxLength(),
@@ -348,6 +351,15 @@ public class CoreModuleProvider extends ModuleProvider {
grpcServer.addHandler(new RemoteServiceHandler(getManager()));
grpcServer.addHandler(new HealthCheckServiceHandler());
+ endpointNameGrouping.startHttpUriRecognitionSvr(
+ getManager()
+ .find(AIPipelineModule.NAME)
+ .provider()
+ .getService(HttpUriRecognition.class),
+ getService(MetadataQueryService.class),
+ moduleConfig.getMaxHttpUrisNumberPerService()
+ );
+
// Disable OAL script has higher priority
oalEngineLoaderService.load(DisableOALDefine.INSTANCE);
@@ -419,7 +431,8 @@ public class CoreModuleProvider extends ModuleProvider {
public String[] requiredModules() {
return new String[] {
TelemetryModule.NAME,
- ConfigurationModule.NAME
+ ConfigurationModule.NAME,
+ AIPipelineModule.NAME
};
}
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/NamingControl.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/NamingControl.java
index 9f21d2eeff..9a3bbf6784 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/NamingControl.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/NamingControl.java
@@ -20,9 +20,9 @@ package org.apache.skywalking.oap.server.core.config;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
-import org.apache.skywalking.oap.server.library.util.StringUtil;
import org.apache.skywalking.oap.server.core.config.group.EndpointNameGrouping;
import org.apache.skywalking.oap.server.library.module.Service;
+import org.apache.skywalking.oap.server.library.util.StringUtil;
/**
* NamingControl provides the service to make the names of service, instance and endpoint following the rules or
@@ -113,6 +113,6 @@ public class NamingControl implements Service {
}
}
- return endpointNameGrouping.format(serviceName, lengthControlledName);
+ return endpointNameGrouping.format(serviceName, lengthControlledName)._1();
}
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRule.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRule.java
index d1fdb985ac..f239d6b38a 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRule.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRule.java
@@ -20,11 +20,14 @@ package org.apache.skywalking.oap.server.core.config.group;
import java.util.HashMap;
import java.util.Map;
+import org.apache.skywalking.oap.server.core.config.group.uri.quickmatch.QuickUriGroupingRule;
import org.apache.skywalking.oap.server.library.util.StringFormatGroup;
/**
* Endpoint group rule hosts all group rules of all services.
+ * Regex grouping has been replaced by {@link QuickUriGroupingRule}
*/
+@Deprecated
public class EndpointGroupingRule {
private Map<String, StringFormatGroup> rules = new HashMap<>();
@@ -35,7 +38,7 @@ public class EndpointGroupingRule {
* @param endpointGroupName represents the logic endpoint name.
* @param ruleRegex match the endpoints which should be in the group name.
*/
- void addRule(String serviceName, String endpointGroupName, String ruleRegex) {
+ public void addRule(String serviceName, String endpointGroupName, String ruleRegex) {
final StringFormatGroup formatGroup = rules.computeIfAbsent(serviceName, name -> new StringFormatGroup());
formatGroup.addRule(endpointGroupName, ruleRegex);
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRuleReader.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRuleReader.java
index a7bc86a67b..2f96668976 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRuleReader.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRuleReader.java
@@ -23,6 +23,7 @@ import java.io.Reader;
import java.util.List;
import java.util.Map;
import java.util.Objects;
+import org.apache.skywalking.oap.server.core.config.group.uri.quickmatch.QuickUriGroupingRule;
import org.apache.skywalking.oap.server.library.util.StringUtil;
import org.yaml.snakeyaml.LoaderOptions;
import org.yaml.snakeyaml.Yaml;
@@ -47,8 +48,8 @@ public class EndpointGroupingRuleReader {
/**
* @return the loaded rules.
*/
- EndpointGroupingRule read() {
- EndpointGroupingRule endpointGroupingRule = new EndpointGroupingRule();
+ QuickUriGroupingRule read() {
+ QuickUriGroupingRule quickUriGroupingRule = new QuickUriGroupingRule();
if (Objects.nonNull(yamlData)) {
List rulesData = (List) yamlData.get("grouping");
@@ -62,19 +63,17 @@ public class EndpointGroupingRuleReader {
final List endpointRules = (List) rule.get("rules");
if (endpointRules != null) {
endpointRules.forEach(endpointRuleObj -> {
- final Map endpointRule = (Map) endpointRuleObj;
- final String endpointLogicGroupName = (String) endpointRule.get("endpoint-name");
- final String groupRegex = (String) endpointRule.get("regex");
- if (StringUtil.isEmpty(endpointLogicGroupName) || StringUtil.isEmpty(groupRegex)) {
+ final String pattern = (String) endpointRuleObj;
+ if (StringUtil.isEmpty(pattern)) {
return;
}
- endpointGroupingRule.addRule(serviceName, endpointLogicGroupName, groupRegex);
+ quickUriGroupingRule.addRule(serviceName, pattern);
});
}
});
}
}
- return endpointGroupingRule;
+ return quickUriGroupingRule;
}
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGrouping.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGrouping.java
index 8456990293..85ea7b0dde 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGrouping.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGrouping.java
@@ -18,32 +18,92 @@
package org.apache.skywalking.oap.server.core.config.group;
+import io.vavr.Tuple2;
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
-import org.apache.skywalking.oap.server.library.util.StringFormatGroup;
+import org.apache.skywalking.oap.server.ai.pipeline.services.api.HttpUriPattern;
+import org.apache.skywalking.oap.server.ai.pipeline.services.api.HttpUriRecognition;
import org.apache.skywalking.oap.server.core.config.group.openapi.EndpointGroupingRule4Openapi;
+import org.apache.skywalking.oap.server.core.config.group.uri.quickmatch.QuickUriGroupingRule;
+import org.apache.skywalking.oap.server.core.query.MetadataQueryService;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.apache.skywalking.oap.server.library.util.RunnableWithExceptionProtection;
+import org.apache.skywalking.oap.server.library.util.StringFormatGroup;
@Slf4j
public class EndpointNameGrouping {
@Setter
- private volatile EndpointGroupingRule endpointGroupingRule;
+ private volatile QuickUriGroupingRule endpointGroupingRule;
@Setter
private volatile EndpointGroupingRule4Openapi endpointGroupingRule4Openapi;
+ @Setter
+ private volatile QuickUriGroupingRule quickUriGroupingRule;
+ /**
+ * Cache the HTTP URIs which are not formatted by the rules per service.
+ * Level one map key is service name, the value is a map of HTTP URI and its count.
+ * Multiple matches will be counted, because it is the pattern that the endpoint is already formatted,
+ * or doesn't need to be formatted.
+ * The repeatable URI is a pattern already.
+ */
+ private ConcurrentHashMap<String, ConcurrentHashMap<String, AtomicInteger>> cachedHttpUris = new ConcurrentHashMap<>();
+ private final AtomicInteger aiPipelineExecutionCounter = new AtomicInteger(0);
+ /**
+ * The max number of HTTP URIs per service for further URI pattern recognition.
+ */
+ private int maxHttpUrisNumberPerService = 3000;
- public String format(String serviceName, String endpointName) {
- String formattedName = endpointName;
+ /**
+ * Format the endpoint name according to the API patterns.
+ *
+ * @param serviceName service name
+ * @param endpointName endpoint name to be formatted.
+ * @return Tuple2 where the first element is the formatted name and the second element is a boolean
+ * represented the endpoint name is formatted or not.
+ */
+ public Tuple2<String, Boolean> format(String serviceName, String endpointName) {
+ Tuple2<String, Boolean> formattedName = new Tuple2<>(endpointName, Boolean.FALSE);
if (endpointGroupingRule4Openapi != null) {
- formattedName = formatByOpenapi(serviceName, formattedName);
+ formattedName = formatByOpenapi(serviceName, endpointName);
+ }
+
+ if (!formattedName._2() && endpointGroupingRule != null) {
+ formattedName = formatByCustom(serviceName, endpointName);
+ }
+
+ if (!formattedName._2() && quickUriGroupingRule != null) {
+ formattedName = formatByQuickUriPattern(serviceName, endpointName);
}
- if (endpointGroupingRule != null) {
- formattedName = formatByCustom(serviceName, formattedName);
+ if (!formattedName._2()) {
+ // Only URI includes '/' will be cached and formatted later.
+ // Otherwise it could be anything, and has no need to be formatted.
+ if (endpointName.indexOf("/") > -1) {
+ ConcurrentHashMap<String, AtomicInteger> svrHttpUris = cachedHttpUris.get(serviceName);
+ if (svrHttpUris == null) {
+ cachedHttpUris.putIfAbsent(serviceName, new ConcurrentHashMap<>());
+ svrHttpUris = cachedHttpUris.get(serviceName);
+ }
+ // Only cache first 2000 URIs per 30 mins.
+ if (svrHttpUris.size() < maxHttpUrisNumberPerService) {
+ final AtomicInteger cachedCount = svrHttpUris.putIfAbsent(endpointName, new AtomicInteger(1));
+ if (null != cachedCount) {
+ cachedCount.incrementAndGet();
+ }
+ }
+ }
}
return formattedName;
}
- private String formatByCustom(String serviceName, String endpointName) {
+ private Tuple2<String, Boolean> formatByCustom(String serviceName, String endpointName) {
final StringFormatGroup.FormatResult formatResult = endpointGroupingRule.format(serviceName, endpointName);
if (log.isDebugEnabled() || log.isTraceEnabled()) {
if (formatResult.isMatch()) {
@@ -54,10 +114,10 @@ public class EndpointNameGrouping {
log.trace("Endpoint {} of Service {} keeps unchanged.", endpointName, serviceName);
}
}
- return formatResult.getName();
+ return new Tuple2<>(formatResult.getReplacedName(), formatResult.isMatch());
}
- private String formatByOpenapi(String serviceName, String endpointName) {
+ private Tuple2<String, Boolean> formatByOpenapi(String serviceName, String endpointName) {
final StringFormatGroup.FormatResult formatResult = endpointGroupingRule4Openapi.format(
serviceName, endpointName);
if (log.isDebugEnabled() || log.isTraceEnabled()) {
@@ -69,6 +129,77 @@ public class EndpointNameGrouping {
log.trace("Endpoint {} of Service {} keeps unchanged.", endpointName, serviceName);
}
}
- return formatResult.getName();
+ return new Tuple2<>(formatResult.getName(), formatResult.isMatch());
+ }
+
+ private Tuple2<String, Boolean> formatByQuickUriPattern(String serviceName, String endpointName) {
+ final StringFormatGroup.FormatResult formatResult = quickUriGroupingRule.format(serviceName, endpointName);
+ if (log.isDebugEnabled() || log.isTraceEnabled()) {
+ if (formatResult.isMatch()) {
+ log.debug(
+ "Endpoint {} of Service {} has been renamed in group {} by AI/ML-powered URI pattern recognition",
+ endpointName, serviceName, formatResult.getName()
+ );
+ } else {
+ log.trace("Endpoint {} of Service {} keeps unchanged.", endpointName, serviceName);
+ }
+ }
+ return new Tuple2<>(formatResult.getName(), formatResult.isMatch());
+ }
+
+ public void startHttpUriRecognitionSvr(final HttpUriRecognition httpUriRecognitionSvr,
+ final MetadataQueryService metadataQueryService,
+ int maxEndpointCandidatePerSvr) {
+ this.maxHttpUrisNumberPerService = maxEndpointCandidatePerSvr;
+ if (!httpUriRecognitionSvr.isInitialized()) {
+ return;
+ }
+ this.quickUriGroupingRule = new QuickUriGroupingRule();
+ Executors.newSingleThreadScheduledExecutor()
+ .scheduleWithFixedDelay(
+ new RunnableWithExceptionProtection(
+ () -> {
+ if (aiPipelineExecutionCounter.incrementAndGet() % 30 == 0) {
+ // Send the cached URIs to the recognition server per 30 mins to build new patterns.
+ cachedHttpUris.forEach((serviceName, httpUris) -> {
+ List<HttpUriRecognition.HTTPUri> uris
+ = httpUris.keySet()
+ .stream()
+ .map(
+ uri -> new HttpUriRecognition.HTTPUri(
+ uri, httpUris.get(uri).get()
+ ))
+ .collect(Collectors.toList());
+ // Reset the cache once the URIs are sent to the recognition server.
+ httpUris.clear();
+ httpUriRecognitionSvr.feedRawData(serviceName, uris);
+ });
+ } else {
+ // Sync with the recognition server per 1 min to get the latest patterns.
+ try {
+ metadataQueryService.listServices(null, null).forEach(
+ service -> {
+ final List<HttpUriPattern> patterns
+ = httpUriRecognitionSvr.fetchAllPatterns(service.getName());
+ if (CollectionUtils.isNotEmpty(patterns)) {
+ StringFormatGroup group = new StringFormatGroup(
+ patterns.size());
+ patterns.forEach(
+ p -> quickUriGroupingRule.addRule(
+ service.getName(), p.getPattern()));
+
+ }
+ }
+ );
+ } catch (IOException e) {
+ log.error("Fail to load all services.", e);
+ }
+
+ }
+ },
+ t -> log.error("Fail to recognize URI patterns.", t)
+ ), 1, 1, TimeUnit.MINUTES
+ );
+
}
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGroupingRuleWatcher.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGroupingRuleWatcher.java
index d2addf82eb..8ee6a134b1 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGroupingRuleWatcher.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGroupingRuleWatcher.java
@@ -23,6 +23,7 @@ import java.io.StringReader;
import lombok.extern.slf4j.Slf4j;
import org.apache.skywalking.oap.server.configuration.api.ConfigChangeWatcher;
import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.config.group.uri.quickmatch.QuickUriGroupingRule;
import org.apache.skywalking.oap.server.library.module.ModuleProvider;
import org.apache.skywalking.oap.server.library.util.ResourceUtils;
@@ -48,7 +49,7 @@ public class EndpointNameGroupingRuleWatcher extends ConfigChangeWatcher {
public void notify(final ConfigChangeEvent value) {
if (value.getEventType().equals(EventType.DELETE)) {
ruleSetting = null;
- grouping.setEndpointGroupingRule(new EndpointGroupingRule());
+ grouping.setEndpointGroupingRule(new QuickUriGroupingRule());
} else {
ruleSetting = value.getNewValue();
grouping.setEndpointGroupingRule(new EndpointGroupingRuleReader(new StringReader(ruleSetting)).read());
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/PatternToken.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/PatternToken.java
new file mode 100644
index 0000000000..313bf3cae9
--- /dev/null
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/PatternToken.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.config.group.uri.quickmatch;
+
+import java.util.List;
+
+public interface PatternToken {
+ boolean isMatch(String name);
+
+ boolean isLeaf();
+
+ String expression();
+
+ void setExpression(String expression);
+
+ List<PatternToken> children();
+
+ default PatternToken find(final PatternToken token) {
+ for (final PatternToken child : children()) {
+ if (child.equals(token)) {
+ return child;
+ }
+ }
+ return null;
+ }
+
+ default PatternToken add(final PatternToken token) {
+ children().add(token);
+ return token;
+ }
+}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/PatternTree.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/PatternTree.java
new file mode 100644
index 0000000000..a930ed11d3
--- /dev/null
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/PatternTree.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.config.group.uri.quickmatch;
+
+import java.util.ArrayList;
+import java.util.List;
+import lombok.EqualsAndHashCode;
+import lombok.ToString;
+import org.apache.skywalking.oap.server.library.util.StringFormatGroup;
+
+@EqualsAndHashCode
+@ToString
+public class PatternTree {
+ private final List<PatternToken> roots;
+
+ public PatternTree() {
+ roots = new ArrayList<>();
+ }
+
+ /**
+ * The pattern is split by /, and each token is a node in the tree. Each node either a literal string or "{var}"
+ * representing a variable.
+ *
+ * @param pattern of URIs
+ */
+ public void addPattern(String pattern) {
+ final String[] tokens = pattern.split("/");
+
+ PatternToken current = null;
+ for (final PatternToken patternToken : roots) {
+ if (patternToken.isMatch(tokens[0])) {
+ current = patternToken;
+ break;
+ }
+ }
+
+ if (current == null) {
+ current = new StringToken(tokens[0]);
+ roots.add(current);
+ }
+
+ if (tokens.length == 1) {
+ current.setExpression(pattern);
+ return;
+ }
+
+ for (int i = 1; i < tokens.length; i++) {
+ final String token = tokens[i];
+ PatternToken newToken;
+ if (VarToken.VAR_TOKEN.equals(token)) {
+ newToken = new VarToken();
+ } else {
+ newToken = new StringToken(token);
+ }
+ final PatternToken found = current.find(newToken);
+ if (found == null) {
+ current = current.add(newToken);
+ } else {
+ current = found;
+ }
+ }
+ current.setExpression(pattern);
+ }
+
+ public StringFormatGroup.FormatResult match(String uri) {
+ final String[] slices = uri.split("/");
+ List<PatternToken> current = roots;
+ PatternToken matchedToken = null;
+ for (final String slice : slices) {
+ boolean matched = false;
+ for (final PatternToken patternToken : current) {
+ if (patternToken.isMatch(slice)) {
+ matchedToken = patternToken;
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ return new StringFormatGroup.FormatResult(false, uri, null);
+ }
+ current = matchedToken.children();
+ }
+ if (matchedToken.isLeaf()) {
+ return new StringFormatGroup.FormatResult(true, uri, matchedToken.expression());
+ } else {
+ return new StringFormatGroup.FormatResult(false, uri, null);
+ }
+ }
+}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRule.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/QuickUriGroupingRule.java
similarity index 50%
copy from oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRule.java
copy to oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/QuickUriGroupingRule.java
index d1fdb985ac..10cf4df043 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRule.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/QuickUriGroupingRule.java
@@ -16,39 +16,24 @@
*
*/
-package org.apache.skywalking.oap.server.core.config.group;
+package org.apache.skywalking.oap.server.core.config.group.uri.quickmatch;
import java.util.HashMap;
import java.util.Map;
import org.apache.skywalking.oap.server.library.util.StringFormatGroup;
-/**
- * Endpoint group rule hosts all group rules of all services.
- */
-public class EndpointGroupingRule {
- private Map<String, StringFormatGroup> rules = new HashMap<>();
+public class QuickUriGroupingRule {
+ private Map<String, PatternTree> rules = new HashMap<>();
- /**
- * Add a new rule to the context.
- *
- * @param serviceName of the new rule
- * @param endpointGroupName represents the logic endpoint name.
- * @param ruleRegex match the endpoints which should be in the group name.
- */
- void addRule(String serviceName, String endpointGroupName, String ruleRegex) {
- final StringFormatGroup formatGroup = rules.computeIfAbsent(serviceName, name -> new StringFormatGroup());
- formatGroup.addRule(endpointGroupName, ruleRegex);
+ public void addRule(String serviceName, String pattern) {
+ final PatternTree patternTree = rules.computeIfAbsent(serviceName, name -> new PatternTree());
+ patternTree.addPattern(pattern);
}
- /**
- * @param service of the given endpoint belonged.
- * @param endpointName to do group checking.
- * @return group result and new endpoint name if rule matched.
- */
public StringFormatGroup.FormatResult format(String service, String endpointName) {
- final StringFormatGroup stringFormatGroup = rules.get(service);
- if (stringFormatGroup != null) {
- return stringFormatGroup.format(endpointName);
+ final PatternTree patternTree = rules.get(service);
+ if (patternTree != null) {
+ return patternTree.match(endpointName);
} else {
return new StringFormatGroup.FormatResult(false, endpointName, endpointName);
}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/StringToken.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/StringToken.java
new file mode 100644
index 0000000000..350502ae68
--- /dev/null
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/StringToken.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.config.group.uri.quickmatch;
+
+import java.util.ArrayList;
+import java.util.List;
+import lombok.EqualsAndHashCode;
+import lombok.Setter;
+import lombok.ToString;
+
+@EqualsAndHashCode(of = "value")
+@ToString
+public class StringToken implements PatternToken {
+ private final String value;
+ private final List<PatternToken> children;
+ @Setter
+ private String expression;
+
+ public StringToken(final String value) {
+ this.value = value;
+ children = new ArrayList<>();
+ }
+
+ @Override
+ public boolean isMatch(final String name) {
+ return value.equals(name);
+ }
+
+ @Override
+ public boolean isLeaf() {
+ return expression != null;
+ }
+
+ @Override
+ public String expression() {
+ return expression;
+ }
+
+ @Override
+ public List<PatternToken> children() {
+ return children;
+ }
+}
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/VarToken.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/VarToken.java
new file mode 100644
index 0000000000..c668fd5ae9
--- /dev/null
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/VarToken.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.config.group.uri.quickmatch;
+
+import java.util.ArrayList;
+import java.util.List;
+import lombok.EqualsAndHashCode;
+import lombok.Setter;
+import lombok.ToString;
+
+@EqualsAndHashCode(of = "")
+@ToString
+public class VarToken implements PatternToken {
+ public static final String VAR_TOKEN = "{var}";
+ private List<PatternToken> children = new ArrayList<>();
+ @Setter
+ private String expression;
+
+ /**
+ * @return TRUE for any input string.
+ */
+ @Override
+ public boolean isMatch(final String name) {
+ return true;
+ }
+
+ @Override
+ public boolean isLeaf() {
+ return expression != null;
+ }
+
+ @Override
+ public String expression() {
+ return expression;
+ }
+
+ @Override
+ public List<PatternToken> children() {
+ return children;
+ }
+}
diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRuleReaderTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRuleReaderTest.java
index 56d56742bc..d07abfe702 100644
--- a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRuleReaderTest.java
+++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/EndpointGroupingRuleReaderTest.java
@@ -18,6 +18,7 @@
package org.apache.skywalking.oap.server.core.config.group;
+import org.apache.skywalking.oap.server.core.config.group.uri.quickmatch.QuickUriGroupingRule;
import org.apache.skywalking.oap.server.library.util.StringFormatGroup;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -29,11 +30,11 @@ public class EndpointGroupingRuleReaderTest {
.getClassLoader()
.getResourceAsStream(
"endpoint-name-grouping.yml"));
- final EndpointGroupingRule rule = reader.read();
+ final QuickUriGroupingRule rule = reader.read();
StringFormatGroup.FormatResult formatResult = rule.format("serviceA", "/prod/123");
Assertions.assertTrue(formatResult.isMatch());
- Assertions.assertEquals("/prod/{id}", formatResult.getName());
+ Assertions.assertEquals("/prod/{var}", formatResult.getReplacedName());
formatResult = rule.format("serviceA", "/prod/");
Assertions.assertFalse(formatResult.isMatch());
diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGroupingRuleWatcherTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGroupingRuleWatcherTest.java
index ddf3a10087..1814d49fbb 100644
--- a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGroupingRuleWatcherTest.java
+++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGroupingRuleWatcherTest.java
@@ -71,44 +71,21 @@ public class EndpointNameGroupingRuleWatcherTest {
return new String[0];
}
}, endpointNameGrouping);
- Assertions.assertEquals("/prod/{id}", endpointNameGrouping.format("serviceA", "/prod/123"));
+ Assertions.assertEquals("/prod/{var}", endpointNameGrouping.format("serviceA", "/prod/123")._1());
watcher.notify(new ConfigChangeWatcher.ConfigChangeEvent(
- "# Licensed to the Apache Software Foundation (ASF) under one or more\n" +
- "# contributor license agreements. See the NOTICE file distributed with\n" +
- "# this work for additional information regarding copyright ownership.\n" +
- "# The ASF licenses this file to You under the Apache License, Version 2.0\n" +
- "# (the \"License\"); you may not use this file except in compliance with\n" +
- "# the License. You may obtain a copy of the License at\n" +
- "#\n" +
- "# http://www.apache.org/licenses/LICENSE-2.0\n" +
- "#\n" +
- "# Unless required by applicable law or agreed to in writing, software\n" +
- "# distributed under the License is distributed on an \"AS IS\" BASIS,\n" +
- "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n" +
- "# See the License for the specific language governing permissions and\n" +
- "# limitations under the License.\n" +
- "\n" +
- "# Endpoint name grouping rules.\n" +
- "# In most cased, endpoint name should be detected by agents or service mesh automatically, and aggregate the metrics based\n" +
- "# on the name.\n" +
- "# But, in some cases, application put the parameter in the endpoint name, such as putting order id in the URI, like\n" +
- "# /prod/ORDER123, /prod/ORDER123.\n" +
- "# This grouping file provides the regex based definition capability to merge those endpoints into a group by better and\n" +
- "# more meaningful aggregation metrics.\n" +
- "\n" +
"grouping:\n" +
" # Endpoint of the service would follow the following rules\n" +
" - service-name: serviceA\n" +
" rules:\n" +
- " - endpoint-name: /prod/order-id\n" +
- " regex: \\/prod\\/.+"
+ " - /prod/{var}\n" +
+ " - /prod/{var}/info\n"
, ConfigChangeWatcher.EventType.MODIFY
));
- Assertions.assertEquals("/prod/order-id", endpointNameGrouping.format("serviceA", "/prod/123"));
+ Assertions.assertEquals("/prod/{var}/info", endpointNameGrouping.format("serviceA", "/prod/123/info")._1());
watcher.notify(new ConfigChangeWatcher.ConfigChangeEvent("", ConfigChangeWatcher.EventType.DELETE));
- Assertions.assertEquals("/prod/123", endpointNameGrouping.format("serviceA", "/prod/123"));
+ Assertions.assertEquals("/prod/123", endpointNameGrouping.format("serviceA", "/prod/123")._1());
}
}
diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/openapi/EndpointGroupingRuleReader4OpenapiTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/openapi/EndpointGroupingRuleReader4OpenapiTest.java
index 34dc36b84b..8a4d504d3a 100644
--- a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/openapi/EndpointGroupingRuleReader4OpenapiTest.java
+++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/openapi/EndpointGroupingRuleReader4OpenapiTest.java
@@ -36,62 +36,62 @@ public class EndpointGroupingRuleReader4OpenapiTest {
//default x-sw-service-name x-sw-endpoint-name-match-rule and x-sw-endpoint-name-format
// test direct lookup
- String endpointName = nameGrouping.format("serviceA", "GET:/products");
+ String endpointName = nameGrouping.format("serviceA", "GET:/products")._1();
Assertions.assertEquals("GET:/products", endpointName);
- endpointName = nameGrouping.format("serviceA", "GET:/products/123");
+ endpointName = nameGrouping.format("serviceA", "GET:/products/123")._1();
Assertions.assertEquals("GET:/products/{id}", endpointName);
- endpointName = nameGrouping.format("serviceA", "GET:/products/123/abc/ef");
+ endpointName = nameGrouping.format("serviceA", "GET:/products/123/abc/ef")._1();
Assertions.assertEquals("GET:/products/123/abc/ef", endpointName);
- endpointName = nameGrouping.format("serviceA", "GET:/products/123/relatedProducts");
+ endpointName = nameGrouping.format("serviceA", "GET:/products/123/relatedProducts")._1();
Assertions.assertEquals("GET:/products/{id}/relatedProducts", endpointName);
- endpointName = nameGrouping.format("serviceA", "GET:/products/1/relatedProducts");
+ endpointName = nameGrouping.format("serviceA", "GET:/products/1/relatedProducts")._1();
Assertions.assertEquals("GET:/products/{id}/relatedProducts", endpointName);
//test custom x-sw-service-name same x-sw-endpoint-name-match-rule and x-sw-endpoint-name-format
- endpointName = nameGrouping.format("serviceA-1", "POST:/customer");
+ endpointName = nameGrouping.format("serviceA-1", "POST:/customer")._1();
Assertions.assertEquals("POST:/customer", endpointName);
- endpointName = nameGrouping.format("serviceA-1", "<GET>:/customers/1");
+ endpointName = nameGrouping.format("serviceA-1", "<GET>:/customers/1")._1();
Assertions.assertEquals("<GET>:/customers/{id}", endpointName);
//test different x-sw-endpoint-name-match-rule and x-sw-endpoint-name-format
- endpointName = nameGrouping.format("serviceB", "GET:/products");
+ endpointName = nameGrouping.format("serviceB", "GET:/products")._1();
Assertions.assertEquals("/products:<GET>", endpointName);
- endpointName = nameGrouping.format("serviceB", "GET:/products/asia/cn");
+ endpointName = nameGrouping.format("serviceB", "GET:/products/asia/cn")._1();
Assertions.assertEquals("/products/{region}/{country}:<GET>", endpointName);
//test match priority, not match /products/{region}/{country}:<GET>
- endpointName = nameGrouping.format("serviceB", "GET:/products/12/relatedProducts");
+ endpointName = nameGrouping.format("serviceB", "GET:/products/12/relatedProducts")._1();
Assertions.assertEquals("/products/{id}/relatedProducts:<GET>", endpointName);
//test not match, return the origin
- endpointName = nameGrouping.format("serviceA", "GET:/products/");
+ endpointName = nameGrouping.format("serviceA", "GET:/products/")._1();
Assertions.assertNotEquals("GET:/products", endpointName);
- endpointName = nameGrouping.format("serviceA", "GET:/products/123/");
+ endpointName = nameGrouping.format("serviceA", "GET:/products/123/")._1();
Assertions.assertEquals("GET:/products/123/", endpointName);
- endpointName = nameGrouping.format("serviceC", "GET:/products/123");
+ endpointName = nameGrouping.format("serviceC", "GET:/products/123")._1();
Assertions.assertEquals("GET:/products/123", endpointName);
- endpointName = nameGrouping.format("serviceA", "GET:/products/1/ratings/123");
+ endpointName = nameGrouping.format("serviceA", "GET:/products/1/ratings/123")._1();
Assertions.assertEquals("GET:/products/1/ratings/123", endpointName);
- endpointName = nameGrouping.format("serviceA-1", "<GET>:/customers/1/123");
+ endpointName = nameGrouping.format("serviceA-1", "<GET>:/customers/1/123")._1();
Assertions.assertEquals("<GET>:/customers/1/123", endpointName);
- endpointName = nameGrouping.format("serviceB", "/products/:<GET>");
+ endpointName = nameGrouping.format("serviceB", "/products/:<GET>")._1();
Assertions.assertEquals("/products/:<GET>", endpointName);
- endpointName = nameGrouping.format("serviceB", "{GET}:/products");
+ endpointName = nameGrouping.format("serviceB", "{GET}:/products")._1();
Assertions.assertEquals("{GET}:/products", endpointName);
- endpointName = nameGrouping.format("serviceB", "/products/1/2/3:<GET>");
+ endpointName = nameGrouping.format("serviceB", "/products/1/2/3:<GET>")._1();
Assertions.assertEquals("/products/1/2/3:<GET>", endpointName);
}
diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/openapi/EndpointNameGroupingRule4OpenapiWatcherTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/openapi/EndpointNameGroupingRule4OpenapiWatcherTest.java
index dd9e9c2ff5..518d61e76c 100644
--- a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/openapi/EndpointNameGroupingRule4OpenapiWatcherTest.java
+++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/openapi/EndpointNameGroupingRule4OpenapiWatcherTest.java
@@ -73,7 +73,7 @@ public class EndpointNameGroupingRule4OpenapiWatcherTest {
return new String[0];
}
}, endpointNameGrouping);
- Assertions.assertEquals("GET:/products/{id}", endpointNameGrouping.format("serviceA", "GET:/products/123"));
+ Assertions.assertEquals("GET:/products/{id}", endpointNameGrouping.format("serviceA", "GET:/products/123")._1());
Map<String, ConfigChangeWatcher.ConfigChangeEvent> groupItems = new HashMap<>();
groupItems.put(
@@ -250,12 +250,12 @@ public class EndpointNameGroupingRule4OpenapiWatcherTest {
);
watcher.notifyGroup(groupItems);
- Assertions.assertEquals("GET:/products/{order-id}", endpointNameGrouping.format("serviceA", "GET:/products/123"));
+ Assertions.assertEquals("GET:/products/{order-id}", endpointNameGrouping.format("serviceA", "GET:/products/123")._1());
groupItems.put("serviceA.productAPI-v1", new ConfigChangeWatcher.ConfigChangeEvent("", ConfigChangeWatcher.EventType.DELETE));
watcher.notifyGroup(groupItems);
- Assertions.assertEquals("GET:/products/123", endpointNameGrouping.format("serviceA", "GET:/products/123"));
+ Assertions.assertEquals("GET:/products/123", endpointNameGrouping.format("serviceA", "GET:/products/123")._1());
}
}
diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/PatternTreeTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/PatternTreeTest.java
new file mode 100644
index 0000000000..469741b040
--- /dev/null
+++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/config/group/uri/quickmatch/PatternTreeTest.java
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.config.group.uri.quickmatch;
+
+import java.lang.reflect.Field;
+import java.util.List;
+import org.apache.skywalking.oap.server.library.util.StringFormatGroup;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class PatternTreeTest {
+ @Test
+ public void testTreeBuild() throws NoSuchFieldException, IllegalAccessException {
+ PatternTree tree = new PatternTree();
+ tree.addPattern("/products/{var}");
+ tree.addPattern("/products/{var}/detail");
+ tree.addPattern("/sales/{var}");
+ tree.addPattern("/employees/{var}/profile");
+
+ final Field rootField = PatternTree.class.getDeclaredField("roots");
+ rootField.setAccessible(true);
+ final List<PatternToken> roots = (List<PatternToken>) rootField.get(tree);
+ final PatternToken root = roots.get(0);
+
+ Assertions.assertEquals(new StringToken(""), root);
+ Assertions.assertEquals(3, root.children().size());
+ final PatternToken prodToken = root.children().get(0);
+ Assertions.assertEquals(new StringToken("products"), prodToken);
+ Assertions.assertEquals(1, prodToken.children().size());
+ final PatternToken prodVarToken = prodToken.children().get(0);
+ Assertions.assertEquals(new VarToken(), prodVarToken);
+ final PatternToken detailToken = prodVarToken.children().get(0);
+ Assertions.assertEquals(new StringToken("detail"), detailToken);
+
+ final PatternToken salesToken = root.children().get(1);
+ Assertions.assertEquals(new StringToken("sales"), salesToken);
+ Assertions.assertEquals(1, salesToken.children().size());
+
+ final PatternToken employeeToken = root.children().get(2);
+ Assertions.assertEquals(new StringToken("employees"), employeeToken);
+ Assertions.assertEquals(1, employeeToken.children().size());
+ }
+
+ @Test
+ public void testPatternMatch() {
+ PatternTree tree = new PatternTree();
+ tree.addPattern("/products/{var}");
+ tree.addPattern("/products/{var}/detail");
+ tree.addPattern("/sales/{var}");
+ tree.addPattern("/employees/{var}/profile");
+
+ StringFormatGroup.FormatResult result;
+ result = tree.match("/products/123");
+ Assertions.assertTrue(result.isMatch());
+ Assertions.assertEquals("/products/{var}", result.getReplacedName());
+
+ result = tree.match("/products/123/detail");
+ Assertions.assertTrue(result.isMatch());
+ Assertions.assertEquals("/products/{var}/detail", result.getReplacedName());
+
+ result = tree.match("/employees/skywalking/profile");
+ Assertions.assertTrue(result.isMatch());
+
+ // URI doesn't have / as prefix
+ result = tree.match("products/123/detail");
+ Assertions.assertFalse(result.isMatch());
+
+ // URI has extra suffix
+ result = tree.match("/products/123/detail/extra");
+ Assertions.assertFalse(result.isMatch());
+ }
+
+ @Test
+ public void testGetPostPatternMatch() {
+ PatternTree tree = new PatternTree();
+ tree.addPattern("GET:/products/{var}");
+ tree.addPattern("POST:/products/{var}/detail");
+ tree.addPattern("POST:/sales/{var}");
+ tree.addPattern("GET:/employees/{var}/profile");
+
+ StringFormatGroup.FormatResult result;
+ result = tree.match("GET:/products/123");
+ Assertions.assertTrue(result.isMatch());
+ Assertions.assertEquals("GET:/products/{var}", result.getReplacedName());
+ }
+}
\ No newline at end of file
diff --git a/oap-server/server-core/src/test/resources/endpoint-name-grouping.yml b/oap-server/server-core/src/test/resources/endpoint-name-grouping.yml
index 63bddcba2d..00d26af615 100644
--- a/oap-server/server-core/src/test/resources/endpoint-name-grouping.yml
+++ b/oap-server/server-core/src/test/resources/endpoint-name-grouping.yml
@@ -20,10 +20,10 @@
# /prod/ORDER123, /prod/ORDER456.
# This grouping file provides the regex based definition capability to merge those endpoints into a group by better and
# more meaningful aggregation metrics.
+# {var} represents any variable string in the URI.
grouping:
# Endpoint of the service would follow the following rules
- service-name: serviceA
rules:
- - endpoint-name: /prod/{id}
- regex: \/prod\/.+
\ No newline at end of file
+ - /prod/{var}
\ No newline at end of file
diff --git a/oap-server/server-library/library-util/src/main/java/org/apache/skywalking/oap/server/library/util/StringFormatGroup.java b/oap-server/server-library/library-util/src/main/java/org/apache/skywalking/oap/server/library/util/StringFormatGroup.java
index 72a716757a..285ac872ca 100644
--- a/oap-server/server-library/library-util/src/main/java/org/apache/skywalking/oap/server/library/util/StringFormatGroup.java
+++ b/oap-server/server-library/library-util/src/main/java/org/apache/skywalking/oap/server/library/util/StringFormatGroup.java
@@ -22,6 +22,7 @@ import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.regex.Pattern;
+import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.ToString;
@@ -38,6 +39,10 @@ public class StringFormatGroup {
rules = new ArrayList<>();
}
+ public StringFormatGroup(int size) {
+ rules = new ArrayList<>(size);
+ }
+
/**
* Add a new match rule. The rule will follow the order of being added.
*
@@ -75,6 +80,8 @@ public class StringFormatGroup {
@Getter
@RequiredArgsConstructor
+ @EqualsAndHashCode
+ @ToString
public static class FormatResult {
private final boolean match;
private final String name;
diff --git a/oap-server/server-starter/src/main/resources/application.yml b/oap-server/server-starter/src/main/resources/application.yml
index 0abb6ca834..37aa72ca25 100644
--- a/oap-server/server-starter/src/main/resources/application.yml
+++ b/oap-server/server-starter/src/main/resources/application.yml
@@ -129,6 +129,8 @@ core:
prepareThreads: ${SW_CORE_PREPARE_THREADS:2}
# Turn it on then automatically grouping endpoint by the given OpenAPI definitions.
enableEndpointNameGroupingByOpenapi: ${SW_CORE_ENABLE_ENDPOINT_NAME_GROUPING_BY_OPENAPI:true}
+ # The max number of HTTP URIs per service for further URI pattern recognition.
+ maxHttpUrisNumberPerService: ${SW_CORE_MAX_HTTP_URIS_NUMBER_PER_SVR:3000}
storage:
selector: ${SW_STORAGE:h2}
elasticsearch:
@@ -550,3 +552,9 @@ aws-firehose:
enableTLS: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_ENABLE_TLS:false}
tlsKeyPath: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_TLS_KEY_PATH:}
tlsCertChainPath: ${SW_RECEIVER_AWS_FIREHOSE_HTTP_TLS_CERT_CHAIN_PATH:}
+
+ai-pipeline:
+ selector: ${SW_AI_PIPELINE:default}
+ default:
+ uriRecognitionServerAddr: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_ADDR:}
+ uriRecognitionServerPort: ${SW_AI_PIPELINE_URI_RECOGNITION_SERVER_PORT:17128}
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/endpoint-name-grouping.yml b/oap-server/server-starter/src/main/resources/endpoint-name-grouping.yml
index 9695ba7f05..ad4e18183b 100644
--- a/oap-server/server-starter/src/main/resources/endpoint-name-grouping.yml
+++ b/oap-server/server-starter/src/main/resources/endpoint-name-grouping.yml
@@ -18,12 +18,12 @@
# on the name.
# But, in some cases, application puts the parameter in the endpoint name, such as putting order id in the URI, like
# /prod/ORDER123, /prod/ORDER456.
-# This grouping file provides the regex based definition capability to merge those endpoints into a group by better and
+# This grouping file provides the self-implemented algorithm to merge those endpoints into a group by better and
# more meaningful aggregation metrics.
+# {var} represents any variable string in the URI.
#grouping:
# # Endpoint of the service would follow the following rules
# - service-name: serviceA
# rules:
-# - endpoint-name: /prod/{id}
-# regex: \/prod\/.+
\ No newline at end of file
+# - /prod/{var}
\ No newline at end of file