You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2023/01/05 04:18:09 UTC
[GitHub] [skywalking] hanahmily opened a new pull request, #10233: Improve BanyanDB measure schema
hanahmily opened a new pull request, #10233:
URL: https://github.com/apache/skywalking/pull/10233
## Remove OAL and MAL generated measures' ID tag
The previous default `ID` tag is removed from measures created by OAL and MAL. `multiGet` and `metricsQuery` methods will leverage *series id* and *time bucket* to retrieve measure data.
## Introduce `Banyandb.MeasureField` annotation
The new annotation marks a column as a measure field instead of a tag. Now a typical measure looks like below:
```yaml
entity:
tagNames:
- entity_id
fields:
- compressionMethod: COMPRESSION_METHOD_ZSTD
encodingMethod: ENCODING_METHOD_UNSPECIFIED
fieldType: FIELD_TYPE_STRING
name: value
- compressionMethod: COMPRESSION_METHOD_ZSTD
encodingMethod: ENCODING_METHOD_GORILLA
fieldType: FIELD_TYPE_INT
name: precision
- compressionMethod: COMPRESSION_METHOD_ZSTD
encodingMethod: ENCODING_METHOD_UNSPECIFIED
fieldType: FIELD_TYPE_STRING
name: dataset
interval: 1m
metadata:
group: measure-minute
name: service_percentile_minute
tagFamilies:
- name: default
tags:
- indexedOnly: false
name: entity_id
type: TAG_TYPE_STRING
```
The measure contains a single tag named `entity_id`, which belongs to the `entity`. No index is created.
Furthermore, if we implement #10229, the `dataset` and `value` fields could get encoded.
## Add `requiredNoIndexingForBanyanDB` to `ScopeDefaultColumn.DefinedByField`
Some fields in `Source` don't need indexing. The new `requiredNoIndexingForBanyanDB` could indicate the OAL parser adding the `BanyanDB.NoIndexing` annotation to generated metrics. For example, `service_id` in `instance_clr_cpu_minute` should not be indexed. Now the measure's schema could be
```yaml
entity:
tagNames:
- entity_id
fields:
- compressionMethod: COMPRESSION_METHOD_ZSTD
encodingMethod: ENCODING_METHOD_GORILLA
fieldType: FIELD_TYPE_INT
name: value
- compressionMethod: COMPRESSION_METHOD_ZSTD
encodingMethod: ENCODING_METHOD_GORILLA
fieldType: FIELD_TYPE_INT
name: total
interval: 1m
metadata:
createRevision: "296"
group: measure-minute
id: 0
modRevision: "296"
name: service_instance_cpm_minute
tagFamilies:
- name: storage-only
tags:
- indexedOnly: false
name: entity_id
type: TAG_TYPE_STRING
- indexedOnly: false
name: service_id
type: TAG_TYPE_STRING
```
There is no tag to be indexed.
## Indexed measures remained
```
name: cache_access_sla_minute-index-rule-binding
name: cache_access_sla_minute
name: cache_read_sla_minute-index-rule-binding
name: cache_read_sla_minute
name: cache_write_sla_minute-index-rule-binding
name: cache_write_sla_minute
name: database_access_sla_minute-index-rule-binding
name: database_access_sla_minute
name: endpoint_relation_server_side_minute-index-rule-binding
name: endpoint_relation_server_side_minute
name: endpoint_relation_sla_minute-index-rule-binding
name: endpoint_relation_sla_minute
name: endpoint_sla_minute-index-rule-binding
name: endpoint_sla_minute
name: mq_endpoint_consume_sla_minute-index-rule-binding
name: mq_endpoint_consume_sla_minute
name: mq_endpoint_produce_sla_minute-index-rule-binding
name: mq_endpoint_produce_sla_minute
name: mq_service_consume_sla_minute-index-rule-binding
name: mq_service_consume_sla_minute
name: mq_service_produce_sla_minute-index-rule-binding
name: mq_service_produce_sla_minute
name: process_relation_client_side_minute-index-rule-binding
name: process_relation_client_side_minute
name: process_relation_server_side_minute-index-rule-binding
name: process_relation_server_side_minute
name: service_instance_relation_client_call_sla_minute-index-rule-binding
name: service_instance_relation_client_call_sla_minute
name: service_instance_relation_client_side_minute-index-rule-binding
name: service_instance_relation_client_side_minute
name: service_instance_relation_server_call_sla_minute-index-rule-binding
name: service_instance_relation_server_call_sla_minute
name: service_instance_relation_server_side_minute-index-rule-binding
name: service_instance_relation_server_side_minute
name: service_instance_sla_minute-index-rule-binding
name: service_instance_sla_minute
name: service_relation_client_call_sla_minute-index-rule-binding
name: service_relation_client_call_sla_minute
name: service_relation_client_side_minute-index-rule-binding
name: service_relation_client_side_minute
name: service_relation_server_call_sla_minute-index-rule-binding
name: service_relation_server_call_sla_minute
name: service_relation_server_side_minute-index-rule-binding
name: service_relation_server_side_minute
name: service_sla_minute-index-rule-binding
name: service_sla_minute
```
`relation` metrics have `source` and `dest` tags that need to be indexed.
`sla` metrics are from `PercentMetrics` whose `mark` column doesn't marked as `storageOnly` https://github.com/apache/skywalking/blob/f3b567160c/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java#L46
@wu-sheng ^^ is this intended or by mistake?
## Add `BanyanDB.StoreIDTag` annotation
Even though we remove most `ID` tags, the process traffic still needs the `ID` tag. There is a scenario to query a process by ID. But the process id is a hashed result and can not be analyzed to a structured object. So I add this new annotation to store the id as a tag for querying.
@mrproliu @wu-sheng The banyandb also encodes `name` and `instanceId` to internal ID(SeriesID) by the xxHash algorithm. We could pass the processId to the server as the internal id. The server will skip the encoding process by storing the ingested id directly. Do you think it makes sense?
- [x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #10105 .
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#issuecomment-1374418634
@wankai123 There are some SeriesID changes, could you check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062487009
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -34,14 +34,16 @@ public class ScopeDefaultColumn {
private String columnName;
private Class<?> type;
private boolean isID;
+ private boolean banyandbNoIndexing;
private int length;
- public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length) {
+ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length, boolean banyandbNoIndexing) {
Review Comment:
`isID` is storage-related, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063169557
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
@interface TimestampColumn {
String value();
}
+
+ /**
+ * MeasureField defines a column as a measure's field.
+ * @since 9.4.0
+ */
+ @Target({ElementType.FIELD})
+ @Retention(RetentionPolicy.RUNTIME)
+ @interface MeasureField {
+ }
+
+ /**
+ * StoreIDTag indicates a metric store its ID as a tag for searching.
+ * @Since 9.4.0
+ */
+ @Target({ElementType.TYPE})
+ @Retention(RetentionPolicy.RUNTIME)
+ @interface StoreIDTag {
Review Comment:
```suggestion
@interface StoreIDAsTag {
```
I think this name is more accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063952705
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
- keywordColumn == null ? false : true
+ keywordColumn != null
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1064020990
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java:
##########
@@ -92,7 +92,7 @@ public class ServiceTraffic extends Metrics {
@Setter
@Getter
@Column(columnName = LAYER)
- @BanyanDB.SeriesID(index = 1)
+ @BanyanDB.SeriesID(index = 0)
Review Comment:
What is the reason of the layer first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063163647
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
- keywordColumn == null ? false : true
+ keywordColumn != null
Review Comment:
Maybe that is some of your local change. I tried formating locally, nothing changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062485652
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
- keywordColumn == null ? false : true
+ keywordColumn != null
Review Comment:
`checkStyle` reports this line should be simplified as this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#issuecomment-1374427573
@hanahmily I think this resolves #10033 too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063952830
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
@interface TimestampColumn {
String value();
}
+
+ /**
+ * MeasureField defines a column as a measure's field.
Review Comment:
I have added more details to the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063967929
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/BanyanDBModelExtension.java:
##########
@@ -37,4 +37,12 @@ public class BanyanDBModelExtension {
@Setter
private String timestampColumn;
+ /**
+ * shouldStoreIDTag indicates whether a metric store its ID as a tag.
+ * The installer will create a virtual string ID tag with a tree index rule.
+ */
+ @Getter
+ @Setter
+ private boolean shouldStoreIDTag;
Review Comment:
It is better to remove `should` prefix. The generated method is `isShouldStoreIDTag`, which is wrong in the English grammar.
```suggestion
private boolean storeIDTag;
```
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/BanyanDBModelExtension.java:
##########
@@ -37,4 +37,12 @@ public class BanyanDBModelExtension {
@Setter
private String timestampColumn;
+ /**
+ * shouldStoreIDTag indicates whether a metric store its ID as a tag.
Review Comment:
```suggestion
* shouldStoreIDTag indicates whether a metric stores its ID as a tag.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1064110751
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java:
##########
@@ -92,7 +92,7 @@ public class ServiceTraffic extends Metrics {
@Setter
@Getter
@Column(columnName = LAYER)
- @BanyanDB.SeriesID(index = 1)
+ @BanyanDB.SeriesID(index = 0)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063969794
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -52,7 +52,7 @@ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, bo
/**
* Dynamic active means this column is only activated through core setting explicitly.
*
- * @return
+ * @return false: Inactive dynamic column. true: Active dynamic column.
Review Comment:
This comment seems a little strange.
```suggestion
* @return FALSE: this column is not going to be added to the final generated metric as a column.
* TRUE: this column could be added as a column if core/activeExtraModelColumns == true.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#issuecomment-1371773059
> sla metrics are from PercentMetrics whose mark column doesn't marked as storageOnly https://github.com/apache/skywalking/blob/f3b567160c/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java#L46
This is a bug, please fix as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062289431
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -52,14 +54,22 @@ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, bo
/**
* Dynamic active means this column is only activated through core setting explicitly.
*
- * @return
+ * @return false: Inactive dynamic column. true: Active dynamic column.
*/
boolean requireDynamicActive() default false;
/**
* Define column length, only effective when the type is String.
*/
int length() default 256;
+
+
+ /**
+ * Active indexing for a column(tag) in BanyanDB.
+ *
+ * @return false: Indexing the column, true: No indexing the column.
+ */
+ boolean requiredNoIndexingForBanyanDB() default false;
Review Comment:
`DefinedByField` is not storage sensitive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#issuecomment-1375060898
This PR reverts the UI module unexpectedly, https://github.com/apache/skywalking/pull/10249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063093347
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -52,14 +54,22 @@ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, bo
/**
* Dynamic active means this column is only activated through core setting explicitly.
*
- * @return
+ * @return false: Inactive dynamic column. true: Active dynamic column.
*/
boolean requireDynamicActive() default false;
/**
* Define column length, only effective when the type is String.
*/
int length() default 256;
+
+
+ /**
+ * Active indexing for a column(tag) in BanyanDB.
+ *
+ * @return false: Indexing the column, true: No indexing the column.
+ */
+ boolean requiredNoIndexingForBanyanDB() default false;
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063093180
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -34,14 +34,16 @@ public class ScopeDefaultColumn {
private String columnName;
private Class<?> type;
private boolean isID;
+ private boolean banyandbNoIndexing;
private int length;
- public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length) {
+ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length, boolean banyandbNoIndexing) {
Review Comment:
Dropped it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063178954
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
@interface TimestampColumn {
String value();
}
+
+ /**
+ * MeasureField defines a column as a measure's field.
Review Comment:
I will update the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng merged pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng merged PR #10233:
URL: https://github.com/apache/skywalking/pull/10233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062291113
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
- keywordColumn == null ? false : true
+ keywordColumn != null
Review Comment:
Why is this format different?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062484743
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -34,14 +34,16 @@ public class ScopeDefaultColumn {
private String columnName;
private Class<?> type;
private boolean isID;
+ private boolean banyandbNoIndexing;
private int length;
- public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length) {
+ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length, boolean banyandbNoIndexing) {
Review Comment:
`service_id` is defined by `ScopeDefaultColumn`. IIUC, `ScopeDefaultColumn.columnName` defines the column name in the storage layer. Essentially, I tend to mark `service_id` as a non-indexed tag. Adding a new field seems relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063181201
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
- keywordColumn == null ? false : true
+ keywordColumn != null
Review Comment:
My IDE might suggest this change. But simple is better than complex. This tweak makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063968980
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,29 @@ enum IndexType {
@interface TimestampColumn {
String value();
}
+
+ /**
+ * MeasureField defines a column as a measure's field.
+ *
+ * Annotated: the column is a measure field.
+ * Unannotated: the column is a measure tag.
+ * storageOnly=true: the column is a measure tag which is not indexed.
+ * storageOnly=false: the column is a measure tag which is indexed.
+ * indexOnly=true: the column is a measure tag which is indexed, but not stored.
+ * indexOnly=false: the column is a measure tag which is indexed and stored.
Review Comment:
```suggestion
* storageOnly=true: the column is a measure tag that is not indexed.
* storageOnly=false: the column is a measure tag that is indexed.
* indexOnly=true: the column is a measure tag that is indexed, but not stored.
* indexOnly=false: the column is a measure tag that is indexed and stored.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1064064959
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java:
##########
@@ -92,7 +92,7 @@ public class ServiceTraffic extends Metrics {
@Setter
@Getter
@Column(columnName = LAYER)
- @BanyanDB.SeriesID(index = 1)
+ @BanyanDB.SeriesID(index = 0)
Review Comment:
Searching SeriesID supports prefix scanning. OAP uses finding by `layer` a lot. Based on that, moving the `layer` to the first place will improve the search performance a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063182808
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
- keywordColumn == null ? false : true
+ keywordColumn != null
Review Comment:
I am not saying the code change, you are changing format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063952830
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
@interface TimestampColumn {
String value();
}
+
+ /**
+ * MeasureField defines a column as a measure's field.
Review Comment:
Add more details to comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1064066206
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java:
##########
@@ -92,7 +92,7 @@ public class ServiceTraffic extends Metrics {
@Setter
@Getter
@Column(columnName = LAYER)
- @BanyanDB.SeriesID(index = 1)
+ @BanyanDB.SeriesID(index = 0)
Review Comment:
Make sense. Let's add a little comments to seriesID index.
We should indicate the index matters for prefix searching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063171191
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
@interface TimestampColumn {
String value();
}
+
+ /**
+ * MeasureField defines a column as a measure's field.
Review Comment:
I think we need to explain if it is not annotated, what a column would be processed.
Such as
1. storageOnly=true without annotated
2. indexOnly=true without annotated
3. storageOnly=true with annotated
4. indexOnly=true with annotated
5. storageOnly=false and indexOnly=false with annotated
5. indexOnly=false and indexOnly=false without annotated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062288897
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -34,14 +34,16 @@ public class ScopeDefaultColumn {
private String columnName;
private Class<?> type;
private boolean isID;
+ private boolean banyandbNoIndexing;
private int length;
- public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length) {
+ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length, boolean banyandbNoIndexing) {
Review Comment:
This seems an unreasonable change. `ScopeDefaultColumn` is never storage related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org