You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2023/01/05 04:18:09 UTC

[GitHub] [skywalking] hanahmily opened a new pull request, #10233: Improve BanyanDB measure schema

hanahmily opened a new pull request, #10233:
URL: https://github.com/apache/skywalking/pull/10233

   ## Remove OAL and MAL generated measures' ID tag
   
   The previous default `ID` tag is removed from measures created by OAL and MAL. `multiGet` and `metricsQuery` methods will leverage *series id* and *time bucket* to retrieve measure data.
   
   ## Introduce `Banyandb.MeasureField` annotation
   
   The new annotation marks a column as a measure field instead of a tag. Now a typical measure looks like below:
   
   ```yaml
   entity:
       tagNames:
       - entity_id
     fields:
     - compressionMethod: COMPRESSION_METHOD_ZSTD
       encodingMethod: ENCODING_METHOD_UNSPECIFIED
       fieldType: FIELD_TYPE_STRING
       name: value
     - compressionMethod: COMPRESSION_METHOD_ZSTD
       encodingMethod: ENCODING_METHOD_GORILLA
       fieldType: FIELD_TYPE_INT
       name: precision
     - compressionMethod: COMPRESSION_METHOD_ZSTD
       encodingMethod: ENCODING_METHOD_UNSPECIFIED
       fieldType: FIELD_TYPE_STRING
       name: dataset
     interval: 1m
     metadata:
       group: measure-minute
       name: service_percentile_minute
     tagFamilies:
     - name: default
       tags:
       - indexedOnly: false
         name: entity_id
         type: TAG_TYPE_STRING
   ```
   
   The measure contains a single tag named `entity_id`, which belongs to the `entity`.  No index is created. 
   
   Furthermore, if we implement #10229, the `dataset` and `value` fields could get encoded.
   
   ## Add `requiredNoIndexingForBanyanDB` to `ScopeDefaultColumn.DefinedByField`
   
   Some fields in `Source` don't need indexing. The new `requiredNoIndexingForBanyanDB` could indicate the OAL parser adding the `BanyanDB.NoIndexing` annotation to generated metrics. For example, `service_id` in `instance_clr_cpu_minute` should not be indexed. Now the measure's schema could be 
   
   ```yaml
   entity:
       tagNames:
       - entity_id
     fields:
     - compressionMethod: COMPRESSION_METHOD_ZSTD
       encodingMethod: ENCODING_METHOD_GORILLA
       fieldType: FIELD_TYPE_INT
       name: value
     - compressionMethod: COMPRESSION_METHOD_ZSTD
       encodingMethod: ENCODING_METHOD_GORILLA
       fieldType: FIELD_TYPE_INT
       name: total
     interval: 1m
     metadata:
       createRevision: "296"
       group: measure-minute
       id: 0
       modRevision: "296"
       name: service_instance_cpm_minute
     tagFamilies:
     - name: storage-only
       tags:
       - indexedOnly: false
         name: entity_id
         type: TAG_TYPE_STRING
       - indexedOnly: false
         name: service_id
         type: TAG_TYPE_STRING
   ```
   
   There is no tag to be indexed.
   
   ## Indexed measures remained
   
   ```
    name: cache_access_sla_minute-index-rule-binding
       name: cache_access_sla_minute
       name: cache_read_sla_minute-index-rule-binding
       name: cache_read_sla_minute
       name: cache_write_sla_minute-index-rule-binding
       name: cache_write_sla_minute
       name: database_access_sla_minute-index-rule-binding
       name: database_access_sla_minute
       name: endpoint_relation_server_side_minute-index-rule-binding
       name: endpoint_relation_server_side_minute
       name: endpoint_relation_sla_minute-index-rule-binding
       name: endpoint_relation_sla_minute
       name: endpoint_sla_minute-index-rule-binding
       name: endpoint_sla_minute
       name: mq_endpoint_consume_sla_minute-index-rule-binding
       name: mq_endpoint_consume_sla_minute
       name: mq_endpoint_produce_sla_minute-index-rule-binding
       name: mq_endpoint_produce_sla_minute
       name: mq_service_consume_sla_minute-index-rule-binding
       name: mq_service_consume_sla_minute
       name: mq_service_produce_sla_minute-index-rule-binding
       name: mq_service_produce_sla_minute
       name: process_relation_client_side_minute-index-rule-binding
       name: process_relation_client_side_minute
       name: process_relation_server_side_minute-index-rule-binding
       name: process_relation_server_side_minute
       name: service_instance_relation_client_call_sla_minute-index-rule-binding
       name: service_instance_relation_client_call_sla_minute
       name: service_instance_relation_client_side_minute-index-rule-binding
       name: service_instance_relation_client_side_minute
       name: service_instance_relation_server_call_sla_minute-index-rule-binding
       name: service_instance_relation_server_call_sla_minute
       name: service_instance_relation_server_side_minute-index-rule-binding
       name: service_instance_relation_server_side_minute
       name: service_instance_sla_minute-index-rule-binding
       name: service_instance_sla_minute
       name: service_relation_client_call_sla_minute-index-rule-binding
       name: service_relation_client_call_sla_minute
       name: service_relation_client_side_minute-index-rule-binding
       name: service_relation_client_side_minute
       name: service_relation_server_call_sla_minute-index-rule-binding
       name: service_relation_server_call_sla_minute
       name: service_relation_server_side_minute-index-rule-binding
       name: service_relation_server_side_minute
       name: service_sla_minute-index-rule-binding
       name: service_sla_minute
   ```
   
   `relation` metrics have `source` and `dest` tags that need to be indexed. 
   
   `sla` metrics are from `PercentMetrics` whose `mark` column doesn't marked as `storageOnly` https://github.com/apache/skywalking/blob/f3b567160c/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java#L46
   
   @wu-sheng ^^ is this intended or by mistake?
   
   ## Add `BanyanDB.StoreIDTag` annotation
   
   Even though we remove most `ID` tags, the process traffic still needs the `ID` tag. There is a scenario to query a process by ID. But the process id is a hashed result and can not be analyzed to a structured object. So I add this new annotation to store the id as a tag for querying.
   
   @mrproliu @wu-sheng  The banyandb also encodes `name` and `instanceId` to internal ID(SeriesID) by the xxHash algorithm. We could pass the processId to the server as the internal id. The server will skip the encoding process by storing the ingested id directly. Do you think it makes sense?
   
   - [x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #10105 .
   - [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#issuecomment-1374418634

   @wankai123 There are some SeriesID changes, could you check?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062487009


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -34,14 +34,16 @@ public class ScopeDefaultColumn {
     private String columnName;
     private Class<?> type;
     private boolean isID;
+    private boolean banyandbNoIndexing;
     private int length;
 
-    public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length) {
+    public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length, boolean banyandbNoIndexing) {

Review Comment:
   `isID` is storage-related, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063169557


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
     @interface TimestampColumn {
         String value();
     }
+
+    /**
+     * MeasureField defines a column as a measure's field.
+     * @since 9.4.0
+     */
+    @Target({ElementType.FIELD})
+    @Retention(RetentionPolicy.RUNTIME)
+    @interface MeasureField {
+    }
+
+    /**
+     * StoreIDTag indicates a metric store its ID as a tag for searching.
+     * @Since 9.4.0
+     */
+    @Target({ElementType.TYPE})
+    @Retention(RetentionPolicy.RUNTIME)
+    @interface StoreIDTag {

Review Comment:
   ```suggestion
       @interface StoreIDAsTag {
   ```
   
   I think this name is more accurate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063952705


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
                 ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
                     elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
                     elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
-                    keywordColumn == null ? false : true
+                        keywordColumn != null

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1064020990


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java:
##########
@@ -92,7 +92,7 @@ public class ServiceTraffic extends Metrics {
     @Setter
     @Getter
     @Column(columnName = LAYER)
-    @BanyanDB.SeriesID(index = 1)
+    @BanyanDB.SeriesID(index = 0)

Review Comment:
   What is the reason of the layer first? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063163647


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
                 ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
                     elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
                     elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
-                    keywordColumn == null ? false : true
+                        keywordColumn != null

Review Comment:
   Maybe that is some of your local change. I tried formating locally, nothing changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062485652


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
                 ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
                     elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
                     elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
-                    keywordColumn == null ? false : true
+                        keywordColumn != null

Review Comment:
   `checkStyle` reports this line should be simplified as this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#issuecomment-1374427573

   @hanahmily I think this resolves #10033 too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063952830


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
     @interface TimestampColumn {
         String value();
     }
+
+    /**
+     * MeasureField defines a column as a measure's field.

Review Comment:
   I have added more details to the comments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063967929


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/BanyanDBModelExtension.java:
##########
@@ -37,4 +37,12 @@ public class BanyanDBModelExtension {
     @Setter
     private String timestampColumn;
 
+    /**
+     * shouldStoreIDTag indicates whether a metric store its ID as a tag.
+     * The installer will create a virtual string ID tag with a tree index rule.
+     */
+    @Getter
+    @Setter
+    private boolean shouldStoreIDTag;

Review Comment:
   It is better to remove `should` prefix. The generated method is `isShouldStoreIDTag`, which is wrong in the English grammar.
   
   ```suggestion
       private boolean storeIDTag;
   ```



##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/BanyanDBModelExtension.java:
##########
@@ -37,4 +37,12 @@ public class BanyanDBModelExtension {
     @Setter
     private String timestampColumn;
 
+    /**
+     * shouldStoreIDTag indicates whether a metric store its ID as a tag.

Review Comment:
   ```suggestion
        * shouldStoreIDTag indicates whether a metric stores its ID as a tag.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1064110751


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java:
##########
@@ -92,7 +92,7 @@ public class ServiceTraffic extends Metrics {
     @Setter
     @Getter
     @Column(columnName = LAYER)
-    @BanyanDB.SeriesID(index = 1)
+    @BanyanDB.SeriesID(index = 0)

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063969794


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -52,7 +52,7 @@ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, bo
         /**
          * Dynamic active means this column is only activated through core setting explicitly.
          *
-         * @return
+         * @return false: Inactive dynamic column. true: Active dynamic column.

Review Comment:
   This comment seems a little strange. 
   
   ```suggestion
            * @return FALSE: this column is not going to be added to the final generated metric as a column.
            *         TRUE: this column could be added as a column if core/activeExtraModelColumns == true.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#issuecomment-1371773059

   > sla metrics are from PercentMetrics whose mark column doesn't marked as storageOnly https://github.com/apache/skywalking/blob/f3b567160c/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/PercentMetrics.java#L46
   
   This is a bug, please fix as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062289431


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -52,14 +54,22 @@ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, bo
         /**
          * Dynamic active means this column is only activated through core setting explicitly.
          *
-         * @return
+         * @return false: Inactive dynamic column. true: Active dynamic column.
          */
         boolean requireDynamicActive() default false;
 
         /**
          * Define column length, only effective when the type is String.
          */
         int length() default 256;
+
+
+        /**
+         * Active indexing for a column(tag) in BanyanDB.
+         *
+         * @return false: Indexing the column, true: No indexing the column.
+         */
+        boolean requiredNoIndexingForBanyanDB() default false;

Review Comment:
   `DefinedByField` is not storage sensitive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] kezhenxu94 commented on pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#issuecomment-1375060898

   This PR reverts the UI module unexpectedly, https://github.com/apache/skywalking/pull/10249


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063093347


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -52,14 +54,22 @@ public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, bo
         /**
          * Dynamic active means this column is only activated through core setting explicitly.
          *
-         * @return
+         * @return false: Inactive dynamic column. true: Active dynamic column.
          */
         boolean requireDynamicActive() default false;
 
         /**
          * Define column length, only effective when the type is String.
          */
         int length() default 256;
+
+
+        /**
+         * Active indexing for a column(tag) in BanyanDB.
+         *
+         * @return false: Indexing the column, true: No indexing the column.
+         */
+        boolean requiredNoIndexingForBanyanDB() default false;

Review Comment:
   Removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063093180


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -34,14 +34,16 @@ public class ScopeDefaultColumn {
     private String columnName;
     private Class<?> type;
     private boolean isID;
+    private boolean banyandbNoIndexing;
     private int length;
 
-    public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length) {
+    public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length, boolean banyandbNoIndexing) {

Review Comment:
   Dropped it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063178954


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
     @interface TimestampColumn {
         String value();
     }
+
+    /**
+     * MeasureField defines a column as a measure's field.

Review Comment:
   I will update the comments. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng merged pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng merged PR #10233:
URL: https://github.com/apache/skywalking/pull/10233


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062291113


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
                 ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
                     elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
                     elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
-                    keywordColumn == null ? false : true
+                        keywordColumn != null

Review Comment:
   Why is this format different?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062484743


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -34,14 +34,16 @@ public class ScopeDefaultColumn {
     private String columnName;
     private Class<?> type;
     private boolean isID;
+    private boolean banyandbNoIndexing;
     private int length;
 
-    public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length) {
+    public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length, boolean banyandbNoIndexing) {

Review Comment:
   `service_id` is defined by `ScopeDefaultColumn`.  IIUC, `ScopeDefaultColumn.columnName` defines the column name in the storage layer. Essentially, I tend to mark `service_id` as a non-indexed tag. Adding a new field seems relevant. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063181201


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
                 ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
                     elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
                     elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
-                    keywordColumn == null ? false : true
+                        keywordColumn != null

Review Comment:
   My IDE might suggest this change. But simple is better than complex. This tweak makes sense to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063968980


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,29 @@ enum IndexType {
     @interface TimestampColumn {
         String value();
     }
+
+    /**
+     * MeasureField defines a column as a measure's field.
+     *
+     * Annotated: the column is a measure field.
+     * Unannotated: the column is a measure tag.
+     *   storageOnly=true: the column is a measure tag which is not indexed.
+     *   storageOnly=false: the column is a measure tag which is indexed.
+     *     indexOnly=true: the column is a measure tag which is indexed, but not stored.
+     *     indexOnly=false: the column is a measure tag which is indexed and stored.

Review Comment:
   ```suggestion
        *   storageOnly=true: the column is a measure tag that is not indexed.
        *   storageOnly=false: the column is a measure tag that is indexed.
        *   indexOnly=true: the column is a measure tag that is indexed, but not stored.
        *   indexOnly=false: the column is a measure tag that is indexed and stored.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1064064959


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java:
##########
@@ -92,7 +92,7 @@ public class ServiceTraffic extends Metrics {
     @Setter
     @Getter
     @Column(columnName = LAYER)
-    @BanyanDB.SeriesID(index = 1)
+    @BanyanDB.SeriesID(index = 0)

Review Comment:
   Searching SeriesID supports prefix scanning. OAP uses finding by `layer` a lot. Based on that, moving the `layer` to the first place will improve the search performance a bit. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063182808


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -190,7 +194,7 @@ private void retrieval(final Class<?> clazz,
                 ElasticSearchExtension elasticSearchExtension = new ElasticSearchExtension(
                     elasticSearchAnalyzer == null ? null : elasticSearchAnalyzer.analyzer(),
                     elasticSearchColumn == null ? null : elasticSearchColumn.columnAlias(),
-                    keywordColumn == null ? false : true
+                        keywordColumn != null

Review Comment:
   I am not saying the code change, you are changing format.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] hanahmily commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
hanahmily commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063952830


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
     @interface TimestampColumn {
         String value();
     }
+
+    /**
+     * MeasureField defines a column as a measure's field.

Review Comment:
   Add more details to comments.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1064066206


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/service/ServiceTraffic.java:
##########
@@ -92,7 +92,7 @@ public class ServiceTraffic extends Metrics {
     @Setter
     @Getter
     @Column(columnName = LAYER)
-    @BanyanDB.SeriesID(index = 1)
+    @BanyanDB.SeriesID(index = 0)

Review Comment:
   Make sense. Let's add a little comments to seriesID index.
   We should indicate the index matters for prefix searching.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1063171191


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/BanyanDB.java:
##########
@@ -131,4 +131,22 @@ enum IndexType {
     @interface TimestampColumn {
         String value();
     }
+
+    /**
+     * MeasureField defines a column as a measure's field.

Review Comment:
   I think we need to explain if it is not annotated, what a column would be processed.
   Such as
   1. storageOnly=true without annotated
   2. indexOnly=true without annotated
   3. storageOnly=true with annotated
   4. indexOnly=true with annotated
   5. storageOnly=false and indexOnly=false with annotated
   5. indexOnly=false and indexOnly=false without annotated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #10233: Improve BanyanDB measure schema

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #10233:
URL: https://github.com/apache/skywalking/pull/10233#discussion_r1062288897


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ScopeDefaultColumn.java:
##########
@@ -34,14 +34,16 @@ public class ScopeDefaultColumn {
     private String columnName;
     private Class<?> type;
     private boolean isID;
+    private boolean banyandbNoIndexing;
     private int length;
 
-    public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length) {
+    public ScopeDefaultColumn(String fieldName, String columnName, Class<?> type, boolean isID, int length, boolean banyandbNoIndexing) {

Review Comment:
   This seems an unreasonable change. `ScopeDefaultColumn` is never storage related.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org