You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/05/31 14:05:04 UTC

[GitHub] [skywalking] kezhenxu94 opened a new pull request, #9151: Add instance properties extractor in MAL

kezhenxu94 opened a new pull request, #9151:
URL: https://github.com/apache/skywalking/pull/9151

   <!--
       ⚠️ Please make sure to read this template first, pull requests that don't accord with this template
       maybe closed without notice.
       Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
       Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
   -->
   
   <!-- ==== πŸ› Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist πŸ‘‡ ====
   ### Fix <bug description or the bug issue number or bug issue link>
   - [ ] Add a unit test to verify that the fix works.
   - [ ] Explain briefly why the bug exists and how to fix it.
        ==== πŸ› Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist πŸ‘† ==== -->
   
   <!-- ==== πŸ“ˆ Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist πŸ‘‡ ====
   ### Improve the performance of <class or module or ...>
   - [ ] Add a benchmark for the improvement, refer to [the existing ones](https://github.com/apache/skywalking/blob/master/apm-commons/apm-datacarrier/src/test/java/org/apache/skywalking/apm/commons/datacarrier/LinkedArrayBenchmark.java)
   - [ ] The benchmark result.
   ```text
   <Paste the benchmark results here>
   ```
   - [ ] Links/URLs to the theory proof or discussion articles/blogs. <links/URLs here>
        ==== πŸ“ˆ Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist πŸ‘† ==== -->
   
   <!-- ==== πŸ†• Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist πŸ‘‡ ====
   ### <Feature description>
   - [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
   - [ ] Update the documentation to include this new feature.
   - [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
   - [ ] If it's UI related, attach the screenshots below.
        ==== πŸ†• Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist πŸ‘† ==== -->
   
   - [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
   - [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/changelog/docs/en/changes/changes.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886280847


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java:
##########
@@ -79,7 +79,7 @@ public boolean combine(final Metrics metrics) {
         final InstanceTraffic instanceTraffic = (InstanceTraffic) metrics;
         this.lastPingTimestamp = instanceTraffic.getLastPingTimestamp();
         if (instanceTraffic.getProperties() != null && instanceTraffic.getProperties().size() > 0) {
-            this.properties = instanceTraffic.getProperties();
+            instanceTraffic.getProperties().entrySet().forEach(it -> this.properties.add(it.getKey(), it.getValue()));

Review Comment:
   > this.properties = instanceTraffic.getProperties();
   
   The previous version is overriding, how your new codes could make override happens?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886243606


##########
docs/en/concepts-and-designs/mal.md:
##########
@@ -239,13 +239,13 @@ last_server_state_sync_time_in_seconds.tagEqual('production', 'catalog').downsam
 They extract level relevant labels from metric labels, then informs the meter-system the level and [layer](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java) to which this metric belongs.
 
  - `service([svc_label1, svc_label2...], Layer)` extracts service level labels from the array argument, extracts layer from `Layer` argument.
- - `instance([svc_label1, svc_label2...], [ins_label1, ins_label2...], Layer)` extracts service level labels from the first array argument, 
-                                                                        extracts instance level labels from the second array argument, extracts layer from `Layer` argument.
- - `endpoint([svc_label1, svc_label2...], [ep_label1, ep_label2...])` extracts service level labels from the first array argument, 
+ - `instance([svc_label1, svc_label2...], [ins_label1, ins_label2...], Layer, Closure<Map<String, String>> propertiesExtractor)` extracts service level labels from the first array argument,

Review Comment:
   Yes. It's optional. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wankai123 commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wankai123 commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r885695973


##########
oap-server/server-starter/src/main/resources/otel-oc-rules/k8s-instance.yaml:
##########
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+filter: "{ tags -> tags.job_name in [ 'kubernetes-cadvisor', 'kube-state-metrics' ] }" # The OpenTelemetry job name
+expSuffix: |-
+  service(['cluster' , 'service'], '::', Layer.K8S_SERVICE)
+  .instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, { tags -> ['pod': tags.pod, 'namespace': tags.namespace] })
+metricPrefix: k8s_service_instance
+metricsRules:
+  - name: pod_instance_status
+    exp: kube_pod_status_phase.retagByK8sMeta('service' , K8sRetagType.Pod2Service , 'pod' , 'namespace').tagNotEqual('service' , '').sum(['cluster', 'namespace', 'service' , 'pod'])

Review Comment:
   A little curious, why don't we get the pod meta info from the K8s API server directly?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886297784


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java:
##########
@@ -79,7 +79,7 @@ public boolean combine(final Metrics metrics) {
         final InstanceTraffic instanceTraffic = (InstanceTraffic) metrics;
         this.lastPingTimestamp = instanceTraffic.getLastPingTimestamp();
         if (instanceTraffic.getProperties() != null && instanceTraffic.getProperties().size() > 0) {
-            this.properties = instanceTraffic.getProperties();
+            instanceTraffic.getProperties().entrySet().forEach(it -> this.properties.add(it.getKey(), it.getValue()));

Review Comment:
   According to `Builder#storage2Entity`, this `properties` could be null, when we need to read an empty properties from the database. Please do null check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886327646


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java:
##########
@@ -79,7 +79,7 @@ public boolean combine(final Metrics metrics) {
         final InstanceTraffic instanceTraffic = (InstanceTraffic) metrics;
         this.lastPingTimestamp = instanceTraffic.getLastPingTimestamp();
         if (instanceTraffic.getProperties() != null && instanceTraffic.getProperties().size() > 0) {
-            this.properties = instanceTraffic.getProperties();
+            instanceTraffic.getProperties().entrySet().forEach(it -> this.properties.add(it.getKey(), it.getValue()));

Review Comment:
   > Could it be NPE relative? Once the properties list is null, your codes are not able to make it back anymore.
   
   Ok that’s indeed because of NPE



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng merged pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng merged PR #9151:
URL: https://github.com/apache/skywalking/pull/9151


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r885742315


##########
docs/en/concepts-and-designs/mal.md:
##########
@@ -239,13 +239,13 @@ last_server_state_sync_time_in_seconds.tagEqual('production', 'catalog').downsam
 They extract level relevant labels from metric labels, then informs the meter-system the level and [layer](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java) to which this metric belongs.
 
  - `service([svc_label1, svc_label2...], Layer)` extracts service level labels from the array argument, extracts layer from `Layer` argument.
- - `instance([svc_label1, svc_label2...], [ins_label1, ins_label2...], Layer)` extracts service level labels from the first array argument, 
-                                                                        extracts instance level labels from the second array argument, extracts layer from `Layer` argument.
- - `endpoint([svc_label1, svc_label2...], [ep_label1, ep_label2...])` extracts service level labels from the first array argument, 
+ - `instance([svc_label1, svc_label2...], [ins_label1, ins_label2...], Layer, Closure<Map<String, String>> propertiesExtractor)` extracts service level labels from the first array argument,

Review Comment:
   Is this Closure optional?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r885729204


##########
oap-server/server-starter/src/main/resources/otel-oc-rules/k8s-instance.yaml:
##########
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+filter: "{ tags -> tags.job_name in [ 'kubernetes-cadvisor', 'kube-state-metrics' ] }" # The OpenTelemetry job name
+expSuffix: |-
+  service(['cluster' , 'service'], '::', Layer.K8S_SERVICE)
+  .instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, { tags -> ['pod': tags.pod, 'namespace': tags.namespace] })
+metricPrefix: k8s_service_instance
+metricsRules:
+  - name: pod_instance_status
+    exp: kube_pod_status_phase.retagByK8sMeta('service' , K8sRetagType.Pod2Service , 'pod' , 'namespace').tagNotEqual('service' , '').sum(['cluster', 'namespace', 'service' , 'pod'])

Review Comment:
   I think we don't expose enough API for this, no matter what we do to drop, it is a kind of hijacking.
   
   I don't have much concern about this metric actually. This just adds an extra 1% metric :) 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886300714


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java:
##########
@@ -79,7 +79,7 @@ public boolean combine(final Metrics metrics) {
         final InstanceTraffic instanceTraffic = (InstanceTraffic) metrics;
         this.lastPingTimestamp = instanceTraffic.getLastPingTimestamp();
         if (instanceTraffic.getProperties() != null && instanceTraffic.getProperties().size() > 0) {
-            this.properties = instanceTraffic.getProperties();
+            instanceTraffic.getProperties().entrySet().forEach(it -> this.properties.add(it.getKey(), it.getValue()));

Review Comment:
   Could it be NPE relative? Once the properties list is null, your codes are not able to make it back anymore. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886279223


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java:
##########
@@ -79,7 +79,7 @@ public boolean combine(final Metrics metrics) {
         final InstanceTraffic instanceTraffic = (InstanceTraffic) metrics;
         this.lastPingTimestamp = instanceTraffic.getLastPingTimestamp();
         if (instanceTraffic.getProperties() != null && instanceTraffic.getProperties().size() > 0) {
-            this.properties = instanceTraffic.getProperties();
+            instanceTraffic.getProperties().entrySet().forEach(it -> this.properties.add(it.getKey(), it.getValue()));

Review Comment:
   @kezhenxu94 Have you debugged here? Is `this.properties` updated?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886267254


##########
oap-server/server-starter/src/main/resources/otel-oc-rules/k8s-instance.yaml:
##########
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+filter: "{ tags -> tags.job_name in [ 'kubernetes-cadvisor', 'kube-state-metrics' ] }" # The OpenTelemetry job name
+expSuffix: |-
+  service(['cluster' , 'service'], '::', Layer.K8S_SERVICE)
+  .instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, { tags -> ['pod': tags.pod, 'namespace': tags.namespace] })
+metricPrefix: k8s_service_instance
+metricsRules:
+  - name: pod_instance_status
+    exp: kube_pod_status_phase.retagByK8sMeta('service' , K8sRetagType.Pod2Service , 'pod' , 'namespace').tagNotEqual('service' , '').sum(['cluster', 'namespace', 'service' , 'pod'])

Review Comment:
   see https://github.com/apache/skywalking/pull/9151#discussion_r886266807 
   
   We could create an instance(pod-oriented) dashboard to make this clear. But for now, even without them, I feel it is fine.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wankai123 commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wankai123 commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r885709774


##########
oap-server/server-starter/src/main/resources/otel-oc-rules/k8s-instance.yaml:
##########
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+filter: "{ tags -> tags.job_name in [ 'kubernetes-cadvisor', 'kube-state-metrics' ] }" # The OpenTelemetry job name
+expSuffix: |-
+  service(['cluster' , 'service'], '::', Layer.K8S_SERVICE)
+  .instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, { tags -> ['pod': tags.pod, 'namespace': tags.namespace] })
+metricPrefix: k8s_service_instance
+metricsRules:
+  - name: pod_instance_status
+    exp: kube_pod_status_phase.retagByK8sMeta('service' , K8sRetagType.Pod2Service , 'pod' , 'namespace').tagNotEqual('service' , '').sum(['cluster', 'namespace', 'service' , 'pod'])

Review Comment:
   Oh, I see. This is not for mesh, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886300229


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java:
##########
@@ -79,7 +79,7 @@ public boolean combine(final Metrics metrics) {
         final InstanceTraffic instanceTraffic = (InstanceTraffic) metrics;
         this.lastPingTimestamp = instanceTraffic.getLastPingTimestamp();
         if (instanceTraffic.getProperties() != null && instanceTraffic.getProperties().size() > 0) {
-            this.properties = instanceTraffic.getProperties();
+            instanceTraffic.getProperties().entrySet().forEach(it -> this.properties.add(it.getKey(), it.getValue()));

Review Comment:
   > In today's k8s monitoring, the instance is pod-oriented. In which case, you could create an instance properties update scenario? Could you share that?
   
   I was no saying my case. I was saying the CI test result, which shows the instance properties are overridden. I'll check more details



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r885701746


##########
oap-server/server-starter/src/main/resources/otel-oc-rules/k8s-instance.yaml:
##########
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+filter: "{ tags -> tags.job_name in [ 'kubernetes-cadvisor', 'kube-state-metrics' ] }" # The OpenTelemetry job name
+expSuffix: |-
+  service(['cluster' , 'service'], '::', Layer.K8S_SERVICE)
+  .instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, { tags -> ['pod': tags.pod, 'namespace': tags.namespace] })
+metricPrefix: k8s_service_instance
+metricsRules:
+  - name: pod_instance_status
+    exp: kube_pod_status_phase.retagByK8sMeta('service' , K8sRetagType.Pod2Service , 'pod' , 'namespace').tagNotEqual('service' , '').sum(['cluster', 'namespace', 'service' , 'pod'])

Review Comment:
   @wankai123 What do you mean? This is for a streaming to add instance to represent a pod.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#issuecomment-1143072014

   BTW, could you add e2e verification to do properties check on the mesh case? (We don't have k8s monitoring e2e, I think)
   Is current check engine working well for this list check?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] kezhenxu94 commented on pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#issuecomment-1143051948

   Then this is ready to review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] kezhenxu94 commented on pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#issuecomment-1143058510

   Seems the instance properties are not updated incrementally but overridden instead. That might need more modifications..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r885687439


##########
oap-server/server-starter/src/main/resources/otel-oc-rules/k8s-instance.yaml:
##########
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+filter: "{ tags -> tags.job_name in [ 'kubernetes-cadvisor', 'kube-state-metrics' ] }" # The OpenTelemetry job name
+expSuffix: |-
+  service(['cluster' , 'service'], '::', Layer.K8S_SERVICE)
+  .instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, { tags -> ['pod': tags.pod, 'namespace': tags.namespace] })
+metricPrefix: k8s_service_instance
+metricsRules:
+  - name: pod_instance_status
+    exp: kube_pod_status_phase.retagByK8sMeta('service' , K8sRetagType.Pod2Service , 'pod' , 'namespace').tagNotEqual('service' , '').sum(['cluster', 'namespace', 'service' , 'pod'])

Review Comment:
   This is a demo of how the extractor works (and will be used in on demand logs), my concern here is that we have to generate an unused instance metrics here, I'm trying to add a drop mechanism in MAL so that we can leverage the "side effect" but not store the metrics, would like to know what do you think @wu-sheng @wankai123 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886264935


##########
oap-server/server-starter/src/main/resources/otel-oc-rules/k8s-instance.yaml:
##########
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+filter: "{ tags -> tags.job_name in [ 'kubernetes-cadvisor', 'kube-state-metrics' ] }" # The OpenTelemetry job name
+expSuffix: |-
+  service(['cluster' , 'service'], '::', Layer.K8S_SERVICE)
+  .instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, { tags -> ['pod': tags.pod, 'namespace': tags.namespace] })
+metricPrefix: k8s_service_instance
+metricsRules:
+  - name: pod_instance_status
+    exp: kube_pod_status_phase.retagByK8sMeta('service' , K8sRetagType.Pod2Service , 'pod' , 'namespace').tagNotEqual('service' , '').sum(['cluster', 'namespace', 'service' , 'pod'])

Review Comment:
   > I think we don't expose enough API for this, no matter what we do to drop, it is a kind of hijacking.
   > 
   > I don't have much concern about this metric actually. This just adds an extra 1% metric :)
   
   Hi, I didn't mean to drop the metrics in the streaming process, I meant to drop the metrics just before MAL sends those metrics, see my changes in https://github.com/apache/skywalking/pull/9151/commits/1db9e71c5401a9a4b91536e7b3b397cd7a6769c8 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886266807


##########
oap-server/server-starter/src/main/resources/otel-oc-rules/k8s-instance.yaml:
##########
@@ -17,6 +17,7 @@ filter: "{ tags -> tags.job_name in [ 'kubernetes-cadvisor', 'kube-state-metrics
 expSuffix: |-
   service(['cluster' , 'service'], '::', Layer.K8S_SERVICE)
   .instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, { tags -> ['pod': tags.pod, 'namespace': tags.namespace] })
+  .drop()

Review Comment:
   `drop` is making this strange. 2 perspective
   1. User would ask why we generate a metric and then drop
   2. Meter system engine is still running for this metric. AKA, this `drop` only means not aggregation.
   
   How about we keep this metric for now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9151: Add instance properties extractor in MAL

Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9151:
URL: https://github.com/apache/skywalking/pull/9151#discussion_r886287258


##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/instance/InstanceTraffic.java:
##########
@@ -79,7 +79,7 @@ public boolean combine(final Metrics metrics) {
         final InstanceTraffic instanceTraffic = (InstanceTraffic) metrics;
         this.lastPingTimestamp = instanceTraffic.getLastPingTimestamp();
         if (instanceTraffic.getProperties() != null && instanceTraffic.getProperties().size() > 0) {
-            this.properties = instanceTraffic.getProperties();
+            instanceTraffic.getProperties().entrySet().forEach(it -> this.properties.add(it.getKey(), it.getValue()));

Review Comment:
   In today's k8s monitoring, the instance is pod-oriented. In which case, you could create an instance properties update scenario? Could you share that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org