You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/09/15 06:22:51 UTC

[GitHub] [cloudstack] soreana opened a new pull request #4438: Prometheus exporter enhancement

soreana opened a new pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438


   ## Description
   <!--- Describe your changes in detail -->
   
   In this pull request, I added new functionality to Cloudstack prometheus exporter. To see the differences please check the testing section.
   
   <!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
   <!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
   <!-- Fixes: # -->
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   
   ## How Has This Been Tested?
   <!-- Please describe in detail how you tested your changes. -->
   <!-- Include details of your testing environment, and the tests you ran to -->
   <!-- see how your change affects other areas of the code, etc. -->
   
   This pull request contains seven commits. Except for the dfb35e5224 commit, they are all added new functionality to the Prometheus. In the subsequent sections, I will describe every commit functionality. I tested them in my test environment with three management servers, one DB node (MySQL), and two KVM hypervisor.
   
   ### 1. Export count of total/up/down hosts by tags 0dbe9e78a3660bef73451c6d56f4826509833f2b
   
   1. Enable Prometheus.
   2. Add tag to the host.
   3. Run `curl http://127.0.0.1:9595/metrics | grep cloudstack_hosts_total`
   
   Output Before Changes:
   ```
   cloudstack_hosts_total{zone="mgt122-60",filter="online"} 2
   cloudstack_hosts_total{zone="mgt122-60",filter="offline"} 0
   cloudstack_hosts_total{zone="mgt122-60",filter="total"} 2
   ```
   
   Output After Changes:
   ```
   cloudstack_hosts_total{zone="mgt122-60",filter="online"} 2
   cloudstack_hosts_total{zone="mgt122-60",filter="offline"} 0
   cloudstack_hosts_total{zone="mgt122-60",filter="total"} 2
   cloudstack_hosts_total{zone="mgt122-60",filter="total",tags="tage1"} 1
   cloudstack_hosts_total{zone="mgt122-60",filter="online",tags="tage1"} 1
   cloudstack_hosts_total{zone="mgt122-60",filter="offline",tags="tage1"} 0 
   ```
   ### 2. Export count of vms by state and host tag e6a81d16d9f11db6bb4fd2b0ab38194961ce516b
   
   1. Enable Prometheus.
   2. Add tag to the host.
   3. Run `curl http://127.0.0.1:9595/metrics | grep cloudstack_vms_total_by_tag`
   
   After changes, the following line added to the Prometheus output:
   ```
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="starting",tags="tage1"} 0
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="running",tags="tage1"} 0
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="stopping",tags="tage1"} 0
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="stopped",tags="tage1"} 0
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="destroyed",tags="tage1"} 0
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="expunging",tags="tage1"} 0
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="migrating",tags="tage1"} 0
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="error",tags="tage1"} 0
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="unknown",tags="tage1"} 0
   cloudstack_vms_total_by_tag{zone="mgt122-60",filter="shutdown",tags="tage1"} 0
   ```
   
   ### 3. Add host tags to host cpu/cores/memory usage in Prometheus exporter eefd9f197352653f74aff73ccfffc4dd86d56b0d
   
   1. Enable Prometheus.
   2. Add tag to the host.
   3. Run following command and justify output with the expected results. `curl http://127.0.0.1:9595/metrics | grep cloudstack_host_vms_cores_total`
   4. repeat step three for `cloudstack_host_cpu_usage_mhz_total` and `cloudstack_host_memory_usage_mibs_total`
   
   Output Before Changes:
   ```
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node75",ip="10.135.122.75",filter="used",dedicated="0"} 2
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node75",ip="10.135.122.75",filter="total",dedicated="0"} 4
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node74",ip="10.135.122.74",filter="used",dedicated="0"} 2
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node74",ip="10.135.122.74",filter="total",dedicated="0"} 4
   cloudstack_host_vms_cores_total\{zone="mgt122-60",filter="allocated"} 4
   cloudstack_host_vms_cores_total_by_tag\{zone="mgt122-60",filter="allocated",tags="tage1"} 0
   ```
   
   Output After Changes:
   ```
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node75",ip="10.135.122.75",filter="used",dedicated="0",tags="tage1"} 2
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node75",ip="10.135.122.75",filter="total",dedicated="0",tags="tage1"} 4
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node74",ip="10.135.122.74",filter="used",dedicated="0",tags=""} 2
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node74",ip="10.135.122.74",filter="total",dedicated="0",tags=""} 4
   cloudstack_host_vms_cores_total\{zone="mgt122-60",filter="allocated"} 4 
   ```
   
   ### 4. Cloudstack Prometheus exporter: Add allocated capacity group by host tag. a489e3c6b269279df5fbff32a708d9ed0296a40e
   
   1. Enable Prometheus.
   2. Add tag to the host.
   3. Run `curl http://127.0.0.1:9595/metrics | grep cloudstack_host_vms_cores_total`
   
   Output Before Changes:
   ```
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node75",ip="10.135.122.75",filter="used",dedicated="0"} 2
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node75",ip="10.135.122.75",filter="total",dedicated="0"} 4
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node74",ip="10.135.122.74",filter="used",dedicated="0"} 2
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node74",ip="10.135.122.74",filter="total",dedicated="0"} 4
   cloudstack_host_vms_cores_total\{zone="mgt122-60",filter="allocated"} 4
   cloudstack_host_vms_cores_total_by_tag\{zone="mgt122-60",filter="allocated",tags="tage1"} 0
   ```
   
   Output After Changes:
   ```
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node75",ip="10.135.122.75",filter="used",dedicated="0",tags="tage1"} 2
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node75",ip="10.135.122.75",filter="total",dedicated="0",tags="tage1"} 4
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node74",ip="10.135.122.74",filter="used",dedicated="0",tags=""} 2
   cloudstack_host_vms_cores_total\{zone="mgt122-60",hostname="node74",ip="10.135.122.74",filter="total",dedicated="0",tags=""} 4
   cloudstack_host_vms_cores_total\{zone="mgt122-60",filter="allocated"} 4
   cloudstack_host_vms_cores_total_by_tag\{zone="mgt122-60",filter="allocated",tags="tage1"} 0
   ```
   
   ### 5. Show count of Active domains on grafana de08479da13b7b3f3eb467fc3798c6734f0e6fb7
   
   ============== Scenario One ==============
   1. Enable Prometheus.
   2. Run `curl http://127.0.0.1:9595/metrics | grep cloudstack_active_domains_total`. Output is:
   ```
   cloudstack_active_domains_total{zone="mgt122-60"} 1
   ```
   3. Create a new domain
   4. Repeat step two. The output will not change.
   5. Add a new account to the domain created in step three.
   6. Repeat step two. The output will change to:
   ```
   cloudstack_active_domains_total{zone="mgt122-60"} 2
   ```
   ============== Scenario Two ==============
   1. Use previous environment
   2. Disable all account in domain created in step 3 of Scenario one.
   3. Repeat step two of Scenario one. The output will change to:
   ``` 
   cloudstack_active_domains_total{zone="mgt122-60"} 1
   ```
   
   ### 6. Show count of Active accounts and vms by size on grafana d7aa19f0f850dfd5eea5c4f51a6529d39c2daf88
   ============== Scenario One ==============
   1. Enable Prometheus.
   2. Run `curl http://127.0.0.1:9595/metrics | grep cloudstack_active_accounts_total`. output is:
   ```
   cloudstack_active_accounts_total{zone="mgt122-60"} 1
   ```
   3. Create a new account
   4. Repeat step two. The output will change to:
   ```
   cloudstack_active_accounts_total\{zone="mgt122-60"} 2
   ```
   ============== Scenario Two ==============
   1. Enable Prometheus.
   2. Run `curl http://127.0.0.1:9595/metrics | grep cloudstack_vms_total_by_size`. output is:
   ```
   cloudstack_vms_total_by_size\{zone="mgt122-60",cpu="1",memory="512"} 2
   ```
   3. Add new instance with different offering
   4. Repeat step two. The output will change to:
   ```
   cloudstack_vms_total_by_size{zone="mgt122-60",cpu="1",memory="512"} 2
   cloudstack_vms_total_by_size\{zone="mgt122-60",cpu="1",memory="1024"} 1
   ```
   
   <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/master/CONTRIBUTING.md) document -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-925663618


   @sureshanaparti thanks!
   
   I have added the spaces.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-993205624


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-993360703


   @sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-919734933


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-921025789


   <b>Trillian test result (tid-2077)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 41441 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4438-t2077-kvm-centos7.zip
   Smoke tests completed. 85 look OK, 4 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_01_add_primary_storage_disabled_host | `Error` | 1.22 | test_primary_storage.py
   test_01_primary_storage_nfs | `Error` | 0.13 | test_primary_storage.py
   ContextSuite context=TestStorageTags>:setup | `Error` | 0.23 | test_primary_storage.py
   test_02_list_snapshots_with_removed_data_store | `Error` | 1.31 | test_snapshots.py
   test_01_secure_vm_migration | `Error` | 164.57 | test_vm_life_cycle.py
   test_02_unsecure_vm_migration | `Error` | 276.33 | test_vm_life_cycle.py
   test_03_secured_to_nonsecured_vm_migration | `Error` | 148.04 | test_vm_life_cycle.py
   test_08_migrate_vm | `Error` | 44.80 | test_vm_life_cycle.py
   test_hostha_enable_ha_when_host_in_maintenance | `Error` | 307.19 | test_hostha_kvm.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-923886665


   @nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] GutoVeronezi commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
GutoVeronezi commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r714714855



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -423,17 +524,23 @@ public String toMetricsString() {
         String zoneUuid;
         String state;
         int total;
+        String hosttags;
 
-        public ItemHost(final String zn, final String zu, final String st, int cnt) {
+        public ItemHost(final String zn, final String zu, final String st, int cnt, final String tags) {
             super("cloudstack_hosts_total");
             zoneName = zn;
             zoneUuid = zu;
             state = st;
             total = cnt;
+            hosttags = tags;
         }
 
         @Override
         public String toMetricsString() {
+            if (! Strings.isNullOrEmpty(hosttags)) {

Review comment:
       @soreana, thanks!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-923818191


   <b>Trillian test result (tid-2135)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 50191 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4438-t2135-kvm-centos7.zip
   Smoke tests completed. 90 look OK, 3 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_deploy_vm_start_failure | `Error` | 61.27 | test_deploy_vm.py
   test_deploy_vm_volume_creation_failure | `Error` | 61.36 | test_deploy_vm.py
   test_vm_ha | `Error` | 59.33 | test_vm_ha.py
   test_vm_sync | `Error` | 129.03 | test_vm_sync.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-895413003


   @nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-926333580


   @sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-923228903


   @nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-895430066


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 814


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-1044149630


   @nvazquez I was working on the https://github.com/apache/cloudstack/pull/4774 this week. It is almost complete. Will pick this one next week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-1065800649


   Hi @soreana please advise when this one is ready, thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-920552261


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-919041460


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 1237


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-919735503


   @sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-872352305


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-895412836


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r768479275



##########
File path: engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java
##########
@@ -802,6 +802,27 @@ public Long countByZoneAndState(long zoneId, State state) {
         return customSearch(sc, null).get(0);
     }
 
+    @Override
+    public Long countByZoneAndStateAndHostTag(long dcId, State state, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        String sql = "SELECT COUNT(1) FROM vm_instance vi "
+                + "JOIN service_offering so ON vi.service_offering_id=so.id "
+                + "JOIN vm_template vt ON vi.vm_template_id = vt.id "
+                + "WHERE vi.data_center_id= " + dcId + " AND vi.state = '" + state + "' AND vi.removed IS NULL "
+                + "AND (so.host_tag='" + hostTag + "' OR vt.template_tag='" + hostTag + "')";

Review comment:
       I'm working on this one. I'm not good at writing search criteria. I hope I finish that later today or tomorrow. 🤞 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r770514716



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -201,6 +252,18 @@ private void addVMMetrics(final List<Item> metricsList, final long dcId, final S
             }
             metricsList.add(new ItemVM(zoneName, zoneUuid, state.name().toLowerCase(), count));
         }
+
+        List<String> allHostTags = hostDao.listAll().stream()
+                .flatMap( h -> _hostTagsDao.gethostTags(h.getId()).stream())
+                .distinct()
+                .collect(Collectors.toList());
+
+        for (final State state : State.values()) {
+            for (final String hosttag : allHostTags) {
+                final Long count = vmDao.countByZoneAndStateAndHostTag(dcId, state, hosttag);
+                metricsList.add(new ItemVMByTag(zoneName, zoneUuid, state.name().toLowerCase(), count, hosttag));
+            }
+        }

Review comment:
       should be extracted

##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -131,6 +142,24 @@ private void addHostMetrics(final List<Item> metricsList, final long dcId, final
             int isDedicated = (dr != null) ? 1 : 0;
             metricsList.add(new ItemHostIsDedicated(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), isDedicated));
 
+            List<String> hostTags = _hostTagsDao.gethostTags(host.getId());
+            String hosttags = StringUtils.join(hostTags, ",");
+            for (String tag : hostTags) {
+                Integer current = totalHosts.get(tag) != null ? totalHosts.get(tag) : 0;
+                totalHosts.put(tag, current + 1);
+            }
+            if (host.getStatus() == Status.Up) {
+                for (String tag : hostTags) {
+                    Integer current = upHosts.get(tag) != null ? upHosts.get(tag) : 0;
+                    upHosts.put(tag, current + 1);
+                }
+            } else if (host.getStatus() == Status.Disconnected || host.getStatus() == Status.Down) {
+                for (String tag : hostTags) {
+                    Integer current = downHosts.get(tag) != null ? downHosts.get(tag) : 0;
+                    downHosts.put(tag, current + 1);
+                }
+            }

Review comment:
       i feel i have seen this code before and made some suggestions to extract it.
   ```suggestion
               String hosttags = markTagMaps(totalHosts, upHosts, downHosts);
   ```
   ```
       private String markTagMaps(Map<String, Integer> totalHost, Map<String, Integer> upHost, Map<String, Integer> downHost) {
               List<String> hostTags = _hostTagsDao.gethostTags(host.getId());
               String hosttags = StringUtils.join(hostTags, ",");
               markTags(hostTags, totalHosts);
               if (host.getStatus() == Status.Up) {
                   markTags(hostTags, upHosts);
               } else if (host.getStatus() == Status.Disconnected || host.getStatus() == Status.Down) {
                   markTags(hostTags, downHosts);
               }
       }
   ```
   and
   ```
       private void markTags(List<String> tags, Map<String, Integer> tagMap) {
                   for (String tag : tags) {
                       Integer current = tagMap.get(tag) != null ? tagMap.get(tag) : 0;
                       tagMap.put(tag, current + 1);
                   }
       }
   ```
   
   further on n the method some extractions should be made as well

##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -144,53 +173,75 @@ private void addHostMetrics(final List<Item> metricsList, final long dcId, final
             final String cpuFactor = String.valueOf(CapacityManager.CpuOverprovisioningFactor.valueIn(host.getClusterId()));
             final CapacityVO cpuCapacity = capacityDao.findByHostIdType(host.getId(), Capacity.CAPACITY_TYPE_CPU);
             if (cpuCapacity != null) {
-                metricsList.add(new ItemHostCpu(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), cpuFactor, USED, cpuCapacity.getUsedCapacity()));
-                metricsList.add(new ItemHostCpu(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), cpuFactor, TOTAL, cpuCapacity.getTotalCapacity()));
+                metricsList.add(new ItemHostCpu(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), cpuFactor, USED, cpuCapacity.getUsedCapacity(), isDedicated, hosttags));
+                metricsList.add(new ItemHostCpu(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), cpuFactor, TOTAL, cpuCapacity.getTotalCapacity(), isDedicated, hosttags));
             } else {
-                metricsList.add(new ItemHostCpu(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), cpuFactor, USED, 0L));
-                metricsList.add(new ItemHostCpu(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), cpuFactor, TOTAL, 0L));
+                metricsList.add(new ItemHostCpu(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), cpuFactor, USED, 0L, isDedicated, hosttags));
+                metricsList.add(new ItemHostCpu(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), cpuFactor, TOTAL, 0L, isDedicated, hosttags));
             }
 
             final String memoryFactor = String.valueOf(CapacityManager.MemOverprovisioningFactor.valueIn(host.getClusterId()));
             final CapacityVO memCapacity = capacityDao.findByHostIdType(host.getId(), Capacity.CAPACITY_TYPE_MEMORY);
             if (memCapacity != null) {
-                metricsList.add(new ItemHostMemory(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), memoryFactor, USED, memCapacity.getUsedCapacity(), isDedicated));
-                metricsList.add(new ItemHostMemory(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), memoryFactor, TOTAL, memCapacity.getTotalCapacity(), isDedicated));
+                metricsList.add(new ItemHostMemory(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), memoryFactor, USED, memCapacity.getUsedCapacity(), isDedicated, hosttags));
+                metricsList.add(new ItemHostMemory(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), memoryFactor, TOTAL, memCapacity.getTotalCapacity(), isDedicated, hosttags));
             } else {
-                metricsList.add(new ItemHostMemory(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), memoryFactor, USED, 0L, isDedicated));
-                metricsList.add(new ItemHostMemory(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), memoryFactor, TOTAL, 0L, isDedicated));
+                metricsList.add(new ItemHostMemory(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), memoryFactor, USED, 0L, isDedicated, hosttags));
+                metricsList.add(new ItemHostMemory(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), memoryFactor, TOTAL, 0L, isDedicated, hosttags));
             }
 
             metricsList.add(new ItemHostVM(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), vmDao.listByHostId(host.getId()).size()));
 
             final CapacityVO coreCapacity = capacityDao.findByHostIdType(host.getId(), Capacity.CAPACITY_TYPE_CPU_CORE);
             if (coreCapacity != null) {
-                metricsList.add(new ItemVMCore(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), USED, coreCapacity.getUsedCapacity(), isDedicated));
-                metricsList.add(new ItemVMCore(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), TOTAL, coreCapacity.getTotalCapacity(), isDedicated));
+                metricsList.add(new ItemVMCore(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), USED, coreCapacity.getUsedCapacity(), isDedicated, hosttags));
+                metricsList.add(new ItemVMCore(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), TOTAL, coreCapacity.getTotalCapacity(), isDedicated, hosttags));
             } else {
-                metricsList.add(new ItemVMCore(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), USED, 0L, isDedicated));
-                metricsList.add(new ItemVMCore(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), TOTAL, 0L, isDedicated));
+                metricsList.add(new ItemVMCore(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), USED, 0L, isDedicated, hosttags));
+                metricsList.add(new ItemVMCore(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), TOTAL, 0L, isDedicated, hosttags));
             }
         }
 
         final List<CapacityDaoImpl.SummedCapacity> cpuCapacity = capacityDao.findCapacityBy((int) Capacity.CAPACITY_TYPE_CPU, dcId, null, null);
         if (cpuCapacity != null && cpuCapacity.size() > 0) {
-            metricsList.add(new ItemHostCpu(zoneName, zoneUuid, null, null, null, null, ALLOCATED, cpuCapacity.get(0).getAllocatedCapacity() != null ? cpuCapacity.get(0).getAllocatedCapacity() : 0));
+            metricsList.add(new ItemHostCpu(zoneName, zoneUuid, null, null, null, null, ALLOCATED, cpuCapacity.get(0).getAllocatedCapacity() != null ? cpuCapacity.get(0).getAllocatedCapacity() : 0, 0, ""));
         }
 
         final List<CapacityDaoImpl.SummedCapacity> memCapacity = capacityDao.findCapacityBy((int) Capacity.CAPACITY_TYPE_MEMORY, dcId, null, null);
         if (memCapacity != null && memCapacity.size() > 0) {
-            metricsList.add(new ItemHostMemory(zoneName, zoneUuid, null, null, null, null, ALLOCATED, memCapacity.get(0).getAllocatedCapacity() != null ? memCapacity.get(0).getAllocatedCapacity() : 0, 0));
+            metricsList.add(new ItemHostMemory(zoneName, zoneUuid, null, null, null, null, ALLOCATED, memCapacity.get(0).getAllocatedCapacity() != null ? memCapacity.get(0).getAllocatedCapacity() : 0, 0, ""));
         }
 
         final List<CapacityDaoImpl.SummedCapacity> coreCapacity = capacityDao.findCapacityBy((int) Capacity.CAPACITY_TYPE_CPU_CORE, dcId, null, null);
         if (coreCapacity != null && coreCapacity.size() > 0) {
-            metricsList.add(new ItemVMCore(zoneName, zoneUuid, null, null, null, ALLOCATED, coreCapacity.get(0).getAllocatedCapacity() != null ? coreCapacity.get(0).getAllocatedCapacity() : 0, 0));
+            metricsList.add(new ItemVMCore(zoneName, zoneUuid, null, null, null, ALLOCATED, coreCapacity.get(0).getAllocatedCapacity() != null ? coreCapacity.get(0).getAllocatedCapacity() : 0, 0, ""));
+        }
+
+        metricsList.add(new ItemHost(zoneName, zoneUuid, ONLINE, up, null));
+        metricsList.add(new ItemHost(zoneName, zoneUuid, OFFLINE, down, null));
+        metricsList.add(new ItemHost(zoneName, zoneUuid, TOTAL, total, null));
+        for (Map.Entry<String, Integer> entry : totalHosts.entrySet()) {
+            String tag = entry.getKey();
+            Integer count = entry.getValue();
+            metricsList.add(new ItemHost(zoneName, zoneUuid, TOTAL, count, tag));
+            if (upHosts.get(tag) != null) {
+                metricsList.add(new ItemHost(zoneName, zoneUuid, ONLINE, upHosts.get(tag), tag));
+            } else {
+                metricsList.add(new ItemHost(zoneName, zoneUuid, ONLINE, 0, tag));
+            }
+            if (downHosts.get(tag) != null) {
+                metricsList.add(new ItemHost(zoneName, zoneUuid, OFFLINE, downHosts.get(tag), tag));
+            } else {
+                metricsList.add(new ItemHost(zoneName, zoneUuid, OFFLINE, 0, tag));
+            }
+        }
+        for (Map.Entry<String, Integer> entry : totalHosts.entrySet()) {
+            String tag = entry.getKey();
+            Ternary<Long, Long, Long> allocatedCapacityByTag = capacityDao.findCapacityByZoneAndHostTag(dcId, tag);
+            metricsList.add(new ItemVMCore(zoneName, zoneUuid, null, null, null, ALLOCATED, allocatedCapacityByTag.first(), 0, tag));
+            metricsList.add(new ItemHostCpu(zoneName, zoneUuid, null, null, null, null, ALLOCATED, allocatedCapacityByTag.second(),0, tag));
+            metricsList.add(new ItemHostMemory(zoneName, zoneUuid, null, null, null, null, ALLOCATED, allocatedCapacityByTag.third(), 0, tag));

Review comment:
       thsi should be extracted




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] GutoVeronezi commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
GutoVeronezi commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r714719902



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -460,14 +567,20 @@ public ItemVMCore(final String zn, final String zu, final String hn, final Strin
                 core = cr;
             }
             isDedicated = dedicated;
+            hosttags = tags;
         }
 
         @Override
         public String toMetricsString() {
-            if (Strings.isNullOrEmpty(hostName) && Strings.isNullOrEmpty(ip)) {
-                return String.format("%s{zone=\"%s\",filter=\"%s\"} %d", name, zoneName, filter, core);
+            if (StringUtils.isEmpty(hostName) && StringUtils.isEmpty(ip)) {

Review comment:
       We could use `StringUtils#isAllEmpty`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r714776010



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -524,14 +647,20 @@ public ItemHostMemory(final String zn, final String zu, final String hn, final S
             filter = fl;
             miBytes = membytes / (1024.0 * 1024.0);
             isDedicated = dedicated;
+            hosttags = tags;
         }
 
         @Override
         public String toMetricsString() {
-            if (Strings.isNullOrEmpty(hostName) && Strings.isNullOrEmpty(ip)) {
-                return String.format("%s{zone=\"%s\",filter=\"%s\"} %.2f", name, zoneName, filter, miBytes);
+            if (StringUtils.isEmpty(hostName) && StringUtils.isEmpty(ip)) {

Review comment:
       Cool, I didn't know it exists :D




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] rhtyd commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-972645481


   @soreana can you fix the conflicts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-928965818


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-926347899


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 1384


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] GutoVeronezi commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
GutoVeronezi commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r714190830



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -423,17 +524,23 @@ public String toMetricsString() {
         String zoneUuid;
         String state;
         int total;
+        String hosttags;
 
-        public ItemHost(final String zn, final String zu, final String st, int cnt) {
+        public ItemHost(final String zn, final String zu, final String st, int cnt, final String tags) {
             super("cloudstack_hosts_total");
             zoneName = zn;
             zoneUuid = zu;
             state = st;
             total = cnt;
+            hosttags = tags;
         }
 
         @Override
         public String toMetricsString() {
+            if (! Strings.isNullOrEmpty(hosttags)) {

Review comment:
       @nvazquez thanks!
   
   To contextualize, recently we finished a voting defining `commons.lang3` as the default string lib ([VOTE thread](https://lists.apache.org/thread.html/r806cd10b3de645c150e5e0e3d845c5a380a700197143f57f0834d758%40%3Cdev.cloudstack.apache.org%3E) & [RESULT thread](https://lists.apache.org/thread.html/r5898f717d506a0e160b9dc766b65769e2fe2f07f9cba70825843e272%40%3Cdev.cloudstack.apache.org%3E)). The code and the docs will be changed soon.
   
   In this case, `!Strings.isNullOrEmpty(hosttags)` can be replaced with `StringUtils.isNotEmpty(hosttags)`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r714636652



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -423,17 +524,23 @@ public String toMetricsString() {
         String zoneUuid;
         String state;
         int total;
+        String hosttags;
 
-        public ItemHost(final String zn, final String zu, final String st, int cnt) {
+        public ItemHost(final String zn, final String zu, final String st, int cnt, final String tags) {
             super("cloudstack_hosts_total");
             zoneName = zn;
             zoneUuid = zu;
             state = st;
             total = cnt;
+            hosttags = tags;
         }
 
         @Override
         public String toMetricsString() {
+            if (! Strings.isNullOrEmpty(hosttags)) {

Review comment:
       
   @GutoVeronezi and @nvazquez thanks!
   
   I have refactored the code and used `commons.lang3` in [this commit](https://github.com/apache/cloudstack/pull/4438/commits/aaa1bcfa12732bd7069c31119b709a7b3416e931).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r768478844



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -115,6 +122,10 @@ private void addHostMetrics(final List<Item> metricsList, final long dcId, final
         int total = 0;
         int up = 0;
         int down = 0;
+        Map<String, Integer> total_hosts = new HashMap<>();

Review comment:
       @DaanHoogland I was thinking about refactoring the whole Prometheus plugin. If I touch one method, I have to change everything. I love to do that, but it would be out of the scope of this pr. I've created #5773 for it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-993205710


   @sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r836358399



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -131,6 +142,24 @@ private void addHostMetrics(final List<Item> metricsList, final long dcId, final
             int isDedicated = (dr != null) ? 1 : 0;
             metricsList.add(new ItemHostIsDedicated(zoneName, zoneUuid, host.getName(), host.getUuid(), host.getPrivateIpAddress(), isDedicated));
 
+            List<String> hostTags = _hostTagsDao.gethostTags(host.getId());
+            String hosttags = StringUtils.join(hostTags, ",");
+            for (String tag : hostTags) {
+                Integer current = totalHosts.get(tag) != null ? totalHosts.get(tag) : 0;
+                totalHosts.put(tag, current + 1);
+            }
+            if (host.getStatus() == Status.Up) {
+                for (String tag : hostTags) {
+                    Integer current = upHosts.get(tag) != null ? upHosts.get(tag) : 0;
+                    upHosts.put(tag, current + 1);
+                }
+            } else if (host.getStatus() == Status.Disconnected || host.getStatus() == Status.Down) {
+                for (String tag : hostTags) {
+                    Integer current = downHosts.get(tag) != null ? downHosts.get(tag) : 0;
+                    downHosts.put(tag, current + 1);
+                }
+            }

Review comment:
       Yeap, You have seen it in the https://github.com/apache/cloudstack/pull/5773 pr :)
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-872352497


   @nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-928965818


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-929522009


   <b>Trillian test result (tid-2225)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 35752 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4438-t2225-kvm-centos7.zip
   Smoke tests completed. 89 look OK, 0 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-924447821


   <b>Trillian test result (tid-2151)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 36230 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4438-t2151-kvm-centos7.zip
   Smoke tests completed. 89 look OK, 0 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r714149917



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -423,17 +524,23 @@ public String toMetricsString() {
         String zoneUuid;
         String state;
         int total;
+        String hosttags;
 
-        public ItemHost(final String zn, final String zu, final String st, int cnt) {
+        public ItemHost(final String zn, final String zu, final String st, int cnt, final String tags) {
             super("cloudstack_hosts_total");
             zoneName = zn;
             zoneUuid = zu;
             state = st;
             total = cnt;
+            hosttags = tags;
         }
 
         @Override
         public String toMetricsString() {
+            if (! Strings.isNullOrEmpty(hosttags)) {

Review comment:
       Can we use the `commons.lang3` library instead on these checks? cc. @GutoVeronezi 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-923222402


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-923928064


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 1347


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-925124484


   Ping for review @rhtyd @DaanHoogland @weizhouapache @sureshanaparti - regarding manual tests I think the manual tests provided by @soreana should be enough 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-895483586


   @nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-872375993


   @nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-983478567


   @rhtyd I've resolved the conflict. Should I rebase and make a one commit?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-983526533


   @weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-871925334


   @nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r716416502



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -115,6 +121,10 @@ private void addHostMetrics(final List<Item> metricsList, final long dcId, final
         int total = 0;
         int up = 0;
         int down = 0;
+        Map<String, Integer> total_hosts = new HashMap<>();
+        Map<String, Integer> up_hosts = new HashMap<>();
+        Map<String, Integer> down_hosts = new HashMap<>();

Review comment:
       ```suggestion
           Map<String, Integer> downHosts = new HashMap<>();
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] rhtyd commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-927683390


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r714537580



##########
File path: engine/schema/src/main/java/com/cloud/capacity/dao/CapacityDaoImpl.java
##########
@@ -411,6 +416,41 @@ public static void setValues(PreparedStatement preparedStatement, Object... valu
 
     }
 
+    @Override
+    public Ternary<Long, Long, Long> findCapacityByZoneAndHostTag(Long zoneId, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        PreparedStatement pstmt;
+
+        StringBuilder allocatedSql = new StringBuilder(LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE);
+        if (hostTag != null && ! hostTag.isEmpty()) {
+            allocatedSql.append(LEFT_JOIN_VM_TEMPLATE);
+        }
+        allocatedSql.append(WHERE_STATE_IS_NOT_DESTRUCTIVE);
+        if (zoneId != null){

Review comment:
       ```suggestion
           if (zoneId != null) {
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] GutoVeronezi commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
GutoVeronezi commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r714716487



##########
File path: engine/schema/src/main/java/com/cloud/capacity/dao/CapacityDaoImpl.java
##########
@@ -411,6 +416,41 @@ public static void setValues(PreparedStatement preparedStatement, Object... valu
 
     }
 
+    @Override
+    public Ternary<Long, Long, Long> findCapacityByZoneAndHostTag(Long zoneId, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        PreparedStatement pstmt;
+
+        StringBuilder allocatedSql = new StringBuilder(LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE);
+        if (hostTag != null && ! hostTag.isEmpty()) {

Review comment:
       We could use `StringUtils#isNotEmpty`.

##########
File path: engine/schema/src/main/java/com/cloud/capacity/dao/CapacityDaoImpl.java
##########
@@ -411,6 +416,41 @@ public static void setValues(PreparedStatement preparedStatement, Object... valu
 
     }
 
+    @Override
+    public Ternary<Long, Long, Long> findCapacityByZoneAndHostTag(Long zoneId, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        PreparedStatement pstmt;
+
+        StringBuilder allocatedSql = new StringBuilder(LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE);
+        if (hostTag != null && ! hostTag.isEmpty()) {
+            allocatedSql.append(LEFT_JOIN_VM_TEMPLATE);
+        }
+        allocatedSql.append(WHERE_STATE_IS_NOT_DESTRUCTIVE);
+        if (zoneId != null) {
+            allocatedSql.append(" AND vi.data_center_id = ?");
+        }
+        if (hostTag != null && ! hostTag.isEmpty()) {

Review comment:
       We could use `StringUtils#isNotEmpty`.

##########
File path: engine/schema/src/main/java/com/cloud/vm/dao/UserVmDaoImpl.java
##########
@@ -692,4 +703,35 @@ public boolean remove(Long id) {
 
         return vmsDetailByNames;
     }
+
+    @Override
+    public List<Ternary<Integer, Integer, Integer>> countVmsBySize(long dcId, int limit) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        String sql = "SELECT cpu,ram_size,count(1) AS count FROM (SELECT * FROM user_vm_view WHERE data_center_id = ? AND state NOT IN ('Destroyed', 'Error', 'Expunging') GROUP BY id) AS uvv GROUP BY cpu,ram_size ORDER BY count DESC ";
+        if (limit >= 0)
+            sql = sql + "limit " + limit;
+        PreparedStatement pstmt = null;
+        List<Ternary<Integer, Integer, Integer>> result = new ArrayList<>();
+        try {
+            pstmt = txn.prepareAutoCloseStatement(sql);
+            pstmt.setLong(1, dcId);
+            ResultSet rs = pstmt.executeQuery();
+            while (rs.next()) {
+                result.add(new Ternary<Integer, Integer, Integer>(rs.getInt(1), rs.getInt(2), rs.getInt(3)));
+            }
+        } catch (Exception e) {
+            s_logger.warn("Error counting vms by size", e);

Review comment:
       We could add some context to this log, like `dcId`.

##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -524,14 +647,20 @@ public ItemHostMemory(final String zn, final String zu, final String hn, final S
             filter = fl;
             miBytes = membytes / (1024.0 * 1024.0);
             isDedicated = dedicated;
+            hosttags = tags;
         }
 
         @Override
         public String toMetricsString() {
-            if (Strings.isNullOrEmpty(hostName) && Strings.isNullOrEmpty(ip)) {
-                return String.format("%s{zone=\"%s\",filter=\"%s\"} %.2f", name, zoneName, filter, miBytes);
+            if (StringUtils.isEmpty(hostName) && StringUtils.isEmpty(ip)) {

Review comment:
       We could use `StringUtils#isAllEmpty`.

##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -460,14 +567,20 @@ public ItemVMCore(final String zn, final String zu, final String hn, final Strin
                 core = cr;
             }
             isDedicated = dedicated;
+            hosttags = tags;
         }
 
         @Override
         public String toMetricsString() {
-            if (Strings.isNullOrEmpty(hostName) && Strings.isNullOrEmpty(ip)) {
-                return String.format("%s{zone=\"%s\",filter=\"%s\"} %d", name, zoneName, filter, core);
+            if (StringUtils.isEmpty(hostName) && StringUtils.isEmpty(ip)) {

Review comment:
       We could use StringUtils#isAllEmpty.

##########
File path: engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java
##########
@@ -802,6 +802,27 @@ public Long countByZoneAndState(long zoneId, State state) {
         return customSearch(sc, null).get(0);
     }
 
+    @Override
+    public Long countByZoneAndStateAndHostTag(long dcId, State state, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        String sql = "SELECT COUNT(1) FROM vm_instance vi "
+                + "JOIN service_offering so ON vi.service_offering_id=so.id "
+                + "JOIN vm_template vt ON vi.vm_template_id = vt.id "
+                + "WHERE vi.data_center_id= " + dcId + " AND vi.state = '" + state + "' AND vi.removed IS NULL "
+                + "AND (so.host_tag='" + hostTag + "' OR vt.template_tag='" + hostTag + "')";
+        PreparedStatement pstmt = null;
+        try {
+            pstmt = txn.prepareAutoCloseStatement(sql);
+            ResultSet rs = pstmt.executeQuery();
+            if (rs.next()) {
+                return rs.getLong(1);
+            }
+        } catch (Exception e) {
+            s_logger.warn("Error counting vms by host tag", e);

Review comment:
       We could add some context to this log, like `dcId` and `hostTag`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-927794103


   @soreana a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-927724407


   Packaging result: :heavy_multiplication_x: el7 :heavy_multiplication_x: el8 :heavy_multiplication_x: debian :heavy_multiplication_x: suse15. SL-JID 1405


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-928966472






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-983519520


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 1809


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-994925020


   @soreana a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-994924803


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-994971090


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 1907


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-1043767227


   Hi @soreana can you please check the outstanding comments?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-927812089


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 1410


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-928966472


   @sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-993878357


   <b>Trillian test result (tid-2657)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 30759 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4438-t2657-kvm-centos7.zip
   Smoke tests completed. 88 look OK, 3 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_DeployVmAntiAffinityGroup | `Error` | 35.48 | test_affinity_groups.py
   test_DeployVmAntiAffinityGroup_in_project | `Error` | 73.45 | test_affinity_groups_projects.py
   test_hostha_enable_ha_when_host_disabled | `Error` | 0.55 | test_hostha_kvm.py
   test_hostha_enable_ha_when_host_in_maintenance | `Error` | 303.81 | test_hostha_kvm.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-993278810


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 1889


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-927685523


   @rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] rhtyd commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-918847201


   @blueorangutan package 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-914922403


   @rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-923931714


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-872019217


   Hey @nvazquez
   Yes, it is ready for review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-872372185


   Packaging result: :heavy_check_mark: centos7 :heavy_check_mark: centos8 :heavy_check_mark: debian. SL-JID 452


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-923932599


   @nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] rhtyd commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-926569591






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti closed pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti closed pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-871958629


   Packaging result: :heavy_multiplication_x: centos7 :heavy_check_mark: centos8 :heavy_multiplication_x: debian. SL-JID 444


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-895482971


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-871925134


   Hi @soreana is this PR ready for review?
   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-872685512


   <b>Trillian test result (tid-1191)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 38732 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4438-t1191-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Smoke tests completed. 88 look OK, 0 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-927793961


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-983488074


   @weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-993359857


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] GabrielBrascher commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r776757233



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -201,6 +252,18 @@ private void addVMMetrics(final List<Item> metricsList, final long dcId, final S
             }
             metricsList.add(new ItemVM(zoneName, zoneUuid, state.name().toLowerCase(), count));
         }
+
+        List<String> allHostTags = hostDao.listAll().stream()
+                .flatMap( h -> _hostTagsDao.gethostTags(h.getId()).stream())
+                .distinct()
+                .collect(Collectors.toList());
+
+        for (final State state : State.values()) {
+            for (final String hosttag : allHostTags) {
+                final Long count = vmDao.countByZoneAndStateAndHostTag(dcId, state, hosttag);
+                metricsList.add(new ItemVMByTag(zoneName, zoneUuid, state.name().toLowerCase(), count, hosttag));
+            }
+        }

Review comment:
       @soreana @DaanHoogland +1 on extracting these code lines.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-920552476


   @nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r662942437



##########
File path: engine/schema/src/main/java/com/cloud/capacity/dao/CapacityDaoImpl.java
##########
@@ -411,14 +414,50 @@ public static void setValues(PreparedStatement preparedStatement, Object... valu
 
     }
 
+    @Override
+    public Ternary<Long, Long, Long> findCapacityByZoneAndHostTag(Long zoneId, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        PreparedStatement pstmt = null;
+        List<SummedCapacity> results = new ArrayList<SummedCapacity>();
+
+        StringBuilder allocatedSql = new StringBuilder(LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE_PART1);
+        if (hostTag != null && ! hostTag.isEmpty()) {
+            allocatedSql.append("LEFT JOIN vm_template ON vm_template.id = vi.vm_template_id ");

Review comment:
       ```suggestion
               private static final String LEFT_JOIN_VMTEMPLATE = "LEFT JOIN vm_template ON vm_template.id = vi.vm_template_id ";
   ```
   up top and here
   ```suggestion
               allocatedSql.append(LEFT_JOIN_VMTEMPLATE);
   ```

##########
File path: engine/schema/src/main/java/com/cloud/capacity/dao/CapacityDaoImpl.java
##########
@@ -187,16 +188,18 @@
             +
             "from op_host_capacity capacity where cluster_id = ? and capacity_type = ?;";
 
-    private static final String LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE = "SELECT v.data_center_id, SUM(cpu) AS cpucore, " +
+    private static final String LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE_PART1 = "SELECT v.data_center_id, SUM(cpu) AS cpucore, " +
                 "SUM(cpu * speed) AS cpu, SUM(ram_size * 1024 * 1024) AS memory " +
                 "FROM (SELECT vi.data_center_id, (CASE WHEN ISNULL(service_offering.cpu) THEN custom_cpu.value ELSE service_offering.cpu end) AS cpu, " +
                 "(CASE WHEN ISNULL(service_offering.speed) THEN custom_speed.value ELSE service_offering.speed end) AS speed, " +
                 "(CASE WHEN ISNULL(service_offering.ram_size) THEN custom_ram_size.value ELSE service_offering.ram_size end) AS ram_size " +
-                "FROM (((vm_instance vi LEFT JOIN service_offering ON(((vi.service_offering_id = service_offering.id))) " +
-                "LEFT JOIN user_vm_details custom_cpu ON(((custom_cpu.vm_id = vi.id) AND (custom_cpu.name = 'CpuNumber')))) " +
-                "LEFT JOIN user_vm_details custom_speed ON(((custom_speed.vm_id = vi.id) AND (custom_speed.name = 'CpuSpeed')))) " +
-                "LEFT JOIN user_vm_details custom_ram_size ON(((custom_ram_size.vm_id = vi.id) AND (custom_ram_size.name = 'memory')))) " +
-                "WHERE ISNULL(vi.removed) AND vi.state NOT IN ('Destroyed', 'Error', 'Expunging')";
+                "FROM vm_instance vi LEFT JOIN service_offering ON(((vi.service_offering_id = service_offering.id))) " +
+                "LEFT JOIN user_vm_details custom_cpu ON(((custom_cpu.vm_id = vi.id) AND (custom_cpu.name = 'CpuNumber'))) " +
+                "LEFT JOIN user_vm_details custom_speed ON(((custom_speed.vm_id = vi.id) AND (custom_speed.name = 'CpuSpeed'))) " +
+                "LEFT JOIN user_vm_details custom_ram_size ON(((custom_ram_size.vm_id = vi.id) AND (custom_ram_size.name = 'memory'))) ";
+
+    private static final String LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE_PART2 =

Review comment:
       and then of course
   ```suggestion
       private static final String WHERE_STATE_IS_NOT_DESTRUCTIVE =
   ```
   or something like that

##########
File path: engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java
##########
@@ -802,6 +802,27 @@ public Long countByZoneAndState(long zoneId, State state) {
         return customSearch(sc, null).get(0);
     }
 
+    @Override
+    public Long countByZoneAndStateAndHostTag(long dcId, State state, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        String sql = "SELECT COUNT(1) FROM vm_instance vi "
+                + "JOIN service_offering so ON vi.service_offering_id=so.id "
+                + "JOIN vm_template vt ON vi.vm_template_id = vt.id "
+                + "WHERE vi.data_center_id= " + dcId + " AND vi.state = '" + state + "' AND vi.removed IS NULL "
+                + "AND (so.host_tag='" + hostTag + "' OR vt.template_tag='" + hostTag + "')";

Review comment:
       Why not define this with prepared statement syntax? these parameters are very suitable for that
   ```suggestion
           String sql = "SELECT COUNT(1) FROM vm_instance vi "
                   + "JOIN service_offering so ON vi.service_offering_id=so.id "
                   + "JOIN vm_template vt ON vi.vm_template_id = vt.id "
                   + "WHERE vi.data_center_id= ? AND vi.state = ? AND vi.removed IS NULL "
                   + "AND (so.host_tag = ? OR vt.template_tag = ? )";
   ```
   and later assign `dcId`, `state`, `hostTag` and again `state`. It is saver and if executed a second time in the same db engine also quicker.

##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -115,6 +122,10 @@ private void addHostMetrics(final List<Item> metricsList, final long dcId, final
         int total = 0;
         int up = 0;
         int down = 0;
+        Map<String, Integer> total_hosts = new HashMap<>();

Review comment:
       before you touched this method it was already 80 lines, can you please refactor at least the new bits out into smaller chunks?
   thanks

##########
File path: engine/schema/src/main/java/com/cloud/capacity/dao/CapacityDaoImpl.java
##########
@@ -187,16 +188,18 @@
             +
             "from op_host_capacity capacity where cluster_id = ? and capacity_type = ?;";
 
-    private static final String LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE = "SELECT v.data_center_id, SUM(cpu) AS cpucore, " +
+    private static final String LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE_PART1 = "SELECT v.data_center_id, SUM(cpu) AS cpucore, " +

Review comment:
       now we are touching this anyway, 
   ```suggestion
       private static final String SELECT_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE = "SELECT v.data_center_id, SUM(cpu) AS cpucore, " +
   ```
   is maybe a better name




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-927939021


   Could someone run the test?
   It seems that @blueorangutan is angry with me. :D


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-872375407


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r714536656



##########
File path: engine/schema/src/main/java/com/cloud/capacity/dao/CapacityDaoImpl.java
##########
@@ -411,6 +416,41 @@ public static void setValues(PreparedStatement preparedStatement, Object... valu
 
     }
 
+    @Override
+    public Ternary<Long, Long, Long> findCapacityByZoneAndHostTag(Long zoneId, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        PreparedStatement pstmt;
+
+        StringBuilder allocatedSql = new StringBuilder(LIST_ALLOCATED_CAPACITY_GROUP_BY_CAPACITY_AND_ZONE);
+        if (hostTag != null && ! hostTag.isEmpty()) {
+            allocatedSql.append(LEFT_JOIN_VM_TEMPLATE);
+        }
+        allocatedSql.append(WHERE_STATE_IS_NOT_DESTRUCTIVE);
+        if (zoneId != null){
+            allocatedSql.append(" AND vi.data_center_id = ?");
+        }
+        if (hostTag != null && ! hostTag.isEmpty()) {
+            allocatedSql.append(" AND (vm_template.template_tag = '").append(hostTag).append("'");
+            allocatedSql.append(" OR service_offering.host_tag = '").append(hostTag).append("')");
+        }
+        allocatedSql.append(" ) AS v GROUP BY v.data_center_id");
+
+        try {
+            // add allocated capacity of zone in result
+            pstmt = txn.prepareAutoCloseStatement(allocatedSql.toString());
+            if (zoneId != null){

Review comment:
       ```suggestion
               if (zoneId != null) {
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-914965839


   Packaging result: :heavy_multiplication_x: el7 :heavy_check_mark: el8 :heavy_multiplication_x: debian :heavy_check_mark: suse15. SL-JID 1165


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-927829827


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-926569993


   @rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] weizhouapache commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-983524470


   @blueorangutan test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-984001082


   <b>Trillian test result (tid-2600)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 30460 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4438-t2600-kvm-centos7.zip
   Smoke tests completed. 91 look OK, 0 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-1081531573


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] rhtyd commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-914921541


   @blueorangutan package 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-918848245


   @rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] nvazquez commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-923885652


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r716416355



##########
File path: plugins/integrations/prometheus/src/main/java/org/apache/cloudstack/metrics/PrometheusExporterImpl.java
##########
@@ -115,6 +121,10 @@ private void addHostMetrics(final List<Item> metricsList, final long dcId, final
         int total = 0;
         int up = 0;
         int down = 0;
+        Map<String, Integer> total_hosts = new HashMap<>();
+        Map<String, Integer> up_hosts = new HashMap<>();

Review comment:
       ```suggestion
           Map<String, Integer> upHosts = new HashMap<>();
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r761753466



##########
File path: engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java
##########
@@ -802,6 +802,27 @@ public Long countByZoneAndState(long zoneId, State state) {
         return customSearch(sc, null).get(0);
     }
 
+    @Override
+    public Long countByZoneAndStateAndHostTag(long dcId, State state, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        String sql = "SELECT COUNT(1) FROM vm_instance vi "
+                + "JOIN service_offering so ON vi.service_offering_id=so.id "
+                + "JOIN vm_template vt ON vi.vm_template_id = vt.id "
+                + "WHERE vi.data_center_id= " + dcId + " AND vi.state = '" + state + "' AND vi.removed IS NULL "
+                + "AND (so.host_tag='" + hostTag + "' OR vt.template_tag='" + hostTag + "')";

Review comment:
       @soreana can you answer me on my question above?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on a change in pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on a change in pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#discussion_r769760141



##########
File path: engine/schema/src/main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java
##########
@@ -802,6 +802,27 @@ public Long countByZoneAndState(long zoneId, State state) {
         return customSearch(sc, null).get(0);
     }
 
+    @Override
+    public Long countByZoneAndStateAndHostTag(long dcId, State state, String hostTag) {
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        String sql = "SELECT COUNT(1) FROM vm_instance vi "
+                + "JOIN service_offering so ON vi.service_offering_id=so.id "
+                + "JOIN vm_template vt ON vi.vm_template_id = vt.id "
+                + "WHERE vi.data_center_id= " + dcId + " AND vi.state = '" + state + "' AND vi.removed IS NULL "
+                + "AND (so.host_tag='" + hostTag + "' OR vt.template_tag='" + hostTag + "')";

Review comment:
       @DaanHoogland I gave up using search criteria and used prepared statements. SearchCriteria with two join statements were over complicated for me.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-926333222


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-926964144


   <b>Trillian test result (tid-2205)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 38829 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4438-t2205-kvm-centos7.zip
   Smoke tests completed. 89 look OK, 0 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] soreana commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
soreana commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-926594213


   > @soreana can you share how this was tested and any test report/cases/logs?
   
   @rhtyd The First six commits in this pr contain changes to the Prometheus exporter. For each commit, I wrote a test case in the pr description. Those are actual steps I followed when I was testing my changes. You don't have to build each commit separately to test it. The remaining commits contain changes requested by the community.
   
   We have this code in our production environment and have used this output for years. For example, we used numbers 3 and 4 to monitor our host allocation statistics based on their tags.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] sureshanaparti commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-925565331


   > Ping for review @rhtyd @DaanHoogland @weizhouapache @sureshanaparti - regarding manual tests I think the manual tests provided by @soreana should be enough
   
   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-1081573925


   Packaging result: :heavy_multiplication_x: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 3003


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [cloudstack] blueorangutan commented on pull request #4438: Prometheus exporter enhancement

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4438:
URL: https://github.com/apache/cloudstack/pull/4438#issuecomment-1081533534


   @soreana a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org