[GitHub] GabrielBrascher opened a new pull request #3078: Add influxdb to statscollector

[GitBox <gi...@apache.org>]

GabrielBrascher opened a new pull request #3078: Add influxdb to statscollector
URL: https://github.com/apache/cloudstack/pull/3078
 
 
   ## Description
   <!--- Describe your changes in detail -->
   Add Support for InfluxDB on StatsCollector.
   
   The collector is defined on the URI scheme. The _Global Settings_ configuration with the name `stats.output.uri` is used to set the URI to send StatsCollector statistics to.
   
   Example: `graphite://graphite-hostaddress:port` or `influxdb://influxdb-hostaddress/dbname`. Note that the port is optional, if not added the default port for the respective collector (graphite or influxdb) will be used. Additionally, the database name `dbname` is  also optional; default db name is `cloudstack`. Note that one  must create and configure the database if using influxdb.
   
   - Host stats
   For the host metrics it creates an InfluxDB point with the measure `host_stats` and containing the following tags and fields:
   Tags:
   -- host_id
   -- uuid
   -- pod_id
   -- data_center_id
   Fields:
   -- total_memory_kbs
   -- free_memory_kbs
   -- cpu_utilization
   -- cpus
   -- cpu_sockets
   -- network_read_kbs
   -- network_write_kbs
   
   - VM stats
   It creates an InfluxDB point with the measure `vm_stats` and containing the following tags and fields:
   Tags:
   -- vm_id
   -- uuid
   -- pod_id
   -- data_center_id
   -- host_id
   Fields:
   -- total_memory_kbs
   -- free_memory_kbs
   -- memory_target_kbs
   -- cpu_utilization
   -- cpus
   -- network_read_kbs
   -- network_write_kbs
   -- disk_read_kbs
   -- disk_read_iops
   -- disk_write_kbs
   -- disk_write_iops
   
   **Note:** As the current stats collector supports only KVM, this extension supports only KVM. I am planning on enhance and refactor the StatsCollector architecture on the future. Due to the inner class structure, test case for some methods will not be implemented. On the future it will be necessary to refactor the whole StatsCOllector architecture and extract inner classes.
   
   <!-- For new features, provide link to FS, dev ML discussion etc. -->
   <!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->
   
   <!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
   <!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
   <!-- Fixes: # -->
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   
   ## Screenshots (if appropriate):
   
   ## How Has This Been Tested?
   <!-- Please describe in detail how you tested your changes. -->
   <!-- Include details of your testing environment, and the tests you ran to -->
   <!-- see how your change affects other areas of the code, etc. -->
   Apart from the JUnit tests, the following tests have been manually done, asserting the expected behavior.
   Test 1: correct scheme and address, database "cloudstack" exists, default port 8086 is open and listening  (influxdb://172.16.0.10, influxdb://172.16.0.10/cloudstack, or influxdb://172.16.0.10:8086/cloudstack)
   - Result: CloudStack starts correctly, data are stored in the database cloudstack (the "cloudstack" is already configured on the influxdb before testing) and no error/warning logs.
   
   Test 2: `stats.output.uri` has wrong URI scheme (http://172.16.0.10 or influxd://172.16.0.10)
    - Result: Cannot store data on the InfluxDB host, logs the following Errors
   ```
   ERROR  [c.c.s.StatsCollector] (main:null) (logid:) influxd is not a valid protocol for external statistics. No statistics will be send.
   (...)
   ERROR [c.c.s.StatsCollector] (main:null) (logid:) Failed to parse external statistics URI: Cannot define a port for the Stats Collector service. The configured URI in stats.output.uri is not supported. Please configure as the following examples: graphite://graphite-hostaddress:port, or influxdb://influxdb-hostaddress:port. Note that the port is optional, if not added the default port for the respective collector (graphite or influxdb) will be used.: influxd://172.16.0.10
   ```
   
   Test 3: correct URI scheme but wrong address (influxdb://172.16.0.11)
   - Result: Cannot store data on the InfluxDB host, logs the following error
   ```
   ERROR [c.c.s.StatsCollector] (StatsCollector-6:ctx-bf6d551c) (logid:d5c92b19) Error trying to retrieve host stats
   org.influxdb.InfluxDBIOException: java.net.NoRouteToHostException: No route to host (Host unreachable)
   ```
   
   Test 4: correct URI scheme and address, database does not exist (influxdb://172.16.0.11/nonexistingdb)
   - Result: Cannot store data on the InfluxDB host, logs the following Errors
   ```
   ERROR [c.c.s.StatsCollector] (StatsCollector-1:ctx-5921793c) (logid:d4bab167) Error trying to retrieve host stats
   com.cloud.utils.exception.CloudRuntimeException: Database with name nonexistingdb does not exist in influxdb host 172.16.0.10:8086
   
   ```
   Test 5: correct URI scheme and address, database exists, customized port is not correctily configured (influxdb://172.16.0.10:8888/)
   - Result: Cannot store data on the InfluxDB host, logs the following Errors
   ```
   ERROR [c.c.s.StatsCollector] (StatsCollector-1:ctx-b2371dde) (logid:209a4b10) Error trying to retrieve host stats
   org.influxdb.InfluxDBIOException: java.net.ConnectException: Failed to connect to /172.16.0.10:8888
   
   ```
   Test 6: edit variable to null/empty
   - Result: it does not write in the external stats collector and does not log exceptions, as expected.
   
   <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/master/CONTRIBUTING.md) document -->
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services