[jira] [Commented] (SOLR-13579) Create resource management API

["Andrzej Bialecki (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-13579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878603#comment-16878603 ] 

Andrzej Bialecki  commented on SOLR-13579:
------------------------------------------

This patch contains a draft of the API, with some details of the implementation fleshed out, up for discussion - there are no tests and no integration with any existing component yet. :)

A high-level design overview:
 * A {{ResourceManager}} manages multiple named pools of resources (flat hierarchy for now). A default instance of {{ResourceManager}} would be created at a {{CoreContainer}} level so that it can manage global limits for a Solr node.
 * Each pool knows how to perform a single specific type of management. This handling is actually performed by a {{ResourceManagerPlugin}}, which knows what monitored values to retrieve from resources, and knows how to adjust the controlled parameters of managed resources.
 * There can be multiple pools of the same type (under different names) - they will likely differ in their parameter. Eg. document cache size may be checked every 1 min and have one limit, but the query / filter cache size may use different parameters, even though the set of monitored parameters and controlled parameters are the same (hence the same type).
 * {{ResourceManager}} is responsible for periodically executing the {{ResourceManagerPlugin}} of each of the pools, so that it can verify and adjust the resources it manages in the pool.
 * Each pool has its own parameters - currently the only global parameter is scheduleDelaySeconds, which determines how often the pool will run the management plugin to verify and adjust the resource usage.
 * Each pool also has plugin-specific parameters, most notably the limits - eg. max total cache size, which the CacheManagerPlugin knows how to use in order to adjust cache sizes.
 * Each managed resource can be managed by multiple types of plugins and it may appear in multiple pools (of different types). This reflects the fact that a single component may have multiple aspects of resource management - eg. cache mgmt, cpu, threads, etc.

The patch also contains an example implementation of a management plugin - {{CacheManagerPlugin}}. This plugin uses the API to enforce global limits on the cache size. It knows how to retrieve and calculate the current resource usage, as reported by the monitored values, and then it adjusts the controlled limits of each resource to bring the usage back to the total values that fit within the limits defined by the pool. In this case the pool can define global limits on the cache {{size}} and {{maxRamMB}} (and these are also the parameters to control for each cache), and the plugin uses {{size}} and {{ramBytesUsed}} for monitoring the actual resource consumption.

Obviously {{SolrCache}} doesn't implement this API yet, but it's relatively easy to add.

I'd appreciate review, comments and suggestions.

> Create resource management API
> ------------------------------
>
>                 Key: SOLR-13579
>                 URL: https://issues.apache.org/jira/browse/SOLR-13579
>             Project: Solr
>          Issue Type: Sub-task
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Andrzej Bialecki 
>            Assignee: Andrzej Bialecki 
>            Priority: Major
>         Attachments: SOLR-13579.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org