[jira] [Commented] (SOLR-14749) Provide a clean API for cluster-level event processing

["Andrzej Bialecki (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192921#comment-17192921 ] 

Andrzej Bialecki commented on SOLR-14749:
-----------------------------------------

[~ichattopadhyaya] as I explained in the comments that follow yours in the PR, we actually need such coordination in Solr core. The concept of cluster singleton instances is important, there are quite a few components currently that work like this, and they all need this functionality. If Solr core doesn't provide this then each plugin author will have to come up with their own hacky and messy and buggy scheme.

If the package / plugin infra becomes more robust to support transitive dependencies then we can extract the actual implementation of ClusterSingleton management into its own plugin - but still, the interface needs to be in the core, with a default implementation.

Again, this is not just for autoscaling - autoscaling will be just one user of the ClusterSingleton concept.

I removed the Scheduler part from PR-1758. Do you still strongly object to the current state of the PR, Ishan? If so, could you please outline technical reasons for your objections?

> Provide a clean API for cluster-level event processing
> ------------------------------------------------------
>
>                 Key: SOLR-14749
>                 URL: https://issues.apache.org/jira/browse/SOLR-14749
>             Project: Solr
>          Issue Type: Improvement
>          Components: AutoScaling
>            Reporter: Andrzej Bialecki
>            Assignee: Andrzej Bialecki
>            Priority: Major
>              Labels: clean-api
>          Time Spent: 9h
>  Remaining Estimate: 0h
>
> This is a companion issue to SOLR-14613 and it aims at providing a clean, strongly typed API for the functionality formerly known as "triggers" - that is, a component for generating cluster-level events corresponding to changes in the cluster state, and a pluggable API for processing these events.
> The 8x triggers have been removed so this functionality is currently missing in 9.0. However, this functionality is crucial for implementing the automatic collection repair and re-balancing as the cluster state changes (nodes going down / up, becoming overloaded / unused / decommissioned, etc).
> For this reason we need this API and a default implementation of triggers that at least can perform automatic collection repair (maintaining the desired replication factor in presence of live node changes).
> As before, the actual changes to the collections will be executed using existing CollectionAdmin API, which in turn may use the placement plugins from SOLR-14613.
> h3. Division of responsibility
>  * built-in Solr components (non-pluggable):
>  ** cluster state monitoring and event generation,
>  ** simple scheduler to periodically generate scheduled events
>  * plugins:
>  ** automatic collection repair on {{nodeLost}} events (provided by default)
>  ** re-balancing of replicas (periodic or on {{nodeAdded}} events)
>  ** reporting (eg. requesting additional node provisioning)
>  ** scheduled maintenance (eg. removing inactive shards after split)
> h3. Other considerations
> These plugins (unlike the placement plugins) need to execute on one designated node in the cluster. Currently the easiest way to implement this is to run them on the Overseer leader node.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org