[jira] [Commented] (SOLR-7789) Introduce a ConfigSet management API

["Gregory Chanan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704171#comment-14704171 ] 

Gregory Chanan commented on SOLR-7789:
--------------------------------------

Here is a patch that implements a ConfigSets API for SolrCloud with two entry points:

- CREATE: Create a ConfigSet, based on an existing ConfigSet
Parameters:
||Key||Type||Required||Default||Description||
|name|String|Yes| |ConfigSet to be created|
|baseConfigSet|String|Yes| |ConfigSet to copy as a base|
|configSetProp._name=value_|String|No| |ConfigSet property from base to override|

- DELETE: Delete a ConfigSet
Parameters:
||Key||Type||Required||Default||Description||
|name|String|Yes| |ConfigSet to be deleted|

The implementation is based on the Collections API and in fact uses the same OverseerProcessor at this time.  The handling code between the ConfigSets API and the CollectionsAPI is completely separate so it should be easy to move to its own OverseerProcessor in the future.  Using one OverseerProcessor seemed reasonable given the above operations should be fast.

Some notes:
1) DELETE does not check that a ConfigSet is not referenced.  This is equivalent to the existing method of deleting a ConfigSet via the zkcli.  The exception to this rule is with task exclusivity: a ConfigSet deletion won't go through if there is an on going operation referencing that ConfigSet (e.g. as a Base ConfigSet in a create).  There is a bit of complexity in implementing that efficiently (simulating a read/write lock for exclusivity so we don't block on multiple ConfigSets being reated with the same Base ConfigSet).

2) The name for the ConfigSet properties file is always assumed to be "configsetprops.json".  This is necessary to handle deletions correctly; if we allowed the user to specify in the API call what the file was called they could point at another or non-existing file to be able to avoid the immutable check.  This is a little strange because the CoreDescriptor lets you override the name of this file, but I guess this is equivalent to solrconfig.xml and schema.xml being special in ZK even though they can also be overriden in the CoreDescriptor.

3) The handling of the Overseer.getConfigSetQueue isn't as nice as I would like.  Because we use the same OverseerProcessor, we need some way of figuring out which MessageHandler to dispatch to; we do this by prefixing the action in ZK with "configsets:".  Both the ConfigSetsHandler and the OverseerConfigSetsMessageHandler need to be aware of this.  Ideally, the queue returned from Overseer.getConfigSetQueue would just handle this when an item was inserted into the queue, so only the Overseer has to worry about how the messages are actually handled.  This was difficult to do given the DistributedQueue API is just byte arrays.  I think the correct thing to do here is change at least the offer API to be JSON based -- all the non-test calls already immediately convert JSON to bytes anyway -- but this seemed like a larger change for this patch.

Some testing notes:
T1) I wanted to be able to test the case where the CREATE failed half way through, so we end up with partial state in ZK that needed to be cleaned up.  I found the easiest way to do this was to override the ZKDatabase that the ZKTestServer uses.  So, you can now get/set the ZKDatabase in the ZKTestServer and pass your own ZKTestServer to the MiniSolrCloudCluster.

T2) There's also an "exclusivity" test to ensure that actions on the same ConfigSet don't trample each other.  This is done by concurrently doing DELETEs/CREATEs and ensuring the exceptions indicate that the operations ran sequentially.

> Introduce a ConfigSet management API
> ------------------------------------
>
>                 Key: SOLR-7789
>                 URL: https://issues.apache.org/jira/browse/SOLR-7789
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Gregory Chanan
>            Assignee: Gregory Chanan
>         Attachments: SOLR-7789.patch
>
>
> SOLR-5955 describes a feature to automatically create a ConfigSet, based on another one, from a collection API call (i.e. one step collection creation).  Discussion there yielded SOLR-7742, Immutable ConfigSet support.  To close the loop, we need support for a ConfigSet management API.
> The simplest ConfigSet API could have one operation:
> create a new config set, based on an existing one, possible modifying the ConfigSet properties.  Note you need to be able to modify the ConfigSet properties at creation time because otherwise Immutable could not be changed.
> Another logical operation to support is ConfigSet deletion; that may be more complicated to implement than creation because you need to handle the case where a collection is already using the configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org