[jira] [Commented] (SOLR-12357) TRA: Pre-emptively create next collection

["David Smiley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-12357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539223#comment-16539223 ] 

David Smiley commented on SOLR-12357:
-------------------------------------

I think we don't need to tie this issue to SOLR-12521.  Lets just proceed with improving TimeRoutedAliasUpdateProcessor to pre-emptively/asynchronously submit the command.  It's true that SOLR-12521 would be especially nice once this issue, SOLR-12357, is done but not strictly required/dependent.  Until we address SOLR-12521, we'll end up creating a gotcha here wherein the router.autoDeleteAge will be evaluated with a reference point that is further in the future than the current time.  

Here are some suggested docs on preemptive creation:
"preemptiveCreateWindowMs":  If a document arrives with a timestamp that is within this configured number of milliseconds before the end time of the most recent collection, then the next collection will be created asynchronously.  This is said to be preemptive collection creation as it is created ahead of time.  _Without_ this setting, collections are created synchronously and thus block the flow of documents until the collection is created.  Preemptive creation reduces these hiccups.  If set to enough time (perhaps an hour or more) then if there are problems creating a collection, this window of time might be enough to take corrective action.  However during this window of time, the collection is consuming resources without being used, and new documents will tend to be routed through it only to be routed elsewhere.  Also, note that {{router.autoDeleteAge}} is currently evaluated relative to the date of a newly created collection, and so you may want to increase the delete age by the preemptive window amount so that the oldest collection isn't deleted too soon.

> TRA: Pre-emptively create next collection 
> ------------------------------------------
>
>                 Key: SOLR-12357
>                 URL: https://issues.apache.org/jira/browse/SOLR-12357
>             Project: Solr
>          Issue Type: Sub-task
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SolrCloud
>            Reporter: David Smiley
>            Priority: Major
>
> When adding data to a Time Routed Alias (TRA), we sometimes need to create new collections.  Today we only do this synchronously – on-demand when a document is coming in.  But this can add delays as the documents inbound are held up for a collection to be created.  And, there may be a problem like a lack of resources (e.g. ample SolrCloud nodes with space) that the policy framework defines.  Such problems could be rectified sooner rather than later assume there is log alerting in place (definitely out of scope here).
> Pre-emptive TRA collection needs a time window configuration parameter, perhaps named something like "preemptiveCreateWindowMs".  If a document's timestamp is within this time window _from the end time of the head/lead collection_ then the collection can be created pre-eptively.  If no data is being sent to the TRA, no collections will be auto created, nor will it happen if older data is being added.  It may be convenient to effectively limit this time setting to the _smaller_ of this value and the TRA interval window, which I think is a fine limitation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org