Flipping data dirs for an (/multiple) SolrCore without affecting search / IndexReaders

[Kay Kay <ka...@gmail.com>]

We have an architecture where we want to flip the solr data.dir (massive 
dataset) while running and serving search requests with minimal downtime.

Some additional requirements.

* While ideally - we want the Solr Search clients to continue to serve 
from the indices as soon as possible -the overriding requirement is that 
the downtime for the Search Solr instances should be as less as 
possible.    So when a new set of (Lucene) indices come in - the 
algorithm we are experimenting with:

- create a new solrcore instance with the revised data directory.
- warm up the solrcore instance with some test queries.
- register the new solrcore instance with the same name as the old one, 
so that all new queries from the clients are to the new SolrCore instance.
- As part of register (String, SolrCore, boolean ) - the III parameter 
when set to false , closes the core connection.

   I am trying to understand more about the first and the fourth ( last) 
steps.

1) What is the fastest / best possible way to get step 1 done ,through a 
pluggable architecture.

Currently - I have written a request handler as follows, that takes care 
of creating the core. What is the best way to change dataDir (got as 
input from SolrQueryRequest) before creating SolrCore-s.

public class CustomRequestHandler extends RequestHandlerBase implements 
SolrCoreAware
{
  private CoreDescriptor coreDescriptor;

  private String         coreName;

  @Override
  public void handleRequestBody(SolrQueryRequest req,
                                SolrQueryResponse rsp) throws Exception
  {
    CoreContainer container = this.coreDescriptor.getCoreContainer();
    // TODO: Parse XML to extract data
    // container.reload(this.coreName);

    // or
    // 2.
    // TODO: Set the new configuration for the data directory / before 
creating the new core.
    SolrCore newCore = container.create(this.coreDescriptor);
    container.register(this.coreName, newCore, false);
  }


  @Override
  public void inform(SolrCore core)
  {
    coreDescriptor = core.getCoreDescriptor();
    coreName = core.getName();
  }
}


2) When a close() happens on an existing SolrCore - what happens when 
there is a long running IndexReader query on that SolrCore . Is that 
terminated abruptly / would the close wait until the IndexReaders 
completes the Query.



* The same process is repeated potentially for multiple SolrCores as 
well, with additional closeHooks that might do some heavy i/o tasks -  
talking over the network etc.
Right now - these long running processes are done in an independent 
thread so that they do not block SolrCore.close() with the currently 
nightly builds.

Re: Flipping data dirs for an (/multiple) SolrCore without affecting search / IndexReaders

[Noble Paul നോബിള്‍ नोब्ळ् <no...@gmail.com>]

Hi Kay,

do you wish to change data dir or just indexdir.
It is possible to switch indexDir w/ minimal overhead.

dataDir can contain a index.properties file which can contain a
property called 'index' it can point to your new index .

but you will have to find a way to populate this index

ReplicationHandler uses this feature.

This may not be a recommended behavior and i'm not sure how you can use this
look at SnapPuller#modifyIndexProps()

--Noble

On Fri, Jan 9, 2009 at 2:12 AM, Kay Kay <ka...@gmail.com> wrote:
> Chris Hostetter wrote:
>>
>> : We have an architecture where we want to flip the solr data.dir (massive
>> : dataset) while running and serving search requests with minimal
>> downtime.
>>        ...
>>
>> : 1) What is the fastest / best possible way to get step 1 done ,through a
>> : pluggable architecture.
>> : : Currently - I have written a request handler as follows, that takes
>> care of
>> : creating the core. What is the best way to change dataDir (got as input
>> from
>> : SolrQueryRequest) before creating SolrCore-s.
>>
>> you shouldn't need any custom plugin code to achieve this ... what you
>> describe wounds like exactly what the SWAP command on the CoreAdmin handler
>> was designed for -- CREATE your new core (using the new data dir) warm it
>> however you want (either via the solrconfig.xml or by explicitly hitting it
>> with queries) and once it's warmed up send the SWAP command to replace the
>> existing core with the name you want to use.
>>
>> : 2) When a close() happens on an existing SolrCore - what happens when
>> there is
>> : a long running IndexReader query on that SolrCore . Is that terminated
>> : abruptly / would the close wait until the IndexReaders completes the
>> Query.
>>
>> any existing uses of the Core will continue to finish (i'm not sure of hte
>> timeline of your question, ut i'm guessing this was before the recent jira
>> issue about the close() method and ref counts where this was better
>> explained, correct?)
>>
>>
>>
>> -Hoss
>>
>>
>>
>
> Thanks Hoss for the explanation regarding changing the data directory. Yes -
> this was before the jira issue discussions for the close() method.
>



-- 
--Noble Paul

Re: Flipping data dirs for an (/multiple) SolrCore without affecting search / IndexReaders

[Kay Kay <ka...@gmail.com>]

Chris Hostetter wrote:
> : We have an architecture where we want to flip the solr data.dir (massive
> : dataset) while running and serving search requests with minimal downtime.
> 	...
>
> : 1) What is the fastest / best possible way to get step 1 done ,through a
> : pluggable architecture.
> : 
> : Currently - I have written a request handler as follows, that takes care of
> : creating the core. What is the best way to change dataDir (got as input from
> : SolrQueryRequest) before creating SolrCore-s.
>
> you shouldn't need any custom plugin code to achieve this ... what you 
> describe wounds like exactly what the SWAP command on the CoreAdmin 
> handler was designed for -- CREATE your new core (using the new data dir) 
> warm it however you want (either via the solrconfig.xml or by explicitly 
> hitting it with queries) and once it's warmed up send the SWAP command to 
> replace the existing core with the name you want to use.
>
> : 2) When a close() happens on an existing SolrCore - what happens when there is
> : a long running IndexReader query on that SolrCore . Is that terminated
> : abruptly / would the close wait until the IndexReaders completes the Query.
>
> any existing uses of the Core will continue to finish (i'm not sure of hte 
> timeline of your question, ut i'm guessing this was before the recent jira 
> issue about the close() method and ref counts where this was better 
> explained, correct?)
>
>
>
> -Hoss
>
>
>   
Thanks Hoss for the explanation regarding changing the data directory. 
Yes - this was before the jira issue discussions for the close() method. 

Re: Flipping data dirs for an (/multiple) SolrCore without affecting search / IndexReaders

[Chris Hostetter <ho...@fucit.org>]

: We have an architecture where we want to flip the solr data.dir (massive
: dataset) while running and serving search requests with minimal downtime.
	...

: 1) What is the fastest / best possible way to get step 1 done ,through a
: pluggable architecture.
: 
: Currently - I have written a request handler as follows, that takes care of
: creating the core. What is the best way to change dataDir (got as input from
: SolrQueryRequest) before creating SolrCore-s.

you shouldn't need any custom plugin code to achieve this ... what you 
describe wounds like exactly what the SWAP command on the CoreAdmin 
handler was designed for -- CREATE your new core (using the new data dir) 
warm it however you want (either via the solrconfig.xml or by explicitly 
hitting it with queries) and once it's warmed up send the SWAP command to 
replace the existing core with the name you want to use.

: 2) When a close() happens on an existing SolrCore - what happens when there is
: a long running IndexReader query on that SolrCore . Is that terminated
: abruptly / would the close wait until the IndexReaders completes the Query.

any existing uses of the Core will continue to finish (i'm not sure of hte 
timeline of your question, ut i'm guessing this was before the recent jira 
issue about the close() method and ref counts where this was better 
explained, correct?)



-Hoss