You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nizan Grauer <ni...@yahoo-inc.com> on 2010/11/09 09:35:39 UTC
Dynamic creating of cores in solr
Hi,
I'm not sure this is the right mail to write to, hopefully you can help or direct me to the right person
I'm using solr - one master with 17 slaves in the server and using solrj as the java client
Currently there's only one core in all of them (master and slaves) - only the cpaCore.
I thought about using multi-cores solr, but I have some problems with that.
I don't know in advance which cores I'd need -
When my java program runs, I call for documents to be index to a certain url, which contains the core name, and I might create a url based on core that is not yet created. For example:
Calling to index - http://localhost:8080/cpaCore - existing core, everything as usual
Calling to index - http://localhost:8080/newCore - server realizes there's no core "newCore", creates it and indexes to it. After that - also creates the new core in the slaves
Calling to index - http://localhost:8080/newCore - existing core, everything as usual
What I'd like to have on the server side to do is realize by itself if the cores exists or not, and if not - create it
One other restriction - I can't change anything in the client side - calling to the server can only make the calls it's doing now - for index and search, and cannot make calls for cores creation via the CoreAdminHandler. All I can do is something in the server itself
What can I do to get it done? Write some RequestHandler? REquestProcessor? Any other option?
Thanks, nizan
Re: Dynamic creating of cores in solr
Posted by Ken Krugler <kk...@transpac.com>.
On Nov 10, 2010, at 12:30pm, Bob Sandiford wrote:
> Why not use replication? Call it inexperience...
>
> We're really early into working with and fully understanding Solr
> and the best way to approach various issues. I did mention that
> this was a prototype and non-production code, so I'm covered,
> though :)
>
> We'll take a look at the replication feature...
Replication doesn't replicate the top-level solr.xml file that defines
available cores, so if dynamic cores is a requirement then your custom
code isn't wasted :)
-- Ken
>> -----Original Message-----
>> From: Jonathan Rochkind [mailto:rochkind@jhu.edu]
>> Sent: Wednesday, November 10, 2010 3:26 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Dynamic creating of cores in solr
>>
>> You could use the actual built-in Solr replication feature to
>> accomplish
>> that same function -- complete re-index to a 'master', and then when
>> finished, trigger replication to the 'slave', with the 'slave' being
>> the
>> live index that actually serves your applications.
>>
>> I am curious if there was any reason you chose to roll your own
>> solution
>> using JSolr and dynamic creation of cores, instead of simply using
>> the
>> replication feature. Were there any downsides of using the
>> replication
>> feature for this purpose that you amerliorated through your solution?
>>
>> Jonathan
>>
>> Bob Sandiford wrote:
>>> We also use SolrJ, and have a dynamically created Core capability -
>> where we don't know in advance what the Cores will be that we
>> require.
>>>
>>> We almost always do a complete index build, and if there's a
>>> previous
>> instance of that index, it needs to be available during a complete
>> index build, so we have two cores per index, and switch them as
>> required at the end of an indexing run.
>>>
>>> Here's a summary of how we do it (we're in an early prototype /
>> implementation right now - this isn't production quality code - as
>> you
>> can tell from our voluminous javadocs on the methods...)
>>>
>>> 1) Identify if the core exists, and if not, create it:
>>>
>>> /**
>>> * This method instantiates two SolrServer objects, solr and
>> indexCore. It requires that
>>> * indexName be set before calling.
>>> */
>>> private void initSolrServer() throws IOException
>>> {
>>> String baseUrl = "http://localhost:8983/solr/";
>>> solr = new CommonsHttpSolrServer(baseUrl);
>>>
>>> String indexCoreName = indexName +
>> SolrConstants.SUFFIX_INDEX; // SUFIX_INDEX = "_INDEX"
>>> String indexCoreUrl = baseUrl + indexCoreName;
>>>
>>> // Here we create two cores for the indexName, if they don't
>> already exist - the live core used
>>> // for searching and a second core used for indexing. After
>> indexing, the two will be switched so the
>>> // just-indexed core will become the live core. The way that
>> core swapping works, the live core will always
>>> // be named [indexName] and the indexing core will always be
>> named [indexname]_INDEX, but the
>>> // dataDir of each core will alternate between [indexName]_1
>> and [indexName]_2.
>>> createCoreIfNeeded(indexName, indexName + "_1", solr);
>>> createCoreIfNeeded(indexCoreName, indexName + "_2", solr);
>>> indexCore = new CommonsHttpSolrServer(indexCoreUrl);
>>> }
>>>
>>>
>>> /**
>>> * Create a core if it does not already exists. Returns true if a
>> new core was created, false otherwise.
>>> */
>>> private boolean createCoreIfNeeded(String coreName, String
>> dataDir, SolrServer server) throws IOException
>>> {
>>> boolean coreExists = true;
>>> try
>>> {
>>> // SolrJ provides no direct method to check if a core
>> exists, but getStatus will
>>> // return an empty list for any core that doesn't.
>>> CoreAdminResponse statusResponse =
>> CoreAdminRequest.getStatus(coreName, server);
>>> coreExists =
>> statusResponse.getCoreStatus(coreName).size() > 0;
>>> if(!coreExists)
>>> {
>>> // Create the core
>>> LOG.info("Creating Solr core: " + coreName);
>>> CoreAdminRequest.Create create = new
>> CoreAdminRequest.Create();
>>> create.setCoreName(coreName);
>>> create.setInstanceDir(".");
>>> create.setDataDir(dataDir);
>>> create.process(server);
>>> }
>>> }
>>> catch (SolrServerException e)
>>> {
>>> e.printStackTrace();
>>> }
>>> return !coreExists;
>>> }
>>>
>>>
>>> 2) Do the index, clearing it first if it's a complete rebuild:
>>>
>>> [snip]
>>> if (fullIndex)
>>> {
>>> try
>>> {
>>> indexCore.deleteByQuery("*:*");
>>> }
>>> catch (SolrServerException e)
>>> {
>>> e.printStackTrace(); //To change body of catch
>> statement use File | Settings | File Templates.
>>> }
>>> }
>>> [snip]
>>>
>>> various logic, then (we submit batches of 100 :
>>>
>>> [snip]
>>> List<SolrInputDocument> docList =
>> b.getSolrInputDocumentList();
>>> UpdateResponse rsp;
>>> try
>>> {
>>> rsp = indexCore.add(docList);
>>> rsp = indexCore.commit();
>>> }
>>> catch (IOException e)
>>> {
>>> LOG.warn("Error commiting documents", e);
>>> }
>>> catch (SolrServerException e)
>>> {
>>> LOG.warn("Error commiting documents", e);
>>> }
>>> [snip]
>>>
>>> 3) optimize, then swap cores:
>>>
>>> private void optimizeCore()
>>> {
>>> try
>>> {
>>> indexCore.optimize();
>>> }
>>> catch(SolrServerException e)
>>> {
>>> LOG.warn("Error while optimizing core", e);
>>> }
>>> catch(IOException e)
>>> {
>>> LOG.warn("Error while optimizing core", e);
>>> }
>>> }
>>>
>>> private void swapCores()
>>> {
>>> String liveCore = indexName;
>>> String indexCore = indexName + SolrConstants.SUFFIX_INDEX; //
>> SUFFIX_INDEX = "_INDEX"
>>> LOG.info("Swapping Solr cores: " + indexCore + ", " +
>> liveCore);
>>> CoreAdminRequest request = new CoreAdminRequest();
>>> request.setAction(CoreAdminAction.SWAP);
>>> request.setCoreName(indexCore);
>>> request.setOtherCoreName(liveCore);
>>> try
>>> {
>>> request.process(solr);
>>> }
>>> catch (SolrServerException e)
>>> {
>>> e.printStackTrace();
>>> }
>>> catch (IOException e)
>>> {
>>> e.printStackTrace();
>>> }
>>> }
>>>
>>>
>>> And that's about it.
>>>
>>> You could adjust the above so there's only one core per index that
>> you want - if you don't do complete reindexes, and don't need the
>> index
>> to always be searchable.
>>>
>>> Hope that helps...
>>>
>>>
>>> Bob Sandiford | Lead Software Engineer | SirsiDynix
>>> P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
>>> www.sirsidynix.com
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Nizan Grauer [mailto:nizang@yahoo-inc.com]
>>>> Sent: Tuesday, November 09, 2010 3:36 AM
>>>> To: solr-user@lucene.apache.org
>>>> Subject: Dynamic creating of cores in solr
>>>>
>>>> Hi,
>>>>
>>>> I'm not sure this is the right mail to write to, hopefully you can
>> help
>>>> or direct me to the right person
>>>>
>>>> I'm using solr - one master with 17 slaves in the server and using
>>>> solrj as the java client
>>>>
>>>> Currently there's only one core in all of them (master and
>>>> slaves) -
>>>> only the cpaCore.
>>>>
>>>> I thought about using multi-cores solr, but I have some problems
>> with
>>>> that.
>>>>
>>>> I don't know in advance which cores I'd need -
>>>>
>>>> When my java program runs, I call for documents to be index to a
>>>> certain url, which contains the core name, and I might create a url
>>>> based on core that is not yet created. For example:
>>>>
>>>> Calling to index - http://localhost:8080/cpaCore - existing core,
>>>> everything as usual
>>>> Calling to index - http://localhost:8080/newCore - server realizes
>>>> there's no core "newCore", creates it and indexes to it. After that
>> -
>>>> also creates the new core in the slaves
>>>> Calling to index - http://localhost:8080/newCore - existing core,
>>>> everything as usual
>>>>
>>>> What I'd like to have on the server side to do is realize by itself
>> if
>>>> the cores exists or not, and if not - create it
>>>>
>>>> One other restriction - I can't change anything in the client
>>>> side -
>>>> calling to the server can only make the calls it's doing now - for
>>>> index and search, and cannot make calls for cores creation via the
>>>> CoreAdminHandler. All I can do is something in the server itself
>>>>
>>>> What can I do to get it done? Write some RequestHandler?
>>>> REquestProcessor? Any other option?
>>>>
>>>> Thanks, nizan
>>>>
>>>
>>>
>>>
>
--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
RE: Dynamic creating of cores in solr
Posted by Bob Sandiford <bo...@sirsidynix.com>.
Why not use replication? Call it inexperience...
We're really early into working with and fully understanding Solr and the best way to approach various issues. I did mention that this was a prototype and non-production code, so I'm covered, though :)
We'll take a look at the replication feature...
Thanks!
Bob Sandiford | Lead Software Engineer | SirsiDynix
P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
www.sirsidynix.com
> -----Original Message-----
> From: Jonathan Rochkind [mailto:rochkind@jhu.edu]
> Sent: Wednesday, November 10, 2010 3:26 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Dynamic creating of cores in solr
>
> You could use the actual built-in Solr replication feature to
> accomplish
> that same function -- complete re-index to a 'master', and then when
> finished, trigger replication to the 'slave', with the 'slave' being
> the
> live index that actually serves your applications.
>
> I am curious if there was any reason you chose to roll your own
> solution
> using JSolr and dynamic creation of cores, instead of simply using the
> replication feature. Were there any downsides of using the replication
> feature for this purpose that you amerliorated through your solution?
>
> Jonathan
>
> Bob Sandiford wrote:
> > We also use SolrJ, and have a dynamically created Core capability -
> where we don't know in advance what the Cores will be that we require.
> >
> > We almost always do a complete index build, and if there's a previous
> instance of that index, it needs to be available during a complete
> index build, so we have two cores per index, and switch them as
> required at the end of an indexing run.
> >
> > Here's a summary of how we do it (we're in an early prototype /
> implementation right now - this isn't production quality code - as you
> can tell from our voluminous javadocs on the methods...)
> >
> > 1) Identify if the core exists, and if not, create it:
> >
> > /**
> > * This method instantiates two SolrServer objects, solr and
> indexCore. It requires that
> > * indexName be set before calling.
> > */
> > private void initSolrServer() throws IOException
> > {
> > String baseUrl = "http://localhost:8983/solr/";
> > solr = new CommonsHttpSolrServer(baseUrl);
> >
> > String indexCoreName = indexName +
> SolrConstants.SUFFIX_INDEX; // SUFIX_INDEX = "_INDEX"
> > String indexCoreUrl = baseUrl + indexCoreName;
> >
> > // Here we create two cores for the indexName, if they don't
> already exist - the live core used
> > // for searching and a second core used for indexing. After
> indexing, the two will be switched so the
> > // just-indexed core will become the live core. The way that
> core swapping works, the live core will always
> > // be named [indexName] and the indexing core will always be
> named [indexname]_INDEX, but the
> > // dataDir of each core will alternate between [indexName]_1
> and [indexName]_2.
> > createCoreIfNeeded(indexName, indexName + "_1", solr);
> > createCoreIfNeeded(indexCoreName, indexName + "_2", solr);
> > indexCore = new CommonsHttpSolrServer(indexCoreUrl);
> > }
> >
> >
> > /**
> > * Create a core if it does not already exists. Returns true if a
> new core was created, false otherwise.
> > */
> > private boolean createCoreIfNeeded(String coreName, String
> dataDir, SolrServer server) throws IOException
> > {
> > boolean coreExists = true;
> > try
> > {
> > // SolrJ provides no direct method to check if a core
> exists, but getStatus will
> > // return an empty list for any core that doesn't.
> > CoreAdminResponse statusResponse =
> CoreAdminRequest.getStatus(coreName, server);
> > coreExists =
> statusResponse.getCoreStatus(coreName).size() > 0;
> > if(!coreExists)
> > {
> > // Create the core
> > LOG.info("Creating Solr core: " + coreName);
> > CoreAdminRequest.Create create = new
> CoreAdminRequest.Create();
> > create.setCoreName(coreName);
> > create.setInstanceDir(".");
> > create.setDataDir(dataDir);
> > create.process(server);
> > }
> > }
> > catch (SolrServerException e)
> > {
> > e.printStackTrace();
> > }
> > return !coreExists;
> > }
> >
> >
> > 2) Do the index, clearing it first if it's a complete rebuild:
> >
> > [snip]
> > if (fullIndex)
> > {
> > try
> > {
> > indexCore.deleteByQuery("*:*");
> > }
> > catch (SolrServerException e)
> > {
> > e.printStackTrace(); //To change body of catch
> statement use File | Settings | File Templates.
> > }
> > }
> > [snip]
> >
> > various logic, then (we submit batches of 100 :
> >
> > [snip]
> > List<SolrInputDocument> docList =
> b.getSolrInputDocumentList();
> > UpdateResponse rsp;
> > try
> > {
> > rsp = indexCore.add(docList);
> > rsp = indexCore.commit();
> > }
> > catch (IOException e)
> > {
> > LOG.warn("Error commiting documents", e);
> > }
> > catch (SolrServerException e)
> > {
> > LOG.warn("Error commiting documents", e);
> > }
> > [snip]
> >
> > 3) optimize, then swap cores:
> >
> > private void optimizeCore()
> > {
> > try
> > {
> > indexCore.optimize();
> > }
> > catch(SolrServerException e)
> > {
> > LOG.warn("Error while optimizing core", e);
> > }
> > catch(IOException e)
> > {
> > LOG.warn("Error while optimizing core", e);
> > }
> > }
> >
> > private void swapCores()
> > {
> > String liveCore = indexName;
> > String indexCore = indexName + SolrConstants.SUFFIX_INDEX; //
> SUFFIX_INDEX = "_INDEX"
> > LOG.info("Swapping Solr cores: " + indexCore + ", " +
> liveCore);
> > CoreAdminRequest request = new CoreAdminRequest();
> > request.setAction(CoreAdminAction.SWAP);
> > request.setCoreName(indexCore);
> > request.setOtherCoreName(liveCore);
> > try
> > {
> > request.process(solr);
> > }
> > catch (SolrServerException e)
> > {
> > e.printStackTrace();
> > }
> > catch (IOException e)
> > {
> > e.printStackTrace();
> > }
> > }
> >
> >
> > And that's about it.
> >
> > You could adjust the above so there's only one core per index that
> you want - if you don't do complete reindexes, and don't need the index
> to always be searchable.
> >
> > Hope that helps...
> >
> >
> > Bob Sandiford | Lead Software Engineer | SirsiDynix
> > P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
> > www.sirsidynix.com
> >
> >
> >
> >> -----Original Message-----
> >> From: Nizan Grauer [mailto:nizang@yahoo-inc.com]
> >> Sent: Tuesday, November 09, 2010 3:36 AM
> >> To: solr-user@lucene.apache.org
> >> Subject: Dynamic creating of cores in solr
> >>
> >> Hi,
> >>
> >> I'm not sure this is the right mail to write to, hopefully you can
> help
> >> or direct me to the right person
> >>
> >> I'm using solr - one master with 17 slaves in the server and using
> >> solrj as the java client
> >>
> >> Currently there's only one core in all of them (master and slaves) -
> >> only the cpaCore.
> >>
> >> I thought about using multi-cores solr, but I have some problems
> with
> >> that.
> >>
> >> I don't know in advance which cores I'd need -
> >>
> >> When my java program runs, I call for documents to be index to a
> >> certain url, which contains the core name, and I might create a url
> >> based on core that is not yet created. For example:
> >>
> >> Calling to index - http://localhost:8080/cpaCore - existing core,
> >> everything as usual
> >> Calling to index - http://localhost:8080/newCore - server realizes
> >> there's no core "newCore", creates it and indexes to it. After that
> -
> >> also creates the new core in the slaves
> >> Calling to index - http://localhost:8080/newCore - existing core,
> >> everything as usual
> >>
> >> What I'd like to have on the server side to do is realize by itself
> if
> >> the cores exists or not, and if not - create it
> >>
> >> One other restriction - I can't change anything in the client side -
> >> calling to the server can only make the calls it's doing now - for
> >> index and search, and cannot make calls for cores creation via the
> >> CoreAdminHandler. All I can do is something in the server itself
> >>
> >> What can I do to get it done? Write some RequestHandler?
> >> REquestProcessor? Any other option?
> >>
> >> Thanks, nizan
> >>
> >
> >
> >
Re: Dynamic creating of cores in solr
Posted by Jonathan Rochkind <ro...@jhu.edu>.
You could use the actual built-in Solr replication feature to accomplish
that same function -- complete re-index to a 'master', and then when
finished, trigger replication to the 'slave', with the 'slave' being the
live index that actually serves your applications.
I am curious if there was any reason you chose to roll your own solution
using JSolr and dynamic creation of cores, instead of simply using the
replication feature. Were there any downsides of using the replication
feature for this purpose that you amerliorated through your solution?
Jonathan
Bob Sandiford wrote:
> We also use SolrJ, and have a dynamically created Core capability - where we don't know in advance what the Cores will be that we require.
>
> We almost always do a complete index build, and if there's a previous instance of that index, it needs to be available during a complete index build, so we have two cores per index, and switch them as required at the end of an indexing run.
>
> Here's a summary of how we do it (we're in an early prototype / implementation right now - this isn't production quality code - as you can tell from our voluminous javadocs on the methods...)
>
> 1) Identify if the core exists, and if not, create it:
>
> /**
> * This method instantiates two SolrServer objects, solr and indexCore. It requires that
> * indexName be set before calling.
> */
> private void initSolrServer() throws IOException
> {
> String baseUrl = "http://localhost:8983/solr/";
> solr = new CommonsHttpSolrServer(baseUrl);
>
> String indexCoreName = indexName + SolrConstants.SUFFIX_INDEX; // SUFIX_INDEX = "_INDEX"
> String indexCoreUrl = baseUrl + indexCoreName;
>
> // Here we create two cores for the indexName, if they don't already exist - the live core used
> // for searching and a second core used for indexing. After indexing, the two will be switched so the
> // just-indexed core will become the live core. The way that core swapping works, the live core will always
> // be named [indexName] and the indexing core will always be named [indexname]_INDEX, but the
> // dataDir of each core will alternate between [indexName]_1 and [indexName]_2.
> createCoreIfNeeded(indexName, indexName + "_1", solr);
> createCoreIfNeeded(indexCoreName, indexName + "_2", solr);
> indexCore = new CommonsHttpSolrServer(indexCoreUrl);
> }
>
>
> /**
> * Create a core if it does not already exists. Returns true if a new core was created, false otherwise.
> */
> private boolean createCoreIfNeeded(String coreName, String dataDir, SolrServer server) throws IOException
> {
> boolean coreExists = true;
> try
> {
> // SolrJ provides no direct method to check if a core exists, but getStatus will
> // return an empty list for any core that doesn't.
> CoreAdminResponse statusResponse = CoreAdminRequest.getStatus(coreName, server);
> coreExists = statusResponse.getCoreStatus(coreName).size() > 0;
> if(!coreExists)
> {
> // Create the core
> LOG.info("Creating Solr core: " + coreName);
> CoreAdminRequest.Create create = new CoreAdminRequest.Create();
> create.setCoreName(coreName);
> create.setInstanceDir(".");
> create.setDataDir(dataDir);
> create.process(server);
> }
> }
> catch (SolrServerException e)
> {
> e.printStackTrace();
> }
> return !coreExists;
> }
>
>
> 2) Do the index, clearing it first if it's a complete rebuild:
>
> [snip]
> if (fullIndex)
> {
> try
> {
> indexCore.deleteByQuery("*:*");
> }
> catch (SolrServerException e)
> {
> e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
> }
> }
> [snip]
>
> various logic, then (we submit batches of 100 :
>
> [snip]
> List<SolrInputDocument> docList = b.getSolrInputDocumentList();
> UpdateResponse rsp;
> try
> {
> rsp = indexCore.add(docList);
> rsp = indexCore.commit();
> }
> catch (IOException e)
> {
> LOG.warn("Error commiting documents", e);
> }
> catch (SolrServerException e)
> {
> LOG.warn("Error commiting documents", e);
> }
> [snip]
>
> 3) optimize, then swap cores:
>
> private void optimizeCore()
> {
> try
> {
> indexCore.optimize();
> }
> catch(SolrServerException e)
> {
> LOG.warn("Error while optimizing core", e);
> }
> catch(IOException e)
> {
> LOG.warn("Error while optimizing core", e);
> }
> }
>
> private void swapCores()
> {
> String liveCore = indexName;
> String indexCore = indexName + SolrConstants.SUFFIX_INDEX; // SUFFIX_INDEX = "_INDEX"
> LOG.info("Swapping Solr cores: " + indexCore + ", " + liveCore);
> CoreAdminRequest request = new CoreAdminRequest();
> request.setAction(CoreAdminAction.SWAP);
> request.setCoreName(indexCore);
> request.setOtherCoreName(liveCore);
> try
> {
> request.process(solr);
> }
> catch (SolrServerException e)
> {
> e.printStackTrace();
> }
> catch (IOException e)
> {
> e.printStackTrace();
> }
> }
>
>
> And that's about it.
>
> You could adjust the above so there's only one core per index that you want - if you don't do complete reindexes, and don't need the index to always be searchable.
>
> Hope that helps...
>
>
> Bob Sandiford | Lead Software Engineer | SirsiDynix
> P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
> www.sirsidynix.com
>
>
>
>> -----Original Message-----
>> From: Nizan Grauer [mailto:nizang@yahoo-inc.com]
>> Sent: Tuesday, November 09, 2010 3:36 AM
>> To: solr-user@lucene.apache.org
>> Subject: Dynamic creating of cores in solr
>>
>> Hi,
>>
>> I'm not sure this is the right mail to write to, hopefully you can help
>> or direct me to the right person
>>
>> I'm using solr - one master with 17 slaves in the server and using
>> solrj as the java client
>>
>> Currently there's only one core in all of them (master and slaves) -
>> only the cpaCore.
>>
>> I thought about using multi-cores solr, but I have some problems with
>> that.
>>
>> I don't know in advance which cores I'd need -
>>
>> When my java program runs, I call for documents to be index to a
>> certain url, which contains the core name, and I might create a url
>> based on core that is not yet created. For example:
>>
>> Calling to index - http://localhost:8080/cpaCore - existing core,
>> everything as usual
>> Calling to index - http://localhost:8080/newCore - server realizes
>> there's no core "newCore", creates it and indexes to it. After that -
>> also creates the new core in the slaves
>> Calling to index - http://localhost:8080/newCore - existing core,
>> everything as usual
>>
>> What I'd like to have on the server side to do is realize by itself if
>> the cores exists or not, and if not - create it
>>
>> One other restriction - I can't change anything in the client side -
>> calling to the server can only make the calls it's doing now - for
>> index and search, and cannot make calls for cores creation via the
>> CoreAdminHandler. All I can do is something in the server itself
>>
>> What can I do to get it done? Write some RequestHandler?
>> REquestProcessor? Any other option?
>>
>> Thanks, nizan
>>
>
>
>
RE: Dynamic creating of cores in solr
Posted by Bob Sandiford <bo...@sirsidynix.com>.
We also use SolrJ, and have a dynamically created Core capability - where we don't know in advance what the Cores will be that we require.
We almost always do a complete index build, and if there's a previous instance of that index, it needs to be available during a complete index build, so we have two cores per index, and switch them as required at the end of an indexing run.
Here's a summary of how we do it (we're in an early prototype / implementation right now - this isn't production quality code - as you can tell from our voluminous javadocs on the methods...)
1) Identify if the core exists, and if not, create it:
/**
* This method instantiates two SolrServer objects, solr and indexCore. It requires that
* indexName be set before calling.
*/
private void initSolrServer() throws IOException
{
String baseUrl = "http://localhost:8983/solr/";
solr = new CommonsHttpSolrServer(baseUrl);
String indexCoreName = indexName + SolrConstants.SUFFIX_INDEX; // SUFIX_INDEX = "_INDEX"
String indexCoreUrl = baseUrl + indexCoreName;
// Here we create two cores for the indexName, if they don't already exist - the live core used
// for searching and a second core used for indexing. After indexing, the two will be switched so the
// just-indexed core will become the live core. The way that core swapping works, the live core will always
// be named [indexName] and the indexing core will always be named [indexname]_INDEX, but the
// dataDir of each core will alternate between [indexName]_1 and [indexName]_2.
createCoreIfNeeded(indexName, indexName + "_1", solr);
createCoreIfNeeded(indexCoreName, indexName + "_2", solr);
indexCore = new CommonsHttpSolrServer(indexCoreUrl);
}
/**
* Create a core if it does not already exists. Returns true if a new core was created, false otherwise.
*/
private boolean createCoreIfNeeded(String coreName, String dataDir, SolrServer server) throws IOException
{
boolean coreExists = true;
try
{
// SolrJ provides no direct method to check if a core exists, but getStatus will
// return an empty list for any core that doesn't.
CoreAdminResponse statusResponse = CoreAdminRequest.getStatus(coreName, server);
coreExists = statusResponse.getCoreStatus(coreName).size() > 0;
if(!coreExists)
{
// Create the core
LOG.info("Creating Solr core: " + coreName);
CoreAdminRequest.Create create = new CoreAdminRequest.Create();
create.setCoreName(coreName);
create.setInstanceDir(".");
create.setDataDir(dataDir);
create.process(server);
}
}
catch (SolrServerException e)
{
e.printStackTrace();
}
return !coreExists;
}
2) Do the index, clearing it first if it's a complete rebuild:
[snip]
if (fullIndex)
{
try
{
indexCore.deleteByQuery("*:*");
}
catch (SolrServerException e)
{
e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
}
}
[snip]
various logic, then (we submit batches of 100 :
[snip]
List<SolrInputDocument> docList = b.getSolrInputDocumentList();
UpdateResponse rsp;
try
{
rsp = indexCore.add(docList);
rsp = indexCore.commit();
}
catch (IOException e)
{
LOG.warn("Error commiting documents", e);
}
catch (SolrServerException e)
{
LOG.warn("Error commiting documents", e);
}
[snip]
3) optimize, then swap cores:
private void optimizeCore()
{
try
{
indexCore.optimize();
}
catch(SolrServerException e)
{
LOG.warn("Error while optimizing core", e);
}
catch(IOException e)
{
LOG.warn("Error while optimizing core", e);
}
}
private void swapCores()
{
String liveCore = indexName;
String indexCore = indexName + SolrConstants.SUFFIX_INDEX; // SUFFIX_INDEX = "_INDEX"
LOG.info("Swapping Solr cores: " + indexCore + ", " + liveCore);
CoreAdminRequest request = new CoreAdminRequest();
request.setAction(CoreAdminAction.SWAP);
request.setCoreName(indexCore);
request.setOtherCoreName(liveCore);
try
{
request.process(solr);
}
catch (SolrServerException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
}
And that's about it.
You could adjust the above so there's only one core per index that you want - if you don't do complete reindexes, and don't need the index to always be searchable.
Hope that helps...
Bob Sandiford | Lead Software Engineer | SirsiDynix
P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
www.sirsidynix.com
> -----Original Message-----
> From: Nizan Grauer [mailto:nizang@yahoo-inc.com]
> Sent: Tuesday, November 09, 2010 3:36 AM
> To: solr-user@lucene.apache.org
> Subject: Dynamic creating of cores in solr
>
> Hi,
>
> I'm not sure this is the right mail to write to, hopefully you can help
> or direct me to the right person
>
> I'm using solr - one master with 17 slaves in the server and using
> solrj as the java client
>
> Currently there's only one core in all of them (master and slaves) -
> only the cpaCore.
>
> I thought about using multi-cores solr, but I have some problems with
> that.
>
> I don't know in advance which cores I'd need -
>
> When my java program runs, I call for documents to be index to a
> certain url, which contains the core name, and I might create a url
> based on core that is not yet created. For example:
>
> Calling to index - http://localhost:8080/cpaCore - existing core,
> everything as usual
> Calling to index - http://localhost:8080/newCore - server realizes
> there's no core "newCore", creates it and indexes to it. After that -
> also creates the new core in the slaves
> Calling to index - http://localhost:8080/newCore - existing core,
> everything as usual
>
> What I'd like to have on the server side to do is realize by itself if
> the cores exists or not, and if not - create it
>
> One other restriction - I can't change anything in the client side -
> calling to the server can only make the calls it's doing now - for
> index and search, and cannot make calls for cores creation via the
> CoreAdminHandler. All I can do is something in the server itself
>
> What can I do to get it done? Write some RequestHandler?
> REquestProcessor? Any other option?
>
> Thanks, nizan