You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Thijs <vo...@gmail.com> on 2009/12/08 10:13:10 UTC
How to setup dynamic multicore replication
Hi
I need some help setting up dynamic multicore replication.
We are changing our setup from a replicated single core index with
multiple document types, as described on the wiki[1], to a dynamic
multicore setup. We need this so that we can display facets with a zero
count that are unique to the document 'type'.
So when indexing new documents we want to create new cores on the fly
using the CoreAdminHandler through SolrJ.
What I can't figure out is how I setup solr.xml and solrconfig.xml so
that a core automatically is also replicated from the master to it's
slaves once it's created.
I have a solr.xml that starts like this:
<?xml version='1.0' encoding='UTF-8'?>
<solr persistent="true">
<cores adminPath="/admin/cores">
</cores>
</solr>
and the replication part of solrconfig.xml
master:
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="master">
<str name="replicateAfter">startup</str>
<str name="replicateAfter">optimize</str>
<str name="confFiles">schema.xml</str>
</lst>
</requestHandler>
slave:
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="slave">
<str name="masterUrl">http://localhost:8081/solr/replication</str>
<str name="pollInterval">00:00:20</str>
</lst>
</requestHandler>
I think I should change the masterUrl in the slave configuration to
something like:
<str
name="masterUrl">http://localhost:8081/solr/${solr.core.name}/replication</str>
So that the replication automatically finds the correct core replication
handler.
But how do I tell the slaves a new core is created, and that is should
start replicating those to?
Thanks in advance.
Thijs
[1]
http://wiki.apache.org/solr/MultipleIndexes#Flattening_Data_Into_a_Single_Index
Re: How to setup dynamic multicore replication
Posted by Thijs <vo...@gmail.com>.
If I for example do:
/select?q=type:book&facet=true&facet.mincount=0&facet.field=title
the titles that are returned for the facet query also contains titles
that are of type dvd. While I only want the unique titles for type book.
On 8-12-2009 12:09, Shalin Shekhar Mangar wrote:
> On Tue, Dec 8, 2009 at 2:43 PM, Thijs<vo...@gmail.com> wrote:
>
>> Hi
>>
>> I need some help setting up dynamic multicore replication.
>>
>> We are changing our setup from a replicated single core index with multiple
>> document types, as described on the wiki[1], to a dynamic multicore setup.
>> We need this so that we can display facets with a zero count that are unique
>> to the document 'type'.
>>
>
> If you go by that wiki link, then there is no need to have multiple cores.
> It basically says that, in some cases, it is possible to flatten multiple
> indexes into one index. Am I missing something?
>
Re: How to setup dynamic multicore replication
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, Dec 8, 2009 at 2:43 PM, Thijs <vo...@gmail.com> wrote:
> Hi
>
> I need some help setting up dynamic multicore replication.
>
> We are changing our setup from a replicated single core index with multiple
> document types, as described on the wiki[1], to a dynamic multicore setup.
> We need this so that we can display facets with a zero count that are unique
> to the document 'type'.
>
If you go by that wiki link, then there is no need to have multiple cores.
It basically says that, in some cases, it is possible to flatten multiple
indexes into one index. Am I missing something?
--
Regards,
Shalin Shekhar Mangar.
Re: How to setup dynamic multicore replication
Posted by Thijs <vo...@gmail.com>.
Hi, Thanks.
That was my second option.
But I was hoping that the master and slaves could find that out for
themselves. As now I have to also have my 'updater software' know about
all the slaves (and maybe even their state). Which it previously had no
idea about.
This way I can't just plugin a 'empty' slave that knows where it's
master is and have it pull in all the required cores and indexes.
Thijs
On 8-12-2009 14:25, Joe Kessel wrote:
>
> Hi,
>
> In my environment I create cores on the fly, then replicate the core to all of the slaves. I first create the core on the master and persist the solr.xml via the CoreAdmin API. I then do the same on each of my slaves. After loading / committing / optimizing the data on the master I send the replication request to each of the slaves. So each slave's replication handler http://<slave_host_port>/solr/<core_name>/replication gets a request to fetch index which includes the master url http://<master_host_port>/solr/<core_name>/replication;
>
>
>
> The slave's solrconf.xml has no mention of the master as it is all done progromatically. You need to specify the core name in the url, and if you haven't created the core on the master it will result in error.
>
>
>
> I don't create a new core everytime I update, but I do have the slaves fetch the index after every update. My first attempt to set the polling did not seem to work, and have not had a chance to revisit. I have not found a way to persist the solrconfig.xml with the updates to the slave list, so the control / management is within my application.
>
>
>
> Hope this highlevel overview helps.
>
>
>
> Joe
>
>
>
>> Date: Tue, 8 Dec 2009 12:42:12 +0100
>> From: vonk.thijs@gmail.com
>> To: solr-user@lucene.apache.org
>> Subject: Re: How to setup dynamic multicore replication
>>
>> But the slave never gets the message that a core is created...
>> at least not in my setup...
>> So it never starts replicating...
>>
>>
>> On 8-12-2009 12:13, Noble Paul നോബിള് नोब्ळ् wrote:
>>> On Tue, Dec 8, 2009 at 2:43 PM, Thijs<vo...@gmail.com> wrote:
>>>> Hi
>>>>
>>>> I need some help setting up dynamic multicore replication.
>>>>
>>>> We are changing our setup from a replicated single core index with multiple
>>>> document types, as described on the wiki[1], to a dynamic multicore setup.
>>>> We need this so that we can display facets with a zero count that are unique
>>>> to the document 'type'.
>>>>
>>>> So when indexing new documents we want to create new cores on the fly using
>>>> the CoreAdminHandler through SolrJ.
>>>>
>>>> What I can't figure out is how I setup solr.xml and solrconfig.xml so that a
>>>> core automatically is also replicated from the master to it's slaves once
>>>> it's created.
>>>>
>>>> I have a solr.xml that starts like this:
>>>>
>>>> <?xml version='1.0' encoding='UTF-8'?>
>>>> <solr persistent="true">
>>>> <cores adminPath="/admin/cores">
>>>> </cores>
>>>> </solr>
>>>>
>>>> and the replication part of solrconfig.xml
>>>> master:
>>>> <requestHandler name="/replication" class="solr.ReplicationHandler">
>>>> <lst name="master">
>>>> <str name="replicateAfter">startup</str>
>>>> <str name="replicateAfter">optimize</str>
>>>> <str name="confFiles">schema.xml</str>
>>>> </lst>
>>>> </requestHandler>
>>>>
>>>> slave:
>>>> <requestHandler name="/replication" class="solr.ReplicationHandler">
>>>> <lst name="slave">
>>>> <str name="masterUrl">http://localhost:8081/solr/replication</str>
>>>> <str name="pollInterval">00:00:20</str>
>>>> </lst>
>>>> </requestHandler>
>>>>
>>>> I think I should change the masterUrl in the slave configuration to
>>>> something like:
>>>> <str
>>>> name="masterUrl">http://localhost:8081/solr/${solr.core.name}/replication</str>
>>>> So that the replication automatically finds the correct core replication
>>>> handler.
>>> if you have dynamically created cores this is the solution.
>>>>
>>>> But how do I tell the slaves a new core is created, and that is should start
>>>> replicating those to?
>>>>
>>>> Thanks in advance.
>>>>
>>>> Thijs
>>>>
>>>> [1]
>>>> http://wiki.apache.org/solr/MultipleIndexes#Flattening_Data_Into_a_Single_Index
>>>>
>>>>
>>>
>>>
>>>
>>
>
> _________________________________________________________________
> Windows Live Hotmail is faster and more secure than ever.
> http://www.microsoft.com/windows/windowslive/hotmail_bl1/hotmail_bl1.aspx?ocid=PID23879::T:WLMTAGL:ON:WL:en-ww:WM_IMHM_1:092009
RE: How to setup dynamic multicore replication
Posted by Joe Kessel <is...@hotmail.com>.
Hi,
In my environment I create cores on the fly, then replicate the core to all of the slaves. I first create the core on the master and persist the solr.xml via the CoreAdmin API. I then do the same on each of my slaves. After loading / committing / optimizing the data on the master I send the replication request to each of the slaves. So each slave's replication handler http://<slave_host_port>/solr/<core_name>/replication gets a request to fetch index which includes the master url http://<master_host_port>/solr/<core_name>/replication;
The slave's solrconf.xml has no mention of the master as it is all done progromatically. You need to specify the core name in the url, and if you haven't created the core on the master it will result in error.
I don't create a new core everytime I update, but I do have the slaves fetch the index after every update. My first attempt to set the polling did not seem to work, and have not had a chance to revisit. I have not found a way to persist the solrconfig.xml with the updates to the slave list, so the control / management is within my application.
Hope this highlevel overview helps.
Joe
> Date: Tue, 8 Dec 2009 12:42:12 +0100
> From: vonk.thijs@gmail.com
> To: solr-user@lucene.apache.org
> Subject: Re: How to setup dynamic multicore replication
>
> But the slave never gets the message that a core is created...
> at least not in my setup...
> So it never starts replicating...
>
>
> On 8-12-2009 12:13, Noble Paul നോബിള് नोब्ळ् wrote:
> > On Tue, Dec 8, 2009 at 2:43 PM, Thijs<vo...@gmail.com> wrote:
> >> Hi
> >>
> >> I need some help setting up dynamic multicore replication.
> >>
> >> We are changing our setup from a replicated single core index with multiple
> >> document types, as described on the wiki[1], to a dynamic multicore setup.
> >> We need this so that we can display facets with a zero count that are unique
> >> to the document 'type'.
> >>
> >> So when indexing new documents we want to create new cores on the fly using
> >> the CoreAdminHandler through SolrJ.
> >>
> >> What I can't figure out is how I setup solr.xml and solrconfig.xml so that a
> >> core automatically is also replicated from the master to it's slaves once
> >> it's created.
> >>
> >> I have a solr.xml that starts like this:
> >>
> >> <?xml version='1.0' encoding='UTF-8'?>
> >> <solr persistent="true">
> >> <cores adminPath="/admin/cores">
> >> </cores>
> >> </solr>
> >>
> >> and the replication part of solrconfig.xml
> >> master:
> >> <requestHandler name="/replication" class="solr.ReplicationHandler">
> >> <lst name="master">
> >> <str name="replicateAfter">startup</str>
> >> <str name="replicateAfter">optimize</str>
> >> <str name="confFiles">schema.xml</str>
> >> </lst>
> >> </requestHandler>
> >>
> >> slave:
> >> <requestHandler name="/replication" class="solr.ReplicationHandler">
> >> <lst name="slave">
> >> <str name="masterUrl">http://localhost:8081/solr/replication</str>
> >> <str name="pollInterval">00:00:20</str>
> >> </lst>
> >> </requestHandler>
> >>
> >> I think I should change the masterUrl in the slave configuration to
> >> something like:
> >> <str
> >> name="masterUrl">http://localhost:8081/solr/${solr.core.name}/replication</str>
> >> So that the replication automatically finds the correct core replication
> >> handler.
> > if you have dynamically created cores this is the solution.
> >>
> >> But how do I tell the slaves a new core is created, and that is should start
> >> replicating those to?
> >>
> >> Thanks in advance.
> >>
> >> Thijs
> >>
> >> [1]
> >> http://wiki.apache.org/solr/MultipleIndexes#Flattening_Data_Into_a_Single_Index
> >>
> >>
> >
> >
> >
>
_________________________________________________________________
Windows Live Hotmail is faster and more secure than ever.
http://www.microsoft.com/windows/windowslive/hotmail_bl1/hotmail_bl1.aspx?ocid=PID23879::T:WLMTAGL:ON:WL:en-ww:WM_IMHM_1:092009
Re: How to setup dynamic multicore replication
Posted by Thijs <vo...@gmail.com>.
But the slave never gets the message that a core is created...
at least not in my setup...
So it never starts replicating...
On 8-12-2009 12:13, Noble Paul നോബിള് नोब्ळ् wrote:
> On Tue, Dec 8, 2009 at 2:43 PM, Thijs<vo...@gmail.com> wrote:
>> Hi
>>
>> I need some help setting up dynamic multicore replication.
>>
>> We are changing our setup from a replicated single core index with multiple
>> document types, as described on the wiki[1], to a dynamic multicore setup.
>> We need this so that we can display facets with a zero count that are unique
>> to the document 'type'.
>>
>> So when indexing new documents we want to create new cores on the fly using
>> the CoreAdminHandler through SolrJ.
>>
>> What I can't figure out is how I setup solr.xml and solrconfig.xml so that a
>> core automatically is also replicated from the master to it's slaves once
>> it's created.
>>
>> I have a solr.xml that starts like this:
>>
>> <?xml version='1.0' encoding='UTF-8'?>
>> <solr persistent="true">
>> <cores adminPath="/admin/cores">
>> </cores>
>> </solr>
>>
>> and the replication part of solrconfig.xml
>> master:
>> <requestHandler name="/replication" class="solr.ReplicationHandler">
>> <lst name="master">
>> <str name="replicateAfter">startup</str>
>> <str name="replicateAfter">optimize</str>
>> <str name="confFiles">schema.xml</str>
>> </lst>
>> </requestHandler>
>>
>> slave:
>> <requestHandler name="/replication" class="solr.ReplicationHandler">
>> <lst name="slave">
>> <str name="masterUrl">http://localhost:8081/solr/replication</str>
>> <str name="pollInterval">00:00:20</str>
>> </lst>
>> </requestHandler>
>>
>> I think I should change the masterUrl in the slave configuration to
>> something like:
>> <str
>> name="masterUrl">http://localhost:8081/solr/${solr.core.name}/replication</str>
>> So that the replication automatically finds the correct core replication
>> handler.
> if you have dynamically created cores this is the solution.
>>
>> But how do I tell the slaves a new core is created, and that is should start
>> replicating those to?
>>
>> Thanks in advance.
>>
>> Thijs
>>
>> [1]
>> http://wiki.apache.org/solr/MultipleIndexes#Flattening_Data_Into_a_Single_Index
>>
>>
>
>
>
Re: How to setup dynamic multicore replication
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
On Tue, Dec 8, 2009 at 2:43 PM, Thijs <vo...@gmail.com> wrote:
> Hi
>
> I need some help setting up dynamic multicore replication.
>
> We are changing our setup from a replicated single core index with multiple
> document types, as described on the wiki[1], to a dynamic multicore setup.
> We need this so that we can display facets with a zero count that are unique
> to the document 'type'.
>
> So when indexing new documents we want to create new cores on the fly using
> the CoreAdminHandler through SolrJ.
>
> What I can't figure out is how I setup solr.xml and solrconfig.xml so that a
> core automatically is also replicated from the master to it's slaves once
> it's created.
>
> I have a solr.xml that starts like this:
>
> <?xml version='1.0' encoding='UTF-8'?>
> <solr persistent="true">
> <cores adminPath="/admin/cores">
> </cores>
> </solr>
>
> and the replication part of solrconfig.xml
> master:
> <requestHandler name="/replication" class="solr.ReplicationHandler">
> <lst name="master">
> <str name="replicateAfter">startup</str>
> <str name="replicateAfter">optimize</str>
> <str name="confFiles">schema.xml</str>
> </lst>
> </requestHandler>
>
> slave:
> <requestHandler name="/replication" class="solr.ReplicationHandler">
> <lst name="slave">
> <str name="masterUrl">http://localhost:8081/solr/replication</str>
> <str name="pollInterval">00:00:20</str>
> </lst>
> </requestHandler>
>
> I think I should change the masterUrl in the slave configuration to
> something like:
> <str
> name="masterUrl">http://localhost:8081/solr/${solr.core.name}/replication</str>
> So that the replication automatically finds the correct core replication
> handler.
if you have dynamically created cores this is the solution.
>
> But how do I tell the slaves a new core is created, and that is should start
> replicating those to?
>
> Thanks in advance.
>
> Thijs
>
> [1]
> http://wiki.apache.org/solr/MultipleIndexes#Flattening_Data_Into_a_Single_Index
>
>
--
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com