You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bryan Talbot <bt...@aeriagames.com> on 2009/05/12 19:12:38 UTC
Replication master+slave
For replication in 1.4, the wiki at http://wiki.apache.org/solr/SolrReplication
says that a node can be both the master and a slave:
A node can act as both master and slave. In that case both the master
and slave configuration lists need to be present inside the
ReplicationHandler requestHandler in the solrconfig.xml.
What does this mean? Does the core then poll itself for updates?
I'd like to have a single set of configuration files that are shared
by masters and slaves and avoid duplicating configuration details in
multiple files (one for master and one for slave) to ease management
and failover. Is this possible?
When I attempt to setup a multi server master-slave configuration and
include both master and slave replication configuration options, I
into some problems. I'm running a nightly build from May 7.
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<lst name="master">
<str name="replicateAfter">commit</str>
</lst>
<lst name="slave">
<str name="masterUrl">http://master_core01:8983/solr/core01/
replication</str>
<str name="pollInterval">00:00:60</str>
</lst>
</requestHandler>
When the replication admin page (http://master_core01:8983/solr/core01/
admin/replication/index.jsp) is visited, the severe error show below
appears in the solr log. The server is otherwise idle so there is no
reason all threads should be busy unless the replication code is
getting itself into a loop.
What's the right way to do this?
May 11, 2009 8:01:22 PM org.apache.tomcat.util.threads.ThreadPool
logFull
SEVERE: All threads (150) are currently busy, waiting. Increase
maxThreads (150) or check the servlet status
May 11, 2009 8:01:41 PM org.apache.solr.handler.ReplicationHandler
getReplicationDetails
WARNING: Exception while invoking a 'details' method on master
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:
218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:
237)
at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at
org
.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:
1116)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager
$
HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:
1413)
at
org
.apache
.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:
1973)
at
org
.apache
.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:
1735)
at
org
.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:
1098)
at
org
.apache
.commons
.httpclient
.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org
.apache
.commons
.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:
171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:
323)
at
org
.apache.solr.handler.SnapPuller.getNamedListResponse(SnapPuller.java:
183)
at
org.apache.solr.handler.SnapPuller.getCommandResponse(SnapPuller.java:
178)
at
org
.apache
.solr
.handler
.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:555)
at
org
.apache
.solr
.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:
147)
at
org
.apache
.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1330)
at
org
.apache.jsp.admin.replication.index_jsp.executeCommand(index_jsp.java:
34)
at
org.apache.jsp.admin.replication.index_jsp._jspService(index_jsp.java:
208)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org
.apache
.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:331)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:329)
at
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
269)
at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org
.apache
.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:
679)
at
org
.apache
.catalina
.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:
461)
at
org
.apache
.catalina
.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:399)
at
org
.apache
.catalina
.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
at
org
.apache
.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:273)
at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
215)
at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org
.apache
.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
213)
at
org
.apache
.catalina.core.StandardContextValve.invoke(StandardContextValve.java:
172)
at
org
.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
127)
at
org
.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
117)
at
org
.apache
.catalina
.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:
482)
at
org
.apache
.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
174)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
875)
at org.apache.coyote.http11.Http11BaseProtocol
$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org
.apache
.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org
.apache
.tomcat
.util
.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:
81)
at org.apache.tomcat.util.threads.ThreadPool
$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)
-Bryan
Re: Replication master+slave
Posted by Michael Ludwig <ml...@as-guides.com>.
Bryan Talbot schrieb:
> So how are people managing solrconfig.xml files which are largely the
> same other than differences for replication?
>
> I don't think it's a "good thing" to maintain two copies of the same
> file and I'd like to avoid that. Maybe enabling the XInclude feature
> in DocumentBuilders would make it possible to modularize configuration
> files to make this possible?
This is already possible using the XML feature called "entities",
more precisely "external general parsed entities" (EGPE). I've never
seen a parser that doesn't do entities.
C:\MILU\dev\XML # type egpe-net.xml
<!DOCTYPE Urmel [
<!ENTITY egpe_from_the_net
SYSTEM "http://lobster.as-guides.com/ds/solr.schema.ent" >
<!ENTITY egpe_from_the_local_disk
SYSTEM "egpe-local.ent" >
]>
<Urmel>
&egpe_from_the_net;
&egpe_from_the_local_disk;
</Urmel>
C:\MILU\dev\XML # type egpe-local.ent
<eins/>
<zwei/>
<drei/>
Michael Ludwig
Re: Replication master+slave
Posted by Peter Wolanin <pe...@acquia.com>.
Indeed - that looks nice - having some kind of conditional includes
would make many things easier.
-Peter
On Wed, May 13, 2009 at 4:22 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>
> This looks nice and simple. I don't know enough about this stuff to see any issues. If there are no issues.....?
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: Bryan Talbot <bt...@aeriagames.com>
>> To: solr-user@lucene.apache.org
>> Sent: Wednesday, May 13, 2009 11:26:41 AM
>> Subject: Re: Replication master+slave
>>
>> I see that Nobel's final comment in SOLR-1154 is that config files need to be
>> able to include snippets from external files. In my limited testing, a simple
>> patch to enable XInclude support seems to work.
>>
>>
>>
>> --- src/java/org/apache/solr/core/Config.java (revision 774137)
>> +++ src/java/org/apache/solr/core/Config.java (working copy)
>> @@ -100,8 +100,10 @@
>> if (lis == null) {
>> lis = loader.openConfig(name);
>> }
>> - javax.xml.parsers.DocumentBuilder builder =
>> DocumentBuilderFactory.newInstance().newDocumentBuilder();
>> - doc = builder.parse(lis);
>> + javax.xml.parsers.DocumentBuilderFactory dbf =
>> DocumentBuilderFactory.newInstance();
>> + dbf.setNamespaceAware(true);
>> + dbf.setXIncludeAware(true);
>> + doc = dbf.newDocumentBuilder().parse(lis);
>>
>> DOMUtil.substituteProperties(doc, loader.getCoreProperties());
>> } catch (ParserConfigurationException e) {
>>
>>
>>
>> This allows a clause like this to include the contents of replication.xml if it
>> exists. If it's not found an exception will be thrown.
>>
>>
>>
>> href="http://localhost:8983/solr/corename/admin/file/?file=replication.xml"
>> xmlns:xi="http://www.w3.org/2001/XInclude">
>>
>>
>>
>> If the file is optional and no exception should be thrown if the file is
>> missing, simply include a fallback action: in this case the fallback is empty
>> and does nothing.
>>
>>
>>
>> href="http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml"
>> xmlns:xi="http://www.w3.org/2001/XInclude">
>>
>>
>>
>>
>> -Bryan
>>
>>
>>
>>
>> On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote:
>>
>> > I was looking at the same problem, and had a discussion with Noble. You can
>> > use a hack to achieve what you want, see
>> >
>> > https://issues.apache.org/jira/browse/SOLR-1154
>> >
>> > Thanks,
>> >
>> > Jianhan
>> >
>> >
>> > On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot wrote:
>> >
>> >> So how are people managing solrconfig.xml files which are largely the same
>> >> other than differences for replication?
>> >>
>> >> I don't think it's a "good thing" to maintain two copies of the same file
>> >> and I'd like to avoid that. Maybe enabling the XInclude feature in
>> >> DocumentBuilders would make it possible to modularize configuration files to
>> >> make this possible?
>> >>
>> >>
>> >>
>> http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)
>> >>
>> >>
>> >> -Bryan
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote:
>> >>
>> >> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
>> >>>> wrote:
>> >>>
>> >>> For replication in 1.4, the wiki at
>> >>>> http://wiki.apache.org/solr/SolrReplication says that a node can be both
>> >>>> the master and a slave:
>> >>>>
>> >>>> A node can act as both master and slave. In that case both the master and
>> >>>> slave configuration lists need to be present inside the
>> >>>> ReplicationHandler
>> >>>> requestHandler in the solrconfig.xml.
>> >>>>
>> >>>> What does this mean? Does the core then poll itself for updates?
>> >>>>
>> >>>
>> >>>
>> >>> No. This type of configuration is meant for "repeaters". Suppose there are
>> >>> slaves in multiple data-centers (say data center A and B). There is always
>> >>> a
>> >>> single master (say in A). One of the slaves in B is used as a master for
>> >>> the
>> >>> other slaves in B. Therefore, this one slave in B is both a master as well
>> >>> as the slave.
>> >>>
>> >>>
>> >>>
>> >>>> I'd like to have a single set of configuration files that are shared by
>> >>>> masters and slaves and avoid duplicating configuration details in
>> >>>> multiple
>> >>>> files (one for master and one for slave) to ease management and failover.
>> >>>> Is this possible?
>> >>>>
>> >>>>
>> >>> You wouldn't want the master to be a slave. So I guess you'd need to have
>> >>> a
>> >>> separate file. Also, it needs to be a separate file so that the slave does
>> >>> not become a master when the solrconfig.xml is replicated.
>> >>>
>> >>>
>> >>>
>> >>>> When I attempt to setup a multi server master-slave configuration and
>> >>>> include both master and slave replication configuration options, I into
>> >>>> some
>> >>>> problems. I'm running a nightly build from May 7.
>> >>>>
>> >>>>
>> >>> Not sure what happened. Is that the url for this solr (meaning same solr
>> >>> url
>> >>> is master and slave of itself)? If yes, that is not a valid configuration.
>> >>>
>> >>> --
>> >>> Regards,
>> >>> Shalin Shekhar Mangar.
>> >>>
>> >>
>> >>
>
>
--
Peter M. Wolanin, Ph.D.
Momentum Specialist, Acquia. Inc.
peter.wolanin@acquia.com
Re: Replication master+slave
Posted by Otis Gospodnetic <ot...@yahoo.com>.
This looks nice and simple. I don't know enough about this stuff to see any issues. If there are no issues.....?
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Bryan Talbot <bt...@aeriagames.com>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, May 13, 2009 11:26:41 AM
> Subject: Re: Replication master+slave
>
> I see that Nobel's final comment in SOLR-1154 is that config files need to be
> able to include snippets from external files. In my limited testing, a simple
> patch to enable XInclude support seems to work.
>
>
>
> --- src/java/org/apache/solr/core/Config.java (revision 774137)
> +++ src/java/org/apache/solr/core/Config.java (working copy)
> @@ -100,8 +100,10 @@
> if (lis == null) {
> lis = loader.openConfig(name);
> }
> - javax.xml.parsers.DocumentBuilder builder =
> DocumentBuilderFactory.newInstance().newDocumentBuilder();
> - doc = builder.parse(lis);
> + javax.xml.parsers.DocumentBuilderFactory dbf =
> DocumentBuilderFactory.newInstance();
> + dbf.setNamespaceAware(true);
> + dbf.setXIncludeAware(true);
> + doc = dbf.newDocumentBuilder().parse(lis);
>
> DOMUtil.substituteProperties(doc, loader.getCoreProperties());
> } catch (ParserConfigurationException e) {
>
>
>
> This allows a clause like this to include the contents of replication.xml if it
> exists. If it's not found an exception will be thrown.
>
>
>
> href="http://localhost:8983/solr/corename/admin/file/?file=replication.xml"
> xmlns:xi="http://www.w3.org/2001/XInclude">
>
>
>
> If the file is optional and no exception should be thrown if the file is
> missing, simply include a fallback action: in this case the fallback is empty
> and does nothing.
>
>
>
> href="http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml"
> xmlns:xi="http://www.w3.org/2001/XInclude">
>
>
>
>
> -Bryan
>
>
>
>
> On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote:
>
> > I was looking at the same problem, and had a discussion with Noble. You can
> > use a hack to achieve what you want, see
> >
> > https://issues.apache.org/jira/browse/SOLR-1154
> >
> > Thanks,
> >
> > Jianhan
> >
> >
> > On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot wrote:
> >
> >> So how are people managing solrconfig.xml files which are largely the same
> >> other than differences for replication?
> >>
> >> I don't think it's a "good thing" to maintain two copies of the same file
> >> and I'd like to avoid that. Maybe enabling the XInclude feature in
> >> DocumentBuilders would make it possible to modularize configuration files to
> >> make this possible?
> >>
> >>
> >>
> http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)
> >>
> >>
> >> -Bryan
> >>
> >>
> >>
> >>
> >>
> >> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote:
> >>
> >> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
> >>>> wrote:
> >>>
> >>> For replication in 1.4, the wiki at
> >>>> http://wiki.apache.org/solr/SolrReplication says that a node can be both
> >>>> the master and a slave:
> >>>>
> >>>> A node can act as both master and slave. In that case both the master and
> >>>> slave configuration lists need to be present inside the
> >>>> ReplicationHandler
> >>>> requestHandler in the solrconfig.xml.
> >>>>
> >>>> What does this mean? Does the core then poll itself for updates?
> >>>>
> >>>
> >>>
> >>> No. This type of configuration is meant for "repeaters". Suppose there are
> >>> slaves in multiple data-centers (say data center A and B). There is always
> >>> a
> >>> single master (say in A). One of the slaves in B is used as a master for
> >>> the
> >>> other slaves in B. Therefore, this one slave in B is both a master as well
> >>> as the slave.
> >>>
> >>>
> >>>
> >>>> I'd like to have a single set of configuration files that are shared by
> >>>> masters and slaves and avoid duplicating configuration details in
> >>>> multiple
> >>>> files (one for master and one for slave) to ease management and failover.
> >>>> Is this possible?
> >>>>
> >>>>
> >>> You wouldn't want the master to be a slave. So I guess you'd need to have
> >>> a
> >>> separate file. Also, it needs to be a separate file so that the slave does
> >>> not become a master when the solrconfig.xml is replicated.
> >>>
> >>>
> >>>
> >>>> When I attempt to setup a multi server master-slave configuration and
> >>>> include both master and slave replication configuration options, I into
> >>>> some
> >>>> problems. I'm running a nightly build from May 7.
> >>>>
> >>>>
> >>> Not sure what happened. Is that the url for this solr (meaning same solr
> >>> url
> >>> is master and slave of itself)? If yes, that is not a valid configuration.
> >>>
> >>> --
> >>> Regards,
> >>> Shalin Shekhar Mangar.
> >>>
> >>
> >>
Re: Replication master+slave
Posted by Bryan Talbot <bt...@aeriagames.com>.
https://issues.apache.org/jira/browse/SOLR-1167
-Bryan
On May 13, 2009, at May 13, 7:20 PM, Otis Gospodnetic wrote:
>
> Bryan, maybe it's time to stick this in JIRA?
> http://wiki.apache.org/solr/HowToContribute
>
> Thanks,
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: Bryan Talbot <bt...@aeriagames.com>
>> To: solr-user@lucene.apache.org
>> Sent: Wednesday, May 13, 2009 10:11:21 PM
>> Subject: Re: Replication master+slave
>>
>> I think the patch I included earlier covers solr core, but it looks
>> like at
>> least some other extensions (DIH) create and use their own XML
>> parser. So, if
>> this functionality is to extend to all XML files, those will need
>> similar
>> patches.
>>
>> Here's one for DIH:
>>
>> --- src/main/java/org/apache/solr/handler/dataimport/
>> DataImporter.java
>> (revision 774137)
>> +++ src/main/java/org/apache/solr/handler/dataimport/
>> DataImporter.java (working
>> copy)
>> @@ -148,8 +148,10 @@
>> void loadDataConfig(String configFile) {
>>
>> try {
>> - DocumentBuilder builder = DocumentBuilderFactory.newInstance()
>> - .newDocumentBuilder();
>> + DocumentBuilderFactory dbf =
>> DocumentBuilderFactory.newInstance();
>> + dbf.setNamespaceAware(true);
>> + dbf.setXIncludeAware(true);
>> + DocumentBuilder builder = dbf.newDocumentBuilder();
>> Document document = builder.parse(new InputSource(new
>> StringReader(
>> configFile)));
>>
>>
>>
>> The only down side I can see to this is it doesn't offer very
>> expressive
>> conditional inclusion: the file is included if it's present
>> otherwise fallback
>> inclusions can be used. It's also specific to XML files and
>> obviously won't
>> work for other types of configuration files. However, it is simple
>> and
>> effective.
>>
>>
>> -Bryan
>>
>>
>>
>>
>> On May 13, 2009, at May 13, 6:36 PM, Otis Gospodnetic wrote:
>>
>>>
>>> Coincidentally, from
>> http://www.cloudera.com/blog/2009/05/07/what%E2%80%99s-new-in-hadoop-core-020/
>> :
>>>
>>> "Hadoop configuration files now support XInclude elements for
>>> including
>> portions of another configuration file (HADOOP-4944). This
>> mechanism allows you
>> to make configuration files more modular and reusable."
>>>
>>> So "others are doing it, too".
>>>
>>> Otis
>>> --
>>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>>
>>>
>>>
>>> ----- Original Message ----
>>>> From: Bryan Talbot
>>>> To: solr-user@lucene.apache.org
>>>> Sent: Wednesday, May 13, 2009 11:26:41 AM
>>>> Subject: Re: Replication master+slave
>>>>
>>>> I see that Nobel's final comment in SOLR-1154 is that config
>>>> files need to be
>>>> able to include snippets from external files. In my limited
>>>> testing, a
>> simple
>>>> patch to enable XInclude support seems to work.
>>>>
>>>>
>>>>
>>>> --- src/java/org/apache/solr/core/Config.java (revision 774137)
>>>> +++ src/java/org/apache/solr/core/Config.java (working copy)
>>>> @@ -100,8 +100,10 @@
>>>> if (lis == null) {
>>>> lis = loader.openConfig(name);
>>>> }
>>>> - javax.xml.parsers.DocumentBuilder builder =
>>>> DocumentBuilderFactory.newInstance().newDocumentBuilder();
>>>> - doc = builder.parse(lis);
>>>> + javax.xml.parsers.DocumentBuilderFactory dbf =
>>>> DocumentBuilderFactory.newInstance();
>>>> + dbf.setNamespaceAware(true);
>>>> + dbf.setXIncludeAware(true);
>>>> + doc = dbf.newDocumentBuilder().parse(lis);
>>>>
>>>> DOMUtil.substituteProperties(doc, loader.getCoreProperties());
>>>> } catch (ParserConfigurationException e) {
>>>>
>>>>
>>>>
>>>> This allows a clause like this to include the contents of
>>>> replication.xml if
>> it
>>>> exists. If it's not found an exception will be thrown.
>>>>
>>>>
>>>>
>>>> href="http://localhost:8983/solr/corename/admin/file/?file=replication.xml
>>>> "
>>>> xmlns:xi="http://www.w3.org/2001/XInclude">
>>>>
>>>>
>>>>
>>>> If the file is optional and no exception should be thrown if the
>>>> file is
>>>> missing, simply include a fallback action: in this case the
>>>> fallback is empty
>>>> and does nothing.
>>>>
>>>>
>>>>
>>>> href="http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml
>>>> "
>>>> xmlns:xi="http://www.w3.org/2001/XInclude">
>>>>
>>>>
>>>>
>>>>
>>>> -Bryan
>>>>
>>>>
>>>>
>>>>
>>>> On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote:
>>>>
>>>>> I was looking at the same problem, and had a discussion with
>>>>> Noble. You can
>>>>> use a hack to achieve what you want, see
>>>>>
>>>>> https://issues.apache.org/jira/browse/SOLR-1154
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jianhan
>>>>>
>>>>>
>>>>> On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot wrote:
>>>>>
>>>>>> So how are people managing solrconfig.xml files which are
>>>>>> largely the same
>>>>>> other than differences for replication?
>>>>>>
>>>>>> I don't think it's a "good thing" to maintain two copies of the
>>>>>> same file
>>>>>> and I'd like to avoid that. Maybe enabling the XInclude
>>>>>> feature in
>>>>>> DocumentBuilders would make it possible to modularize
>>>>>> configuration files
>> to
>>>>>> make this possible?
>>>>>>
>>>>>>
>>>>>>
>>>>
>> http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)
>>>>>>
>>>>>>
>>>>>> -Bryan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar
>>>>>> wrote:
>>>>>>
>>>>>> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
>>>>>>>> wrote:
>>>>>>>
>>>>>>> For replication in 1.4, the wiki at
>>>>>>>> http://wiki.apache.org/solr/SolrReplication says that a node
>>>>>>>> can be both
>>>>>>>> the master and a slave:
>>>>>>>>
>>>>>>>> A node can act as both master and slave. In that case both
>>>>>>>> the master and
>>>>>>>> slave configuration lists need to be present inside the
>>>>>>>> ReplicationHandler
>>>>>>>> requestHandler in the solrconfig.xml.
>>>>>>>>
>>>>>>>> What does this mean? Does the core then poll itself for
>>>>>>>> updates?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> No. This type of configuration is meant for "repeaters".
>>>>>>> Suppose there are
>>>>>>> slaves in multiple data-centers (say data center A and B).
>>>>>>> There is always
>>>>>>> a
>>>>>>> single master (say in A). One of the slaves in B is used as a
>>>>>>> master for
>>>>>>> the
>>>>>>> other slaves in B. Therefore, this one slave in B is both a
>>>>>>> master as well
>>>>>>> as the slave.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I'd like to have a single set of configuration files that are
>>>>>>>> shared by
>>>>>>>> masters and slaves and avoid duplicating configuration
>>>>>>>> details in
>>>>>>>> multiple
>>>>>>>> files (one for master and one for slave) to ease management
>>>>>>>> and failover.
>>>>>>>> Is this possible?
>>>>>>>>
>>>>>>>>
>>>>>>> You wouldn't want the master to be a slave. So I guess you'd
>>>>>>> need to have
>>>>>>> a
>>>>>>> separate file. Also, it needs to be a separate file so that
>>>>>>> the slave does
>>>>>>> not become a master when the solrconfig.xml is replicated.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> When I attempt to setup a multi server master-slave
>>>>>>>> configuration and
>>>>>>>> include both master and slave replication configuration
>>>>>>>> options, I into
>>>>>>>> some
>>>>>>>> problems. I'm running a nightly build from May 7.
>>>>>>>>
>>>>>>>>
>>>>>>> Not sure what happened. Is that the url for this solr (meaning
>>>>>>> same solr
>>>>>>> url
>>>>>>> is master and slave of itself)? If yes, that is not a valid
>>>>>>> configuration.
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shalin Shekhar Mangar.
>>>>>>>
>>>>>>
>>>>>>
>>>
>
Re: Replication master+slave
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
There's a related issue open.
https://issues.apache.org/jira/browse/SOLR-712
On Thu, May 14, 2009 at 7:50 AM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
>
> Bryan, maybe it's time to stick this in JIRA?
> http://wiki.apache.org/solr/HowToContribute
>
> Thanks,
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
> > From: Bryan Talbot <bt...@aeriagames.com>
> > To: solr-user@lucene.apache.org
> > Sent: Wednesday, May 13, 2009 10:11:21 PM
> > Subject: Re: Replication master+slave
> >
> > I think the patch I included earlier covers solr core, but it looks like
> at
> > least some other extensions (DIH) create and use their own XML parser.
> So, if
> > this functionality is to extend to all XML files, those will need similar
> > patches.
> >
> > Here's one for DIH:
> >
> > --- src/main/java/org/apache/solr/handler/dataimport/DataImporter.java
> > (revision 774137)
> > +++ src/main/java/org/apache/solr/handler/dataimport/DataImporter.java
> (working
> > copy)
> > @@ -148,8 +148,10 @@
> > void loadDataConfig(String configFile) {
> >
> > try {
> > - DocumentBuilder builder = DocumentBuilderFactory.newInstance()
> > - .newDocumentBuilder();
> > + DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
> > + dbf.setNamespaceAware(true);
> > + dbf.setXIncludeAware(true);
> > + DocumentBuilder builder = dbf.newDocumentBuilder();
> > Document document = builder.parse(new InputSource(new
> StringReader(
> > configFile)));
> >
> >
> >
> > The only down side I can see to this is it doesn't offer very expressive
> > conditional inclusion: the file is included if it's present otherwise
> fallback
> > inclusions can be used. It's also specific to XML files and obviously
> won't
> > work for other types of configuration files. However, it is simple and
> > effective.
> >
> >
> > -Bryan
> >
> >
> >
> >
> > On May 13, 2009, at May 13, 6:36 PM, Otis Gospodnetic wrote:
> >
> > >
> > > Coincidentally, from
> >
> http://www.cloudera.com/blog/2009/05/07/what%E2%80%99s-new-in-hadoop-core-020/:
> > >
> > > "Hadoop configuration files now support XInclude elements for including
> > portions of another configuration file (HADOOP-4944). This mechanism
> allows you
> > to make configuration files more modular and reusable."
> > >
> > > So "others are doing it, too".
> > >
> > > Otis
> > > --
> > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > >
> > >
> > >
> > > ----- Original Message ----
> > >> From: Bryan Talbot
> > >> To: solr-user@lucene.apache.org
> > >> Sent: Wednesday, May 13, 2009 11:26:41 AM
> > >> Subject: Re: Replication master+slave
> > >>
> > >> I see that Nobel's final comment in SOLR-1154 is that config files
> need to be
> > >> able to include snippets from external files. In my limited testing,
> a
> > simple
> > >> patch to enable XInclude support seems to work.
> > >>
> > >>
> > >>
> > >> --- src/java/org/apache/solr/core/Config.java (revision 774137)
> > >> +++ src/java/org/apache/solr/core/Config.java (working copy)
> > >> @@ -100,8 +100,10 @@
> > >> if (lis == null) {
> > >> lis = loader.openConfig(name);
> > >> }
> > >> - javax.xml.parsers.DocumentBuilder builder =
> > >> DocumentBuilderFactory.newInstance().newDocumentBuilder();
> > >> - doc = builder.parse(lis);
> > >> + javax.xml.parsers.DocumentBuilderFactory dbf =
> > >> DocumentBuilderFactory.newInstance();
> > >> + dbf.setNamespaceAware(true);
> > >> + dbf.setXIncludeAware(true);
> > >> + doc = dbf.newDocumentBuilder().parse(lis);
> > >>
> > >> DOMUtil.substituteProperties(doc, loader.getCoreProperties());
> > >> } catch (ParserConfigurationException e) {
> > >>
> > >>
> > >>
> > >> This allows a clause like this to include the contents of
> replication.xml if
> > it
> > >> exists. If it's not found an exception will be thrown.
> > >>
> > >>
> > >>
> > >> href="
> http://localhost:8983/solr/corename/admin/file/?file=replication.xml"
> > >> xmlns:xi="http://www.w3.org/2001/XInclude">
> > >>
> > >>
> > >>
> > >> If the file is optional and no exception should be thrown if the file
> is
> > >> missing, simply include a fallback action: in this case the fallback
> is empty
> > >> and does nothing.
> > >>
> > >>
> > >>
> > >> href="
> http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml"
> > >> xmlns:xi="http://www.w3.org/2001/XInclude">
> > >>
> > >>
> > >>
> > >>
> > >> -Bryan
> > >>
> > >>
> > >>
> > >>
> > >> On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote:
> > >>
> > >>> I was looking at the same problem, and had a discussion with Noble.
> You can
> > >>> use a hack to achieve what you want, see
> > >>>
> > >>> https://issues.apache.org/jira/browse/SOLR-1154
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jianhan
> > >>>
> > >>>
> > >>> On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot wrote:
> > >>>
> > >>>> So how are people managing solrconfig.xml files which are largely
> the same
> > >>>> other than differences for replication?
> > >>>>
> > >>>> I don't think it's a "good thing" to maintain two copies of the same
> file
> > >>>> and I'd like to avoid that. Maybe enabling the XInclude feature in
> > >>>> DocumentBuilders would make it possible to modularize configuration
> files
> > to
> > >>>> make this possible?
> > >>>>
> > >>>>
> > >>>>
> > >>
> >
> http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)<http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware%28boolean%29>
> > >>>>
> > >>>>
> > >>>> -Bryan
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote:
> > >>>>
> > >>>> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
> > >>>>>> wrote:
> > >>>>>
> > >>>>> For replication in 1.4, the wiki at
> > >>>>>> http://wiki.apache.org/solr/SolrReplication says that a node can
> be both
> > >>>>>> the master and a slave:
> > >>>>>>
> > >>>>>> A node can act as both master and slave. In that case both the
> master and
> > >>>>>> slave configuration lists need to be present inside the
> > >>>>>> ReplicationHandler
> > >>>>>> requestHandler in the solrconfig.xml.
> > >>>>>>
> > >>>>>> What does this mean? Does the core then poll itself for updates?
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>> No. This type of configuration is meant for "repeaters". Suppose
> there are
> > >>>>> slaves in multiple data-centers (say data center A and B). There is
> always
> > >>>>> a
> > >>>>> single master (say in A). One of the slaves in B is used as a
> master for
> > >>>>> the
> > >>>>> other slaves in B. Therefore, this one slave in B is both a master
> as well
> > >>>>> as the slave.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>> I'd like to have a single set of configuration files that are
> shared by
> > >>>>>> masters and slaves and avoid duplicating configuration details in
> > >>>>>> multiple
> > >>>>>> files (one for master and one for slave) to ease management and
> failover.
> > >>>>>> Is this possible?
> > >>>>>>
> > >>>>>>
> > >>>>> You wouldn't want the master to be a slave. So I guess you'd need
> to have
> > >>>>> a
> > >>>>> separate file. Also, it needs to be a separate file so that the
> slave does
> > >>>>> not become a master when the solrconfig.xml is replicated.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>> When I attempt to setup a multi server master-slave configuration
> and
> > >>>>>> include both master and slave replication configuration options, I
> into
> > >>>>>> some
> > >>>>>> problems. I'm running a nightly build from May 7.
> > >>>>>>
> > >>>>>>
> > >>>>> Not sure what happened. Is that the url for this solr (meaning same
> solr
> > >>>>> url
> > >>>>> is master and slave of itself)? If yes, that is not a valid
> configuration.
> > >>>>>
> > >>>>> --
> > >>>>> Regards,
> > >>>>> Shalin Shekhar Mangar.
> > >>>>>
> > >>>>
> > >>>>
> > >
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: Replication master+slave
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Bryan, maybe it's time to stick this in JIRA?
http://wiki.apache.org/solr/HowToContribute
Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Bryan Talbot <bt...@aeriagames.com>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, May 13, 2009 10:11:21 PM
> Subject: Re: Replication master+slave
>
> I think the patch I included earlier covers solr core, but it looks like at
> least some other extensions (DIH) create and use their own XML parser. So, if
> this functionality is to extend to all XML files, those will need similar
> patches.
>
> Here's one for DIH:
>
> --- src/main/java/org/apache/solr/handler/dataimport/DataImporter.java
> (revision 774137)
> +++ src/main/java/org/apache/solr/handler/dataimport/DataImporter.java (working
> copy)
> @@ -148,8 +148,10 @@
> void loadDataConfig(String configFile) {
>
> try {
> - DocumentBuilder builder = DocumentBuilderFactory.newInstance()
> - .newDocumentBuilder();
> + DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
> + dbf.setNamespaceAware(true);
> + dbf.setXIncludeAware(true);
> + DocumentBuilder builder = dbf.newDocumentBuilder();
> Document document = builder.parse(new InputSource(new StringReader(
> configFile)));
>
>
>
> The only down side I can see to this is it doesn't offer very expressive
> conditional inclusion: the file is included if it's present otherwise fallback
> inclusions can be used. It's also specific to XML files and obviously won't
> work for other types of configuration files. However, it is simple and
> effective.
>
>
> -Bryan
>
>
>
>
> On May 13, 2009, at May 13, 6:36 PM, Otis Gospodnetic wrote:
>
> >
> > Coincidentally, from
> http://www.cloudera.com/blog/2009/05/07/what%E2%80%99s-new-in-hadoop-core-020/ :
> >
> > "Hadoop configuration files now support XInclude elements for including
> portions of another configuration file (HADOOP-4944). This mechanism allows you
> to make configuration files more modular and reusable."
> >
> > So "others are doing it, too".
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: Bryan Talbot
> >> To: solr-user@lucene.apache.org
> >> Sent: Wednesday, May 13, 2009 11:26:41 AM
> >> Subject: Re: Replication master+slave
> >>
> >> I see that Nobel's final comment in SOLR-1154 is that config files need to be
> >> able to include snippets from external files. In my limited testing, a
> simple
> >> patch to enable XInclude support seems to work.
> >>
> >>
> >>
> >> --- src/java/org/apache/solr/core/Config.java (revision 774137)
> >> +++ src/java/org/apache/solr/core/Config.java (working copy)
> >> @@ -100,8 +100,10 @@
> >> if (lis == null) {
> >> lis = loader.openConfig(name);
> >> }
> >> - javax.xml.parsers.DocumentBuilder builder =
> >> DocumentBuilderFactory.newInstance().newDocumentBuilder();
> >> - doc = builder.parse(lis);
> >> + javax.xml.parsers.DocumentBuilderFactory dbf =
> >> DocumentBuilderFactory.newInstance();
> >> + dbf.setNamespaceAware(true);
> >> + dbf.setXIncludeAware(true);
> >> + doc = dbf.newDocumentBuilder().parse(lis);
> >>
> >> DOMUtil.substituteProperties(doc, loader.getCoreProperties());
> >> } catch (ParserConfigurationException e) {
> >>
> >>
> >>
> >> This allows a clause like this to include the contents of replication.xml if
> it
> >> exists. If it's not found an exception will be thrown.
> >>
> >>
> >>
> >> href="http://localhost:8983/solr/corename/admin/file/?file=replication.xml"
> >> xmlns:xi="http://www.w3.org/2001/XInclude">
> >>
> >>
> >>
> >> If the file is optional and no exception should be thrown if the file is
> >> missing, simply include a fallback action: in this case the fallback is empty
> >> and does nothing.
> >>
> >>
> >>
> >> href="http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml"
> >> xmlns:xi="http://www.w3.org/2001/XInclude">
> >>
> >>
> >>
> >>
> >> -Bryan
> >>
> >>
> >>
> >>
> >> On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote:
> >>
> >>> I was looking at the same problem, and had a discussion with Noble. You can
> >>> use a hack to achieve what you want, see
> >>>
> >>> https://issues.apache.org/jira/browse/SOLR-1154
> >>>
> >>> Thanks,
> >>>
> >>> Jianhan
> >>>
> >>>
> >>> On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot wrote:
> >>>
> >>>> So how are people managing solrconfig.xml files which are largely the same
> >>>> other than differences for replication?
> >>>>
> >>>> I don't think it's a "good thing" to maintain two copies of the same file
> >>>> and I'd like to avoid that. Maybe enabling the XInclude feature in
> >>>> DocumentBuilders would make it possible to modularize configuration files
> to
> >>>> make this possible?
> >>>>
> >>>>
> >>>>
> >>
> http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)
> >>>>
> >>>>
> >>>> -Bryan
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote:
> >>>>
> >>>> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
> >>>>>> wrote:
> >>>>>
> >>>>> For replication in 1.4, the wiki at
> >>>>>> http://wiki.apache.org/solr/SolrReplication says that a node can be both
> >>>>>> the master and a slave:
> >>>>>>
> >>>>>> A node can act as both master and slave. In that case both the master and
> >>>>>> slave configuration lists need to be present inside the
> >>>>>> ReplicationHandler
> >>>>>> requestHandler in the solrconfig.xml.
> >>>>>>
> >>>>>> What does this mean? Does the core then poll itself for updates?
> >>>>>>
> >>>>>
> >>>>>
> >>>>> No. This type of configuration is meant for "repeaters". Suppose there are
> >>>>> slaves in multiple data-centers (say data center A and B). There is always
> >>>>> a
> >>>>> single master (say in A). One of the slaves in B is used as a master for
> >>>>> the
> >>>>> other slaves in B. Therefore, this one slave in B is both a master as well
> >>>>> as the slave.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> I'd like to have a single set of configuration files that are shared by
> >>>>>> masters and slaves and avoid duplicating configuration details in
> >>>>>> multiple
> >>>>>> files (one for master and one for slave) to ease management and failover.
> >>>>>> Is this possible?
> >>>>>>
> >>>>>>
> >>>>> You wouldn't want the master to be a slave. So I guess you'd need to have
> >>>>> a
> >>>>> separate file. Also, it needs to be a separate file so that the slave does
> >>>>> not become a master when the solrconfig.xml is replicated.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> When I attempt to setup a multi server master-slave configuration and
> >>>>>> include both master and slave replication configuration options, I into
> >>>>>> some
> >>>>>> problems. I'm running a nightly build from May 7.
> >>>>>>
> >>>>>>
> >>>>> Not sure what happened. Is that the url for this solr (meaning same solr
> >>>>> url
> >>>>> is master and slave of itself)? If yes, that is not a valid configuration.
> >>>>>
> >>>>> --
> >>>>> Regards,
> >>>>> Shalin Shekhar Mangar.
> >>>>>
> >>>>
> >>>>
> >
Re: Replication master+slave
Posted by Bryan Talbot <bt...@aeriagames.com>.
I think the patch I included earlier covers solr core, but it looks
like at least some other extensions (DIH) create and use their own XML
parser. So, if this functionality is to extend to all XML files,
those will need similar patches.
Here's one for DIH:
--- src/main/java/org/apache/solr/handler/dataimport/
DataImporter.java (revision 774137)
+++ src/main/java/org/apache/solr/handler/dataimport/
DataImporter.java (working copy)
@@ -148,8 +148,10 @@
void loadDataConfig(String configFile) {
try {
- DocumentBuilder builder = DocumentBuilderFactory.newInstance()
- .newDocumentBuilder();
+ DocumentBuilderFactory dbf =
DocumentBuilderFactory.newInstance();
+ dbf.setNamespaceAware(true);
+ dbf.setXIncludeAware(true);
+ DocumentBuilder builder = dbf.newDocumentBuilder();
Document document = builder.parse(new InputSource(new
StringReader(
configFile)));
The only down side I can see to this is it doesn't offer very
expressive conditional inclusion: the file is included if it's present
otherwise fallback inclusions can be used. It's also specific to XML
files and obviously won't work for other types of configuration
files. However, it is simple and effective.
-Bryan
On May 13, 2009, at May 13, 6:36 PM, Otis Gospodnetic wrote:
>
> Coincidentally, from http://www.cloudera.com/blog/2009/05/07/what%E2%80%99s-new-in-hadoop-core-020/
> :
>
> "Hadoop configuration files now support XInclude elements for
> including portions of another configuration file (HADOOP-4944). This
> mechanism allows you to make configuration files more modular and
> reusable."
>
> So "others are doing it, too".
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: Bryan Talbot <bt...@aeriagames.com>
>> To: solr-user@lucene.apache.org
>> Sent: Wednesday, May 13, 2009 11:26:41 AM
>> Subject: Re: Replication master+slave
>>
>> I see that Nobel's final comment in SOLR-1154 is that config files
>> need to be
>> able to include snippets from external files. In my limited
>> testing, a simple
>> patch to enable XInclude support seems to work.
>>
>>
>>
>> --- src/java/org/apache/solr/core/Config.java (revision 774137)
>> +++ src/java/org/apache/solr/core/Config.java (working copy)
>> @@ -100,8 +100,10 @@
>> if (lis == null) {
>> lis = loader.openConfig(name);
>> }
>> - javax.xml.parsers.DocumentBuilder builder =
>> DocumentBuilderFactory.newInstance().newDocumentBuilder();
>> - doc = builder.parse(lis);
>> + javax.xml.parsers.DocumentBuilderFactory dbf =
>> DocumentBuilderFactory.newInstance();
>> + dbf.setNamespaceAware(true);
>> + dbf.setXIncludeAware(true);
>> + doc = dbf.newDocumentBuilder().parse(lis);
>>
>> DOMUtil.substituteProperties(doc, loader.getCoreProperties());
>> } catch (ParserConfigurationException e) {
>>
>>
>>
>> This allows a clause like this to include the contents of
>> replication.xml if it
>> exists. If it's not found an exception will be thrown.
>>
>>
>>
>> href="http://localhost:8983/solr/corename/admin/file/?file=replication.xml
>> "
>> xmlns:xi="http://www.w3.org/2001/XInclude">
>>
>>
>>
>> If the file is optional and no exception should be thrown if the
>> file is
>> missing, simply include a fallback action: in this case the
>> fallback is empty
>> and does nothing.
>>
>>
>>
>> href="http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml
>> "
>> xmlns:xi="http://www.w3.org/2001/XInclude">
>>
>>
>>
>>
>> -Bryan
>>
>>
>>
>>
>> On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote:
>>
>>> I was looking at the same problem, and had a discussion with
>>> Noble. You can
>>> use a hack to achieve what you want, see
>>>
>>> https://issues.apache.org/jira/browse/SOLR-1154
>>>
>>> Thanks,
>>>
>>> Jianhan
>>>
>>>
>>> On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot wrote:
>>>
>>>> So how are people managing solrconfig.xml files which are largely
>>>> the same
>>>> other than differences for replication?
>>>>
>>>> I don't think it's a "good thing" to maintain two copies of the
>>>> same file
>>>> and I'd like to avoid that. Maybe enabling the XInclude feature in
>>>> DocumentBuilders would make it possible to modularize
>>>> configuration files to
>>>> make this possible?
>>>>
>>>>
>>>>
>> http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)
>>>>
>>>>
>>>> -Bryan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote:
>>>>
>>>> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
>>>>>> wrote:
>>>>>
>>>>> For replication in 1.4, the wiki at
>>>>>> http://wiki.apache.org/solr/SolrReplication says that a node
>>>>>> can be both
>>>>>> the master and a slave:
>>>>>>
>>>>>> A node can act as both master and slave. In that case both the
>>>>>> master and
>>>>>> slave configuration lists need to be present inside the
>>>>>> ReplicationHandler
>>>>>> requestHandler in the solrconfig.xml.
>>>>>>
>>>>>> What does this mean? Does the core then poll itself for updates?
>>>>>>
>>>>>
>>>>>
>>>>> No. This type of configuration is meant for "repeaters". Suppose
>>>>> there are
>>>>> slaves in multiple data-centers (say data center A and B). There
>>>>> is always
>>>>> a
>>>>> single master (say in A). One of the slaves in B is used as a
>>>>> master for
>>>>> the
>>>>> other slaves in B. Therefore, this one slave in B is both a
>>>>> master as well
>>>>> as the slave.
>>>>>
>>>>>
>>>>>
>>>>>> I'd like to have a single set of configuration files that are
>>>>>> shared by
>>>>>> masters and slaves and avoid duplicating configuration details in
>>>>>> multiple
>>>>>> files (one for master and one for slave) to ease management and
>>>>>> failover.
>>>>>> Is this possible?
>>>>>>
>>>>>>
>>>>> You wouldn't want the master to be a slave. So I guess you'd
>>>>> need to have
>>>>> a
>>>>> separate file. Also, it needs to be a separate file so that the
>>>>> slave does
>>>>> not become a master when the solrconfig.xml is replicated.
>>>>>
>>>>>
>>>>>
>>>>>> When I attempt to setup a multi server master-slave
>>>>>> configuration and
>>>>>> include both master and slave replication configuration
>>>>>> options, I into
>>>>>> some
>>>>>> problems. I'm running a nightly build from May 7.
>>>>>>
>>>>>>
>>>>> Not sure what happened. Is that the url for this solr (meaning
>>>>> same solr
>>>>> url
>>>>> is master and slave of itself)? If yes, that is not a valid
>>>>> configuration.
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Shalin Shekhar Mangar.
>>>>>
>>>>
>>>>
>
Re: Replication master+slave
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Coincidentally, from http://www.cloudera.com/blog/2009/05/07/what%E2%80%99s-new-in-hadoop-core-020/ :
"Hadoop configuration files now support XInclude elements for including portions of another configuration file (HADOOP-4944). This mechanism allows you to make configuration files more modular and reusable."
So "others are doing it, too".
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Bryan Talbot <bt...@aeriagames.com>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, May 13, 2009 11:26:41 AM
> Subject: Re: Replication master+slave
>
> I see that Nobel's final comment in SOLR-1154 is that config files need to be
> able to include snippets from external files. In my limited testing, a simple
> patch to enable XInclude support seems to work.
>
>
>
> --- src/java/org/apache/solr/core/Config.java (revision 774137)
> +++ src/java/org/apache/solr/core/Config.java (working copy)
> @@ -100,8 +100,10 @@
> if (lis == null) {
> lis = loader.openConfig(name);
> }
> - javax.xml.parsers.DocumentBuilder builder =
> DocumentBuilderFactory.newInstance().newDocumentBuilder();
> - doc = builder.parse(lis);
> + javax.xml.parsers.DocumentBuilderFactory dbf =
> DocumentBuilderFactory.newInstance();
> + dbf.setNamespaceAware(true);
> + dbf.setXIncludeAware(true);
> + doc = dbf.newDocumentBuilder().parse(lis);
>
> DOMUtil.substituteProperties(doc, loader.getCoreProperties());
> } catch (ParserConfigurationException e) {
>
>
>
> This allows a clause like this to include the contents of replication.xml if it
> exists. If it's not found an exception will be thrown.
>
>
>
> href="http://localhost:8983/solr/corename/admin/file/?file=replication.xml"
> xmlns:xi="http://www.w3.org/2001/XInclude">
>
>
>
> If the file is optional and no exception should be thrown if the file is
> missing, simply include a fallback action: in this case the fallback is empty
> and does nothing.
>
>
>
> href="http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml"
> xmlns:xi="http://www.w3.org/2001/XInclude">
>
>
>
>
> -Bryan
>
>
>
>
> On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote:
>
> > I was looking at the same problem, and had a discussion with Noble. You can
> > use a hack to achieve what you want, see
> >
> > https://issues.apache.org/jira/browse/SOLR-1154
> >
> > Thanks,
> >
> > Jianhan
> >
> >
> > On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot wrote:
> >
> >> So how are people managing solrconfig.xml files which are largely the same
> >> other than differences for replication?
> >>
> >> I don't think it's a "good thing" to maintain two copies of the same file
> >> and I'd like to avoid that. Maybe enabling the XInclude feature in
> >> DocumentBuilders would make it possible to modularize configuration files to
> >> make this possible?
> >>
> >>
> >>
> http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)
> >>
> >>
> >> -Bryan
> >>
> >>
> >>
> >>
> >>
> >> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote:
> >>
> >> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
> >>>> wrote:
> >>>
> >>> For replication in 1.4, the wiki at
> >>>> http://wiki.apache.org/solr/SolrReplication says that a node can be both
> >>>> the master and a slave:
> >>>>
> >>>> A node can act as both master and slave. In that case both the master and
> >>>> slave configuration lists need to be present inside the
> >>>> ReplicationHandler
> >>>> requestHandler in the solrconfig.xml.
> >>>>
> >>>> What does this mean? Does the core then poll itself for updates?
> >>>>
> >>>
> >>>
> >>> No. This type of configuration is meant for "repeaters". Suppose there are
> >>> slaves in multiple data-centers (say data center A and B). There is always
> >>> a
> >>> single master (say in A). One of the slaves in B is used as a master for
> >>> the
> >>> other slaves in B. Therefore, this one slave in B is both a master as well
> >>> as the slave.
> >>>
> >>>
> >>>
> >>>> I'd like to have a single set of configuration files that are shared by
> >>>> masters and slaves and avoid duplicating configuration details in
> >>>> multiple
> >>>> files (one for master and one for slave) to ease management and failover.
> >>>> Is this possible?
> >>>>
> >>>>
> >>> You wouldn't want the master to be a slave. So I guess you'd need to have
> >>> a
> >>> separate file. Also, it needs to be a separate file so that the slave does
> >>> not become a master when the solrconfig.xml is replicated.
> >>>
> >>>
> >>>
> >>>> When I attempt to setup a multi server master-slave configuration and
> >>>> include both master and slave replication configuration options, I into
> >>>> some
> >>>> problems. I'm running a nightly build from May 7.
> >>>>
> >>>>
> >>> Not sure what happened. Is that the url for this solr (meaning same solr
> >>> url
> >>> is master and slave of itself)? If yes, that is not a valid configuration.
> >>>
> >>> --
> >>> Regards,
> >>> Shalin Shekhar Mangar.
> >>>
> >>
> >>
Re: Replication master+slave
Posted by Bryan Talbot <bt...@aeriagames.com>.
I see that Nobel's final comment in SOLR-1154 is that config files
need to be able to include snippets from external files. In my
limited testing, a simple patch to enable XInclude support seems to
work.
--- src/java/org/apache/solr/core/Config.java (revision 774137)
+++ src/java/org/apache/solr/core/Config.java (working copy)
@@ -100,8 +100,10 @@
if (lis == null) {
lis = loader.openConfig(name);
}
- javax.xml.parsers.DocumentBuilder builder =
DocumentBuilderFactory.newInstance().newDocumentBuilder();
- doc = builder.parse(lis);
+ javax.xml.parsers.DocumentBuilderFactory dbf =
DocumentBuilderFactory.newInstance();
+ dbf.setNamespaceAware(true);
+ dbf.setXIncludeAware(true);
+ doc = dbf.newDocumentBuilder().parse(lis);
DOMUtil.substituteProperties(doc, loader.getCoreProperties());
} catch (ParserConfigurationException e) {
This allows a clause like this to include the contents of
replication.xml if it exists. If it's not found an exception will be
thrown.
<!-- include external file to define replication configuration -->
<xi:include href="http://localhost:8983/solr/corename/admin/file/?file=replication.xml
"
xmlns:xi="http://www.w3.org/2001/XInclude">
</xi:include>
If the file is optional and no exception should be thrown if the file
is missing, simply include a fallback action: in this case the
fallback is empty and does nothing.
<!-- include external file to define replication configuration -->
<xi:include href="http://localhost:8983/solr/forum_en/admin/file/?file=replication.xml
"
xmlns:xi="http://www.w3.org/2001/XInclude">
<xi:fallback/>
</xi:include>
-Bryan
On May 12, 2009, at May 12, 8:05 PM, Jian Han Guo wrote:
> I was looking at the same problem, and had a discussion with Noble.
> You can
> use a hack to achieve what you want, see
>
> https://issues.apache.org/jira/browse/SOLR-1154
>
> Thanks,
>
> Jianhan
>
>
> On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot
> <bt...@aeriagames.com>wrote:
>
>> So how are people managing solrconfig.xml files which are largely
>> the same
>> other than differences for replication?
>>
>> I don't think it's a "good thing" to maintain two copies of the
>> same file
>> and I'd like to avoid that. Maybe enabling the XInclude feature in
>> DocumentBuilders would make it possible to modularize configuration
>> files to
>> make this possible?
>>
>>
>> http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)
>> <http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware%28boolean%29
>> >
>>
>>
>> -Bryan
>>
>>
>>
>>
>>
>> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote:
>>
>> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
>> <btalbot@aeriagames.com
>>>> wrote:
>>>
>>> For replication in 1.4, the wiki at
>>>> http://wiki.apache.org/solr/SolrReplication says that a node can
>>>> be both
>>>> the master and a slave:
>>>>
>>>> A node can act as both master and slave. In that case both the
>>>> master and
>>>> slave configuration lists need to be present inside the
>>>> ReplicationHandler
>>>> requestHandler in the solrconfig.xml.
>>>>
>>>> What does this mean? Does the core then poll itself for updates?
>>>>
>>>
>>>
>>> No. This type of configuration is meant for "repeaters". Suppose
>>> there are
>>> slaves in multiple data-centers (say data center A and B). There
>>> is always
>>> a
>>> single master (say in A). One of the slaves in B is used as a
>>> master for
>>> the
>>> other slaves in B. Therefore, this one slave in B is both a master
>>> as well
>>> as the slave.
>>>
>>>
>>>
>>>> I'd like to have a single set of configuration files that are
>>>> shared by
>>>> masters and slaves and avoid duplicating configuration details in
>>>> multiple
>>>> files (one for master and one for slave) to ease management and
>>>> failover.
>>>> Is this possible?
>>>>
>>>>
>>> You wouldn't want the master to be a slave. So I guess you'd need
>>> to have
>>> a
>>> separate file. Also, it needs to be a separate file so that the
>>> slave does
>>> not become a master when the solrconfig.xml is replicated.
>>>
>>>
>>>
>>>> When I attempt to setup a multi server master-slave configuration
>>>> and
>>>> include both master and slave replication configuration options,
>>>> I into
>>>> some
>>>> problems. I'm running a nightly build from May 7.
>>>>
>>>>
>>> Not sure what happened. Is that the url for this solr (meaning
>>> same solr
>>> url
>>> is master and slave of itself)? If yes, that is not a valid
>>> configuration.
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>>
Re: Replication master+slave
Posted by Jian Han Guo <ji...@gmail.com>.
I was looking at the same problem, and had a discussion with Noble. You can
use a hack to achieve what you want, see
https://issues.apache.org/jira/browse/SOLR-1154
Thanks,
Jianhan
On Tue, May 12, 2009 at 5:13 PM, Bryan Talbot <bt...@aeriagames.com>wrote:
> So how are people managing solrconfig.xml files which are largely the same
> other than differences for replication?
>
> I don't think it's a "good thing" to maintain two copies of the same file
> and I'd like to avoid that. Maybe enabling the XInclude feature in
> DocumentBuilders would make it possible to modularize configuration files to
> make this possible?
>
>
> http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)<http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware%28boolean%29>
>
>
> -Bryan
>
>
>
>
>
> On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote:
>
> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot <btalbot@aeriagames.com
>> >wrote:
>>
>> For replication in 1.4, the wiki at
>>> http://wiki.apache.org/solr/SolrReplication says that a node can be both
>>> the master and a slave:
>>>
>>> A node can act as both master and slave. In that case both the master and
>>> slave configuration lists need to be present inside the
>>> ReplicationHandler
>>> requestHandler in the solrconfig.xml.
>>>
>>> What does this mean? Does the core then poll itself for updates?
>>>
>>
>>
>> No. This type of configuration is meant for "repeaters". Suppose there are
>> slaves in multiple data-centers (say data center A and B). There is always
>> a
>> single master (say in A). One of the slaves in B is used as a master for
>> the
>> other slaves in B. Therefore, this one slave in B is both a master as well
>> as the slave.
>>
>>
>>
>>> I'd like to have a single set of configuration files that are shared by
>>> masters and slaves and avoid duplicating configuration details in
>>> multiple
>>> files (one for master and one for slave) to ease management and failover.
>>> Is this possible?
>>>
>>>
>> You wouldn't want the master to be a slave. So I guess you'd need to have
>> a
>> separate file. Also, it needs to be a separate file so that the slave does
>> not become a master when the solrconfig.xml is replicated.
>>
>>
>>
>>> When I attempt to setup a multi server master-slave configuration and
>>> include both master and slave replication configuration options, I into
>>> some
>>> problems. I'm running a nightly build from May 7.
>>>
>>>
>> Not sure what happened. Is that the url for this solr (meaning same solr
>> url
>> is master and slave of itself)? If yes, that is not a valid configuration.
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
>
Re: Replication master+slave
Posted by Bryan Talbot <bt...@aeriagames.com>.
So how are people managing solrconfig.xml files which are largely the
same other than differences for replication?
I don't think it's a "good thing" to maintain two copies of the same
file and I'd like to avoid that. Maybe enabling the XInclude feature
in DocumentBuilders would make it possible to modularize configuration
files to make this possible?
http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/DocumentBuilderFactory.html#setXIncludeAware(boolean)
-Bryan
On May 12, 2009, at May 12, 11:43 AM, Shalin Shekhar Mangar wrote:
> On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot
> <bt...@aeriagames.com>wrote:
>
>> For replication in 1.4, the wiki at
>> http://wiki.apache.org/solr/SolrReplication says that a node can be
>> both
>> the master and a slave:
>>
>> A node can act as both master and slave. In that case both the
>> master and
>> slave configuration lists need to be present inside the
>> ReplicationHandler
>> requestHandler in the solrconfig.xml.
>>
>> What does this mean? Does the core then poll itself for updates?
>
>
> No. This type of configuration is meant for "repeaters". Suppose
> there are
> slaves in multiple data-centers (say data center A and B). There is
> always a
> single master (say in A). One of the slaves in B is used as a master
> for the
> other slaves in B. Therefore, this one slave in B is both a master
> as well
> as the slave.
>
>
>>
>> I'd like to have a single set of configuration files that are
>> shared by
>> masters and slaves and avoid duplicating configuration details in
>> multiple
>> files (one for master and one for slave) to ease management and
>> failover.
>> Is this possible?
>>
>
> You wouldn't want the master to be a slave. So I guess you'd need to
> have a
> separate file. Also, it needs to be a separate file so that the
> slave does
> not become a master when the solrconfig.xml is replicated.
>
>
>>
>> When I attempt to setup a multi server master-slave configuration and
>> include both master and slave replication configuration options, I
>> into some
>> problems. I'm running a nightly build from May 7.
>>
>
> Not sure what happened. Is that the url for this solr (meaning same
> solr url
> is master and slave of itself)? If yes, that is not a valid
> configuration.
>
> --
> Regards,
> Shalin Shekhar Mangar.
Re: Replication master+slave
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, May 12, 2009 at 10:42 PM, Bryan Talbot <bt...@aeriagames.com>wrote:
> For replication in 1.4, the wiki at
> http://wiki.apache.org/solr/SolrReplication says that a node can be both
> the master and a slave:
>
> A node can act as both master and slave. In that case both the master and
> slave configuration lists need to be present inside the ReplicationHandler
> requestHandler in the solrconfig.xml.
>
> What does this mean? Does the core then poll itself for updates?
No. This type of configuration is meant for "repeaters". Suppose there are
slaves in multiple data-centers (say data center A and B). There is always a
single master (say in A). One of the slaves in B is used as a master for the
other slaves in B. Therefore, this one slave in B is both a master as well
as the slave.
>
> I'd like to have a single set of configuration files that are shared by
> masters and slaves and avoid duplicating configuration details in multiple
> files (one for master and one for slave) to ease management and failover.
> Is this possible?
>
You wouldn't want the master to be a slave. So I guess you'd need to have a
separate file. Also, it needs to be a separate file so that the slave does
not become a master when the solrconfig.xml is replicated.
>
> When I attempt to setup a multi server master-slave configuration and
> include both master and slave replication configuration options, I into some
> problems. I'm running a nightly build from May 7.
>
Not sure what happened. Is that the url for this solr (meaning same solr url
is master and slave of itself)? If yes, that is not a valid configuration.
--
Regards,
Shalin Shekhar Mangar.