You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by bsargurunathan <bs...@gmail.com> on 2012/08/28 11:54:38 UTC
Solr Master/Slave setup issue
Hi Guys,
I have a problem with Solr Master/Slave setup. Please clarify me.
I added the following things in my Master solrconfig.xml
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<lst name="master">
<str name="replicateAfter">commit</str>
<str name="replicateAfter">startup</str>
<str name="confFiles">schema.xml,stopwords.txt</str>
</lst>
</requestHandler>
And I added following things in Slave solrconfig.xml,
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<lst name="slave">
<str
name="masterUrl">http://172.16.17.64:8080/ndxservice/registry/replication/</str>
<str name="pollInterval">00:00:60</str>
</lst>
</requestHandler>
So, my problem is, I want to avoid the hot coded masterUrl.
It should be coming by dynamically, how it is possible?
Because I am testing with different environment (i.e)BETA,UAT,PROD so I dont
want to change that setting manually for every war.
Thanks,
Guru
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Master-Slave-setup-issue-tp4003667.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Master/Slave setup issue
Posted by Chris Hostetter <ho...@fucit.org>.
: <str name="enable">${enable.slave:false}</str>
: <str
: name="masterUrl">http://localhost:8080/ndxservice/registry/replication</str>
...
: In above config, I did the enable/disable of solr/master using
: solrcore.properties file,
: but I want set up the masterUrl from java code or my app setting.xml. How it
: is really possible?
exactly the same way, something like...
<str name="masterUrl">${master.replication.url:http://default-url:8080/solr/replication}</str>
...and then specify master.replication.url at run time the same way you
specify enable.slave.
-Hoss
Re: Solr Master/Slave setup issue
Posted by Ahmet Arslan <io...@yahoo.com>.
--- On Tue, 8/28/12, bsargurunathan <bs...@gmail.com> wrote:
> From: bsargurunathan <bs...@gmail.com>
> Subject: Re: Solr Master/Slave setup issue
> To: solr-user@lucene.apache.org
> Date: Tuesday, August 28, 2012, 8:54 PM
> Hi iorixxx,
> Thanks for the updates, its really helpful for me. But I
> want one more
> clarification,
>
> I am setting up the solrcofig.xml like as follows,
> <requestHandler name="/replication"
> class="solr.ReplicationHandler" >
> <lst
> name="master">
> <str
> name="enable">${enable.master:false}</str>
> <str
> name="replicateAfter">commit</str>
> <str
> name="confFiles">schema.xml,stopwords.txt</str>
> </lst>
> <lst
> name="slave">
> <str
> name="enable">${enable.slave:false}</str>
>
> <str
> name="masterUrl">http://localhost:8080/ndxservice/registry/replication</str>
>
> <str
> name="pollInterval">00:00:5</str>
> </lst>
> </requestHandler>
> In above config, I did the enable/disable of solr/master
> using
> solrcore.properties file,
> but I want set up the masterUrl from java code or my app
> setting.xml. How it
> is really possible?
It is possible for an one time replication.
"Force a fetchindex on slave from master command: http://slave_host:port/solr/replication?command=fetchindex
It is possible to pass on extra attribute 'masterUrl' or other attributes like 'compression' (or any other parameter which is specified in the <lst name="slave"> tag) to do a one time replication from a master. This obviates the need for hardcoding the master in the slave. "
In past I implemented a request handler that writes/updates solrcore.properties file. That can be an option too. (Where $MASTER_CORE_URL) is defined in a solrcore.properties)
Re: Solr Master/Slave setup issue
Posted by bsargurunathan <bs...@gmail.com>.
Hi iorixxx,
Thanks for the updates, its really helpful for me. But I want one more
clarification,
I am setting up the solrcofig.xml like as follows,
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<lst name="master">
<str name="enable">${enable.master:false}</str>
<str name="replicateAfter">commit</str>
<str name="confFiles">schema.xml,stopwords.txt</str>
</lst>
<lst name="slave">
<str name="enable">${enable.slave:false}</str>
<str
name="masterUrl">http://localhost:8080/ndxservice/registry/replication</str>
<str name="pollInterval">00:00:5</str>
</lst>
</requestHandler>
In above config, I did the enable/disable of solr/master using
solrcore.properties file,
but I want set up the masterUrl from java code or my app setting.xml. How it
is really possible?
Thanks,
Guru
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Master-Slave-setup-issue-tp4003667p4003755.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Master/Slave setup issue
Posted by Ahmet Arslan <io...@yahoo.com>.
> Can you please share some samples for
> that?
> I cant see anywhere that solrcore.properties file in my
> solr-3.5 package.
> Can you please explain somewhat depth?
You need to create it by yourself. Create a solrcore.properties file under conf directory.
This example talks about using same solrconfig.xml file for both slave and master.
http://wiki.apache.org/solr/SolrReplication#enable.2BAC8-disable_master.2BAC8-slave_in_a_node
Re: Solr Master/Slave setup issue
Posted by bsargurunathan <bs...@gmail.com>.
Can you please share some samples for that?
I cant see anywhere that solrcore.properties file in my solr-3.5 package.
Can you please explain somewhat depth?
Thanks,
Guru
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Master-Slave-setup-issue-tp4003667p4003675.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Shard Replicas sharing files
Posted by Christian von Wendt-Jensen <Ch...@infopaq.com>.
I was playing with the thought because want to find a way to easily rebalance hardware resources. And it takes a lot of time to move replicas around because the shards we are serving are many (25 and counting) and large (50GB). If all replicas could share a single fileset, it would be so easy to set up new replicas and move them around.
We are using Solr 3.6, and are of course aware of the activity on SolrCloud and ElasticSearch. My problem is that we are not likely to migrate within the first year, and I need a solution for easier administration.
The files could be placed on a SAN with very high bandwidth. I understand that a SAN in practice can compete with local harddrives when it comes to bandwidth. So in the assumption that I/O is not a bottleneck, what other problems would arise?
I can think of some:
* The replica uses a file lock effectively preventing other instances to access the index. This would be a showstopper in it self, unless there is a workaround.
* The replicas filelist is in memory, and it is therefore not possible to have instant synchronization, as the replica think it should copy the delta from the master.
Med venlig hilsen / Best Regards
Christian von Wendt-Jensen
IT Team Lead, Customer Solutions
Infopaq International A/S
Kgs. Nytorv 22
DK-1050 København K
Phone +45 36 99 00 00
Mobile +45 31 17 10 07
Email christian.sonne.jensen@infopaq.com<ma...@infopaq.com>
Web www.infopaq.com<http://www.infopaq.com/>
DISCLAIMER:
This e-mail and accompanying documents contain privileged confidential information. The information is intended only for the recipient(s) named. Any unauthorised disclosure, copying, distribution, exploitation or the taking of any action in reliance of the content of this e-mail is strictly prohibited. If you have received this e-mail in error we would be obliged if you would delete the e-mail and attachments and notify the dispatcher by return e-mail or at +45 36 99 00 00
P Please consider the environment before printing this mail note.
From: Erick Erickson <er...@gmail.com>>
Reply-To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Date: Thu, 30 Aug 2012 03:33:09 +0200
To: "solr-user@lucene.apache.org<ma...@lucene.apache.org>" <so...@lucene.apache.org>>
Subject: Re: Solr Shard Replicas sharing files
Possible, kinda maybe. But then all of the SolrCloud goodness that's
there for HA/DR goes out the window because the shared index (actually
the hardware it's on) becomes a single point of failure. On the other
hand, you're using the word replica but not explicitly talking about
SolrCloud, so I guess this is just about standard master/slave
situations...
Where the answer is that it's generally not a great idea to share
indexes like this. The disk I/O becomes your bottleneck with all those
slaves asking to pull what then need off the disk at once, every time
it is committed to compounded with network latency.
But I have to ask, is this just a theoretical question or is it really
something you're having trouble with in production?
And the idea of a "replication tree", where N slaves get their index
from the master, then M slaves get their index from the first N slaves
sounds like a "repeater" setup, see:
http://wiki.apache.org/solr/SolrReplication#Setting_up_a_Repeater
Best
Erick
On Wed, Aug 29, 2012 at 4:23 AM, Christian von Wendt-Jensen
<Ch...@infopaq.com>> wrote:
Hi,
I was wondering if it was possible to let all replicas of a shard share the physical lucene files. In that way you would only need one set of files on a shared storage, and then setup as many replicas as needed without copying files around. This would make it very fast to optimize and rebalance hardware resources as more shards are added.
What I visioning was a setup with one master doing all the indexing. Then all the shard replicas are installed as a string of slaves setup as both master and slave, such that the first replica replicates directly from the master. The next replica replicates from the first replica and so on.
In this way only the first replica need to write indexfiles. When the next replica is triggered to replicate it will find that all files are up to date, and then you issue a "commit" to reload the index in memory, thereby being up-to-date. The master's commit triggers a cascade of replication, which are all up-to-date immediately, and then it is a matter of few seconds for the slaves to be in sync with the master.
Taking this though further, the first replica could actually access the master's index files directly, and then be up-to-date without copying any files.
Would this setup be possible?
Med venlig hilsen / Best Regards
Christian von Wendt-Jensen
IT Team Lead, Customer Solutions
Infopaq International A/S
Kgs. Nytorv 22
DK-1050 København K
Phone +45 36 99 00 00
Mobile +45 31 17 10 07
Email christian.sonne.jensen@infopaq.com<ma...@infopaq.com>
Web www.infopaq.com<http://www.infopaq.com/>
Re: Solr Shard Replicas sharing files
Posted by Erick Erickson <er...@gmail.com>.
Possible, kinda maybe. But then all of the SolrCloud goodness that's
there for HA/DR goes out the window because the shared index (actually
the hardware it's on) becomes a single point of failure. On the other
hand, you're using the word replica but not explicitly talking about
SolrCloud, so I guess this is just about standard master/slave
situations...
Where the answer is that it's generally not a great idea to share
indexes like this. The disk I/O becomes your bottleneck with all those
slaves asking to pull what then need off the disk at once, every time
it is committed to compounded with network latency.
But I have to ask, is this just a theoretical question or is it really
something you're having trouble with in production?
And the idea of a "replication tree", where N slaves get their index
from the master, then M slaves get their index from the first N slaves
sounds like a "repeater" setup, see:
http://wiki.apache.org/solr/SolrReplication#Setting_up_a_Repeater
Best
Erick
On Wed, Aug 29, 2012 at 4:23 AM, Christian von Wendt-Jensen
<Ch...@infopaq.com> wrote:
> Hi,
>
> I was wondering if it was possible to let all replicas of a shard share the physical lucene files. In that way you would only need one set of files on a shared storage, and then setup as many replicas as needed without copying files around. This would make it very fast to optimize and rebalance hardware resources as more shards are added.
>
> What I visioning was a setup with one master doing all the indexing. Then all the shard replicas are installed as a string of slaves setup as both master and slave, such that the first replica replicates directly from the master. The next replica replicates from the first replica and so on.
>
> In this way only the first replica need to write indexfiles. When the next replica is triggered to replicate it will find that all files are up to date, and then you issue a "commit" to reload the index in memory, thereby being up-to-date. The master's commit triggers a cascade of replication, which are all up-to-date immediately, and then it is a matter of few seconds for the slaves to be in sync with the master.
>
> Taking this though further, the first replica could actually access the master's index files directly, and then be up-to-date without copying any files.
>
> Would this setup be possible?
>
>
>
> Med venlig hilsen / Best Regards
>
> Christian von Wendt-Jensen
> IT Team Lead, Customer Solutions
>
> Infopaq International A/S
> Kgs. Nytorv 22
> DK-1050 København K
>
> Phone +45 36 99 00 00
> Mobile +45 31 17 10 07
> Email christian.sonne.jensen@infopaq.com<ma...@infopaq.com>
> Web www.infopaq.com<http://www.infopaq.com/>
>
Solr Shard Replicas sharing files
Posted by Christian von Wendt-Jensen <Ch...@infopaq.com>.
Hi,
I was wondering if it was possible to let all replicas of a shard share the physical lucene files. In that way you would only need one set of files on a shared storage, and then setup as many replicas as needed without copying files around. This would make it very fast to optimize and rebalance hardware resources as more shards are added.
What I visioning was a setup with one master doing all the indexing. Then all the shard replicas are installed as a string of slaves setup as both master and slave, such that the first replica replicates directly from the master. The next replica replicates from the first replica and so on.
In this way only the first replica need to write indexfiles. When the next replica is triggered to replicate it will find that all files are up to date, and then you issue a "commit" to reload the index in memory, thereby being up-to-date. The master's commit triggers a cascade of replication, which are all up-to-date immediately, and then it is a matter of few seconds for the slaves to be in sync with the master.
Taking this though further, the first replica could actually access the master's index files directly, and then be up-to-date without copying any files.
Would this setup be possible?
Med venlig hilsen / Best Regards
Christian von Wendt-Jensen
IT Team Lead, Customer Solutions
Infopaq International A/S
Kgs. Nytorv 22
DK-1050 København K
Phone +45 36 99 00 00
Mobile +45 31 17 10 07
Email christian.sonne.jensen@infopaq.com<ma...@infopaq.com>
Web www.infopaq.com<http://www.infopaq.com/>
Re: Solr Master/Slave setup issue
Posted by Ahmet Arslan <io...@yahoo.com>.
> And I added following things in Slave solrconfig.xml,
>
> <requestHandler name="/replication"
> class="solr.ReplicationHandler" >
> <lst name="slave">
> <str
> name="masterUrl">http://172.16.17.64:8080/ndxservice/registry/replication/</str>
> <str
> name="pollInterval">00:00:60</str>
> </lst>
> </requestHandler>
>
> So, my problem is, I want to avoid the hot coded masterUrl.
> It should be coming by dynamically, how it is possible?
> Because I am testing with different environment
> (i.e)BETA,UAT,PROD so I dont
> want to change that setting manually for every war.
May be you can use solrcore.properties file for this.
http://search-lucene.com/m/vUmKw17NnUj