You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by James Brady <ja...@gmail.com> on 2008/04/29 21:09:49 UTC
Master / slave setup with multicore
Hi all,
I'm aiming to use the new multicore features in development versions
of Solr. My ideal setup would be to have master / slave servers on the
same machine, snapshotting across from the 'write' to the 'read'
server at intervals.
This was all fine with Solr 1.2, but the rsync & snappuller
configuration doesn't seem to be set up to allow for multicore
replication in 1.3.
The rsyncd.conf file allows for several data directories to be
defined, but the snappuller script only handles a single directory,
expecting the Lucene index to be directly inside that directory.
What's the best practice / best suggestions for replicating a
multicore update server out to search servers?
Thanks,
James
Re: Distributed Search (shard) w/ Multicore?
Posted by Yonik Seeley <yo...@apache.org>.
Try adding echoParams=all to the request.
Maybe there is a default rows=0 or something.
Are you using a recent version of Solr?
-Yonik
On Fri, May 2, 2008 at 4:30 PM, Jon Baer <jo...@gmail.com> wrote:
> Sorry about that, Im sending something simple like:
>
>http://search.company.com:8115/solr/search/players?q=Smith&shards=box1:8115/search/players,box2:8115/search/players
>
> Im getting back:
>
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">18</int>
> </lst>
> <result name="response" numFound="13" start="0" />
> </response>
>
> Identical schemas, it found the correct 13 but no docs attached. In the
> logs I can see the results come back (w/ wt=javabin&isShared=true) ...
>
> - Jon
>
>
>
> On May 2, 2008, at 3:41 PM, Yonik Seeley wrote:
>
>
> > On Fri, May 2, 2008 at 3:36 PM, Jon Baer <jo...@gmail.com> wrote:
> >
> > > Im trying to figure out if I can do this or if something else needs to
> be
> > > set, trying to run a query over multiple cores w/ the shard param? I
> seem
> > > to be getting the correct number of results back but no data ... any
> ideas?
> > >
> >
> > Should work OK (note that schemas should match across cores...
> > distributed search is not federated search).
> > You might need to be a little more explicit about what you are sending
> > and what you are getting back (the actual URL of the request, and the
> > actual XML of the response).
> >
> > -Yonik
Re: Distributed Search (shard) w/ Multicore?
Posted by Jon Baer <jo...@gmail.com>.
Sorry about that, Im sending something simple like:
http://search.company.com:8115/solr/search/players?q=Smith&shards=box1:8115/search/players,box2:8115/search/players
Im getting back:
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">18</int>
</lst>
<result name="response" numFound="13" start="0" />
</response>
Identical schemas, it found the correct 13 but no docs attached. In
the logs I can see the results come back (w/
wt=javabin&isShared=true) ...
- Jon
On May 2, 2008, at 3:41 PM, Yonik Seeley wrote:
> On Fri, May 2, 2008 at 3:36 PM, Jon Baer <jo...@gmail.com> wrote:
>> Im trying to figure out if I can do this or if something else needs
>> to be
>> set, trying to run a query over multiple cores w/ the shard param?
>> I seem
>> to be getting the correct number of results back but no data ...
>> any ideas?
>
> Should work OK (note that schemas should match across cores...
> distributed search is not federated search).
> You might need to be a little more explicit about what you are sending
> and what you are getting back (the actual URL of the request, and the
> actual XML of the response).
>
> -Yonik
Re: Distributed Search (shard) w/ Multicore?
Posted by Yonik Seeley <yo...@apache.org>.
On Fri, May 2, 2008 at 3:36 PM, Jon Baer <jo...@gmail.com> wrote:
> Im trying to figure out if I can do this or if something else needs to be
> set, trying to run a query over multiple cores w/ the shard param? I seem
> to be getting the correct number of results back but no data ... any ideas?
Should work OK (note that schemas should match across cores...
distributed search is not federated search).
You might need to be a little more explicit about what you are sending
and what you are getting back (the actual URL of the request, and the
actual XML of the response).
-Yonik
Distributed Search (shard) w/ Multicore?
Posted by Jon Baer <jo...@gmail.com>.
Hi,
Im trying to figure out if I can do this or if something else needs to
be set, trying to run a query over multiple cores w/ the shard param?
I seem to be getting the correct number of results back but no
data ... any ideas?
Thanks.
- Jon
Re: Master / slave setup with multicore
Posted by James Brady <ja...@gmail.com>.
Ah, wait, my fault - I didn't have the right Solr port configured in
the slave, so snapinstaller was commiting the master :/
Thanks,
James
On 2 May 2008, at 09:17, Bill Au wrote:
> snapinstall calls commit to trigger Solr to use the new index. Do
> you see
> the commit request in your Solr log? Anything in the snapinstaller
> log?
>
> Bill
>
> On Thu, May 1, 2008 at 8:35 PM, James Brady <james.colin.brady@gmail.com
> >
> wrote:
>
>> Hi Ryan, thanks for that!
>>
>> I have one outstanding question: when I take a snapshot on the
>> master,
>> snappull and snapinstall on the slave, the new index is not being
>> used:
>> restarting the slave server does pick up the changes, however.
>>
>> Has anyone else had this problem with recent development builds?
>>
>> In case anyone is trying to do multicore replication, here some of
>> the
>> things I've done to get it working.. These could go on the wiki
>> somewhere,
>> what do people think?
>>
>> Obviously, have as much shared configuration as possible is ideal.
>> On the
>> master, I have core-specific:
>> - scripts.conf, for webapp_name, master_data_dir and
>> master_status_dir
>> - solrconfig.xml, for the post-commit and post-optimise snapshooter
>> locations
>>
>> On the slave, I have core-specific:
>> -scripts.conf, as above
>>
>> I've also customised snappuller to accept a different rsync module
>> name
>> (hard coded to 'solr' at present). This module name is set in the
>> slave
>> scripts.conf
>>
>> James
>>
>>
>> On 29 Apr 2008, at 13:44, Ryan McKinley wrote:
>>
>>
>>> On Apr 29, 2008, at 3:09 PM, James Brady wrote:
>>>
>>>> Hi all,
>>>> I'm aiming to use the new multicore features in development
>>>> versions
>>>> of Solr. My ideal setup would be to have master / slave servers
>>>> on the same
>>>> machine, snapshotting across from the 'write' to the 'read'
>>>> server at
>>>> intervals.
>>>>
>>>> This was all fine with Solr 1.2, but the rsync & snappuller
>>>> configuration doesn't seem to be set up to allow for multicore
>>>> replication
>>>> in 1.3.
>>>>
>>>> The rsyncd.conf file allows for several data directories to be
>>>> defined, but the snappuller script only handles a single directory,
>>>> expecting the Lucene index to be directly inside that directory.
>>>>
>>>> What's the best practice / best suggestions for replicating a
>>>> multicore update server out to search servers?
>>>>
>>>>
>>> Currently, for multicore replication you will need to install the
>>> snap*
>>> scripts for _each_ core. The scripts all expect a single core so
>>> for
>>> multiple cores, you will just need to install it multiple times.
>>>
>>> ryan
>>>
>>
>>
Re: Master / slave setup with multicore
Posted by Bill Au <bi...@gmail.com>.
snapinstall calls commit to trigger Solr to use the new index. Do you see
the commit request in your Solr log? Anything in the snapinstaller log?
Bill
On Thu, May 1, 2008 at 8:35 PM, James Brady <ja...@gmail.com>
wrote:
> Hi Ryan, thanks for that!
>
> I have one outstanding question: when I take a snapshot on the master,
> snappull and snapinstall on the slave, the new index is not being used:
> restarting the slave server does pick up the changes, however.
>
> Has anyone else had this problem with recent development builds?
>
> In case anyone is trying to do multicore replication, here some of the
> things I've done to get it working.. These could go on the wiki somewhere,
> what do people think?
>
> Obviously, have as much shared configuration as possible is ideal. On the
> master, I have core-specific:
> - scripts.conf, for webapp_name, master_data_dir and master_status_dir
> - solrconfig.xml, for the post-commit and post-optimise snapshooter
> locations
>
> On the slave, I have core-specific:
> -scripts.conf, as above
>
> I've also customised snappuller to accept a different rsync module name
> (hard coded to 'solr' at present). This module name is set in the slave
> scripts.conf
>
> James
>
>
> On 29 Apr 2008, at 13:44, Ryan McKinley wrote:
>
>
> > On Apr 29, 2008, at 3:09 PM, James Brady wrote:
> >
> > > Hi all,
> > > I'm aiming to use the new multicore features in development versions
> > > of Solr. My ideal setup would be to have master / slave servers on the same
> > > machine, snapshotting across from the 'write' to the 'read' server at
> > > intervals.
> > >
> > > This was all fine with Solr 1.2, but the rsync & snappuller
> > > configuration doesn't seem to be set up to allow for multicore replication
> > > in 1.3.
> > >
> > > The rsyncd.conf file allows for several data directories to be
> > > defined, but the snappuller script only handles a single directory,
> > > expecting the Lucene index to be directly inside that directory.
> > >
> > > What's the best practice / best suggestions for replicating a
> > > multicore update server out to search servers?
> > >
> > >
> > Currently, for multicore replication you will need to install the snap*
> > scripts for _each_ core. The scripts all expect a single core so for
> > multiple cores, you will just need to install it multiple times.
> >
> > ryan
> >
>
>
Re: Master / slave setup with multicore
Posted by James Brady <ja...@gmail.com>.
Hi Ryan, thanks for that!
I have one outstanding question: when I take a snapshot on the master,
snappull and snapinstall on the slave, the new index is not being
used: restarting the slave server does pick up the changes, however.
Has anyone else had this problem with recent development builds?
In case anyone is trying to do multicore replication, here some of the
things I've done to get it working.. These could go on the wiki
somewhere, what do people think?
Obviously, have as much shared configuration as possible is ideal. On
the master, I have core-specific:
- scripts.conf, for webapp_name, master_data_dir and master_status_dir
- solrconfig.xml, for the post-commit and post-optimise snapshooter
locations
On the slave, I have core-specific:
-scripts.conf, as above
I've also customised snappuller to accept a different rsync module
name (hard coded to 'solr' at present). This module name is set in the
slave scripts.conf
James
On 29 Apr 2008, at 13:44, Ryan McKinley wrote:
>
> On Apr 29, 2008, at 3:09 PM, James Brady wrote:
>> Hi all,
>> I'm aiming to use the new multicore features in development
>> versions of Solr. My ideal setup would be to have master / slave
>> servers on the same machine, snapshotting across from the 'write'
>> to the 'read' server at intervals.
>>
>> This was all fine with Solr 1.2, but the rsync & snappuller
>> configuration doesn't seem to be set up to allow for multicore
>> replication in 1.3.
>>
>> The rsyncd.conf file allows for several data directories to be
>> defined, but the snappuller script only handles a single directory,
>> expecting the Lucene index to be directly inside that directory.
>>
>> What's the best practice / best suggestions for replicating a
>> multicore update server out to search servers?
>>
>
> Currently, for multicore replication you will need to install the
> snap* scripts for _each_ core. The scripts all expect a single core
> so for multiple cores, you will just need to install it multiple
> times.
>
> ryan
Re: Master / slave setup with multicore
Posted by Ryan McKinley <ry...@gmail.com>.
On Apr 29, 2008, at 3:09 PM, James Brady wrote:
> Hi all,
> I'm aiming to use the new multicore features in development versions
> of Solr. My ideal setup would be to have master / slave servers on
> the same machine, snapshotting across from the 'write' to the 'read'
> server at intervals.
>
> This was all fine with Solr 1.2, but the rsync & snappuller
> configuration doesn't seem to be set up to allow for multicore
> replication in 1.3.
>
> The rsyncd.conf file allows for several data directories to be
> defined, but the snappuller script only handles a single directory,
> expecting the Lucene index to be directly inside that directory.
>
> What's the best practice / best suggestions for replicating a
> multicore update server out to search servers?
>
Currently, for multicore replication you will need to install the
snap* scripts for _each_ core. The scripts all expect a single core
so for multiple cores, you will just need to install it multiple times.
ryan