You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2010/04/28 00:00:21 UTC
Replicate cores from master to slave
If I create a new core on a Solr master, is there a way to instruct a
Solr slave to replicate the new core?
Re: Replicate cores from master to slave
Posted by Jason Rutherglen <ja...@gmail.com>.
Actually, it simply a matter of adding a core via core admin handler.
It's not hard to accomplish, though, I guess it's not a part of Solr
today to perform this task.
On Wed, Apr 28, 2010 at 8:50 AM, Jon Baer <jo...@gmail.com> wrote:
> Correct me if Im wrong but I think the problem here is that while there is a "fetchindex" command in replication the handler and the master/slave setup pertain to the core config.
>
> For example for this to work properly the solr.xml configuration would need to setup some type of "global" replication handler for all cores.
>
> Just throwing that out there since Im not sure even the ZooKeeper setup would include something like this.
>
> - Jon
>
> On Apr 28, 2010, at 10:14 AM, Jason Rutherglen wrote:
>
>> I guess I didn't explain it properly. I want to create a core on
>> the master, and then have N slaves also (aka replicate) create
>> those new core(s) on the slave servers, then of course, begin to
>> replicate (yeah, got that part). There doesn't appear to be
>> anything today that does this, it's unclear how/if Solr Cloud
>> enables the described functionality because, while yes Solr
>> Cloud can be used to *create* new cores out of a config in ZK,
>> that isn't what I'm asking. I'll open an issue and submit a
>> patch?
>>
>> On Wed, Apr 28, 2010 at 1:53 AM, Yonik Seeley
>> <yo...@lucidimagination.com> wrote:
>>> On Tue, Apr 27, 2010 at 9:56 PM, Chris Hostetter
>>> <ho...@fucit.org> wrote:
>>>> but as i understand the new cloud stuff (by which i mean: i don't
>>>> understand the new cloud stuff, but i've heard rumors) this will be
>>>> possible with that functionality.
>>>
>>> Yeah, that should be the goal.
>>> The solrconfig.xml, etc, are stored in zookeeper, so creating a new
>>> core is (should be) trivial.
>>> There's not anything in cloud yet to automatically set up the
>>> master/slave replication relationship though.
>>>
>>> -Yonik
>>> Apache Lucene Eurocon 2010
>>> 18-21 May 2010 | Prague
>>>
>
>
Re: Replicate cores from master to slave
Posted by Jon Baer <jo...@gmail.com>.
Correct me if Im wrong but I think the problem here is that while there is a "fetchindex" command in replication the handler and the master/slave setup pertain to the core config.
For example for this to work properly the solr.xml configuration would need to setup some type of "global" replication handler for all cores.
Just throwing that out there since Im not sure even the ZooKeeper setup would include something like this.
- Jon
On Apr 28, 2010, at 10:14 AM, Jason Rutherglen wrote:
> I guess I didn't explain it properly. I want to create a core on
> the master, and then have N slaves also (aka replicate) create
> those new core(s) on the slave servers, then of course, begin to
> replicate (yeah, got that part). There doesn't appear to be
> anything today that does this, it's unclear how/if Solr Cloud
> enables the described functionality because, while yes Solr
> Cloud can be used to *create* new cores out of a config in ZK,
> that isn't what I'm asking. I'll open an issue and submit a
> patch?
>
> On Wed, Apr 28, 2010 at 1:53 AM, Yonik Seeley
> <yo...@lucidimagination.com> wrote:
>> On Tue, Apr 27, 2010 at 9:56 PM, Chris Hostetter
>> <ho...@fucit.org> wrote:
>>> but as i understand the new cloud stuff (by which i mean: i don't
>>> understand the new cloud stuff, but i've heard rumors) this will be
>>> possible with that functionality.
>>
>> Yeah, that should be the goal.
>> The solrconfig.xml, etc, are stored in zookeeper, so creating a new
>> core is (should be) trivial.
>> There's not anything in cloud yet to automatically set up the
>> master/slave replication relationship though.
>>
>> -Yonik
>> Apache Lucene Eurocon 2010
>> 18-21 May 2010 | Prague
>>
Re: Replicate cores from master to slave
Posted by Jason Rutherglen <ja...@gmail.com>.
In this case, there's a index/core created per day. Each new day/core
needs to fairly immediately after creation, replicate to the slave for
queries. Is there some other mechanism in trunk/cloud that would
solve this?
On Wed, Apr 28, 2010 at 9:22 AM, Yonik Seeley
<yo...@lucidimagination.com> wrote:
> On Wed, Apr 28, 2010 at 10:14 AM, Jason Rutherglen
> <ja...@gmail.com> wrote:
>> I guess I didn't explain it properly. I want to create a core on
>> the master, and then have N slaves also (aka replicate) create
>> those new core(s) on the slave servers, then of course, begin to
>> replicate (yeah, got that part). There doesn't appear to be
>> anything today that does this, it's unclear how/if Solr Cloud
>> enables the described functionality because, while yes Solr
>> Cloud can be used to *create* new cores out of a config in ZK,
>> that isn't what I'm asking. I'll open an issue and submit a
>> patch?
>
> I guess the point was more that a concept of "replicating cores"
> doesn't seem like the direction that Solr is currently going. So
> perhaps we should back up and figure out what this new core is and why
> it's being created. Is it simply a new shard (a new piece of the
> whole index that shares the same config), or an entirely new
> collection?
>
> -Yonik
> Apache Lucene Eurocon 2010
> 18-21 May 2010 | Prague
>
Re: Replicate cores from master to slave
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Apr 28, 2010 at 10:14 AM, Jason Rutherglen
<ja...@gmail.com> wrote:
> I guess I didn't explain it properly. I want to create a core on
> the master, and then have N slaves also (aka replicate) create
> those new core(s) on the slave servers, then of course, begin to
> replicate (yeah, got that part). There doesn't appear to be
> anything today that does this, it's unclear how/if Solr Cloud
> enables the described functionality because, while yes Solr
> Cloud can be used to *create* new cores out of a config in ZK,
> that isn't what I'm asking. I'll open an issue and submit a
> patch?
I guess the point was more that a concept of "replicating cores"
doesn't seem like the direction that Solr is currently going. So
perhaps we should back up and figure out what this new core is and why
it's being created. Is it simply a new shard (a new piece of the
whole index that shares the same config), or an entirely new
collection?
-Yonik
Apache Lucene Eurocon 2010
18-21 May 2010 | Prague
Re: Replicate cores from master to slave
Posted by Jason Rutherglen <ja...@gmail.com>.
I guess I didn't explain it properly. I want to create a core on
the master, and then have N slaves also (aka replicate) create
those new core(s) on the slave servers, then of course, begin to
replicate (yeah, got that part). There doesn't appear to be
anything today that does this, it's unclear how/if Solr Cloud
enables the described functionality because, while yes Solr
Cloud can be used to *create* new cores out of a config in ZK,
that isn't what I'm asking. I'll open an issue and submit a
patch?
On Wed, Apr 28, 2010 at 1:53 AM, Yonik Seeley
<yo...@lucidimagination.com> wrote:
> On Tue, Apr 27, 2010 at 9:56 PM, Chris Hostetter
> <ho...@fucit.org> wrote:
>> but as i understand the new cloud stuff (by which i mean: i don't
>> understand the new cloud stuff, but i've heard rumors) this will be
>> possible with that functionality.
>
> Yeah, that should be the goal.
> The solrconfig.xml, etc, are stored in zookeeper, so creating a new
> core is (should be) trivial.
> There's not anything in cloud yet to automatically set up the
> master/slave replication relationship though.
>
> -Yonik
> Apache Lucene Eurocon 2010
> 18-21 May 2010 | Prague
>
Re: Replicate cores from master to slave
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Tue, Apr 27, 2010 at 9:56 PM, Chris Hostetter
<ho...@fucit.org> wrote:
> but as i understand the new cloud stuff (by which i mean: i don't
> understand the new cloud stuff, but i've heard rumors) this will be
> possible with that functionality.
Yeah, that should be the goal.
The solrconfig.xml, etc, are stored in zookeeper, so creating a new
core is (should be) trivial.
There's not anything in cloud yet to automatically set up the
master/slave replication relationship though.
-Yonik
Apache Lucene Eurocon 2010
18-21 May 2010 | Prague
Re: Replicate cores from master to slave
Posted by Chris Hostetter <ho...@fucit.org>.
: If I create a new core on a Solr master, is there a way to instruct a
: Solr slave to replicate the new core?
replication is handled by a handler running inside the core, so the slave
has to have the core running beore it can start replicating.
but as i understand the new cloud stuff (by which i mean: i don't
understand the new cloud stuff, but i've heard rumors) this will be
possible with that functionality.
-Hoss