replication should include the schema also

[Noble Paul നോബിള്‍ नोब्ळ् <no...@gmail.com>]

The current Solr replication just copy the data directory . So if the
schema changes and I do a re-index it will blissfully copy the index
and the slaves will fail because of incompatible schema.

So the steps we follow are
 * Stop rsync on slaves
 * Update the master with new schema
 * re-index data
 * forEach slave
 ** Kill the slave
 ** clean the data directory
 ** install the new schema
 ** restart
 ** do a manual snappull

The amount of work the admin needs to do is quite significant
(depending on the no:of slaves). These are manual steps and very error
prone

The solution :
Make the replication mechanism handle the schema replication also. So
all I need to do is to just change the master and the slaves synch
automatically

What is a good way to implement this?

We have an idea along the following lines

This should involve changes to the snapshooter and snappuller scripts
and the snapinstaller components

Everytime the snapshooter takes a snapshot it must keep the timestamps
of schema.xml and elevate.xml (all the files which might affect the
runtime behavior in slaves)
For subsequent snapshots if the timestamps of any of them is changed
it must copy the all of them also for replication.

The snappuller copies the new directory as usual

The snapinstaller checks if these config files are present ,

if yes,
 * It can create a temporary core
 * install the changed index and configuration
 * load it completely and swap it out with the original core

--Noble

Re: replication should include the schema also

[Noble Paul നോബിള്‍ नोब्ळ् <no...@gmail.com>]

are u using a trunk version? did u try the new replication feature
http://wiki.apache.org/solr/SolrReplication it supports solrconfig
replication automatically

On Fri, Feb 13, 2009 at 6:56 AM, sunnyfr <jo...@gmail.com> wrote:
>
> Hi,
>
> What do you mean by clean the data directory on the slave servers?
> Do you have remove everything from it and then start a new rsyncd ???
> and turn on snappuller cronjob ??
> thanks a lot,
>
>
> Noble Paul നോബിള്‍  नोब्ळ् wrote:
>>
>> The current Solr replication just copy the data directory . So if the
>> schema changes and I do a re-index it will blissfully copy the index
>> and the slaves will fail because of incompatible schema.
>>
>> So the steps we follow are
>>  * Stop rsync on slaves
>>  * Update the master with new schema
>>  * re-index data
>>  * forEach slave
>>  ** Kill the slave
>>  ** clean the data directory
>>  ** install the new schema
>>  ** restart
>>  ** do a manual snappull
>>
>> The amount of work the admin needs to do is quite significant
>> (depending on the no:of slaves). These are manual steps and very error
>> prone
>>
>> The solution :
>> Make the replication mechanism handle the schema replication also. So
>> all I need to do is to just change the master and the slaves synch
>> automatically
>>
>> What is a good way to implement this?
>>
>> We have an idea along the following lines
>>
>> This should involve changes to the snapshooter and snappuller scripts
>> and the snapinstaller components
>>
>> Everytime the snapshooter takes a snapshot it must keep the timestamps
>> of schema.xml and elevate.xml (all the files which might affect the
>> runtime behavior in slaves)
>> For subsequent snapshots if the timestamps of any of them is changed
>> it must copy the all of them also for replication.
>>
>> The snappuller copies the new directory as usual
>>
>> The snapinstaller checks if these config files are present ,
>>
>> if yes,
>>  * It can create a temporary core
>>  * install the changed index and configuration
>>  * load it completely and swap it out with the original core
>>
>> --Noble
>>
>>
>
> --
> View this message in context: http://www.nabble.com/replication-should-include-the-schema-also-tp16851477p21995813.html
> Sent from the Solr - Dev mailing list archive at Nabble.com.
>
>



-- 
--Noble Paul

Re: replication should include the schema also

[sunnyfr <jo...@gmail.com>]

Hi,

What do you mean by clean the data directory on the slave servers?
Do you have remove everything from it and then start a new rsyncd ??? 
and turn on snappuller cronjob ??
thanks a lot,


Noble Paul നോബിള്‍  नोब्ळ् wrote:
> 
> The current Solr replication just copy the data directory . So if the
> schema changes and I do a re-index it will blissfully copy the index
> and the slaves will fail because of incompatible schema.
> 
> So the steps we follow are
>  * Stop rsync on slaves
>  * Update the master with new schema
>  * re-index data
>  * forEach slave
>  ** Kill the slave
>  ** clean the data directory
>  ** install the new schema
>  ** restart
>  ** do a manual snappull
> 
> The amount of work the admin needs to do is quite significant
> (depending on the no:of slaves). These are manual steps and very error
> prone
> 
> The solution :
> Make the replication mechanism handle the schema replication also. So
> all I need to do is to just change the master and the slaves synch
> automatically
> 
> What is a good way to implement this?
> 
> We have an idea along the following lines
> 
> This should involve changes to the snapshooter and snappuller scripts
> and the snapinstaller components
> 
> Everytime the snapshooter takes a snapshot it must keep the timestamps
> of schema.xml and elevate.xml (all the files which might affect the
> runtime behavior in slaves)
> For subsequent snapshots if the timestamps of any of them is changed
> it must copy the all of them also for replication.
> 
> The snappuller copies the new directory as usual
> 
> The snapinstaller checks if these config files are present ,
> 
> if yes,
>  * It can create a temporary core
>  * install the changed index and configuration
>  * load it completely and swap it out with the original core
> 
> --Noble
> 
> 

-- 
View this message in context: http://www.nabble.com/replication-should-include-the-schema-also-tp16851477p21995813.html
Sent from the Solr - Dev mailing list archive at Nabble.com.