You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by mlathe <ml...@gmail.com> on 2009/06/09 23:55:20 UTC
Re: snapshot config value for Solr 1.4 Replication
Does anyone have comments on this?
Is this on the wrong list? should it be solr-user?
Thanks
--Matthias
mlathe wrote:
>
> Hi All,
> I'm doing some proof of concept work with Solr Replication
> http://wiki.apache.org/solr/SolrReplication
>
> If you dig through the ReplicationHandler code you will see that the
> master node's config can include "replicateAfter" and "snapshot", like
> this:
> <lst name="master">
> <str name="replicateAfter">startup,commit</str>
> <str name="snapshot">startup,commit</str>
> <str name="confFiles">schema.xml,stopwords.txt,synonyms.txt</str>
> </lst>
>
> Does anyone understand what the snapshot values do? it's not defined in
> the wiki documentation.
>
> Thanks
> --Matthias
>
>
--
View this message in context: http://www.nabble.com/snapshot-config-value-for-Solr-1.4-Replication-tp23788960p23952016.html
Sent from the Solr - Dev mailing list archive at Nabble.com.
Re: snapshot config value for Solr 1.4 Replication
Posted by mlathe <ml...@gmail.com>.
Shalin Shekhar Mangar wrote:
>
> Please note that snapshots are needed only if you want backups.
> Replication
> does not need to create snapshots.
>
Thanks. I updated the Wiki page
http://wiki.apache.org/solr/SolrReplication
--
View this message in context: http://www.nabble.com/snapshot-config-value-for-Solr-1.4-Replication-tp23788960p23967397.html
Sent from the Solr - Dev mailing list archive at Nabble.com.
Re: snapshot config value for Solr 1.4 Replication
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Wed, Jun 10, 2009 at 11:37 PM, Mark Miller <ma...@gmail.com>wrote:
>
> The wiki is quite confusing on this.
>
> Doesnt it talk about the slave pulling a snapshot (or snap pull)? What does
> it pull if you don't create a snapshot?
>
> I think the terminology around snapshot needs to be cleaned up.
>
Yes, the snapshot terminology is an artifact of the rsync based replication.
ReplicationHandler does not need to create the snapshot directories that
were necessary for rsync replication. It replicates directly out of the
index using the Lucene index commit points. An option to create the snapshot
directories is still provided for people who used to rely on the snapshot
directories for backup.
It does need to be cleaned up though. I'll try to take a shot at it.
--
Regards,
Shalin Shekhar Mangar.
Re: snapshot config value for Solr 1.4 Replication
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
Mark ,
I have opened an issue SOLR-1216
let us make the necessary changes right away
thanks,
On Fri, Jun 12, 2009 at 4:23 AM, Mark Miller<ma...@gmail.com> wrote:
> +1 Noble. I think we should be careful about using the word snap or
> snapshot.
>
> The reason I say its confusing is that: at the top of the wiki, it said "no
> snapshots" - but indeed there were snapshots, just not in the same sense as
> before. Shalin also mentioned that
> this new replication method doesnt need to take a snapshot to replicate, but
> in my mind, the commit point is a snapshot of the index (terminology thats
> been used in the Lucene world). The only difference from the old scripts
> replication snapshot is that you dont copy it out to a different folder with
> this new method - to me, its a snapshot maintained in the same folder.
>
> So in my mind, snapshot is just so overloaded, and almost more confusing
> than helpful.
>
> Its also not very clear that you don't need to call snappull unless you want
> backups. Its also not clear where those backups will go (to someone reading
> the wiki), or how they are managed, or rather you can replicate from a
> backup
> rather than the live index, etc. Its also not very clear how
> "enable/disable" replicate relate to "enable/disable" polling. I know
> because I saw the email about it, but a user reading the wiki would have
> trouble. Its also not clear that replication defaults to on as well as
> polling (I think?).
>
> It might also talk about what happens if you shutdown during replication.
> Will Solr wait for the replication to finish? Will it be cut off? If its cut
> off, are the temp files cleaned up later?
>
> Not to be over critical, or come down all at once, I was just going over the
> page with someone trying to learn it recently and these are the type of
> issues that came up. Just some suggestions of what we could address,
> but I'm not nearly as familiar with it as you guys are.
>
> - Mark
>
> Noble Paul നോബിള് नोब्ळ् wrote:
>>
>> how about renaming the 'snappull' command to 'synchronize' or 'sync'
>>
>> this is the only place where it uses the term 'snap'
>>
>>
>> On Wed, Jun 10, 2009 at 11:37 PM, Mark Miller<ma...@gmail.com>
>> wrote:
>>
>>>
>>> Shalin Shekhar Mangar wrote:
>>>
>>>>
>>>> 2009/6/10 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
>>>>
>>>>
>>>>
>>>>>
>>>>> <str name="snapshot">startup</str>
>>>>> <str name="snapshot">commit</str>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> Please note that snapshots are needed only if you want backups.
>>>> Replication
>>>> does not need to create snapshots.
>>>>
>>>>
>>>>
>>>
>>> The wiki is quite confusing on this.
>>>
>>> Doesnt it talk about the slave pulling a snapshot (or snap pull)? What
>>> does
>>> it pull if you don't create a snapshot?
>>>
>>> I think the terminology around snapshot needs to be cleaned up.
>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: snapshot config value for Solr 1.4 Replication
Posted by Mark Miller <ma...@gmail.com>.
+1 Noble. I think we should be careful about using the word snap or
snapshot.
The reason I say its confusing is that: at the top of the wiki, it said
"no snapshots" - but indeed there were snapshots, just not in the same
sense as before. Shalin also mentioned that
this new replication method doesnt need to take a snapshot to replicate,
but in my mind, the commit point is a snapshot of the index (terminology
thats been used in the Lucene world). The only difference from the old
scripts replication snapshot is that you dont copy it out to a different
folder with this new method - to me, its a snapshot maintained in the
same folder.
So in my mind, snapshot is just so overloaded, and almost more confusing
than helpful.
Its also not very clear that you don't need to call snappull unless you
want backups. Its also not clear where those backups will go (to someone
reading the wiki), or how they are managed, or rather you can replicate
from a backup
rather than the live index, etc. Its also not very clear how
"enable/disable" replicate relate to "enable/disable" polling. I know
because I saw the email about it, but a user reading the wiki would have
trouble. Its also not clear that replication defaults to on as well as
polling (I think?).
It might also talk about what happens if you shutdown during
replication. Will Solr wait for the replication to finish? Will it be
cut off? If its cut off, are the temp files cleaned up later?
Not to be over critical, or come down all at once, I was just going over
the page with someone trying to learn it recently and these are the type
of issues that came up. Just some suggestions of what we could address,
but I'm not nearly as familiar with it as you guys are.
- Mark
Noble Paul നോബിള് नोब्ळ् wrote:
> how about renaming the 'snappull' command to 'synchronize' or 'sync'
>
> this is the only place where it uses the term 'snap'
>
>
> On Wed, Jun 10, 2009 at 11:37 PM, Mark Miller<ma...@gmail.com> wrote:
>
>> Shalin Shekhar Mangar wrote:
>>
>>> 2009/6/10 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
>>>
>>>
>>>
>>>> <str name="snapshot">startup</str>
>>>> <str name="snapshot">commit</str>
>>>>
>>>>
>>>>
>>>>
>>> Please note that snapshots are needed only if you want backups.
>>> Replication
>>> does not need to create snapshots.
>>>
>>>
>>>
>> The wiki is quite confusing on this.
>>
>> Doesnt it talk about the slave pulling a snapshot (or snap pull)? What does
>> it pull if you don't create a snapshot?
>>
>> I think the terminology around snapshot needs to be cleaned up.
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>>
>
>
>
>
--
- Mark
http://www.lucidimagination.com
Re: snapshot config value for Solr 1.4 Replication
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
how about renaming the 'snappull' command to 'synchronize' or 'sync'
this is the only place where it uses the term 'snap'
On Wed, Jun 10, 2009 at 11:37 PM, Mark Miller<ma...@gmail.com> wrote:
> Shalin Shekhar Mangar wrote:
>>
>> 2009/6/10 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
>>
>>
>>>
>>> <str name="snapshot">startup</str>
>>> <str name="snapshot">commit</str>
>>>
>>>
>>>
>>
>> Please note that snapshots are needed only if you want backups.
>> Replication
>> does not need to create snapshots.
>>
>>
>
> The wiki is quite confusing on this.
>
> Doesnt it talk about the slave pulling a snapshot (or snap pull)? What does
> it pull if you don't create a snapshot?
>
> I think the terminology around snapshot needs to be cleaned up.
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: snapshot config value for Solr 1.4 Replication
Posted by Mark Miller <ma...@gmail.com>.
Shalin Shekhar Mangar wrote:
> 2009/6/10 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
>
>
>> <str name="snapshot">startup</str>
>> <str name="snapshot">commit</str>
>>
>>
>>
> Please note that snapshots are needed only if you want backups. Replication
> does not need to create snapshots.
>
>
The wiki is quite confusing on this.
Doesnt it talk about the slave pulling a snapshot (or snap pull)? What
does it pull if you don't create a snapshot?
I think the terminology around snapshot needs to be cleaned up.
--
- Mark
http://www.lucidimagination.com
Re: snapshot config value for Solr 1.4 Replication
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
2009/6/10 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
>
> <str name="snapshot">startup</str>
> <str name="snapshot">commit</str>
>
>
Please note that snapshots are needed only if you want backups. Replication
does not need to create snapshots.
--
Regards,
Shalin Shekhar Mangar.
Re: snapshot config value for Solr 1.4 Replication
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
On Wed, Jun 10, 2009 at 3:25 AM, mlathe<ml...@gmail.com> wrote:
>
> Does anyone have comments on this?
> Is this on the wrong list? should it be solr-user?
This questions could go to solr-user
>
> Thanks
> --Matthias
>
>
> mlathe wrote:
>>
>> Hi All,
>> I'm doing some proof of concept work with Solr Replication
>> http://wiki.apache.org/solr/SolrReplication
>>
>> If you dig through the ReplicationHandler code you will see that the
>> master node's config can include "replicateAfter" and "snapshot", like
>> this:
>> <lst name="master">
this is wrong
>> <str name="replicateAfter">startup,commit</str>
this is right
<str name="replicateAfter">startup</str>
<str name="replicateAfter">commit</str>
this is wrong
>> <str name="snapshot">startup,commit</str>
this right
<str name="snapshot">startup</str>
<str name="snapshot">commit</str>
>> <str name="confFiles">schema.xml,stopwords.txt,synonyms.txt</str>
>> </lst>
>>
>> Does anyone understand what the snapshot values do? it's not defined in
>> the wiki documentation.
>>
>> Thanks
>> --Matthias
>>
>>
>
> --
> View this message in context: http://www.nabble.com/snapshot-config-value-for-Solr-1.4-Replication-tp23788960p23952016.html
> Sent from the Solr - Dev mailing list archive at Nabble.com.
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com