You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by pof <Me...@gmail.com> on 2009/06/09 01:44:22 UTC
Re: How to distribute lucene using rsync
Is their a way to address this without Solr? I was looking at Solr today and
it looks like a lot of piss-farting around just to backup an index. I also
found their explaination on
http://wiki.apache.org/solr/CollectionDistribution poorly explained, so I am
a bit confused with the whole process.
Grant Ingersoll-6 wrote:
>
> Yes, see Solr. It takes care of all this, and then some, for you.
>
> -Grant
>
> On Aug 25, 2007, at 9:52 PM, varma d wrote:
>
>> Hi Lucene gurus,
>> I am newbie and i have a question on transfering index
>> directories across
>> multiple machines. Whenever i update/add any new documents to the
>> existing
>> index, then it is generating new .cfs file. Because of this , i
>> ended up
>> doing complete transfer (instead of diff) as the file names got
>> changed. Is
>> there any way to make the .cfs file name as fixed. I would like to
>> know if
>> anyone successfully synced index directories using rsync.
>>
>> Thanks for ur time
>> varma
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/How-to-distribute-lucene-using-rsync-tp12331421p23934073.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to distribute lucene using rsync
Posted by Ian Lea <ia...@gmail.com>.
Unless you optimize it or are doing weird things with merge factors
you won't get completely new files every time you update an index.
Some files will change, or be created, or deleted, and some won't.
Then you can just copy them wherever you want using rsync or whatever
you like. We use rsync, mainly on non compound indexes although also
on compound, and it works well.
$ rsync -av --delete /local/index/dir-or-tree desthost::lucene
--
Ian.
On Tue, Jun 9, 2009 at 12:44 AM, pof<Me...@gmail.com> wrote:
>
> Is their a way to address this without Solr? I was looking at Solr today and
> it looks like a lot of piss-farting around just to backup an index. I also
> found their explaination on
> http://wiki.apache.org/solr/CollectionDistribution poorly explained, so I am
> a bit confused with the whole process.
>
>
> Grant Ingersoll-6 wrote:
>>
>> Yes, see Solr. It takes care of all this, and then some, for you.
>>
>> -Grant
>>
>> On Aug 25, 2007, at 9:52 PM, varma d wrote:
>>
>>> Hi Lucene gurus,
>>> I am newbie and i have a question on transfering index
>>> directories across
>>> multiple machines. Whenever i update/add any new documents to the
>>> existing
>>> index, then it is generating new .cfs file. Because of this , i
>>> ended up
>>> doing complete transfer (instead of diff) as the file names got
>>> changed. Is
>>> there any way to make the .cfs file name as fixed. I would like to
>>> know if
>>> anyone successfully synced index directories using rsync.
>>>
>>> Thanks for ur time
>>> varma
>>
>> --------------------------
>> Grant Ingersoll
>> http://lucene.grantingersoll.com
>>
>> Lucene Helpful Hints:
>> http://wiki.apache.org/lucene-java/BasicsOfPerformance
>> http://wiki.apache.org/lucene-java/LuceneFAQ
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/How-to-distribute-lucene-using-rsync-tp12331421p23934073.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to distribute lucene using rsync
Posted by Michael McCandless <lu...@mikemccandless.com>.
Note that when using rsync, you must first close the IndexWriter, else
the copy can be corrupt.
If having to close IndexWriter (and stop indexing) is a hassle, then
you should use SnapshotDeletionPolicy; it was created exactly for this
reason (to take a backup of the index even while further indexing is
proceeding).
Mike
On Mon, Jun 8, 2009 at 7:44 PM, pof<Me...@gmail.com> wrote:
>
> Is their a way to address this without Solr? I was looking at Solr today and
> it looks like a lot of piss-farting around just to backup an index. I also
> found their explaination on
> http://wiki.apache.org/solr/CollectionDistribution poorly explained, so I am
> a bit confused with the whole process.
>
>
> Grant Ingersoll-6 wrote:
>>
>> Yes, see Solr. It takes care of all this, and then some, for you.
>>
>> -Grant
>>
>> On Aug 25, 2007, at 9:52 PM, varma d wrote:
>>
>>> Hi Lucene gurus,
>>> I am newbie and i have a question on transfering index
>>> directories across
>>> multiple machines. Whenever i update/add any new documents to the
>>> existing
>>> index, then it is generating new .cfs file. Because of this , i
>>> ended up
>>> doing complete transfer (instead of diff) as the file names got
>>> changed. Is
>>> there any way to make the .cfs file name as fixed. I would like to
>>> know if
>>> anyone successfully synced index directories using rsync.
>>>
>>> Thanks for ur time
>>> varma
>>
>> --------------------------
>> Grant Ingersoll
>> http://lucene.grantingersoll.com
>>
>> Lucene Helpful Hints:
>> http://wiki.apache.org/lucene-java/BasicsOfPerformance
>> http://wiki.apache.org/lucene-java/LuceneFAQ
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/How-to-distribute-lucene-using-rsync-tp12331421p23934073.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org