You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ken Krugler <kk...@transpac.com> on 2007/02/13 16:33:26 UTC
Question re snapinstaller
Hi all,
In looking at the snapinstaller script, it seems to do the following:
1. Copy a new index directory from the master to the slave's Solr
data directory, giving it a name "index.tmp<number>".
2. Delete the current index directory ("index").
3. Rename the temp index directory to be "index".
Then the commit script will send a <commit/> POST to the
.../solr/update service, and the new index gets swapped into use.
I feel like I must be missing something, because it seems like any
request that's in the middle of being processed between step #2 and
the end of a successful swap could fail due to the index changing
underneath. Any insights here?
Thanks,
-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"
Re: Question re snapinstaller
Posted by Yonik Seeley <yo...@apache.org>.
On 2/13/07, Bill Au <bi...@gmail.com> wrote:
> Solr snapshots are created using hard links. The file is not deleted as
> long as there is 1 or more
> link to it.
Or a process that holds it open. It would work even if there were no
links in the filesystem because the IndexReader would still be holding
them open.
-Yonik
Re: Question re snapinstaller
Posted by Bill Au <bi...@gmail.com>.
Solr snapshots are created using hard links. The file is not deleted as
long as there is 1 or more
link to it.
Bill
On 2/13/07, Mike Klaas <mi...@gmail.com> wrote:
>
> On 2/13/07, Ken Krugler <kk...@transpac.com> wrote:
>
> > >A Lucene's IndexReader opens all index files it needs when it is
> instantiated.
> > >Changes to a Lucene index via IndexWriter never change an existing
> > >file... new files are always created.
> > >Put the two together and it allows an IndexWriter (or anything else,
> > >like snapinstaller) to change the index in the background without
> > >impact to open readers.
> >
> > Right, but from looking at the snapinstaller script it seemed as
> > though the deletion of the "index" directory would be deleting files
> > out from under the IndexReader.
>
> Files aren't truly deleted until no process has an open file descriptor:
>
>
> http://users.actcom.co.il/~choo/lupg/tutorials/handling-files/handling-files.html#sys_file_unlink
>
> -Mike
>
Re: Question re snapinstaller
Posted by Mike Klaas <mi...@gmail.com>.
On 2/13/07, Ken Krugler <kk...@transpac.com> wrote:
> >A Lucene's IndexReader opens all index files it needs when it is instantiated.
> >Changes to a Lucene index via IndexWriter never change an existing
> >file... new files are always created.
> >Put the two together and it allows an IndexWriter (or anything else,
> >like snapinstaller) to change the index in the background without
> >impact to open readers.
>
> Right, but from looking at the snapinstaller script it seemed as
> though the deletion of the "index" directory would be deleting files
> out from under the IndexReader.
Files aren't truly deleted until no process has an open file descriptor:
http://users.actcom.co.il/~choo/lupg/tutorials/handling-files/handling-files.html#sys_file_unlink
-Mike
Re: Question re snapinstaller
Posted by Ken Krugler <kk...@transpac.com>.
>On 2/13/07, Ken Krugler <kk...@transpac.com> wrote:
>>Hi all,
>>
>>In looking at the snapinstaller script, it seems to do the following:
>>
>>1. Copy a new index directory from the master to the slave's Solr
>>data directory, giving it a name "index.tmp<number>".
>>
>>2. Delete the current index directory ("index").
>>
>>3. Rename the temp index directory to be "index".
>>
>>Then the commit script will send a <commit/> POST to the
>>.../solr/update service, and the new index gets swapped into use.
>>
>>I feel like I must be missing something, because it seems like any
>>request that's in the middle of being processed between step #2 and
>>the end of a successful swap could fail due to the index changing
>>underneath. Any insights here?
>
>A Lucene's IndexReader opens all index files it needs when it is instantiated.
>Changes to a Lucene index via IndexWriter never change an existing
>file... new files are always created.
>Put the two together and it allows an IndexWriter (or anything else,
>like snapinstaller) to change the index in the background without
>impact to open readers.
Right, but from looking at the snapinstaller script it seemed as
though the deletion of the "index" directory would be deleting files
out from under the IndexReader.
Thanks,
-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"
Re: Question re snapinstaller
Posted by Yonik Seeley <yo...@apache.org>.
On 2/13/07, Ken Krugler <kk...@transpac.com> wrote:
> Hi all,
>
> In looking at the snapinstaller script, it seems to do the following:
>
> 1. Copy a new index directory from the master to the slave's Solr
> data directory, giving it a name "index.tmp<number>".
>
> 2. Delete the current index directory ("index").
>
> 3. Rename the temp index directory to be "index".
>
> Then the commit script will send a <commit/> POST to the
> .../solr/update service, and the new index gets swapped into use.
>
> I feel like I must be missing something, because it seems like any
> request that's in the middle of being processed between step #2 and
> the end of a successful swap could fail due to the index changing
> underneath. Any insights here?
A Lucene's IndexReader opens all index files it needs when it is instantiated.
Changes to a Lucene index via IndexWriter never change an existing
file... new files are always created.
Put the two together and it allows an IndexWriter (or anything else,
like snapinstaller) to change the index in the background without
impact to open readers.
-Yonik