You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tracy Flynn <tr...@zoemail.net> on 2007/12/11 03:03:51 UTC

Replication hooks

Hi,

I'm interested in setting up simple replication. I've reviewed all the  
Wiki information, looked at the scripts etc. and understand most of  
what I see.

There are some references to  'hooks in the code'  for both the master  
and slave nodes for handling replication. I've searched the 1.2 and  
trunk code bases for obvious phrases, but I can't identify these hooks.

Can someone please point me to the correct place(s) to look?

Thanks,

Tracy Flynn

Re: Replication hooks - changing the index while the slave is running ...

Posted by Tracy Flynn <tr...@zoemail.net>.
Lance,

A belated thanks for explaining the reason that the delete works for  
replacing the slave index copies.

Tracy


On Dec 17, 2007, at 6:05 PM, Norskog, Lance wrote:

> It works via two Unix file system tricks.
>
> 1) Files are not directly bound with with filenames, instead there  
> is a
> layer of indirection called an 'inode'. So, multiple file and  
> directory
> names point to the same physical file. The "." and ".." directory
> entries are implemented this way.
>
> 2) Physical files are bound to all open file descriptors even after
> there are no file names for the files. So, file data exists until all
> file names are gone AND all open files are gone.
>
> Lance
>
> -----Original Message-----
> From: Tracy Flynn [mailto:tracysflynn.concur89394@zoemail.net]
> Sent: Saturday, December 15, 2007 7:36 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Replication hooks - changing the index while the slave is
> running ...
>
> That helps
>
> Thanks for the prompt reply
>
> On Dec 15, 2007, at 10:15 AM, Yonik Seeley wrote:
>
>> On Dec 14, 2007 7:36 PM, Tracy Flynn
>> <tr...@zoemail.net> wrote:
>>> 1) The existing index(es) being used by the Solr slave instance are
>>> physically deleted
>>> 2) The new index snapshots are renamed/moved from their temporary
>>> installation location to the default index location
>>> 3) The slave is sent a 'commit' to force a new IndexReader to start
>>> to read the new index.
>>>
>>> What happens to search requests against the existing/old index  
>>> during
>
>>> step 1) and between steps 1 and 2?
>>
>> Search requests will still work on the old searcher/index.
>>
>>> Where do they get information if
>>> they need to go to disk for results that are not cached? Do they a)
>>> hang b) produce no results c) error in some other way?
>>
>> A lucene IndexReader keeps all the files open that aren't loaded into
>> memory... and external deletion has no effect on the ability to keep
>> reading these open files (they aren't really deleted yet).
>>
>> -Yonik
>


RE: Replication hooks - changing the index while the slave is running ...

Posted by "Norskog, Lance" <la...@divvio.com>.
It works via two Unix file system tricks. 

1) Files are not directly bound with with filenames, instead there is a
layer of indirection called an 'inode'. So, multiple file and directory
names point to the same physical file. The "." and ".." directory
entries are implemented this way. 

2) Physical files are bound to all open file descriptors even after
there are no file names for the files. So, file data exists until all
file names are gone AND all open files are gone.

Lance

-----Original Message-----
From: Tracy Flynn [mailto:tracysflynn.concur89394@zoemail.net] 
Sent: Saturday, December 15, 2007 7:36 AM
To: solr-user@lucene.apache.org
Subject: Re: Replication hooks - changing the index while the slave is
running ...

That helps

Thanks for the prompt reply

On Dec 15, 2007, at 10:15 AM, Yonik Seeley wrote:

> On Dec 14, 2007 7:36 PM, Tracy Flynn
> <tr...@zoemail.net> wrote:
>> 1) The existing index(es) being used by the Solr slave instance are 
>> physically deleted
>> 2) The new index snapshots are renamed/moved from their temporary 
>> installation location to the default index location
>> 3) The slave is sent a 'commit' to force a new IndexReader to start 
>> to read the new index.
>>
>> What happens to search requests against the existing/old index during

>> step 1) and between steps 1 and 2?
>
> Search requests will still work on the old searcher/index.
>
>> Where do they get information if
>> they need to go to disk for results that are not cached? Do they a) 
>> hang b) produce no results c) error in some other way?
>
> A lucene IndexReader keeps all the files open that aren't loaded into 
> memory... and external deletion has no effect on the ability to keep 
> reading these open files (they aren't really deleted yet).
>
> -Yonik


Re: Replication hooks - changing the index while the slave is running ...

Posted by Tracy Flynn <tr...@zoemail.net>.
That helps

Thanks for the prompt reply

On Dec 15, 2007, at 10:15 AM, Yonik Seeley wrote:

> On Dec 14, 2007 7:36 PM, Tracy Flynn
> <tr...@zoemail.net> wrote:
>> 1) The existing index(es) being used by the Solr slave instance are
>> physically deleted
>> 2) The new index snapshots are renamed/moved from their temporary
>> installation location to the default index location
>> 3) The slave is sent a 'commit' to force a new IndexReader to start  
>> to
>> read the new index.
>>
>> What happens to search requests against the existing/old index during
>> step 1) and between steps 1 and 2?
>
> Search requests will still work on the old searcher/index.
>
>> Where do they get information if
>> they need to go to disk for results that are not cached? Do they a)
>> hang b) produce no results c) error in some other way?
>
> A lucene IndexReader keeps all the files open that aren't loaded into
> memory... and external deletion has no effect on the ability to keep
> reading these open files (they aren't really deleted yet).
>
> -Yonik


Re: Replication hooks - changing the index while the slave is running ...

Posted by Yonik Seeley <yo...@apache.org>.
On Dec 14, 2007 7:36 PM, Tracy Flynn
<tr...@zoemail.net> wrote:
> 1) The existing index(es) being used by the Solr slave instance are
> physically deleted
> 2) The new index snapshots are renamed/moved from their temporary
> installation location to the default index location
> 3) The slave is sent a 'commit' to force a new IndexReader to start to
> read the new index.
>
> What happens to search requests against the existing/old index during
> step 1) and between steps 1 and 2?

Search requests will still work on the old searcher/index.

> Where do they get information if
> they need to go to disk for results that are not cached? Do they a)
> hang b) produce no results c) error in some other way?

A lucene IndexReader keeps all the files open that aren't loaded into
memory... and external deletion has no effect on the ability to keep
reading these open files (they aren't really deleted yet).

-Yonik

Re: Replication hooks - changing the index while the slave is running ...

Posted by Tracy Flynn <tr...@zoemail.net>.
I've looked at everything suggested - documentation, code and scripts  
and I follow almost everything that's happening.

If I understand correctly, when the updated snapshot is installed on  
the slave:

1) The existing index(es) being used by the Solr slave instance are  
physically deleted
2) The new index snapshots are renamed/moved from their temporary  
installation location to the default index location
3) The slave is sent a 'commit' to force a new IndexReader to start to  
read the new index.

What happens to search requests against the existing/old index during  
step 1) and between steps 1 and 2?  Where do they get information if  
they need to go to disk for results that are not cached? Do they a)  
hang b) produce no results c) error in some other way?


Regards,

Tracy

On Dec 11, 2007, at 9:11 AM, Tracy Flynn wrote:

> That's what I was after.
>
> As always, thanks for the quick response.
>
> Tracy
>
> On Dec 11, 2007, at 12:18 AM, Yonik Seeley wrote:
>
>> On Dec 10, 2007 11:22 PM, climbingrose <cl...@gmail.com>  
>> wrote:
>>> I think there is a event listener interface for hooking into Solr  
>>> events
>>> such as post commit, post optimise and open new searcher. I can't  
>>> remember
>>> on top of my head but if you do a search for *EventListener in  
>>> Eclipse,
>>> you'll find it.
>>> The Wiki shows how to trigger snapshooter after each commit and  
>>> optimise.
>>> You should be able to follow this example to create your own  
>>> listener.
>>
>> Right... you shouldn't need to implement your own listeners though.
>> Search for postCommit in the example solrconfig.xml
>>
>> -Yonik
>


Re: Replication hooks

Posted by Tracy Flynn <tr...@zoemail.net>.
That's what I was after.

As always, thanks for the quick response.

Tracy

On Dec 11, 2007, at 12:18 AM, Yonik Seeley wrote:

> On Dec 10, 2007 11:22 PM, climbingrose <cl...@gmail.com> wrote:
>> I think there is a event listener interface for hooking into Solr  
>> events
>> such as post commit, post optimise and open new searcher. I can't  
>> remember
>> on top of my head but if you do a search for *EventListener in  
>> Eclipse,
>> you'll find it.
>> The Wiki shows how to trigger snapshooter after each commit and  
>> optimise.
>> You should be able to follow this example to create your own  
>> listener.
>
> Right... you shouldn't need to implement your own listeners though.
> Search for postCommit in the example solrconfig.xml
>
> -Yonik


Re: Replication hooks

Posted by Yonik Seeley <yo...@apache.org>.
On Dec 10, 2007 11:22 PM, climbingrose <cl...@gmail.com> wrote:
> I think there is a event listener interface for hooking into Solr events
> such as post commit, post optimise and open new searcher. I can't remember
> on top of my head but if you do a search for *EventListener in Eclipse,
> you'll find it.
> The Wiki shows how to trigger snapshooter after each commit and optimise.
> You should be able to follow this example to create your own listener.

Right... you shouldn't need to implement your own listeners though.
Search for postCommit in the example solrconfig.xml

-Yonik

Re: Replication hooks

Posted by climbingrose <cl...@gmail.com>.
I think there is a event listener interface for hooking into Solr events
such as post commit, post optimise and open new searcher. I can't remember
on top of my head but if you do a search for *EventListener in Eclipse,
you'll find it.
The Wiki shows how to trigger snapshooter after each commit and optimise.
You should be able to follow this example to create your own listener.

On Dec 11, 2007 1:03 PM, Tracy Flynn <tr...@zoemail.net>
wrote:

> Hi,
>
> I'm interested in setting up simple replication. I've reviewed all the
> Wiki information, looked at the scripts etc. and understand most of
> what I see.
>
> There are some references to  'hooks in the code'  for both the master
> and slave nodes for handling replication. I've searched the 1.2 and
> trunk code bases for obvious phrases, but I can't identify these hooks.
>
> Can someone please point me to the correct place(s) to look?
>
> Thanks,
>
> Tracy Flynn
>



-- 
Regards,

Cuong Hoang