You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucy.apache.org by goran kent <go...@gmail.com> on 2011/11/01 10:37:30 UTC

[lucy-user] Managing fresh/stale indexes

Hi,

I'm pondering about the best approach to handle fresh/stale indexes
while searches are happening (ie, we don't want to disrupt an active
search).

Let's say you have index1 which is stale, but actively being searched on.
I want to automate the process of switching in a fresh index1.fresh
(symlink or move).  Let's say index1 is being hammered by active
searches: what's the safest way to switch in index1.fresh?  Have an
intermediary?:

index1   --  a symlink to either:

index1.a/
or
index1.b/

(and a control file which indicates the last stale one so the link
cycles through them correctly -- and perhaps a lockfile which tells
searchers: hey, hold for 1s while I reassign this symlink; but perhaps
this is overcomplicating things)

What will happen if a search is currently working on index1 (a symlink
to index1.a, eg) and my automation removes the link and reassigns it
to index1.b?

The contents of index1.a might be cached in RAM, etc.

...or should I not worry about it and let the OS handle these concerns?

Re: [lucy-user] Managing fresh/stale indexes

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Wed, Nov 02, 2011 at 09:29:28AM +0200, goran kent wrote:
> As I'm typing this though, I'm thinking, what about
> Lucy::Search::IndexSearcher->serve?
> It's in a loop hiding it's internals - how do I get *it* to honour my
> $idx_lockfile?
>
> Hack LucyX/Remote/SearchServer.pm's sub serve()?  Perhaps between
> 
> read( $client_sock, $buf, $len );
> + sleep while $idx_lockfile is extant
> my $response   = $dispatch{$method}->( $self, thaw($buf) );

Well, we're up against two problems here.

First, as described earlier, we have the issue that Lucy's index opening
mechanism is not really designed to cope with full-index swapouts.

Second, SearchServer/SearchClient is not really set up to take advantage of
Lucy's near-real-time capabilities.  It was written a long time ago, and was
designed with static index data in mind.  It was anticipated that you might
restart the service and show users a new view of the index once per day, once
per hour or once per minute -- but not that SearchServer would be expected to
provide near-instantaneous availability of updates on cue.

I think that in order to get near-real-time happening with
SearchServer/SearchClient, we may need to enhance SearchServer to handle more
than one Searcher.

Instead of having SearchServer's constructor require a Searcher, we could have
it require an index path.  Then, when a SearchClient connects, SearchServer
could open a fresh IndexSearcher, add it to a hash keyed by snapshot number,
and pass back the snapshot number identifier to the SearchClient.

All subsequent search requests from the SearchClient would need to include
this snapshot number identifier.  And when the SearchClient decides that it's
finished and sends back a "done" command to the SearchServer, only the
IndexSearcher associated with that snapshot number gets zapped (or has its
refcount decremented, if we end up needing refcounting as I suspect we might).

[ Aside to Nate Kurz: This is *precisely* the kind of construct we'd hoped to
enable when we were working on Lucy's "cheap Searcher" model, no? :) ]

> My final question:  Peter says "For full index swap-outs, I just let
> existing Searchers finish and re-open
> themselves when they realize the underlying index is different" --
> peter are you saying LucyX::Remote::SearchServer does this
> automatically on it's own without intervention needed, 

SearchServer does not auto-update when the index changes, and it would be bad
if it did.

    http://incubator.apache.org/lucy/docs/perl/Lucy/Docs/DocIDs.html#Document-ids-are-ephemeral
    
Searcher objects in Lucy always represent a point-in-time view of the indexed
corpus.  You don't want the content associated with a particular document id
to change in between performing a search and fetching a document!

> urgh, this is getting dirty ugly.
>
> /throws toys out cot, sobs

LOL, I understand the frustrations, but honestly you're right up against a
very exciting problem set.

We've wanted for a while now to build a distributed layer for Lucy that
handles sharding and replication: something that uses techniques similar to
Solr, but much thinner than Solr -- no REST interface, no extra query types,
no PDF parsing, etc -- and that fully exploits Lucy's excellent near-real-time
fundamentals.

Here at Eventful, we have our own proprietary distributed layer;
unfortunately, it's optimized for some specific requirements and it wouldn't
be easy to adapt for general use.  Nevertheless, we have the experience of
having built such a layer, and my colleagues and I had a very animated
lunchtime conversation today about how best to design a generalized tool that
would work for users like yourself.  If you want to work on such a tool for
Lucy, you're going to find collaborators!

Marvin Humphrey


Re: [lucy-user] Managing fresh/stale indexes

Posted by goran kent <go...@gmail.com>.
s/wrapper/monitor_script/g

On 11/2/11, goran kent <go...@gmail.com> wrote:
> Thanks to all for responding.
>
> OK, to summarise (and in the interests of keeping things simple):
>
> merging/updating indexers:
> -------------------------------------
>
>   touch $idx_lockfile
>   rm -rf $index
>   mv $fresh $index
>   unlink $idx_lockfile
>   touch $idx_has_changed
>   # ie, don't bother with symlinks and whatnot since active searchers
> have cached everything.
>
> searchers:
> --------------
>
> loop
>   $lockfound = 0;
>   while (-f $idx_lockfile) {
>       # pause for 100ms
>       select(undef, undef, undef, 0.1); }
>       $lockfound = 1;
>   }
>
>   if ($lockfound) {
>       $lockfound = 0;
>       # restart_searcher
>       goto restart;
>   }
> end
>
> As I'm typing this though, I'm thinking, what about
> Lucy::Search::IndexSearcher->serve?
> It's in a loop hiding it's internals - how do I get *it* to honour my
> $idx_lockfile?
>
> Hack LucyX/Remote/SearchServer.pm's sub serve()?  Perhaps between
>
> read( $client_sock, $buf, $len );
> + sleep while $idx_lockfile is extant
> my $response   = $dispatch{$method}->( $self, thaw($buf) );
>
> My final question:  Peter says "For full index swap-outs, I just let
> existing Searchers finish and re-open
> themselves when they realize the underlying index is different" --
> peter are you saying LucyX::Remote::SearchServer does this
> automatically on it's own without intervention needed, or should I
> have a wrapper with LucyX::Remote::SearchServer ensconced which
> monitors the indexes and kills/restarts LucyX::Remote::SearchServer
> when a changed index is detected (-f $idx_has_changed)?
>
> urgh, this is getting dirty ugly.
>
> /throws toys out cot, sobs
>
> On 11/2/11, Peter Karman <pe...@peknet.com> wrote:
>> Marvin Humphrey wrote on 11/1/11 12:36 PM:
>>> On Tue, Nov 01, 2011 at 11:37:30AM +0200, goran kent wrote:
>>>> I'm pondering about the best approach to handle fresh/stale indexes
>>>> while searches are happening (ie, we don't want to disrupt an active
>>>> search).
>>>
>>> Once an IndexSearcher has opened successfully, it no longer needs access
>>> to
>>> the index dir[1] -- you can actually wipe the index dir and the
>>> IndexSearcher
>>> will keep functioning indefinitely.  This is achieved by caching open
>>> filehandles to all files the IndexSearcher could ever need access to, so
>>> that
>>> it will still have access even if the files are deleted[2].
>>>
>>
>> fwiw, this is exactly what I do.
>>
>> For incremental updates, my Searcher wrapper code[1] detects when the
>> index
>> has
>> changed and re-opens itself.
>>
>> For full index swap-outs, I just let existing Searchers finish and
>> re-open
>> themselves when they realize the underlying index is different. When the
>> open
>> filehandles eventually close, then the filesystem actually removes the
>> old
>> index.
>>
>> [1] https://metacpan.org/module/SWISH::Prog::Lucy::Searcher
>>
>> --
>> Peter Karman  .  http://peknet.com/  .  peter@peknet.com
>>
>

Re: [lucy-user] Managing fresh/stale indexes

Posted by goran kent <go...@gmail.com>.
Thanks to all for responding.

OK, to summarise (and in the interests of keeping things simple):

merging/updating indexers:
-------------------------------------

  touch $idx_lockfile
  rm -rf $index
  mv $fresh $index
  unlink $idx_lockfile
  touch $idx_has_changed
  # ie, don't bother with symlinks and whatnot since active searchers
have cached everything.

searchers:
--------------

loop
  $lockfound = 0;
  while (-f $idx_lockfile) {
      # pause for 100ms
      select(undef, undef, undef, 0.1); }
      $lockfound = 1;
  }

  if ($lockfound) {
      $lockfound = 0;
      # restart_searcher
      goto restart;
  }
end

As I'm typing this though, I'm thinking, what about
Lucy::Search::IndexSearcher->serve?
It's in a loop hiding it's internals - how do I get *it* to honour my
$idx_lockfile?

Hack LucyX/Remote/SearchServer.pm's sub serve()?  Perhaps between

read( $client_sock, $buf, $len );
+ sleep while $idx_lockfile is extant
my $response   = $dispatch{$method}->( $self, thaw($buf) );

My final question:  Peter says "For full index swap-outs, I just let
existing Searchers finish and re-open
themselves when they realize the underlying index is different" --
peter are you saying LucyX::Remote::SearchServer does this
automatically on it's own without intervention needed, or should I
have a wrapper with LucyX::Remote::SearchServer ensconced which
monitors the indexes and kills/restarts LucyX::Remote::SearchServer
when a changed index is detected (-f $idx_has_changed)?

urgh, this is getting dirty ugly.

/throws toys out cot, sobs

On 11/2/11, Peter Karman <pe...@peknet.com> wrote:
> Marvin Humphrey wrote on 11/1/11 12:36 PM:
>> On Tue, Nov 01, 2011 at 11:37:30AM +0200, goran kent wrote:
>>> I'm pondering about the best approach to handle fresh/stale indexes
>>> while searches are happening (ie, we don't want to disrupt an active
>>> search).
>>
>> Once an IndexSearcher has opened successfully, it no longer needs access
>> to
>> the index dir[1] -- you can actually wipe the index dir and the
>> IndexSearcher
>> will keep functioning indefinitely.  This is achieved by caching open
>> filehandles to all files the IndexSearcher could ever need access to, so
>> that
>> it will still have access even if the files are deleted[2].
>>
>
> fwiw, this is exactly what I do.
>
> For incremental updates, my Searcher wrapper code[1] detects when the index
> has
> changed and re-opens itself.
>
> For full index swap-outs, I just let existing Searchers finish and re-open
> themselves when they realize the underlying index is different. When the
> open
> filehandles eventually close, then the filesystem actually removes the old
> index.
>
> [1] https://metacpan.org/module/SWISH::Prog::Lucy::Searcher
>
> --
> Peter Karman  .  http://peknet.com/  .  peter@peknet.com
>

Re: [lucy-user] Managing fresh/stale indexes

Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 11/1/11 12:36 PM:
> On Tue, Nov 01, 2011 at 11:37:30AM +0200, goran kent wrote:
>> I'm pondering about the best approach to handle fresh/stale indexes
>> while searches are happening (ie, we don't want to disrupt an active
>> search).
> 
> Once an IndexSearcher has opened successfully, it no longer needs access to
> the index dir[1] -- you can actually wipe the index dir and the IndexSearcher
> will keep functioning indefinitely.  This is achieved by caching open
> filehandles to all files the IndexSearcher could ever need access to, so that
> it will still have access even if the files are deleted[2].
>  

fwiw, this is exactly what I do.

For incremental updates, my Searcher wrapper code[1] detects when the index has
changed and re-opens itself.

For full index swap-outs, I just let existing Searchers finish and re-open
themselves when they realize the underlying index is different. When the open
filehandles eventually close, then the filesystem actually removes the old index.

[1] https://metacpan.org/module/SWISH::Prog::Lucy::Searcher

-- 
Peter Karman  .  http://peknet.com/  .  peter@peknet.com

Re: [lucy-user] Managing fresh/stale indexes

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, Nov 01, 2011 at 11:37:30AM +0200, goran kent wrote:
> I'm pondering about the best approach to handle fresh/stale indexes
> while searches are happening (ie, we don't want to disrupt an active
> search).

Once an IndexSearcher has opened successfully, it no longer needs access to
the index dir[1] -- you can actually wipe the index dir and the IndexSearcher
will keep functioning indefinitely.  This is achieved by caching open
filehandles to all files the IndexSearcher could ever need access to, so that
it will still have access even if the files are deleted[2].
 
> Let's say you have index1 which is stale, but actively being searched on.  I
> want to automate the process of switching in a fresh index1.fresh (symlink
> or move).  Let's say index1 is being hammered by active searches: what's the
> safest way to switch in index1.fresh?  Have an intermediary?:
>
> index1   --  a symlink to either:
> 
> index1.a/
> or
> index1.b/

Swapping out a symlink for a new one is not totally safe.  It will not affect
any existing IndexSearchers which are already open, but if the swap happens
while an attempt to open a new IndexSearcher is underway, you will get an
exception.

Lucy has a retry mechanism[3] which fires in case the index is modified and a
file needed by a particular snapshot gets deleted before the opening attempt
completes successfully.  However, that retry code works by looking for a more
recent snapshot file, as measured by the generation number embedded in the
snapshot file's name.  It will give up if it can't find such a newer snapshot
file, and thus it will not attempt to read a fresh new index which suddenly
appears in the location it was trying to open.

Put another way, the retry mechanism is only there to guarantee that opening
an index is never disrupted by updates.  It's not there to support what you
are trying to do.

> (and a control file which indicates the last stale one so the link
> cycles through them correctly -- and perhaps a lockfile which tells
> searchers: hey, hold for 1s while I reassign this symlink; but perhaps
> this is overcomplicating things)

To guard against the possible interruption of an open, you may need something
like that.

I presume you are planning on making live updates to indexes you are searching
with SearchServer/SearchClient/PolySearcher, though, right?  In that case, you
may need a more sophisticated mechanism in order to move the entire cluster
forward without disrupting any searches which are underway (including fetching
docs and highlighting).

Marvin Humphrey

[1] This will change if we ever support non-compound index file formats,
    because then e.g. the Open_In() calls in SegPostingList and SegLexicon's
    constructors would need to operate on real files.  However, there are no
    plans to support the non-compound format, because Lucy uses way, way more
    "files" than Lucene.

[2] On Unix.  On Windows, you can't delete a file that's got a filehandle open
    against it, so the mechanism is slightly different.  Regardless, the
    robustness of an open IndexSearcher is something you can count on across
    all operating systems.

[3] The retry logic is in PolyReader_do_open():
    http://svn.apache.org/viewvc/incubator/lucy/trunk/core/Lucy/Index/PolyReader.c?view=markup#l275