You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Doug Cutting <cu...@apache.org> on 2006/01/04 19:30:01 UTC

Re: indexreader refresh

Amol Bhutada wrote:
> If I have a reader and searcher on a indexdata folder and another 
> indexwriter writing documents to the same indexdata folder, do I need to 
> close existing reader and searcher and create new so that newly indexed 
> data comes into search effect?

[ moved from user to dev list]

This is a frequent request.  While opening an all-new IndexReader is 
effective, it is not always efficient.  It might be nice to support a 
more efficient means of re-opening an index.

Perhaps we should add a few new IndexReader methods, as follows:

/** If <code>reader</code>'s index has not been changed, return
   * <code>reader</code>, otherwise return a new {@link IndexReader}
   * reading the new latest of the index
   */
public static IndexReader open(IndexReader reader) {
   if (isCurrent()) {
     // unchanged: return existing
     return reader;
   }

   // try to incrementally create new reader
   IndexReader result = reader.reopen(reader);
   if (result != null) {
     return result;
   }

   // punt, opening an entirely new reader
   return IndexReader.open(reader.directory());
}

/** Return a new IndexReader reading the current state
   * of the index, re-using reader's resources, or null if this
   * is not possible.
   */
protected IndexReader reopen(IndexReader reader) {
   return null;
}

Then we can add implementations of reopen to SegmentReader and 
MultiReader that attempt to re-use the existing, already opened 
segments.  This should mostly be simple, but there are a few tricky 
issues, like detecting whether an already-open segment has had 
deletions, and deciding when to close obsolete segments.

Does this sound like it would make a good addition?  Does someone want 
to volunteer to implement it?

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: indexreader refresh

Posted by Doug Cutting <cu...@apache.org>.
Yes, that's a good start.  Your patch does not handle deletions 
correctly.  If a segment has had deletions since it was opened then its 
deletions file needs to be re-read.  I also think returning a new 
IndexReader is preferable to modifying one, since an IndexReader is 
often used as a cache key, and caches should be invalidated when an 
IndexReader is re-opened.

Robert Engels wrote:
> I proposed and posted a patch for this long ago. Only thing missing would be
> some sort of reference courting for segments (rather than the 'stayopen'
> flag).
> 
>   /**
>    * reopens the IndexReader, possibly reusing the segments for greater
> efficiency. The original IndexReader instance
>    * is closed, and the reference is no longer valid
>    *
>    * @return the new IndexReader
>    */
>   public IndexReader reopen() throws IOException {
>       if(!(this instanceof MultiReader))
>           return IndexReader.open(directory);
> 
>       MultiReader mr = (MultiReader) this;
> 
>       final IndexReader[] oldreaders = mr.getReaders();
>       final boolean[] stayopen = new boolean[oldreaders.length];
> 
>       synchronized (directory) {			  // in- & inter-process sync
>           return (IndexReader)new Lock.With(
>               directory.makeLock(IndexWriter.COMMIT_LOCK_NAME),
>               IndexWriter.COMMIT_LOCK_TIMEOUT) {
>               public Object doBody() throws IOException {
>                 SegmentInfos infos = new SegmentInfos();
>                 infos.read(directory);
>                 if (infos.size() == 1) {		  // index is optimized
>                   return new SegmentReader(infos, infos.info(0),
> closeDirectory);
>                 } else {
>                   IndexReader[] readers = new IndexReader[infos.size()];
>                   for (int i = 0; i < infos.size(); i++) {
>                       for(int j=0;j<oldreaders.length;j++) {
>                           SegmentReader sr = (SegmentReader) oldreaders[j];
>                           if(sr.si.name.equals(infos.info(i).name)) {
>                               readers[i]=sr;
>                               stayopen[j]=true;
>                           }
>                       }
>                       if(readers[i]==null)
>                           readers[i] = new SegmentReader(infos.info(i));
>                   }
> 
>                   for(int i=0;i<stayopen.length;i++)
>                       if(!stayopen[i])
>                           oldreaders[i].close();
> 
>                   return new MultiReader(directory, infos, closeDirectory,
> readers);
>                 }
>               }
>             }.run();
>         }
>   }
> 
> -----Original Message-----
> From: Doug Cutting [mailto:cutting@apache.org]
> Sent: Wednesday, January 04, 2006 12:30 PM
> To: java-dev@lucene.apache.org
> Subject: Re: indexreader refresh
> 
> 
> Amol Bhutada wrote:
> 
>>If I have a reader and searcher on a indexdata folder and another
>>indexwriter writing documents to the same indexdata folder, do I need to
>>close existing reader and searcher and create new so that newly indexed
>>data comes into search effect?
> 
> 
> [ moved from user to dev list]
> 
> This is a frequent request.  While opening an all-new IndexReader is
> effective, it is not always efficient.  It might be nice to support a
> more efficient means of re-opening an index.
> 
> Perhaps we should add a few new IndexReader methods, as follows:
> 
> /** If <code>reader</code>'s index has not been changed, return
>    * <code>reader</code>, otherwise return a new {@link IndexReader}
>    * reading the new latest of the index
>    */
> public static IndexReader open(IndexReader reader) {
>    if (isCurrent()) {
>      // unchanged: return existing
>      return reader;
>    }
> 
>    // try to incrementally create new reader
>    IndexReader result = reader.reopen(reader);
>    if (result != null) {
>      return result;
>    }
> 
>    // punt, opening an entirely new reader
>    return IndexReader.open(reader.directory());
> }
> 
> /** Return a new IndexReader reading the current state
>    * of the index, re-using reader's resources, or null if this
>    * is not possible.
>    */
> protected IndexReader reopen(IndexReader reader) {
>    return null;
> }
> 
> Then we can add implementations of reopen to SegmentReader and
> MultiReader that attempt to re-use the existing, already opened
> segments.  This should mostly be simple, but there are a few tricky
> issues, like detecting whether an already-open segment has had
> deletions, and deciding when to close obsolete segments.
> 
> Does this sound like it would make a good addition?  Does someone want
> to volunteer to implement it?
> 
> Doug
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


RE: indexreader refresh

Posted by Robert Engels <re...@ix.netcom.com>.
I proposed and posted a patch for this long ago. Only thing missing would be
some sort of reference courting for segments (rather than the 'stayopen'
flag).

  /**
   * reopens the IndexReader, possibly reusing the segments for greater
efficiency. The original IndexReader instance
   * is closed, and the reference is no longer valid
   *
   * @return the new IndexReader
   */
  public IndexReader reopen() throws IOException {
      if(!(this instanceof MultiReader))
          return IndexReader.open(directory);

      MultiReader mr = (MultiReader) this;

      final IndexReader[] oldreaders = mr.getReaders();
      final boolean[] stayopen = new boolean[oldreaders.length];

      synchronized (directory) {			  // in- & inter-process sync
          return (IndexReader)new Lock.With(
              directory.makeLock(IndexWriter.COMMIT_LOCK_NAME),
              IndexWriter.COMMIT_LOCK_TIMEOUT) {
              public Object doBody() throws IOException {
                SegmentInfos infos = new SegmentInfos();
                infos.read(directory);
                if (infos.size() == 1) {		  // index is optimized
                  return new SegmentReader(infos, infos.info(0),
closeDirectory);
                } else {
                  IndexReader[] readers = new IndexReader[infos.size()];
                  for (int i = 0; i < infos.size(); i++) {
                      for(int j=0;j<oldreaders.length;j++) {
                          SegmentReader sr = (SegmentReader) oldreaders[j];
                          if(sr.si.name.equals(infos.info(i).name)) {
                              readers[i]=sr;
                              stayopen[j]=true;
                          }
                      }
                      if(readers[i]==null)
                          readers[i] = new SegmentReader(infos.info(i));
                  }

                  for(int i=0;i<stayopen.length;i++)
                      if(!stayopen[i])
                          oldreaders[i].close();

                  return new MultiReader(directory, infos, closeDirectory,
readers);
                }
              }
            }.run();
        }
  }

-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Wednesday, January 04, 2006 12:30 PM
To: java-dev@lucene.apache.org
Subject: Re: indexreader refresh


Amol Bhutada wrote:
> If I have a reader and searcher on a indexdata folder and another
> indexwriter writing documents to the same indexdata folder, do I need to
> close existing reader and searcher and create new so that newly indexed
> data comes into search effect?

[ moved from user to dev list]

This is a frequent request.  While opening an all-new IndexReader is
effective, it is not always efficient.  It might be nice to support a
more efficient means of re-opening an index.

Perhaps we should add a few new IndexReader methods, as follows:

/** If <code>reader</code>'s index has not been changed, return
   * <code>reader</code>, otherwise return a new {@link IndexReader}
   * reading the new latest of the index
   */
public static IndexReader open(IndexReader reader) {
   if (isCurrent()) {
     // unchanged: return existing
     return reader;
   }

   // try to incrementally create new reader
   IndexReader result = reader.reopen(reader);
   if (result != null) {
     return result;
   }

   // punt, opening an entirely new reader
   return IndexReader.open(reader.directory());
}

/** Return a new IndexReader reading the current state
   * of the index, re-using reader's resources, or null if this
   * is not possible.
   */
protected IndexReader reopen(IndexReader reader) {
   return null;
}

Then we can add implementations of reopen to SegmentReader and
MultiReader that attempt to re-use the existing, already opened
segments.  This should mostly be simple, but there are a few tricky
issues, like detecting whether an already-open segment has had
deletions, and deciding when to close obsolete segments.

Does this sound like it would make a good addition?  Does someone want
to volunteer to implement it?

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org