Re-load Suggester question

[Brent Montrose <br...@gmail.com>]

Hello

I've recently implemented a FuzzySuggester and an AnalyingInFixSuggester
with an web app.  It works great, and performs well.

I'd like to be able to reload or re-build the underlying indexes with an
updated dictionary periodically throughout the day without having to
restart the app.

I've tried calling the load() method on the FuzzySuggester and the build()
method on the AnalyzingInFixSuggester on 'built' suggesters, but based on
heap memory usage, it appears as though the indexes are not being replaced.
 In other words, when one the suggesters is initially constructed and
built, I see heap memory usage go up by approximately 20 MB.  When, I
invoked the build() or load() method a second time, heap memory goes up by
another 20MB.

A code snippet is below:

     StandardAnalyzer sa = new StandardAnalyzer(Version.LUCENE_44);
     File tempDir = new File("C:/index/infixsuggestor/");
     AnalyzingInfixSuggester suggester = new
AnalyzingInfixSuggester(Version.LUCENE_44, tempDir, sa, sa, 3);
    FileDictionary fDictionary = new FileDictionary(new
FileReader("dictionary.txt") );
    suggester.build(fDictionary);

time passes....the dictionary.txt file has been updated, need to do a
complete reload of it.

     suggester.build(fDictionary);

is this correct?  Should a 'new' suggester be created instead, and the old
one discarded?

Thanks

Re: Re-load Suggester question

[Michael McCandless <lu...@mikemccandless.com>]

In general, the build() method fully replaces all internal suggester
state every time you call it.

Ie, a whole new FST is built, or a whole new index is created
(AnalyzingInfixSuggester).

The build() process is not incremental, although with
AnalyzingInfixSuggester this is in principle easy to do (it's just
using a "normal" Lucene index under the hood).

I think in general we need to improve the suggester APIs, somehow.
E.g. instead of "inverted" iterator API (build) we should have
something more like an IndexWriter, where you add your suggestions to
it, and then you call .finish() or something to finish building it.

Mike McCandless

http://blog.mikemccandless.com


On Wed, Aug 7, 2013 at 4:59 PM, Brent Montrose <br...@gmail.com> wrote:
> Hello
>
> I've recently implemented a FuzzySuggester and an AnalyingInFixSuggester
> with an web app.  It works great, and performs well.
>
> I'd like to be able to reload or re-build the underlying indexes with an
> updated dictionary periodically throughout the day without having to
> restart the app.
>
> I've tried calling the load() method on the FuzzySuggester and the build()
> method on the AnalyzingInFixSuggester on 'built' suggesters, but based on
> heap memory usage, it appears as though the indexes are not being replaced.
>  In other words, when one the suggesters is initially constructed and
> built, I see heap memory usage go up by approximately 20 MB.  When, I
> invoked the build() or load() method a second time, heap memory goes up by
> another 20MB.
>
> A code snippet is below:
>
>      StandardAnalyzer sa = new StandardAnalyzer(Version.LUCENE_44);
>      File tempDir = new File("C:/index/infixsuggestor/");
>      AnalyzingInfixSuggester suggester = new
> AnalyzingInfixSuggester(Version.LUCENE_44, tempDir, sa, sa, 3);
>     FileDictionary fDictionary = new FileDictionary(new
> FileReader("dictionary.txt") );
>     suggester.build(fDictionary);
>
> time passes....the dictionary.txt file has been updated, need to do a
> complete reload of it.
>
>      suggester.build(fDictionary);
>
> is this correct?  Should a 'new' suggester be created instead, and the old
> one discarded?
>
> Thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org