You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Siraj Haider <si...@jobdiva.com> on 2016/12/14 20:14:39 UTC
Warming Indexes
Hi,
We are in the process of moving from lucene 2.9 to 6.3. We are going to use SearcherManager and SearcherFactory classes for the first time and are wondering on how to warm our indexes the first time. Below are the snippets of code, where we create our IndexWriter and SearcherManager. And then we use searcherManager.acquire() to get IndexSearcher. Can somebody please guide us on how to implement search warming? Would we need to warm each IndexSearcher that we get from .acquire()?
IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new File(_indexLocation).toPath()), writer_config);
SearcherManager searcherManager = new SearcherManager(indexWriter, new SearcherFactory());
------
Regards
-Siraj Haider
(212) 306-0154
________________________________
This electronic mail message and any attachments may contain information which is privileged, sensitive and/or otherwise exempt from disclosure under applicable law. The information is intended only for the use of the individual or entity named as the addressee above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution (electronic or otherwise) or forwarding of, or the taking of any action in reliance on, the contents of this transmission is strictly prohibited. If you have received this electronic transmission in error, please notify us by telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
RE: Warming Indexes
Posted by Siraj Haider <si...@jobdiva.com>.
Hi Uwe,
Below is the code that shows how we are opening writer and searchermanager:
LimitTokenCountAnalyzer limit_analyzer = new LimitTokenCountAnalyzer(analyzer, 100000, true);
IndexWriterConfig writer_config = new IndexWriterConfig(limit_analyzer);
writer_config.setRAMBufferSizeMB(320);
writer_config.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
JobDivaIndexWarmer indexWarmer = new JobDivaIndexWarmer();
writer_config.setMergedSegmentWarmer(indexWarmer);
if (_useCompoundIndexFormat==1) writer_config.setUseCompoundFile(true);
else writer_config.setUseCompoundFile(false);
indexWriter = new IndexWriter(fsDir, writer_config);
searcherManager = new SearcherManager(indexWriter, new SearcherFactory());
And then we keep that indexwriter alive and after each batch of document we do the following:
indexWriter.commit();
searcherManager.maybeRefresh();
Should these sequence of events cause the MergedSegmentWarmer to get called?
------
Regards
-Siraj Haider
(212) 306-0154
-----Original Message-----
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Wednesday, December 21, 2016 11:37 AM
To: java-user@lucene.apache.org
Subject: RE: Warming Indexes
Hi,
The warmup only happens if you reopen the searcher using the NRT APIs or searcher manager. If you just index with a single IndexWriter that has no open NRT readers, nothing will happen.
To warmup when the application starts, I'd suggest to use the first IndexSearcher. The warmer in IndexWriter is just there to run queries on new index segments that get live through SearcherManager or other NRT APIs.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Siraj Haider [mailto:siraj@jobdiva.com]
> Sent: Wednesday, December 21, 2016 3:58 PM
> To: java-user@lucene.apache.org
> Subject: RE: Warming Indexes
>
> Thanks Uwe, I implemented the interface and am printing some lines to
> see if the warmup is happening, but I never see those prints in the
> log after some documents are indexed. Another question is how to
> warmup the index when I open the index first time, i.e. when the
> application starts. Should I simply run warmup queries on the acquired
> IndexSearcher from SearcherManager or is there a better way to accomplish that.
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, December 19, 2016 1:29 PM
> To: java-user@lucene.apache.org
> Subject: RE: Warming Indexes
>
> Hi,
>
> Just implement this interface and override the sole abstract method.
> The argument is a LeafReader that you can quickly wrap with an
> IndexSearcher (LeafReader is a subclass of IndexReader, so it's
> standard pattern) and execute one or multiple queries on it. This
> depends on your use case. Do whatever you want with the to-be warmed LeafReader.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Siraj Haider [mailto:siraj@jobdiva.com]
> > Sent: Monday, December 19, 2016 4:10 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: Warming Indexes
> >
> > Thanks for the information Uwe, it was very helpful. Do you have any
> > example code implementing IndexWriter.IndexReaderWarmer class? I am
> > having difficulty finding any examples on internet.
> >
> > ------
> > Regards
> > -Siraj Haider
> > (212) 306-0154
> >
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Wednesday, December 14, 2016 3:34 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: Warming Indexes
> >
> > Hi,
> >
> > How about subclassing SearcherFactory and include warming there?
> >
> https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/Sea
> > r
> > cherFactory.html
> >
> > Refer especially to the last bullet point. As described there to do
> > warming in the background without blocking in the near realtime
> > case, use warmers on
> > IndexWriter.setMergedSegmentWarmer() instead. This has the effect
> > that only new index segments are warmed as needed instead of the
> > whole index on every reopen.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> > > -----Original Message-----
> > > From: Siraj Haider [mailto:siraj@jobdiva.com]
> > > Sent: Wednesday, December 14, 2016 9:15 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Warming Indexes
> > >
> > > Hi,
> > >
> > > We are in the process of moving from lucene 2.9 to 6.3. We are
> > > going to use SearcherManager and SearcherFactory classes for the
> > > first time and are wondering on how to warm our indexes the first time.
> > > Below are the snippets of code, where we create our IndexWriter
> > > and
> > SearcherManager.
> > > And then we use searcherManager.acquire() to get IndexSearcher.
> > > Can somebody please guide us on how to implement search warming?
> > > Would
> > we
> > > need to warm each IndexSearcher that we get from .acquire()?
> > >
> > > IndexWriter indexWriter = new IndexWriter(new
> > > NIOFSDirectory(new File(_indexLocation).toPath()), writer_config);
> > > SearcherManager searcherManager = new
> > SearcherManager(indexWriter,
> > > new SearcherFactory());
> > >
> > >
> > > ------
> > > Regards
> > > -Siraj Haider
> > > (212) 306-0154
> > >
> > >
> > > ________________________________
> > >
> > > This electronic mail message and any attachments may contain
> > > information which is privileged, sensitive and/or otherwise exempt
> > > from disclosure under applicable law. The information is intended
> > > only for the use of the individual or entity named as the addressee above.
> > > If you are not the intended recipient, you are hereby notified
> > > that any disclosure, copying, distribution (electronic or
> > > otherwise) or forwarding of, or the taking of any action in
> > > reliance on, the contents of this transmission is strictly
> > > prohibited. If you have received this electronic transmission in
> > > error, please notify us by telephone, facsimile, or e-mail as
> > > noted above to arrange for the return of
> > any electronic mail or attachments. Thank You.
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ________________________________
> >
> > This electronic mail message and any attachments may contain
> > information which is privileged, sensitive and/or otherwise exempt
> > from disclosure under applicable law. The information is intended
> > only for the use of the individual or entity named as the addressee above.
> > If you are not the intended recipient, you are hereby notified that
> > any disclosure, copying, distribution (electronic or otherwise) or
> > forwarding of, or the taking of any action in reliance on, the
> > contents of this transmission is strictly prohibited. If you have
> > received this electronic transmission in error, please notify us by
> > telephone, facsimile, or e-mail as noted above to arrange for the
> > return of
> any electronic mail or attachments. Thank You.
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain
> information which is privileged, sensitive and/or otherwise exempt
> from disclosure under applicable law. The information is intended only
> for the use of the individual or entity named as the addressee above.
> If you are not the intended recipient, you are hereby notified that
> any disclosure, copying, distribution (electronic or otherwise) or
> forwarding of, or the taking of any action in reliance on, the
> contents of this transmission is strictly prohibited. If you have
> received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
________________________________
This electronic mail message and any attachments may contain information which is privileged, sensitive and/or otherwise exempt from disclosure under applicable law. The information is intended only for the use of the individual or entity named as the addressee above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution (electronic or otherwise) or forwarding of, or the taking of any action in reliance on, the contents of this transmission is strictly prohibited. If you have received this electronic transmission in error, please notify us by telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
RE: Warming Indexes
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
The warmup only happens if you reopen the searcher using the NRT APIs or searcher manager. If you just index with a single IndexWriter that has no open NRT readers, nothing will happen.
To warmup when the application starts, I'd suggest to use the first IndexSearcher. The warmer in IndexWriter is just there to run queries on new index segments that get live through SearcherManager or other NRT APIs.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Siraj Haider [mailto:siraj@jobdiva.com]
> Sent: Wednesday, December 21, 2016 3:58 PM
> To: java-user@lucene.apache.org
> Subject: RE: Warming Indexes
>
> Thanks Uwe, I implemented the interface and am printing some lines to see if
> the warmup is happening, but I never see those prints in the log after some
> documents are indexed. Another question is how to warmup the index when
> I open the index first time, i.e. when the application starts. Should I simply
> run warmup queries on the acquired IndexSearcher from SearcherManager
> or is there a better way to accomplish that.
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, December 19, 2016 1:29 PM
> To: java-user@lucene.apache.org
> Subject: RE: Warming Indexes
>
> Hi,
>
> Just implement this interface and override the sole abstract method. The
> argument is a LeafReader that you can quickly wrap with an IndexSearcher
> (LeafReader is a subclass of IndexReader, so it's standard pattern) and
> execute one or multiple queries on it. This depends on your use case. Do
> whatever you want with the to-be warmed LeafReader.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Siraj Haider [mailto:siraj@jobdiva.com]
> > Sent: Monday, December 19, 2016 4:10 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: Warming Indexes
> >
> > Thanks for the information Uwe, it was very helpful. Do you have any
> > example code implementing IndexWriter.IndexReaderWarmer class? I am
> > having difficulty finding any examples on internet.
> >
> > ------
> > Regards
> > -Siraj Haider
> > (212) 306-0154
> >
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Wednesday, December 14, 2016 3:34 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: Warming Indexes
> >
> > Hi,
> >
> > How about subclassing SearcherFactory and include warming there?
> >
> https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/Sea
> > r
> > cherFactory.html
> >
> > Refer especially to the last bullet point. As described there to do
> > warming in the background without blocking in the near realtime case,
> > use warmers on
> > IndexWriter.setMergedSegmentWarmer() instead. This has the effect that
> > only new index segments are warmed as needed instead of the whole
> > index on every reopen.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> > > -----Original Message-----
> > > From: Siraj Haider [mailto:siraj@jobdiva.com]
> > > Sent: Wednesday, December 14, 2016 9:15 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Warming Indexes
> > >
> > > Hi,
> > >
> > > We are in the process of moving from lucene 2.9 to 6.3. We are going
> > > to use SearcherManager and SearcherFactory classes for the first
> > > time and are wondering on how to warm our indexes the first time.
> > > Below are the snippets of code, where we create our IndexWriter and
> > SearcherManager.
> > > And then we use searcherManager.acquire() to get IndexSearcher. Can
> > > somebody please guide us on how to implement search warming? Would
> > we
> > > need to warm each IndexSearcher that we get from .acquire()?
> > >
> > > IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> > > File(_indexLocation).toPath()), writer_config);
> > > SearcherManager searcherManager = new
> > SearcherManager(indexWriter,
> > > new SearcherFactory());
> > >
> > >
> > > ------
> > > Regards
> > > -Siraj Haider
> > > (212) 306-0154
> > >
> > >
> > > ________________________________
> > >
> > > This electronic mail message and any attachments may contain
> > > information which is privileged, sensitive and/or otherwise exempt
> > > from disclosure under applicable law. The information is intended
> > > only for the use of the individual or entity named as the addressee above.
> > > If you are not the intended recipient, you are hereby notified that
> > > any disclosure, copying, distribution (electronic or otherwise) or
> > > forwarding of, or the taking of any action in reliance on, the
> > > contents of this transmission is strictly prohibited. If you have
> > > received this electronic transmission in error, please notify us by
> > > telephone, facsimile, or e-mail as noted above to arrange for the
> > > return of
> > any electronic mail or attachments. Thank You.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ________________________________
> >
> > This electronic mail message and any attachments may contain
> > information which is privileged, sensitive and/or otherwise exempt
> > from disclosure under applicable law. The information is intended only
> > for the use of the individual or entity named as the addressee above.
> > If you are not the intended recipient, you are hereby notified that
> > any disclosure, copying, distribution (electronic or otherwise) or
> > forwarding of, or the taking of any action in reliance on, the
> > contents of this transmission is strictly prohibited. If you have
> > received this electronic transmission in error, please notify us by
> > telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain information
> which is privileged, sensitive and/or otherwise exempt from disclosure under
> applicable law. The information is intended only for the use of the individual
> or entity named as the addressee above. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying, distribution
> (electronic or otherwise) or forwarding of, or the taking of any action in
> reliance on, the contents of this transmission is strictly prohibited. If you
> have received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Warming Indexes
Posted by Siraj Haider <si...@jobdiva.com>.
Thanks Uwe, I implemented the interface and am printing some lines to see if the warmup is happening, but I never see those prints in the log after some documents are indexed. Another question is how to warmup the index when I open the index first time, i.e. when the application starts. Should I simply run warmup queries on the acquired IndexSearcher from SearcherManager or is there a better way to accomplish that.
------
Regards
-Siraj Haider
(212) 306-0154
-----Original Message-----
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Monday, December 19, 2016 1:29 PM
To: java-user@lucene.apache.org
Subject: RE: Warming Indexes
Hi,
Just implement this interface and override the sole abstract method. The argument is a LeafReader that you can quickly wrap with an IndexSearcher (LeafReader is a subclass of IndexReader, so it's standard pattern) and execute one or multiple queries on it. This depends on your use case. Do whatever you want with the to-be warmed LeafReader.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Siraj Haider [mailto:siraj@jobdiva.com]
> Sent: Monday, December 19, 2016 4:10 PM
> To: java-user@lucene.apache.org
> Subject: RE: Warming Indexes
>
> Thanks for the information Uwe, it was very helpful. Do you have any
> example code implementing IndexWriter.IndexReaderWarmer class? I am
> having difficulty finding any examples on internet.
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Wednesday, December 14, 2016 3:34 PM
> To: java-user@lucene.apache.org
> Subject: RE: Warming Indexes
>
> Hi,
>
> How about subclassing SearcherFactory and include warming there?
> https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/Sea
> r
> cherFactory.html
>
> Refer especially to the last bullet point. As described there to do
> warming in the background without blocking in the near realtime case,
> use warmers on
> IndexWriter.setMergedSegmentWarmer() instead. This has the effect that
> only new index segments are warmed as needed instead of the whole
> index on every reopen.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Siraj Haider [mailto:siraj@jobdiva.com]
> > Sent: Wednesday, December 14, 2016 9:15 PM
> > To: java-user@lucene.apache.org
> > Subject: Warming Indexes
> >
> > Hi,
> >
> > We are in the process of moving from lucene 2.9 to 6.3. We are going
> > to use SearcherManager and SearcherFactory classes for the first
> > time and are wondering on how to warm our indexes the first time.
> > Below are the snippets of code, where we create our IndexWriter and
> SearcherManager.
> > And then we use searcherManager.acquire() to get IndexSearcher. Can
> > somebody please guide us on how to implement search warming? Would
> we
> > need to warm each IndexSearcher that we get from .acquire()?
> >
> > IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> > File(_indexLocation).toPath()), writer_config);
> > SearcherManager searcherManager = new
> SearcherManager(indexWriter,
> > new SearcherFactory());
> >
> >
> > ------
> > Regards
> > -Siraj Haider
> > (212) 306-0154
> >
> >
> > ________________________________
> >
> > This electronic mail message and any attachments may contain
> > information which is privileged, sensitive and/or otherwise exempt
> > from disclosure under applicable law. The information is intended
> > only for the use of the individual or entity named as the addressee above.
> > If you are not the intended recipient, you are hereby notified that
> > any disclosure, copying, distribution (electronic or otherwise) or
> > forwarding of, or the taking of any action in reliance on, the
> > contents of this transmission is strictly prohibited. If you have
> > received this electronic transmission in error, please notify us by
> > telephone, facsimile, or e-mail as noted above to arrange for the
> > return of
> any electronic mail or attachments. Thank You.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain
> information which is privileged, sensitive and/or otherwise exempt
> from disclosure under applicable law. The information is intended only
> for the use of the individual or entity named as the addressee above.
> If you are not the intended recipient, you are hereby notified that
> any disclosure, copying, distribution (electronic or otherwise) or
> forwarding of, or the taking of any action in reliance on, the
> contents of this transmission is strictly prohibited. If you have
> received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
________________________________
This electronic mail message and any attachments may contain information which is privileged, sensitive and/or otherwise exempt from disclosure under applicable law. The information is intended only for the use of the individual or entity named as the addressee above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution (electronic or otherwise) or forwarding of, or the taking of any action in reliance on, the contents of this transmission is strictly prohibited. If you have received this electronic transmission in error, please notify us by telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Warming Indexes
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
Just implement this interface and override the sole abstract method. The argument is a LeafReader that you can quickly wrap with an IndexSearcher (LeafReader is a subclass of IndexReader, so it's standard pattern) and execute one or multiple queries on it. This depends on your use case. Do whatever you want with the to-be warmed LeafReader.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Siraj Haider [mailto:siraj@jobdiva.com]
> Sent: Monday, December 19, 2016 4:10 PM
> To: java-user@lucene.apache.org
> Subject: RE: Warming Indexes
>
> Thanks for the information Uwe, it was very helpful. Do you have any
> example code implementing IndexWriter.IndexReaderWarmer class? I am
> having difficulty finding any examples on internet.
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Wednesday, December 14, 2016 3:34 PM
> To: java-user@lucene.apache.org
> Subject: RE: Warming Indexes
>
> Hi,
>
> How about subclassing SearcherFactory and include warming there?
> https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/Sear
> cherFactory.html
>
> Refer especially to the last bullet point. As described there to do warming in
> the background without blocking in the near realtime case, use warmers on
> IndexWriter.setMergedSegmentWarmer() instead. This has the effect that
> only new index segments are warmed as needed instead of the whole index
> on every reopen.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Siraj Haider [mailto:siraj@jobdiva.com]
> > Sent: Wednesday, December 14, 2016 9:15 PM
> > To: java-user@lucene.apache.org
> > Subject: Warming Indexes
> >
> > Hi,
> >
> > We are in the process of moving from lucene 2.9 to 6.3. We are going
> > to use SearcherManager and SearcherFactory classes for the first time
> > and are wondering on how to warm our indexes the first time. Below are
> > the snippets of code, where we create our IndexWriter and
> SearcherManager.
> > And then we use searcherManager.acquire() to get IndexSearcher. Can
> > somebody please guide us on how to implement search warming? Would
> we
> > need to warm each IndexSearcher that we get from .acquire()?
> >
> > IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> > File(_indexLocation).toPath()), writer_config);
> > SearcherManager searcherManager = new
> SearcherManager(indexWriter,
> > new SearcherFactory());
> >
> >
> > ------
> > Regards
> > -Siraj Haider
> > (212) 306-0154
> >
> >
> > ________________________________
> >
> > This electronic mail message and any attachments may contain
> > information which is privileged, sensitive and/or otherwise exempt
> > from disclosure under applicable law. The information is intended only
> > for the use of the individual or entity named as the addressee above.
> > If you are not the intended recipient, you are hereby notified that
> > any disclosure, copying, distribution (electronic or otherwise) or
> > forwarding of, or the taking of any action in reliance on, the
> > contents of this transmission is strictly prohibited. If you have
> > received this electronic transmission in error, please notify us by
> > telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain information
> which is privileged, sensitive and/or otherwise exempt from disclosure under
> applicable law. The information is intended only for the use of the individual
> or entity named as the addressee above. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying, distribution
> (electronic or otherwise) or forwarding of, or the taking of any action in
> reliance on, the contents of this transmission is strictly prohibited. If you
> have received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Warming Indexes
Posted by Siraj Haider <si...@jobdiva.com>.
Thanks for the information Uwe, it was very helpful. Do you have any example code implementing IndexWriter.IndexReaderWarmer class? I am having difficulty finding any examples on internet.
------
Regards
-Siraj Haider
(212) 306-0154
-----Original Message-----
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Wednesday, December 14, 2016 3:34 PM
To: java-user@lucene.apache.org
Subject: RE: Warming Indexes
Hi,
How about subclassing SearcherFactory and include warming there?
https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/SearcherFactory.html
Refer especially to the last bullet point. As described there to do warming in the background without blocking in the near realtime case, use warmers on IndexWriter.setMergedSegmentWarmer() instead. This has the effect that only new index segments are warmed as needed instead of the whole index on every reopen.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Siraj Haider [mailto:siraj@jobdiva.com]
> Sent: Wednesday, December 14, 2016 9:15 PM
> To: java-user@lucene.apache.org
> Subject: Warming Indexes
>
> Hi,
>
> We are in the process of moving from lucene 2.9 to 6.3. We are going
> to use SearcherManager and SearcherFactory classes for the first time
> and are wondering on how to warm our indexes the first time. Below are
> the snippets of code, where we create our IndexWriter and SearcherManager.
> And then we use searcherManager.acquire() to get IndexSearcher. Can
> somebody please guide us on how to implement search warming? Would we
> need to warm each IndexSearcher that we get from .acquire()?
>
> IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> File(_indexLocation).toPath()), writer_config);
> SearcherManager searcherManager = new SearcherManager(indexWriter,
> new SearcherFactory());
>
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain
> information which is privileged, sensitive and/or otherwise exempt
> from disclosure under applicable law. The information is intended only
> for the use of the individual or entity named as the addressee above.
> If you are not the intended recipient, you are hereby notified that
> any disclosure, copying, distribution (electronic or otherwise) or
> forwarding of, or the taking of any action in reliance on, the
> contents of this transmission is strictly prohibited. If you have
> received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
________________________________
This electronic mail message and any attachments may contain information which is privileged, sensitive and/or otherwise exempt from disclosure under applicable law. The information is intended only for the use of the individual or entity named as the addressee above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution (electronic or otherwise) or forwarding of, or the taking of any action in reliance on, the contents of this transmission is strictly prohibited. If you have received this electronic transmission in error, please notify us by telephone, facsimile, or e-mail as noted above to arrange for the return of any electronic mail or attachments. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Warming Indexes
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
How about subclassing SearcherFactory and include warming there?
https://lucene.apache.org/core/6_3_0/core/org/apache/lucene/search/SearcherFactory.html
Refer especially to the last bullet point. As described there to do warming in the background without blocking in the near realtime case, use warmers on IndexWriter.setMergedSegmentWarmer() instead. This has the effect that only new index segments are warmed as needed instead of the whole index on every reopen.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Siraj Haider [mailto:siraj@jobdiva.com]
> Sent: Wednesday, December 14, 2016 9:15 PM
> To: java-user@lucene.apache.org
> Subject: Warming Indexes
>
> Hi,
>
> We are in the process of moving from lucene 2.9 to 6.3. We are going to use
> SearcherManager and SearcherFactory classes for the first time and are
> wondering on how to warm our indexes the first time. Below are the
> snippets of code, where we create our IndexWriter and SearcherManager.
> And then we use searcherManager.acquire() to get IndexSearcher. Can
> somebody please guide us on how to implement search warming? Would we
> need to warm each IndexSearcher that we get from .acquire()?
>
> IndexWriter indexWriter = new IndexWriter(new NIOFSDirectory(new
> File(_indexLocation).toPath()), writer_config);
> SearcherManager searcherManager = new SearcherManager(indexWriter,
> new SearcherFactory());
>
>
> ------
> Regards
> -Siraj Haider
> (212) 306-0154
>
>
> ________________________________
>
> This electronic mail message and any attachments may contain information
> which is privileged, sensitive and/or otherwise exempt from disclosure under
> applicable law. The information is intended only for the use of the individual
> or entity named as the addressee above. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying, distribution
> (electronic or otherwise) or forwarding of, or the taking of any action in
> reliance on, the contents of this transmission is strictly prohibited. If you
> have received this electronic transmission in error, please notify us by
> telephone, facsimile, or e-mail as noted above to arrange for the return of
> any electronic mail or attachments. Thank You.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org