You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michael McCandless <lu...@mikemccandless.com> on 2010/02/27 11:36:54 UTC
Re: custom FieldCache cost too much time. how can I preload the the
custom fieldCache when new segment exits!
How are you opening a new reader?
If it's a near-real-time reader (IndexWriter.getReader), or you use
IndexReader.reopen, it should only be the newly created segments that
have to generate the field cache entry, which most of the time should
be fast.
If you are already using those APIs and its still not fast enough,
then you should just warm the reader before using it (additionally,
for a near-real-time reader you should warm newly merged segments by
installing a mergedSegmentWarmer on the writer).
Mike
On Sat, Feb 27, 2010 at 3:35 AM, luocanrao <lu...@sohu.com> wrote:
> custom FieldCache cost too much time.
> So every first time,reopen the new reader ,it interfere the performance of
> search
> I hope someone can tell me,how can I preload the the custom fieldCache when
> new segment exits!
> Thanks again!
>
> here is source , In FieldComparator.setNextReader method
> ((C2CFieldManager)fieldManager).lCommID =
> FieldCache.DEFAULT.getLongs(reader, "1",new LongParser(){
> public long parseLong(String documentIDStr)
> {
> documentIDStr =
> documentIDStr.substring(16);
> long documentID =
> Long.parseLong(documentIDStr,16);
> return documentID;
> }
>
> });
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: custom FieldCache cost too much time. how can I preload the the
custom fieldCache when new segment exits!
Posted by Michael McCandless <lu...@mikemccandless.com>.
If you look at the javadocs for IndexWriter it explains how to do it.
You just provide a class that implements the warm method, and inside
that method you do whatever app specific things you need to do to the
provided IndexReader to warm it.
Note that the SearcherManager class from LIA2 handles setting up the
MergedSegmentWarmer, if you implement the warm method.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
答复: custom FieldCache cost too much time. how can I preload the the custom fieldCache when new segment exits!
Posted by luocanrao <lu...@sohu.com>.
Ps:in our evrioment
a document have more than ten field,in a short time,may be have many update.
installing a mergedSegmentWarmer on the writer,can you give me a small case,
thanks very much!
-----邮件原件-----
发件人: luocanrao [mailto:luocan19826164@sohu.com]
发送时间: 2010年2月27日 19:09
收件人: java-user@lucene.apache.org
主题: 答复: custom FieldCache cost too much time. how can I preload the the custom fieldCache when new segment exits!
I set merge factor 4, every five minute I reopen the reader.
yes most of the time is very fast. But sometimes it is very slow.
For example,when start the program,the first query cosume 10s!
So when newly created segment generated,the query cosume more than 1s.
Our performance is key point.
Sorry ,my English is not good!
I hope I can preload custom fieldCache in a another thread,not the query thread.
So I will not have performace issue.
-----邮件原件-----
发件人: Michael McCandless [mailto:lucene@mikemccandless.com]
发送时间: 2010年2月27日 18:37
收件人: java-user@lucene.apache.org
主题: Re: custom FieldCache cost too much time. how can I preload the the custom fieldCache when new segment exits!
How are you opening a new reader?
If it's a near-real-time reader (IndexWriter.getReader), or you use
IndexReader.reopen, it should only be the newly created segments that
have to generate the field cache entry, which most of the time should
be fast.
If you are already using those APIs and its still not fast enough,
then you should just warm the reader before using it (additionally,
for a near-real-time reader you should warm newly merged segments by
installing a mergedSegmentWarmer on the writer).
Mike
On Sat, Feb 27, 2010 at 3:35 AM, luocanrao <lu...@sohu.com> wrote:
> custom FieldCache cost too much time.
> So every first time,reopen the new reader ,it interfere the performance of
> search
> I hope someone can tell me,how can I preload the the custom fieldCache when
> new segment exits!
> Thanks again!
>
> here is source , In FieldComparator.setNextReader method
> ((C2CFieldManager)fieldManager).lCommID =
> FieldCache.DEFAULT.getLongs(reader, "1",new LongParser(){
> public long parseLong(String documentIDStr)
> {
> documentIDStr =
> documentIDStr.substring(16);
> long documentID =
> Long.parseLong(documentIDStr,16);
> return documentID;
> }
>
> });
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
答复: custom FieldCache cost too much time. how can I preload the the custom fieldCache when new segment exits!
Posted by luocanrao <lu...@sohu.com>.
I set merge factor 4, every five minute I reopen the reader.
yes most of the time is very fast. But sometimes it is very slow.
For example,when start the program,the first query cosume 10s!
So when newly created segment generated,the query cosume more than 1s.
Our performance is key point.
Sorry ,my English is not good!
I hope I can preload custom fieldCache in a another thread,not the query thread.
So I will not have performace issue.
-----邮件原件-----
发件人: Michael McCandless [mailto:lucene@mikemccandless.com]
发送时间: 2010年2月27日 18:37
收件人: java-user@lucene.apache.org
主题: Re: custom FieldCache cost too much time. how can I preload the the custom fieldCache when new segment exits!
How are you opening a new reader?
If it's a near-real-time reader (IndexWriter.getReader), or you use
IndexReader.reopen, it should only be the newly created segments that
have to generate the field cache entry, which most of the time should
be fast.
If you are already using those APIs and its still not fast enough,
then you should just warm the reader before using it (additionally,
for a near-real-time reader you should warm newly merged segments by
installing a mergedSegmentWarmer on the writer).
Mike
On Sat, Feb 27, 2010 at 3:35 AM, luocanrao <lu...@sohu.com> wrote:
> custom FieldCache cost too much time.
> So every first time,reopen the new reader ,it interfere the performance of
> search
> I hope someone can tell me,how can I preload the the custom fieldCache when
> new segment exits!
> Thanks again!
>
> here is source , In FieldComparator.setNextReader method
> ((C2CFieldManager)fieldManager).lCommID =
> FieldCache.DEFAULT.getLongs(reader, "1",new LongParser(){
> public long parseLong(String documentIDStr)
> {
> documentIDStr =
> documentIDStr.substring(16);
> long documentID =
> Long.parseLong(documentIDStr,16);
> return documentID;
> }
>
> });
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org