You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ar...@equifax.com on 2008/08/11 23:05:24 UTC

Field sizes: maxFieldLength

Hi all -

I know in advance that each of the fileds i index doesnt go more than 
1000, Can i gain any performance improvement while writing the index by 
limiting the maxFieldLength to 200? 

tx
Regards, 
Aravind R Yarram 
This message contains information from Equifax Inc. which may be confidential and privileged.  If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited.  If you have received this transmission in error, please notify by e-mail postmaster@equifax.com.

Re: Field sizes: maxFieldLength

Posted by Chris Hostetter <ho...@fucit.org>.
: In-Reply-To: <48...@gmail.com>
: Subject: Field sizes: maxFieldLength

http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists

When starting a new discussion on a mailing list, please do not reply to 
an existing message, instead start a fresh email.  Even if you change the 
subject line of your email, other mail headers still track which thread 
you replied to and your question is "hidden" in that thread and gets less 
attention.   It makes following discussions in the mailing list archives 
particularly difficult.
See Also:  http://en.wikipedia.org/wiki/Thread_hijacking




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Field sizes: maxFieldLength

Posted by Mark Miller <ma...@gmail.com>.
No. Its just a simple number of terms taken from the document limiter used
on the assumption that text that long stops adding value, or starts adding
noise, or starts accelerating into diminishing returns at that point. No
optimizations are used based on it.

On Mon, Aug 11, 2008 at 5:20 PM, <Ar...@equifax.com> wrote:

> tx for the response but i think i didnt make my question clear...
>
> If i am indexing a filed that can at the max contain 1000 fileds, does it
> help in improving performance if i let Lucene know IN ADVANCE about 1000?
>
>
>
>
>
>
> Mark Miller <ma...@gmail.com>
> 08/11/2008 05:13 PM
> Please respond to
> java-user@lucene.apache.org
>
>
> To
> java-user@lucene.apache.org
> cc
>
> Subject
> Re: Field sizes: maxFieldLength
>
>
>
>
>
>
> Aravind.Yarram@equifax.com wrote:
> > Hi all -
> >
> > I know in advance that each of the fileds i index doesnt go more than
> > 1000, Can i gain any performance improvement while writing the index by
> > limiting the maxFieldLength to 200?
> >
> > tx
> > Regards,
> > Aravind R Yarram
> > This message contains information from Equifax Inc. which may be
> confidential and privileged.  If you are not an intended recipient, please
> refrain from any disclosure, copying, distribution or use of this
> information and note that such actions are prohibited.  If you have
> received this transmission in error, please notify by e-mail
> postmaster@equifax.com.
> >
> >
> Its 10000. Sure, if you have a lot of docs between 200 and 10000,
> indexing less will be faster. But you will only be able to search on
> those first 200 tokens for any doc longer.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> This message contains information from Equifax Inc. which may be
> confidential and privileged.  If you are not an intended recipient, please
> refrain from any disclosure, copying, distribution or use of this
> information and note that such actions are prohibited.  If you have received
> this transmission in error, please notify by e-mail postmaster@equifax.com
> .
>

Re: Field sizes: maxFieldLength

Posted by Mark Miller <ma...@gmail.com>.
The gist is: it doesn't help. That simply cuts long documents off at the 
knees on the assumption that its long enough already, that more won't 
add much value (and may add noise?). Its not used for any sort of 
optimizations...its a straight, just use the first n tokens from a document.

Aravind.Yarram@equifax.com wrote:
> tx for the response but i think i didnt make my question clear...
>
> If i am indexing a filed that can at the max contain 1000 fileds, does it 
> help in improving performance if i let Lucene know IN ADVANCE about 1000?
>
>
>
>
>
>
> Mark Miller <ma...@gmail.com> 
> 08/11/2008 05:13 PM
> Please respond to
> java-user@lucene.apache.org
>
>
> To
> java-user@lucene.apache.org
> cc
>
> Subject
> Re: Field sizes: maxFieldLength
>
>
>
>
>
>
> Aravind.Yarram@equifax.com wrote:
>   
>> Hi all -
>>
>> I know in advance that each of the fileds i index doesnt go more than 
>> 1000, Can i gain any performance improvement while writing the index by 
>> limiting the maxFieldLength to 200? 
>>
>> tx
>> Regards, 
>> Aravind R Yarram 
>> This message contains information from Equifax Inc. which may be 
>>     
> confidential and privileged.  If you are not an intended recipient, please 
> refrain from any disclosure, copying, distribution or use of this 
> information and note that such actions are prohibited.  If you have 
> received this transmission in error, please notify by e-mail 
> postmaster@equifax.com.
>   
>>     
> Its 10000. Sure, if you have a lot of docs between 200 and 10000, 
> indexing less will be faster. But you will only be able to search on 
> those first 200 tokens for any doc longer.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> This message contains information from Equifax Inc. which may be confidential and privileged.  If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited.  If you have received this transmission in error, please notify by e-mail postmaster@equifax.com.
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Field sizes: maxFieldLength

Posted by Ar...@equifax.com.
tx for the response but i think i didnt make my question clear...

If i am indexing a filed that can at the max contain 1000 fileds, does it 
help in improving performance if i let Lucene know IN ADVANCE about 1000?






Mark Miller <ma...@gmail.com> 
08/11/2008 05:13 PM
Please respond to
java-user@lucene.apache.org


To
java-user@lucene.apache.org
cc

Subject
Re: Field sizes: maxFieldLength






Aravind.Yarram@equifax.com wrote:
> Hi all -
>
> I know in advance that each of the fileds i index doesnt go more than 
> 1000, Can i gain any performance improvement while writing the index by 
> limiting the maxFieldLength to 200? 
>
> tx
> Regards, 
> Aravind R Yarram 
> This message contains information from Equifax Inc. which may be 
confidential and privileged.  If you are not an intended recipient, please 
refrain from any disclosure, copying, distribution or use of this 
information and note that such actions are prohibited.  If you have 
received this transmission in error, please notify by e-mail 
postmaster@equifax.com.
>
> 
Its 10000. Sure, if you have a lot of docs between 200 and 10000, 
indexing less will be faster. But you will only be able to search on 
those first 200 tokens for any doc longer.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org



This message contains information from Equifax Inc. which may be confidential and privileged.  If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited.  If you have received this transmission in error, please notify by e-mail postmaster@equifax.com.

Re: Field sizes: maxFieldLength

Posted by Mark Miller <ma...@gmail.com>.
Aravind.Yarram@equifax.com wrote:
> Hi all -
>
> I know in advance that each of the fileds i index doesnt go more than 
> 1000, Can i gain any performance improvement while writing the index by 
> limiting the maxFieldLength to 200? 
>
> tx
> Regards, 
> Aravind R Yarram 
> This message contains information from Equifax Inc. which may be confidential and privileged.  If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited.  If you have received this transmission in error, please notify by e-mail postmaster@equifax.com.
>
>   
Its 10000. Sure, if you have a lot of docs between 200 and 10000, 
indexing less will be faster. But you will only be able to search on 
those first 200 tokens for any doc longer.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org