the impact of thousands of field in a single document

[Fa...@emc.com]

Hi,

Due to requirement, we need to construct a Lucene document with tens of
thousands of Field. Did anyone try this? What's the performance penalty
comparing with one single field to store all tokens for both indexing
and searching?

Thanks,
Li 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: the impact of thousands of field in a single document

[Yonik Seeley <ys...@gmail.com>]

On Wed, Feb 18, 2009 at 3:26 AM,  <Fa...@emc.com> wrote:
> Due to requirement, we need to construct a Lucene document with tens of
> thousands of Field. Did anyone try this? What's the performance penalty
> comparing with one single field to store all tokens for both indexing
> and searching?

It's doable.
Searching performance won't suffer at all.
Retrieving stored fields (after a search) will suffer somewhat.
Indexing performance will suffer somewhat.

The main problem will be norms, which aren't sparse and take up one
byte per document per field in the index, unless you omit the norms.

-Yonik
Lucene/Solr? http://www.lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org