Upper limit on number of Fields

[Joel Halbert <jo...@su3analytics.com>]

Hi,

Is there any practical limit on the number of fields that can be
maintained on an index?

My index looks something like this, 1 million documents. For each group
of 1000 documents I might have 10 indexed fields. This would mean in
total about 10000 fields. Am I going to run into any issues here?

The index in Ram based.

Rgs,
Joel 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Upper limit on number of Fields

[Mark Miller <ma...@gmail.com>]

In my experience, the main issue to be concerned about with tons of 
fields is norms. You'll likely have to turn them off for most of the 
fields unless you have plenty of RAM to burn. They are stored in byte 
arrays of size maxdoc for each field (eg non sparse). Other than that, I 
don't think there is a pronounced penalty in my experience for tons of 
fields. 10,000 fields is probably more than I have ever seen though, so 
do report back how it goes.


-- 
- Mark

http://www.lucidimagination.com



Joel Halbert wrote:
> Hi,
>
> Is there any practical limit on the number of fields that can be
> maintained on an index?
>
> My index looks something like this, 1 million documents. For each group
> of 1000 documents I might have 10 indexed fields. This would mean in
> total about 10000 fields. Am I going to run into any issues here?
>
> The index in Ram based.
>
> Rgs,
> Joel 
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Upper limit on number of Fields

[Karl Wettin <ka...@gmail.com>]

15 feb 2009 kl. 16.27 skrev Joel Halbert:

> Is there any practical limit on the number of fields that can be
> maintained on an index?
>
> My index looks something like this, 1 million documents. For each  
> group
> of 1000 documents I might have 10 indexed fields. This would mean in
> total about 10000 fields. Am I going to run into any issues here?
>
> The index in Ram based.


There is a resource cost to handle identical term values in several  
fields rather than keeping them in a single field as there is one  
index per field. You might however save CPU at query time by dividing  
them up in multiple fields as a field index would be smaller and thus  
faster to navigate.  But at the same time Lucene is more or less  
logarithmic when it comes to speed so you might actually save both  
speed and resources by using the same field if you can.

I don't think there will be a problem. A million documents might  
however consume a lot of RAM. Did you measure how much slower it is on  
FS? The difference really should not be that great, especially when  
the index is that large. Also, reports (including my own) indicate  
that running the index on SSD is  as fast as running in RAM so you  
might want to look in to how much bang for the buck you get by hosting  
in RAM rather than on SSD.

Hope this helps,

        karl

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org