dynamicField Scaling

[mark angelillo <li...@snooth.com>]

Hello,

I've got a Solr index running and I want to use a dynamicField to  
store n different sorting fields. The field that is used to actually  
sort the results will be determined by the application that is  
querying the index.

I'm wondering if anyone has done something similar to this, or if  
anyone has an idea of how Solr will perform as the number n of  
sorting fields grows larger. Is there a way to make sure this doesn't  
start to slow the index down? Is there any information out there  
about the number of dynamicFields that can be declared in this way  
before the entire index suffers? Is there such a limit?

(I'm assuming the number of documents in the index will eventually be  
around 500k -- perhaps more in the future.)

TIA,
Mark Angelillo

Re: dynamicField Scaling

[mark angelillo <li...@snooth.com>]

On Mar 7, 2007, at 2:17 PM, Mike Klaas wrote:

> On 3/7/07, mark angelillo <li...@snooth.com> wrote:
>> Hello,
>>
>> I've got a Solr index running and I want to use a dynamicField to
>> store n different sorting fields. The field that is used to actually
>> sort the results will be determined by the application that is
>> querying the index.
>>
>> I'm wondering if anyone has done something similar to this, or if
>> anyone has an idea of how Solr will perform as the number n of
>> sorting fields grows larger. Is there a way to make sure this doesn't
>> start to slow the index down? Is there any information out there
>> about the number of dynamicFields that can be declared in this way
>> before the entire index suffers? Is there such a limit?
>
> It's not realy about the number of dynamic fields.  The key variable
> is the number of sort fields.  To sort efficiently, solr needs to
> maintain a cache of field values.  This consumes memory per-field on
> the order of
>
> D x S + U
>
> where D is the document count, S is the the size of the data type (eg.
> 4bytes for ints, 8 bytes for doubles, 4/8 bytes for anything else
> [pointers]), and U is the cumulative size of the unique field values
> (if sorting on a non-primitive type, like Strings).
>
> If you have sufficient memory to store this data for each field you
> are sorting on, you shouldn't have any problems.
>
> best,
> -Mike


Okay, makes sense.

Thanks,
Mark

Re: dynamicField Scaling

[Mike Klaas <mi...@gmail.com>]

On 3/7/07, mark angelillo <li...@snooth.com> wrote:
> Hello,
>
> I've got a Solr index running and I want to use a dynamicField to
> store n different sorting fields. The field that is used to actually
> sort the results will be determined by the application that is
> querying the index.
>
> I'm wondering if anyone has done something similar to this, or if
> anyone has an idea of how Solr will perform as the number n of
> sorting fields grows larger. Is there a way to make sure this doesn't
> start to slow the index down? Is there any information out there
> about the number of dynamicFields that can be declared in this way
> before the entire index suffers? Is there such a limit?

It's not realy about the number of dynamic fields.  The key variable
is the number of sort fields.  To sort efficiently, solr needs to
maintain a cache of field values.  This consumes memory per-field on
the order of

D x S + U

where D is the document count, S is the the size of the data type (eg.
4bytes for ints, 8 bytes for doubles, 4/8 bytes for anything else
[pointers]), and U is the cumulative size of the unique field values
(if sorting on a non-primitive type, like Strings).

If you have sufficient memory to store this data for each field you
are sorting on, you shouldn't have any problems.

best,
-Mike