You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Damjan Cvetan <da...@cvetan.si.INVALID> on 2024/05/10 08:29:29 UTC
Solr resource recommendations
Dear Solr users,
I'm reaching out for any recommendation on efficient resource needs for the
collection I'm running. I hope some of you can share some info on your
resources and collection size which I can compare to.
I'm trying to optimize the setup with various settings as suggested within
"Solr optimization" articles but I'm wondering how much CPUs, RAM and Solr
nodes should have been enough.
I have 2 collections, each of 175 GB in size where the average
document size is 180kB. One collection is using stemming filters while the
other does not. I'm running all of it on 2 nodes, each having 6 CPUs and
80GB of RAM. Collections are split into 2 shards, one shard per node. I'm
doing query, filter query, faceeting, sorting and range queries. It was all
working fine but last year the collection got bigger and response time got
up to 30s.
Thank you all.
Regards,
Damjan
Re: Solr resource recommendations
Posted by ufuk yılmaz <uy...@vivaldi.net.INVALID>.
Hi Damjan,
Just a reminder, you need to reindex if you change an existing field definition to have docValues, since they are created and written to disk during indexing.
Sincerely,
Ufuk
—
> On May 11, 2024, at 4:52, Damjan Cvetan <da...@cvetan.si.invalid> wrote:
>
> Hi Ufuk,
>
> Thank you for pointing me to the docValues. I have some fields there added
> later in the project used for faceting and not having docValues enabled.
> I'll try that out.
>
> I'm not using the JSON Facet API but a traditional faceting approach. I see
> there are facet.method options (enum, fc, fcs), I'm using the fc option as
> default. I see that JSON Facet API provides more methods. I'll test them
> out on my collection before doing any code changes in the project.
>
> I'll report back with my findings. Thank you.
>
> Regards,
> Damjan
>
> On Fri, 10 May 2024 at 19:24, ufuk yılmaz <uy...@vivaldi.net.invalid>
> wrote:
>
>> Hi Damjan,
>>
>> Do you make proper use of docvalues in your schema?
>>
>> Also for faceting, did you check out the “method” parameter of the json
>> facet api?
>>
>> Those two made the biggest impact for me in a situation similar to yours
>> in the past.
>>
>> Regards,
>> Ufuk
>> —
>>
>>> On May 10, 2024, at 17:29, Damjan Cvetan <da...@cvetan.si.invalid>
>> wrote:
>>>
>>> Dear Solr users,
>>>
>>> I'm reaching out for any recommendation on efficient resource needs for
>> the
>>> collection I'm running. I hope some of you can share some info on your
>>> resources and collection size which I can compare to.
>>>
>>> I'm trying to optimize the setup with various settings as suggested
>> within
>>> "Solr optimization" articles but I'm wondering how much CPUs, RAM and
>> Solr
>>> nodes should have been enough.
>>>
>>> I have 2 collections, each of 175 GB in size where the average
>>> document size is 180kB. One collection is using stemming filters while
>> the
>>> other does not. I'm running all of it on 2 nodes, each having 6 CPUs and
>>> 80GB of RAM. Collections are split into 2 shards, one shard per node. I'm
>>> doing query, filter query, faceeting, sorting and range queries. It was
>> all
>>> working fine but last year the collection got bigger and response time
>> got
>>> up to 30s.
>>>
>>> Thank you all.
>>>
>>> Regards,
>>> Damjan
>>
>>
Re: Solr resource recommendations
Posted by Damjan Cvetan <da...@cvetan.si.INVALID>.
Hi Ufuk,
Thank you for pointing me to the docValues. I have some fields there added
later in the project used for faceting and not having docValues enabled.
I'll try that out.
I'm not using the JSON Facet API but a traditional faceting approach. I see
there are facet.method options (enum, fc, fcs), I'm using the fc option as
default. I see that JSON Facet API provides more methods. I'll test them
out on my collection before doing any code changes in the project.
I'll report back with my findings. Thank you.
Regards,
Damjan
On Fri, 10 May 2024 at 19:24, ufuk yılmaz <uy...@vivaldi.net.invalid>
wrote:
> Hi Damjan,
>
> Do you make proper use of docvalues in your schema?
>
> Also for faceting, did you check out the “method” parameter of the json
> facet api?
>
> Those two made the biggest impact for me in a situation similar to yours
> in the past.
>
> Regards,
> Ufuk
> —
>
> > On May 10, 2024, at 17:29, Damjan Cvetan <da...@cvetan.si.invalid>
> wrote:
> >
> > Dear Solr users,
> >
> > I'm reaching out for any recommendation on efficient resource needs for
> the
> > collection I'm running. I hope some of you can share some info on your
> > resources and collection size which I can compare to.
> >
> > I'm trying to optimize the setup with various settings as suggested
> within
> > "Solr optimization" articles but I'm wondering how much CPUs, RAM and
> Solr
> > nodes should have been enough.
> >
> > I have 2 collections, each of 175 GB in size where the average
> > document size is 180kB. One collection is using stemming filters while
> the
> > other does not. I'm running all of it on 2 nodes, each having 6 CPUs and
> > 80GB of RAM. Collections are split into 2 shards, one shard per node. I'm
> > doing query, filter query, faceeting, sorting and range queries. It was
> all
> > working fine but last year the collection got bigger and response time
> got
> > up to 30s.
> >
> > Thank you all.
> >
> > Regards,
> > Damjan
>
>
Re: Solr resource recommendations
Posted by ufuk yılmaz <uy...@vivaldi.net.INVALID>.
Hi Damjan,
Do you make proper use of docvalues in your schema?
Also for faceting, did you check out the “method” parameter of the json facet api?
Those two made the biggest impact for me in a situation similar to yours in the past.
Regards,
Ufuk
—
> On May 10, 2024, at 17:29, Damjan Cvetan <da...@cvetan.si.invalid> wrote:
>
> Dear Solr users,
>
> I'm reaching out for any recommendation on efficient resource needs for the
> collection I'm running. I hope some of you can share some info on your
> resources and collection size which I can compare to.
>
> I'm trying to optimize the setup with various settings as suggested within
> "Solr optimization" articles but I'm wondering how much CPUs, RAM and Solr
> nodes should have been enough.
>
> I have 2 collections, each of 175 GB in size where the average
> document size is 180kB. One collection is using stemming filters while the
> other does not. I'm running all of it on 2 nodes, each having 6 CPUs and
> 80GB of RAM. Collections are split into 2 shards, one shard per node. I'm
> doing query, filter query, faceeting, sorting and range queries. It was all
> working fine but last year the collection got bigger and response time got
> up to 30s.
>
> Thank you all.
>
> Regards,
> Damjan