Slow query response.

[Modassar Ather <mo...@gmail.com>]

Hi,

I have a field f which is defined as follows.
<fieldType name="string" class="solr.StrField" sortMissingLast="true"
omitNorms="true"/>

Solr-5.2.1 is used. The index is spread across 12 shards (no replica) and
the index size on each node is around 100 GB.

When I search for 50 thousand values (ORed) in the field f it takes almost
around 45 to 55 seconds.
Per my understanding it is too slow. Kindly share your thoughts on this
behavior and provide your suggestions.

Thanks,
Modassar

Re: Slow query response.

[Modassar Ather <mo...@gmail.com>]

Thanks Jack for your response.

The users of our application can enter a list of ids which the UI caps at
50k. All the ids are valid and match documents. We do faceting, grouping
etc. on the result set of up to 50k documents.
I checked and found that the query is not very resource intensive. It is
not eating up lot of CPU%, IO% or memory.

Regards,
Modassar

On Thu, Dec 17, 2015 at 8:44 PM, Jack Krupansky <ja...@gmail.com>
wrote:

> A single query with tens of thousands of terms is very clearly a misuse of
> Solr. If it happens to work at all, consider yourself lucky. Are you using
> a standard Solr query parser or the terms query parser that lets you write
> a raw list of terms to OR.
>
> Are your nodes CPU-bound or I/O-bound during those 50-second intervals? My
> bet is that your index does not fit fully in memory, causing lots of I/O to
> repeatedly page in portions of the index and probably additional CPU usage
> as well.
>
> How many rows are you returning on each query? Are you using all these
> terms just to filter a smaller query or to return a large bulk of
> documents?
>
>
> -- Jack Krupansky
>
> On Thu, Dec 17, 2015 at 7:01 AM, Modassar Ather <mo...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have a field f which is defined as follows.
> > <fieldType name="string" class="solr.StrField" sortMissingLast="true"
> > omitNorms="true"/>
> >
> > Solr-5.2.1 is used. The index is spread across 12 shards (no replica) and
> > the index size on each node is around 100 GB.
> >
> > When I search for 50 thousand values (ORed) in the field f it takes
> almost
> > around 45 to 55 seconds.
> > Per my understanding it is too slow. Kindly share your thoughts on this
> > behavior and provide your suggestions.
> >
> > Thanks,
> > Modassar
> >
>

Re: Slow query response.

[Jack Krupansky <ja...@gmail.com>]

A single query with tens of thousands of terms is very clearly a misuse of
Solr. If it happens to work at all, consider yourself lucky. Are you using
a standard Solr query parser or the terms query parser that lets you write
a raw list of terms to OR.

Are your nodes CPU-bound or I/O-bound during those 50-second intervals? My
bet is that your index does not fit fully in memory, causing lots of I/O to
repeatedly page in portions of the index and probably additional CPU usage
as well.

How many rows are you returning on each query? Are you using all these
terms just to filter a smaller query or to return a large bulk of documents?


-- Jack Krupansky

On Thu, Dec 17, 2015 at 7:01 AM, Modassar Ather <mo...@gmail.com>
wrote:

> Hi,
>
> I have a field f which is defined as follows.
> <fieldType name="string" class="solr.StrField" sortMissingLast="true"
> omitNorms="true"/>
>
> Solr-5.2.1 is used. The index is spread across 12 shards (no replica) and
> the index size on each node is around 100 GB.
>
> When I search for 50 thousand values (ORed) in the field f it takes almost
> around 45 to 55 seconds.
> Per my understanding it is too slow. Kindly share your thoughts on this
> behavior and provide your suggestions.
>
> Thanks,
> Modassar
>