You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sundeep T <su...@gmail.com> on 2017/11/20 17:33:42 UTC
Trailing wild card searches very slow in Solr
Hi,
We have several indexed string fields which is not tokenized and does not
have docValues enabled.
When we do trailing wildcard searches on these fields they are running very
slow. We were thinking that since this field is indexed, such queries
should be running pretty quickly. We are using Solr 6.6.1. Anyone has ideas
on why these queries are running slow and if there are any ways to speed
them up?
Thanks
Sundeep
Re: Trailing wild card searches very slow in Solr
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/20/2017 12:50 PM, Sundeep T wrote:
> I initially asked this question regarding leading wildcards. This was a
> typo, and what I meant was trailing wild card queries were slow. So queries
> like text:'hello*" are slow. We were expecting since the string field is
> already indexed, the searches should be fast, but that seems to be not the
> case
The following is my understanding of wildcard queries:
Let's say that the wildcard "hello*" matches one million terms on the
text field you're searching. I have no idea what the actual number
would be ... one million is arbitrary.
Behind the scenes, that query object will quite literally have one
million different terms in it, and ALL of those terms will be checked
against the index to obtain the list of matching documents. Even if
each check is pretty fast, doing a million of them to satisfy the query
is going to take some time. Also, it's going to take some time to
gather the one million matching terms in the first place. The term
gathering step is particularly slow with leading wildcards, but even
trailing wildcards can result in slow queries.
Wildcards that only match a few terms might be quick, but predicting the
number of matching terms BEFORE the query is executed is difficult.
Thanks,
Shawn
Re: Trailing wild card searches very slow in Solr
Posted by Erick Erickson <er...@gmail.com>.
At first glance you have a mis-configured setup. The most glaring
issue is that you're trying to search a 150G index in 1G of memory.
bq: String field (not tokenized) is docValues=true, indexed=true and stored=true
OK, this is kind of unusual to query but if the field just contains
single tokens it's probably OK.
bq: Field is almost unique in the index, around 80 million are unique
This is a _lot_ of unique fields, but as long as your wildcard
searches don't actually match too many values (say 1,000 or so) it
should be OK.
bq: no commits on index
Huh? Then you can't search. I suspect you have autocommit settings in
your solrconfig.xml file?
bq: solr jvm heap 1GB
This is far too small. It's a miracle it works at all.
bq: index size on disk is around 150GB
I pretty much guarantee your heap is undersized for an index that size.
bq: q=myfield:abc* has QTime=17-20secs after filecache on OS is primed
How many terms does abc* match? That's the biggest question in terms
of perfirmance.
But really, I expect even if you created an OR clause with, say, 50
terms in it it would perform poorly. My guess is that you don't have
nearly enough memory for your Solr instance.
You didn't include the results of adding &debug=query, perhaps you
can't due to corporate policy. But you _can_ scrub the
parsedQuery_tostring bits of the return.
But really, don't do much until you give your Solr instance enough
memory to work with.
Best,
Erick
On Mon, Nov 20, 2017 at 5:26 PM, Sundeep T <su...@gmail.com> wrote:
> Hi Erick,
>
> Thanks for the reply. Here are more details on our setup -
>
> *Setup/schema details -*
>
> 100 million doc solr core
>
> String field (not tokenized) is docValues=true, indexed=true and stored=true
>
> Field is almost unique in the index, around 80 million are unique
>
> no commits on index
>
> all caches disabled in solrconfig.xml
>
> solr jvm heap 1GB
>
> single solr core in jvm
>
> solr core is not optimized and has about 50 segment files some up to 5GB
>
> index size on disk is around 150GB
>
> solr v6.5.0
>
>
>
> *Performance -*
>
>
> q=myfield:abc* has QTime=30secs+ first time
>
> q=myfield:abc* has QTime=17-20secs after filecache on OS is primed
>
>
> Thanks
> Sundeep
>
>
> On Mon, Nov 20, 2017 at 12:16 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> Well, define "slow". Conceptually a large OR clause is created that
>> contains all the terms that start with the indicated text. (actually a
>> PrefixQuery should be formed).
>>
>> That said, I'd expect hello* to be reasonably fast as not many terms
>> _probably_ start with 'hello'. Not the same at all for, say, h*.
>>
>> You might review: https://wiki.apache.org/solr/UsingMailingLists,
>> you're not really providing much information to go on here.
>>
>> What is the result of adding &debug=query? Particularly it would be
>> useful to see the parsed query.
>>
>> Are all such queries slow? What happens if you submit hel* followed by
>> hello*, the first one will bring the underlying index structures into
>> memory, for all we know this could simply be an autowarming issue.
>>
>> Are you indexing at the same time? Do you have a short autocommit interval?
>>
>> What version of Solr?
>>
>> Details matter.
>> Best,
>> Erick
>>
>> On Mon, Nov 20, 2017 at 11:50 AM, Sundeep T <su...@gmail.com> wrote:
>> > Hi Erick.
>> >
>> > I initially asked this question regarding leading wildcards. This was a
>> > typo, and what I meant was trailing wild card queries were slow. So
>> queries
>> > like text:'hello*" are slow. We were expecting since the string field is
>> > already indexed, the searches should be fast, but that seems to be not
>> the
>> > case
>> >
>> > Thanks
>> > Sundeep
>> >
>> > On Mon, Nov 20, 2017 at 9:39 AM, Erick Erickson <erickerickson@gmail.com
>> >
>> > wrote:
>> >
>> >> You already asked that question and got several answers, did you not
>> >> see them? If you did see them, what is unclear?
>> >>
>> >> Best,
>> >> Erick
>> >>
>> >> On Mon, Nov 20, 2017 at 9:33 AM, Sundeep T <su...@gmail.com>
>> wrote:
>> >> > Hi,
>> >> >
>> >> > We have several indexed string fields which is not tokenized and does
>> not
>> >> > have docValues enabled.
>> >> >
>> >> > When we do trailing wildcard searches on these fields they are running
>> >> very
>> >> > slow. We were thinking that since this field is indexed, such queries
>> >> > should be running pretty quickly. We are using Solr 6.6.1. Anyone has
>> >> ideas
>> >> > on why these queries are running slow and if there are any ways to
>> speed
>> >> > them up?
>> >> >
>> >> > Thanks
>> >> > Sundeep
>> >>
>>
Re: Trailing wild card searches very slow in Solr
Posted by Sundeep T <su...@gmail.com>.
Hi Erick,
Thanks for the reply. Here are more details on our setup -
*Setup/schema details -*
100 million doc solr core
String field (not tokenized) is docValues=true, indexed=true and stored=true
Field is almost unique in the index, around 80 million are unique
no commits on index
all caches disabled in solrconfig.xml
solr jvm heap 1GB
single solr core in jvm
solr core is not optimized and has about 50 segment files some up to 5GB
index size on disk is around 150GB
solr v6.5.0
*Performance -*
q=myfield:abc* has QTime=30secs+ first time
q=myfield:abc* has QTime=17-20secs after filecache on OS is primed
Thanks
Sundeep
On Mon, Nov 20, 2017 at 12:16 PM, Erick Erickson <er...@gmail.com>
wrote:
> Well, define "slow". Conceptually a large OR clause is created that
> contains all the terms that start with the indicated text. (actually a
> PrefixQuery should be formed).
>
> That said, I'd expect hello* to be reasonably fast as not many terms
> _probably_ start with 'hello'. Not the same at all for, say, h*.
>
> You might review: https://wiki.apache.org/solr/UsingMailingLists,
> you're not really providing much information to go on here.
>
> What is the result of adding &debug=query? Particularly it would be
> useful to see the parsed query.
>
> Are all such queries slow? What happens if you submit hel* followed by
> hello*, the first one will bring the underlying index structures into
> memory, for all we know this could simply be an autowarming issue.
>
> Are you indexing at the same time? Do you have a short autocommit interval?
>
> What version of Solr?
>
> Details matter.
> Best,
> Erick
>
> On Mon, Nov 20, 2017 at 11:50 AM, Sundeep T <su...@gmail.com> wrote:
> > Hi Erick.
> >
> > I initially asked this question regarding leading wildcards. This was a
> > typo, and what I meant was trailing wild card queries were slow. So
> queries
> > like text:'hello*" are slow. We were expecting since the string field is
> > already indexed, the searches should be fast, but that seems to be not
> the
> > case
> >
> > Thanks
> > Sundeep
> >
> > On Mon, Nov 20, 2017 at 9:39 AM, Erick Erickson <erickerickson@gmail.com
> >
> > wrote:
> >
> >> You already asked that question and got several answers, did you not
> >> see them? If you did see them, what is unclear?
> >>
> >> Best,
> >> Erick
> >>
> >> On Mon, Nov 20, 2017 at 9:33 AM, Sundeep T <su...@gmail.com>
> wrote:
> >> > Hi,
> >> >
> >> > We have several indexed string fields which is not tokenized and does
> not
> >> > have docValues enabled.
> >> >
> >> > When we do trailing wildcard searches on these fields they are running
> >> very
> >> > slow. We were thinking that since this field is indexed, such queries
> >> > should be running pretty quickly. We are using Solr 6.6.1. Anyone has
> >> ideas
> >> > on why these queries are running slow and if there are any ways to
> speed
> >> > them up?
> >> >
> >> > Thanks
> >> > Sundeep
> >>
>
Re: Trailing wild card searches very slow in Solr
Posted by Erick Erickson <er...@gmail.com>.
Well, define "slow". Conceptually a large OR clause is created that
contains all the terms that start with the indicated text. (actually a
PrefixQuery should be formed).
That said, I'd expect hello* to be reasonably fast as not many terms
_probably_ start with 'hello'. Not the same at all for, say, h*.
You might review: https://wiki.apache.org/solr/UsingMailingLists,
you're not really providing much information to go on here.
What is the result of adding &debug=query? Particularly it would be
useful to see the parsed query.
Are all such queries slow? What happens if you submit hel* followed by
hello*, the first one will bring the underlying index structures into
memory, for all we know this could simply be an autowarming issue.
Are you indexing at the same time? Do you have a short autocommit interval?
What version of Solr?
Details matter.
Best,
Erick
On Mon, Nov 20, 2017 at 11:50 AM, Sundeep T <su...@gmail.com> wrote:
> Hi Erick.
>
> I initially asked this question regarding leading wildcards. This was a
> typo, and what I meant was trailing wild card queries were slow. So queries
> like text:'hello*" are slow. We were expecting since the string field is
> already indexed, the searches should be fast, but that seems to be not the
> case
>
> Thanks
> Sundeep
>
> On Mon, Nov 20, 2017 at 9:39 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> You already asked that question and got several answers, did you not
>> see them? If you did see them, what is unclear?
>>
>> Best,
>> Erick
>>
>> On Mon, Nov 20, 2017 at 9:33 AM, Sundeep T <su...@gmail.com> wrote:
>> > Hi,
>> >
>> > We have several indexed string fields which is not tokenized and does not
>> > have docValues enabled.
>> >
>> > When we do trailing wildcard searches on these fields they are running
>> very
>> > slow. We were thinking that since this field is indexed, such queries
>> > should be running pretty quickly. We are using Solr 6.6.1. Anyone has
>> ideas
>> > on why these queries are running slow and if there are any ways to speed
>> > them up?
>> >
>> > Thanks
>> > Sundeep
>>
Re: Trailing wild card searches very slow in Solr
Posted by Sundeep T <su...@gmail.com>.
Hi Erick.
I initially asked this question regarding leading wildcards. This was a
typo, and what I meant was trailing wild card queries were slow. So queries
like text:'hello*" are slow. We were expecting since the string field is
already indexed, the searches should be fast, but that seems to be not the
case
Thanks
Sundeep
On Mon, Nov 20, 2017 at 9:39 AM, Erick Erickson <er...@gmail.com>
wrote:
> You already asked that question and got several answers, did you not
> see them? If you did see them, what is unclear?
>
> Best,
> Erick
>
> On Mon, Nov 20, 2017 at 9:33 AM, Sundeep T <su...@gmail.com> wrote:
> > Hi,
> >
> > We have several indexed string fields which is not tokenized and does not
> > have docValues enabled.
> >
> > When we do trailing wildcard searches on these fields they are running
> very
> > slow. We were thinking that since this field is indexed, such queries
> > should be running pretty quickly. We are using Solr 6.6.1. Anyone has
> ideas
> > on why these queries are running slow and if there are any ways to speed
> > them up?
> >
> > Thanks
> > Sundeep
>
Re: Trailing wild card searches very slow in Solr
Posted by Erick Erickson <er...@gmail.com>.
You already asked that question and got several answers, did you not
see them? If you did see them, what is unclear?
Best,
Erick
On Mon, Nov 20, 2017 at 9:33 AM, Sundeep T <su...@gmail.com> wrote:
> Hi,
>
> We have several indexed string fields which is not tokenized and does not
> have docValues enabled.
>
> When we do trailing wildcard searches on these fields they are running very
> slow. We were thinking that since this field is indexed, such queries
> should be running pretty quickly. We are using Solr 6.6.1. Anyone has ideas
> on why these queries are running slow and if there are any ways to speed
> them up?
>
> Thanks
> Sundeep