Facet

[William Bell <bi...@gmail.com>]

We increased our number of terms (String) in a facet by 50,000.  Now we are
getting an error when we facet by this field - so we switched it to
facet.method=enum, and now the results come back. However, when we put it
into production we literally hit a wall (CPU went to 100% for 16 cores)
after about 30 minutes live.

We tried adding more machines to reduce the CPU, but it did not help. I am
wondering if CPU is a symptom... So we increased memory from 12GB to 16GB.
(the core size is 10GB). This is Solr 4.10.3.

What are some ideas? We are going to try docValues on the field. Does
anyone know if method=fc or method=enum works for docValue? I cannot find
any documentation on that.

We are thinking of splitting the field into 2 fields (fielda, fieldb). At
least the number will be less, but not sure if it will help memory?

The weird thing is for the first 30 minutes things are performing great.
Literally at like 10% CPU across 16 cores, not much memory and normal GC.

Originally the facet was a method=fc. Is there an issue with enum? We have
facet.threads=20 set, and not sure this is wise for a enum ?

What would you look at? Would you recommend turning off different caches
like fieldCache or filterCache?

-- 
Bill Bell
billnbell@gmail.com
cell 720-256-8076

Re: Facet

[Erick Erickson <er...@gmail.com>]

fc.method=enum will create an entry in the filter cache for each and
every value. But since the filterCache is bounded, each result will
pretty much be thrown away immediately. At least that's what I
remember.

Which neatly accounts for your issue I think; you're spending a huge
amount of time/cycles calculating filterCache entries to just throw
them away. If you increased your filterCache size to (shudder) 300K+ I
think your performance would be fine after the first one, but I
really, really, really doubt you can do that.

You say "Now we are getting an error". What's the error? I'm guessing OOM...

Faceting really wasn't built for very high cardinality fields. If this
is a reporting kind of thing, and you have the option of using 5.1
(coming Real Soon Now), you might get some usage out of "streaming
aggregation", which is way cool. But it's not going to give you
sub-second responses though.

Best,
Erick

On Sun, Apr 5, 2015 at 2:59 PM, Toke Eskildsen <te...@statsbiblioteket.dk> wrote:
> Bill Bell <bi...@gmail.com> wrote:
>> The limit is set to -1. But the average result is 300.
>
> Okay, better. Well, somewhat better. But unless your values are very well distributed, I would guess that your worst case is very high. Have you checked if your performance problems are for specific queries?
>
> One way is to look through your solr.log for high QTimes and see if that correlates with large result sets. My guess (still assuming distributed search) is that lines containing __terms (indicating the fine count phase of distributed faceting) will have higher QTimes that the other queries.
>
>> Would creating 900 fields be better ?
>> Then I could just put the prefix in the field name.
>
> With fc, there is an constant overhead for each field that you facet on. 900 fields would take up much more memory than a single field with all the values. I don't think that enum leaves structures in memory, but I doubt that it would be better than using a single field and facet.prefix.
>
>> So far I heard solcloud, docvalues as viable solutions. Stay away from enum.
>
> SolrCloud is not a solution to faceting as such. There is a performance penalty when switching from single-shard to SolrCloud, especially for the fairly large facet result sets that you have. I just guessed that you were using SolrCloud already.
>
> A quick test: Try setting facet.limit=10 and run some tests. If performance is fine for that and you're using multiple shards, then your performance (at least for faceting) would probably be a lot higher with just a single shard.
>
> - Toke Eskildsen

Re: Facet

[Toke Eskildsen <te...@statsbiblioteket.dk>]

Bill Bell <bi...@gmail.com> wrote:
> The limit is set to -1. But the average result is 300.

Okay, better. Well, somewhat better. But unless your values are very well distributed, I would guess that your worst case is very high. Have you checked if your performance problems are for specific queries?

One way is to look through your solr.log for high QTimes and see if that correlates with large result sets. My guess (still assuming distributed search) is that lines containing __terms (indicating the fine count phase of distributed faceting) will have higher QTimes that the other queries.

> Would creating 900 fields be better ?
> Then I could just put the prefix in the field name.

With fc, there is an constant overhead for each field that you facet on. 900 fields would take up much more memory than a single field with all the values. I don't think that enum leaves structures in memory, but I doubt that it would be better than using a single field and facet.prefix.

> So far I heard solcloud, docvalues as viable solutions. Stay away from enum.

SolrCloud is not a solution to faceting as such. There is a performance penalty when switching from single-shard to SolrCloud, especially for the fairly large facet result sets that you have. I just guessed that you were using SolrCloud already.

A quick test: Try setting facet.limit=10 and run some tests. If performance is fine for that and you're using multiple shards, then your performance (at least for faceting) would probably be a lot higher with just a single shard.

- Toke Eskildsen

Re: Facet

[Bill Bell <bi...@gmail.com>]

Ok

Clarification

The limit is set to -1. But the average result is 300. 

The amount of strings stored in the field increased a lot. Like 250k to 350k. But the amount coming out is limited by facet.prefix. 

Would creating 900 fields be better ? Then I could just put the prefix in the field name. Like this: proc_ps122

Thoughts ?

So far I heard solcloud, docvalues as viable solutions. Stay away from enum.

Bill Bell
Sent from mobile


> On Apr 5, 2015, at 2:56 AM, Toke Eskildsen <te...@statsbiblioteket.dk> wrote:
> 
> William Bell <bi...@gmail.com> wrote:
> Sent: 05 April 2015 06:20
> To: solr-user@lucene.apache.org
> Subject: Facet
> 
>> We increased our number of terms (String) in a facet by 50,000.
> 
> Do you mean facet.limit=50000?
> 
>> Now we are getting an error when we facet by this field - so we switched it to
>> facet.method=enum, and now the results come back. However, when we put
>> it into production we literally hit a wall (CPU went to 100% for 16 cores)
>> after about 30 minutes live.
> 
> It was strange that enum worked. Internally, the difference between facet.limit=100 and facet.limit=50000 is quite small. The real hits are for fine-counting within SolrCloud and serializing the result in order to deliver it to the client. I thought enum behaved the same as fc with regard to those two.
> 
>> We tried adding more machines to reduce the CPU, but it did not help.
> 
> Sounds like SolrCloud. More machines does not help here, it might even be worse. What happens is that distributed faceting is two-phase, where the second phase is fine-counting. The fine-counting essentially makes all shards perform micro-searches for a large part of the terms returned: Your shards are bogged down by tens of thousands of small searches.
> 
> If you are feeling adventurous, you can try putting
> http://tokee.github.io/lucene-solr/
> on a test-installation (I am the author). It changes the way the fine-counting is done.
> 
> 
> Depending on your container, you might need to raise the internal limits for GET-communication. Tomcat has a default of 2MB somewhere (sorry, don't remember the details), which is not a lot for 50,000 values.
> 
>> What are some ideas? We are going to try docValues on the field. Does
>> anyone know if method=fc or method=enum works for docValue? I cannot find
>> any documentation on that.
> 
> If DocValues are enabled, fc will use them. It does not change anything for enum. But I would argue against enum for anything in the thousands anyway.
> 
>> We are thinking of splitting the field into 2 fields (fielda, fieldb). At
>> least the number will be less, but not sure if it will help memory?
> 
> The killer is the number of terms requested/returned.
> 
>> The weird thing is for the first 30 minutes things are performing great.
>> Literally at like 10% CPU across 16 cores, not much memory and normal GC.
> 
> It might be because you have just been lucky. Take a look at
> https://twitter.com/anjacks0n/status/509284768035262464
> for how different performance can be for different result set sizes.
> 
>> Originally the facet was a method=fc. Is there an issue with enum? We have
>> facet.threads=20 set, and not sure this is wise for a enum ?
> 
> Facet threading does not thread within each field, it just means that multiple fields are processed in parallel.
> 
> - Toke Eskildsen

Re: Facet

[Toke Eskildsen <te...@statsbiblioteket.dk>]

William Bell <bi...@gmail.com> wrote:
Sent: 05 April 2015 06:20
To: solr-user@lucene.apache.org
Subject: Facet

> We increased our number of terms (String) in a facet by 50,000.

Do you mean facet.limit=50000?

>  Now we are getting an error when we facet by this field - so we switched it to
> facet.method=enum, and now the results come back. However, when we put
> it into production we literally hit a wall (CPU went to 100% for 16 cores)
> after about 30 minutes live.

It was strange that enum worked. Internally, the difference between facet.limit=100 and facet.limit=50000 is quite small. The real hits are for fine-counting within SolrCloud and serializing the result in order to deliver it to the client. I thought enum behaved the same as fc with regard to those two.

> We tried adding more machines to reduce the CPU, but it did not help.

Sounds like SolrCloud. More machines does not help here, it might even be worse. What happens is that distributed faceting is two-phase, where the second phase is fine-counting. The fine-counting essentially makes all shards perform micro-searches for a large part of the terms returned: Your shards are bogged down by tens of thousands of small searches.

If you are feeling adventurous, you can try putting
http://tokee.github.io/lucene-solr/
on a test-installation (I am the author). It changes the way the fine-counting is done.


Depending on your container, you might need to raise the internal limits for GET-communication. Tomcat has a default of 2MB somewhere (sorry, don't remember the details), which is not a lot for 50,000 values.

> What are some ideas? We are going to try docValues on the field. Does
> anyone know if method=fc or method=enum works for docValue? I cannot find
> any documentation on that.

If DocValues are enabled, fc will use them. It does not change anything for enum. But I would argue against enum for anything in the thousands anyway.

> We are thinking of splitting the field into 2 fields (fielda, fieldb). At
> least the number will be less, but not sure if it will help memory?

The killer is the number of terms requested/returned.

> The weird thing is for the first 30 minutes things are performing great.
> Literally at like 10% CPU across 16 cores, not much memory and normal GC.

It might be because you have just been lucky. Take a look at
https://twitter.com/anjacks0n/status/509284768035262464
for how different performance can be for different result set sizes.

> Originally the facet was a method=fc. Is there an issue with enum? We have
> facet.threads=20 set, and not sure this is wise for a enum ?

Facet threading does not thread within each field, it just means that multiple fields are processed in parallel.

- Toke Eskildsen