You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alfonso Muñoz-Pomer Fuentes <am...@ebi.ac.uk> on 2018/02/20 11:44:23 UTC
What is “high cardinality” in facet streams?
Hi,
We have a query that we can resolve using either facet or search with rollup. In the Stream Source Reference section of Solr’s Reference Guide (https://lucene.apache.org/solr/guide/7_1/stream-source-reference.html#facet) it says “To support high cardinality aggregations see the rollup function”. I was wondering what it’s considered “high cardinality”. If it serves, our query returns up to 60k results. I haven’t got to do any benchmarking to see if there’s any difference, though, because facet so far performs very well, but I don’t know if I’m near the “tipping point”. Any feedback would be appreciated.
Many thanks in advance.
--
Alfonso Muñoz-Pomer Fuentes
Senior Lead Software Engineer @ Expression Atlas Team
European Bioinformatics Institute (EMBL-EBI)
European Molecular Biology Laboratory
Tel:+ 44 (0) 1223 49 2633
Skype: amunozpomer
Re: What is “high cardinality” in facet streams?
Posted by Alfonso Muñoz-Pomer Fuentes <am...@ebi.ac.uk>.
Right now we’re sharding the collection as we hit performance issues in the past with legacy Solr (i.e. a single Solr core), and also we’re experimenting a bit to see which replication factor we can get away with (in terms of resources and cost). Unfortunately, PSQL isn’t yet an option due to the lack of point field support, which we’re using in our schema (https://issues.apache.org/jira/browse/SOLR-10427).
Thanks for pointing at the parallel function. What I don’t understand, though, is if I don’t use the parallel decorator, my query isn’t distributed across my cluster nodes (e.g. I have four shards and no replicas)?
> On 22 Feb 2018, at 03:01, Joel Bernstein <jo...@gmail.com> wrote:
>
> With Streaming Expressions you have options for speeding up large
> aggregations.
>
> 1) Shard
> 2) Use the parallel function to run the aggregation in parallel.
> 3) Add more replicas
>
> When you use the parallel function the same aggregation can be pulled from
> every shard and every shard replica in the cluster.
>
> The parallel SQL interface supports a map_reduce aggregation mode where you
> can specific then number of parallel workers. If a SQL group by query works
> for you that might be the easiest way to go. The docs have good coverage of
> this topic.
>
>
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Wed, Feb 21, 2018 at 8:43 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 2/21/2018 12:08 PM, Alfonso Muñoz-Pomer Fuentes wrote:
>>> Some more details about my collection:
>>> - Approximately 200M documents
>>> - 1.2M different values in the field I’m faceting over
>>>
>>> The query I’m doing is over a single bucket, which after applying q and
>> fq the 1.2M values are reduced to, at most 60K (often times half that
>> value). From your replies I assume I’m not going to hit a bottleneck any
>> time soon. Thanks a lot.
>>
>> Two hundred million documents is going to be a pretty big index even if
>> the documents are small. The server is going to need a lot of spare
>> memory (not assigned to programs) for good general performance.
>>
>> As I understand it, facet performance is going to be heavily determined
>> by the 1.2 million unique values in the field you're using. Facet
>> performance is probably going to be very similar whether your query
>> matches 60K or 1 million.
>>
>> Thanks,
>> Shawn
>>
>>
--
Alfonso Muñoz-Pomer Fuentes
Senior Lead Software Engineer @ Expression Atlas Team
European Bioinformatics Institute (EMBL-EBI)
European Molecular Biology Laboratory
Tel:+ 44 (0) 1223 49 2633
Skype: amunozpomer
Re: What is “high cardinality” in facet streams?
Posted by Joel Bernstein <jo...@gmail.com>.
With Streaming Expressions you have options for speeding up large
aggregations.
1) Shard
2) Use the parallel function to run the aggregation in parallel.
3) Add more replicas
When you use the parallel function the same aggregation can be pulled from
every shard and every shard replica in the cluster.
The parallel SQL interface supports a map_reduce aggregation mode where you
can specific then number of parallel workers. If a SQL group by query works
for you that might be the easiest way to go. The docs have good coverage of
this topic.
Joel Bernstein
http://joelsolr.blogspot.com/
On Wed, Feb 21, 2018 at 8:43 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 2/21/2018 12:08 PM, Alfonso Muñoz-Pomer Fuentes wrote:
> > Some more details about my collection:
> > - Approximately 200M documents
> > - 1.2M different values in the field I’m faceting over
> >
> > The query I’m doing is over a single bucket, which after applying q and
> fq the 1.2M values are reduced to, at most 60K (often times half that
> value). From your replies I assume I’m not going to hit a bottleneck any
> time soon. Thanks a lot.
>
> Two hundred million documents is going to be a pretty big index even if
> the documents are small. The server is going to need a lot of spare
> memory (not assigned to programs) for good general performance.
>
> As I understand it, facet performance is going to be heavily determined
> by the 1.2 million unique values in the field you're using. Facet
> performance is probably going to be very similar whether your query
> matches 60K or 1 million.
>
> Thanks,
> Shawn
>
>
Re: What is “high cardinality” in facet streams?
Posted by Alfonso Muñoz-Pomer Fuentes <am...@ebi.ac.uk>.
All in all the index is about 250GB and it’s sharded in two dedicated VMs with 24GB of memory and it’s performing ok so far (queries take about 7 seconds, the worst cases about 10). At some point in the past we needed to transition to SolrCloud because a single Solr core, of course, wouldn’t scale.
> On 22 Feb 2018, at 01:43, Shawn Heisey <ap...@elyograg.org> wrote:
>
> On 2/21/2018 12:08 PM, Alfonso Muñoz-Pomer Fuentes wrote:
>> Some more details about my collection:
>> - Approximately 200M documents
>> - 1.2M different values in the field I’m faceting over
>>
>> The query I’m doing is over a single bucket, which after applying q and fq the 1.2M values are reduced to, at most 60K (often times half that value). From your replies I assume I’m not going to hit a bottleneck any time soon. Thanks a lot.
>
> Two hundred million documents is going to be a pretty big index even if
> the documents are small. The server is going to need a lot of spare
> memory (not assigned to programs) for good general performance.
>
> As I understand it, facet performance is going to be heavily determined
> by the 1.2 million unique values in the field you're using. Facet
> performance is probably going to be very similar whether your query
> matches 60K or 1 million.
>
> Thanks,
> Shawn
>
--
Alfonso Muñoz-Pomer Fuentes
Senior Lead Software Engineer @ Expression Atlas Team
European Bioinformatics Institute (EMBL-EBI)
European Molecular Biology Laboratory
Tel:+ 44 (0) 1223 49 2633
Skype: amunozpomer
Re: What is “high cardinality” in facet streams?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/21/2018 12:08 PM, Alfonso Muñoz-Pomer Fuentes wrote:
> Some more details about my collection:
> - Approximately 200M documents
> - 1.2M different values in the field I’m faceting over
>
> The query I’m doing is over a single bucket, which after applying q and fq the 1.2M values are reduced to, at most 60K (often times half that value). From your replies I assume I’m not going to hit a bottleneck any time soon. Thanks a lot.
Two hundred million documents is going to be a pretty big index even if
the documents are small. The server is going to need a lot of spare
memory (not assigned to programs) for good general performance.
As I understand it, facet performance is going to be heavily determined
by the 1.2 million unique values in the field you're using. Facet
performance is probably going to be very similar whether your query
matches 60K or 1 million.
Thanks,
Shawn
Re: What is “high cardinality” in facet streams?
Posted by Alfonso Muñoz-Pomer Fuentes <am...@ebi.ac.uk>.
Thank you for all the information. As usual I learn a lot about Solr and data modeling with each reply.
Some more details about my collection:
- Approximately 200M documents
- 1.2M different values in the field I’m faceting over
The query I’m doing is over a single bucket, which after applying q and fq the 1.2M values are reduced to, at most 60K (often times half that value). From your replies I assume I’m not going to hit a bottleneck any time soon. Thanks a lot.
> On 20 Feb 2018, at 18:03, Joel Bernstein <jo...@gmail.com> wrote:
>
> The rollup streaming expression rolls up aggregations on a stream that has
> been sorted by the group by fields. This is basically a MapReduce reduce
> operation and can work with extremely high cardinality (basically
> unlimited). The rollup function is designed to rollup data produced by the
> /export handler which can also sort data sets with very high cardinality.
> The docs should describe the correct usage of the rollup expression with
> the /export handler.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Tue, Feb 20, 2018 at 11:10 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 2/20/2018 4:44 AM, Alfonso Muñoz-Pomer Fuentes wrote:
>>
>>> We have a query that we can resolve using either facet or search with
>>> rollup. In the Stream Source Reference section of Solr’s Reference Guide (
>>> https://lucene.apache.org/solr/guide/7_1/stream-source-refe
>>> rence.html#facet) it says “To support high cardinality aggregations see
>>> the rollup function”. I was wondering what it’s considered “high
>>> cardinality”. If it serves, our query returns up to 60k results. I haven’t
>>> got to do any benchmarking to see if there’s any difference, though,
>>> because facet so far performs very well, but I don’t know if I’m near the
>>> “tipping point”. Any feedback would be appreciated.
>>>
>>
>> There's no hard and fast rule for this. The tipping point is going to be
>> different for every use case. With a little bit of information about your
>> setup, experienced users can make an educated guess about whether or not
>> performance will be good, but cannot say with absolute certainty what
>> you're going to run into.
>>
>> Let's start with some definitions, which you may or may not already know:
>>
>> https://en.wikipedia.org/wiki/Cardinality_(data_modeling)
>> https://en.wikipedia.org/wiki/Cardinality
>>
>> You haven't said how many unique values are in your field. The only
>> information I have from you is 60K results from your queries, which may or
>> may not have any bearing on the total number of documents in your index, or
>> the total number of unique values in the field you're using for faceting.
>> So the next paragraph may or may not apply to your index.
>>
>> In general, 60,000 unique values in a field would be considered very low
>> cardinality, because computers can typically operate on 60,000 values
>> *very* quickly, unless the size of each value is enormous. But if the
>> index has 60,000 total documents, then *in relation to other data*, the
>> cardinality is very high, even though most people would say the opposite.
>> Sixty thousand documents or unique values is almost always a very small
>> index, not prone to performance issues.
>>
>> The warnings about cardinality in the Solr documentation mostly refer to
>> *absolute* cardinality -- how many unique values there are in a field,
>> regardless of the actual number of documents. If there are millions or
>> billions of unique values, then operations like facets, grouping, sorting,
>> etc are probably going to be slow. If there are a lot less, such as
>> thousands or only a handful, then those operations are likely to be very
>> fast, because the computer will have less information it must process.
>>
>> Thanks,
>> Shawn
>>
>>
--
Alfonso Muñoz-Pomer Fuentes
Senior Lead Software Engineer @ Expression Atlas Team
European Bioinformatics Institute (EMBL-EBI)
European Molecular Biology Laboratory
Tel:+ 44 (0) 1223 49 2633
Skype: amunozpomer
Re: What is “high cardinality” in facet streams?
Posted by Joel Bernstein <jo...@gmail.com>.
The rollup streaming expression rolls up aggregations on a stream that has
been sorted by the group by fields. This is basically a MapReduce reduce
operation and can work with extremely high cardinality (basically
unlimited). The rollup function is designed to rollup data produced by the
/export handler which can also sort data sets with very high cardinality.
The docs should describe the correct usage of the rollup expression with
the /export handler.
Joel Bernstein
http://joelsolr.blogspot.com/
On Tue, Feb 20, 2018 at 11:10 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 2/20/2018 4:44 AM, Alfonso Muñoz-Pomer Fuentes wrote:
>
>> We have a query that we can resolve using either facet or search with
>> rollup. In the Stream Source Reference section of Solr’s Reference Guide (
>> https://lucene.apache.org/solr/guide/7_1/stream-source-refe
>> rence.html#facet) it says “To support high cardinality aggregations see
>> the rollup function”. I was wondering what it’s considered “high
>> cardinality”. If it serves, our query returns up to 60k results. I haven’t
>> got to do any benchmarking to see if there’s any difference, though,
>> because facet so far performs very well, but I don’t know if I’m near the
>> “tipping point”. Any feedback would be appreciated.
>>
>
> There's no hard and fast rule for this. The tipping point is going to be
> different for every use case. With a little bit of information about your
> setup, experienced users can make an educated guess about whether or not
> performance will be good, but cannot say with absolute certainty what
> you're going to run into.
>
> Let's start with some definitions, which you may or may not already know:
>
> https://en.wikipedia.org/wiki/Cardinality_(data_modeling)
> https://en.wikipedia.org/wiki/Cardinality
>
> You haven't said how many unique values are in your field. The only
> information I have from you is 60K results from your queries, which may or
> may not have any bearing on the total number of documents in your index, or
> the total number of unique values in the field you're using for faceting.
> So the next paragraph may or may not apply to your index.
>
> In general, 60,000 unique values in a field would be considered very low
> cardinality, because computers can typically operate on 60,000 values
> *very* quickly, unless the size of each value is enormous. But if the
> index has 60,000 total documents, then *in relation to other data*, the
> cardinality is very high, even though most people would say the opposite.
> Sixty thousand documents or unique values is almost always a very small
> index, not prone to performance issues.
>
> The warnings about cardinality in the Solr documentation mostly refer to
> *absolute* cardinality -- how many unique values there are in a field,
> regardless of the actual number of documents. If there are millions or
> billions of unique values, then operations like facets, grouping, sorting,
> etc are probably going to be slow. If there are a lot less, such as
> thousands or only a handful, then those operations are likely to be very
> fast, because the computer will have less information it must process.
>
> Thanks,
> Shawn
>
>
Re: What is “high cardinality” in facet streams?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/20/2018 4:44 AM, Alfonso Muñoz-Pomer Fuentes wrote:
> We have a query that we can resolve using either facet or search with rollup. In the Stream Source Reference section of Solr’s Reference Guide (https://lucene.apache.org/solr/guide/7_1/stream-source-reference.html#facet) it says “To support high cardinality aggregations see the rollup function”. I was wondering what it’s considered “high cardinality”. If it serves, our query returns up to 60k results. I haven’t got to do any benchmarking to see if there’s any difference, though, because facet so far performs very well, but I don’t know if I’m near the “tipping point”. Any feedback would be appreciated.
There's no hard and fast rule for this. The tipping point is going to
be different for every use case. With a little bit of information about
your setup, experienced users can make an educated guess about whether
or not performance will be good, but cannot say with absolute certainty
what you're going to run into.
Let's start with some definitions, which you may or may not already know:
https://en.wikipedia.org/wiki/Cardinality_(data_modeling)
https://en.wikipedia.org/wiki/Cardinality
You haven't said how many unique values are in your field. The only
information I have from you is 60K results from your queries, which may
or may not have any bearing on the total number of documents in your
index, or the total number of unique values in the field you're using
for faceting. So the next paragraph may or may not apply to your index.
In general, 60,000 unique values in a field would be considered very low
cardinality, because computers can typically operate on 60,000 values
*very* quickly, unless the size of each value is enormous. But if the
index has 60,000 total documents, then *in relation to other data*, the
cardinality is very high, even though most people would say the
opposite. Sixty thousand documents or unique values is almost always a
very small index, not prone to performance issues.
The warnings about cardinality in the Solr documentation mostly refer to
*absolute* cardinality -- how many unique values there are in a field,
regardless of the actual number of documents. If there are millions or
billions of unique values, then operations like facets, grouping,
sorting, etc are probably going to be slow. If there are a lot less,
such as thousands or only a handful, then those operations are likely to
be very fast, because the computer will have less information it must
process.
Thanks,
Shawn