Re: Too many values for UnInvertedField faceting on field topic

[Michael Jakl <ja...@gmail.com>]

Hi!

On Wed, Feb 29, 2012 at 22:21, Emmanuel Espina <es...@gmail.com> wrote:
> No. But probably we can find another way to do what you want. Please
> describe the problem and include some "numbers" to give us an idea of
> the sizes that you are handling. Number of documents, size of the
> index, etc.

Thank you! Our Solr holds currently about 168Mio documents. From each
of these documents we extract the most important keywords and store
them in a multivalued field (topic). Our goal is to provide faceted
navigation through these topics. The topic field holds roughly 5
values per doc, but I wasn't able to compute the correct number right
now.

The use cases require that the facets have to be calculated fast
enough so that they can be answered in reasonable time (1-2 secs)
which we were able to do with a 192GB RAM machine and regular warming.

Splitting the Solr into a few smaller ones (even on the same machine)
seems to be the most promising way, but I've been shying away from it
for some reasons: higher complexity, a huge reimport (though, I could
split the current index), some components didn't support it when we
were starting (Grouping was only introduced with 3.5 IIRC). I've
tested the sharding approach and it was a bit slower than the one huge
index approach.

I'd be happy to hear some suggestions,
Michael

Re: Too many values for UnInvertedField faceting on field topic

[Michael Jakl <ja...@gmail.com>]

Hi!

On Thu, Mar 1, 2012 at 23:54, Yonik Seeley <yo...@lucidimagination.com> wrote:
> On Thu, Mar 1, 2012 at 3:34 AM, Michael Jakl <ja...@gmail.com> wrote:
>> The topic field holds roughly 5
>> values per doc, but I wasn't able to compute the correct number right
>> now.
>
> How many unique values for that field in the whole index?
> If you have log output (or output from the stats page for
> fieldValueCache) that should tell you exactly.

I'm sorry, I've already reduced the size of the index and I'm in the
process of splitting it into a few shards. Solr couldn't build the
fieldValueCache for this particular field (that's where the exception
came from).

Thanks,
Michael

Re: Too many values for UnInvertedField faceting on field topic

[Yonik Seeley <yo...@lucidimagination.com>]

On Thu, Mar 1, 2012 at 3:34 AM, Michael Jakl <ja...@gmail.com> wrote:
> The topic field holds roughly 5
> values per doc, but I wasn't able to compute the correct number right
> now.

How many unique values for that field in the whole index?
If you have log output (or output from the stats page for
fieldValueCache) that should tell you exactly.

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10