FW: Performance of filterCache for Faceting - single value & tokenized

[Fuad Efendi <fu...@efendi.ca>]

Ciao-Ciao!

I did something strange and website (www.tokenizer.org) performs 1000 times
faster now (it is still in my basement via ASDL 600kbps upload asychronous)
Thank you for supporting SOLR!

filterCache:
Size: 1051311

What I did: single-valued fields for Category and ItemName. Category field
is tokenized (with custom analyzer), and I updated only 30% of Lucene index,
but it was more than enough for huge performance improvements. Before that,
due to some mistakes in parsing HTML and etc., I had *multivalued* field for
facets; I tried to minimize amount of tokens (sorry for possibly wrong
terminology). Field for facets is still tokenized, but it is single-value.

Before: after Commit/Optimize and Server Restart first query took 5-7
minutes to execute
After: only 2 seconds!

I was browsing Lucene fieldCache, unfortunately it's not applicable for
tokenized fields... 

filterCache size is almost the same as before, but it works much faster.


Thanks,

Fuad Efendi
(416)993-2060(cell)
(416)761-1940(home)
Tokenizer Inc.
http://www.tokenizer.org