[Solr 3.5] Facets and stats become a lot slower during concurrent inserts

[Lyuba Romanchuk <ly...@gmail.com>]

Hi,

I test facets and stats in Solr 3.5 and I see that queries are running a
lot slower during inserts into index with more than 15M documents .
If I stop to insert new documents facet/stats queries run 10-1000 times
faster than with concurrent inserts.
I don't see this degradation in Lucene.

Could you please explain what may cause this?
Is it Solr related issue only?

Thank you for help.

Best regards,
Lyuba

Re: [Solr 3.5] Facets and stats become a lot slower during concurrent inserts

[Chris Hostetter <ho...@fucit.org>]

: autoCommit is disabled in solrconfig.xml and I use
: SolrServer::addBeans(beans, 100) for inserts.

have you looked at the javadocs for that method?

https://lucene.apache.org/solr/api/org/apache/solr/client/solrj/SolrServer.html#addBean%28java.lang.Object,%20int%29

  public UpdateResponse addBean(Object obj,
                                int commitWithinMs)
  Parameters:
    obj - the input bean
    commitWithinMs - max time (in ms) before a commit will happen

...so you are in fact asking solr to do a commit within 0.1 second of 
every document you add.

: > Are you also doing commits (or have autocommit enabled)?
: > The first time a facet command is used for a field after a commit,
: > certain data structures need to be constructed.
: > To avoid slow first requests like this, you can add a request that
: > does the faceting as a static warming query that will be run before
: > any live queries use the new searcher.


-Hoss

Re: [Solr 3.5] Facets and stats become a lot slower during concurrent inserts

[Lyuba Romanchuk <ly...@gmail.com>]

autoCommit is disabled in solrconfig.xml and I use
SolrServer::addBeans(beans, 100) for inserts.
I need to insert new documents continually in high rate with concurrent
running queries.

Best regards,
Lyuba

On Tue, Dec 27, 2011 at 6:15 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:

> On Tue, Dec 27, 2011 at 10:43 AM, Lyuba Romanchuk
> <ly...@gmail.com> wrote:
> > I test facets and stats in Solr 3.5 and I see that queries are running a
> > lot slower during inserts into index with more than 15M documents .
>
> Are you also doing commits (or have autocommit enabled)?
> The first time a facet command is used for a field after a commit,
> certain data structures need to be constructed.
> To avoid slow first requests like this, you can add a request that
> does the faceting as a static warming query that will be run before
> any live queries use the new searcher.
>
> -Yonik
> http://www.lucidimagination.com
>

Re: [Solr 3.5] Facets and stats become a lot slower during concurrent inserts

[Yonik Seeley <yo...@lucidimagination.com>]

On Tue, Dec 27, 2011 at 10:43 AM, Lyuba Romanchuk
<ly...@gmail.com> wrote:
> I test facets and stats in Solr 3.5 and I see that queries are running a
> lot slower during inserts into index with more than 15M documents .

Are you also doing commits (or have autocommit enabled)?
The first time a facet command is used for a field after a commit,
certain data structures need to be constructed.
To avoid slow first requests like this, you can add a request that
does the faceting as a static warming query that will be run before
any live queries use the new searcher.

-Yonik
http://www.lucidimagination.com