HttpSolrServer commit is taking more time

[Venkateswarlu Bommineni <bv...@gmail.com>]

Hi,

I am new to Solr so need your help in solving below issue.

I am using SolrJ to add and commit the files into Solr.

But Solr commit is taking a long time.

for example: for 14000 records it is taking 4 min.

below is the code snippet.


*static final HttpSolrServer server = new HttpSolrServer(urlString);*
*server.add(<List of documents>);*
*server.commit();*

Could you please help me in resolving the issue.\


Thanks,
Venkat.

Re: HttpSolrServer commit is taking more time

[Venkateswarlu Bommineni <bv...@gmail.com>]

Thanks for the reply.

I can see same configuration as given in mail in Solr configuration file
But I can see same performance issues while querying also through solrJ.

Thanks,
Venkat.

On 21 Apr 2017 9:30 am, "Shawn Heisey" <ap...@elyograg.org> wrote:

> On 4/20/2017 9:23 PM, Venkateswarlu Bommineni wrote:
> > I am new to Solr so need your help in solving below issue.
> >
> > I am using SolrJ to add and commit the files into Solr.
> >
> > But Solr commit is taking a long time.
> >
> > for example: for 14000 records it is taking 4 min.
>
> Usually, extreme commit times like this have one of two causes:
>
> 1) The caches are very large and have a large autowarmCount.
>
> 2) The Solr heap is way too small, and the JVM is doing constant garbage
> collections.
>
> If queries are not having slowness issues, I would bet on option 1,
> although you may in fact be running into BOTH problems.
>
> This is what a typical example config looks like for one of the Solr
> caches:
>
>     <filterCache class="solr.FastLRUCache"
>                  size="512"
>                  initialSize="512"
>                  autowarmCount="0"/>
>
>
> This cache has a size of 512, but autowarmCount is zero.  This means
> that when a new searcher is created by a commit, none of the entries in
> the filterCache on the old searcher will make it to the cache on the new
> searcher.  If you change the autowarmCount value to say 4, then the top
> 4 filter queries in the cache will be re-executed on the new searcher,
> prepopulating the new cache with four entries.  If each of those four
> filters takes ten seconds to run, then warming that cache will take 40
> seconds.  I'm betting that somebody changed the autowarmCount values on
> the Solr caches to a high number in your configuration.    If that's the
> case, lower the number and reload/restart.
>
> Thanks,
> Shawn
>
>

Re: HttpSolrServer commit is taking more time

[Shawn Heisey <ap...@elyograg.org>]

On 4/20/2017 9:23 PM, Venkateswarlu Bommineni wrote:
> I am new to Solr so need your help in solving below issue.
>
> I am using SolrJ to add and commit the files into Solr.
>
> But Solr commit is taking a long time.
>
> for example: for 14000 records it is taking 4 min.

Usually, extreme commit times like this have one of two causes:

1) The caches are very large and have a large autowarmCount.

2) The Solr heap is way too small, and the JVM is doing constant garbage
collections.

If queries are not having slowness issues, I would bet on option 1,
although you may in fact be running into BOTH problems.

This is what a typical example config looks like for one of the Solr caches:

    <filterCache class="solr.FastLRUCache"
                 size="512"
                 initialSize="512"
                 autowarmCount="0"/>


This cache has a size of 512, but autowarmCount is zero.  This means
that when a new searcher is created by a commit, none of the entries in
the filterCache on the old searcher will make it to the cache on the new
searcher.  If you change the autowarmCount value to say 4, then the top
4 filter queries in the cache will be re-executed on the new searcher,
prepopulating the new cache with four entries.  If each of those four
filters takes ten seconds to run, then warming that cache will take 40
seconds.  I'm betting that somebody changed the autowarmCount values on
the Solr caches to a high number in your configuration.    If that's the
case, lower the number and reload/restart.

Thanks,
Shawn