You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Midas A <te...@gmail.com> on 2016/08/09 05:47:52 UTC
commit it taking 1300 ms
Hi ,
commit is taking more than 1300 ms . what should i check on server.
below is my configuration .
<autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>
${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
Re: commit it taking 1300 ms
Posted by Pushkar Raste <pu...@gmail.com>.
It would be worth looking into iostats of your disks.
On Aug 22, 2016 10:11 AM, "Alessandro Benedetti" <ab...@apache.org>
wrote:
> I agree with the suggestions so far.
> The cache auto-warming doesn't seem the problem as the index is not massive
> and the auto-warm is for only 10 docs.
> Are you using any warming query for the new searcher ?
>
> Are you using soft or hard commit ?
> This can make the difference ( soft are much cheaper, not free but cheaper)
> .
> You said :
> " Actually earlier it was taking less but suddenly it has increased "
>
> What happened ?
> Anyway, there are a lot of questions to answer before we can help you...
>
> Cheers
>
> On Fri, Aug 12, 2016 at 4:58 AM, Esther-Melaine Quansah <
> esther.quansah@lucidworks.com> wrote:
>
> > Midas,
> >
> > I’d like further clarification as well. Are you sending commits along
> with
> > each document that you’re POSTing to Solr? If so, you’re essentially
> either
> > opening a new searcher or flushing to disk with each POST which could
> > explain latency between each request.
> >
> > Thanks,
> >
> > Esther
> > > On Aug 11, 2016, at 12:19 PM, Erick Erickson <er...@gmail.com>
> > wrote:
> > >
> > > bq: we post json documents through the curl it takes the time (same
> > time i
> > > would like to say that we are not hard committing ). that curl takes
> time
> > > i.e. 1.3 sec.
> > >
> > > OK, I'm really confused. _what_ is taking 1.3 seconds? When you said
> > > commit, I was thinking of Solr's commit operation, which is totally
> > distinct
> > > from just adding a doc to the index. But I read the above statement
> > > as you're saying it takes 1.3 seconds just to send a doc to Solr.
> > >
> > > Let's see the exact curl command you're using please?
> > >
> > > Best,
> > > Erick
> > >
> > >
> > > On Thu, Aug 11, 2016 at 5:32 AM, Emir Arnautovic
> > > <em...@sematext.com> wrote:
> > >> Hi Midas,
> > >>
> > >> 1. How many indexing threads?
> > >> 2. Do you batch documents and what is your batch size?
> > >> 3. How frequently do you commit?
> > >>
> > >> I would recommend:
> > >> 1. Move commits to Solr (set auto soft commit to max allowed time)
> > >> 2. Use batches (bulks)
> > >> 3. tune bulk size and number of threads to achieve max performance.
> > >>
> > >> Thanks,
> > >> Emir
> > >>
> > >>
> > >>
> > >> On 11.08.2016 08:21, Midas A wrote:
> > >>>
> > >>> Emir,
> > >>>
> > >>> other queries:
> > >>>
> > >>> a) Solr cloud : NO
> > >>> b) <filterCache class="solr.FastLRUCache"
> > >>> size="5000" initialSize="5000" autowarmCount="10"/>
> > >>> c) <queryResultCache class="solr.LRUCache"
> > >>> size="1000" initialSize="1000" autowarmCount="10"/>
> > >>> d) <documentCache class="solr.LRUCache"
> > >>> size="1000" initialSize="1000" autowarmCount="10"/>
> > >>> e) we are using multi threaded system.
> > >>>
> > >>> On Thu, Aug 11, 2016 at 11:48 AM, Midas A <te...@gmail.com>
> > wrote:
> > >>>
> > >>>> Emir,
> > >>>>
> > >>>> we post json documents through the curl it takes the time (same
> time i
> > >>>> would like to say that we are not hard committing ). that curl takes
> > time
> > >>>> i.e. 1.3 sec.
> > >>>>
> > >>>> On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic <
> > >>>> emir.arnautovic@sematext.com> wrote:
> > >>>>
> > >>>>> Hi Midas,
> > >>>>>
> > >>>>> According to your autocommit configuration and your worry about
> > commit
> > >>>>> time I assume that you are doing explicit commits from client code
> > and
> > >>>>> that
> > >>>>> 1.3s is client observed commit time. If that is the case, than it
> > might
> > >>>>> be
> > >>>>> opening searcher that is taking time.
> > >>>>>
> > >>>>> How do you index data - single threaded or multithreaded? How
> > frequently
> > >>>>> do you commit from client? Can you let Solr do soft commits instead
> > of
> > >>>>> explicitly committing? Do you have warmup queries? Is this
> SolrCloud?
> > >>>>> What
> > >>>>> is number of servers (what spec), shards, docs?
> > >>>>>
> > >>>>> In any case monitoring can give you more info about server/Solr
> > behavior
> > >>>>> and help you diagnose issues more easily/precisely. One such
> > monitoring
> > >>>>> tool is our SPM <http://sematext.com/spm>.
> > >>>>>
> > >>>>> Regards,
> > >>>>> Emir
> > >>>>>
> > >>>>> --
> > >>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > >>>>> Solr & Elasticsearch Support * http://sematext.com/
> > >>>>>
> > >>>>> On 10.08.2016 05:20, Midas A wrote:
> > >>>>>
> > >>>>>> Thanks for replying
> > >>>>>>
> > >>>>>> index size:9GB
> > >>>>>> 2000 docs/sec.
> > >>>>>>
> > >>>>>> Actually earlier it was taking less but suddenly it has increased
> .
> > >>>>>>
> > >>>>>> Currently we do not have any monitoring tool.
> > >>>>>>
> > >>>>>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
> > >>>>>> emir.arnautovic@sematext.com> wrote:
> > >>>>>>
> > >>>>>> Hi Midas,
> > >>>>>>>
> > >>>>>>> Can you give us more details on your index: size, number of new
> > docs
> > >>>>>>> between commits. Why do you think 1.3s for commit is to much and
> > why
> > >>>>>>> do
> > >>>>>>> you
> > >>>>>>> need it to take less? Did you do any system/Solr monitoring?
> > >>>>>>>
> > >>>>>>> Emir
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On 09.08.2016 14:10, Midas A wrote:
> > >>>>>>>
> > >>>>>>> please reply it is urgent.
> > >>>>>>>>
> > >>>>>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi ,
> > >>>>>>>>
> > >>>>>>>>> commit is taking more than 1300 ms . what should i check on
> > server.
> > >>>>>>>>>
> > >>>>>>>>> below is my configuration .
> > >>>>>>>>>
> > >>>>>>>>> <autoCommit> <maxTime>${solr.autoCommit.
> maxTime:15000}</maxTime>
> > <
> > >>>>>>>>> openSearcher>false</openSearcher> </autoCommit>
> <autoSoftCommit>
> > >>>>>>>>> <maxTime>
> > >>>>>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>
> > >>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > >>>>>>> Solr & Elasticsearch Support * http://sematext.com/
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>
> > >> --
> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > >> Solr & Elasticsearch Support * http://sematext.com/
> > >>
> >
> >
>
>
> --
> --------------------------
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>
Re: commit it taking 1300 ms
Posted by Alessandro Benedetti <ab...@apache.org>.
I agree with the suggestions so far.
The cache auto-warming doesn't seem the problem as the index is not massive
and the auto-warm is for only 10 docs.
Are you using any warming query for the new searcher ?
Are you using soft or hard commit ?
This can make the difference ( soft are much cheaper, not free but cheaper)
.
You said :
" Actually earlier it was taking less but suddenly it has increased "
What happened ?
Anyway, there are a lot of questions to answer before we can help you...
Cheers
On Fri, Aug 12, 2016 at 4:58 AM, Esther-Melaine Quansah <
esther.quansah@lucidworks.com> wrote:
> Midas,
>
> I’d like further clarification as well. Are you sending commits along with
> each document that you’re POSTing to Solr? If so, you’re essentially either
> opening a new searcher or flushing to disk with each POST which could
> explain latency between each request.
>
> Thanks,
>
> Esther
> > On Aug 11, 2016, at 12:19 PM, Erick Erickson <er...@gmail.com>
> wrote:
> >
> > bq: we post json documents through the curl it takes the time (same
> time i
> > would like to say that we are not hard committing ). that curl takes time
> > i.e. 1.3 sec.
> >
> > OK, I'm really confused. _what_ is taking 1.3 seconds? When you said
> > commit, I was thinking of Solr's commit operation, which is totally
> distinct
> > from just adding a doc to the index. But I read the above statement
> > as you're saying it takes 1.3 seconds just to send a doc to Solr.
> >
> > Let's see the exact curl command you're using please?
> >
> > Best,
> > Erick
> >
> >
> > On Thu, Aug 11, 2016 at 5:32 AM, Emir Arnautovic
> > <em...@sematext.com> wrote:
> >> Hi Midas,
> >>
> >> 1. How many indexing threads?
> >> 2. Do you batch documents and what is your batch size?
> >> 3. How frequently do you commit?
> >>
> >> I would recommend:
> >> 1. Move commits to Solr (set auto soft commit to max allowed time)
> >> 2. Use batches (bulks)
> >> 3. tune bulk size and number of threads to achieve max performance.
> >>
> >> Thanks,
> >> Emir
> >>
> >>
> >>
> >> On 11.08.2016 08:21, Midas A wrote:
> >>>
> >>> Emir,
> >>>
> >>> other queries:
> >>>
> >>> a) Solr cloud : NO
> >>> b) <filterCache class="solr.FastLRUCache"
> >>> size="5000" initialSize="5000" autowarmCount="10"/>
> >>> c) <queryResultCache class="solr.LRUCache"
> >>> size="1000" initialSize="1000" autowarmCount="10"/>
> >>> d) <documentCache class="solr.LRUCache"
> >>> size="1000" initialSize="1000" autowarmCount="10"/>
> >>> e) we are using multi threaded system.
> >>>
> >>> On Thu, Aug 11, 2016 at 11:48 AM, Midas A <te...@gmail.com>
> wrote:
> >>>
> >>>> Emir,
> >>>>
> >>>> we post json documents through the curl it takes the time (same time i
> >>>> would like to say that we are not hard committing ). that curl takes
> time
> >>>> i.e. 1.3 sec.
> >>>>
> >>>> On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic <
> >>>> emir.arnautovic@sematext.com> wrote:
> >>>>
> >>>>> Hi Midas,
> >>>>>
> >>>>> According to your autocommit configuration and your worry about
> commit
> >>>>> time I assume that you are doing explicit commits from client code
> and
> >>>>> that
> >>>>> 1.3s is client observed commit time. If that is the case, than it
> might
> >>>>> be
> >>>>> opening searcher that is taking time.
> >>>>>
> >>>>> How do you index data - single threaded or multithreaded? How
> frequently
> >>>>> do you commit from client? Can you let Solr do soft commits instead
> of
> >>>>> explicitly committing? Do you have warmup queries? Is this SolrCloud?
> >>>>> What
> >>>>> is number of servers (what spec), shards, docs?
> >>>>>
> >>>>> In any case monitoring can give you more info about server/Solr
> behavior
> >>>>> and help you diagnose issues more easily/precisely. One such
> monitoring
> >>>>> tool is our SPM <http://sematext.com/spm>.
> >>>>>
> >>>>> Regards,
> >>>>> Emir
> >>>>>
> >>>>> --
> >>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> >>>>> Solr & Elasticsearch Support * http://sematext.com/
> >>>>>
> >>>>> On 10.08.2016 05:20, Midas A wrote:
> >>>>>
> >>>>>> Thanks for replying
> >>>>>>
> >>>>>> index size:9GB
> >>>>>> 2000 docs/sec.
> >>>>>>
> >>>>>> Actually earlier it was taking less but suddenly it has increased .
> >>>>>>
> >>>>>> Currently we do not have any monitoring tool.
> >>>>>>
> >>>>>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
> >>>>>> emir.arnautovic@sematext.com> wrote:
> >>>>>>
> >>>>>> Hi Midas,
> >>>>>>>
> >>>>>>> Can you give us more details on your index: size, number of new
> docs
> >>>>>>> between commits. Why do you think 1.3s for commit is to much and
> why
> >>>>>>> do
> >>>>>>> you
> >>>>>>> need it to take less? Did you do any system/Solr monitoring?
> >>>>>>>
> >>>>>>> Emir
> >>>>>>>
> >>>>>>>
> >>>>>>> On 09.08.2016 14:10, Midas A wrote:
> >>>>>>>
> >>>>>>> please reply it is urgent.
> >>>>>>>>
> >>>>>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi ,
> >>>>>>>>
> >>>>>>>>> commit is taking more than 1300 ms . what should i check on
> server.
> >>>>>>>>>
> >>>>>>>>> below is my configuration .
> >>>>>>>>>
> >>>>>>>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> <
> >>>>>>>>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit>
> >>>>>>>>> <maxTime>
> >>>>>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>
> >>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> >>>>>>> Solr & Elasticsearch Support * http://sematext.com/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>
> >> --
> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> Solr & Elasticsearch Support * http://sematext.com/
> >>
>
>
--
--------------------------
Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: commit it taking 1300 ms
Posted by Esther-Melaine Quansah <es...@lucidworks.com>.
Midas,
I’d like further clarification as well. Are you sending commits along with each document that you’re POSTing to Solr? If so, you’re essentially either opening a new searcher or flushing to disk with each POST which could explain latency between each request.
Thanks,
Esther
> On Aug 11, 2016, at 12:19 PM, Erick Erickson <er...@gmail.com> wrote:
>
> bq: we post json documents through the curl it takes the time (same time i
> would like to say that we are not hard committing ). that curl takes time
> i.e. 1.3 sec.
>
> OK, I'm really confused. _what_ is taking 1.3 seconds? When you said
> commit, I was thinking of Solr's commit operation, which is totally distinct
> from just adding a doc to the index. But I read the above statement
> as you're saying it takes 1.3 seconds just to send a doc to Solr.
>
> Let's see the exact curl command you're using please?
>
> Best,
> Erick
>
>
> On Thu, Aug 11, 2016 at 5:32 AM, Emir Arnautovic
> <em...@sematext.com> wrote:
>> Hi Midas,
>>
>> 1. How many indexing threads?
>> 2. Do you batch documents and what is your batch size?
>> 3. How frequently do you commit?
>>
>> I would recommend:
>> 1. Move commits to Solr (set auto soft commit to max allowed time)
>> 2. Use batches (bulks)
>> 3. tune bulk size and number of threads to achieve max performance.
>>
>> Thanks,
>> Emir
>>
>>
>>
>> On 11.08.2016 08:21, Midas A wrote:
>>>
>>> Emir,
>>>
>>> other queries:
>>>
>>> a) Solr cloud : NO
>>> b) <filterCache class="solr.FastLRUCache"
>>> size="5000" initialSize="5000" autowarmCount="10"/>
>>> c) <queryResultCache class="solr.LRUCache"
>>> size="1000" initialSize="1000" autowarmCount="10"/>
>>> d) <documentCache class="solr.LRUCache"
>>> size="1000" initialSize="1000" autowarmCount="10"/>
>>> e) we are using multi threaded system.
>>>
>>> On Thu, Aug 11, 2016 at 11:48 AM, Midas A <te...@gmail.com> wrote:
>>>
>>>> Emir,
>>>>
>>>> we post json documents through the curl it takes the time (same time i
>>>> would like to say that we are not hard committing ). that curl takes time
>>>> i.e. 1.3 sec.
>>>>
>>>> On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic <
>>>> emir.arnautovic@sematext.com> wrote:
>>>>
>>>>> Hi Midas,
>>>>>
>>>>> According to your autocommit configuration and your worry about commit
>>>>> time I assume that you are doing explicit commits from client code and
>>>>> that
>>>>> 1.3s is client observed commit time. If that is the case, than it might
>>>>> be
>>>>> opening searcher that is taking time.
>>>>>
>>>>> How do you index data - single threaded or multithreaded? How frequently
>>>>> do you commit from client? Can you let Solr do soft commits instead of
>>>>> explicitly committing? Do you have warmup queries? Is this SolrCloud?
>>>>> What
>>>>> is number of servers (what spec), shards, docs?
>>>>>
>>>>> In any case monitoring can give you more info about server/Solr behavior
>>>>> and help you diagnose issues more easily/precisely. One such monitoring
>>>>> tool is our SPM <http://sematext.com/spm>.
>>>>>
>>>>> Regards,
>>>>> Emir
>>>>>
>>>>> --
>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>
>>>>> On 10.08.2016 05:20, Midas A wrote:
>>>>>
>>>>>> Thanks for replying
>>>>>>
>>>>>> index size:9GB
>>>>>> 2000 docs/sec.
>>>>>>
>>>>>> Actually earlier it was taking less but suddenly it has increased .
>>>>>>
>>>>>> Currently we do not have any monitoring tool.
>>>>>>
>>>>>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
>>>>>> emir.arnautovic@sematext.com> wrote:
>>>>>>
>>>>>> Hi Midas,
>>>>>>>
>>>>>>> Can you give us more details on your index: size, number of new docs
>>>>>>> between commits. Why do you think 1.3s for commit is to much and why
>>>>>>> do
>>>>>>> you
>>>>>>> need it to take less? Did you do any system/Solr monitoring?
>>>>>>>
>>>>>>> Emir
>>>>>>>
>>>>>>>
>>>>>>> On 09.08.2016 14:10, Midas A wrote:
>>>>>>>
>>>>>>> please reply it is urgent.
>>>>>>>>
>>>>>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi ,
>>>>>>>>
>>>>>>>>> commit is taking more than 1300 ms . what should i check on server.
>>>>>>>>>
>>>>>>>>> below is my configuration .
>>>>>>>>>
>>>>>>>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
>>>>>>>>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit>
>>>>>>>>> <maxTime>
>>>>>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>
>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
Re: commit it taking 1300 ms
Posted by Erick Erickson <er...@gmail.com>.
bq: we post json documents through the curl it takes the time (same time i
would like to say that we are not hard committing ). that curl takes time
i.e. 1.3 sec.
OK, I'm really confused. _what_ is taking 1.3 seconds? When you said
commit, I was thinking of Solr's commit operation, which is totally distinct
from just adding a doc to the index. But I read the above statement
as you're saying it takes 1.3 seconds just to send a doc to Solr.
Let's see the exact curl command you're using please?
Best,
Erick
On Thu, Aug 11, 2016 at 5:32 AM, Emir Arnautovic
<em...@sematext.com> wrote:
> Hi Midas,
>
> 1. How many indexing threads?
> 2. Do you batch documents and what is your batch size?
> 3. How frequently do you commit?
>
> I would recommend:
> 1. Move commits to Solr (set auto soft commit to max allowed time)
> 2. Use batches (bulks)
> 3. tune bulk size and number of threads to achieve max performance.
>
> Thanks,
> Emir
>
>
>
> On 11.08.2016 08:21, Midas A wrote:
>>
>> Emir,
>>
>> other queries:
>>
>> a) Solr cloud : NO
>> b) <filterCache class="solr.FastLRUCache"
>> size="5000" initialSize="5000" autowarmCount="10"/>
>> c) <queryResultCache class="solr.LRUCache"
>> size="1000" initialSize="1000" autowarmCount="10"/>
>> d) <documentCache class="solr.LRUCache"
>> size="1000" initialSize="1000" autowarmCount="10"/>
>> e) we are using multi threaded system.
>>
>> On Thu, Aug 11, 2016 at 11:48 AM, Midas A <te...@gmail.com> wrote:
>>
>>> Emir,
>>>
>>> we post json documents through the curl it takes the time (same time i
>>> would like to say that we are not hard committing ). that curl takes time
>>> i.e. 1.3 sec.
>>>
>>> On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic <
>>> emir.arnautovic@sematext.com> wrote:
>>>
>>>> Hi Midas,
>>>>
>>>> According to your autocommit configuration and your worry about commit
>>>> time I assume that you are doing explicit commits from client code and
>>>> that
>>>> 1.3s is client observed commit time. If that is the case, than it might
>>>> be
>>>> opening searcher that is taking time.
>>>>
>>>> How do you index data - single threaded or multithreaded? How frequently
>>>> do you commit from client? Can you let Solr do soft commits instead of
>>>> explicitly committing? Do you have warmup queries? Is this SolrCloud?
>>>> What
>>>> is number of servers (what spec), shards, docs?
>>>>
>>>> In any case monitoring can give you more info about server/Solr behavior
>>>> and help you diagnose issues more easily/precisely. One such monitoring
>>>> tool is our SPM <http://sematext.com/spm>.
>>>>
>>>> Regards,
>>>> Emir
>>>>
>>>> --
>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>> On 10.08.2016 05:20, Midas A wrote:
>>>>
>>>>> Thanks for replying
>>>>>
>>>>> index size:9GB
>>>>> 2000 docs/sec.
>>>>>
>>>>> Actually earlier it was taking less but suddenly it has increased .
>>>>>
>>>>> Currently we do not have any monitoring tool.
>>>>>
>>>>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
>>>>> emir.arnautovic@sematext.com> wrote:
>>>>>
>>>>> Hi Midas,
>>>>>>
>>>>>> Can you give us more details on your index: size, number of new docs
>>>>>> between commits. Why do you think 1.3s for commit is to much and why
>>>>>> do
>>>>>> you
>>>>>> need it to take less? Did you do any system/Solr monitoring?
>>>>>>
>>>>>> Emir
>>>>>>
>>>>>>
>>>>>> On 09.08.2016 14:10, Midas A wrote:
>>>>>>
>>>>>> please reply it is urgent.
>>>>>>>
>>>>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi ,
>>>>>>>
>>>>>>>> commit is taking more than 1300 ms . what should i check on server.
>>>>>>>>
>>>>>>>> below is my configuration .
>>>>>>>>
>>>>>>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
>>>>>>>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit>
>>>>>>>> <maxTime>
>>>>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>
>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>>
>>>>>>
>>>>>>
>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
Re: commit it taking 1300 ms
Posted by Emir Arnautovic <em...@sematext.com>.
Hi Midas,
1. How many indexing threads?
2. Do you batch documents and what is your batch size?
3. How frequently do you commit?
I would recommend:
1. Move commits to Solr (set auto soft commit to max allowed time)
2. Use batches (bulks)
3. tune bulk size and number of threads to achieve max performance.
Thanks,
Emir
On 11.08.2016 08:21, Midas A wrote:
> Emir,
>
> other queries:
>
> a) Solr cloud : NO
> b) <filterCache class="solr.FastLRUCache"
> size="5000" initialSize="5000" autowarmCount="10"/>
> c) <queryResultCache class="solr.LRUCache"
> size="1000" initialSize="1000" autowarmCount="10"/>
> d) <documentCache class="solr.LRUCache"
> size="1000" initialSize="1000" autowarmCount="10"/>
> e) we are using multi threaded system.
>
> On Thu, Aug 11, 2016 at 11:48 AM, Midas A <te...@gmail.com> wrote:
>
>> Emir,
>>
>> we post json documents through the curl it takes the time (same time i
>> would like to say that we are not hard committing ). that curl takes time
>> i.e. 1.3 sec.
>>
>> On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic <
>> emir.arnautovic@sematext.com> wrote:
>>
>>> Hi Midas,
>>>
>>> According to your autocommit configuration and your worry about commit
>>> time I assume that you are doing explicit commits from client code and that
>>> 1.3s is client observed commit time. If that is the case, than it might be
>>> opening searcher that is taking time.
>>>
>>> How do you index data - single threaded or multithreaded? How frequently
>>> do you commit from client? Can you let Solr do soft commits instead of
>>> explicitly committing? Do you have warmup queries? Is this SolrCloud? What
>>> is number of servers (what spec), shards, docs?
>>>
>>> In any case monitoring can give you more info about server/Solr behavior
>>> and help you diagnose issues more easily/precisely. One such monitoring
>>> tool is our SPM <http://sematext.com/spm>.
>>>
>>> Regards,
>>> Emir
>>>
>>> --
>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>> On 10.08.2016 05:20, Midas A wrote:
>>>
>>>> Thanks for replying
>>>>
>>>> index size:9GB
>>>> 2000 docs/sec.
>>>>
>>>> Actually earlier it was taking less but suddenly it has increased .
>>>>
>>>> Currently we do not have any monitoring tool.
>>>>
>>>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
>>>> emir.arnautovic@sematext.com> wrote:
>>>>
>>>> Hi Midas,
>>>>> Can you give us more details on your index: size, number of new docs
>>>>> between commits. Why do you think 1.3s for commit is to much and why do
>>>>> you
>>>>> need it to take less? Did you do any system/Solr monitoring?
>>>>>
>>>>> Emir
>>>>>
>>>>>
>>>>> On 09.08.2016 14:10, Midas A wrote:
>>>>>
>>>>> please reply it is urgent.
>>>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com> wrote:
>>>>>>
>>>>>> Hi ,
>>>>>>
>>>>>>> commit is taking more than 1300 ms . what should i check on server.
>>>>>>>
>>>>>>> below is my configuration .
>>>>>>>
>>>>>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
>>>>>>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit>
>>>>>>> <maxTime>
>>>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>
>>>>>
>>>>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: commit it taking 1300 ms
Posted by Midas A <te...@gmail.com>.
Emir,
other queries:
a) Solr cloud : NO
b) <filterCache class="solr.FastLRUCache"
size="5000" initialSize="5000" autowarmCount="10"/>
c) <queryResultCache class="solr.LRUCache"
size="1000" initialSize="1000" autowarmCount="10"/>
d) <documentCache class="solr.LRUCache"
size="1000" initialSize="1000" autowarmCount="10"/>
e) we are using multi threaded system.
On Thu, Aug 11, 2016 at 11:48 AM, Midas A <te...@gmail.com> wrote:
> Emir,
>
> we post json documents through the curl it takes the time (same time i
> would like to say that we are not hard committing ). that curl takes time
> i.e. 1.3 sec.
>
> On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic <
> emir.arnautovic@sematext.com> wrote:
>
>> Hi Midas,
>>
>> According to your autocommit configuration and your worry about commit
>> time I assume that you are doing explicit commits from client code and that
>> 1.3s is client observed commit time. If that is the case, than it might be
>> opening searcher that is taking time.
>>
>> How do you index data - single threaded or multithreaded? How frequently
>> do you commit from client? Can you let Solr do soft commits instead of
>> explicitly committing? Do you have warmup queries? Is this SolrCloud? What
>> is number of servers (what spec), shards, docs?
>>
>> In any case monitoring can give you more info about server/Solr behavior
>> and help you diagnose issues more easily/precisely. One such monitoring
>> tool is our SPM <http://sematext.com/spm>.
>>
>> Regards,
>> Emir
>>
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>> On 10.08.2016 05:20, Midas A wrote:
>>
>>> Thanks for replying
>>>
>>> index size:9GB
>>> 2000 docs/sec.
>>>
>>> Actually earlier it was taking less but suddenly it has increased .
>>>
>>> Currently we do not have any monitoring tool.
>>>
>>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
>>> emir.arnautovic@sematext.com> wrote:
>>>
>>> Hi Midas,
>>>>
>>>> Can you give us more details on your index: size, number of new docs
>>>> between commits. Why do you think 1.3s for commit is to much and why do
>>>> you
>>>> need it to take less? Did you do any system/Solr monitoring?
>>>>
>>>> Emir
>>>>
>>>>
>>>> On 09.08.2016 14:10, Midas A wrote:
>>>>
>>>> please reply it is urgent.
>>>>>
>>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com> wrote:
>>>>>
>>>>> Hi ,
>>>>>
>>>>>> commit is taking more than 1300 ms . what should i check on server.
>>>>>>
>>>>>> below is my configuration .
>>>>>>
>>>>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
>>>>>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit>
>>>>>> <maxTime>
>>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>
>>>>
>>>>
>
Re: commit it taking 1300 ms
Posted by Midas A <te...@gmail.com>.
Emir,
we post json documents through the curl it takes the time (same time i
would like to say that we are not hard committing ). that curl takes time
i.e. 1.3 sec.
On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic <
emir.arnautovic@sematext.com> wrote:
> Hi Midas,
>
> According to your autocommit configuration and your worry about commit
> time I assume that you are doing explicit commits from client code and that
> 1.3s is client observed commit time. If that is the case, than it might be
> opening searcher that is taking time.
>
> How do you index data - single threaded or multithreaded? How frequently
> do you commit from client? Can you let Solr do soft commits instead of
> explicitly committing? Do you have warmup queries? Is this SolrCloud? What
> is number of servers (what spec), shards, docs?
>
> In any case monitoring can give you more info about server/Solr behavior
> and help you diagnose issues more easily/precisely. One such monitoring
> tool is our SPM <http://sematext.com/spm>.
>
> Regards,
> Emir
>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
> On 10.08.2016 05:20, Midas A wrote:
>
>> Thanks for replying
>>
>> index size:9GB
>> 2000 docs/sec.
>>
>> Actually earlier it was taking less but suddenly it has increased .
>>
>> Currently we do not have any monitoring tool.
>>
>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
>> emir.arnautovic@sematext.com> wrote:
>>
>> Hi Midas,
>>>
>>> Can you give us more details on your index: size, number of new docs
>>> between commits. Why do you think 1.3s for commit is to much and why do
>>> you
>>> need it to take less? Did you do any system/Solr monitoring?
>>>
>>> Emir
>>>
>>>
>>> On 09.08.2016 14:10, Midas A wrote:
>>>
>>> please reply it is urgent.
>>>>
>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com> wrote:
>>>>
>>>> Hi ,
>>>>
>>>>> commit is taking more than 1300 ms . what should i check on server.
>>>>>
>>>>> below is my configuration .
>>>>>
>>>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
>>>>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit>
>>>>> <maxTime>
>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
>>>>>
>>>>>
>>>>>
>>>>> --
>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>>
>>>
Re: commit it taking 1300 ms
Posted by Emir Arnautovic <em...@sematext.com>.
Hi Midas,
According to your autocommit configuration and your worry about commit
time I assume that you are doing explicit commits from client code and
that 1.3s is client observed commit time. If that is the case, than it
might be opening searcher that is taking time.
How do you index data - single threaded or multithreaded? How frequently
do you commit from client? Can you let Solr do soft commits instead of
explicitly committing? Do you have warmup queries? Is this SolrCloud?
What is number of servers (what spec), shards, docs?
In any case monitoring can give you more info about server/Solr behavior
and help you diagnose issues more easily/precisely. One such monitoring
tool is our SPM <http://sematext.com/spm>.
Regards,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On 10.08.2016 05:20, Midas A wrote:
> Thanks for replying
>
> index size:9GB
> 2000 docs/sec.
>
> Actually earlier it was taking less but suddenly it has increased .
>
> Currently we do not have any monitoring tool.
>
> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
> emir.arnautovic@sematext.com> wrote:
>
>> Hi Midas,
>>
>> Can you give us more details on your index: size, number of new docs
>> between commits. Why do you think 1.3s for commit is to much and why do you
>> need it to take less? Did you do any system/Solr monitoring?
>>
>> Emir
>>
>>
>> On 09.08.2016 14:10, Midas A wrote:
>>
>>> please reply it is urgent.
>>>
>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com> wrote:
>>>
>>> Hi ,
>>>> commit is taking more than 1300 ms . what should i check on server.
>>>>
>>>> below is my configuration .
>>>>
>>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
>>>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit>
>>>> <maxTime>
>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
>>>>
>>>>
>>>>
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
Re: commit it taking 1300 ms
Posted by Midas A <te...@gmail.com>.
Thanks for replying
index size:9GB
2000 docs/sec.
Actually earlier it was taking less but suddenly it has increased .
Currently we do not have any monitoring tool.
On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
emir.arnautovic@sematext.com> wrote:
> Hi Midas,
>
> Can you give us more details on your index: size, number of new docs
> between commits. Why do you think 1.3s for commit is to much and why do you
> need it to take less? Did you do any system/Solr monitoring?
>
> Emir
>
>
> On 09.08.2016 14:10, Midas A wrote:
>
>> please reply it is urgent.
>>
>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com> wrote:
>>
>> Hi ,
>>>
>>> commit is taking more than 1300 ms . what should i check on server.
>>>
>>> below is my configuration .
>>>
>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
>>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit>
>>> <maxTime>
>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
>>>
>>>
>>>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
Re: commit it taking 1300 ms
Posted by Emir Arnautovic <em...@sematext.com>.
Hi Midas,
Can you give us more details on your index: size, number of new docs
between commits. Why do you think 1.3s for commit is to much and why do
you need it to take less? Did you do any system/Solr monitoring?
Emir
On 09.08.2016 14:10, Midas A wrote:
> please reply it is urgent.
>
> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com> wrote:
>
>> Hi ,
>>
>> commit is taking more than 1300 ms . what should i check on server.
>>
>> below is my configuration .
>>
>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
>> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>
>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
>>
>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: commit it taking 1300 ms
Posted by Midas A <te...@gmail.com>.
please reply it is urgent.
On Tue, Aug 9, 2016 at 11:17 AM, Midas A <te...@gmail.com> wrote:
> Hi ,
>
> commit is taking more than 1300 ms . what should i check on server.
>
> below is my configuration .
>
> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> <
> openSearcher>false</openSearcher> </autoCommit> <autoSoftCommit> <maxTime>
> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit>
>
>