You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Gili Nachum <gi...@gmail.com> on 2015/02/08 12:48:37 UTC
43sec commit duration - blocked by index merge events?
Hello,
During a load test I noticed a commit that took 43 seconds to complete
(client hard complete).
Is this to be expected? What's causing it?
I have a pair of machines hosting a 128M docs collection (8 shards,
replication factor=2).
Could it be merges? In Lucene merges happen async of commit statements, but
reading Solr's doc for Update Hanlder
<https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig>
it sounds like hard commits do wait for merges to occur: *" The tradeoff is
that a soft commit gives you faster visibility because it's not waiting for
background merges to finish."*
Thanks.
Re: 43sec commit duration - blocked by index merge events?
Posted by Timothy Potter <th...@gmail.com>.
I think Mark found something similar -
https://issues.apache.org/jira/browse/SOLR-6838
On Sat, Feb 14, 2015 at 2:05 AM, Erick Erickson <er...@gmail.com>
wrote:
> Exactly how are you issuing the commit? I'm assuming you're
> using SolrJ. the server.commit(whatever, true) waits for the searcher
> to be opened before returning. This includes (I believe) warmup
> times. It could be that the warmup times are huge in your case, the
> solr logs should show you the autowarm times for a new searcher.
>
> Best,
> Erick
>
> On Fri, Feb 13, 2015 at 2:53 PM, Jack Krupansky
> <ja...@gmail.com> wrote:
> > I wasn't able to follow Otis' answer but... the purpose of commit is to
> > make make recent document changes (since the last commit) visible to
> > queries, and has nothing to do with merging of segments. IOW, take the
> new
> > segment that is being created and not yet ready for use by query, and
> > finish it so that query can access it. Soft commit vs. hard commit is
> > simply a matter of whether Solr will wait for the I/O to write the new
> > segment to disk to complete. Merging is an independent, background
> > procedure (thread) that merges existing segments. It does seem odd that
> the
> > cited doc does say that soft commit waits for background merges! (Hoss??)
> >
> > -- Jack Krupansky
> >
> > On Fri, Feb 13, 2015 at 4:47 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> >> Check
> >> http://search-lucene.com/?q=commit+wait+block&fc_type=mail+_hash_+user
> >>
> >> e.g. http://search-lucene.com/m/QTPa7Sqx81
> >>
> >> Otis
> >> --
> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> Solr & Elasticsearch Support * http://sematext.com/
> >>
> >>
> >> On Fri, Feb 13, 2015 at 8:50 AM, Gili Nachum <gi...@gmail.com>
> wrote:
> >>
> >> > Thanks Otis, can you confirm that a commit call will wait for merges
> to
> >> > complete before returning?
> >> >
> >> > On Thu, Feb 12, 2015 at 8:46 PM, Otis Gospodnetic <
> >> > otis.gospodnetic@gmail.com> wrote:
> >> >
> >> > > If you are using Solr and SPM for Solr, you can check a report that
> >> shows
> >> > > the # of files in an index and the report that shows you the max
> >> docs-num
> >> > > docs delta. If you see the # of files drop during a commit, that's
> a
> >> > > merge. If you see a big delta change, that's probably a merge, too.
> >> > >
> >> > > You could also jstack or kill -3 the JVM and see where it's spending
> >> its
> >> > > time to give you some ideas what's going on inside.
> >> > >
> >> > > HTH.
> >> > >
> >> > > Otis
> >> > > --
> >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> >> > > Solr & Elasticsearch Support * http://sematext.com/
> >> > >
> >> > >
> >> > > On Sun, Feb 8, 2015 at 6:48 AM, Gili Nachum <gi...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Hello,
> >> > > >
> >> > > > During a load test I noticed a commit that took 43 seconds to
> >> complete
> >> > > > (client hard complete).
> >> > > > Is this to be expected? What's causing it?
> >> > > > I have a pair of machines hosting a 128M docs collection (8
> shards,
> >> > > > replication factor=2).
> >> > > >
> >> > > > Could it be merges? In Lucene merges happen async of commit
> >> statements,
> >> > > but
> >> > > > reading Solr's doc for Update Hanlder
> >> > > > <
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
> >> > > > >
> >> > > > it sounds like hard commits do wait for merges to occur: *" The
> >> > tradeoff
> >> > > is
> >> > > > that a soft commit gives you faster visibility because it's not
> >> waiting
> >> > > for
> >> > > > background merges to finish."*
> >> > > > Thanks.
> >> > > >
> >> > >
> >> >
> >>
>
Re: 43sec commit duration - blocked by index merge events?
Posted by Erick Erickson <er...@gmail.com>.
Exactly how are you issuing the commit? I'm assuming you're
using SolrJ. the server.commit(whatever, true) waits for the searcher
to be opened before returning. This includes (I believe) warmup
times. It could be that the warmup times are huge in your case, the
solr logs should show you the autowarm times for a new searcher.
Best,
Erick
On Fri, Feb 13, 2015 at 2:53 PM, Jack Krupansky
<ja...@gmail.com> wrote:
> I wasn't able to follow Otis' answer but... the purpose of commit is to
> make make recent document changes (since the last commit) visible to
> queries, and has nothing to do with merging of segments. IOW, take the new
> segment that is being created and not yet ready for use by query, and
> finish it so that query can access it. Soft commit vs. hard commit is
> simply a matter of whether Solr will wait for the I/O to write the new
> segment to disk to complete. Merging is an independent, background
> procedure (thread) that merges existing segments. It does seem odd that the
> cited doc does say that soft commit waits for background merges! (Hoss??)
>
> -- Jack Krupansky
>
> On Fri, Feb 13, 2015 at 4:47 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
>> Check
>> http://search-lucene.com/?q=commit+wait+block&fc_type=mail+_hash_+user
>>
>> e.g. http://search-lucene.com/m/QTPa7Sqx81
>>
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On Fri, Feb 13, 2015 at 8:50 AM, Gili Nachum <gi...@gmail.com> wrote:
>>
>> > Thanks Otis, can you confirm that a commit call will wait for merges to
>> > complete before returning?
>> >
>> > On Thu, Feb 12, 2015 at 8:46 PM, Otis Gospodnetic <
>> > otis.gospodnetic@gmail.com> wrote:
>> >
>> > > If you are using Solr and SPM for Solr, you can check a report that
>> shows
>> > > the # of files in an index and the report that shows you the max
>> docs-num
>> > > docs delta. If you see the # of files drop during a commit, that's a
>> > > merge. If you see a big delta change, that's probably a merge, too.
>> > >
>> > > You could also jstack or kill -3 the JVM and see where it's spending
>> its
>> > > time to give you some ideas what's going on inside.
>> > >
>> > > HTH.
>> > >
>> > > Otis
>> > > --
>> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> > > Solr & Elasticsearch Support * http://sematext.com/
>> > >
>> > >
>> > > On Sun, Feb 8, 2015 at 6:48 AM, Gili Nachum <gi...@gmail.com>
>> > wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > During a load test I noticed a commit that took 43 seconds to
>> complete
>> > > > (client hard complete).
>> > > > Is this to be expected? What's causing it?
>> > > > I have a pair of machines hosting a 128M docs collection (8 shards,
>> > > > replication factor=2).
>> > > >
>> > > > Could it be merges? In Lucene merges happen async of commit
>> statements,
>> > > but
>> > > > reading Solr's doc for Update Hanlder
>> > > > <
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
>> > > > >
>> > > > it sounds like hard commits do wait for merges to occur: *" The
>> > tradeoff
>> > > is
>> > > > that a soft commit gives you faster visibility because it's not
>> waiting
>> > > for
>> > > > background merges to finish."*
>> > > > Thanks.
>> > > >
>> > >
>> >
>>
Re: 43sec commit duration - blocked by index merge events?
Posted by Jack Krupansky <ja...@gmail.com>.
I wasn't able to follow Otis' answer but... the purpose of commit is to
make make recent document changes (since the last commit) visible to
queries, and has nothing to do with merging of segments. IOW, take the new
segment that is being created and not yet ready for use by query, and
finish it so that query can access it. Soft commit vs. hard commit is
simply a matter of whether Solr will wait for the I/O to write the new
segment to disk to complete. Merging is an independent, background
procedure (thread) that merges existing segments. It does seem odd that the
cited doc does say that soft commit waits for background merges! (Hoss??)
-- Jack Krupansky
On Fri, Feb 13, 2015 at 4:47 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Check
> http://search-lucene.com/?q=commit+wait+block&fc_type=mail+_hash_+user
>
> e.g. http://search-lucene.com/m/QTPa7Sqx81
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, Feb 13, 2015 at 8:50 AM, Gili Nachum <gi...@gmail.com> wrote:
>
> > Thanks Otis, can you confirm that a commit call will wait for merges to
> > complete before returning?
> >
> > On Thu, Feb 12, 2015 at 8:46 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > If you are using Solr and SPM for Solr, you can check a report that
> shows
> > > the # of files in an index and the report that shows you the max
> docs-num
> > > docs delta. If you see the # of files drop during a commit, that's a
> > > merge. If you see a big delta change, that's probably a merge, too.
> > >
> > > You could also jstack or kill -3 the JVM and see where it's spending
> its
> > > time to give you some ideas what's going on inside.
> > >
> > > HTH.
> > >
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > > On Sun, Feb 8, 2015 at 6:48 AM, Gili Nachum <gi...@gmail.com>
> > wrote:
> > >
> > > > Hello,
> > > >
> > > > During a load test I noticed a commit that took 43 seconds to
> complete
> > > > (client hard complete).
> > > > Is this to be expected? What's causing it?
> > > > I have a pair of machines hosting a 128M docs collection (8 shards,
> > > > replication factor=2).
> > > >
> > > > Could it be merges? In Lucene merges happen async of commit
> statements,
> > > but
> > > > reading Solr's doc for Update Hanlder
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
> > > > >
> > > > it sounds like hard commits do wait for merges to occur: *" The
> > tradeoff
> > > is
> > > > that a soft commit gives you faster visibility because it's not
> waiting
> > > for
> > > > background merges to finish."*
> > > > Thanks.
> > > >
> > >
> >
>
Re: 43sec commit duration - blocked by index merge events?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Check http://search-lucene.com/?q=commit+wait+block&fc_type=mail+_hash_+user
e.g. http://search-lucene.com/m/QTPa7Sqx81
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Fri, Feb 13, 2015 at 8:50 AM, Gili Nachum <gi...@gmail.com> wrote:
> Thanks Otis, can you confirm that a commit call will wait for merges to
> complete before returning?
>
> On Thu, Feb 12, 2015 at 8:46 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > If you are using Solr and SPM for Solr, you can check a report that shows
> > the # of files in an index and the report that shows you the max docs-num
> > docs delta. If you see the # of files drop during a commit, that's a
> > merge. If you see a big delta change, that's probably a merge, too.
> >
> > You could also jstack or kill -3 the JVM and see where it's spending its
> > time to give you some ideas what's going on inside.
> >
> > HTH.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Sun, Feb 8, 2015 at 6:48 AM, Gili Nachum <gi...@gmail.com>
> wrote:
> >
> > > Hello,
> > >
> > > During a load test I noticed a commit that took 43 seconds to complete
> > > (client hard complete).
> > > Is this to be expected? What's causing it?
> > > I have a pair of machines hosting a 128M docs collection (8 shards,
> > > replication factor=2).
> > >
> > > Could it be merges? In Lucene merges happen async of commit statements,
> > but
> > > reading Solr's doc for Update Hanlder
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
> > > >
> > > it sounds like hard commits do wait for merges to occur: *" The
> tradeoff
> > is
> > > that a soft commit gives you faster visibility because it's not waiting
> > for
> > > background merges to finish."*
> > > Thanks.
> > >
> >
>
Re: 43sec commit duration - blocked by index merge events?
Posted by Gili Nachum <gi...@gmail.com>.
Thanks Otis, can you confirm that a commit call will wait for merges to
complete before returning?
On Thu, Feb 12, 2015 at 8:46 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> If you are using Solr and SPM for Solr, you can check a report that shows
> the # of files in an index and the report that shows you the max docs-num
> docs delta. If you see the # of files drop during a commit, that's a
> merge. If you see a big delta change, that's probably a merge, too.
>
> You could also jstack or kill -3 the JVM and see where it's spending its
> time to give you some ideas what's going on inside.
>
> HTH.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Sun, Feb 8, 2015 at 6:48 AM, Gili Nachum <gi...@gmail.com> wrote:
>
> > Hello,
> >
> > During a load test I noticed a commit that took 43 seconds to complete
> > (client hard complete).
> > Is this to be expected? What's causing it?
> > I have a pair of machines hosting a 128M docs collection (8 shards,
> > replication factor=2).
> >
> > Could it be merges? In Lucene merges happen async of commit statements,
> but
> > reading Solr's doc for Update Hanlder
> > <
> >
> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
> > >
> > it sounds like hard commits do wait for merges to occur: *" The tradeoff
> is
> > that a soft commit gives you faster visibility because it's not waiting
> for
> > background merges to finish."*
> > Thanks.
> >
>
Re: 43sec commit duration - blocked by index merge events?
Posted by Otis Gospodnetic <ot...@gmail.com>.
If you are using Solr and SPM for Solr, you can check a report that shows
the # of files in an index and the report that shows you the max docs-num
docs delta. If you see the # of files drop during a commit, that's a
merge. If you see a big delta change, that's probably a merge, too.
You could also jstack or kill -3 the JVM and see where it's spending its
time to give you some ideas what's going on inside.
HTH.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Sun, Feb 8, 2015 at 6:48 AM, Gili Nachum <gi...@gmail.com> wrote:
> Hello,
>
> During a load test I noticed a commit that took 43 seconds to complete
> (client hard complete).
> Is this to be expected? What's causing it?
> I have a pair of machines hosting a 128M docs collection (8 shards,
> replication factor=2).
>
> Could it be merges? In Lucene merges happen async of commit statements, but
> reading Solr's doc for Update Hanlder
> <
> https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
> >
> it sounds like hard commits do wait for merges to occur: *" The tradeoff is
> that a soft commit gives you faster visibility because it's not waiting for
> background merges to finish."*
> Thanks.
>