You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mou <mo...@gmail.com> on 2012/07/17 04:43:06 UTC
Using Solr 3.4 running on tomcat7 - very slow search
Hi,
Our index is divided into two shards and each of them has 120M docs , total
size 75G in each core.
The server is a pretty good one , jvm is given memory of 70G and about same
is left for OS (SLES 11) .
We use all dynamic fields except th eunique id and are using long queries
but almost all of them are filter queires, Each query may have 10 -30 fq
parameters.
When I tested the index ( same size) but with max heap size 40 G, queries
were blazing fast. I used solrmeter to load test and it was happily serving
12000 queries or more per min with avg 65 ms qtime.We had an excellent
filtercache hit ratio.
This index is only used for searching and being replicated every 7 sec from
the master.
But now in production server it is horribly slow and taking 5 mins(qtime) to
return a query ( same query).
What could go wrong?
Really appreciate your suggestions on debugging this thing..
--
View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Using Solr 3.4 running on tomcat7 - very slow search
Posted by Mou <mo...@gmail.com>.
Increasing the polling interval does help. But the requirement is to get a
document indexed and searchable instantly ( sounds like RTS), 30 sec is
acceptable.I need to look at Solr NRT and cloud.
I created a new core to accept daily updates and replicate every 10 sec.
Two other cores with 234 Million documents are configured to replicate only
once a day.
I am feeding all three cores but two big cores are not replicating. While
searching I am running a group.field on my unique id and taking the most
updated one. Right now it looks fine.Every day I am going to delete the
last day's records from the daily update.
I am planning to use rsync for replication, it will be fusion IO to fusion
IO , so hopefully will be very fast. What do you think ?
We use windows service ( written in dot net C#) to feed the data using REST
call. That is really fast , we can feed more than 15 Million data in a day
to two cores easily. I am using solr config autocommit = 5 sec
I could not figure out how I was able to achieve those numbers in my test
environment, all configuration were same except I had lot less memory in
test ! I am trying to find out what I am missing in other configuration.
My SLES kernel version is different in production, its a 3.0.* , test was
2.6.* but I do not think that can cause a problem.
Thank you again,
Mou
On Wed, Jul 18, 2012 at 6:26 PM, Erick Erickson [via Lucene] <
ml-node+s472066n3995861h45@n3.nabble.com> wrote:
> Replication will indeed be incremental. But if you commit too often (and
> committing too often a common mistake) then the merging will
> eventually merge everything into new segments and the whole thing will
> be replicated.
>
> Additionally, optimizing (or forceMerge in 4.x) will make a single segment
> and force the entire index to replicate.
>
> You should emphatically _not_ have to have two cores. Solr is built to
> handle replication etc. I suspect your committing too often or some
> other mis-configuration and you're creating a problem for yourself.
>
> Here's what I'd do:
> 1> increase the polling interval to, say, 10 minutes (or however long you
> can
> live with stale data) on the slave.
>
> 2> decrease the commits you're doing. This could involve the autocommit
> options
> you might have set in solrconfig.xml. It could be your client (don't
> know how you're
> indexing, solrJ?) and the commitWithin parameter. Could be you're
> optimizing (if you
> are, stop it!).
>
> Note that ramBufferSizeMB has no influence on how often things are
> _committed_.
> When this limit is exceeded, the accumulated indexing data is written
> to the currently-open
> segment. Multiple flushes can go to the _same_ segment. The write-once
> nature of
> segments means that after a segment is closed (through a commit), it
> is not changed. But
> a segment that is not closed may be written to multiple times until it's
> closed.
>
> HTH
> Erick
>
> On Wed, Jul 18, 2012 at 1:25 PM, Mou <[hidden email]<http://user/SendEmail.jtp?type=node&node=3995861&i=0>>
> wrote:
>
> > Hi Eric,
> >
> > I totally agree. That's what I also figured ultimately. One thing I am
> not
> > clear. The replication is supposed to be incremental ? But looks like
> it
> > is trying to replicate the whole index. May be I am changing the index
> so
> > frequently, it is triggering auto merge and a full replication ? I am
> > thinking in right direction?
> >
> > I see that when I start the solr search instance before I start feeding
> the
> > solr Index, my searches are fine BUT it is using the old searcher so I
> am
> > not seeing the updates in the result.
> >
> > So now I am trying to change my architecture. I am going to have a core
> > dedicated to receive daily updates, which is going to be 5 million docs
> and
> > size is going to be little less than 5 G, which is small and replication
> > will be faster?
> >
> > I will search both the cores i.e. old data and the daily updates and do
> a
> > field collapsing on my unique id so that I do not return duplicate
> results
> > .I haven't tried grouping results ; so not sure about the performance.
> Any
> > suggestion ?
> >
> > Eventually I have to use Solr trunk like you suggested.
> >
> > Thank you for your help,
> >
> > On Wed, Jul 18, 2012 at 10:28 AM, Erick Erickson [via Lucene] <
> > [hidden email] <http://user/SendEmail.jtp?type=node&node=3995861&i=1>>
> wrote:
> >
> >> bq: This index is only used for searching and being replicated every 7
> sec
> >> from
> >> the master.
> >>
> >> This is a red-flag. 7 second replication times are likely forcing your
> >> app to spend
> >> all its time opening new searchers. Your cached filter queries are
> >> likely rarely being re-used
> >> because they're being thrown away every 7 seconds. This assumes you're
> >> changing your master index frequently.
> >>
> >> If you need near real time, consider Solr trunk and SolrCloud, but
> >> trying to simulate
> >> NRT with very short replication intervals is usually a bad idea.
> >>
> >> A quick test would be to disable replication for a bit (or lengthen it
> >> to, say, 10 minutes)
> >>
> >> Best
> >> Erick
> >>
> >> On Tue, Jul 17, 2012 at 10:47 PM, Fuad Efendi <[hidden email]<
> http://user/SendEmail.jtp?type=node&node=3995754&i=0>>
> >> wrote:
> >>
> >> >
> >> >> FWIW, when asked at what point one would want to split JVMs and
> shard,
> >> >> on the same machine, Grant Ingersoll mentioned 16GB, and precisely
> for
> >> >> GC cost reasons. You're way above that.
> >> >
> >> > - his index is 75G, and Grant mentioned RAM heap size; we can use
> >> terabytes
> >> > of index with 16Gb memory.
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >> ------------------------------
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >>
> >>
> http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995754.html
> >> To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow
> search, click
> >> here<
>
> >> .
> >> NAML<
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> >>
> >
> >
> > --
> > View this message in context:
> http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995774.html
>
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995861.html
> To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow search, click
> here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg1MTA5MTUw>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995873.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Using Solr 3.4 running on tomcat7 - very slow search
Posted by Erick Erickson <er...@gmail.com>.
Replication will indeed be incremental. But if you commit too often (and
committing too often a common mistake) then the merging will
eventually merge everything into new segments and the whole thing will
be replicated.
Additionally, optimizing (or forceMerge in 4.x) will make a single segment
and force the entire index to replicate.
You should emphatically _not_ have to have two cores. Solr is built to
handle replication etc. I suspect your committing too often or some
other mis-configuration and you're creating a problem for yourself.
Here's what I'd do:
1> increase the polling interval to, say, 10 minutes (or however long you can
live with stale data) on the slave.
2> decrease the commits you're doing. This could involve the autocommit options
you might have set in solrconfig.xml. It could be your client (don't
know how you're
indexing, solrJ?) and the commitWithin parameter. Could be you're
optimizing (if you
are, stop it!).
Note that ramBufferSizeMB has no influence on how often things are _committed_.
When this limit is exceeded, the accumulated indexing data is written
to the currently-open
segment. Multiple flushes can go to the _same_ segment. The write-once nature of
segments means that after a segment is closed (through a commit), it
is not changed. But
a segment that is not closed may be written to multiple times until it's closed.
HTH
Erick
On Wed, Jul 18, 2012 at 1:25 PM, Mou <mo...@gmail.com> wrote:
> Hi Eric,
>
> I totally agree. That's what I also figured ultimately. One thing I am not
> clear. The replication is supposed to be incremental ? But looks like it
> is trying to replicate the whole index. May be I am changing the index so
> frequently, it is triggering auto merge and a full replication ? I am
> thinking in right direction?
>
> I see that when I start the solr search instance before I start feeding the
> solr Index, my searches are fine BUT it is using the old searcher so I am
> not seeing the updates in the result.
>
> So now I am trying to change my architecture. I am going to have a core
> dedicated to receive daily updates, which is going to be 5 million docs and
> size is going to be little less than 5 G, which is small and replication
> will be faster?
>
> I will search both the cores i.e. old data and the daily updates and do a
> field collapsing on my unique id so that I do not return duplicate results
> .I haven't tried grouping results ; so not sure about the performance. Any
> suggestion ?
>
> Eventually I have to use Solr trunk like you suggested.
>
> Thank you for your help,
>
> On Wed, Jul 18, 2012 at 10:28 AM, Erick Erickson [via Lucene] <
> ml-node+s472066n3995754h28@n3.nabble.com> wrote:
>
>> bq: This index is only used for searching and being replicated every 7 sec
>> from
>> the master.
>>
>> This is a red-flag. 7 second replication times are likely forcing your
>> app to spend
>> all its time opening new searchers. Your cached filter queries are
>> likely rarely being re-used
>> because they're being thrown away every 7 seconds. This assumes you're
>> changing your master index frequently.
>>
>> If you need near real time, consider Solr trunk and SolrCloud, but
>> trying to simulate
>> NRT with very short replication intervals is usually a bad idea.
>>
>> A quick test would be to disable replication for a bit (or lengthen it
>> to, say, 10 minutes)
>>
>> Best
>> Erick
>>
>> On Tue, Jul 17, 2012 at 10:47 PM, Fuad Efendi <[hidden email]<http://user/SendEmail.jtp?type=node&node=3995754&i=0>>
>> wrote:
>>
>> >
>> >> FWIW, when asked at what point one would want to split JVMs and shard,
>> >> on the same machine, Grant Ingersoll mentioned 16GB, and precisely for
>> >> GC cost reasons. You're way above that.
>> >
>> > - his index is 75G, and Grant mentioned RAM heap size; we can use
>> terabytes
>> > of index with 16Gb memory.
>> >
>> >
>> >
>> >
>> >
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995754.html
>> To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow search, click
>> here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg1MTA5MTUw>
>> .
>> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995774.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Using Solr 3.4 running on tomcat7 - very slow search
Posted by Mou <mo...@gmail.com>.
Hi Eric,
I totally agree. That's what I also figured ultimately. One thing I am not
clear. The replication is supposed to be incremental ? But looks like it
is trying to replicate the whole index. May be I am changing the index so
frequently, it is triggering auto merge and a full replication ? I am
thinking in right direction?
I see that when I start the solr search instance before I start feeding the
solr Index, my searches are fine BUT it is using the old searcher so I am
not seeing the updates in the result.
So now I am trying to change my architecture. I am going to have a core
dedicated to receive daily updates, which is going to be 5 million docs and
size is going to be little less than 5 G, which is small and replication
will be faster?
I will search both the cores i.e. old data and the daily updates and do a
field collapsing on my unique id so that I do not return duplicate results
.I haven't tried grouping results ; so not sure about the performance. Any
suggestion ?
Eventually I have to use Solr trunk like you suggested.
Thank you for your help,
On Wed, Jul 18, 2012 at 10:28 AM, Erick Erickson [via Lucene] <
ml-node+s472066n3995754h28@n3.nabble.com> wrote:
> bq: This index is only used for searching and being replicated every 7 sec
> from
> the master.
>
> This is a red-flag. 7 second replication times are likely forcing your
> app to spend
> all its time opening new searchers. Your cached filter queries are
> likely rarely being re-used
> because they're being thrown away every 7 seconds. This assumes you're
> changing your master index frequently.
>
> If you need near real time, consider Solr trunk and SolrCloud, but
> trying to simulate
> NRT with very short replication intervals is usually a bad idea.
>
> A quick test would be to disable replication for a bit (or lengthen it
> to, say, 10 minutes)
>
> Best
> Erick
>
> On Tue, Jul 17, 2012 at 10:47 PM, Fuad Efendi <[hidden email]<http://user/SendEmail.jtp?type=node&node=3995754&i=0>>
> wrote:
>
> >
> >> FWIW, when asked at what point one would want to split JVMs and shard,
> >> on the same machine, Grant Ingersoll mentioned 16GB, and precisely for
> >> GC cost reasons. You're way above that.
> >
> > - his index is 75G, and Grant mentioned RAM heap size; we can use
> terabytes
> > of index with 16Gb memory.
> >
> >
> >
> >
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995754.html
> To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow search, click
> here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg1MTA5MTUw>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995774.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Using Solr 3.4 running on tomcat7 - very slow search
Posted by Erick Erickson <er...@gmail.com>.
bq: This index is only used for searching and being replicated every 7 sec from
the master.
This is a red-flag. 7 second replication times are likely forcing your
app to spend
all its time opening new searchers. Your cached filter queries are
likely rarely being re-used
because they're being thrown away every 7 seconds. This assumes you're
changing your master index frequently.
If you need near real time, consider Solr trunk and SolrCloud, but
trying to simulate
NRT with very short replication intervals is usually a bad idea.
A quick test would be to disable replication for a bit (or lengthen it
to, say, 10 minutes)
Best
Erick
On Tue, Jul 17, 2012 at 10:47 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>
>> FWIW, when asked at what point one would want to split JVMs and shard,
>> on the same machine, Grant Ingersoll mentioned 16GB, and precisely for
>> GC cost reasons. You're way above that.
>
> - his index is 75G, and Grant mentioned RAM heap size; we can use terabytes
> of index with 16Gb memory.
>
>
>
>
>
RE: Using Solr 3.4 running on tomcat7 - very slow search
Posted by Fuad Efendi <fu...@efendi.ca>.
> FWIW, when asked at what point one would want to split JVMs and shard,
> on the same machine, Grant Ingersoll mentioned 16GB, and precisely for
> GC cost reasons. You're way above that.
- his index is 75G, and Grant mentioned RAM heap size; we can use terabytes
of index with 16Gb memory.
Re: Using Solr 3.4 running on tomcat7 - very slow search
Posted by Mou <mo...@gmail.com>.
Brian,
Thanks again.
swappiness is set to 60 and from vmstat , I can see no swapping is going on.
Also I am using fusion IO SSD for storing my index.
I also used the visualVM and it shows me that it is blocked on
lock=org.apache.lucene.index.SegmentCoreReaders@299172a7.
Any clue?
On Mon, Jul 16, 2012 at 10:38 PM, Bryan Loofbourrow [via Lucene] <
ml-node+s472066n3995452h66@n3.nabble.com> wrote:
> Another thing you may wish to ponder is this blog entry from Mike
> McCandless:
> http://blog.mikemccandless.com/2011/04/just-say-no-to-swapping.html
>
> In it, he discusses the poor interaction between OS swapping, and
> long-neglected allocations in a JVM. You're on Linux, which has decent
> control over swapping decisions, so you may find that a tweak is in order,
> especially if you can discover evidence that the hard drive is being
> worked hard during GC. If the problem exists, it might be especially
> pronounced in your large JVM.
>
> I have no direct evidence of thrashing during GC (I am not sure how to go
> about gathering such evidence), but I have seen, on a Windows machine, a
> Tomcat running Solr refuse to shut down for many minutes, while a Resource
> Monitor session reports that that same Tomcat process is frantically
> reading from the page file the whole time. So there is something besides
> plausibility to the idea.
>
> -- Bryan
>
> > -----Original Message-----
> > From: Mou [mailto:[hidden email]<http://user/SendEmail.jtp?type=node&node=3995452&i=0>]
>
> > Sent: Monday, July 16, 2012 9:09 PM
> > To: [hidden email]<http://user/SendEmail.jtp?type=node&node=3995452&i=1>
> > Subject: Re: Using Solr 3.4 running on tomcat7 - very slow search
> >
> > Thanks Brian. Excellent suggestion.
> >
> > I haven't used VisualVM before but I am going to use it to see where CPU
> > is
> > going. I saw that CPU is overly used. I haven't seen so much CPU use in
> > testing.
> > Although I think GC is not a problem, splitting the jvm per shard would
> be
> > a good idea.
> >
> >
> > On Mon, Jul 16, 2012 at 9:44 PM, Bryan Loofbourrow [via Lucene] <
> > [hidden email] <http://user/SendEmail.jtp?type=node&node=3995452&i=2>>
> wrote:
> >
> > > 5 min is ridiculously long for a query that used to take 65ms. That
> > ought
> > > to be a great clue. The only two things I've seen that could cause
> that
> > > are thrashing, or GC. Hard to see how it could be thrashing, given
> your
> > > hardware, so I'd initially suspect GC.
> > >
> > > Aim VisualVM at the JVM. It shows how much CPU goes to GC over time,
> in
>
> > a
> > > nice blue line. And if it's not GC, try out its Sampler tab, and see
> > where
> > > the CPU is spending its time.
> > >
> > > FWIW, when asked at what point one would want to split JVMs and shard,
> > on
> > > the same machine, Grant Ingersoll mentioned 16GB, and precisely for GC
> > > cost reasons. You're way above that. Maybe multiple JVMs and sharding,
> > > even on the same machine, would serve you better than a monster 70GB
> > JVM.
> > >
> > > -- Bryan
> > >
> > > > -----Original Message-----
> > > > From: Mou [mailto:[hidden
> > email]<http://user/SendEmail.jtp?type=node&node=3995446&i=0>]
> > >
> > > > Sent: Monday, July 16, 2012 7:43 PM
> > > > To: [hidden
> > email]<http://user/SendEmail.jtp?type=node&node=3995446&i=1>
> > > > Subject: Using Solr 3.4 running on tomcat7 - very slow search
> > > >
> > > > Hi,
> > > >
> > > > Our index is divided into two shards and each of them has 120M docs
> ,
> > > > total
> > > > size 75G in each core.
> > > > The server is a pretty good one , jvm is given memory of 70G and
> about
> > > > same
> > > > is left for OS (SLES 11) .
> > > >
> > > > We use all dynamic fields except th eunique id and are using long
> > > queries
> > > > but almost all of them are filter queires, Each query may have 10
> -30
> > fq
> > > > parameters.
> > > >
> > > > When I tested the index ( same size) but with max heap size 40 G,
> > > queries
> > >
> > > > were blazing fast. I used solrmeter to load test and it was happily
> > > > serving
> > > > 12000 queries or more per min with avg 65 ms qtime.We had an
> excellent
> > > > filtercache hit ratio.
> > > >
> > > > This index is only used for searching and being replicated every 7
> sec
>
> > > > from
> > > > the master.
> > > >
> > > > But now in production server it is horribly slow and taking 5
> > > mins(qtime)
> > >
> > > > to
> > > > return a query ( same query).
> > > > What could go wrong?
> > > >
> > > > Really appreciate your suggestions on debugging this thing..
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > http://lucene.472066.n3.nabble.com/Using-
> > > > Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436.html
> > > > Sent from the Solr - User mailing list archive at Nabble.com.
> > >
> > >
> > > ------------------------------
> > > If you reply to this email, your message will be added to the
> > discussion
> > > below:
> > >
> > > http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-
> > very-slow-search-tp3995436p3995446.html
> > > To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow
> > search, click
> > >
> >
> here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=uns
> >
> ubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg
> > 1MTA5MTUw>
> > > .
> > >
> >
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=mac
> >
> ro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespace
> > s.BasicNamespace-nabble.view.web.template.NabbleNamespace-
> >
> nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21na
> > bble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-
> > send_instant_email%21nabble%3Aemail.naml>
> > >
> >
> >
> > --
> > View this message in context: http://lucene.472066.n3.nabble.com/Using-
> > Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995449.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995452.html
> To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow search, click
> here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg1MTA5MTUw>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995631.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Using Solr 3.4 running on tomcat7 - very slow search
Posted by Bryan Loofbourrow <bl...@knowledgemosaic.com>.
Another thing you may wish to ponder is this blog entry from Mike
McCandless:
http://blog.mikemccandless.com/2011/04/just-say-no-to-swapping.html
In it, he discusses the poor interaction between OS swapping, and
long-neglected allocations in a JVM. You're on Linux, which has decent
control over swapping decisions, so you may find that a tweak is in order,
especially if you can discover evidence that the hard drive is being
worked hard during GC. If the problem exists, it might be especially
pronounced in your large JVM.
I have no direct evidence of thrashing during GC (I am not sure how to go
about gathering such evidence), but I have seen, on a Windows machine, a
Tomcat running Solr refuse to shut down for many minutes, while a Resource
Monitor session reports that that same Tomcat process is frantically
reading from the page file the whole time. So there is something besides
plausibility to the idea.
-- Bryan
> -----Original Message-----
> From: Mou [mailto:mounandi@gmail.com]
> Sent: Monday, July 16, 2012 9:09 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Using Solr 3.4 running on tomcat7 - very slow search
>
> Thanks Brian. Excellent suggestion.
>
> I haven't used VisualVM before but I am going to use it to see where CPU
> is
> going. I saw that CPU is overly used. I haven't seen so much CPU use in
> testing.
> Although I think GC is not a problem, splitting the jvm per shard would
be
> a good idea.
>
>
> On Mon, Jul 16, 2012 at 9:44 PM, Bryan Loofbourrow [via Lucene] <
> ml-node+s472066n3995446h80@n3.nabble.com> wrote:
>
> > 5 min is ridiculously long for a query that used to take 65ms. That
> ought
> > to be a great clue. The only two things I've seen that could cause
that
> > are thrashing, or GC. Hard to see how it could be thrashing, given
your
> > hardware, so I'd initially suspect GC.
> >
> > Aim VisualVM at the JVM. It shows how much CPU goes to GC over time,
in
> a
> > nice blue line. And if it's not GC, try out its Sampler tab, and see
> where
> > the CPU is spending its time.
> >
> > FWIW, when asked at what point one would want to split JVMs and shard,
> on
> > the same machine, Grant Ingersoll mentioned 16GB, and precisely for GC
> > cost reasons. You're way above that. Maybe multiple JVMs and sharding,
> > even on the same machine, would serve you better than a monster 70GB
> JVM.
> >
> > -- Bryan
> >
> > > -----Original Message-----
> > > From: Mou [mailto:[hidden
> email]<http://user/SendEmail.jtp?type=node&node=3995446&i=0>]
> >
> > > Sent: Monday, July 16, 2012 7:43 PM
> > > To: [hidden
> email]<http://user/SendEmail.jtp?type=node&node=3995446&i=1>
> > > Subject: Using Solr 3.4 running on tomcat7 - very slow search
> > >
> > > Hi,
> > >
> > > Our index is divided into two shards and each of them has 120M docs
,
> > > total
> > > size 75G in each core.
> > > The server is a pretty good one , jvm is given memory of 70G and
about
> > > same
> > > is left for OS (SLES 11) .
> > >
> > > We use all dynamic fields except th eunique id and are using long
> > queries
> > > but almost all of them are filter queires, Each query may have 10
-30
> fq
> > > parameters.
> > >
> > > When I tested the index ( same size) but with max heap size 40 G,
> > queries
> >
> > > were blazing fast. I used solrmeter to load test and it was happily
> > > serving
> > > 12000 queries or more per min with avg 65 ms qtime.We had an
excellent
> > > filtercache hit ratio.
> > >
> > > This index is only used for searching and being replicated every 7
sec
> > > from
> > > the master.
> > >
> > > But now in production server it is horribly slow and taking 5
> > mins(qtime)
> >
> > > to
> > > return a query ( same query).
> > > What could go wrong?
> > >
> > > Really appreciate your suggestions on debugging this thing..
> > >
> > >
> > >
> > > --
> > > View this message in context:
> http://lucene.472066.n3.nabble.com/Using-
> > > Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the
> discussion
> > below:
> >
> > http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-
> very-slow-search-tp3995436p3995446.html
> > To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow
> search, click
> >
>
here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=uns
>
ubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg
> 1MTA5MTUw>
> > .
> >
>
NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=mac
>
ro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespace
> s.BasicNamespace-nabble.view.web.template.NabbleNamespace-
>
nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21na
> bble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-
> send_instant_email%21nabble%3Aemail.naml>
> >
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Using-
> Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995449.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Using Solr 3.4 running on tomcat7 - very slow search
Posted by Mou <mo...@gmail.com>.
Thanks Brian. Excellent suggestion.
I haven't used VisualVM before but I am going to use it to see where CPU is
going. I saw that CPU is overly used. I haven't seen so much CPU use in
testing.
Although I think GC is not a problem, splitting the jvm per shard would be
a good idea.
On Mon, Jul 16, 2012 at 9:44 PM, Bryan Loofbourrow [via Lucene] <
ml-node+s472066n3995446h80@n3.nabble.com> wrote:
> 5 min is ridiculously long for a query that used to take 65ms. That ought
> to be a great clue. The only two things I've seen that could cause that
> are thrashing, or GC. Hard to see how it could be thrashing, given your
> hardware, so I'd initially suspect GC.
>
> Aim VisualVM at the JVM. It shows how much CPU goes to GC over time, in a
> nice blue line. And if it's not GC, try out its Sampler tab, and see where
> the CPU is spending its time.
>
> FWIW, when asked at what point one would want to split JVMs and shard, on
> the same machine, Grant Ingersoll mentioned 16GB, and precisely for GC
> cost reasons. You're way above that. Maybe multiple JVMs and sharding,
> even on the same machine, would serve you better than a monster 70GB JVM.
>
> -- Bryan
>
> > -----Original Message-----
> > From: Mou [mailto:[hidden email]<http://user/SendEmail.jtp?type=node&node=3995446&i=0>]
>
> > Sent: Monday, July 16, 2012 7:43 PM
> > To: [hidden email]<http://user/SendEmail.jtp?type=node&node=3995446&i=1>
> > Subject: Using Solr 3.4 running on tomcat7 - very slow search
> >
> > Hi,
> >
> > Our index is divided into two shards and each of them has 120M docs ,
> > total
> > size 75G in each core.
> > The server is a pretty good one , jvm is given memory of 70G and about
> > same
> > is left for OS (SLES 11) .
> >
> > We use all dynamic fields except th eunique id and are using long
> queries
> > but almost all of them are filter queires, Each query may have 10 -30 fq
> > parameters.
> >
> > When I tested the index ( same size) but with max heap size 40 G,
> queries
>
> > were blazing fast. I used solrmeter to load test and it was happily
> > serving
> > 12000 queries or more per min with avg 65 ms qtime.We had an excellent
> > filtercache hit ratio.
> >
> > This index is only used for searching and being replicated every 7 sec
> > from
> > the master.
> >
> > But now in production server it is horribly slow and taking 5
> mins(qtime)
>
> > to
> > return a query ( same query).
> > What could go wrong?
> >
> > Really appreciate your suggestions on debugging this thing..
> >
> >
> >
> > --
> > View this message in context: http://lucene.472066.n3.nabble.com/Using-
> > Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995446.html
> To unsubscribe from Using Solr 3.4 running on tomcat7 - very slow search, click
> here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3995436&code=bW91bmFuZGlAZ21haWwuY29tfDM5OTU0MzZ8Mjg1MTA5MTUw>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Using-Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436p3995449.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Using Solr 3.4 running on tomcat7 - very slow search
Posted by Bryan Loofbourrow <bl...@knowledgemosaic.com>.
5 min is ridiculously long for a query that used to take 65ms. That ought
to be a great clue. The only two things I've seen that could cause that
are thrashing, or GC. Hard to see how it could be thrashing, given your
hardware, so I'd initially suspect GC.
Aim VisualVM at the JVM. It shows how much CPU goes to GC over time, in a
nice blue line. And if it's not GC, try out its Sampler tab, and see where
the CPU is spending its time.
FWIW, when asked at what point one would want to split JVMs and shard, on
the same machine, Grant Ingersoll mentioned 16GB, and precisely for GC
cost reasons. You're way above that. Maybe multiple JVMs and sharding,
even on the same machine, would serve you better than a monster 70GB JVM.
-- Bryan
> -----Original Message-----
> From: Mou [mailto:mounandi@gmail.com]
> Sent: Monday, July 16, 2012 7:43 PM
> To: solr-user@lucene.apache.org
> Subject: Using Solr 3.4 running on tomcat7 - very slow search
>
> Hi,
>
> Our index is divided into two shards and each of them has 120M docs ,
> total
> size 75G in each core.
> The server is a pretty good one , jvm is given memory of 70G and about
> same
> is left for OS (SLES 11) .
>
> We use all dynamic fields except th eunique id and are using long
queries
> but almost all of them are filter queires, Each query may have 10 -30 fq
> parameters.
>
> When I tested the index ( same size) but with max heap size 40 G,
queries
> were blazing fast. I used solrmeter to load test and it was happily
> serving
> 12000 queries or more per min with avg 65 ms qtime.We had an excellent
> filtercache hit ratio.
>
> This index is only used for searching and being replicated every 7 sec
> from
> the master.
>
> But now in production server it is horribly slow and taking 5
mins(qtime)
> to
> return a query ( same query).
> What could go wrong?
>
> Really appreciate your suggestions on debugging this thing..
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Using-
> Solr-3-4-running-on-tomcat7-very-slow-search-tp3995436.html
> Sent from the Solr - User mailing list archive at Nabble.com.