You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sausarkar <sa...@ebay.com> on 2013/02/04 22:48:54 UTC
Really bad query performance for date range queries
we are experiencing very bad performance issues with date range queries. We
have configured the date fields as following: Our queries are rounded every
minute:
qt=ads&debugQuery=false&fl=id,StartDt_t110,.......&fq=Status_i110:2&fq=StartDt_t110:{
NOW/MINUTE-150DAYS TO NOW/MINUTE-90DAYS }&start=0&rows=20 Index size - 10
million documents in Solr 4.1 Start of every minute we see the query speed
to be about 10-12 seconds and increases over time. When the query time jump
to more than 10 seconds the linux *load average spikes up to more than 100
*in a 16 CPU machine. Any one has any suggestions how this can be fixed?
--
View this message in context: http://lucene.472066.n3.nabble.com/Really-bad-query-performance-for-date-range-queries-tp4038435.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Really bad query performance for date range queries
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
Hi Shawn,
I'm running solr in Tomcat on RHEL. It looks like what you're doing is making jHiccup wrap around the whole JVM by doing it that way, is that right? That's pretty cool if so. I'll see if I can set it up in my dev environment tomorrow.
Thanks,
Robi
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org]
Sent: Tuesday, February 05, 2013 2:53 PM
To: solr-user@lucene.apache.org
Subject: Re: Really bad query performance for date range queries
On 2/5/2013 3:19 PM, Petersen, Robert wrote:
> Hi Shawn,
>
> I've looked at the xing JVM before but don't use it. jHiccup looks like a really useful tool. Can you tell us how you are starting it up? Do you start it wrapping the app container (ie tomcat / jetty)?
Instead of just calling /usr/bin/java , I use this in my init script (homegrown for the jetty included in Solr):
/usr/local/bin/jHiccup /usr/bin/java
The jHiccup shell script is in /usr/local/bin, and the two jar files included in the download are in /usr/local/bin/bin.
The shell script that's included looks like it works under cygwin, so you could run it on Windows as long as you've got that. The rest of the shell script looks too complex to easily convert to a Windows batch file.
Thanks,
Shawn
Re: Really bad query performance for date range queries
Posted by Shawn Heisey <so...@elyograg.org>.
On 2/5/2013 3:19 PM, Petersen, Robert wrote:
> Hi Shawn,
>
> I've looked at the xing JVM before but don't use it. jHiccup looks like a really useful tool. Can you tell us how you are starting it up? Do you start it wrapping the app container (ie tomcat / jetty)?
Instead of just calling /usr/bin/java , I use this in my init script
(homegrown for the jetty included in Solr):
/usr/local/bin/jHiccup /usr/bin/java
The jHiccup shell script is in /usr/local/bin, and the two jar files
included in the download are in /usr/local/bin/bin.
The shell script that's included looks like it works under cygwin, so
you could run it on Windows as long as you've got that. The rest of the
shell script looks too complex to easily convert to a Windows batch file.
Thanks,
Shawn
RE: Really bad query performance for date range queries
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
Hi Shawn,
I've looked at the xing JVM before but don't use it. jHiccup looks like a really useful tool. Can you tell us how you are starting it up? Do you start it wrapping the app container (ie tomcat / jetty)?
Thanks
Robi
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org]
Sent: Tuesday, February 05, 2013 1:27 PM
To: solr-user@lucene.apache.org
Subject: Re: Really bad query performance for date range queries
On 2/5/2013 12:51 PM, sausarkar wrote:
> We have a 96GB ram machine with 16 processors. the JVM is set to use 60 GB.
> The test that we are running are purely query there is no indexing going on.
> I dont see garbage collection when I attach visualVM but see frequent
> CPU spikes ~once every minute.
A previous message from you indicates that your index is 12GB. I agree with Erick that this is not very large. The pauses that you have described sound a lot like stop-the-world garbage collection. I've seen very long pauses on an 8GB heap ... I don't even want to think about what could happen on 60GB.
Do you really need a 60GB heap? My dev server handles seven index shards with a 7GB heap and 16GB total RAM. On 4.1 the total index size is is over 100GB. On 4.2-SNAPSHOT the total index size is about 83GB.
Query performance isn't stellar, but it works perfectly. My production servers (running 3.5) have tons of RAM and each one only gets half the index, but they only run with the heap at 8GB. My queries are pretty low volume and not HUGELY complex. Median query time is about 26 milliseconds and 95th percentile is about 950 milliseconds.
Looking at the GC stats in jconsole/jvisualvm, I didn't think I had a GC pause problem, but I was proven wrong when I started correlating all the various logs in my system to load balancer "DOWN" incidents. I saw a pause of 12 seconds once in the GC log - on an 8GB heap.
I was introduced to a very cool program that tracks any kind of pause that's caused by factors outside the Java program, like GC pauses in the JVM or something happening in the OS. This is much easier to interpret than Java's GC logging, and you can get a nice graph from the data.
http://www.azulsystems.com/jHiccup
Using jHiccup, I was able to do a little bit of comparison between different runs. That helped me find some GC tuning parameters that have almost gotten rid of my GC pause problem. I'm constantly working on those parameters. The current values are:
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:NewRatio=3
-XX:MaxTenuringThreshold=8
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+UseLargePages
-XX:+AggressiveOpts
The Xing JVM (made by the company that created jHiccup) apparently has extremely low GC pause characteristics even with giant heaps like yours.
I'm not using it, and I don't know how much it costs.
Thanks,
Shawn
Re: Really bad query performance for date range queries
Posted by Shawn Heisey <so...@elyograg.org>.
On 2/5/2013 12:51 PM, sausarkar wrote:
> We have a 96GB ram machine with 16 processors. the JVM is set to use 60 GB.
> The test that we are running are purely query there is no indexing going on.
> I dont see garbage collection when I attach visualVM but see frequent CPU
> spikes ~once every minute.
A previous message from you indicates that your index is 12GB. I agree
with Erick that this is not very large. The pauses that you have
described sound a lot like stop-the-world garbage collection. I've seen
very long pauses on an 8GB heap ... I don't even want to think about
what could happen on 60GB.
Do you really need a 60GB heap? My dev server handles seven index
shards with a 7GB heap and 16GB total RAM. On 4.1 the total index size
is is over 100GB. On 4.2-SNAPSHOT the total index size is about 83GB.
Query performance isn't stellar, but it works perfectly. My production
servers (running 3.5) have tons of RAM and each one only gets half the
index, but they only run with the heap at 8GB. My queries are pretty
low volume and not HUGELY complex. Median query time is about 26
milliseconds and 95th percentile is about 950 milliseconds.
Looking at the GC stats in jconsole/jvisualvm, I didn't think I had a GC
pause problem, but I was proven wrong when I started correlating all the
various logs in my system to load balancer "DOWN" incidents. I saw a
pause of 12 seconds once in the GC log - on an 8GB heap.
I was introduced to a very cool program that tracks any kind of pause
that's caused by factors outside the Java program, like GC pauses in the
JVM or something happening in the OS. This is much easier to interpret
than Java's GC logging, and you can get a nice graph from the data.
http://www.azulsystems.com/jHiccup
Using jHiccup, I was able to do a little bit of comparison between
different runs. That helped me find some GC tuning parameters that have
almost gotten rid of my GC pause problem. I'm constantly working on
those parameters. The current values are:
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:NewRatio=3
-XX:MaxTenuringThreshold=8
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+UseLargePages
-XX:+AggressiveOpts
The Xing JVM (made by the company that created jHiccup) apparently has
extremely low GC pause characteristics even with giant heaps like yours.
I'm not using it, and I don't know how much it costs.
Thanks,
Shawn
Re: Really bad query performance for date range queries
Posted by sausarkar <sa...@ebay.com>.
We have a 96GB ram machine with 16 processors. the JVM is set to use 60 GB.
The test that we are running are purely query there is no indexing going on.
I dont see garbage collection when I attach visualVM but see frequent CPU
spikes ~once every minute.
--
View this message in context: http://lucene.472066.n3.nabble.com/Really-bad-query-performance-for-date-range-queries-tp4038435p4038633.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Really bad query performance for date range queries
Posted by Erick Erickson <er...@gmail.com>.
On the surface, this doesn't seem right. Your corpus isn't very large and
it sounds like you have some significant hardware supporting it. I'm
guessing you have some other issues here like;
- inappropriate memory allocation
- too-frequent commits
- too many queries for your hardware
- indexing to the same machine
I'd attach a memory profiler to the app and watch that, also are you sure
it's the fq? You should be able to just submit various forms of the query
by hand. You may need to throw more hardware at it.
Best
Erick
On Mon, Feb 4, 2013 at 6:14 PM, sausarkar <sa...@ebay.com> wrote:
> Unfortunately we need data by minute we cannot go hour, is there an option
> for 3 minutes or 5 minutes? something is like NOW/3MIN?
>
> I am also noticing when I generating around 110 queries per second (date
> range ones) after sometime solr does not respond and just freezes. Is there
> a way to cure this?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Really-bad-query-performance-for-date-range-queries-tp4038435p4038455.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Really bad query performance for date range queries
Posted by sausarkar <sa...@ebay.com>.
Unfortunately we need data by minute we cannot go hour, is there an option
for 3 minutes or 5 minutes? something is like NOW/3MIN?
I am also noticing when I generating around 110 queries per second (date
range ones) after sometime solr does not respond and just freezes. Is there
a way to cure this?
--
View this message in context: http://lucene.472066.n3.nabble.com/Really-bad-query-performance-for-date-range-queries-tp4038435p4038455.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Really bad query performance for date range queries
Posted by Jack Krupansky <ja...@basetechnology.com>.
The core concept behind filter queries is that they cache their results
(list of documents) so that they can filter the main query more efficiently.
But is your query keeps changing, like every minute, the cached results need
to get thrown away and recalculated, every minute.
Can you try hourly resolution or even daily resolution instead?
Or, disable caching for the offending filter query, as in:
Example: q=*:*&fq={!cache=false}inStock:true
Example: q=*:*&fq={!frange l=1 u=4 cache=false}sqrt(popularity)
-- Jack Krupansky
-----Original Message-----
From: sausarkar
Sent: Monday, February 04, 2013 4:48 PM
To: solr-user@lucene.apache.org
Subject: Really bad query performance for date range queries
we are experiencing very bad performance issues with date range queries. We
have configured the date fields as following: Our queries are rounded every
minute:
qt=ads&debugQuery=false&fl=id,StartDt_t110,.......&fq=Status_i110:2&fq=StartDt_t110:{
NOW/MINUTE-150DAYS TO NOW/MINUTE-90DAYS }&start=0&rows=20 Index size - 10
million documents in Solr 4.1 Start of every minute we see the query speed
to be about 10-12 seconds and increases over time. When the query time jump
to more than 10 seconds the linux *load average spikes up to more than 100
*in a 16 CPU machine. Any one has any suggestions how this can be fixed?
--
View this message in context:
http://lucene.472066.n3.nabble.com/Really-bad-query-performance-for-date-range-queries-tp4038435.html
Sent from the Solr - User mailing list archive at Nabble.com.