You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Fuad Efendi <fu...@efendi.ca> on 2009/12/20 20:53:45 UTC
SOLR Performance Tuning: Disable INFO Logging.
After researching how to configure default SOLR & Tomcat logging, I finally
disabled INFO-level for SOLR.
And performance improved at least 7 times!!! ('at least 7' because I
restarted server 5 minutes ago; caches are not prepopulated yet)
Before that, I had 300-600 ms in HTTPD log files in average, and 4%-8% I/O
wait whenever "top" commands shows SOLR on top.
Now, I have 50ms-100ms in average (total response time logged by HTTPD).
P.S.
Of course, I am limited in RAM, and I use slow SATA... server is moderately
loaded, 5-10 requests per second.
P.P.S.
And suddenly synchronous I/O by Java/Tomcat Logger slows down performance
much higher than read-only I/O of Lucene.
Fuad Efendi
+1 416-993-2060
http://www.linkedin.com/in/liferay
Tokenizer Inc.
http://www.tokenizer.ca/
Data Mining, Vertical Search
RE: SOLR Performance Tuning: Disable INFO Logging.
Posted by Fuad Efendi <fu...@efendi.ca>.
> Can you quickly explain what you did to disable INFO-Level?
>
> I am from a PHP background and am not so well versed in Tomcat or
> Java. Is this a section in solrconfig.xml or did you have to edit
> Solr Java source and recompile?
1. Create a file called logging.properties with following content (I created it in /home/tomcat/solr folder):
.level=INFO
handlers= java.util.logging.ConsoleHandler, java.util.logging.FileHandler
java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
java.util.logging.FileHandler.level = INFO
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
java.util.logging.ConsoleHandler.level = ALL
org.apache.solr.level=SEVERE
2. Modify file tomcat_installation/bin/catalina.sh to include following (as a first line in script):
JAVA_OPTS="... ... ... -Djava.util.logging.config.file=/home/tomcat/solr/logging.properties"
(this line may include more parameters such as -Xmx8196m for memory, -Dfile.encoding=UTF8 -Dsolr.solr.home=/home/tomcat/solr -Dsolr.data.dir=/home/tomcat/solr for SOLR, etc.)
With these settings, SOLR (and Tomcat) will use standard Java 5/6 logging capabilities. Log output will default to standard /logs folder of Tomcat.
You may find additional logging configuration settings by google for "Java 5 Logging" etc.
>
>
> 2009/12/20 Fuad Efendi <fu...@efendi.ca>:
> > After researching how to configure default SOLR & Tomcat logging, I
> finally
> > disabled INFO-level for SOLR.
> >
> > And performance improved at least 7 times!!! ('at least 7' because I
> > restarted server 5 minutes ago; caches are not prepopulated yet)
> >
> > Before that, I had 300-600 ms in HTTPD log files in average, and 4%-8%
> I/O
> > wait whenever "top" commands shows SOLR on top.
> >
> > Now, I have 50ms-100ms in average (total response time logged by HTTPD).
> >
> >
> > P.S.
> > Of course, I am limited in RAM, and I use slow SATA... server is
> moderately
> > loaded, 5-10 requests per second.
> >
> >
> > P.P.S.
> > And suddenly synchronous I/O by Java/Tomcat Logger slows down
> performance
> > much higher than read-only I/O of Lucene.
> >
> >
> >
> > Fuad Efendi
> > +1 416-993-2060
> > http://www.linkedin.com/in/liferay
> >
> > Tokenizer Inc.
> > http://www.tokenizer.ca/
> > Data Mining, Vertical Search
> >
> >
> >
> >
> >
Fuad Efendi
+1 416-993-2060
http://www.linkedin.com/in/liferay
Tokenizer Inc.
http://www.tokenizer.ca/
Data Mining, Vertical Search
Re: SOLR Performance Tuning: Disable INFO Logging.
Posted by Andrew McCombe <eu...@gmail.com>.
Hi
Can you quickly explain what you did to disable INFO-Level?
I am from a PHP background and am not so well versed in Tomcat or
Java. Is this a section in solrconfig.xml or did you have to edit
Solr Java source and recompile?
Thanks In Advance
Andrew
2009/12/20 Fuad Efendi <fu...@efendi.ca>:
> After researching how to configure default SOLR & Tomcat logging, I finally
> disabled INFO-level for SOLR.
>
> And performance improved at least 7 times!!! ('at least 7' because I
> restarted server 5 minutes ago; caches are not prepopulated yet)
>
> Before that, I had 300-600 ms in HTTPD log files in average, and 4%-8% I/O
> wait whenever "top" commands shows SOLR on top.
>
> Now, I have 50ms-100ms in average (total response time logged by HTTPD).
>
>
> P.S.
> Of course, I am limited in RAM, and I use slow SATA... server is moderately
> loaded, 5-10 requests per second.
>
>
> P.P.S.
> And suddenly synchronous I/O by Java/Tomcat Logger slows down performance
> much higher than read-only I/O of Lucene.
>
>
>
> Fuad Efendi
> +1 416-993-2060
> http://www.linkedin.com/in/liferay
>
> Tokenizer Inc.
> http://www.tokenizer.ca/
> Data Mining, Vertical Search
>
>
>
>
>
RE: SOLR Performance Tuning: Disable INFO Logging.
Posted by Fuad Efendi <fu...@efendi.ca>.
We were talking about GC options a lot; don't forget to enclose following
into "if (log.isInfoEnabled())":
...
final NamedList<Object> responseHeader = new SimpleOrderedMap<Object>();
rsp.add("responseHeader", responseHeader);
NamedList toLog = rsp.getToLog();
//toLog.add("core", getName());
toLog.add("webapp", req.getContext().get("webapp"));
toLog.add("path", req.getContext().get("path"));
toLog.add("params", "{" + req.getParamString() + "}");
handler.handleRequest(req,rsp);
setResponseHeaderValues(handler,req,rsp);
StringBuilder sb = new StringBuilder();
for (int i=0; i<toLog.size(); i++) {
String name = toLog.getName(i);
Object val = toLog.getVal(i);
sb.append(name).append("=").append(val).append(" ");
}
log.info(logid + sb.toString());...
...
-Fuad
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: December-20-09 2:54 PM
> To: solr-user@lucene.apache.org
> Subject: SOLR Performance Tuning: Disable INFO Logging.
>
> After researching how to configure default SOLR & Tomcat logging, I
> finally
> disabled INFO-level for SOLR.
>
> And performance improved at least 7 times!!! ('at least 7' because I
> restarted server 5 minutes ago; caches are not prepopulated yet)
>
> Before that, I had 300-600 ms in HTTPD log files in average, and 4%-8% I/O
> wait whenever "top" commands shows SOLR on top.
>
> Now, I have 50ms-100ms in average (total response time logged by HTTPD).
>
>
> P.S.
> Of course, I am limited in RAM, and I use slow SATA... server is
> moderately
> loaded, 5-10 requests per second.
>
>
> P.P.S.
> And suddenly synchronous I/O by Java/Tomcat Logger slows down performance
> much higher than read-only I/O of Lucene.
>
>
>
> Fuad Efendi
> +1 416-993-2060
> http://www.linkedin.com/in/liferay
>
> Tokenizer Inc.
> http://www.tokenizer.ca/
> Data Mining, Vertical Search
>
>
>