Solr webapp deployment is very slow with in solrconfig.xml

[汤林 <ta...@gmail.com>]

Have opened JIRA for that:

https://issues.apache.org/jira/browse/SOLR-6675


We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr
index(cores) are big(50~100G) each core.

When we start up tomcat, the solr webapp deployment is very slow. From
tomcat's catalina log, every time it takes about 10 minutes to get
deployed. After we analyzing java core dump, we notice it's because the
loading process cannot finish until the MBean calculation for large index
is done.

So we tried to remove the <jmx/> from solrconfig.xml, after that, the
loading of solr webapp only take about 1 minute. So we can sure the MBean
calculation for large index is the root cause.

Could you please point me if there is any async way to do statistic
monitoring without <jmx/> in solrconfig.xml, or let it do calculation after
the deployment? Thanks!

The callstack.png file in the attachment is the call stack of the long
blocking thread which is doing statistics calculation.

The catalina log of tomcat:
INFO: Starting Servlet Engine: Apache Tomcat/7.0.54
Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive
/opt/ibm/solrsearch/tomcat/webapps/solr.war
Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive
/opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms
<---------------- Time taken for solr app Deployment is about 10 minutes
-----------
Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
/opt/ibm/solrsearch/tomcat/webapps/manager
Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
/opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms
Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
/opt/ibm/solrsearch/tomcat/webapps/examples
Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
/opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms
Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
/opt/ibm/solrsearch/tomcat/webapps/docs
Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
/opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms
Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
/opt/ibm/solrsearch/tomcat/webapps/ROOT
Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
/opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms
Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory
/opt/ibm/solrsearch/tomcat/webapps/host-manager
Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
/opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms
Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 601506 ms