Tomcat7 with Solr closes at fixed hours, every time another hour

[Adrian Fita <ad...@gmail.com>]

Hello.

I'm having a Solr running within Tomcat7 and Tomcat is closing at
fixed hours, everytime is a different hour. catalina.log doesn't show
anything other than a clean tomcat shutdown  (no exception or
anything). I would really apreciate some advice on how to debug this.
Tomcat doesn't run anything other than solr.

The Context XML definition for the solr application is the following:

<Context path="/solr" docBase="/opt/solr/webapp">
        <Environment name="solr/home" type="java.lang.String"
value="/opt/solr/solr-home/" override="true"/>
</Context>

Here are some relevant messages from catalina.log:

Aug 12, 2011 5:53:41 PM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {optimize=} 0 6255
Aug 12, 2011 5:53:41 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/update params={wt=json} status=0 QTime=6255
Aug 12, 2011 6:00:01 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8081"]
Aug 12, 2011 6:00:02 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-8010"]
Aug 12, 2011 6:00:03 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Aug 12, 2011 6:00:03 PM org.apache.solr.core.SolrCore close
INFO: []  CLOSING SolrCore org.apache.solr.core.SolrCore@16d7894
Aug 12, 2011 6:00:03 PM org.apache.solr.update.DirectUpdateHandler2 close
INFO: closing DirectUpdateHandler2{commits=9408,autocommits=0,optimizes=9408,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=7500,cumulative_deletesById=1908,cumulative_deletesByQuery=0,cumulative_errors=0}
Aug 12, 2011 6:00:03 PM org.apache.solr.update.DirectUpdateHandler2 close
INFO: closed DirectUpdateHandler2{commits=9408,autocommits=0,optimizes=9408,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=7500,cumulative_deletesById=1908,cumulative_deletesByQuery=0,cumulative_errors=0}
Aug 12, 2011 6:00:03 PM org.apache.solr.core.SolrCore closeSearcher
INFO: [] Closing main searcher on request.
Aug 12, 2011 6:00:03 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@1b0f1ab main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=7958,cumulative_hits=25,cumulative_hitratio=0.00,cumulative_inserts=7935,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=736,cumulative_hits=442,cumulative_hitratio=0.60,cumulative_inserts=294,cumulative_evictions=0}
Aug 12, 2011 6:00:04 PM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/solr] created a ThreadLocal with key of
type [org.apache.solr.schema.DateField.ThreadLocalDateFormat] (value
[org.apache.solr.schema.DateField$ThreadLocalDateFormat@e2d63f]) and a
value of type [org.apache.solr.schema.DateField.ISO8601CanonicalDateFormat]
(value [org.apache.solr.schema.DateField$ISO8601CanonicalDateFormat@6b2ed43a])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
Aug 12, 2011 6:00:04 PM org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/solr] created a ThreadLocal with key of
type [org.apache.solr.schema.DateField.ThreadLocalDateFormat] (value
[org.apache.solr.schema.DateField$ThreadLocalDateFormat@e2d63f]) and a
value of type [org.apache.solr.schema.DateField.ISO8601CanonicalDateFormat]
(value [org.apache.solr.schema.DateField$ISO8601CanonicalDateFormat@6b2ed43a])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
Aug 12, 2011 6:00:04 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8081"]
Aug 12, 2011 6:00:04 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8010"]
Aug 12, 2011 6:00:04 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8081"]
Aug 12, 2011 6:00:04 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-8010"]

--
Fita Adrian

Re: Tomcat7 with Solr closes at fixed hours, every time another hour

[Chris Hostetter <ho...@fucit.org>]

: 
: I'm having a Solr running within Tomcat7 and Tomcat is closing at
: fixed hours, everytime is a different hour. catalina.log doesn't show
: anything other than a clean tomcat shutdown  (no exception or
: anything). I would really apreciate some advice on how to debug this.
: Tomcat doesn't run anything other than solr.

this doesn't appear to be related to Solr.  You can see from your logs 
that the command originats from outside of solr -- I suspect you would see 
the same problem if you ran a tomcat instance on this port w/o using solr 
at all.

My guess is you have a rogue cron command either running on the local 
machine or using the remote shutdown port  telling tomcat to shutdown.  
(perhaps it's looking for tomcat ports whose logs suggest they aren't 
getting a lot of traffic?  or aren't registered with a load balancer?)

You might want to start by making sure you have remote shutdown support 
disabled...
https://tomcat.apache.org/tomcat-7.0-doc/security-howto.html#Server

...and checking the crontab on the local machine to see what runs on the 
hour.



-Hoss