Stopping Solr results in broken Solr - killing me! CoreContainer is either not initialized or shutting down.

[Russell Jurney <ru...@gmail.com>]

Whenever I stop Solr with 'bin/solr stop' or 'sudo service solr stop'
Solr's cores are corrupted and the only way to remedy the situation is to
do a full uninstall and reinstall of all traces of Solr.

Please help, I can't live like this. The error is at the bottom, as it is
long.

This is a standard Ubuntu 18.04 AMI  for AWS on an r5.4xlarge instance with
a 1024GB EBS drive. The OS settings we have altered to get Solr running are:

# Set file limit for Solr higher than 1024

echo "# Solr LIMIT must be higher than 1024" | sudo tee -a
/etc/security/limits.conf

echo "${USER} hard nofile 65535"  | sudo tee -a /etc/security/limits.conf

echo "${USER} soft nofile 65535"  | sudo tee -a /etc/security/limits.conf

echo "${USER} hard nproc 65535"  | sudo tee -a /etc/security/limits.conf

echo "${USER} soft nproc 65535"  | sudo tee -a /etc/security/limits.conf

# Fix available entropy

echo "HRNGDEVICE=/dev/urandom" | sudo tee -a /etc/default/rng-tools

sudo /etc/init.d/rng-tools restart


I am installing Solr like this:

sudo bin/install_solr_service.sh ~/solr-8.5.2.tgz


This is to the point that I can't even restart Solr to set SOLR_HOST for
security reasons. It is madness.

I create the cores with this command:

sudo -u solr `which solr` create_core -c ${CORE} -d ${CORE_CONFIG_DIR}/ -V


I load the cores with this command:

sudo -u solr `which post` -c ${CORE} ${DATA_PATH}


I stop Solr with this command:

sudo solr service stop

# or

sudo /opt/solr/bin/solr stop


/var/solr is owned by user solr. /opt/solr is owned by user solr.

I just find myself wondering - am I making a mistake by not running stop
solr as user solr?

Anyway, I have no idea what is wrong. Please help!

I start Solr like this the most recent time:

sudo -u solr `which solr` start

Waiting up to 180 seconds to see Solr running on port 8983 [|]
Started Solr server on port 8983 (pid=1697). Happy searching!


This happens when I stop solr no matter how I do it, via service or solr
command, as solr or root. Deleting the data directory doesn't help. No
removal of cores helps. I use the script above to remove Solr and then
reinstall it and create/load the cores again and only then does it work :(

Error at http://solr.host:8983/solr/ follows:


HTTP ERROR 404 javax.servlet.UnavailableException: Error processing
the request. CoreContainer is either not initialized or shutting
down.URI:	/solr/STATUS:	404MESSAGE:	javax.servlet.UnavailableException:
Error processing the request. CoreContainer is either not initialized
or shutting down.SERVLET:	default
CAUSED BY:	javax.servlet.ServletException:
javax.servlet.UnavailableException: Error processing the request.
CoreContainer is either not initialized or shutting down.
CAUSED BY:	javax.servlet.UnavailableException: Error processing the
request. CoreContainer is either not initialized or shutting down.
Caused by:
javax.servlet.ServletException: javax.servlet.UnavailableException:
Error processing the request. CoreContainer is either not initialized
or shutting down.
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.servlet.UnavailableException: Error processing the
request. CoreContainer is either not initialized or shutting down.
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:371)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
	at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 18 more


Thanks,
Russell Jurney @rjurney <http://twitter.com/rjurney>
russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
<http://facebook.com/jurney> datasyndrome.com