You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Katherine Mora <KM...@searchtechnologies.com> on 2016/08/12 17:27:41 UTC

ConcurrentModificationException due to high volume of requests

Hi All,

We are using Solr 5.2.1 in a production environment where we have a high volume of requests. We are not having any timeouts or performance issues. However, the logs are filled with this exception.
We've been researching and there isn't a lot of information about this problem or why it happens. We have tried to reproduce it by sending queries using JMeter and it only happens when multiple queries are sent at the same time.

Checking the code we found that this exception is within the core Solr code and part of the exception caught our attention as it is logging debug information (we are using INFO logging level) (org.apache.solr.search.stats.LocalStatsCache.get(LocalStatsCache.java:40)).

Has anyone seen this exception before? Would it be OK to generate a patch? We were thinking about commenting the debug line or adding try/catch statements.

Thank you!


null:java.util.ConcurrentModificationException
         at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
         at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
         at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
         at org.apache.solr.common.params.ModifiableSolrParams.toString(ModifiableSolrParams.java:201)
         at java.lang.String.valueOf(String.java:2849)
         at java.lang.StringBuilder.append(StringBuilder.java:128)
         at org.apache.solr.request.SolrQueryRequestBase.toString(SolrQueryRequestBase.java:165)
         at org.apache.solr.search.stats.LocalStatsCache.get(LocalStatsCache.java:40)
         at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:366)
         at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:255)
         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
         at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
         at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
         at org.eclipse.jetty.server.Server.handle(Server.java:497)
         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
         at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
         at java.lang.Thread.run(Thread.java:745)


KATHERINE MORA
Senior Engineer

RE: ConcurrentModificationException due to high volume of requests

Posted by Katherine Mora <KM...@searchtechnologies.com>.

Thank you Shalin. I've created SOLR-9414

KATHERINE MORA
Senior Engineer

-----Original Message-----
From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com] 
Sent: Saturday, August 13, 2016 7:22 PM
To: solr-user@lucene.apache.org
Subject: Re: ConcurrentModificationException due to high volume of requests

This is a bug. Please open a Jira issue. We need to dig where else this param map is being used. It is possible that there are more such bugs.

On Fri, Aug 12, 2016 at 10:57 PM, Katherine Mora < KMora@searchtechnologies.com> wrote:

> Hi All,
>
> We are using Solr 5.2.1 in a production environment where we have a 
> high volume of requests. We are not having any timeouts or performance issues.
> However, the logs are filled with this exception.
> We've been researching and there isn't a lot of information about this 
> problem or why it happens. We have tried to reproduce it by sending 
> queries using JMeter and it only happens when multiple queries are 
> sent at the same time.
>
> Checking the code we found that this exception is within the core Solr 
> code and part of the exception caught our attention as it is logging 
> debug information (we are using INFO logging level) (org.apache.solr.search.stats.
> LocalStatsCache.get(LocalStatsCache.java:40)).
>
> Has anyone seen this exception before? Would it be OK to generate a patch?
> We were thinking about commenting the debug line or adding try/catch 
> statements.
>
> Thank you!
>
>
> null:java.util.ConcurrentModificationException
>          at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(
> LinkedHashMap.java:394)
>          at java.util.LinkedHashMap$EntryIterator.next(
> LinkedHashMap.java:413)
>          at java.util.LinkedHashMap$EntryIterator.next(
> LinkedHashMap.java:412)
>          at 
> org.apache.solr.common.params.ModifiableSolrParams.toString(
> ModifiableSolrParams.java:201)
>          at java.lang.String.valueOf(String.java:2849)
>          at java.lang.StringBuilder.append(StringBuilder.java:128)
>          at org.apache.solr.request.SolrQueryRequestBase.toString(
> SolrQueryRequestBase.java:165)
>          at org.apache.solr.search.stats.LocalStatsCache.get(
> LocalStatsCache.java:40)
>          at org.apache.solr.handler.component.QueryComponent.
> process(QueryComponent.java:366)
>          at org.apache.solr.handler.component.SearchHandler.
> handleRequestBody(SearchHandler.java:255)
>          at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:143)
>          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>          at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:654)
>          at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:450)
>          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:227)
>          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:196)
>          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1652)
>          at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:585)
>          at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>          at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:577)
>          at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:223)
>          at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
>          at org.eclipse.jetty.server.handler.RequestLogHandler.
> handle(RequestLogHandler.java:95)
>          at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1129)
>          at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:515)
>          at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>          at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1061)
>          at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>          at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:215)
>          at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:110)
>          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
>          at org.eclipse.jetty.server.Server.handle(Server.java:497)
>          at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:310)
>          at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:257)
>          at org.eclipse.jetty.io.AbstractConnection$2.run(
> AbstractConnection.java:540)
>          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:635)
>          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:555)
>          at java.lang.Thread.run(Thread.java:745)
>
>
> KATHERINE MORA
> Senior Engineer
>
>


--
Regards,
Shalin Shekhar Mangar.

Re: ConcurrentModificationException due to high volume of requests

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.

This is a bug. Please open a Jira issue. We need to dig where else this
param map is being used. It is possible that there are more such bugs.

On Fri, Aug 12, 2016 at 10:57 PM, Katherine Mora <
KMora@searchtechnologies.com> wrote:

> Hi All,
>
> We are using Solr 5.2.1 in a production environment where we have a high
> volume of requests. We are not having any timeouts or performance issues.
> However, the logs are filled with this exception.
> We've been researching and there isn't a lot of information about this
> problem or why it happens. We have tried to reproduce it by sending queries
> using JMeter and it only happens when multiple queries are sent at the same
> time.
>
> Checking the code we found that this exception is within the core Solr
> code and part of the exception caught our attention as it is logging debug
> information (we are using INFO logging level) (org.apache.solr.search.stats.
> LocalStatsCache.get(LocalStatsCache.java:40)).
>
> Has anyone seen this exception before? Would it be OK to generate a patch?
> We were thinking about commenting the debug line or adding try/catch
> statements.
>
> Thank you!
>
>
> null:java.util.ConcurrentModificationException
>          at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(
> LinkedHashMap.java:394)
>          at java.util.LinkedHashMap$EntryIterator.next(
> LinkedHashMap.java:413)
>          at java.util.LinkedHashMap$EntryIterator.next(
> LinkedHashMap.java:412)
>          at org.apache.solr.common.params.ModifiableSolrParams.toString(
> ModifiableSolrParams.java:201)
>          at java.lang.String.valueOf(String.java:2849)
>          at java.lang.StringBuilder.append(StringBuilder.java:128)
>          at org.apache.solr.request.SolrQueryRequestBase.toString(
> SolrQueryRequestBase.java:165)
>          at org.apache.solr.search.stats.LocalStatsCache.get(
> LocalStatsCache.java:40)
>          at org.apache.solr.handler.component.QueryComponent.
> process(QueryComponent.java:366)
>          at org.apache.solr.handler.component.SearchHandler.
> handleRequestBody(SearchHandler.java:255)
>          at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:143)
>          at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
>          at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:654)
>          at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:450)
>          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:227)
>          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:196)
>          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1652)
>          at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:585)
>          at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>          at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:577)
>          at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:223)
>          at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
>          at org.eclipse.jetty.server.handler.RequestLogHandler.
> handle(RequestLogHandler.java:95)
>          at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1129)
>          at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:515)
>          at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>          at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1061)
>          at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>          at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:215)
>          at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:110)
>          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
>          at org.eclipse.jetty.server.Server.handle(Server.java:497)
>          at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:310)
>          at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:257)
>          at org.eclipse.jetty.io.AbstractConnection$2.run(
> AbstractConnection.java:540)
>          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:635)
>          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:555)
>          at java.lang.Thread.run(Thread.java:745)
>
>
> KATHERINE MORA
> Senior Engineer
>
>


-- 
Regards,
Shalin Shekhar Mangar.