You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Shawn Heisey <so...@elyograg.org> on 2012/12/21 08:53:21 UTC

Solr 3.5: java.lang.NegativeArraySizeException caused by negative start value

This is on Solr 3.5.0.

We are getting a java.lang.NegativeArraySizeException when our webapp 
sends a query where the start parameter is set to a negative value. 
This seems to set off a denial of service problem within Solr.  I don't 
yet know whether it's a mistake in coding, or whether some malicious 
user has found an attack vector on our site.

After the first exception, another exception 
(org.mortbay.jetty.EofException) appears in the logs with increasing 
frequency.  Within minutes of the first exception, the load balancer 
complains about having no servers available because ping requests are 
failing.

This is distributed search, but the shards parameter is in 
solrconfig.xml, not provided by the client.

Full exception:

Dec 20, 2012 7:41:34 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.NegativeArraySizeException
         at 
org.apache.lucene.util.PriorityQueue.initialize(PriorityQueue.java:108)
         at 
org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:139)
         at 
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:712)
         at 
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:571)
         at 
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:550)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:289)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:208)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372)
         at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
         at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
         at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
         at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
         at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
         at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
         at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
         at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
         at 
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
         at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
         at org.mortbay.jetty.Server.handle(Server.java:326)
         at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
         at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
         at 
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
         at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)


Later exceptions:

Dec 21, 2012 12:24:37 AM org.apache.solr.common.SolrException log
SEVERE: org.mortbay.jetty.EofException
         at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
         at 
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
         at 
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
         at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278)
         at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
         at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
         at 
org.apache.solr.common.util.FastWriter.flush(FastWriter.java:115)
         at 
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:344)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265)
         at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
         at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
         at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
         at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
         at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
         at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
         at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
         at 
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
         at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
         at org.mortbay.jetty.Server.handle(Server.java:326)
         at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
         at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
         at 
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
         at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.net.SocketException: Broken pipe
         at java.net.SocketOutputStream.socketWrite0(Native Method)
         at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
         at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
         at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368)
         at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129)
         at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161)
         at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714)
         ... 25 more

Re: Solr 3.5: java.lang.NegativeArraySizeException caused by negative start value

Posted by Shawn Heisey <so...@elyograg.org>.
On 12/21/2012 3:31 PM, Chris Hostetter wrote:
> : My Solr 4.1 dev server doesn't seem to have a problem handling this.  I see a
> : substantially similar exception when I send a similar request.
> :
> : I have a clarification, at least for 4.1: That specific exception is when it's
> : a large negative number - larger than the numFound of the query.  With a small
> : negative number, it's a different exception.  I won't be trying a small
>
> weird ... definitely seems like something we should just nip in the bud --
> fail hard and fast if start is < 0.
>
> can you file a jira please?

Looks like it was a malicious user trying to break into our site.  That 
user was sending unusual values for our site parameters, including a 
negative page number.  The webapp happily used that value as-is.  The 
inputs are now being scrubbed.

In 3.5, the negative start parameter on the distributed search is 
resulting in the same negative number being moved to the rows parameter 
on the searches being sent to the shards.  If I'm not mistaken, a 
negative number for rows means "all results."  I suspect that the DoS 
problem is because the shards are trying to send all their matching rows 
- the log entries say this is about 8 million on each of six shards.  
This overloads the system, which valiantly tries to make the insane 
request finish, and everything comes to a halt.

Failing hard & fast if there's a negative number on the start parameter 
would stop the entire problem from happening.  I am thinking this means 
that I should not worry about the underlying DoS issue in 3.x and just 
file the first one for 3.6 and 4.0.  Is that right?

I have been experimenting in 4.1 and it doesn't seem to have the denial 
of service problem.  It looks like the entire request gets killed at the 
first exception.

Thanks,
Shawn

Re: Solr 3.5: java.lang.NegativeArraySizeException caused by negative start value

Posted by Chris Hostetter <ho...@fucit.org>.
: My Solr 4.1 dev server doesn't seem to have a problem handling this.  I see a
: substantially similar exception when I send a similar request.
: 
: I have a clarification, at least for 4.1: That specific exception is when it's
: a large negative number - larger than the numFound of the query.  With a small
: negative number, it's a different exception.  I won't be trying a small

weird ... definitely seems like something we should just nip in the bud -- 
fail hard and fast if start is < 0.

can you file a jira please?


-Hoss

Re: Solr 3.5: java.lang.NegativeArraySizeException caused by negative start value

Posted by Shawn Heisey <so...@elyograg.org>.
On 12/21/2012 12:53 AM, Shawn Heisey wrote:
> This is on Solr 3.5.0.
>
> We are getting a java.lang.NegativeArraySizeException when our webapp
> sends a query where the start parameter is set to a negative value. This
> seems to set off a denial of service problem within Solr.  I don't yet
> know whether it's a mistake in coding, or whether some malicious user
> has found an attack vector on our site.

My Solr 4.1 dev server doesn't seem to have a problem handling this.  I 
see a substantially similar exception when I send a similar request.

I have a clarification, at least for 4.1: That specific exception is 
when it's a large negative number - larger than the numFound of the 
query.  With a small negative number, it's a different exception.  I 
won't be trying a small negative number on my 3.5 servers, they're in 
production and I don't want to make them explode.

Thanks,
Shawn