server down caused by complex query

["Jason, Kim" <hi...@gmail.com>]

Hi all

Nowadays our solr server is frequently down.
Because our user send very long and complex queries with asterisk and near
operator.
Sometimes near operator exceeds 1,000 and keywords almost include asterisk.
If such query is sent to server, jvm memory is full. (our jvm memory
allocates 110G.)
After that, server is like down.

We also have old version's k2 engine.
But k2 is not down for same query.
k2 uses more i/o than memory.

Could we control solr memory usage?
Or is there any other solution?
(we are using solr1.4)

Thanks in advance.
Jason

--
View this message in context: http://lucene.472066.n3.nabble.com/server-down-caused-by-complex-query-tp3535506p3535506.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: server down caused by complex query

[Jason <hi...@gmail.com>]

Hellow, Hoss

We're using ComplexPhraseQueryParser and maxBooleanClauses setting is
1000000.
I know maxBooleanClauses is so big.
But we are expert search organization and queries are very complex and
include wildcard.
So we need it.
Our application receives type of queries like ((A* OR B* OR C*,...) n/2 (X*
OR Y* OR Z*,...)) AND (...)  from user.
Then it is converted into solr query like ("A* X*"~2 OR "A* Y*"~2 OR "A*
Z*"~2 OR "B* X*"~2 OR ...) AND (...).
Like above, queries for near expression is written repeatedly.
I expect this is inefficient and why jvm memory is full.

I think surround query parser may our solution.
So now we are customizing surround query parser because it is very limited.


Below is out tomcat setenv...
======================================================================
export CATALINA_OPTS="-Xms112640m -Xmx112640m"
export CATALINA_OPTS="$CATALINA_OPTS -Dserver"
export CATALINA_OPTS="$CATALINA_OPTS
-Djava.library.path=/usr/local/lib:/usr/local/apr/lib"
export CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9014
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false"
export CATALINA_OPTS="$CATALINA_OPTS -Dfile.encoding=utf-8"
export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseConcMarkSweepGC"
======================================================================

Thanks
Jason

--
View this message in context: http://lucene.472066.n3.nabble.com/server-down-caused-by-complex-query-tp3535506p3581218.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: server down caused by complex query

[Chris Hostetter <ho...@fucit.org>]

: Because our user send very long and complex queries with asterisk and near
: operator.
: Sometimes near operator exceeds 1,000 and keywords almost include asterisk.
: If such query is sent to server, jvm memory is full. (our jvm memory

"near" operator isn't something I know of as a built in feature of SOlr 
(definitely not Solr 1.4) ... which query parser are you using?  

what is the value of your <maxBooleanClauses> setting in solrconfig.xml?  
that's the method that should help to limit the risk of query explosion if 
users try to overwelm the server with really large queries, but for 
wildcard and prefix queries (ie: using "*") even Solr 1.4 implemented 
those using "ConstantScoreQuery" instead of using query expansion, so i'm 
no sure how/why a single query could eat up so much ram.

In general, there have been a lot of improvements in memory usage in 
recent versions of Solr, so i suggest you upgrade to Solr 3.5 -- but 
beyond that basic advice any other suggestions will require a *lot* more 
specifics about exactly waht your configs look like, the full requests 
(all params) of queries that are causing you problems, detials on your 
JVM configuration, etc... 

-Hoss