You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ravi Solr <ra...@gmail.com> on 2013/04/19 20:55:36 UTC
Weird query issues
Hello,
We are using Solr 3.6.2 single core ( both index and query on same machine)
and randomly the server fails to query correctly. If we query from the
admin console the query is not even applied and it returns numFound count
equal to total docs in the index as if no query is made, and if use SOLRJ
to query it throws javabin error
Invalid version (expected 2, but 60) or the data in not in 'javabin' format
Once we restart the container everything is back to normal.
In the process of debugging the solr logs I found empty queries like the
one below. Can anybody tell me what can cause empty queries in the log as
given below so trying to see if it may be relateed to the solr issues
[#|2013-04-19T14:10:20.308-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=19;_ThreadName=httpSSLWorkerThread-9001-0;|[core1]
webapp=/solr path=/select params={} hits=21727 status=0 QTime=24 |#]
Would Appreciate any pointers
Thanks
Ravi Kiran Bhaskar
Re: Weird query issues
Posted by Ravi Solr <ra...@gmail.com>.
Thanks Shawn, We are using 3.6.2 client and server. I cleared my browser
cache several times while querying (is that similar to clear cache in
solrconfig.xml ?). The query is logged in the solrj based client's
application container however I see it empty in the solr's application
container...so somehow it is getting swallowed by solr...Iam not able to
figure out how and why ?
Thanks
Ravi Kiran Bhaskar
On Fri, Apr 26, 2013 at 4:33 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/26/2013 1:01 PM, Ravi Solr wrote:
> > Hello Shawn,
> > We found that it is unrelated to the group queries instead more
> > related to the empty queries. Do you happen to know what could cause
> empty
> > queries like the following from SOLRJ ? I can generate similar query via
> > curl hitting the select handler like - http://server:port/solr/select
> >
> >
> server.log_2013-04-26T05-02-22:[#|2013-04-26T04:33:39.065-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=38;_ThreadName=httpSSLWorkerTh
> > read-9001-11;|[xxxcore] webapp=/solr path=/select params={} hits=24099
> > status=0 QTime=19 |#]
> >
> > What we are seeing is a huge number of these empty queries. Once this
> > happens I have observed 2 things
> >
> > 1. even if I query from admin console, irrespective of the query, I get
> > same results as if its a cached page of *:* query. i.e. I cannot see the
> > query I entered in the server log, the query doesn't even come to the
> > server but I get same results as *:*
> >
> > 2. If I query via solrj no results are returned.
> >
> > This has been driving me nuts for almost a week. Any help is greatly
> > appreciated.
>
> Querying from the admin UI and not seeing anything in the server log
> sounds like browser caching. You can turn that off in solrconfig.xml.
>
> I could not duplicate what you're seeing with SolrJ. You didn't say
> what version of SolrJ, so I did this using 3.6.2 (same as your server
> version). I thought maybe if you had a query object that didn't have an
> actual query set, it might do what you're seeing, but that doesn't
> appear to be the case. I don't have a 3.6.2 server to test against, so
> I used my 3.5.0 and 4.2.1 servers.
>
> Test code:
> http://pastie.org/private/bnvurz1f9b9viawgqbxvmq
>
> Solr 4.2.1 log:
> INFO - 2013-04-26 14:17:24.127; org.apache.solr.core.SolrCore; [ncmain]
> webapp=/solr path=/select params={wt=xml&version=2.2} hits=0 status=0
> QTime=20
>
> 3.5.0 server log:
> ----
> Apr 26, 2013 2:20:23 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.NullPointerException
>
> Apr 26, 2013 2:20:23 PM org.apache.solr.core.SolrCore execute
> INFO: [ncmain] webapp=/solr path=/select params={wt=xml&version=2.2}
> status=500 QTime=0
> Apr 26, 2013 2:20:23 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.NullPointerException
> ----
>
>
> Same code without the setParser line:
>
> Solr 4.2.1 log:
> INFO - 2013-04-26 14:14:01.270; org.apache.solr.core.SolrCore; [ncmain]
> webapp=/solr path=/select params={wt=javabin&version=2} hits=0 status=0
> QTime=187
>
> Thanks,
> Shawn
>
>
Re: Weird query issues
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/26/2013 1:01 PM, Ravi Solr wrote:
> Hello Shawn,
> We found that it is unrelated to the group queries instead more
> related to the empty queries. Do you happen to know what could cause empty
> queries like the following from SOLRJ ? I can generate similar query via
> curl hitting the select handler like - http://server:port/solr/select
>
> server.log_2013-04-26T05-02-22:[#|2013-04-26T04:33:39.065-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=38;_ThreadName=httpSSLWorkerTh
> read-9001-11;|[xxxcore] webapp=/solr path=/select params={} hits=24099
> status=0 QTime=19 |#]
>
> What we are seeing is a huge number of these empty queries. Once this
> happens I have observed 2 things
>
> 1. even if I query from admin console, irrespective of the query, I get
> same results as if its a cached page of *:* query. i.e. I cannot see the
> query I entered in the server log, the query doesn't even come to the
> server but I get same results as *:*
>
> 2. If I query via solrj no results are returned.
>
> This has been driving me nuts for almost a week. Any help is greatly
> appreciated.
Querying from the admin UI and not seeing anything in the server log
sounds like browser caching. You can turn that off in solrconfig.xml.
I could not duplicate what you're seeing with SolrJ. You didn't say
what version of SolrJ, so I did this using 3.6.2 (same as your server
version). I thought maybe if you had a query object that didn't have an
actual query set, it might do what you're seeing, but that doesn't
appear to be the case. I don't have a 3.6.2 server to test against, so
I used my 3.5.0 and 4.2.1 servers.
Test code:
http://pastie.org/private/bnvurz1f9b9viawgqbxvmq
Solr 4.2.1 log:
INFO - 2013-04-26 14:17:24.127; org.apache.solr.core.SolrCore; [ncmain]
webapp=/solr path=/select params={wt=xml&version=2.2} hits=0 status=0
QTime=20
3.5.0 server log:
----
Apr 26, 2013 2:20:23 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
Apr 26, 2013 2:20:23 PM org.apache.solr.core.SolrCore execute
INFO: [ncmain] webapp=/solr path=/select params={wt=xml&version=2.2}
status=500 QTime=0
Apr 26, 2013 2:20:23 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
----
Same code without the setParser line:
Solr 4.2.1 log:
INFO - 2013-04-26 14:14:01.270; org.apache.solr.core.SolrCore; [ncmain]
webapp=/solr path=/select params={wt=javabin&version=2} hits=0 status=0
QTime=187
Thanks,
Shawn
Re: Weird query issues
Posted by Ravi Solr <ra...@gmail.com>.
Hello Shawn,
We found that it is unrelated to the group queries instead more
related to the empty queries. Do you happen to know what could cause empty
queries like the following from SOLRJ ? I can generate similar query via
curl hitting the select handler like - http://server:port/solr/select
server.log_2013-04-26T05-02-22:[#|2013-04-26T04:33:39.065-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=38;_ThreadName=httpSSLWorkerTh
read-9001-11;|[xxxcore] webapp=/solr path=/select params={} hits=24099
status=0 QTime=19 |#]
What we are seeing is a huge number of these empty queries. Once this
happens I have observed 2 things
1. even if I query from admin console, irrespective of the query, I get
same results as if its a cached page of *:* query. i.e. I cannot see the
query I entered in the server log, the query doesn't even come to the
server but I get same results as *:*
2. If I query via solrj no results are returned.
This has been driving me nuts for almost a week. Any help is greatly
appreciated.
Thanks
Ravi Kiran Bhaskar
On Sat, Apr 20, 2013 at 10:33 PM, Ravi Solr <ra...@gmail.com> wrote:
> Thanks for your advise Shawn. I have created a JIRA issue SOLR-4743.
>
>
> On Sat, Apr 20, 2013 at 4:32 PM, Shawn Heisey <so...@elyograg.org> wrote:
>
>> On 4/20/2013 9:08 AM, Ravi Solr wrote:
>> > Thanks you very much for responding Shawn. I never use IE, I use
>> firefox.
>> > These are brand new servers and I don't think I am mixing versions. What
>> > made you think I was using the 1.4.1 ?? You are correct in saying that
>> the
>> > server is throwing HTML response since a group query has been failing
>> with
>> > SEVERE error following which the entire instance behaves weirdly until
>> we
>> > restart.
>> >
>> > Its surprising that group query error handling has such glaring issue.
>> If
>> > you specify group=true but don't specify group.query or group.field SOLR
>> > throws a SEVERE exception following which we see the empty queries and
>> > finally no responses via solrj and admin console gives numFound always
>> > equal to total number of docs in index . Looks like the searcher goes
>> for a
>> > spin once it encounters the exception. Such situation should have been
>> > gracefully handled
>>
>> Ah, so what's happening is that after an invalid grouping query, Solr is
>> unstable and stops working right. You should file an issue in Jira,
>> giving as much detail as you can. My last message was almost completely
>> wrong.
>>
>> You are right that it should be gracefully handled, and obviously it is
>> not. For the 3.x Solr versions, grouping did not exist before 3.6. It
>> is a major 4.x feature that was backported. Sometimes such major
>> features depend on significant changes that have not happened on older
>> versions, leading to problems like this. Unfortunately, you could wait
>> quite a while for a fix on 3.6, where active development has stopped.
>>
>> I have no personal experience with grouping, but I just tried the
>> problematic query (adding "&group=true" to one that works) on 4.2.1. It
>> doesn't throw an error, I just get no results. When I follow it with a
>> regular query, everything works perfectly. Would you be able to upgrade
>> to 4.2.1? That's not a trivial thing to do, so hopefully you are
>> already working on upgrading.
>>
>> Thanks,
>> Shawn
>>
>>
>
Re: Weird query issues
Posted by Ravi Solr <ra...@gmail.com>.
Thanks for your advise Shawn. I have created a JIRA issue SOLR-4743.
On Sat, Apr 20, 2013 at 4:32 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/20/2013 9:08 AM, Ravi Solr wrote:
> > Thanks you very much for responding Shawn. I never use IE, I use firefox.
> > These are brand new servers and I don't think I am mixing versions. What
> > made you think I was using the 1.4.1 ?? You are correct in saying that
> the
> > server is throwing HTML response since a group query has been failing
> with
> > SEVERE error following which the entire instance behaves weirdly until we
> > restart.
> >
> > Its surprising that group query error handling has such glaring issue. If
> > you specify group=true but don't specify group.query or group.field SOLR
> > throws a SEVERE exception following which we see the empty queries and
> > finally no responses via solrj and admin console gives numFound always
> > equal to total number of docs in index . Looks like the searcher goes
> for a
> > spin once it encounters the exception. Such situation should have been
> > gracefully handled
>
> Ah, so what's happening is that after an invalid grouping query, Solr is
> unstable and stops working right. You should file an issue in Jira,
> giving as much detail as you can. My last message was almost completely
> wrong.
>
> You are right that it should be gracefully handled, and obviously it is
> not. For the 3.x Solr versions, grouping did not exist before 3.6. It
> is a major 4.x feature that was backported. Sometimes such major
> features depend on significant changes that have not happened on older
> versions, leading to problems like this. Unfortunately, you could wait
> quite a while for a fix on 3.6, where active development has stopped.
>
> I have no personal experience with grouping, but I just tried the
> problematic query (adding "&group=true" to one that works) on 4.2.1. It
> doesn't throw an error, I just get no results. When I follow it with a
> regular query, everything works perfectly. Would you be able to upgrade
> to 4.2.1? That's not a trivial thing to do, so hopefully you are
> already working on upgrading.
>
> Thanks,
> Shawn
>
>
Re: Weird query issues
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/20/2013 9:08 AM, Ravi Solr wrote:
> Thanks you very much for responding Shawn. I never use IE, I use firefox.
> These are brand new servers and I don't think I am mixing versions. What
> made you think I was using the 1.4.1 ?? You are correct in saying that the
> server is throwing HTML response since a group query has been failing with
> SEVERE error following which the entire instance behaves weirdly until we
> restart.
>
> Its surprising that group query error handling has such glaring issue. If
> you specify group=true but don't specify group.query or group.field SOLR
> throws a SEVERE exception following which we see the empty queries and
> finally no responses via solrj and admin console gives numFound always
> equal to total number of docs in index . Looks like the searcher goes for a
> spin once it encounters the exception. Such situation should have been
> gracefully handled
Ah, so what's happening is that after an invalid grouping query, Solr is
unstable and stops working right. You should file an issue in Jira,
giving as much detail as you can. My last message was almost completely
wrong.
You are right that it should be gracefully handled, and obviously it is
not. For the 3.x Solr versions, grouping did not exist before 3.6. It
is a major 4.x feature that was backported. Sometimes such major
features depend on significant changes that have not happened on older
versions, leading to problems like this. Unfortunately, you could wait
quite a while for a fix on 3.6, where active development has stopped.
I have no personal experience with grouping, but I just tried the
problematic query (adding "&group=true" to one that works) on 4.2.1. It
doesn't throw an error, I just get no results. When I follow it with a
regular query, everything works perfectly. Would you be able to upgrade
to 4.2.1? That's not a trivial thing to do, so hopefully you are
already working on upgrading.
Thanks,
Shawn
Re: Weird query issues
Posted by Ravi Solr <ra...@gmail.com>.
Thanks you very much for responding Shawn. I never use IE, I use firefox.
These are brand new servers and I don't think I am mixing versions. What
made you think I was using the 1.4.1 ?? You are correct in saying that the
server is throwing HTML response since a group query has been failing with
SEVERE error following which the entire instance behaves weirdly until we
restart.
Its surprising that group query error handling has such glaring issue. If
you specify group=true but don't specify group.query or group.field SOLR
throws a SEVERE exception following which we see the empty queries and
finally no responses via solrj and admin console gives numFound always
equal to total number of docs in index . Looks like the searcher goes for a
spin once it encounters the exception. Such situation should have been
gracefully handled
[#|2013-04-19T23:47:53.363-0400|SEVERE|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=26;_ThreadName=httpSSLWorkerThread-9001-17;_RequestID=2f
933642-cad0-40e5-86c6-65b00be9bb97;|org.apache.solr.common.SolrException:
Specify at least one field, function or query to group by.
at org.apache.solr.search.Grouping.execute(Grouping.java:228)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:372)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:186)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
at
com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
|#]
[#|2013-04-19T23:47:53.365-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=26;_ThreadName=httpSSLWorkerThread-9001-17;|[core1]
webapp=/solr path=/select
params={q=astronomy\+&rows=10&start=0&facet=true&fq=source:"xxx.com"&fq=locations:("Maryland")&sort=score+desc&group=true}
status=400 QTime=9 |#]
Ravi Kiran Bhaskar
On Fri, Apr 19, 2013 at 5:40 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/19/2013 12:55 PM, Ravi Solr wrote:
>
>> We are using Solr 3.6.2 single core ( both index and query on same
>> machine)
>> and randomly the server fails to query correctly. If we query from the
>> admin console the query is not even applied and it returns numFound count
>> equal to total docs in the index as if no query is made, and if use SOLRJ
>> to query it throws javabin error
>>
>> Invalid version (expected 2, but 60) or the data in not in 'javabin'
>> format
>>
>
> The UI problem is likely a browser issue, but I could be wrong. Some
> browsers, IE in particular, but not limited to that one, have problems with
> the admin UI. Using a different browser or clearing the browser cache can
> sometimes fix those problems.
>
> As for SolrJ, are you using a really old (1.x) SolrJ with Solr 3.6.2? Have
> you ever had Solr 1.x running on the same machine that's now running 3.6.2?
>
> Because the javabin version changed between 1.4.1 and 3.1.0, SolrJ 1.x is
> not compatible with Solr 3.1 and later unless you set the response parser
> on the server object to XML before you try to use it. If you have upgraded
> Solr from an old version, your servlet container (sun-appserver) may have
> some of the old jars remaining from the 1.x install. They must be removed.
>
> To change your SolrJ to use the XML response parser, use code like the
> following:
>
> server.setParser(new XMLResponseParser());
>
> When SolrJ and Solr are both version 3.x or 4.x, you can remove this line.
>
> Another way that you can get the javabin error is when Solr is returning
> an error response, or returning a response that is not an error but is an
> HTML response reporting an unusual circumstance rather than the usual
> javabin. These HTML responses should no longer exist in the newest
> versions of Solr. Do you see any errors or warnings in your server log?
> The server log line you included in your email is not an error.
>
> Thanks,
> Shawn
>
>
Re: Weird query issues
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/19/2013 12:55 PM, Ravi Solr wrote:
> We are using Solr 3.6.2 single core ( both index and query on same machine)
> and randomly the server fails to query correctly. If we query from the
> admin console the query is not even applied and it returns numFound count
> equal to total docs in the index as if no query is made, and if use SOLRJ
> to query it throws javabin error
>
> Invalid version (expected 2, but 60) or the data in not in 'javabin' format
The UI problem is likely a browser issue, but I could be wrong. Some
browsers, IE in particular, but not limited to that one, have problems
with the admin UI. Using a different browser or clearing the browser
cache can sometimes fix those problems.
As for SolrJ, are you using a really old (1.x) SolrJ with Solr 3.6.2?
Have you ever had Solr 1.x running on the same machine that's now
running 3.6.2?
Because the javabin version changed between 1.4.1 and 3.1.0, SolrJ 1.x
is not compatible with Solr 3.1 and later unless you set the response
parser on the server object to XML before you try to use it. If you
have upgraded Solr from an old version, your servlet container
(sun-appserver) may have some of the old jars remaining from the 1.x
install. They must be removed.
To change your SolrJ to use the XML response parser, use code like the
following:
server.setParser(new XMLResponseParser());
When SolrJ and Solr are both version 3.x or 4.x, you can remove this line.
Another way that you can get the javabin error is when Solr is returning
an error response, or returning a response that is not an error but is
an HTML response reporting an unusual circumstance rather than the usual
javabin. These HTML responses should no longer exist in the newest
versions of Solr. Do you see any errors or warnings in your server log?
The server log line you included in your email is not an error.
Thanks,
Shawn