You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Ravi Solr <ra...@gmail.com> on 2012/05/04 19:50:56 UTC

Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Hello,
         We Recently we migrated our SOLR 3.6 server OS from Solaris
to CentOS and from then on we started seeing "Invalid version
(expected 2, but 60)" errors on one of the query servers (oddly one
other query server seems fine). If we restart the server having issue
everything will be alright, but the next day in the morning again we
get the same exception. I made sure that all the client applications
are using SOLR 3.6 version.

The Glassfish on which all the applications  and SOLR are deployed use
Java  1.6.0_29. The only difference I could see

1. The process indexing to the server having issues is using java1.6.0_31
2. The process indexing to the server that DOES NOT have issues is
using java1.6.0_29

Could the Java minor version being greater than the SOLR instance be
the cause of this issue  ???

Can anybody please help me debug this a bit more ? what else can I
look at to understand the underlying problem. The stack trace is given
below


[#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx.xxxx.xxxx.xxxx|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5;
org.apache.solr.client.solrj.SolrServerException: Error executing query
        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
        at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743)
        at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347)
        at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
Caused by: java.lang.RuntimeException: Invalid version (expected 2,
but 60) or the data in not in 'javabin' format
        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
        ... 43 more


Thanking you in anticipation,

Ravi Kiran Bhaskar
Principal Software Engineer
Washington Post Digital
1150 15th Street NW, Washington, DC 20071

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Ravi Solr <ra...@gmail.com>.

Just to give folks an update, we trashed the server having issues and
cloned/rebuild a VM from a sane server and it seems to be running good
for the past 3 days without any issues. We intend to monitor it over
the weekend. If its still stable on Monday, I would blame the issues
it on the server configuration. :-)

Thanks
Ravi Kiran Bhaskar

On Tue, May 15, 2012 at 2:57 PM, Ravi Solr <ra...@gmail.com> wrote:
> I have already triple cross-checked  that all my clients are using
> same version as the server which is 3.6
>
> Thanks
>
> Ravi Kiran
>
> On Tue, May 15, 2012 at 2:09 PM, Ramesh K Balasubramanian
> <be...@yahoo.com> wrote:
>> I have seen similar errors before when the solr version and solrj version in the client don't match.
>>
>> Best Regards,
>> Ramesh

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Ravi Solr <ra...@gmail.com>.

I have already triple cross-checked  that all my clients are using
same version as the server which is 3.6

Thanks

Ravi Kiran

On Tue, May 15, 2012 at 2:09 PM, Ramesh K Balasubramanian
<be...@yahoo.com> wrote:
> I have seen similar errors before when the solr version and solrj version in the client don't match.
>
> Best Regards,
> Ramesh

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Ramesh K Balasubramanian <be...@yahoo.com>.

I have seen similar errors before when the solr version and solrj version in the client don't match.
 
Best Regards,
Ramesh

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Ravi Solr <ra...@gmail.com>.

Hello,
       Unfortunately it seems like I spoke too early. Today morning I
received the same error again even after disabling the iptables. The
weird thing is only one out of 6 or 7 queries fails as evidenced in
the stack traces below. The query below the stack trace gave a
'status=500' subsequent queries look fine


[#|2012-05-15T08:12:38.703-0400|SEVERE|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-8;_RequestID=9f54ea89-357a-4c1b-87a1-fbaacc9fd0ee;|org.apache.solr.common.SolrException
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:275)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
Caused by: java.lang.RuntimeException: Invalid version (expected 2,
but 60) or the data in not in 'javabin' format
        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
        at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:469)
        at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:249)
        at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:129)
        at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:103)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

[#|2012-05-15T08:12:38.704-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-8;|[sitesearchcore]
webapp=/solr-admin path=/select
params={q=dwts&start=0&rows=10&shards=localhost:9001/solr-admin/sitesearchcore,localhost:9001/solr-admin/deathnoticescore&tracking=sitesearch&fl=headline,contenturl,blurb,sourcenav,contenttype,byline,primarysection,smallthumburl,displaydatetime,pubdatetime&facet=true&facet.field=contenttype&facet.mincount=1&facet.method=enum&fq=displaydatetime:[2005-01-01T00:00:00Z+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+24+Hours"}displaydatetime:[NOW/DAY-1DAY+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+7+Days"}displaydatetime:[NOW/DAY-7DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+60+Days"}displaydatetime:[NOW/DAY-60DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+12+Months"}displaydatetime:[NOW/DAY-1YEAR+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"All+Since+2005"}displaydatetime:[*+TO+NOW/DAY%2B1DAY]&wt=javabin&version=2}
status=500 QTime=25 |#]

[#|2012-05-15T08:12:39.905-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=27;_ThreadName=httpSSLWorkerThread-9001-6;|[sitesearchcore]
webapp=/solr-admin path=/admin/ping params={} hits=35257 status=0
QTime=5 |#]

[#|2012-05-15T08:12:39.906-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=27;_ThreadName=httpSSLWorkerThread-9001-6;|[sitesearchcore]
webapp=/solr-admin path=/admin/ping params={} status=0 QTime=6 |#]

[#|2012-05-15T08:12:41.109-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-2;|[spellsuggestcore]
webapp=/solr-admin path=/select
params={q=north+korea&spellcheck=true&spellcheck.count=5&spellcheck.collate=true&spellcheck.maxCollations=5&spellcheck.collateExtendedResults=true&tracking=sitesearch&wt=javabin&version=2}
hits=248 status=0 QTime=2 |#]

[#|2012-05-15T08:12:43.349-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=33;_ThreadName=httpSSLWorkerThread-9001-9;|[deathnoticescore]
webapp=/solr-admin path=/select
params={mm=2<-1+5<-3+6<60%25&tie=0.01&qf=headline^50.0+webheadline^50.0+mobileheadline^50.0+subheadline^50.0+blurb^40.0+bylinetext^30.0+keywordLower^10.0+personLower^10.0+organizationLower^10.0+conceptLower^10.0+subjectLower^10.0+neighborhoodLower^10.0+addressLower^10.0+cityLower^10.0+zipcodeLower^10.0+countyLower^10.0+countryLower^10.0+countrycodeLower^10.0+body^5.0&q.alt=*:*&hl.fl=keyword&defType=edismax&f.body.hl.fragsize=0&pf=headline^55.0+webheadline^55.0+mobileheadline^55.0+subheadline^55.0+blurb^45.0+bylinetext^35.0+keywordLower^15.0+personLower^15.0+organizationLower^15.0+conceptLower^15.0+subjectLower^15.0+neighborhoodLower^15.0+addressLower^15.0+cityLower^15.0+zipcodeLower^15.0+countyLower^15.0+countryLower^15.0+countrycodeLower^15.0+body^10.0&fl=systemid,score&bf=ord(displaydatetime)^1.0&f.text.hl.fragmenter=regex&f.name.hl.alternateField=keyword&ps=5&q=annalynne&start=0&rows=10&tracking=sitesearch&facet=true&facet.field=contenttype&facet.method=enum&fq=displaydatetime:[2005-01-01T00:00:00Z+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+24+Hours"}displaydatetime:[NOW/DAY-1DAY+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+7+Days"}displaydatetime:[NOW/DAY-7DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+60+Days"}displaydatetime:[NOW/DAY-60DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+12+Months"}displaydatetime:[NOW/DAY-1YEAR+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"All+Since+2005"}displaydatetime:[*+TO+NOW/DAY%2B1DAY]&fsv=true&f.contenttype.facet.limit=160&isShard=true&NOW=1337083963338&wt=javabin&version=2}
hits=0 status=0 QTime=4 |#]

[#|2012-05-15T08:12:43.356-0400|INFO|sun-appserver2.1.1|org.apache.solr.core.SolrCore|_ThreadID=28;_ThreadName=httpSSLWorkerThread-9001-7;|[sitesearchcore]
webapp=/solr-admin path=/select
params={mm=2<-1+5<-3+6<60%25&tie=0.01&qf=headline^50.0+webheadline^50.0+mobileheadline^50.0+subheadline^50.0+blurb^40.0+bylinetext^30.0+keywordLower^10.0+personLower^10.0+organizationLower^10.0+conceptLower^10.0+subjectLower^10.0+neighborhoodLower^10.0+addressLower^10.0+cityLower^10.0+zipcodeLower^10.0+countyLower^10.0+countryLower^10.0+countrycodeLower^10.0+body^5.0&q.alt=*:*&hl.fl=keyword&defType=edismax&f.body.hl.fragsize=0&pf=headline^55.0+webheadline^55.0+mobileheadline^55.0+subheadline^55.0+blurb^45.0+bylinetext^35.0+keywordLower^15.0+personLower^15.0+organizationLower^15.0+conceptLower^15.0+subjectLower^15.0+neighborhoodLower^15.0+addressLower^15.0+cityLower^15.0+zipcodeLower^15.0+countyLower^15.0+countryLower^15.0+countrycodeLower^15.0+body^10.0&fl=systemid,score&bf=ord(displaydatetime)^1.0&f.text.hl.fragmenter=regex&f.name.hl.alternateField=keyword&ps=5&q=annalynne&start=0&rows=10&tracking=sitesearch&facet=true&facet.field=contenttype&facet.method=enum&fq=displaydatetime:[2005-01-01T00:00:00Z+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+24+Hours"}displaydatetime:[NOW/DAY-1DAY+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+7+Days"}displaydatetime:[NOW/DAY-7DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+60+Days"}displaydatetime:[NOW/DAY-60DAYS+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"Past+12+Months"}displaydatetime:[NOW/DAY-1YEAR+TO+NOW/DAY%2B1DAY]&facet.query={!ex%3Ddt+key%3D"All+Since+2005"}displaydatetime:[*+TO+NOW/DAY%2B1DAY]&fsv=true&f.contenttype.facet.limit=160&isShard=true&NOW=1337083963338&wt=javabin&version=2}
hits=5 status=0 QTime=11 |#]

Thanks,

Ravi Kiran Bhaskar

On Fri, May 11, 2012 at 11:04 PM, Mark Miller <ma...@gmail.com> wrote:
> Yeah, 9 times out of 10, this error is a 404 - which wouldn't be logged anywhere.
>
> On May 11, 2012, at 6:12 PM, Ravi Solr wrote:
>
>> Guys, just to give you an update, we think we "might" have found the
>> issue. iptables was enabled on one query server and disabled on the
>> other. The server where iptables is enabled is the one having issues,
>> we disabled the iptables today to test out the theory that the
>> iptables might be causing this issue of null/empty response. If the
>> server holds up during the weekend then we have the culprit :-)
>>
>> Thanks to all of you who helped me out. Stay tuned.
>>
>> Ravi Kiran
>>
>> On Fri, May 11, 2012 at 1:23 AM, Shawn Heisey <so...@elyograg.org> wrote:
>>> On 5/10/2012 4:17 PM, Ravi Solr wrote:
>>>>
>>>> Thanks for responding Mr. Heisey... I don't see any parsing errors in
>>>> my log but I see lot of exceptions like the one listed below....once
>>>> an exception like this happens weirdness ensues. For example - To
>>>> check sanity I queried for uniquekey:"111" from the solr admin GUI it
>>>> gave back numFound equal to all docs in that index i.e. its not
>>>> searching for that uniquekey at all, it blindly matched all docs.
>>>> However, once you restart the server the same index without any change
>>>> works perfectly returning only one doc in numFound when you search for
>>>> uniquekey:"111"...I tried everything from reindexing, copying index
>>>> from another sane server, delete entire index and reindex from scratch
>>>> etc but in vain, it works for roughly 24 hours and then starts
>>>> throwing the same error no matter what the query is.
>>>>
>>>>
>>>>
>>>> [#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error
>>>> searching data for section Local
>>>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>>>        at
>>>> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>>>        at
>>>> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>>>        at xxx.xxx.xxx.xxx(FeedController.java:621)
>>>>        at xxx.xxx.xxx.xxx(FeedController.java:402)
>>>
>>>
>>> This is still saying solrj.  Unless I am completely misunderstanding the way
>>> things work, which I will freely admit is possible, this is the client code.
>>>  Do you have anything in the log files from Solr (the server)?  I don't have
>>> a lot of experience with Tomcat, because I run my Solr under jetty as
>>> included in the example.  It looks like the client is running under Tomcat,
>>> though I suppose you might be running Solr under a different container.
>>>
>>> Thanks,
>>> Shawn
>>>
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Mark Miller <ma...@gmail.com>.

Yeah, 9 times out of 10, this error is a 404 - which wouldn't be logged anywhere.

On May 11, 2012, at 6:12 PM, Ravi Solr wrote:

> Guys, just to give you an update, we think we "might" have found the
> issue. iptables was enabled on one query server and disabled on the
> other. The server where iptables is enabled is the one having issues,
> we disabled the iptables today to test out the theory that the
> iptables might be causing this issue of null/empty response. If the
> server holds up during the weekend then we have the culprit :-)
> 
> Thanks to all of you who helped me out. Stay tuned.
> 
> Ravi Kiran
> 
> On Fri, May 11, 2012 at 1:23 AM, Shawn Heisey <so...@elyograg.org> wrote:
>> On 5/10/2012 4:17 PM, Ravi Solr wrote:
>>> 
>>> Thanks for responding Mr. Heisey... I don't see any parsing errors in
>>> my log but I see lot of exceptions like the one listed below....once
>>> an exception like this happens weirdness ensues. For example - To
>>> check sanity I queried for uniquekey:"111" from the solr admin GUI it
>>> gave back numFound equal to all docs in that index i.e. its not
>>> searching for that uniquekey at all, it blindly matched all docs.
>>> However, once you restart the server the same index without any change
>>> works perfectly returning only one doc in numFound when you search for
>>> uniquekey:"111"...I tried everything from reindexing, copying index
>>> from another sane server, delete entire index and reindex from scratch
>>> etc but in vain, it works for roughly 24 hours and then starts
>>> throwing the same error no matter what the query is.
>>> 
>>> 
>>> 
>>> [#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error
>>> searching data for section Local
>>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>>        at
>>> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>>        at
>>> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>>        at xxx.xxx.xxx.xxx(FeedController.java:621)
>>>        at xxx.xxx.xxx.xxx(FeedController.java:402)
>> 
>> 
>> This is still saying solrj.  Unless I am completely misunderstanding the way
>> things work, which I will freely admit is possible, this is the client code.
>>  Do you have anything in the log files from Solr (the server)?  I don't have
>> a lot of experience with Tomcat, because I run my Solr under jetty as
>> included in the example.  It looks like the client is running under Tomcat,
>> though I suppose you might be running Solr under a different container.
>> 
>> Thanks,
>> Shawn
>> 

- Mark Miller
lucidimagination.com

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Ravi Solr <ra...@gmail.com>.

Guys, just to give you an update, we think we "might" have found the
issue. iptables was enabled on one query server and disabled on the
other. The server where iptables is enabled is the one having issues,
we disabled the iptables today to test out the theory that the
iptables might be causing this issue of null/empty response. If the
server holds up during the weekend then we have the culprit :-)

Thanks to all of you who helped me out. Stay tuned.

Ravi Kiran

On Fri, May 11, 2012 at 1:23 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 5/10/2012 4:17 PM, Ravi Solr wrote:
>>
>> Thanks for responding Mr. Heisey... I don't see any parsing errors in
>> my log but I see lot of exceptions like the one listed below....once
>> an exception like this happens weirdness ensues. For example - To
>> check sanity I queried for uniquekey:"111" from the solr admin GUI it
>> gave back numFound equal to all docs in that index i.e. its not
>> searching for that uniquekey at all, it blindly matched all docs.
>> However, once you restart the server the same index without any change
>> works perfectly returning only one doc in numFound when you search for
>> uniquekey:"111"...I tried everything from reindexing, copying index
>> from another sane server, delete entire index and reindex from scratch
>> etc but in vain, it works for roughly 24 hours and then starts
>> throwing the same error no matter what the query is.
>>
>>
>>
>> [#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error
>> searching data for section Local
>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>        at
>> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>        at
>> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>        at xxx.xxx.xxx.xxx(FeedController.java:621)
>>        at xxx.xxx.xxx.xxx(FeedController.java:402)
>
>
> This is still saying solrj.  Unless I am completely misunderstanding the way
> things work, which I will freely admit is possible, this is the client code.
>  Do you have anything in the log files from Solr (the server)?  I don't have
> a lot of experience with Tomcat, because I run my Solr under jetty as
> included in the example.  It looks like the client is running under Tomcat,
> though I suppose you might be running Solr under a different container.
>
> Thanks,
> Shawn
>

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Shawn Heisey <so...@elyograg.org>.

On 5/10/2012 4:17 PM, Ravi Solr wrote:
> Thanks for responding Mr. Heisey... I don't see any parsing errors in
> my log but I see lot of exceptions like the one listed below....once
> an exception like this happens weirdness ensues. For example - To
> check sanity I queried for uniquekey:"111" from the solr admin GUI it
> gave back numFound equal to all docs in that index i.e. its not
> searching for that uniquekey at all, it blindly matched all docs.
> However, once you restart the server the same index without any change
> works perfectly returning only one doc in numFound when you search for
> uniquekey:"111"...I tried everything from reindexing, copying index
> from another sane server, delete entire index and reindex from scratch
> etc but in vain, it works for roughly 24 hours and then starts
> throwing the same error no matter what the query is.
>
>
> [#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error
> searching data for section Local
> org.apache.solr.client.solrj.SolrServerException: Error executing query
> 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
> 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
> 	at xxx.xxx.xxx.xxx(FeedController.java:621)
> 	at xxx.xxx.xxx.xxx(FeedController.java:402)

This is still saying solrj.  Unless I am completely misunderstanding the 
way things work, which I will freely admit is possible, this is the 
client code.  Do you have anything in the log files from Solr (the 
server)?  I don't have a lot of experience with Tomcat, because I run my 
Solr under jetty as included in the example.  It looks like the client 
is running under Tomcat, though I suppose you might be running Solr 
under a different container.

Thanks,
Shawn

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Ravi Solr <ra...@gmail.com>.

Thanks for responding Mr. Heisey... I don't see any parsing errors in
my log but I see lot of exceptions like the one listed below....once
an exception like this happens weirdness ensues. For example - To
check sanity I queried for uniquekey:"111" from the solr admin GUI it
gave back numFound equal to all docs in that index i.e. its not
searching for that uniquekey at all, it blindly matched all docs.
However, once you restart the server the same index without any change
works perfectly returning only one doc in numFound when you search for
uniquekey:"111"...I tried everything from reindexing, copying index
from another sane server, delete entire index and reindex from scratch
etc but in vain, it works for roughly 24 hours and then starts
throwing the same error no matter what the query is.


[#|2012-05-10T13:27:14.071-0400|SEVERE|sun-appserver2.1.1|xxx.xxx.xxx.xxx|_ThreadID=21;_ThreadName=httpSSLWorkerThread-9001-6;_RequestID=d44462e7-576b-4391-a499-c65da33e3293;|Error
searching data for section Local
org.apache.solr.client.solrj.SolrServerException: Error executing query
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
	at xxx.xxx.xxx.xxx(FeedController.java:621)
	at xxx.xxx.xxx.xxx(FeedController.java:402)
	at sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
	at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
	at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
	at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
	at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
	at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
	at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
	at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
	at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
	at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
	at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
	at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
	at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
	at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
	at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
	at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
	at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393)
	at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
	at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
Caused by: java.lang.RuntimeException: Invalid version (expected 2,
but 60) or the data in not in 'javabin' format
	at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
	at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
	... 43 more
|#]


Ravi Kiran Bhaskar

On Thu, May 10, 2012 at 4:33 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 5/10/2012 12:27 PM, Ravi Solr wrote:
>>
>> I clean the entire index and re-indexed it with SOLRJ 3.6. Still I get
>> the same error every single day. How can I see if the container
>> returned partial/nonconforming response since it may be hidden by
>> solrj ?
>
>
> If the server is sending a non-javabin error response that SolrJ doesn't
> parse, the logs from the container that runs Solr will normally give you
> useful information.  Here's the first part of something logged on mine for
> an invalid query - the query sent in only had one double quote.  The log
> actually contains the full Java stacktrace, I just included the first little
> bit:
>
> May 10, 2012 2:13:17 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException:
> org.apache.lucene.queryParser.ParseException: Cannot parse '(  ("trader
> joe's))': Lexical error at line 1, column 20.  Encountered: <EOF> after :
> "\"trader joe\'s))"
>
> Thanks,
> Shawn
>

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Shawn Heisey <so...@elyograg.org>.

On 5/10/2012 12:27 PM, Ravi Solr wrote:
> I clean the entire index and re-indexed it with SOLRJ 3.6. Still I get
> the same error every single day. How can I see if the container
> returned partial/nonconforming response since it may be hidden by
> solrj ?

If the server is sending a non-javabin error response that SolrJ doesn't 
parse, the logs from the container that runs Solr will normally give you 
useful information.  Here's the first part of something logged on mine 
for an invalid query - the query sent in only had one double quote.  The 
log actually contains the full Java stacktrace, I just included the 
first little bit:

May 10, 2012 2:13:17 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: 
org.apache.lucene.queryParser.ParseException: Cannot parse '(  ("trader 
joe's))': Lexical error at line 1, column 20.  Encountered: <EOF> after 
: "\"trader joe\'s))"

Thanks,
Shawn

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Ravi Solr <ra...@gmail.com>.

I clean the entire index and re-indexed it with SOLRJ 3.6. Still I get
the same error every single day. How can I see if the container
returned partial/nonconforming response since it may be hidden by
solrj ?

Thanks

Ravi Kiran Bhaskar

On Mon, May 7, 2012 at 2:16 PM, Ravi Solr <ra...@gmail.com> wrote:
> Hello Mr. Miller and Mr. Erickson,
>              Found yet another inconsistency on the query server that
> might be causing this issue. Today morning also I got a similar error
> as shown in stacktrace below. So I tried querying for that
> "d101dd3a-979a-11e1-927c-291130c98dff" which is our unique key in the
> schema.
>
> On the server having issue it returned more than 10 docs with
> numFound="1051273" and on all other sane servers it returned only 1
> doc with numFound="1". This is really weird, as, we copied the entire
> index from a sane server onto the server having issues now just 2 days
> ago. Do you have any idea why this would happen ?
>
> [#|2012-05-07T12:58:54.055-0400|SEVERE|sun-appserver2.1.1|com.wpost.ipad.feeds.FeedController|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-3;_RequestID=4203e3e5-c39d-4df7-a32a-600d0169c81f;|Error
> searching for thumbnails for d101dd3a-979a-11e1-927c-291130c98dff
> org.apache.solr.client.solrj.SolrServerException: Error executing query
>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>        at xxx.xxx.xxx.xxx.populateThumbnails(FeedController.java:1184)
>        at xxx.xxx.xxx.xxx..findNewsBySection(FeedController.java:509)
>        at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
> ..
> ...
> ..
> Caused by: java.lang.RuntimeException: Invalid version (expected 2,
> but 60) or the data in not in 'javabin' format
>        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
>        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
>        ... 43 more
> |#]
>
> Ravi Kiran Bhaskar
> Principal Software Engineer
> Washington Post Digital
> 1150 15th Street NW, Washington, DC 20071
>
> On Mon, May 7, 2012 at 9:36 AM, Mark Miller <ma...@gmail.com> wrote:
>> Normally this specific error is caused by a non success http error page and response is returned. The response parser tries to parse HTML as javabin.
>>
>> Sent from my iPhone
>>
>> On May 7, 2012, at 7:37 AM, Erick Erickson <er...@gmail.com> wrote:
>>
>>> Well, I'm guessing that the version of Solr (and perhaps there are
>>> classpath issues in here?) are different, somehow, on the machine
>>> slave that is showing the error.
>>>
>>> It's also possible that your config files have a different  LUCENE_VERSION
>>> in them, although I don't think this should really create the errors you're
>>> reporting.
>>>
>>> The thing that leads me in this direction is your statement that things
>>> are fine for a while and then go bad later.  If replication happens just
>>> before you get the index version error, that would point a finger at
>>> something like different Solr versions.
>>>
>>> If there is no replication before this error, then this probably isn't
>>> the problem
>>> and we'll have to look elsewhere...
>>>
>>> But this is all guesswork, just like every bug... things are only obvious after
>>> you find the problem!
>>>
>>> Best
>>> Erick
>>>
>>>
>>> On Sun, May 6, 2012 at 11:08 AM, Ravi Solr <ra...@gmail.com> wrote:
>>>> Thank you very much for responding Mr.Erickson. You may be right on
>>>> old version index, I will reindex. However we have a 2
>>>> separate/disjoint master-slave setup...only one query node/slave has
>>>> this issue. if it was really incompatible indexes why isnt the other
>>>> query server also throwing errors? that's what is throwing my
>>>> debugging thought process off.
>>>>
>>>> Thanks
>>>>
>>>> Ravi Kiran Bhaskar
>>>> Principal Software Engineer
>>>> Washington Post Digital
>>>> 1150 15th Street NW, Washington, DC 20071
>>>>
>>>> On Sat, May 5, 2012 at 12:53 PM, Erick Erickson <er...@gmail.com> wrote:
>>>>> The first thing I'd check is if, in the log, there is a replication happening
>>>>> immediately prior to the error. I confess I'm not entirely up on the
>>>>> version thing, but is it possible you're replicating an index that
>>>>> is built with some other version of Solr?
>>>>>
>>>>> That would at least explain your statement that it runs OK, but then
>>>>> fails sometime later.
>>>>>
>>>>> Best
>>>>> Erick
>>>>>
>>>>> On Fri, May 4, 2012 at 1:50 PM, Ravi Solr <ra...@gmail.com> wrote:
>>>>>> Hello,
>>>>>>         We Recently we migrated our SOLR 3.6 server OS from Solaris
>>>>>> to CentOS and from then on we started seeing "Invalid version
>>>>>> (expected 2, but 60)" errors on one of the query servers (oddly one
>>>>>> other query server seems fine). If we restart the server having issue
>>>>>> everything will be alright, but the next day in the morning again we
>>>>>> get the same exception. I made sure that all the client applications
>>>>>> are using SOLR 3.6 version.
>>>>>>
>>>>>> The Glassfish on which all the applications  and SOLR are deployed use
>>>>>> Java  1.6.0_29. The only difference I could see
>>>>>>
>>>>>> 1. The process indexing to the server having issues is using java1.6.0_31
>>>>>> 2. The process indexing to the server that DOES NOT have issues is
>>>>>> using java1.6.0_29
>>>>>>
>>>>>> Could the Java minor version being greater than the SOLR instance be
>>>>>> the cause of this issue  ???
>>>>>>
>>>>>> Can anybody please help me debug this a bit more ? what else can I
>>>>>> look at to understand the underlying problem. The stack trace is given
>>>>>> below
>>>>>>
>>>>>>
>>>>>> [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx.xxxx.xxxx.xxxx|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5;
>>>>>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>>>>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>>>>>        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>>>>>        at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743)
>>>>>>        at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347)
>>>>>>        at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
>>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
>>>>>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
>>>>>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
>>>>>>        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
>>>>>>        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
>>>>>>        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
>>>>>>        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
>>>>>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
>>>>>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
>>>>>>        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
>>>>>>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
>>>>>>        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
>>>>>>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
>>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>>>        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
>>>>>>        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
>>>>>>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
>>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>>>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>>>>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
>>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>>>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>>>>        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
>>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
>>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
>>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
>>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
>>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
>>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
>>>>>>        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393)
>>>>>>        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
>>>>>>        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
>>>>>> Caused by: java.lang.RuntimeException: Invalid version (expected 2,
>>>>>> but 60) or the data in not in 'javabin' format
>>>>>>        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
>>>>>>        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
>>>>>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
>>>>>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>>>>>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
>>>>>>        ... 43 more
>>>>>>
>>>>>>
>>>>>> Thanking you in anticipation,
>>>>>>
>>>>>> Ravi Kiran Bhaskar
>>>>>> Principal Software Engineer
>>>>>> Washington Post Digital
>>>>>> 1150 15th Street NW, Washington, DC 20071

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Ravi Solr <ra...@gmail.com>.

Hello Mr. Miller and Mr. Erickson,
              Found yet another inconsistency on the query server that
might be causing this issue. Today morning also I got a similar error
as shown in stacktrace below. So I tried querying for that
"d101dd3a-979a-11e1-927c-291130c98dff" which is our unique key in the
schema.

On the server having issue it returned more than 10 docs with
numFound="1051273" and on all other sane servers it returned only 1
doc with numFound="1". This is really weird, as, we copied the entire
index from a sane server onto the server having issues now just 2 days
ago. Do you have any idea why this would happen ?

[#|2012-05-07T12:58:54.055-0400|SEVERE|sun-appserver2.1.1|com.wpost.ipad.feeds.FeedController|_ThreadID=22;_ThreadName=httpSSLWorkerThread-9001-3;_RequestID=4203e3e5-c39d-4df7-a32a-600d0169c81f;|Error
searching for thumbnails for d101dd3a-979a-11e1-927c-291130c98dff
org.apache.solr.client.solrj.SolrServerException: Error executing query
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
	at xxx.xxx.xxx.xxx.populateThumbnails(FeedController.java:1184)
	at xxx.xxx.xxx.xxx..findNewsBySection(FeedController.java:509)
	at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
..
...
..
Caused by: java.lang.RuntimeException: Invalid version (expected 2,
but 60) or the data in not in 'javabin' format
	at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
	at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
	... 43 more
|#]

Ravi Kiran Bhaskar
Principal Software Engineer
Washington Post Digital
1150 15th Street NW, Washington, DC 20071

On Mon, May 7, 2012 at 9:36 AM, Mark Miller <ma...@gmail.com> wrote:
> Normally this specific error is caused by a non success http error page and response is returned. The response parser tries to parse HTML as javabin.
>
> Sent from my iPhone
>
> On May 7, 2012, at 7:37 AM, Erick Erickson <er...@gmail.com> wrote:
>
>> Well, I'm guessing that the version of Solr (and perhaps there are
>> classpath issues in here?) are different, somehow, on the machine
>> slave that is showing the error.
>>
>> It's also possible that your config files have a different  LUCENE_VERSION
>> in them, although I don't think this should really create the errors you're
>> reporting.
>>
>> The thing that leads me in this direction is your statement that things
>> are fine for a while and then go bad later.  If replication happens just
>> before you get the index version error, that would point a finger at
>> something like different Solr versions.
>>
>> If there is no replication before this error, then this probably isn't
>> the problem
>> and we'll have to look elsewhere...
>>
>> But this is all guesswork, just like every bug... things are only obvious after
>> you find the problem!
>>
>> Best
>> Erick
>>
>>
>> On Sun, May 6, 2012 at 11:08 AM, Ravi Solr <ra...@gmail.com> wrote:
>>> Thank you very much for responding Mr.Erickson. You may be right on
>>> old version index, I will reindex. However we have a 2
>>> separate/disjoint master-slave setup...only one query node/slave has
>>> this issue. if it was really incompatible indexes why isnt the other
>>> query server also throwing errors? that's what is throwing my
>>> debugging thought process off.
>>>
>>> Thanks
>>>
>>> Ravi Kiran Bhaskar
>>> Principal Software Engineer
>>> Washington Post Digital
>>> 1150 15th Street NW, Washington, DC 20071
>>>
>>> On Sat, May 5, 2012 at 12:53 PM, Erick Erickson <er...@gmail.com> wrote:
>>>> The first thing I'd check is if, in the log, there is a replication happening
>>>> immediately prior to the error. I confess I'm not entirely up on the
>>>> version thing, but is it possible you're replicating an index that
>>>> is built with some other version of Solr?
>>>>
>>>> That would at least explain your statement that it runs OK, but then
>>>> fails sometime later.
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Fri, May 4, 2012 at 1:50 PM, Ravi Solr <ra...@gmail.com> wrote:
>>>>> Hello,
>>>>>         We Recently we migrated our SOLR 3.6 server OS from Solaris
>>>>> to CentOS and from then on we started seeing "Invalid version
>>>>> (expected 2, but 60)" errors on one of the query servers (oddly one
>>>>> other query server seems fine). If we restart the server having issue
>>>>> everything will be alright, but the next day in the morning again we
>>>>> get the same exception. I made sure that all the client applications
>>>>> are using SOLR 3.6 version.
>>>>>
>>>>> The Glassfish on which all the applications  and SOLR are deployed use
>>>>> Java  1.6.0_29. The only difference I could see
>>>>>
>>>>> 1. The process indexing to the server having issues is using java1.6.0_31
>>>>> 2. The process indexing to the server that DOES NOT have issues is
>>>>> using java1.6.0_29
>>>>>
>>>>> Could the Java minor version being greater than the SOLR instance be
>>>>> the cause of this issue  ???
>>>>>
>>>>> Can anybody please help me debug this a bit more ? what else can I
>>>>> look at to understand the underlying problem. The stack trace is given
>>>>> below
>>>>>
>>>>>
>>>>> [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx.xxxx.xxxx.xxxx|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5;
>>>>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>>>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>>>>        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>>>>        at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743)
>>>>>        at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347)
>>>>>        at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
>>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
>>>>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
>>>>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
>>>>>        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
>>>>>        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
>>>>>        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
>>>>>        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
>>>>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
>>>>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
>>>>>        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
>>>>>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
>>>>>        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
>>>>>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>>        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
>>>>>        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
>>>>>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>>>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>>>        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
>>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
>>>>>        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393)
>>>>>        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
>>>>>        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
>>>>> Caused by: java.lang.RuntimeException: Invalid version (expected 2,
>>>>> but 60) or the data in not in 'javabin' format
>>>>>        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
>>>>>        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
>>>>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
>>>>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>>>>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
>>>>>        ... 43 more
>>>>>
>>>>>
>>>>> Thanking you in anticipation,
>>>>>
>>>>> Ravi Kiran Bhaskar
>>>>> Principal Software Engineer
>>>>> Washington Post Digital
>>>>> 1150 15th Street NW, Washington, DC 20071

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Mark Miller <ma...@gmail.com>.

Normally this specific error is caused by a non success http error page and response is returned. The response parser tries to parse HTML as javabin. 

Sent from my iPhone

On May 7, 2012, at 7:37 AM, Erick Erickson <er...@gmail.com> wrote:

> Well, I'm guessing that the version of Solr (and perhaps there are
> classpath issues in here?) are different, somehow, on the machine
> slave that is showing the error.
> 
> It's also possible that your config files have a different  LUCENE_VERSION
> in them, although I don't think this should really create the errors you're
> reporting.
> 
> The thing that leads me in this direction is your statement that things
> are fine for a while and then go bad later.  If replication happens just
> before you get the index version error, that would point a finger at
> something like different Solr versions.
> 
> If there is no replication before this error, then this probably isn't
> the problem
> and we'll have to look elsewhere...
> 
> But this is all guesswork, just like every bug... things are only obvious after
> you find the problem!
> 
> Best
> Erick
> 
> 
> On Sun, May 6, 2012 at 11:08 AM, Ravi Solr <ra...@gmail.com> wrote:
>> Thank you very much for responding Mr.Erickson. You may be right on
>> old version index, I will reindex. However we have a 2
>> separate/disjoint master-slave setup...only one query node/slave has
>> this issue. if it was really incompatible indexes why isnt the other
>> query server also throwing errors? that's what is throwing my
>> debugging thought process off.
>> 
>> Thanks
>> 
>> Ravi Kiran Bhaskar
>> Principal Software Engineer
>> Washington Post Digital
>> 1150 15th Street NW, Washington, DC 20071
>> 
>> On Sat, May 5, 2012 at 12:53 PM, Erick Erickson <er...@gmail.com> wrote:
>>> The first thing I'd check is if, in the log, there is a replication happening
>>> immediately prior to the error. I confess I'm not entirely up on the
>>> version thing, but is it possible you're replicating an index that
>>> is built with some other version of Solr?
>>> 
>>> That would at least explain your statement that it runs OK, but then
>>> fails sometime later.
>>> 
>>> Best
>>> Erick
>>> 
>>> On Fri, May 4, 2012 at 1:50 PM, Ravi Solr <ra...@gmail.com> wrote:
>>>> Hello,
>>>>         We Recently we migrated our SOLR 3.6 server OS from Solaris
>>>> to CentOS and from then on we started seeing "Invalid version
>>>> (expected 2, but 60)" errors on one of the query servers (oddly one
>>>> other query server seems fine). If we restart the server having issue
>>>> everything will be alright, but the next day in the morning again we
>>>> get the same exception. I made sure that all the client applications
>>>> are using SOLR 3.6 version.
>>>> 
>>>> The Glassfish on which all the applications  and SOLR are deployed use
>>>> Java  1.6.0_29. The only difference I could see
>>>> 
>>>> 1. The process indexing to the server having issues is using java1.6.0_31
>>>> 2. The process indexing to the server that DOES NOT have issues is
>>>> using java1.6.0_29
>>>> 
>>>> Could the Java minor version being greater than the SOLR instance be
>>>> the cause of this issue  ???
>>>> 
>>>> Can anybody please help me debug this a bit more ? what else can I
>>>> look at to understand the underlying problem. The stack trace is given
>>>> below
>>>> 
>>>> 
>>>> [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx.xxxx.xxxx.xxxx|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5;
>>>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>>>        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>>>        at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743)
>>>>        at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347)
>>>>        at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
>>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>>        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
>>>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
>>>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
>>>>        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
>>>>        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
>>>>        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
>>>>        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
>>>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
>>>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
>>>>        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
>>>>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
>>>>        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
>>>>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
>>>>        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
>>>>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>>        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
>>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
>>>>        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393)
>>>>        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
>>>>        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
>>>> Caused by: java.lang.RuntimeException: Invalid version (expected 2,
>>>> but 60) or the data in not in 'javabin' format
>>>>        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
>>>>        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
>>>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
>>>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>>>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
>>>>        ... 43 more
>>>> 
>>>> 
>>>> Thanking you in anticipation,
>>>> 
>>>> Ravi Kiran Bhaskar
>>>> Principal Software Engineer
>>>> Washington Post Digital
>>>> 1150 15th Street NW, Washington, DC 20071

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Erick Erickson <er...@gmail.com>.

Well, I'm guessing that the version of Solr (and perhaps there are
classpath issues in here?) are different, somehow, on the machine
slave that is showing the error.

It's also possible that your config files have a different  LUCENE_VERSION
in them, although I don't think this should really create the errors you're
reporting.

The thing that leads me in this direction is your statement that things
are fine for a while and then go bad later.  If replication happens just
before you get the index version error, that would point a finger at
something like different Solr versions.

If there is no replication before this error, then this probably isn't
the problem
and we'll have to look elsewhere...

But this is all guesswork, just like every bug... things are only obvious after
you find the problem!

Best
Erick


On Sun, May 6, 2012 at 11:08 AM, Ravi Solr <ra...@gmail.com> wrote:
> Thank you very much for responding Mr.Erickson. You may be right on
> old version index, I will reindex. However we have a 2
> separate/disjoint master-slave setup...only one query node/slave has
> this issue. if it was really incompatible indexes why isnt the other
> query server also throwing errors? that's what is throwing my
> debugging thought process off.
>
> Thanks
>
> Ravi Kiran Bhaskar
> Principal Software Engineer
> Washington Post Digital
> 1150 15th Street NW, Washington, DC 20071
>
> On Sat, May 5, 2012 at 12:53 PM, Erick Erickson <er...@gmail.com> wrote:
>> The first thing I'd check is if, in the log, there is a replication happening
>> immediately prior to the error. I confess I'm not entirely up on the
>> version thing, but is it possible you're replicating an index that
>> is built with some other version of Solr?
>>
>> That would at least explain your statement that it runs OK, but then
>> fails sometime later.
>>
>> Best
>> Erick
>>
>> On Fri, May 4, 2012 at 1:50 PM, Ravi Solr <ra...@gmail.com> wrote:
>>> Hello,
>>>         We Recently we migrated our SOLR 3.6 server OS from Solaris
>>> to CentOS and from then on we started seeing "Invalid version
>>> (expected 2, but 60)" errors on one of the query servers (oddly one
>>> other query server seems fine). If we restart the server having issue
>>> everything will be alright, but the next day in the morning again we
>>> get the same exception. I made sure that all the client applications
>>> are using SOLR 3.6 version.
>>>
>>> The Glassfish on which all the applications  and SOLR are deployed use
>>> Java  1.6.0_29. The only difference I could see
>>>
>>> 1. The process indexing to the server having issues is using java1.6.0_31
>>> 2. The process indexing to the server that DOES NOT have issues is
>>> using java1.6.0_29
>>>
>>> Could the Java minor version being greater than the SOLR instance be
>>> the cause of this issue  ???
>>>
>>> Can anybody please help me debug this a bit more ? what else can I
>>> look at to understand the underlying problem. The stack trace is given
>>> below
>>>
>>>
>>> [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx.xxxx.xxxx.xxxx|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5;
>>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>>        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>>        at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743)
>>>        at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347)
>>>        at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
>>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
>>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
>>>        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
>>>        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
>>>        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
>>>        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
>>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
>>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
>>>        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
>>>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
>>>        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
>>>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
>>>        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
>>>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>>        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
>>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
>>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
>>>        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393)
>>>        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
>>>        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
>>> Caused by: java.lang.RuntimeException: Invalid version (expected 2,
>>> but 60) or the data in not in 'javabin' format
>>>        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
>>>        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
>>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
>>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
>>>        ... 43 more
>>>
>>>
>>> Thanking you in anticipation,
>>>
>>> Ravi Kiran Bhaskar
>>> Principal Software Engineer
>>> Washington Post Digital
>>> 1150 15th Street NW, Washington, DC 20071

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Ravi Solr <ra...@gmail.com>.

Thank you very much for responding Mr.Erickson. You may be right on
old version index, I will reindex. However we have a 2
separate/disjoint master-slave setup...only one query node/slave has
this issue. if it was really incompatible indexes why isnt the other
query server also throwing errors? that's what is throwing my
debugging thought process off.

Thanks

Ravi Kiran Bhaskar
Principal Software Engineer
Washington Post Digital
1150 15th Street NW, Washington, DC 20071

On Sat, May 5, 2012 at 12:53 PM, Erick Erickson <er...@gmail.com> wrote:
> The first thing I'd check is if, in the log, there is a replication happening
> immediately prior to the error. I confess I'm not entirely up on the
> version thing, but is it possible you're replicating an index that
> is built with some other version of Solr?
>
> That would at least explain your statement that it runs OK, but then
> fails sometime later.
>
> Best
> Erick
>
> On Fri, May 4, 2012 at 1:50 PM, Ravi Solr <ra...@gmail.com> wrote:
>> Hello,
>>         We Recently we migrated our SOLR 3.6 server OS from Solaris
>> to CentOS and from then on we started seeing "Invalid version
>> (expected 2, but 60)" errors on one of the query servers (oddly one
>> other query server seems fine). If we restart the server having issue
>> everything will be alright, but the next day in the morning again we
>> get the same exception. I made sure that all the client applications
>> are using SOLR 3.6 version.
>>
>> The Glassfish on which all the applications  and SOLR are deployed use
>> Java  1.6.0_29. The only difference I could see
>>
>> 1. The process indexing to the server having issues is using java1.6.0_31
>> 2. The process indexing to the server that DOES NOT have issues is
>> using java1.6.0_29
>>
>> Could the Java minor version being greater than the SOLR instance be
>> the cause of this issue  ???
>>
>> Can anybody please help me debug this a bit more ? what else can I
>> look at to understand the underlying problem. The stack trace is given
>> below
>>
>>
>> [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx.xxxx.xxxx.xxxx|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5;
>> org.apache.solr.client.solrj.SolrServerException: Error executing query
>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>>        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>>        at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743)
>>        at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347)
>>        at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
>>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
>>        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
>>        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
>>        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
>>        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
>>        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
>>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
>>        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
>>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
>>        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
>>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>>        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
>>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
>>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
>>        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393)
>>        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
>>        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
>> Caused by: java.lang.RuntimeException: Invalid version (expected 2,
>> but 60) or the data in not in 'javabin' format
>>        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
>>        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
>>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
>>        ... 43 more
>>
>>
>> Thanking you in anticipation,
>>
>> Ravi Kiran Bhaskar
>> Principal Software Engineer
>> Washington Post Digital
>> 1150 15th Street NW, Washington, DC 20071

Re: Invalid version (expected 2, but 60) on CentOS in production please Help!!!

Posted by Erick Erickson <er...@gmail.com>.

The first thing I'd check is if, in the log, there is a replication happening
immediately prior to the error. I confess I'm not entirely up on the
version thing, but is it possible you're replicating an index that
is built with some other version of Solr?

That would at least explain your statement that it runs OK, but then
fails sometime later.

Best
Erick

On Fri, May 4, 2012 at 1:50 PM, Ravi Solr <ra...@gmail.com> wrote:
> Hello,
>         We Recently we migrated our SOLR 3.6 server OS from Solaris
> to CentOS and from then on we started seeing "Invalid version
> (expected 2, but 60)" errors on one of the query servers (oddly one
> other query server seems fine). If we restart the server having issue
> everything will be alright, but the next day in the morning again we
> get the same exception. I made sure that all the client applications
> are using SOLR 3.6 version.
>
> The Glassfish on which all the applications  and SOLR are deployed use
> Java  1.6.0_29. The only difference I could see
>
> 1. The process indexing to the server having issues is using java1.6.0_31
> 2. The process indexing to the server that DOES NOT have issues is
> using java1.6.0_29
>
> Could the Java minor version being greater than the SOLR instance be
> the cause of this issue  ???
>
> Can anybody please help me debug this a bit more ? what else can I
> look at to understand the underlying problem. The stack trace is given
> below
>
>
> [#|2012-05-04T09:58:43.985-0400|SEVERE|sun-appserver2.1.1|xxx.xxxx.xxxx.xxxx|_ThreadID=32;_ThreadName=httpSSLWorkerThread-9001-7;_RequestID=a19f92cc-2a8c-47e8-b159-a20330f14af5;
> org.apache.solr.client.solrj.SolrServerException: Error executing query
>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
>        at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:311)
>        at com.wpost.ipad.feeds.FeedController.findLinksetNewsBySection(FeedController.java:743)
>        at com.wpost.ipad.feeds.FeedController.findNewsBySection(FeedController.java:347)
>        at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:421)
>        at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:409)
>        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:774)
>        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
>        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
>        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
>        at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
>        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
>        at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
>        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
>        at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
>        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
>        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
>        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
>        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
>        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:670)
>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:601)
>        at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:875)
>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:365)
>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:285)
>        at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:221)
>        at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:393)
>        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
>        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)
> Caused by: java.lang.RuntimeException: Invalid version (expected 2,
> but 60) or the data in not in 'javabin' format
>        at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
>        at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41)
>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:333)
>        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>        at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
>        ... 43 more
>
>
> Thanking you in anticipation,
>
> Ravi Kiran Bhaskar
> Principal Software Engineer
> Washington Post Digital
> 1150 15th Street NW, Washington, DC 20071