You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Shamik Bandopadhyay <sh...@gmail.com> on 2015/03/11 19:02:43 UTC
Solr 5.0 --> "IllegalStateException: unexpected docvalues type NONE"
on result grouping
Hi,
I've a field which is being used for result grouping. Here's the field
definition.
<field name="ADDedup" type="string" indexed="true" stored="true"
multiValued="false" required="false" omitNorms="true" docValues="true"/>
This started once I did a rolling update from 4.7 to 5.0. I started getting
the error on any group by query --> "SolrDispatchFilter null:java.lang.
IllegalStateException: unexpected docvalues type NONE for field 'ADSKDedup'
(expected=SORTED). Use UninvertingReader or dex with docvalues."
Does this mean that I need to re-index documents to get over this error ?
Regards,
Shamik
Re: Solr 5.0 --> "IllegalStateException: unexpected docvalues type
NONE" on result grouping
Posted by shamik <sh...@gmail.com>.
Looks like it's happening for any field which is using docvalues.
java.lang.IllegalStateException: unexpected docvalues type NONE for field
'title_sort' (expected=SORTED). Use UninvertingReader or index with
docvalues.
Any idea ?
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-5-0-IllegalStateException-unexpected-docvalues-type-NONE-on-result-grouping-tp4192477p4192529.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 5.0 --> "IllegalStateException: unexpected docvalues type
NONE" on result grouping
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/11/2015 4:45 PM, shamik wrote:
> <field name="DocumentType" type="string" indexed="true" stored="true"
> multiValued="false" required="false" omitNorms="true" docValues="true"
> /> 3/11/2015, 2:14:30 PM ERROR SolrDispatchFilter
> null:java.lang.IllegalStateException: unexpected docvalues type NONE
> for field 'DocumentType' (expected=SORTED). Use UninvertingReader or
> index with docvalues. null:java.lang.IllegalStateException: unexpected
> docvalues type NONE for field 'DocumentType' (expected=SORTED). Use
> UninvertingReader or index with docvalues.
I admit right up front that I know very little about what might be
happening here ... but I did have one idea. It could be completely wrong.
Is it possible that you have an index that fits the following description?
The field originally did not have docValues. You enabled docValues on
the field in the schema, but there are index segments still in the index
directory from *before* you changed the schema.
If that sounds at all possible, then if you did not fully reindex, there
would be segments with valid documents that do not have docValues. You
should fully reindex and then optimize before upgrading. If you did
fully reindex, but did not optimize, then there might be segments with
*deleted* documents that do not have docValues ... and maybe 4.7 was
fine with that but 5.0 isn't.
Whenever I upgrade Solr, I always reindex from scratch, and often I will
completely delete all the data directories. It takes longer, but then I
know the index is 100% correct for the version and config I'm running.
I'll reiterate that this whole idea could be 100% wrong.
Thanks,
Shawn
Re: Solr 5.0 --> "IllegalStateException: unexpected docvalues type
NONE" on result grouping
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/12/2015 3:36 PM, Alexandre Rafalovitch wrote:
> Manual optimize is no longer needed for modern Solr. It does great
> optimization automatically. The only reason I recommended it here is
> to make sure that all segments are brought up to the latest version
> and the deleted documents are purged. That's something that also would
> happen automatically eventually, but "eventually" was not an option
> for you.
>
> I am glad this helped. I am not 100% sure if you have to do it on each
> shard in SolrCloud mode, but I suspect so.
In SolrCloud, whenever you send an optimize command to any shard replica
in a collection, the entire collection will be optimized. SolrCloud
will do the optimization sequentially, not in parallel. There is
currently no way to optimize only one shard replica, and as far as I
know, there is no way to ask for a parallel optimization.
Alexandre's comments about the necessity of optimization (whether it's
SolrCloud or not) is spot on. The only time that optimization should be
done on a modern Solr index is when you have a lot of deleted documents
and want to clean those up, either to reclaim disk space or remove them
from the relevancy calculation.
Most people do see a performance boost on an optimized index compared to
a non-optimized index, but with a modern Solr install, you might
actually see better performance on a multi-segment index when the
indexing rate is high, because Lucene is moving to a model where there
are per-segment caches that are not invalidated at commit time, only at
merge time.
Thanks,
Shawn
Re: Solr 5.0 --> "IllegalStateException: unexpected docvalues type
NONE" on result grouping
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Manual optimize is no longer needed for modern Solr. It does great
optimization automatically. The only reason I recommended it here is
to make sure that all segments are brought up to the latest version
and the deleted documents are purged. That's something that also would
happen automatically eventually, but "eventually" was not an option
for you.
I am glad this helped. I am not 100% sure if you have to do it on each
shard in SolrCloud mode, but I suspect so.
Regards,
Alex.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 12 March 2015 at 17:24, shamik <sh...@gmail.com> wrote:
> Wow, "optimize" worked like a charm. This really addressed the docvalues
> issue. A follow-up question, is it recommended to run optimize in a
> Production Solr index ? Also, in a Sorl cloud mode, do we need to run
> optimize on each instance / each shard / any instance ?
>
> Appreciate your help Alex.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-5-0-migration-IllegalStateException-unexpected-docvalues-type-NONE-on-fields-using-docvalues-tp4192477p4192732.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 5.0 --> "IllegalStateException: unexpected docvalues type
NONE" on result grouping
Posted by shamik <sh...@gmail.com>.
Wow, "optimize" worked like a charm. This really addressed the docvalues
issue. A follow-up question, is it recommended to run optimize in a
Production Solr index ? Also, in a Sorl cloud mode, do we need to run
optimize on each instance / each shard / any instance ?
Appreciate your help Alex.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-5-0-migration-IllegalStateException-unexpected-docvalues-type-NONE-on-fields-using-docvalues-tp4192477p4192732.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 5.0 --> "IllegalStateException: unexpected docvalues type
NONE" on result grouping
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Do you have any really old segments in that index? Could be worth
trying to optimize them down to one in latest format first.
Like Shawn, this is just a "one more idea" proposal.
Regards,
Alex.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 12 March 2015 at 14:47, shamik <sh...@gmail.com> wrote:
> Well, I think I've narrowed down the issue. The error is happening when I'm
> trying to do a rolling update from Solr 4.7 (which is our current version)
> to 5.0 . I'm able to re-produce this couple of times. If I do a fresh index
> on a 5.0, it works. Not sure if there's any other way to mitigate it.
>
> I'll appreciate if someone can share their experience on the same.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-5-0-migration-IllegalStateException-unexpected-docvalues-type-NONE-on-fields-using-docvalues-tp4192477p4192706.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 5.0 --> "IllegalStateException: unexpected docvalues type
NONE" on result grouping
Posted by shamik <sh...@gmail.com>.
Well, I think I've narrowed down the issue. The error is happening when I'm
trying to do a rolling update from Solr 4.7 (which is our current version)
to 5.0 . I'm able to re-produce this couple of times. If I do a fresh index
on a 5.0, it works. Not sure if there's any other way to mitigate it.
I'll appreciate if someone can share their experience on the same.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-5-0-migration-IllegalStateException-unexpected-docvalues-type-NONE-on-fields-using-docvalues-tp4192477p4192706.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 5.0 --> "IllegalStateException: unexpected docvalues type
NONE" on result grouping
Posted by shamik <sh...@gmail.com>.
Thanks for your reply. Initially, I was under the impression that the issue
is related to grouping as group queries were failing. Later, when I looked
further, I found that it's happening for any field for which the docvalue
has turned on. The second example I took was from another field. Here's a
full stack trace for another field using docvalues.
Field definition :
<field name="DocumentType" type="string" indexed="true" stored="true"
multiValued="false" required="false" omitNorms="true" docValues="true" />
3/11/2015, 2:14:30 PM ERROR SolrDispatchFilter
null:java.lang.IllegalStateException: unexpected docvalues type NONE for
field 'DocumentType' (expected=SORTED). Use UninvertingReader or index with
docvalues.
null:java.lang.IllegalStateException: unexpected docvalues type NONE for
field 'DocumentType' (expected=SORTED). Use UninvertingReader or index with
docvalues.
at org.apache.lucene.index.DocValues.checkField(DocValues.java:208)
at org.apache.lucene.index.DocValues.getSorted(DocValues.java:264)
at
org.apache.lucene.search.FieldComparator$TermOrdValComparator.getSortedDocValues(FieldComparator.java:757)
at
org.apache.lucene.search.FieldComparator$TermOrdValComparator.getLeafComparator(FieldComparator.java:762)
at
org.apache.lucene.search.FieldValueHitQueue.getComparators(FieldValueHitQueue.java:183)
at
org.apache.lucene.search.TopFieldCollector$NonScoringCollector.getLeafCollector(TopFieldCollector.java:141)
at
org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:99)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:583)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:284)
at
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:231)
at
org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1766)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1502)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:586)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:511)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:227)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2006)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
As you suggested, I'm going to try docvalues in a stand-alone 5.0 to see if
it has to do with the upgrade.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-5-0-IllegalStateException-unexpected-docvalues-type-NONE-on-result-grouping-tp4192477p4192537.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 5.0 --> "IllegalStateException: unexpected docvalues type
NONE" on result grouping
Posted by Chris Hostetter <ho...@fucit.org>.
1) the error message you posted doesn't appear to have been copied
verbatim (note: "...or dex...") ... please proivde the *exact* error you
are getting -- ideally with full stack trace from the solr logs.
2) the <field/> you posted doens't match the field name in your error
message
3)providing a <field/> w/o the <fieldType/> makes it hard to guess
exactly what settings you are using (ie: is "docValuesFormat" specified on
the fieldType?)
4) You said you are using result grouping, but you haven't actually shown
us an example of what your request (with all params) looks like ...
there's nohting in your email to even indicate grouping is involved.
Details matter...
https://wiki.apache.org/solr/UsingMailingLists
(PS: assuming no obvious mistakes, it would be helpful to know if you can
reproduce this problem using your same exact same configs & schema with a
newly built 5.0 index, or if it *only* affects you when you upgrade and
use an existing index from 4.7)
: Date: Wed, 11 Mar 2015 11:02:43 -0700
: From: Shamik Bandopadhyay <sh...@gmail.com>
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org
: Subject: Solr 5.0 --> "IllegalStateException: unexpected docvalues type NONE"
: on result grouping
:
: Hi,
:
: I've a field which is being used for result grouping. Here's the field
: definition.
:
: <field name="ADDedup" type="string" indexed="true" stored="true"
: multiValued="false" required="false" omitNorms="true" docValues="true"/>
:
: This started once I did a rolling update from 4.7 to 5.0. I started getting
: the error on any group by query --> "SolrDispatchFilter null:java.lang.
: IllegalStateException: unexpected docvalues type NONE for field 'ADSKDedup'
: (expected=SORTED). Use UninvertingReader or dex with docvalues."
:
: Does this mean that I need to re-index documents to get over this error ?
:
: Regards,
: Shamik
:
-Hoss
http://www.lucidworks.com/