You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Erik Hatcher <er...@ehatchersolutions.com> on 2009/02/01 12:25:56 UTC
Re: Recent document boosting with dismax
Is your date_added field multiValued and you've assigned multiple to
some documents?
Erik
On Jan 31, 2009, at 4:12 PM, James Brady wrote:
> Hi,I'm following the recipe here:
> http://wiki.apache.org/solr/SolrRelevancyFAQ#head-b1b1cdedcb9cd9bfd9c994709b4d7e540359b1fdfor
> boosting recent documents: bf=recip(rord(date_added),1,1000,1000)
>
> On some of my servers I've started getting errors like this:
>
> SEVERE: java.lang.RuntimeException: there are more terms than
> documents in
> field "date_added", but it's impossible to sort on tokenized fields
> at
> org.apache.lucene.search.FieldCacheImpl
> $10.createValue(FieldCacheImpl.java:379)
> at org.apache.lucene.search.FieldCacheImpl
> $Cache.get(FieldCacheImpl.java:72)
> at
> org
> .apache
> .lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:352)
> at
> org
> .apache
> .solr
> .search
> .function.ReverseOrdFieldSource.getValues(ReverseOrdFieldSource.java:
> 55)
> at
> org
> .apache
> .solr
> .search
> .function
> .ReciprocalFloatFunction.getValues(ReciprocalFloatFunction.java:56)
> at
> org.apache.solr.search.function.FunctionQuery
> $AllScorer.<init>(FunctionQuery.java:103)
> at
> org.apache.solr.search.function.FunctionQuery
> $FunctionWeight.scorer(FunctionQuery.java:81)
> at
> org.apache.lucene.search.BooleanQuery
> $BooleanWeight.scorer(BooleanQuery.java:232)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
> 143)
> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
> ...
>
> The date_added field is stored as a vanilla Solr date type:
> <fieldType name="date" class="solr.DateField"
> sortMissingLast="true"
> omitNorms="true"/>
>
> I'm having lots of other problems (un-related) with corrupt indices
> - could
> it be that in running the org.apache.lucene.index.CheckIndex
> utility, and
> losing some documents in the process, the ordinal part of my boost
> function
> is permanently broken?
>
> Thanks!
> James
Re: Recent document boosting with dismax
Posted by James Brady <ja...@gmail.com>.
Great, thanks for that, Chris!
2009/2/3 Chris Hostetter <ho...@fucit.org>
>
> : Hi, no the data_added field was one per document.
>
> i would suggest adding multiValued="false" to your "date" fieldType so
> that Solr can enforce that for you -- otherwise we can't be 100% sure.
>
> if it really is only a single valued field, then i suspect you're right
> about the index corruption being the source of your problem, but it's
> not neccessarily a permenant problem. try optimizing your index, that
> should merge all the segments and purge any terms that aren't actually
> part of live documents (i think) ... if that doesn't work, rebuilding will
> be your best bet (and with that multiValued="false" will error if you
> are inadvertantly sending multiple values per document)
>
> : >> I'm having lots of other problems (un-related) with corrupt indices -
> : >> could
> : >> it be that in running the org.apache.lucene.index.CheckIndex utility,
> and
> : >> losing some documents in the process, the ordinal part of my boost
> : >> function
> : >> is permanently broken?
>
>
>
> -Hoss
>
>
Re: Recent document boosting with dismax
Posted by Chris Hostetter <ho...@fucit.org>.
: Hi, no the data_added field was one per document.
i would suggest adding multiValued="false" to your "date" fieldType so
that Solr can enforce that for you -- otherwise we can't be 100% sure.
if it really is only a single valued field, then i suspect you're right
about the index corruption being the source of your problem, but it's
not neccessarily a permenant problem. try optimizing your index, that
should merge all the segments and purge any terms that aren't actually
part of live documents (i think) ... if that doesn't work, rebuilding will
be your best bet (and with that multiValued="false" will error if you
are inadvertantly sending multiple values per document)
: >> I'm having lots of other problems (un-related) with corrupt indices -
: >> could
: >> it be that in running the org.apache.lucene.index.CheckIndex utility, and
: >> losing some documents in the process, the ordinal part of my boost
: >> function
: >> is permanently broken?
-Hoss
Re: Recent document boosting with dismax
Posted by James Brady <ja...@gmail.com>.
Hi, no the data_added field was one per document.
2009/2/1 Erik Hatcher <er...@ehatchersolutions.com>
> Is your date_added field multiValued and you've assigned multiple to some
> documents?
>
> Erik
>
>
> On Jan 31, 2009, at 4:12 PM, James Brady wrote:
>
> Hi,I'm following the recipe here:
>>
>> http://wiki.apache.org/solr/SolrRelevancyFAQ#head-b1b1cdedcb9cd9bfd9c994709b4d7e540359b1fdfor
>> boosting recent documents: bf=recip(rord(date_added),1,1000,1000)
>>
>> On some of my servers I've started getting errors like this:
>>
>> SEVERE: java.lang.RuntimeException: there are more terms than documents in
>> field "date_added", but it's impossible to sort on tokenized fields
>> at
>>
>> org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:379)
>> at
>> org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
>> at
>>
>> org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:352)
>> at
>>
>> org.apache.solr.search.function.ReverseOrdFieldSource.getValues(ReverseOrdFieldSource.java:55)
>> at
>>
>> org.apache.solr.search.function.ReciprocalFloatFunction.getValues(ReciprocalFloatFunction.java:56)
>> at
>>
>> org.apache.solr.search.function.FunctionQuery$AllScorer.<init>(FunctionQuery.java:103)
>> at
>>
>> org.apache.solr.search.function.FunctionQuery$FunctionWeight.scorer(FunctionQuery.java:81)
>> at
>>
>> org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:232)
>> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:143)
>> at org.apache.lucene.search.Searcher.search(Searcher.java:118)
>> ...
>>
>> The date_added field is stored as a vanilla Solr date type:
>> <fieldType name="date" class="solr.DateField" sortMissingLast="true"
>> omitNorms="true"/>
>>
>> I'm having lots of other problems (un-related) with corrupt indices -
>> could
>> it be that in running the org.apache.lucene.index.CheckIndex utility, and
>> losing some documents in the process, the ordinal part of my boost
>> function
>> is permanently broken?
>>
>> Thanks!
>> James
>>
>
>