You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jonathan Rochkind <ro...@jhu.edu> on 2011/10/27 23:00:21 UTC
changing omitNorms on an already built index
So Solr 1.4. I decided I wanted to change a field to have
omitNorms=true that didn't previously.
So I changed the schema to have omitNorms=true. And I reindexed all
documents.
But it seems to have had absolutely no effect. All relevancy rankings
seem to be the same.
Now, I could have a mistake somewhere else, maybe I didn't do what I
thought.
But I'm wondering if there are any known issues related to this, is
there something special you have to do to change a field from
omitNorms=false to omitNorms=true on an already built index? Other than
re-indexing everything? Any known issues relevant here?
Thanks for any help,
Jonathan
Re: changing omitNorms on an already built index
Posted by Jonathan Rochkind <ro...@jhu.edu>.
On 10/27/2011 9:14 PM, Erick Erickson wrote:
> Well, this could be explained if your fields are very short. Norms
> are encoded into (part of?) a byte, so your ranking may be unaffected.
>
> Try adding debugQuery=on and looking at the explanation. If you've
> really omitted norms, I think you should see clauses like:
>
> 1.0 = fieldNorm(field=features, doc=1)
> in the output, never something like
Thanks, this was very helpful. Indeed with debugQuery on, I get "1.0 =
fieldNorm" on my index with omitNorms for the relevant field, and in my
index without omitNorms for the relevant field, I get a non-unit value
"= fieldNorm", thanks for giving me a way to reassure myself that
omitNorms really is doing it's thing.
Now to dive into my debugQuery and figure out why it doesn't seem to be
having as much effect as I anticipated on relevance!
Re: changing omitNorms on an already built index
Posted by Erick Erickson <er...@gmail.com>.
Well, this could be explained if your fields are very short. Norms
are encoded into (part of?) a byte, so your ranking may be unaffected.
Try adding debugQuery=on and looking at the explanation. If you've
really omitted norms, I think you should see clauses like:
1.0 = fieldNorm(field=features, doc=1)
in the output, never something like
0.25 = fieldNorm(field=features, doc=1)
i.e. in the absence of norm information, 1 is used.
Also, in your index, see if your *.nrm files change in size.
And I recommend that, when you're experimenting, you remove
your entire <solr home>/data/index directory (the directory too,
not just sub-files) before re-indexing. As Simon and Robert say,
eventually the norm data will be purged, but by removing the
directory first, you can look at things like the .nrm file with
confidence that you're not seeing remnants that haven't been
cleaned up quite yet.
Best
Erick
On Thu, Oct 27, 2011 at 5:00 PM, Jonathan Rochkind <ro...@jhu.edu> wrote:
> So Solr 1.4. I decided I wanted to change a field to have omitNorms=true
> that didn't previously.
>
> So I changed the schema to have omitNorms=true. And I reindexed all
> documents.
>
> But it seems to have had absolutely no effect. All relevancy rankings seem
> to be the same.
>
> Now, I could have a mistake somewhere else, maybe I didn't do what I
> thought.
>
> But I'm wondering if there are any known issues related to this, is there
> something special you have to do to change a field from omitNorms=false to
> omitNorms=true on an already built index? Other than re-indexing
> everything? Any known issues relevant here?
>
> Thanks for any help,
>
> Jonathan
>
Re: changing omitNorms on an already built index
Posted by Simon Willnauer <si...@googlemail.com>.
On Fri, Oct 28, 2011 at 12:20 AM, Robert Muir <rc...@gmail.com> wrote:
> On Thu, Oct 27, 2011 at 6:00 PM, Simon Willnauer
> <si...@googlemail.com> wrote:
>> we are not actively removing norms. if you set omitNorms=true and
>> index documents they won't have norms for this field. Yet, other
>> segment still have norms until they get merged with a segment that has
>> no norms for that field ie. omits norms. omitNorms is anti-viral so
>> once you set it to true it will be true for other segment eventually.
>> If you optimize you index you should see that norms go away.
>>
>
> This is only true in trunk (4.x!)
> https://issues.apache.org/jira/browse/LUCENE-2846
ah right, I thought this was ported - nevermind! thanks robert
simon
>
> --
> lucidimagination.com
>
Re: changing omitNorms on an already built index
Posted by Robert Muir <rc...@gmail.com>.
On Thu, Oct 27, 2011 at 6:00 PM, Simon Willnauer
<si...@googlemail.com> wrote:
> we are not actively removing norms. if you set omitNorms=true and
> index documents they won't have norms for this field. Yet, other
> segment still have norms until they get merged with a segment that has
> no norms for that field ie. omits norms. omitNorms is anti-viral so
> once you set it to true it will be true for other segment eventually.
> If you optimize you index you should see that norms go away.
>
This is only true in trunk (4.x!)
https://issues.apache.org/jira/browse/LUCENE-2846
--
lucidimagination.com
Re: changing omitNorms on an already built index
Posted by Simon Willnauer <si...@googlemail.com>.
we are not actively removing norms. if you set omitNorms=true and
index documents they won't have norms for this field. Yet, other
segment still have norms until they get merged with a segment that has
no norms for that field ie. omits norms. omitNorms is anti-viral so
once you set it to true it will be true for other segment eventually.
If you optimize you index you should see that norms go away.
simon
On Thu, Oct 27, 2011 at 11:17 PM, Marc Sturlese <ma...@gmail.com> wrote:
> As far as I know there's no issue about this. You have to reindex and that's
> it.
> In which kind of field are you changing the norms? (You just will see
> changes in text fields)
> Using debugQuery=true you can see how norms affect the score (in case you
> have them not omited)
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/changing-omitNorms-on-an-already-built-index-tp3459132p3459169.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: changing omitNorms on an already built index
Posted by Marc Sturlese <ma...@gmail.com>.
As far as I know there's no issue about this. You have to reindex and that's
it.
In which kind of field are you changing the norms? (You just will see
changes in text fields)
Using debugQuery=true you can see how norms affect the score (in case you
have them not omited)
--
View this message in context: http://lucene.472066.n3.nabble.com/changing-omitNorms-on-an-already-built-index-tp3459132p3459169.html
Sent from the Solr - User mailing list archive at Nabble.com.