NOT_ANALYSED_NO_NORMS should get max field length boost

[Paul Taylor <pa...@fastmail.fm>]

Lucene in Action says you can possibly use NOT_ANALYSED_NO_NORMS when 
indexing fields that arent tokenized, but later says norms are used to 
boost fields with less /single term, so matches based on these single 
term fields would miss out on this boost. Is there a way to use 
NOT_ANALYSED_NO_NORMS on these fields will will mean they end up with 
the best boost (1.0 as default) , and then documents that are analysed 
with norms receive a negative boost (<1.0) if they contain more than one 
token.

I'm not using Document or Field boosting, so seems a bit silly for me to 
store all these norms just to say this field contains a single token and 
therefore should get an addtional boost.

Perhaps Im misundersanding this, and this would work as required.


thanks Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: NOT_ANALYSED_NO_NORMS should get max field length boost

[Paul Taylor <pa...@fastmail.fm>]

Erick Erickson wrote:
> Are you saying that you index the *same* field differently in different
> documents? Or do you index the field in question in the same way in
> all documents?
Same way in all documents
>
> I ask because I'm having a hard time following the logic here. A
> field that is NOT analyzed is an all-or-none match, i.e.
> looking for "paul" in an unanalyzed field "paul taylor" will
> not match, so boosting is pretty irrelevant on that field.
>
But if it does match won't this affect the end score

i.e, consider Doc 1 has unalysed field containing fred and Doc 2 
analysed field contain tom sawyer
and this search unalysedfield:fred OR analysedfield:tom

If it had norms, Doc1 would come first , but without norms they would be 
joint first right ?

>
> Could you boost your *other* fields by less than one to achieve
> the same end?
What are the values, what is the value given to a single field norm boosted

Paul


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: NOT_ANALYSED_NO_NORMS should get max field length boost

[Paul Taylor <pa...@fastmail.fm>]

> On Tue, Jan 12, 2010 at 7:53 AM, Paul Taylor <paul_t100@fastmail.fm 
> <ma...@fastmail.fm>> wrote:
>
>     Lucene in Action says you can possibly use NOT_ANALYSED_NO_NORMS
>     when indexing fields that arent tokenized, but later says norms
>     are used to boost fields with less /single term, so matches based
>     on these single term fields would miss out on this boost. Is there
>     a way to use NOT_ANALYSED_NO_NORMS on these fields will will mean
>     they end up with the best boost (1.0 as default) , and then
>     documents that are analysed with norms receive a negative boost
>     (<1.0) if they contain more than one token.
>
>     I'm not using Document or Field boosting, so seems a bit silly for
>     me to store all these norms just to say this field contains a
>     single token and therefore should get an addtional boost.
>
>     Perhaps Im misundersanding this, and this would work as required.
>
>
>     thanks Paul
>
FYI Looking at DefaultSimailarity() the lengthNorm is 1/sqrt(numTerms), 
so for one term would equal 1. i.e. the same as not having norms, so 
AFAIK there is no difference after all if not using Document or field 
boosting

Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: NOT_ANALYSED_NO_NORMS should get max field length boost

[Erick Erickson <er...@gmail.com>]

Are you saying that you index the *same* field differently in different
documents? Or do you index the field in question in the same way in
all documents?

I ask because I'm having a hard time following the logic here. A
field that is NOT analyzed is an all-or-none match, i.e.
looking for "paul" in an unanalyzed field "paul taylor" will
not match, so boosting is pretty irrelevant on that field.

If you're analyzing the same field for some documents and not
analyzing it for other documents, I don't know what happens, but
it's probably bad.

Could you boost your *other* fields by less than one to achieve
the same end?

If none of this is relevant, could you explain your use case a little
more?

HTH
Erick

On Tue, Jan 12, 2010 at 7:53 AM, Paul Taylor <pa...@fastmail.fm> wrote:

> Lucene in Action says you can possibly use NOT_ANALYSED_NO_NORMS when
> indexing fields that arent tokenized, but later says norms are used to boost
> fields with less /single term, so matches based on these single term fields
> would miss out on this boost. Is there a way to use NOT_ANALYSED_NO_NORMS on
> these fields will will mean they end up with the best boost (1.0 as default)
> , and then documents that are analysed with norms receive a negative boost
> (<1.0) if they contain more than one token.
>
> I'm not using Document or Field boosting, so seems a bit silly for me to
> store all these norms just to say this field contains a single token and
> therefore should get an addtional boost.
>
> Perhaps Im misundersanding this, and this would work as required.
>
>
> thanks Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>