You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Jordi Salvat i Alabart (Created) (JIRA)" <ji...@apache.org> on 2012/04/05 12:05:25 UTC
[jira] [Created] (LUCENE-3957) Document precision requirements of
setBoost calls
Document precision requirements of setBoost calls
-------------------------------------------------
Key: LUCENE-3957
URL: https://issues.apache.org/jira/browse/LUCENE-3957
Project: Lucene - Java
Issue Type: Improvement
Components: general/javadocs
Affects Versions: 3.5
Reporter: Jordi Salvat i Alabart
The behaviour of index-time boosts seems pretty erratic (e.g. a boost of 8.0 produces the exact same score as a boost of 9.0) until you become aware that these factors end up encoded in a single byte, with a three-bit mantissa. This consumed a whole day of research for us, and I still believe we were lucky to spot it, given how deeply dug into the code & documentation this information is.
I suggest adding a small note to the JavaDoc of setBoost methods in Document, Fieldable, FieldInvertState, and possibly AbstractField, Field, and NumericField.
Suggested text:
"Note that all index-time boost values end up encoded using Similarity.encodeNormValue, with a 3-bit mantissa -- so differences in the boost value of less than 25% may easily be rounded away."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3957) Document precision requirements of
setBoost calls
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256760#comment-13256760 ]
Robert Muir commented on LUCENE-3957:
-------------------------------------
I don't understand why its long and winded, its documented in tons of places in lucene,
in-fact its actually over-specified in file-formats, for example, because even in 3.5
the encoding of the normalization byte is an implementation detail of the Similarity:
its just that you can only use a single byte.
In trunk its definitely overspecified since besides the above, the Similarity can use
more than a byte if it wants to.
1. Main website (scoring):
http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/scoring.html
{noformat}
Indexing time boosts are preprocessed for storage efficiency and written to the directory (when writing the document) in a single byte (!) as follows.
...
This composition of 1-byte representation of norms...
...
Encoding and decoding of the resulted float norm in a single byte are done by the static methods of the class Similarity: encodeNorm() and decodeNorm(). Due to loss of precision, it is not guaranteed that decode(encode(x)) = x, e.g. decode(encode(0.89)) = 0.75. At scoring (search) time, this norm is brought into the score of document as norm(t, d), as shown by the formula in Similarity.
{noformat}
2. Main website (file formats):
http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/fileformats.html#Normalization%20Factors
{noformat}
Each byte encodes a floating point value. Bits 0-2 contain the 3-bit mantissa, and bits 3-8 contain the 5-bit exponent.
These are converted to an IEEE single float value as follows:
...
{noformat}
3. Javadocs (Similarity):
http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/api/core/org/apache/lucene/search/Similarity.html
{noformat}
However the resulted norm value is encoded as a single byte before being stored. At search time, the norm byte value is read from the index directory and decoded back to a float norm value. This encoding/decoding, while reducing index size, comes with the price of precision loss...
Compression of norm values to a single byte saves memory at search time, because once a field is referenced at search time, its norms - for all documents - are maintained in memory.
The rationale supporting such lossy compression of norm values is that given the difficulty (and inaccuracy) of users to express their true information need by a query, only big differences matter.
{noformat}
> Document precision requirements of setBoost calls
> -------------------------------------------------
>
> Key: LUCENE-3957
> URL: https://issues.apache.org/jira/browse/LUCENE-3957
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/javadocs
> Affects Versions: 3.5
> Reporter: Jordi Salvat i Alabart
>
> The behaviour of index-time boosts seems pretty erratic (e.g. a boost of 8.0 produces the exact same score as a boost of 9.0) until you become aware that these factors end up encoded in a single byte, with a three-bit mantissa. This consumed a whole day of research for us, and I still believe we were lucky to spot it, given how deeply dug into the code & documentation this information is.
> I suggest adding a small note to the JavaDoc of setBoost methods in Document, Fieldable, FieldInvertState, and possibly AbstractField, Field, and NumericField.
> Suggested text:
> "Note that all index-time boost values end up encoded using Similarity.encodeNormValue, with a 3-bit mantissa -- so differences in the boost value of less than 25% may easily be rounded away."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3957) Document precision requirements of
setBoost calls
Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248215#comment-13248215 ]
Simon Willnauer commented on LUCENE-3957:
-----------------------------------------
bq. I don't think we should put docs there in those classes:
+1 thats a very long standing impl detail. If it is not on the wiki in the FAQ we should add it or maybe add a pitfalls section to the wiki.
> Document precision requirements of setBoost calls
> -------------------------------------------------
>
> Key: LUCENE-3957
> URL: https://issues.apache.org/jira/browse/LUCENE-3957
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/javadocs
> Affects Versions: 3.5
> Reporter: Jordi Salvat i Alabart
>
> The behaviour of index-time boosts seems pretty erratic (e.g. a boost of 8.0 produces the exact same score as a boost of 9.0) until you become aware that these factors end up encoded in a single byte, with a three-bit mantissa. This consumed a whole day of research for us, and I still believe we were lucky to spot it, given how deeply dug into the code & documentation this information is.
> I suggest adding a small note to the JavaDoc of setBoost methods in Document, Fieldable, FieldInvertState, and possibly AbstractField, Field, and NumericField.
> Suggested text:
> "Note that all index-time boost values end up encoded using Similarity.encodeNormValue, with a 3-bit mantissa -- so differences in the boost value of less than 25% may easily be rounded away."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3957) Document precision requirements of
setBoost calls
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247185#comment-13247185 ]
Robert Muir commented on LUCENE-3957:
-------------------------------------
I don't think we should put docs there in those classes:
in trunk you can use however many bytes you want for this,
so the text would be wrong anyway.
its an implementation detail of the default similarity that
it uses a single byte, and it documents it does this already.
> Document precision requirements of setBoost calls
> -------------------------------------------------
>
> Key: LUCENE-3957
> URL: https://issues.apache.org/jira/browse/LUCENE-3957
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/javadocs
> Affects Versions: 3.5
> Reporter: Jordi Salvat i Alabart
>
> The behaviour of index-time boosts seems pretty erratic (e.g. a boost of 8.0 produces the exact same score as a boost of 9.0) until you become aware that these factors end up encoded in a single byte, with a three-bit mantissa. This consumed a whole day of research for us, and I still believe we were lucky to spot it, given how deeply dug into the code & documentation this information is.
> I suggest adding a small note to the JavaDoc of setBoost methods in Document, Fieldable, FieldInvertState, and possibly AbstractField, Field, and NumericField.
> Suggested text:
> "Note that all index-time boost values end up encoded using Similarity.encodeNormValue, with a 3-bit mantissa -- so differences in the boost value of less than 25% may easily be rounded away."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3957) Document precision requirements of
setBoost calls
Posted by "Jordi Salvat i Alabart (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256736#comment-13256736 ]
Jordi Salvat i Alabart commented on LUCENE-3957:
------------------------------------------------
Wherever, but please increase exposure of this detail, for the sake of those who're coming in later.
It's a VERY long and winded way between setting a $docBoost in a SolR dataSource transformer script and the current location of this information. Would be kind of acceptable if the value was held with a couple of decimal digits of precision, but the default (3-bit) implementation doesn't hold even ONE digit, making the behaviour really shocking -- specially since the boost value is not directly visible, but only through its impact on search scores.
> Document precision requirements of setBoost calls
> -------------------------------------------------
>
> Key: LUCENE-3957
> URL: https://issues.apache.org/jira/browse/LUCENE-3957
> Project: Lucene - Java
> Issue Type: Improvement
> Components: general/javadocs
> Affects Versions: 3.5
> Reporter: Jordi Salvat i Alabart
>
> The behaviour of index-time boosts seems pretty erratic (e.g. a boost of 8.0 produces the exact same score as a boost of 9.0) until you become aware that these factors end up encoded in a single byte, with a three-bit mantissa. This consumed a whole day of research for us, and I still believe we were lucky to spot it, given how deeply dug into the code & documentation this information is.
> I suggest adding a small note to the JavaDoc of setBoost methods in Document, Fieldable, FieldInvertState, and possibly AbstractField, Field, and NumericField.
> Suggested text:
> "Note that all index-time boost values end up encoded using Similarity.encodeNormValue, with a 3-bit mantissa -- so differences in the boost value of less than 25% may easily be rounded away."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org