You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Hoss Man (JIRA)" <ji...@apache.org> on 2016/06/03 00:43:59 UTC
[jira] [Updated] (LUCENE-7132) BooleanQuery scores can be diff for
same docs+sim when using coord (disagree with Explanation which doesn't
change)
[ https://issues.apache.org/jira/browse/LUCENE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated LUCENE-7132:
-----------------------------
Attachment: LUCENE-7132.patch
I was reminded of this issue today and pinged [~mikemccand] about it on IRC.
In reviewing the issue myself, to try and describe it to him succinctly, I realized the key bullet points of this bug is:
* It is possible to build an index, _with *no* deleted documents_ such that a particular BooleanQuery returns drastically different scores for some documents depending on if/how the segments in the index have been merged.
* The Explanations for every document matching this BooleanQuery do *not* change based on if/how the segments in the index have been merged -- such that the Explanations can be drastically different then the scores
Couched that way, and with a request from mike to try and confirm if the bug was dependent on using LuceneTestCase.newIndexWriterConfig (i gather he suspected it might be a test only bug) I revamped the previous test class to attempt to more straight forwardly demonstrate the crux of the matter.
There are now two test methods: testScoresWithDefaultIWC & testScoresWithRandomIWC -- both delegating to the same helpe method for the meat of the test, just using different IndexWriterConfigs.
"checkScores" is the meat of both test methods. It builds up an index using the same "RajeshData.txt" and then executes a fixed query against the index, recording the scores of every mathcing document. It then does a {{forceMerge(1,true)}} on the IndexWriter, and re-executes the query comparing the scores of every matching document against the scores from the previous query execution.
If the tests make it this far (and they rarely do) then does the query again, this time checking the "Explanation" for every matching document against the score.
----
With this seed, both tests demonstrate an identical inconsistency between the scores of a document containing body="Microsoft Office 365" between the pre-merge and single-segment indexes....
{noformat}
[junit4] <JUnit4> says ahoj! Master seed: B6DEF72C383813DE
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(29777@localhost).
[junit4] Suite: org.apache.lucene.search.TestBooleanScoreConsistency
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithRandomIWC -Dtests.seed=B6DEF72C383813DE -Dtests.slow=true -Dtests.locale=ar-IQ -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 2.80s | TestBooleanScoreConsistency.testScoresWithRandomIWC <<<
[junit4] > Throwable #1: java.lang.AssertionError: 7614: score doesn't match (previously returned) expected score for Microsoft Office 365 expected:<1.2357021570205688> but was:<0.41190072894096375>
[junit4] > at __randomizedtesting.SeedInfo.seed([B6DEF72C383813DE:2C10E99E7CDA36F8]:0)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExpected(TestBooleanScoreConsistency.java:178)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:137)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithRandomIWC(TestBooleanScoreConsistency.java:85)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithDefaultIWC -Dtests.seed=B6DEF72C383813DE -Dtests.slow=true -Dtests.locale=ar-IQ -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 1.63s | TestBooleanScoreConsistency.testScoresWithDefaultIWC <<<
[junit4] > Throwable #1: java.lang.AssertionError: 7614: score doesn't match (previously returned) expected score for Microsoft Office 365 expected:<1.2357021570205688> but was:<0.41190072894096375>
[junit4] > at __randomizedtesting.SeedInfo.seed([B6DEF72C383813DE:1CF87A1A7D4B3415]:0)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExpected(TestBooleanScoreConsistency.java:178)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:137)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC(TestBooleanScoreConsistency.java:79)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {id=PostingsFormat(name=Direct), body=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=825, maxMBSortInHeap=5.5977062021908015, sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=ar-IQ, timezone=Asia/Jayapura
[junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=266511232,total=336068608
[junit4] 2> NOTE: All tests run in this JVM: [TestBooleanScoreConsistency]
[junit4] Completed [1/1 (1!)] in 4.72s, 2 tests, 2 failures <<< FAILURES!
[junit4]
[junit4]
[junit4] Tests with failures [seed: B6DEF72C383813DE]:
[junit4] - org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithRandomIWC
[junit4] - org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC
[junit4]
[junit4]
[junit4] JVM J0: 0.40 .. 5.70 = 5.30s
[junit4] Execution time total: 5.77 sec.
[junit4] Tests summary: 1 suite, 2 tests, 2 failures
{noformat}
With this seed, the pre-merge and single-segment indexes in both tests return identical scores for all matching docs, but the Explanations (which seem to be correct AFAICT) don't match the scores...
{noformat}
[junit4] <JUnit4> says salut! Master seed: AEB2F91E0E8938CC
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(29716@localhost).
[junit4] Suite: org.apache.lucene.search.TestBooleanScoreConsistency
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithDefaultIWC -Dtests.seed=AEB2F91E0E8938CC -Dtests.slow=true -Dtests.locale=zh -Dtests.timezone=Australia/Sydney -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 4.08s | TestBooleanScoreConsistency.testScoresWithDefaultIWC <<<
[junit4] > Throwable #1: java.lang.AssertionError: 15581: score() and explain() return different values for: Office 365
[junit4] > 0.7475659 = product of:
[junit4] > 2.2426977 = sum of:
[junit4] > 2.2426977 = weight(body:365 in 15581) [ClassicSimilarity], result of:
[junit4] > 2.2426977 = score(doc=15581,freq=1.0), product of:
[junit4] > 0.5300819 = queryWeight, product of:
[junit4] > 6.769362 = idf(docFreq=53, docCount=17297)
[junit4] > 0.07830604 = queryNorm
[junit4] > 4.230851 = fieldWeight in 15581, product of:
[junit4] > 1.0 = tf(freq=1.0), with freq of:
[junit4] > 1.0 = termFreq=1.0
[junit4] > 6.769362 = idf(docFreq=53, docCount=17297)
[junit4] > 0.625 = fieldNorm(doc=15581)
[junit4] > 0.33333334 = coord(1/3)
[junit4] > expected:<0.2491886466741562> but was:<0.7475659251213074>
[junit4] > at __randomizedtesting.SeedInfo.seed([AEB2F91E0E8938CC:49474284BFA1F07]:0)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExplanations(TestBooleanScoreConsistency.java:214)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:138)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC(TestBooleanScoreConsistency.java:79)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithRandomIWC -Dtests.seed=AEB2F91E0E8938CC -Dtests.slow=true -Dtests.locale=zh -Dtests.timezone=Australia/Sydney -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 1.90s | TestBooleanScoreConsistency.testScoresWithRandomIWC <<<
[junit4] > Throwable #1: java.lang.AssertionError: 7614: score() and explain() return different values for: Microsoft Office 365
[junit4] > 1.2357022 = product of:
[junit4] > 1.8535532 = sum of:
[junit4] > 0.05939492 = weight(body:microsoft in 7614) [ClassicSimilarity], result of:
[junit4] > 0.05939492 = score(doc=7614,freq=1.0), product of:
[junit4] > 0.09644668 = queryWeight, product of:
[junit4] > 1.2316633 = idf(docFreq=13720, docCount=17297)
[junit4] > 0.07830604 = queryNorm
[junit4] > 0.6158317 = fieldWeight in 7614, product of:
[junit4] > 1.0 = tf(freq=1.0), with freq of:
[junit4] > 1.0 = termFreq=1.0
[junit4] > 1.2316633 = idf(docFreq=13720, docCount=17297)
[junit4] > 0.5 = fieldNorm(doc=7614)
[junit4] > 1.7941582 = weight(body:365 in 7614) [ClassicSimilarity], result of:
[junit4] > 1.7941582 = score(doc=7614,freq=1.0), product of:
[junit4] > 0.5300819 = queryWeight, product of:
[junit4] > 6.769362 = idf(docFreq=53, docCount=17297)
[junit4] > 0.07830604 = queryNorm
[junit4] > 3.384681 = fieldWeight in 7614, product of:
[junit4] > 1.0 = tf(freq=1.0), with freq of:
[junit4] > 1.0 = termFreq=1.0
[junit4] > 6.769362 = idf(docFreq=53, docCount=17297)
[junit4] > 0.5 = fieldNorm(doc=7614)
[junit4] > 0.6666667 = coord(2/3)
[junit4] > expected:<0.41190072894096375> but was:<1.2357021570205688>
[junit4] > at __randomizedtesting.SeedInfo.seed([AEB2F91E0E8938CC:347CE7AC4A6B1DEA]:0)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExplanations(TestBooleanScoreConsistency.java:214)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:138)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithRandomIWC(TestBooleanScoreConsistency.java:85)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {id=PostingsFormat(name=MockRandom), body=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, maxPointsInLeafNode=127, maxMBSortInHeap=5.70855157410318, sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=zh, timezone=Australia/Sydney
[junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=181104832,total=342360064
[junit4] 2> NOTE: All tests run in this JVM: [TestBooleanScoreConsistency]
[junit4] Completed [1/1 (1!)] in 6.29s, 2 tests, 2 failures <<< FAILURES!
[junit4]
[junit4]
[junit4] Tests with failures [seed: AEB2F91E0E8938CC]:
[junit4] - org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC
[junit4] - org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithRandomIWC
[junit4]
[junit4]
[junit4] JVM J0: 0.41 .. 7.13 = 6.73s
[junit4] Execution time total: 7.18 sec.
[junit4] Tests summary: 1 suite, 2 tests, 2 failures
{noformat}
With this perplexing seed (found using tests.iters) the Randomized IndexWriterConfig manages to pass all checks in the test -- getting scores consistent with the Explanation both before and after the forceMerge -- but the default IndexWriterConfig still demonstrates the problem...
{noformat}
[junit4] <JUnit4> says hallo! Master seed: 29D39C39B6047864:5BA9313A359A1831
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(29867@localhost).
[junit4] Suite: org.apache.lucene.search.TestBooleanScoreConsistency
[junit4] OK 6.16s | TestBooleanScoreConsistency.testScoresWithRandomIWC
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithDefaultIWC -Dtests.seed=29D39C39B6047864 -Dtests.slow=true -Dtests.locale=uk-UA -Dtests.timezone=Indian/Comoro -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 1.26s | TestBooleanScoreConsistency.testScoresWithDefaultIWC <<<
[junit4] > Throwable #1: java.lang.AssertionError: 7614: score doesn't match (previously returned) expected score for Microsoft Office 365 expected:<1.2357021570205688> but was:<0.41190072894096375>
[junit4] > at __randomizedtesting.SeedInfo.seed([29D39C39B6047864:5BA9313A359A1831]:0)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExpected(TestBooleanScoreConsistency.java:178)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:137)
[junit4] > at org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC(TestBooleanScoreConsistency.java:79)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62), sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=uk-UA, timezone=Indian/Comoro
[junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=285815104,total=350748672
[junit4] 2> NOTE: All tests run in this JVM: [TestBooleanScoreConsistency]
[junit4] Completed [1/1 (1!)] in 7.73s, 2 tests, 1 failure <<< FAILURES!
[junit4]
[junit4]
[junit4] Tests with failures [seed: 29D39C39B6047864:5BA9313A359A1831]:
[junit4] - org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC
[junit4]
[junit4]
[junit4] JVM J0: 0.37 .. 8.58 = 8.21s
[junit4] Execution time total: 8.62 sec.
[junit4] Tests summary: 1 suite, 2 tests, 1 failure
{noformat}
> BooleanQuery scores can be diff for same docs+sim when using coord (disagree with Explanation which doesn't change)
> -------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-7132
> URL: https://issues.apache.org/jira/browse/LUCENE-7132
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
> Affects Versions: 5.5
> Reporter: Ahmet Arslan
> Assignee: Steve Rowe
> Attachments: LUCENE-7132.patch, LUCENE-7132.patch, LUCENE-7132.patch, SOLR-8884.patch, SOLR-8884.patch, debug.xml
>
>
> Some of the folks [reported|http://find.searchhub.org/document/80666f5c3b86ddda] that sometimes explain's score can be different than the score requested by fields parameter. Interestingly, Explain's scores would create a different ranking than the original result list. This is something users experience, but it cannot be re-produced deterministically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org