[jira] [Commented] (STANBOL-1104) Improve ranking for multi term OR queries over the SolrYard

["Rupert Westenthaler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STANBOL-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13680941#comment-13680941 ] 

Rupert Westenthaler commented on STANBOL-1104:
----------------------------------------------

Further investigations revealed the following:

(1) The issue with the ranking of the results of the EntityLinkingEngine is that some Entities had matching labels in both the language of the text as well as the fallback language. Other only in one of the two. As the EntityLinkingEngine perfoms queries like 

    {lang1}:"{term1}" OR {lang1}:"{term2}" OR {lang2}:"{term1}" OR {lang2}:"{term2}" 

there are two possibilities for two query terms to match

 (a) both {term1} and {term2} do match in the same language
 (b) a single term matches in {lang1} and {lang2}

Entities that where matching like (b) where causing the impression that the ranking was flawed, but in fact it was not! This is an issue that needs to be solved for the EntityLinker (or the EntitySearcher implementation). There is nothing wrong with the ranking of the query results.



(2) The implementation of the "Term Proximity" performed very well

Test with the Enhancer Stress Test Tool and a Enhancement Chain linking against the freebase.com SolrIndex (~40million entities, ~50GByte on disc) showed no performance deficit when activating this feature (what was surprising).

This approach also provides the expected results for the queries as used by the EntityLinking engine as it create queries like 

    {lang1}:"{term1}" OR {lang1}:"{term2}" OR {lang1}:"{term1} {term2}"~9 OR {lang2}:"{term1}" OR {lang2}:"{term2}" OR {lang2}:"{term1} {term2}"~9

and this query penalizes Entities that match like described in (1.b) against those that match like described in (1.a)



(3) The use of query time boost showed (as expected) no influence on the performance. However using a boost had no influence on the ranking of the results, because both variants (1.a) and (1.b) are boosted in the same way. However this tells that the ranking is mainly influenced by the number of matching query terms and not by the index time document boosts. 

                
> Improve ranking for multi term OR queries over the SolrYard
> -----------------------------------------------------------
>
>                 Key: STANBOL-1104
>                 URL: https://issues.apache.org/jira/browse/STANBOL-1104
>             Project: Stanbol
>          Issue Type: Improvement
>          Components: Entityhub
>            Reporter: Rupert Westenthaler
>            Assignee: Rupert Westenthaler
>
> Test for EntityLinking against big vocabularies (e.g. Freebase with about 40 million entities) have shown that the currently used Solr Queries for multi-token OR queries do not always give the expected ranking of the results because of the following reasons:
> ReferencedSite do use Entity rankings (implemented as index time Document boosts). Those rankings do have an impact on the rankings of query results. On the positive side those rankings ensure that a query for Paris should give Paris, France before Paris, Texas. On the negative for a query for two tokens (e.g. two given names) it might happen that other entities with only one of those terms (e.g. very famous person with one of the two requested given names) are ranked before entities with a lower ranking that do match both terms.
> This is even more likely for terms that are very common in the index, as normalization will reduce the boost for entities with such a term - resulting in the document boost to have an even higher impact.
> The described behavior is especially a problem for the EntityLinkingEngine as its uses exactly such kind of "{term1} OR {term2}" queries to lookup Entities. 
> Possible Solutions include:
> * disable the use of index time document boosts: However this would have a negative impact on every day searches (e.g. for Paris) and is therefore not an option within most scenarios.
> * increase the number of selected entities for the EntityLinkingEngine: currently max(10,2*maxSuggestion) entities are retrieved. Increasing this value would make the engine more resistant to unexpected rankings. However (1) it does not solve (but workaround) the problem; (2) some tests have shown that even increasing the value to 50 does not include the expected result (using the freebase.com index as dataset).
> * explicitly adding a "{token1} {token2}" query term in the EntityLinkingEngine to queries for "{term1}" OR "{term2}". However this would only boost entities where {token1} and {token2} would be in exact that order. Entities containing "{token2} {token1}" or "{token1} {other} {token2} would not get any boost. So this solution will only improve rankings for those cases where the label would also match an AND connected query. 
> * the use of a "Term Proximity" as suggested by [1]:  This ensures that (1) Entities that do only match one of the parsed terms will get no boost from this part of the query, (2) even for entities that match several/all terms the ranking will get improved as the distance within the text will be considered for calculating the ranking. As phrase queries are more complicated to answer it is expected that this will have an impact on the performance.
> * Using a high query time boost for multi term OR queries as suggested by [2]. This would allow to increase the boost given to entities containing {token1} and {token2} and therefore reduce the influence of the index time document boost used to represent the entity ranking. The advantage is that this will not have any performance implications (as it only influences the ranking computation and does not make the query more complex). 
> [1] http://wiki.apache.org/solr/SolrRelevancyCookbook#Term_Proximity
> [2] http://wiki.apache.org/solr/SolrRelevancyCookbook#Ranking_Terms

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira