search engine - Precision, recall

[Itay K <it...@gmail.com>]

Hi,

I'm trying to measure Precision and recall for a search engine which is
crawling data sources of an organization.

Are there any best practices regrading these indexes and specific
industries (e.g. for financial organizations, the recommended percentage
for precision and recall is ~60%).

Is there any best practice in general for the recommended percentage?

I read an article from 2005 regrading measured precision and recall for web
search engines but unfortunately my use case isn't a web application and I
believe that since than a lot has changed.

thanks

Re: search engine - Precision, recall

[Shawn Heisey <ap...@elyograg.org>]

On 7/27/2017 7:20 AM, Itay K wrote:
> I'm trying to measure Precision and recall for a search engine which is
> crawling data sources of an organization.
>
> Are there any best practices regrading these indexes and specific
> industries (e.g. for financial organizations, the recommended percentage
> for precision and recall is ~60%).
>
> Is there any best practice in general for the recommended percentage?
>
> I read an article from 2005 regrading measured precision and recall for web
> search engines but unfortunately my use case isn't a web application and I
> believe that since than a lot has changed.

I don't believe you can assign concrete numbers to these aspects of a
search engine, at least not in a way that has meaning after the query,
the index, or the user's expectation changes.

Recall is all about numbers, but precision is a completely subjective
measurement that is going to vary from person to person.  Results that
are highly relevant for one user might be completely irrelevant for
another, even when both users enter the exact same search terms.

Also, the search terms that one user enters are likely to be different
from the search terms that another user enters, even if they are looking
for exactly the same thing.  I cannot think of a way of calculating
percentages for precision and recall that would give meaningful numbers
when very different searches and expectations must be examined.  A
recall count for one search will have little meaning when compared to
the recall count for a different search ... and as already mentioned,
precision is COMPLETELY subjective.

IMHO, tuning precision and recall is not about getting some calculated
numbers as high as you can.  In order to tune successfully, you have to
know what people are searching for, what they expect to find, and come
up with a configuration that will produce the best balance between
precision and recall when applied to the combination of the data in the
index and what's actually being searched.  Frequently the tuning process
involves educating the users, in addition to (or sometimes instead of)
changing the search engine configuration.  Six months after tuning the
search, as the index and your users change, you may need completely
different settings to get good results.

Changes that affect precision and recall are usually a tradeoff between
those two factors.  Improving one of them will often make the other
worse.  They must be approached with a goal of bringing them into
balance for the searches done by a majority of the system users.

Thanks,
Shawn

Re: search engine - Precision, recall

[Florian Meier <me...@googlemail.com>]

Hi Itay,

in IR research there’s a long tradition (TREC and alike) for measuring the effectiveness of search engines. In this context it is measured by using a so called test collection, which consists of three things:
1. Documents 
2. Topics i.e. information needs/queries of users for these documents 
3. relevance assessment data i.e. which documents are relevant for which topics 

Using the results your search engine returns for each topic, one can calculate Precision and Recall or - depending on the context and use case the searching engine is running in - other measures might be more appropriate. For example in an enterprise context it might be the case that only one document is possible to fulfill the searchers information need. If you find this to be the case for a lot of information needs,  different measures like MRR might be more suitable. 

Moreover there’s also the question on whether you are aiming for high precision or high recall. Balancing both is a hard task and it is up to you and your users needs to find out what is more important for them. E.g. is it critical to not have certain documents? Than high recall might be your aim...

If you are aiming for high precision you could, for example, measure something like Precision@10, which means how many relevant documents are in the top 10 documents returned. For this you don’t need relevance assessment data. However, what you need in any case is an idea what your users are searching for and generate possible test queries from this and also have an idea what documents are relevant. Finally, I thinks it’s not possible to give a recommended percentage cause it depends a lot on your context.

Hope this helps,
Florian


> Am 27.07.2017 um 15:20 schrieb Itay K <it...@gmail.com>:
> 
> Hi,
> 
> I'm trying to measure Precision and recall for a search engine which is
> crawling data sources of an organization.
> 
> Are there any best practices regrading these indexes and specific
> industries (e.g. for financial organizations, the recommended percentage
> for precision and recall is ~60%).
> 
> Is there any best practice in general for the recommended percentage?
> 
> I read an article from 2005 regrading measured precision and recall for web
> search engines but unfortunately my use case isn't a web application and I
> believe that since than a lot has changed.
> 
> thanks