You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Mike Klaas (JIRA)" <ji...@apache.org> on 2006/07/07 02:10:32 UTC

[jira] Updated: (SOLR-24) Add Highlighting to standard request handler

     [ http://issues.apache.org/jira/browse/SOLR-24?page=all ]

Mike Klaas updated SOLR-24:
---------------------------

    Attachment: highlight_patch_v2.diff

Updated highlighting patch.

Numerous improvements:
- uses three parameters: highlight (true/false) highlightFields maxSnippets
- vastly improved defaultField handling (old version ignored some cases of specified default fields, and did the wrong thing entirely for DisMax)
- vastly improved DisMax query handling

Open issues: 
- I needed to subclass a final class in the highlighter contrib package in lucene, which required C&P.  This is a stop-gap measure until the issue is fixed in lucene.
- The interaction between WordDelimiterFilter and Highlighter is not fixed.  This will require further investigation.

I believe the patch is in a commitable state.  If it is committed, I'll open JIRA issues for the open problems above. 

> Add Highlighting to standard request handler
> --------------------------------------------
>
>          Key: SOLR-24
>          URL: http://issues.apache.org/jira/browse/SOLR-24
>      Project: Solr
>         Type: New Feature

>   Components: search
>     Reporter: Mike Klaas
>  Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils.  I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class.  Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values.  (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful.  This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
>    highlight: list of fields to highlight, or highlight the default field if at all present
>   maxSnippets: maximum number of snippets to return for each field
>    highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags.  There are issues with this approach, but are mitigated with the ability to specify a custom Formatter.  Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request handler

Posted by Mike Klaas <mi...@gmail.com>.
On 7/10/06, Chris Hostetter <ho...@fucit.org> wrote:

> excelent point ... my only followup suggestion to my orriginal idea is
> that thanks to the schema, you could acctually check which of the
> returned fields are analyzed/text and highlight those -- but i throw that
> idea out only to see what you think of it ... at the moment you've
> confinved be that the "query fields" are the way to go.

I'd be happy with that approach as well.  I'm going to leave it as
query fields for this patch, and the results of this discussion can be
implemented as an additional patch.  Another option is to define
default highlit fields in the schema.  Perhaps it is moot since most
people will specify the fields they desire.  Does anyone else have
comments?

-Mike

Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request handler

Posted by Chris Hostetter <ho...@fucit.org>.
: > 5) perhaps a better way to pick the "default" fields if highlightFields
: > isn't specified would be using the return fields being used (the "fl"
: > param) rather then the default search field in StandardRequestHandler and
: > the "qf" fields in DisMaxHandler ?
:
: Tempting, but I think that using qf/df provides slightly more natural
: default behaviour.  The returned field list often has many
: non-analyzable or non-text fields which we don't necessarily want to
: summarize, while the query fields are guaranteed to at least
: potentially contain the highlighted term (as we are searching them).
: At least in my use cases, highlighting is a cue that a search has
: found a term, hence it only makes sense to use in searched fields.

excelent point ... my only followup suggestion to my orriginal idea is
that thanks to the schema, you could acctually check which of the
returned fields are analyzed/text and highlight those -- but i throw that
idea out only to see what you think of it ... at the moment you've
confinved be that the "query fields" are the way to go.


-Hoss


Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request handler

Posted by Mike Klaas <mi...@gmail.com>.
A brief follow-up:

On 7/6/06, Chris Hostetter <ho...@fucit.org> wrote:
> 3) using the example app with this patch, i tried both of these...

> ...on my port, i get no highlighting for any doc, even though in both
> cases there are docs in the result that have the simple term "video" in
> the features field ... am i doing something wrong, or is this a bug?

Looking at this, it seems that implementation will only highlight the
first segment of a multi-valued field.  Highlighter behaves similarly

> 4) if you open a LUCENE-XXXX issue for the QueryTermExtractor issue with
> DisjunctionMaxQueries, and attach a patch with testcases, and no one
> objects, i'll commit it :) ... but in the mean time i'm curous what it is
> about QueryTermExtractor that doesn't work -- it looks like it's fallback
> behavior is to use "query.extractTerms" and that should do the same thing
> as your getTermsFromDisjunctionMaxQuery right?

I see the issue now.  Since the first version of the patch and the
current solr trunk, solr has been updated to use a newer Lucene (which
contains the extractTerms fallback).  The old codebase required the
modification.

> 5) perhaps a better way to pick the "default" fields if highlightFields
> isn't specified would be using the return fields being used (the "fl"
> param) rather then the default search field in StandardRequestHandler and
> the "qf" fields in DisMaxHandler ?

Tempting, but I think that using qf/df provides slightly more natural
default behaviour.  The returned field list often has many
non-analyzable or non-text fields which we don't necessarily want to
summarize, while the query fields are guaranteed to at least
potentially contain the highlighted term (as we are searching them).
At least in my use cases, highlighting is a cue that a search has
found a term, hence it only makes sense to use in searched fields.

I'll take a look at supporting multi-valued fields.  It's something
that I would be keen to work on as a further refinement to the base
patch instead of being part of this one, if possible.

cheers,
-Mike

Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request handler

Posted by Mike Klaas <mi...@gmail.com>.
On 7/6/06, Chris Hostetter <ho...@fucit.org> wrote:
>
> Wow, i'd *just* finished reading the last patch completely through, and
> now there's a new one ... fortunately i'm familiar with some of these
> changes (the day someone writes a really good "diff differ" i'll be a very
> happy man)

Hopefully any further changes will be minor!  I  appreciate the
feedback, as I'm relatively new to both solr and lucene.

> A couple of issues that jump out at me (in the new patch)...
>
> 1) you changed the SolrPluginUtils.splitList from ",| " to "," ... this
> eliminates the ability to seperate fields with spaces in query params,
> which i know is fairly common -- was there a particular reason for
> that?

Purely a (shocking) typo.  Thanks.

> 2) if a field doesn't generate any snippets, it's still included in the
> output for that doc ... is this intentional?

By the principle of least surprise, I left in empty fields as it
guaranteed that you received what you asked for.  It makes sense to
drop the field completely if it doesn't exist in the document, though,
or if the value is empty.  It is a minor change if there is a
preference.

> 3) using the example app with this patch, i tried both of these...
>
> http://localhost:8983/solr/select/?indent=on&q=video&fl=id,features&highlight=true&highlightFields=features
> http://localhost:8983/solr/select/?indent=on&q=features:video&fl=id,features&highlight=true&highlightFields=features
>
> ...on my port, i get no highlighting for any doc, even though in both
> cases there are docs in the result that have the simple term "video" in
> the features field ... am i doing something wrong, or is this a bug?

I'll check on this and get back to you.

> 4) if you open a LUCENE-XXXX issue for the QueryTermExtractor issue with
> DisjunctionMaxQueries, and attach a patch with testcases, and no one
> objects, i'll commit it :) ... but in the mean time i'm curous what it is
> about QueryTermExtractor that doesn't work -- it looks like it's fallback
> behavior is to use "query.extractTerms" and that should do the same thing
> as your getTermsFromDisjunctionMaxQuery right?

I'm not sure that I follow.  For DisMax, you could have a
DisjuntionMaxQuery inside a Boolean Query passed to
QueryTermExtractor.  This would eventually make its way to the private
static getTerms method, which switches on the query Class, and does
nothign if the Query is not recognized.  No?

> 5)-7)

I agree completely.

Thanks for your comments, Hoss, I'll get back to you tomorrow.

-Mike

Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request handler

Posted by Chris Hostetter <ho...@fucit.org>.
Wow, i'd *just* finished reading the last patch completely through, and
now there's a new one ... fortunately i'm familiar with some of these
changes (the day someone writes a really good "diff differ" i'll be a very
happy man)


A couple of issues that jump out at me (in the new patch)...

1) you changed the SolrPluginUtils.splitList from ",| " to "," ... this
eliminates the ability to seperate fields with spaces in query params,
which i know is fairly common -- was there a particular reason for
that?

2) if a field doesn't generate any snippets, it's still included in the
output for that doc ... is this intentional?

3) using the example app with this patch, i tried both of these...

http://localhost:8983/solr/select/?indent=on&q=video&fl=id,features&highlight=true&highlightFields=features
http://localhost:8983/solr/select/?indent=on&q=features:video&fl=id,features&highlight=true&highlightFields=features

...on my port, i get no highlighting for any doc, even though in both
cases there are docs in the result that have the simple term "video" in
the features field ... am i doing something wrong, or is this a bug?

4) if you open a LUCENE-XXXX issue for the QueryTermExtractor issue with
DisjunctionMaxQueries, and attach a patch with testcases, and no one
objects, i'll commit it :) ... but in the mean time i'm curous what it is
about QueryTermExtractor that doesn't work -- it looks like it's fallback
behavior is to use "query.extractTerms" and that should do the same thing
as your getTermsFromDisjunctionMaxQuery right?

5) perhaps a better way to pick the "default" fields if highlightFields
isn't specified would be using the return fields being used (the "fl"
param) rather then the default search field in StandardRequestHandler and
the "qf" fields in DisMaxHandler ?

6) one thing to watch out for is schemas with no uniqueKey defined.
currently that produces highlighting like this...
  <lst name="highlighting">
    <lst name="id=null"> ...
...that would probably be better with no name.

7) there is at least one additional place to StandardRequestHandler could
be refactored to use SolrPluginUtils -- that would be setReturnFields
(which i discovered when i saw StandardRequestHandler still has the old
splitList value)


#5-7 above aren't issues, just comments, but if you can clarify #1-4 i'll
happily commit this patch.



-Hoss