You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Mike Klaas (JIRA)" <ji...@apache.org> on 2006/06/23 05:39:29 UTC
[jira] Created: (SOLR-24) Add Highlighting to standard request
handler
Add Highlighting to standard request handler
--------------------------------------------
Key: SOLR-24
URL: http://issues.apache.org/jira/browse/SOLR-24
Project: Solr
Type: New Feature
Components: search
Reporter: Mike Klaas
This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
-----------------------------------
Refactoring StandardRequestHandler:
1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
--------------------------------------
Highlighting:
1. Highlighting is controlled by three request parameters:
highlight: list of fields to highlight, or highlight the default field if at all present
maxSnippets: maximum number of snippets to return for each field
highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Created: (SOLR-24) Add Highlighting to standard request
handler
Posted by Chris Hostetter <ho...@fucit.org>.
: possible Solr's development. Note that there is already at least one
: parameter where presense/absence is significant (debugQuery).
hmmm, yeah ... that's becuase it was designed with a checkbox in mind --
but it breaks if you try to use it with other HTML form elements. Maybe
we should change it too.
: Significant whitespace scares me <g>. If presence/absense of a param
: being significant is problematic, it is better to simply disallow this
: Solr-wide. Mandate that value-less parameters are equivalent to not
: specifying them at all. Things like the highlight parameter can
: accept 'true' as a hint to use the default field for these types of
: uses, or it can be broken down into two parameters
: (highlight=true/false, highlightFields=...).
Good point about whitespace, and more specificly regarding highlighting
you may want to make the change to two params independent of other issues
... if i understand you correctly, you are currently using one param to
indicate both the activation of highlighting and the field to use if the
default field is not desired (empty string meaning use the default field).
This is less flexible hten the two field appraoch, because it prevents the
client from ever being able to say "wether or not highlighting should be
used is up to the server's config, but if highlighting is going to be
used, i'd like this field to be highlighted"
-Hoss
Re: [jira] Created: (SOLR-24) Add Highlighting to standard request handler
Posted by Mike Klaas <mi...@gmail.com>.
On 6/23/06, Chris Hostetter <ho...@fucit.org> wrote:
>
> Mike, I've only had a chance to skim your patch so far, but on the
> surface it looks very good to me. One little thing scares me a little
> bit...
>
> : 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
>
> ...in principal, I agree with the sentiment that "" should be a legal
> value, but the Utils class was implimented this way to support URLs
> generated by HTML form submission, where the param is still included in
> the URL but with an empty value. Changing this would break at least one
> or two Solr uses I know of.
I thought that this might be problematic which is why I made sure to
highlight it. Definitely an issue that should be resolved as early as
possible Solr's development. Note that there is already at least one
parameter where presense/absence is significant (debugQuery).
> Then again, if we want to change it, now would be better then latter
> ... hmmm. Perhaps an alternate approach might be to continue to use the
> default when the param is empty (""), but stop trimming the values before
> the comparison, so that " " is a legal value that prevents the default
> form being used?
Significant whitespace scares me <g>. If presence/absense of a param
being significant is problematic, it is better to simply disallow this
Solr-wide. Mandate that value-less parameters are equivalent to not
specifying them at all. Things like the highlight parameter can
accept 'true' as a hint to use the default field for these types of
uses, or it can be broken down into two parameters
(highlight=true/false, highlightFields=...).
> I'm eager to see what other people on the list (who know more about
> highlighting and are interested in using highlighting with Solr) think
> about your patch ... alas, Erik seems to be quite busy right now, maybe he
> can share some thoughts next week? (fingers crossed)
Sounds good--I'm busy for the next week regardless.
Thanks for your comments!
-Mike
Re: [jira] Created: (SOLR-24) Add Highlighting to standard request
handler
Posted by Chris Hostetter <ho...@fucit.org>.
Mike, I've only had a chance to skim your patch so far, but on the
surface it looks very good to me. One little thing scares me a little
bit...
: 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
...in principal, I agree with the sentiment that "" should be a legal
value, but the Utils class was implimented this way to support URLs
generated by HTML form submission, where the param is still included in
the URL but with an empty value. Changing this would break at least one
or two Solr uses I know of.
Then again, if we want to change it, now would be better then latter
... hmmm. Perhaps an alternate approach might be to continue to use the
default when the param is empty (""), but stop trimming the values before
the comparison, so that " " is a legal value that prevents the default
form being used?
...this is of course a minor bit of paranoia on my part, and
should not detract from the coolness of the rest of the patch.
I'm eager to see what other people on the list (who know more about
highlighting and are interested in using highlighting with Solr) think
about your patch ... alas, Erik seems to be quite busy right now, maybe he
can share some thoughts next week? (fingers crossed)
-Hoss
Re: [jira] Resolved: (SOLR-24) Add Highlighting to standard request
handler
Posted by Chris Hostetter <ho...@fucit.org>.
Thanks for submitting such a great patch mike -- not just the
highlighting, but refactoring StandardRequestHandler to take advantage of
the utilities too. Sweet.
As a side note...
: I wish there was an svn patch command that would do that for me so I
: wouldn't forget...
: I guess I should get used to running "svn status" after a commit and
: see if anything sticks out.
...this is one of the reasosn the "example" directory bugs me. I like to
run "svn status" before i commit anything, and make sure there aren't any
"?" ... but with the example directory containing both commited files and
also temporary build files, it's not easily managed with svn:ignore so a
lot of "?" show up and it's easy to miss seeing the significant ones in
the noise.
...allthough, i guess we could put an svn:ignore "*" pattern on
example/solr/bin and that would solve most of hte problem. thoughts?
-Hoss
Re: [jira] Resolved: (SOLR-24) Add Highlighting to standard request handler
Posted by Yonik Seeley <ys...@gmail.com>.
On 7/13/06, Mike Klaas <mi...@gmail.com> wrote:
> There is a small problem with your commit: HighlighterTest,
> CommonParams, DisMaxParams were not svn add'd to the repository.
Got it... thanks!
I wish there was an svn patch command that would do that for me so I
wouldn't forget...
I guess I should get used to running "svn status" after a commit and
see if anything sticks out.
-Yonik
Re: [jira] Resolved: (SOLR-24) Add Highlighting to standard request handler
Posted by Mike Klaas <mi...@gmail.com>.
On 7/13/06, Yonik Seeley (JIRA) <ji...@apache.org> wrote:
> I just committed this, and opened a lucene bug
> http://issues.apache.org/jira/browse/LUCENE-627
>
> Now we just need to update the wiki!
Great! I'll update the wiki in the next day or so.
There is a small problem with your commit: HighlighterTest,
CommonParams, DisMaxParams were not svn add'd to the repository.
Thanks, Yonik,
-Mike
[jira] Updated: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=all ]
Mike Klaas updated SOLR-24:
---------------------------
Attachment: highlight_patch_v4.diff
Updated patch reflecting comments:
- splitList typo fixed
- fields producing no highlight snippets are not included in result tag
- full support for multi-valued fields, this includes a mutl-value token stream and TextFragmenter which starts fragments on value boundaries
- use of SolrQueryTermExtractor removed (not needed with current version of lucene)
- "id=key" attributes are stripped if the schema contains no uniqueKey
- Factored out response field setting from StandardRequestHandler (note, this is a slight semantic change, as previously the score was only included if "score" was sent in the field list. The current behavious matched DisMaxRequestHandler, but not the documentation. If this behaviour is desirable, both handlers should be modified to comply).
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419367 ]
Hoss Man commented on SOLR-24:
------------------------------
Bill: I beleive what you are seeing is an issue with the Highlighter not playing nice with Analyzers that produce multiple tokens in the same position -- Mike mentioend this in the email discussion, but i assumed it was reagarding using TermPosition information ... in the example app "name" is a stored field, so i'm not sure why the indexing Analyzer would affect highlighting (perhaps it's yet anohter aspect of the highlighter package i don't understand)
Incidently, We might as well capture here a summary of the dicussion from last week...
http://www.nabble.com/-jira--Created%3A-%28SOLR-24%29-Add-Highlighting-to-standard-request-handler-tf1834206.html
1) patch generally look good
2) I'm hesitent to change SolrPluginUtils.getParam()
In this particular case (using the "highlight" param to indicate both that highlighting should be turned on, and to override the default highlighting fields) I think a better API would be two seperate params, and the more i think about it the more i like that solution in general -- much as Mike pointed out that "Significant whitespace scares me" in response to truely ridiculous suggestion i made, significant *empty* space scares me ... so i would vote for changing that and then commiting.
Incidently, i got a response from Erik H giving this patch his thumbs up a few days ago, but looking at the thread now i realize it came staight to me and not the list.
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=all ]
Mike Klaas updated SOLR-24:
---------------------------
Attachment: highlight_patch_v1.diff
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Bill Au (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419345 ]
Bill Au commented on SOLR-24:
-----------------------------
I have come across a test case there the patch doesn't work. I am using the data in example/exampledocs/ipod_other.xml.
After adding the two docs I queried solr using curl:
curl -i "http://localhost:8983/solr/select/?q=ipod&highlight=name&indent=true"
I got the following highlighting data in the response:
<lst name="highlighting">
<lst name="id=IW-02">
<arr name="name">
<str>i<em>iPod</em> & i<em>iPod</em> Mini USB 22.0 Cable</str>
</arr>
</lst>
<lst name="id=F8V7067-APL-KIT">
<arr name="name">
<str>Belkin Mobile Power Cord for i<em>iPod</em> w/ Dock</str>
</arr>
</lst>
</lst>
In both cases, there is an extra i in front of <em>iPod</em>.
My guess is there is something wrong with the TokenStream passed to the Highligher. I am taking a closing look at that now.
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419878 ]
Yonik Seeley commented on SOLR-24:
----------------------------------
I just got a chance to check this out... it's looking good Mike! I think this is something that will be useful to a lot of people.
Perhaps highlight=true could be overloaded later to provide alternate ways of highlighting (giving positions rather than inserting <em> in the values). While <em> is a little flawed since it's problematic in field values that may already contain markup, I think it's probably the best/simplest default that people can use easily.
Regarding the <lst name="id=theuniqueid">... should this be <lst name="theuinqueid"> without including the field name?
Including the unique key field name (which I assume id is), does add a little more info, at the cost of making it harder for clients to use the value.
I like that you leave in empty entries like <lst name="id=VS1GB400C3"/> for documents that there are no highlighter fragments for. This makes it possible to access the highlight info by position rather than relying on id correlation.
I wasn't able to get multi-valued fields working though. Even if multiple values matched, I would only get a single <str> back in the highlight section.
Example with the demo document set:
http://localhost:8983/solr/select?q=features:(USB+video+color+includes+MB+memory+display+photo)&indent=on&highlight=true&highlightFields=features
<lst name="highlighting">
<lst name="id=MA147LL/A">
<arr name="features">
<str>Stores up to 1515,000 songs, 2525,000 <em>photos</em>, or 150 hours of <em>video</em></str>
</arr>
</lst>
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419885 ]
Mike Klaas commented on SOLR-24:
--------------------------------
I also agree about removing the "id=" (I was just blithly following the example of the debug data here).
Another note to reviewers: in the process of Gap-enabled the text fragmenter, I introduced a bug that caused over-long fragments in regular fields. This has been fix on my local copy.
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=all ]
Mike Klaas updated SOLR-24:
---------------------------
Attachment: highlight_patch_v3.diff
The previous patch didn't apply cleanly to trunk due to svn keyword. This version omits a docstring change.
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419995 ]
Mike Klaas commented on SOLR-24:
--------------------------------
> This was done because there is an optimization that can use a filter (the set of all documents matching a query) to satisfy > a sorted query if scores aren't needed. This bypasses re-executing the query.
I'll ensure this skippage is possible with StandardrequestHandler. It should probably be added to DisMax too, in that case.
> Regarding gaps... I can see how one would need to rely on a position gap when using term-vectors... but when > > re-analyzing stored fields, they are already discrete. Is the problem caused by the hilighter architecture (I haven't used it >before)?
Highlighter takes a tokenstream and a piece of text and fragments it. The fragments are scored to determined which top set of them to return. The only way for it to work on multiple token streams is to invoke it multiple times (in which case we have to find a way of merging the highlighting output from each in a nice way), or fooling highlighter into thinking it is a signle stream (ensuring separation among the various parts), which is attractive since Highlighter compares the fragments from all the parts and picks the globally highest scoring fragments.
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419398 ]
Mike Klaas commented on SOLR-24:
--------------------------------
Just to follow up, I'll be creating an new patch which implements the changes in highlighting parameters (this should be ready in a few days).
The issue with the doubled tokens only occurs when analyzing a stored field for highlighting (it does not occur when using Term Position vectors). Stored fields need to be analyzed on the fly to do highlighting matching to the query term.
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by Chris Hostetter <ho...@fucit.org>.
: > - Factored out response field setting from StandardRequestHandler (note, this is a slight semantic change, as previously
: > the score was only included if "score" was sent in the field list.
:
: This was done because there is an optimization that can use a filter
: (the set of all documents matching a query) to satisfy a sorted query if
: scores aren't needed. This bypasses re-executing the query.
yeah .. i'd forgotten about that ... off the cuff we can probably deal
with this by using the utility method to set field list on the response,
and then before executing the search, check if "score" is in
SolrQueryResponse.getReturnFields() and if it is set the flag
-Hoss
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419886 ]
Yonik Seeley commented on SOLR-24:
----------------------------------
> - Factored out response field setting from StandardRequestHandler (note, this is a slight semantic change, as previously
> the score was only included if "score" was sent in the field list.
This was done because there is an optimization that can use a filter (the set of all documents matching a query) to satisfy a sorted query if scores aren't needed. This bypasses re-executing the query.
> I also agree about removing the "id=" (I was just blithly following the example of the debug data here).
Yeah, I really only meant for the debug stuff to be human-readable (and more likely to change in the future).
Regarding gaps... I can see how one would need to rely on a position gap when using term-vectors... but when re-analyzing stored fields, they are already discrete. Is the problem caused by the hilighter architecture (I haven't used it before)?
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=all ]
Yonik Seeley resolved SOLR-24:
------------------------------
Resolution: Fixed
Assign To: Yonik Seeley
I just committed this, and opened a lucene bug
http://issues.apache.org/jira/browse/LUCENE-627
Now we just need to update the wiki!
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Assignee: Yonik Seeley
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff, highlight_v5.patch
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Bill Au (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419559 ]
Bill Au commented on SOLR-24:
-----------------------------
The TokenStream looks OK. It contains "i", "pod", and "ipod".
I also tried using different query terms.
q=pod produces the same output as q=ipod.
q=iPod produces output with the extra "i" highlighted as well, ie
<lst name="highlighting">
<lst name="id=IW-02">
<arr name="name">
<str><em>i</em><em>iPod</em> & <em>i</em><em>iPod</em> Mini USB 22.0 Cable</str>
</arr>
</lst>
<lst name="id=F8V7067-APL-KIT">
<arr name="name">
<str>Belkin Mobile Power Cord for <em>i</em><em>iPod</em> w/ Dock</str>
</arr>
</lst>
</lst>
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419883 ]
Mike Klaas commented on SOLR-24:
--------------------------------
Interesting... Highlighter by default merges contiguous fragments (and fragments can be contiguous regardless of the gap between them).
I made a change to disable fragment merging and the bahaviour is more natural. I'll post a modified patch once we determine what othe rmodifications are necessary.
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12420892 ]
Yonik Seeley commented on SOLR-24:
----------------------------------
+1 for committing this.
Excellent job Mike!
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff, highlight_v5.patch
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419879 ]
Yonik Seeley commented on SOLR-24:
----------------------------------
> I wasn't able to get multi-valued fields working though.
As I started browsing the code I ran across the maxSnippets param and cranked it up... now all the snippets appear as a single string rather than multiple strings in the array. Am I doing something wrong?
Example:
<lst name="id=MA147LL/A">
<arr name="features">
<str>
iiTunes, Podcasts, AudiobooksStores up to 1515,000 songs, 2525,000 <em>photos</em>, or 150 hours of <em>video</em>22.5-inch, 320x240 <em>color</em> TFT LCD <em>display</em> with LED backlightUp to 20 hours of battery lifePlays AAC, MP3, WAV, AIFF, Audible, Apple Lossless, H.264 <em>video</em>Notes, Calendar, Phone book, Hold button, Date <em>display</em>, <em>Photo</em> wallet, BuiltBuilt-in games, JPEG <em>photo</em> playback, Upgradeable firmware, <em>USB</em> 22.0 compatibility, Playback speed control, Rechargeable capability, Battery level indication
</str>
</arr>
</lst>
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request handler
Posted by Mike Klaas <mi...@gmail.com>.
On 7/10/06, Chris Hostetter <ho...@fucit.org> wrote:
> excelent point ... my only followup suggestion to my orriginal idea is
> that thanks to the schema, you could acctually check which of the
> returned fields are analyzed/text and highlight those -- but i throw that
> idea out only to see what you think of it ... at the moment you've
> confinved be that the "query fields" are the way to go.
I'd be happy with that approach as well. I'm going to leave it as
query fields for this patch, and the results of this discussion can be
implemented as an additional patch. Another option is to define
default highlit fields in the schema. Perhaps it is moot since most
people will specify the fields they desire. Does anyone else have
comments?
-Mike
Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request
handler
Posted by Chris Hostetter <ho...@fucit.org>.
: > 5) perhaps a better way to pick the "default" fields if highlightFields
: > isn't specified would be using the return fields being used (the "fl"
: > param) rather then the default search field in StandardRequestHandler and
: > the "qf" fields in DisMaxHandler ?
:
: Tempting, but I think that using qf/df provides slightly more natural
: default behaviour. The returned field list often has many
: non-analyzable or non-text fields which we don't necessarily want to
: summarize, while the query fields are guaranteed to at least
: potentially contain the highlighted term (as we are searching them).
: At least in my use cases, highlighting is a cue that a search has
: found a term, hence it only makes sense to use in searched fields.
excelent point ... my only followup suggestion to my orriginal idea is
that thanks to the schema, you could acctually check which of the
returned fields are analyzed/text and highlight those -- but i throw that
idea out only to see what you think of it ... at the moment you've
confinved be that the "query fields" are the way to go.
-Hoss
Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request handler
Posted by Mike Klaas <mi...@gmail.com>.
A brief follow-up:
On 7/6/06, Chris Hostetter <ho...@fucit.org> wrote:
> 3) using the example app with this patch, i tried both of these...
> ...on my port, i get no highlighting for any doc, even though in both
> cases there are docs in the result that have the simple term "video" in
> the features field ... am i doing something wrong, or is this a bug?
Looking at this, it seems that implementation will only highlight the
first segment of a multi-valued field. Highlighter behaves similarly
> 4) if you open a LUCENE-XXXX issue for the QueryTermExtractor issue with
> DisjunctionMaxQueries, and attach a patch with testcases, and no one
> objects, i'll commit it :) ... but in the mean time i'm curous what it is
> about QueryTermExtractor that doesn't work -- it looks like it's fallback
> behavior is to use "query.extractTerms" and that should do the same thing
> as your getTermsFromDisjunctionMaxQuery right?
I see the issue now. Since the first version of the patch and the
current solr trunk, solr has been updated to use a newer Lucene (which
contains the extractTerms fallback). The old codebase required the
modification.
> 5) perhaps a better way to pick the "default" fields if highlightFields
> isn't specified would be using the return fields being used (the "fl"
> param) rather then the default search field in StandardRequestHandler and
> the "qf" fields in DisMaxHandler ?
Tempting, but I think that using qf/df provides slightly more natural
default behaviour. The returned field list often has many
non-analyzable or non-text fields which we don't necessarily want to
summarize, while the query fields are guaranteed to at least
potentially contain the highlighted term (as we are searching them).
At least in my use cases, highlighting is a cue that a search has
found a term, hence it only makes sense to use in searched fields.
I'll take a look at supporting multi-valued fields. It's something
that I would be keen to work on as a further refinement to the base
patch instead of being part of this one, if possible.
cheers,
-Mike
Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request handler
Posted by Mike Klaas <mi...@gmail.com>.
On 7/6/06, Chris Hostetter <ho...@fucit.org> wrote:
>
> Wow, i'd *just* finished reading the last patch completely through, and
> now there's a new one ... fortunately i'm familiar with some of these
> changes (the day someone writes a really good "diff differ" i'll be a very
> happy man)
Hopefully any further changes will be minor! I appreciate the
feedback, as I'm relatively new to both solr and lucene.
> A couple of issues that jump out at me (in the new patch)...
>
> 1) you changed the SolrPluginUtils.splitList from ",| " to "," ... this
> eliminates the ability to seperate fields with spaces in query params,
> which i know is fairly common -- was there a particular reason for
> that?
Purely a (shocking) typo. Thanks.
> 2) if a field doesn't generate any snippets, it's still included in the
> output for that doc ... is this intentional?
By the principle of least surprise, I left in empty fields as it
guaranteed that you received what you asked for. It makes sense to
drop the field completely if it doesn't exist in the document, though,
or if the value is empty. It is a minor change if there is a
preference.
> 3) using the example app with this patch, i tried both of these...
>
> http://localhost:8983/solr/select/?indent=on&q=video&fl=id,features&highlight=true&highlightFields=features
> http://localhost:8983/solr/select/?indent=on&q=features:video&fl=id,features&highlight=true&highlightFields=features
>
> ...on my port, i get no highlighting for any doc, even though in both
> cases there are docs in the result that have the simple term "video" in
> the features field ... am i doing something wrong, or is this a bug?
I'll check on this and get back to you.
> 4) if you open a LUCENE-XXXX issue for the QueryTermExtractor issue with
> DisjunctionMaxQueries, and attach a patch with testcases, and no one
> objects, i'll commit it :) ... but in the mean time i'm curous what it is
> about QueryTermExtractor that doesn't work -- it looks like it's fallback
> behavior is to use "query.extractTerms" and that should do the same thing
> as your getTermsFromDisjunctionMaxQuery right?
I'm not sure that I follow. For DisMax, you could have a
DisjuntionMaxQuery inside a Boolean Query passed to
QueryTermExtractor. This would eventually make its way to the private
static getTerms method, which switches on the query Class, and does
nothign if the Query is not recognized. No?
> 5)-7)
I agree completely.
Thanks for your comments, Hoss, I'll get back to you tomorrow.
-Mike
Re: [jira] Updated: (SOLR-24) Add Highlighting to standard request
handler
Posted by Chris Hostetter <ho...@fucit.org>.
Wow, i'd *just* finished reading the last patch completely through, and
now there's a new one ... fortunately i'm familiar with some of these
changes (the day someone writes a really good "diff differ" i'll be a very
happy man)
A couple of issues that jump out at me (in the new patch)...
1) you changed the SolrPluginUtils.splitList from ",| " to "," ... this
eliminates the ability to seperate fields with spaces in query params,
which i know is fairly common -- was there a particular reason for
that?
2) if a field doesn't generate any snippets, it's still included in the
output for that doc ... is this intentional?
3) using the example app with this patch, i tried both of these...
http://localhost:8983/solr/select/?indent=on&q=video&fl=id,features&highlight=true&highlightFields=features
http://localhost:8983/solr/select/?indent=on&q=features:video&fl=id,features&highlight=true&highlightFields=features
...on my port, i get no highlighting for any doc, even though in both
cases there are docs in the result that have the simple term "video" in
the features field ... am i doing something wrong, or is this a bug?
4) if you open a LUCENE-XXXX issue for the QueryTermExtractor issue with
DisjunctionMaxQueries, and attach a patch with testcases, and no one
objects, i'll commit it :) ... but in the mean time i'm curous what it is
about QueryTermExtractor that doesn't work -- it looks like it's fallback
behavior is to use "query.extractTerms" and that should do the same thing
as your getTermsFromDisjunctionMaxQuery right?
5) perhaps a better way to pick the "default" fields if highlightFields
isn't specified would be using the return fields being used (the "fl"
param) rather then the default search field in StandardRequestHandler and
the "qf" fields in DisMaxHandler ?
6) one thing to watch out for is schemas with no uniqueKey defined.
currently that produces highlighting like this...
<lst name="highlighting">
<lst name="id=null"> ...
...that would probably be better with no name.
7) there is at least one additional place to StandardRequestHandler could
be refactored to use SolrPluginUtils -- that would be setReturnFields
(which i discovered when i saw StandardRequestHandler still has the old
splitList value)
#5-7 above aren't issues, just comments, but if you can clarify #1-4 i'll
happily commit this patch.
-Hoss
[jira] Updated: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=all ]
Mike Klaas updated SOLR-24:
---------------------------
Attachment: highlight_patch_v2.diff
Updated highlighting patch.
Numerous improvements:
- uses three parameters: highlight (true/false) highlightFields maxSnippets
- vastly improved defaultField handling (old version ignored some cases of specified default fields, and did the wrong thing entirely for DisMax)
- vastly improved DisMax query handling
Open issues:
- I needed to subclass a final class in the highlighter contrib package in lucene, which required C&P. This is a stop-gap measure until the issue is fixed in lucene.
- The interaction between WordDelimiterFilter and Highlighter is not fixed. This will require further investigation.
I believe the patch is in a commitable state. If it is committed, I'll open JIRA issues for the open problems above.
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=all ]
Mike Klaas updated SOLR-24:
---------------------------
Attachment: highlight_v5.patch
(hopefully) complete patch.
- removed "id=" from document highlighting entries
- ensured the multi-value snippets do not get concatenated
- in StandardRequestHandler and DisMaxHandler, the score flag is set iff it is in the responsefields parameter
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff, highlight_patch_v4.diff, highlight_v5.patch
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-24) Add Highlighting to standard request
handler
Posted by "Bill Au (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-24?page=comments#action_12419736 ]
Bill Au commented on SOLR-24:
-----------------------------
My guess is that the Highlighter doesn't like the fact that the
TokenStream produced by the WordDelimiterFilter contains
multiple Tokens with the same starting offset.
> Add Highlighting to standard request handler
> --------------------------------------------
>
> Key: SOLR-24
> URL: http://issues.apache.org/jira/browse/SOLR-24
> Project: Solr
> Type: New Feature
> Components: search
> Reporter: Mike Klaas
> Attachments: highlight_patch_v1.diff, highlight_patch_v2.diff, highlight_patch_v3.diff
>
> This patch adds highlighting functionality to solr request handlers it also refactors StandardRequestHandler to use the common functionality provided in SolrPluginUtils. I'd have preferred to do two separate patches, but creating two mutually-dependent patches on a repo without being able to commit a revision was daunting.
> -----------------------------------
> Refactoring StandardRequestHandler:
> 1. Moved solr.util.CommonParams to its own class. Removed DisMax-specific parameters, and placed in a subclass.
> 2. StandardRequestHandler uses CommonParams to store config-time parameter values (new feature)
> 3. StandardRequestHandler uses SolrPluginUtils methods for duplicate functionality
> 4. Some of said SPU methods have grown a "params" parameter to enable them to use default values. (Note: instead of passing this around, something like a RequestHelper class which carries the SolrRequest and Param values would be useful. This class could house the utility methods that require Request parameters).
> 5. SolrPluginUtils.getParam() only uses the default parameter if it is null, not blank.
> --------------------------------------
> Highlighting:
> 1. Highlighting is controlled by three request parameters:
> highlight: list of fields to highlight, or highlight the default field if at all present
> maxSnippets: maximum number of snippets to return for each field
> highlightFormatterClass: 'solr.<classname>' or full package path of highlight.Formatter subclass to use in highlighting.
> 2. Default formatter is to use <em> tags. There are issues with this approach, but are mitigated with the ability to specify a custom Formatter. Definately should consider alternatives (a custom xml approach to denote highlit regions will require some Highlighter package hackery).
> 3. Document summaries are returned as a separate element under <response> format is still up for discussion.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira