Support for custom Fragmenter when Highlighting

[Andrew May <am...@ingenta.com>]

Hi,

I've been trying out the highlighting added to Solr recently (thanks - it's really 
useful), and I found myself wanting to highlight entire fields rather than just generating 
fragments. i.e. I want to use the Lucene NullFragmenter rather than the default GapFragmenter.

I've made some changes locally to support custom fragmenters in the same way that custom 
formatters are supported (i.e. either configured in solrconfig.xml or a parameter in the 
request).

Before I submit a patch, I wanted to check a few things:

In CommonParams, the variables added for highlighting were declared static (unlike the 
existing variables for the field list etc.). I made these non-static and everything still 
compiles. Is that OK?

In SolrPluginUtils I added "getHighlighter(Query query, SolrQueryRequest request, 
CommonParams params)". The idea was to move all the parameter checking into this method 
instead of having it in doStandardHighlighting(), and have this as the single place 
highlighters are generated.
However, I didn't remove getDefaultHighlighter(Query query) because this is used by 
getHighlights(DocList docs, String[] fieldNames, Query query, SolrIndexSearcher searcher, 
int numFragments), which doesn't have access to the SolrQueryRequest or CommonParams. I 
can't see anywhere this getHighlights method is used, but I didn't want to remove it.

In SolrPluginUtils I did an Eclipse "organise imports" (force of habit). This has 
re-arranged and shortened the list of imports considerably (there were some unused 
imports, and also I've got Eclipse configured to use * if there is more than 1 import from 
a package).

Basically I have a really hard time not fiddling with other people's code - it annoys the 
hell out of some of my colleagues.

So, should I submit the patch as is (via JIRA I assume?), or should I change it to make 
the minimal number of changes from the original classes?

Regards,

Andrew

Re: Support for custom Fragmenter when Highlighting

[Yonik Seeley <ys...@gmail.com>]

On 7/18/06, Chris Hostetter <ho...@fucit.org> wrote:
>
> : Field properties could be made more than a convention... the
> : f.field.property=val stuff could be parsed out to easier-to-use
> : objects by PluginUtils.  But they are optional, and I don't see
> : f.foo.fragsize=0 being more complicated than having to write your own
> : custom query handler ;-)
>
> certainly ... i'm just not clear if there are a lot of cases where that
> much flexability is really needed at query time -- as opposed to in the
> solrconfig where it can more easily be expressed.

Multiple sites could query a common search collection, or probably
more likely, different pages on the same site may want different
fragment sizes.   As a common example, look at google groups... there
are small fragments in the summary page, and then large fragments (or
the complete message) highlighted when you go to the thread view.

I could even see a site wanting the user to be able to customize the sizes.

I guess it is primarily an issue of aesthetics... we do want to avoid
making things ugly (query args, configuration, etc) while we make
things as powerful and flexible as possible.

-Yonik

Re: Support for custom Fragmenter when Highlighting

[Andrew May <am...@ingenta.com>]

Mike Klaas wrote:
> On 7/19/06, Andrew May <am...@ingenta.com> wrote:
> 
>> (although I have
>> wondered whether it would be possible for the default highlighted 
>> fields to be derived
>> from the query in some way).
> 
> It wouldn't be too hard to extract <field>:<term> queries in
> StandardRequestHandler.  Do you have anything specific in mind?  Hoss
> and I discussed using all analyzed stored fields as a reasonable
> default.
> 

I was sort of thinking that if the query was "frogs" then the default field would be 
highlighted, but if the query was "+title:frogs +author:May" then title and author would 
be highlighted. However, I don't know if it gets complicated with dynamic fields.

>> But if that's not a common requirement it could be worked around by 
>> copying the field and
>> having different highlighting settings.
> 
> I think the best way to achieve this is to configure two separate
> request handlers in solrconfig.xml with the same request handler class
> but with different default parameters.
> 

That hadn't occurred to me.

I'm going to work on the assumption that it's OK for some aspects of highlighting to only 
be configurable in solrconfig.xml, and see what I can produce. I'll try to put together a 
patch that can go in JIRA and then it can be discussed further.

-Andrew

Re: Support for custom Fragmenter when Highlighting

[Mike Klaas <mi...@gmail.com>]

On 7/19/06, Andrew May <am...@ingenta.com> wrote:

> (although I have
> wondered whether it would be possible for the default highlighted fields to be derived
> from the query in some way).

It wouldn't be too hard to extract <field>:<term> queries in
StandardRequestHandler.  Do you have anything specific in mind?  Hoss
and I discussed using all analyzed stored fields as a reasonable
default.

<>
> Having said that, it is possible that I might have a requirement to have different
> presentations of the same search results - perhaps the search results page has a fragment
> of the "title" highlighted, but the detail page needs the entire title highlighted. Or two
> different sites with different styles of results sharing the same Solr instance.
>
> But if that's not a common requirement it could be worked around by copying the field and
> having different highlighting settings.

I think the best way to achieve this is to configure two separate
request handlers in solrconfig.xml with the same request handler class
but with different default parameters.

-Mike

Re: Support for custom Fragmenter when Highlighting

[Andrew May <am...@ingenta.com>]

Chris Hostetter wrote:
> certainly ... i'm just not clear if there are a lot of cases where that
> much flexability is really needed at query time -- as opposed to in the
> solrconfig where it can more easily be expressed.
> 
> I guess in my mind people are more likley to say "in this application,
> anybody who queries is probably going to want highlighting xyz for field
> pdq and highlighting abc for field hij, and property qrs for field mno --
> so i'll put that in the solrconfig."
> 

For my own uses, I want to statically configure the highlighting settings and just turn it 
on and off and specify which fields I want highlighted in the results (although I have 
wondered whether it would be possible for the default highlighted fields to be derived 
from the query in some way).

Having said that, it is possible that I might have a requirement to have different 
presentations of the same search results - perhaps the search results page has a fragment 
of the "title" highlighted, but the detail page needs the entire title highlighted. Or two 
different sites with different styles of results sharing the same Solr instance.

But if that's not a common requirement it could be worked around by copying the field and 
having different highlighting settings.

I'm happy to continue poking around in the Solr code as I'm the one who brought this up, 
but I'll probably get out of my depth if I try to do anything more complicated, and I'm 
reluctant to impose my style of configuration if it doesn't match the consensus (assuming 
we can come to one). It does seem that most of the complexity here is in the configuration 
and not the actual highlighting work.

-Andrew

Re: Support for custom Fragmenter when Highlighting

[Chris Hostetter <ho...@fucit.org>]

: Field properties could be made more than a convention... the
: f.field.property=val stuff could be parsed out to easier-to-use
: objects by PluginUtils.  But they are optional, and I don't see
: f.foo.fragsize=0 being more complicated than having to write your own
: custom query handler ;-)

certainly ... i'm just not clear if there are a lot of cases where that
much flexability is really needed at query time -- as opposed to in the
solrconfig where it can more easily be expressed.

I guess in my mind people are more likley to say "in this application,
anybody who queries is probably going to want highlighting xyz for field
pdq and highlighting abc for field hij, and property qrs for field mno --
so i'll put that in the solrconfig."



-Hoss

Re: Support for custom Fragmenter when Highlighting

[Yonik Seeley <ys...@gmail.com>]

On 7/18/06, Chris Hostetter <ho...@fucit.org> wrote:
> Adding a "namespace" of query params for dealing with highlighting (or
> anything else) seems kind of complicated and confusing ... is that level
> of granularity of control at query time really that usefull conpared to
> the complexity of understanding it?  -- or would it make more sense to
> confine that level of customizability to the solrconfig init params of the
> requestHandler (where more complex data structures can already be
> expressed) ?

That's why good defaults are important.
If the defaults are working fine for you, there is no difference
between greater customizability and less.

The namespace thing is just a convention to aid understanding in
general....  what query parameters apply to what.

Field properties could be made more than a convention... the
f.field.property=val stuff could be parsed out to easier-to-use
objects by PluginUtils.  But they are optional, and I don't see
f.foo.fragsize=0 being more complicated than having to write your own
custom query handler ;-)

-Yonik

Re: Support for custom Fragmenter when Highlighting

[Chris Hostetter <ho...@fucit.org>]

: Have you considered making the query language an XML-base POST'd
: language, like updates?  As the language increases in power, this
: could improve manageability.  It would also correspond directly with
: directives in solrconfig.xml.  A subset of the query language could be
: allowed in a GET request.

FYI: There is an xml parser in the lucene contrib directory that allows
for a very customizable xml based query syntax -- i believe it is flexible
enough that other types of options could be expressed in it.

in general, i think the issue of query params and query param names for
the "built in" handlers isn't that big of a deal ... the
StandardRequestHandler doesn't have to support every possible feature of
Solr via query params -- we just wnat to make sure to support the major
ones with easy to use and intuative param names.  If people want really
specialized behavior, they can write a specialized plugin (so it is
important to make the functionality easy to use via utilities)

Adding a "namespace" of query params for dealing with highlighting (or
anything else) seems kind of complicated and confusing ... is that level
of granularity of control at query time really that usefull conpared to
the complexity of understanding it?  -- or would it make more sense to
confine that level of customizability to the solrconfig init params of the
requestHandler (where more complex data structures can already be
expressed) ?





-Hoss

Re: Support for custom Fragmenter when Highlighting

[Mike Klaas <mi...@gmail.com>]

On 7/18/06, Yonik Seeley <ys...@gmail.com> wrote:

<>
> I wonder what the default fragment size should be?

FWIW, the current default is 100 and this seems to do a decent job.
We should keep in mind that future Fragmenters might be able to be
more intelligent (take periods into account, for instance).  So the
fragment size might be more of a fragment size hint.

> #model things as properties on fields (with f. being the field namespace)
>
> f.foo.fragsize=0
> f.bar.fragsize=1000
> f.*.fragsize=100   #the default

I like this option the best, though the wildcard specification might
get out of hand.

There could be a top-level namespace:
hl.fragsize = 100 #default
And field-level overrides precisely matching the top-level general params:
f.foo.hl.fragsize = 0

> I wouldn't be putting as much thought into this, but this issue will
> probably come up again and be applicable beyond the specification of
> fragment sizes.
>
> Thoughts?

Have you considered making the query language an XML-base POST'd
language, like updates?  As the language increases in power, this
could improve manageability.  It would also correspond directly with
directives in solrconfig.xml.  A subset of the query language could be
allowed in a GET request.

cheers,
-Mike

Re: Support for custom Fragmenter when Highlighting

[Yonik Seeley <ys...@gmail.com>]

> This makes me think that perhaps I was taking the wrong tack by specifying the Fragmenter
> class, and that it would be better to specify the fragment length. If length <= 0 then use
> the NullFragmenter, otherwise use the GapFragmenter, supplying the length value to it.
> This has the same issue of wanting to configure it per-field.

I wonder what the default fragment size should be?

Brainstorming of different ways one could specify per-field params:
Query args are proliferating... at first blush, I'm liking the idea of
using some "namespaces" to make it easier to keep track of them.

hl.fragsize=100   #specify a highlighter fragsize default for all fields

#field specific frag sizes in a style of named params for each field
highlightFields=foo(fragsize:1000,
highlighter=...),bar(fragsize:1000),*(fragsize:100)
#probably too much syntax and too much extra parsing required.

#field specific frag sizes
hl.foo.fragsize=0      # field "foo" specific fragment size
hl.bar.fragsize=1000  # field "bar" specific fragment size

*OR*

hl.fragsize.foo=0
hl.fragsize.bar=1000

*OR*

hl.fragsize=foo:0,bar:1000
#I'm not sure if I like the secondary parsing needed for a format like this...
#the previous formats also required some parsing to recognize the param
#started with hl.fragsize and then to extract the field name.

*OR*

#model things as properties on fields (with f. being the field namespace)

f.foo.fragsize=0
f.bar.fragsize=1000
f.*.fragsize=100   #the default

I wouldn't be putting as much thought into this, but this issue will
probably come up again and be applicable beyond the specification of
fragment sizes.

Thoughts?

-Yonik

Re: Support for custom Fragmenter when Highlighting

[Andrew May <am...@ingenta.com>]

Mike Klaas wrote:
> My main comment before seeing the patch is that the Fragmenter desired
> might be different for different fields.  In my use, I'd want
> GapFragmenter for the main text field and NullFragmenter for the title
> field.  This complexity is why I didn't implement this in the first
> patch.  If it is supported by your approach, that would be great.

I'm afraid it doesn't do that, which I agree would be more useful. For what I'm working 
on, I don't know whether I'm likely to want to mix the two types of highlighting (I'm 
still waiting for some real requirements...), but it's a possibility.

I guess it really needs a configurable global default, and then per-field defaults 
(presumably set in schema.xml). I'm not sure how practical it would be to specify 
fragmenters for each field as part of the query string - that would be kind of messy.

I see Yonik has commented on this:

 > If there were a way to specify fragment sizes, one could pick a very
 > large fragment size, or there could be a special fragment size (like 0
 > or -1) that means "don't fragment".  This might be simpler than having
 > some kind of special "don't fragment" parameter in addition to
 > fragment size parameters.

This makes me think that perhaps I was taking the wrong tack by specifying the Fragmenter 
class, and that it would be better to specify the fragment length. If length <= 0 then use 
the NullFragmenter, otherwise use the GapFragmenter, supplying the length value to it. 
This has the same issue of wanting to configure it per-field.

-Andrew

Re: Support for custom Fragmenter when Highlighting

[Yonik Seeley <ys...@gmail.com>]

I just recently added some tasks to the wiki task-list related to highlighter:

    *  Allow specification of begin and end markup for highlighted
regions, perhaps with namespace like query params highlight.start=<b>
highlight.end=</b>
    * Allow specification of default highlighter markup via init
params in solrconfig.xml
    * decide on the best default... <B></B>, <em></em>, or <SPAN
class="highlight"></SPAN>

If there were a way to specify fragment sizes, one could pick a very
large fragment size, or there could be a special fragment size (like 0
or -1) that means "don't fragment".  This might be simpler than having
some kind of special "don't fragment" parameter in addition to
fragment size parameters.

> Sounds like insofar as removing imports is concerned.  I'm not sure
> what the Solr policy on asterisk imports is--I personally eschew them.

I personally don't care.  I use IntelliJ, and by default it collapses
all the imports behind a "+", so I never see them unless I want to.
It does the same with the license statements at the top of the file
too (which will be going shortly, BTW).

Try not to do other formatting changes with a patch though... it makes
it harder to review since it's harder to see what actually changed.
Big formatting changes should be a separate patch.

-Yonik

Re: Support for custom Fragmenter when Highlighting

[Mike Klaas <mi...@gmail.com>]

On 7/18/06, Andrew May <am...@ingenta.com> wrote:
> I've been trying out the highlighting added to Solr recently (thanks - it's really
> useful), and I found myself wanting to highlight entire fields rather than just generating
> fragments. i.e. I want to use the Lucene NullFragmenter rather than the default GapFragmenter.

Cool, this was something I knew would be needed soon.  It makes sense
for fields like title, where the intent is purely highlighting and not
summarization.

<>
> In CommonParams, the variables added for highlighting were declared static (unlike the
> existing variables for the field list etc.). I made these non-static and everything still
> compiles. Is that OK?

They should be non-static

> In SolrPluginUtils I added "getHighlighter(Query query, SolrQueryRequest request,
> CommonParams params)". The idea was to move all the parameter checking into this method
> instead of having it in doStandardHighlighting(), and have this as the single place
> highlighters are generated.
> However, I didn't remove getDefaultHighlighter(Query query) because this is used by
> getHighlights(DocList docs, String[] fieldNames, Query query, SolrIndexSearcher searcher,
> int numFragments), which doesn't have access to the SolrQueryRequest or CommonParams. I
> can't see anywhere this getHighlights method is used, but I didn't want to remove it.

It is safe to remove that--the main use to to perform highlighting
simply outside the confines of a request, but that can be accomplished
easily-enough with the other getHighlights method.

> In SolrPluginUtils I did an Eclipse "organise imports" (force of habit). This has
> re-arranged and shortened the list of imports considerably (there were some unused
> imports, and also I've got Eclipse configured to use * if there is more than 1 import from
> a package).

Sounds like insofar as removing imports is concerned.  I'm not sure
what the Solr policy on asterisk imports is--I personally eschew them.

> So, should I submit the patch as is (via JIRA I assume?), or should I change it to make
> the minimal number of changes from the original classes?

I'd say get an initial reaction here then submit a patch.  It is
easier to work with a JIRA issue in place.

My main comment before seeing the patch is that the Fragmenter desired
might be different for different fields.  In my use, I'd want
GapFragmenter for the main text field and NullFragmenter for the title
field.  This complexity is why I didn't implement this in the first
patch.  If it is supported by your approach, that would be great.

-Mike

Re: Support for custom Fragmenter when Highlighting

[Yonik Seeley <ys...@gmail.com>]

On 7/18/06, Andrew May <am...@ingenta.com> wrote:
> In CommonParams, the variables added for highlighting were declared static (unlike the
> existing variables for the field list etc.). I made these non-static and everything still
> compiles. Is that OK?

Thanks for pointing it out, I just fixed this.

-Yonik