You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Koji Sekiguchi <ko...@r.email.ne.jp> on 2011/12/26 12:55:41 UTC

feature of FST version of SynonymFilter affects Highlighter

I found that SynonymFilter javadoc says:

"Matches single or multi word synonyms in a token stream.
This token stream cannot properly handle position increments != 1"

I think due to the feature, Highlighter doesn't work properly in some cases:

http://www.lucidimagination.com/search/document/c3ed1e0a2b12ddfa#c3ed1e0a2b12ddfa

https://issues.apache.org/jira/browse/SOLR-2845

Can we remove the restriction in some future?

If not, I'd propose we have an option to choose SlowSynonymFilterFactory explicitly
in schema.xml (we can choose it by setting luceneMatchVersion to 33,
but it is global).

koji
-- 
http://www.rondhuit.com/en/

Re: feature of FST version of SynonymFilter affects Highlighter

Posted by Robert Muir <rc...@gmail.com>.
On Mon, Dec 26, 2011 at 10:54 AM, Koji Sekiguchi <ko...@r.email.ne.jp> wrote:

> I don't have JUnit test case. What I tried was:
>
> I have indexing time synonym definition:
>
> nhl, national hockey league
>
> and I indexed "I like national hockey league".
>
> Then I searched nhl with hl=on, I got an unwanted highlight snippet
> "I like <em>national</em> hockey league".
>
> But if I set luceneMatchVersion to LUCENE_33 and re-indexed,
> I got an expected result "I like <em>national hockey league</em>".
>

Thanks Koji, I'll see if I can create a test case for this later
today. SynonymFilter could have a bug with the offsets.

-- 
lucidimagination.com

Re: feature of FST version of SynonymFilter affects Highlighter

Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(11/12/26 23:58), Robert Muir wrote:
> The old one didn't really handle this correctly either.

Sorry, I jumped the gun! I should say offsets, not positions!
So please ignore what I quoted in javadoc in my previous mail, sorry!

> Koji, what is the highlighting problem? Can we have a test case?

I don't have JUnit test case. What I tried was:

I have indexing time synonym definition:

nhl, national hockey league

and I indexed "I like national hockey league".

Then I searched nhl with hl=on, I got an unwanted highlight snippet
"I like <em>national</em> hockey league".

But if I set luceneMatchVersion to LUCENE_33 and re-indexed,
I got an expected result "I like <em>national hockey league</em>".

koji
-- 
http://www.rondhuit.com/en/

Re: feature of FST version of SynonymFilter affects Highlighter

Posted by Robert Muir <rc...@gmail.com>.
The old one didn't really handle this correctly either.

Koji, what is the highlighting problem? Can we have a test case?

2011/12/26 Koji Sekiguchi <ko...@r.email.ne.jp>:
> I found that SynonymFilter javadoc says:
>
> "Matches single or multi word synonyms in a token stream.
> This token stream cannot properly handle position increments != 1"
>
> I think due to the feature, Highlighter doesn't work properly in some cases:
>
> http://www.lucidimagination.com/search/document/c3ed1e0a2b12ddfa#c3ed1e0a2b12ddfa
>
> https://issues.apache.org/jira/browse/SOLR-2845
>
> Can we remove the restriction in some future?
>
> If not, I'd propose we have an option to choose SlowSynonymFilterFactory explicitly
> in schema.xml (we can choose it by setting luceneMatchVersion to 33,
> but it is global).
>
> koji
> --
> http://www.rondhuit.com/en/



-- 
lucidimagination.com