HTMLStripWhitespaceTokenizerFactory problem with highlighting

[Daniel Alheiros <Da...@bbc.co.uk>]

Hi

I'm using the released version 1.2.

I was using HTMLStripWhitespaceTokenizerFactory to remove from my index some
rubish (html tags that are not relevant in my structure), but it is making
the highlighting fail in some conditions. It seems to me that it's not
keeping track of the proper positions of tokens when using this HTML
Stripper.... So the emphasis tags appear at wrong positions.

I'm sure it's related to it, because when I replaced it by the standard
WhitespaceTokenizerFactory it worked properly.

Is it a known bug?

Regards,
Daniel


http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
					

Re: HTMLStripWhitespaceTokenizerFactory problem with highlighting

[Chris Hostetter <ho...@fucit.org>]

: I was using HTMLStripWhitespaceTokenizerFactory to remove from my index some
: rubish (html tags that are not relevant in my structure), but it is making
: the highlighting fail in some conditions. It seems to me that it's not

: Is it a known bug?

SOLR-42


-Hoss