You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Batzenmann <ax...@freiheit.com> on 2008/09/30 13:28:53 UTC
Howto concatenate tokens at index time (without spaces)
Hi,
I'm looking for a way to create a fieldtype which will apart from the
whitespacedtokenized tokens also store concatenated versions of the tokens.
The ShingleFilter does s.th. very similar but keeps spaces in between words.
In german a shoe(Schuh) you wear in your 'spare time'(Freizeit) is actually
a "Freizeitschuh" and not a "Freizeit Schuh".
The WorddelimterFactory could be incorporated for this as well if the space
character could be configured as delimiter-character - but these cant be
configured at all or am I wrong?
Synonyms are in my opinion not the solution for this, as this it is imho
aboslutely not neccessary to persist any data for this requirement.
cheers, Axel
--
View this message in context: http://www.nabble.com/Howto-concatenate-tokens-at-index-time-%28without-spaces%29-tp19740271p19740271.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Howto concatenate tokens at index time (without spaces)
Posted by Batzenmann <ax...@freiheit.com>.
Otis Gospodnetic wrote:
>
> I haven't used the German analyzer (either Snowball or the one we have in
> Lucene's contrib), but have you checked if that does the trick of keeping
> words together?
>
I'm not sure how this can work out with words that are space separated,
especially since we use a whitespacetokenizer first in the filter chain.
I solved the problem for now by applying the follwing filter:
public class ConcatFilter extends TokenFilter {
private Token _last;
private Queue<Token> _concatVersions = new LinkedList<Token>();
public ConcatFilter(TokenStream input) {
super(input);
}
@Override
public Token next() throws IOException {
final Token next = input.next();
if ( next != null ) {
if ( _last != null ) {
final String concatStr = _last.termText() + next.termText();
_concatVersions.add(new Token(concatStr, 0,
concatStr.length()));
}
_last = next;
return next;
} else if ( ! _concatVersions.isEmpty() ) {
return _concatVersions.poll();
}
return null;
}
}
--
View this message in context: http://www.nabble.com/Howto-concatenate-tokens-at-index-time-%28without-spaces%29-tp19740271p19756337.html
Sent from the Solr - User mailing list archive at Nabble.com.