Re: Howto concatenate tokens at index time (without spaces)

[Batzenmann <ax...@freiheit.com>]

Otis Gospodnetic wrote:
> 
> I haven't used the German analyzer (either Snowball or the one we have in
> Lucene's contrib), but have you checked if that does the trick of keeping
> words together?
> 
I'm not sure how this can work out with words that are space separated,
especially since we use a whitespacetokenizer first in the filter chain.

I solved the problem for now by applying the follwing filter:

public class ConcatFilter extends TokenFilter {
    private Token _last;
    private Queue<Token> _concatVersions = new LinkedList<Token>(); 

    public ConcatFilter(TokenStream input) {
        super(input);
    }

    @Override
    public Token next() throws IOException {
        final Token next = input.next();
        if ( next != null ) {
            if ( _last != null ) {
                final String concatStr = _last.termText() + next.termText();
                _concatVersions.add(new Token(concatStr, 0,
concatStr.length()));
            }
            _last = next;
            return next;
        } else if ( ! _concatVersions.isEmpty() ) {
            return _concatVersions.poll();
        }
        return null;
    }
}
-- 
View this message in context: http://www.nabble.com/Howto-concatenate-tokens-at-index-time-%28without-spaces%29-tp19740271p19756337.html
Sent from the Solr - User mailing list archive at Nabble.com.