You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Batzenmann <ax...@freiheit.com> on 2008/09/30 13:28:53 UTC

Howto concatenate tokens at index time (without spaces)

Hi,

I'm looking for a way to create a fieldtype which will apart from the
whitespacedtokenized tokens also store concatenated versions of the tokens.

The ShingleFilter does s.th. very similar but keeps spaces in between words.
In german a shoe(Schuh) you wear in your 'spare time'(Freizeit) is actually
a "Freizeitschuh" and not a "Freizeit Schuh".
The WorddelimterFactory could be incorporated for this as well if the space
character could be configured as delimiter-character - but these cant be
configured at all or am I wrong?

Synonyms are in my opinion not the solution for this, as this it is imho
aboslutely not neccessary to persist any data for this 	requirement.

cheers, Axel
-- 
View this message in context: http://www.nabble.com/Howto-concatenate-tokens-at-index-time-%28without-spaces%29-tp19740271p19740271.html
Sent from the Solr - User mailing list archive at Nabble.com.


Re: Howto concatenate tokens at index time (without spaces)

Posted by Batzenmann <ax...@freiheit.com>.

Otis Gospodnetic wrote:
> 
> I haven't used the German analyzer (either Snowball or the one we have in
> Lucene's contrib), but have you checked if that does the trick of keeping
> words together?
> 
I'm not sure how this can work out with words that are space separated,
especially since we use a whitespacetokenizer first in the filter chain.

I solved the problem for now by applying the follwing filter:

public class ConcatFilter extends TokenFilter {
    private Token _last;
    private Queue<Token> _concatVersions = new LinkedList<Token>(); 

    public ConcatFilter(TokenStream input) {
        super(input);
    }

    @Override
    public Token next() throws IOException {
        final Token next = input.next();
        if ( next != null ) {
            if ( _last != null ) {
                final String concatStr = _last.termText() + next.termText();
                _concatVersions.add(new Token(concatStr, 0,
concatStr.length()));
            }
            _last = next;
            return next;
        } else if ( ! _concatVersions.isEmpty() ) {
            return _concatVersions.poll();
        }
        return null;
    }
}
-- 
View this message in context: http://www.nabble.com/Howto-concatenate-tokens-at-index-time-%28without-spaces%29-tp19740271p19756337.html
Sent from the Solr - User mailing list archive at Nabble.com.