Generic RemoveDuplicatesTokenFilter

[pravesh <su...@yahoo.com>]

Hi All,

Currently, the SOLR's existing 
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.RemoveDuplicatesTokenFilterFactory
RemoveDuplicatesTokenFilter   filters the duplicate tokens with the same
text and logical at the same position.

In my case, if the same term appears duplicate one after the other then i
need to remove all duplicates and consume only single occurance of the term
(even if the positionincrementgap ==1).

For e.g. the input stream is as:  /quick brown brown brown fox jumps jumps
over the little little lazy brown dog/
Then the output shld be:  quick brown fox jumps over the little lazy brown
dog.

To acheive this, I implemented my own version of
/RemoveDuplicatesTokenFilter/ with overridden /process()/ method as:

  protected Token process(Token t) throws IOException {
	  Token nextTok = peek(1);
	  if(t!=null && nextTok!=null){
	         if(t.termText().equalsIgnoreCase(nextTok.termText())){
		    return null;
		  }
          }
	  return t;
  }

The above implementation works as per desired and the continuous duplicates
are getting removed :)

Any advice/feedback for the above implementation :)

Regards
Pravesh

--
View this message in context: http://lucene.472066.n3.nabble.com/Generic-RemoveDuplicatesTokenFilter-tp3581656p3581656.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Generic RemoveDuplicatesTokenFilter

[Erick Erickson <er...@gmail.com>]

@Pravesh
That look reasonable. Of course you could extend it
to do many things. I'm assuming you've just created
a plugin that you use rather than compile this into
the Solr code, right?

@astubbs
I'd probably use a TokenFilter(Factory) implementation
as a plugin as I think pravesh has.
It would also be possible to use a TokenizerFactory,
depending on where you need this to happen, but FilterFactory
is my first choice, they allow more flexibility.

I rather doubt a patch on this order will make it into the code,
it's rather special-purpose.

Best
Erick

On Wed, Jan 4, 2012 at 12:32 PM, astubbs <an...@gmail.com> wrote:
> That's exactly what I need. I'm using phonetic tokens on ngrams, and there's
> lots of dupes. Can you submit it as a patch? What's the easiest way to get
> this into my solr?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Generic-RemoveDuplicatesTokenFilter-tp3581656p3632499.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Generic RemoveDuplicatesTokenFilter

[astubbs <an...@gmail.com>]

That's exactly what I need. I'm using phonetic tokens on ngrams, and there's
lots of dupes. Can you submit it as a patch? What's the easiest way to get
this into my solr?

--
View this message in context: http://lucene.472066.n3.nabble.com/Generic-RemoveDuplicatesTokenFilter-tp3581656p3632499.html
Sent from the Solr - User mailing list archive at Nabble.com.