phrase filter

[emagdnim paradox <bw...@yahoo.com>]

Hey all,

When I analyze my text during indexing, I want to sometimes include stop words. For example, if the phrase is "in box" I want to include "in", but if it's "in blah" I don't. Also, I was considering keeping "in box" as one token, since this seems to make more semantic sense and would allow me to do searches without using nearness.

1. Is there an existing analyzer I can use to do this?
2. Supposing no analyzer like this exists, how would I go about doing this? I was thinking about creating a filter which does:
* Increment
* See if next token is the start of a two-word phrase. If not, just return the token, if it is:
* Increment again
* See if these two tokens together are a two-word phrase. If so, return them both as one token, else rewind and just return the first token.

What I'm having trouble with is that last piece. Do I just set TermAttribute to contain both tokens together? Or use position increment attribute somehow? How can I "rewind" if it turns out I actually wanted to return them as two separate tokens?

Thanks,
-Ben


      

RE: phrase filter

[Digy <di...@gmail.com>]

1. Of course No.
2. there isn't any method similar to "rewind". You have to use some
structures/logic to implement this.
I am not sure whether it can be helpful as a sample but below is an analyzer
that emits every char as a separate token.


https://svn.apache.org/repos/asf/lucene/lucene.net/trunk/C%23/contrib/Contri
b.Net/Contrib.Net/Analysis/Ext/Analysis.Ext.cs

DIGY



-----Original Message-----
From: emagdnim paradox [mailto:bwsithspawn00@yahoo.com] 
Sent: Wednesday, April 21, 2010 8:03 PM
To: lucene-net-user@lucene.apache.org
Subject: phrase filter

Hey all,

When I analyze my text during indexing, I want to sometimes include stop
words. For example, if the phrase is "in box" I want to include "in", but if
it's "in blah" I don't. Also, I was considering keeping "in box" as one
token, since this seems to make more semantic sense and would allow me to do
searches without using nearness.

1. Is there an existing analyzer I can use to do this?
2. Supposing no analyzer like this exists, how would I go about doing this?
I was thinking about creating a filter which does:
* Increment
* See if next token is the start of a two-word phrase. If not, just return
the token, if it is:
* Increment again
* See if these two tokens together are a two-word phrase. If so, return them
both as one token, else rewind and just return the first token.

What I'm having trouble with is that last piece. Do I just set TermAttribute
to contain both tokens together? Or use position increment attribute
somehow? How can I "rewind" if it turns out I actually wanted to return them
as two separate tokens?

Thanks,
-Ben