You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by John Powers <jp...@configureone.com> on 2006/04/17 18:59:15 UTC
hypens
Hello,
If I have a user search for "b-trunk" I would like them to be able to
find "b-trunk" (with hypen). I would also like someone searching for
"b trunk" to also find "b-trunk".
On the other side, if someone searches for 12412 I would like them to be
able to find 12412-235, 12412-121, 12412-etc... as well as letting
someone type in 12412-235 directly and get a good result list: the one
item would be best, but a larger list with that one on top is good too.
So for now I am using the standardanalyzer. I do a search for what
they give me in quotes on all fields as well as the same thing w/o
quotes. When I print out the final query the half of the overall query
in quotes seems to have the hypens stripped out, but the w/o quotes
version doesn't...so this lets me find what I want. But I have each
search phrase in the final query twice now. it seems to work fine,
but it seems pretty inelegant--unelegant even.
It seems like I can't just strip out the hypens, nor keep them. I am
storing the name as keyword, but everything else as Text. I thought
that would matter but a description or keyword or other field may have
something like "this also relates to 23523-235" so if someone was
searching for 23523 I would also want this in the list... and if they
searched for the 23523-235 then I would also want this still. So I
don't know if its solvable by the type of field I use to index it. Or
do I have to store each field twice with different analyzer? That seems
just as clumsy as my double-search solution.
Any thoughts?
Re: hypens
Posted by karl wettin <ka...@snigel.net>.
17 apr 2006 kl. 18.59 skrev John Powers:
> Hello,
>
> If I have a user search for "b-trunk" I would like them to be able to
> find "b-trunk" (with hypen). I would also like someone searching for
> "b trunk" to also find "b-trunk".
If you don't care about spans, make a filter that rebuilds the token
at index time. It's a bit quick and dirty, but I do things like this
in some cases without any major problems.
Below code builds [btrunk] [b-trunk] [b] [trunk].
I would not recommend you to do this without considering what you do.
public class LowASCIIDashWordFilter extends TokenFilter {
private static Pattern p = Pattern.compile("(\\w+)-(\\w+)");
/** Construct a token stream filtering the given input. */
public LowASCIIDashWordFilter(TokenStream input) {
super(input);
}
private LinkedList<Token> buf;
/** Returns the next token in the stream, or null at EOS. */
public Token next() throws IOException {
Token next;
if (buf == null) {
buf = new LinkedList<Token>();
next = input.next();
while (next != null) {
Matcher m = p.matcher(next.termText());
if (m.matches()) {
buf.add(new Token(m.group(1) + m.group(2),
m.start(1), m.end(2), "composite dashword"));
buf.add(new Token(m.group(1), m.start(1), m.end
(1), "left dashword"));
buf.add(new Token(m.group(2), m.start(2), m.end
(2), "right dashword"));
}
next = input.next();
}
}
if (buf.size() > 0) {
return buf.removeFirst();
} else {
return null;
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: hypens
Posted by Ramana Jelda <ra...@ciao-group.com>.
Hi,
I would use index & search analyzers in this case..
"b-trunk" is analyzed & indexed as b,btrunk,trunk
Search term "b-trunk" is anlayzed using search analyzer as "btrunk" and
searched. U will find the result..
Similarly for 12412-235, 12412-121, 12412-etc , indexed as
12412,12412235,235 etc....
So obviously it will find 12412 search term.
Good luck,
Jelda
> -----Original Message-----
> From: John Powers [mailto:jpowers@configureone.com]
> Sent: Monday, April 17, 2006 6:59 PM
> To: java-user@lucene.apache.org
> Subject: hypens
>
> Hello,
>
>
>
> If I have a user search for "b-trunk" I would like them to be able to
>
> find "b-trunk" (with hypen). I would also like someone searching for
>
> "b trunk" to also find "b-trunk".
>
>
>
> On the other side, if someone searches for 12412 I would like
> them to be
>
> able to find 12412-235, 12412-121, 12412-etc... as well
> as letting
>
> someone type in 12412-235 directly and get a good result
> list: the one item would be best, but a larger list with that
> one on top is good too.
>
>
>
> So for now I am using the standardanalyzer. I do a search for what
>
> they give me in quotes on all fields as well as the same
> thing w/o quotes. When I print out the final query the half
> of the overall query in quotes seems to have the hypens
> stripped out, but the w/o quotes
>
> version doesn't...so this lets me find what I want. But I have each
>
> search phrase in the final query twice now. it seems to work fine,
>
> but it seems pretty inelegant--unelegant even.
>
>
>
> It seems like I can't just strip out the hypens, nor keep
> them. I am
>
> storing the name as keyword, but everything else as Text. I thought
>
> that would matter but a description or keyword or other field
> may have something like "this also relates to 23523-235" so
> if someone was searching for 23523 I would also want this in
> the list... and if they
>
> searched for the 23523-235 then I would also want this still. So I
>
> don't know if its solvable by the type of field I use to
> index it. Or
>
> do I have to store each field twice with different analyzer?
> That seems just as clumsy as my double-search solution.
>
>
>
> Any thoughts?
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org