You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Paul Smith <PS...@tenfold.com> on 2005/04/13 01:42:23 UTC
Hungarian notation analyzer and phrase queries
I am writing a document management system for my company, and many of
our feature names are in Hungarian notation (PowerQuery,
TransactionManager, etc.). This can make it hard to find some things
with a default analyzer.
I'd like to be able to index text like "Use PowerQuery for advanced
searches" and be able to find it with "use power query for advanced
searches". Note the space between power and query.
I have written a custom analyzer to tokenize PowerQuery into 'power',
'query, and 'powerquery' and change the position increment to 0, but I
don't quite get the desired behavior. The phrase query "use power query
for advanced searches" does not match, but "use query for advanced
searches", and "use power for advanced searches" do.
Any ideas?
I noticed that Dave at tropo.com has written a JavaDocAnalyzer that has
the same problem. Go to searchmorph.com and search for "An instance of
HashMap has two parameters" and "An instance of Hash Map has two
parameters"
I realize that with my custom analyzer I can find it without using a
phrase query, but it would be nice.
Thanks,
Paul
http://www.tenfold.com
**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the TenFold Postmaster (postmaster@tenfold.com).
**********************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Hungarian notation analyzer and phrase queries
Posted by "Peter Hotm. N�rregaard" <ng...@hotmail.com>.
Chris wrote
>...As Erik points out in that thread, when dealing with a dictionary of
>"singleword" => ["multi" "word"], and ["multi" "word"] => "singleword"
>synonyms a very good/simple approach is to use an analyzer that allways
>normalizes down to the single word version (as a single token)
>
>This allows you to leave the position incriment alone, and get Span/Phrase
>queries to work just fine.
But be aware of the drawbacks from replacing multiple words with its
corresponding acryonym/alias during analysis. The analyzis at search-time
cannot perform the normalization to a single word if the rest of the words
in the search string are not recognized:
- Prefix search: [cyber] [ca*] would not match [cybercafe]
- Wildcard search: [cyber] [ca?e] would not match [cybercafe]
- Fuzzy search: [cyber] [cage~] would not match [cybercafe]
I am not sure that this is correct but perhaps someone in this forum can.
/Peter
_________________________________________________________________
Del din verden med MSN Spaces http://spaces.msn.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Hungarian notation analyzer and phrase queries
Posted by Chris Hostetter <ho...@fucit.org>.
This question is very similar to a recent/current thread sith the subject
"How to include a multi-word synonym to a word when indexing?" ...
http://www.mail-archive.com/java-user@lucene.apache.org/msg00546.html
...As Erik points out in that thread, when dealing with a dictionary of
"singleword" => ["multi" "word"], and ["multi" "word"] => "singleword"
synonyms a very good/simple approach is to use an analyzer that allways
normalizes down to the single word version (as a single token)
This allows you to leave the position incriment alone, and get Span/Phrase
queries to work just fine.
In your case, if the source text being indexed contains allready contains
only the "SingleWord" forms in "HungarianNotation", you don't need to
start out with a dictionary -- you can build one while indexing using a
TokenFilter (ie: if a token starts with capital, and contains capital,
split on capitals and log to dictionary file). Then at search time, use a
different TokenFilter that looks up token sequences in your dictionary
and replaces them with the single token version.
If you have mixed usage in the source text being indexed, you may have to
do two passes to build a dictionary, then index.
: Date: Tue, 12 Apr 2005 17:42:23 -0600
: From: Paul Smith <PS...@tenfold.com>
: Reply-To: java-user@lucene.apache.org
: To: java-user@lucene.apache.org
: Subject: Hungarian notation analyzer and phrase queries
:
: I am writing a document management system for my company, and many of
: our feature names are in Hungarian notation (PowerQuery,
: TransactionManager, etc.). This can make it hard to find some things
: with a default analyzer.
:
: I'd like to be able to index text like "Use PowerQuery for advanced
: searches" and be able to find it with "use power query for advanced
: searches". Note the space between power and query.
:
: I have written a custom analyzer to tokenize PowerQuery into 'power',
: 'query, and 'powerquery' and change the position increment to 0, but I
: don't quite get the desired behavior. The phrase query "use power query
: for advanced searches" does not match, but "use query for advanced
: searches", and "use power for advanced searches" do.
:
: Any ideas?
:
: I noticed that Dave at tropo.com has written a JavaDocAnalyzer that has
: the same problem. Go to searchmorph.com and search for "An instance of
: HashMap has two parameters" and "An instance of Hash Map has two
: parameters"
:
: I realize that with my custom analyzer I can find it without using a
: phrase query, but it would be nice.
:
: Thanks,
: Paul
:
: http://www.tenfold.com
:
: **********************************************************************
: This email and any files transmitted with it are confidential and
: intended solely for the use of the individual or entity to whom they
: are addressed. If you have received this email in error please notify
: the TenFold Postmaster (postmaster@tenfold.com).
: **********************************************************************
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re[2]: multi word synonym (was Hungarian notation analyzer and phrase queries)
Posted by Sven Duzont <sv...@keljob.com>.
Hello,
What about the solution to index every multi-word synonym as a single
token ?
Example :
Phrase to index : "i love jsp and tomcat"
Synonyms : "jsp" = "java server pages" = "javaserver pages"
Tokens : i love jsp and tomcat
java server pages
javaserver pages
Position : 0 1 2 3 4
This solution will have the advantage to solve the phrase query problems.
One will have also to rewrite queries before parsing it with the
QueryParser. for instance the query (tomcat jsp) will be rewrited as
(tomcat (jsp OR "java server pages" OR "javaserver pages"))
Any thoughts ?
Thanks in advance
---
Sven
mercredi 13 avril 2005, 19:36:44, vous avez écrit:
CH> : Another approach would be to index this as:
CH> :
CH> : token: use power query for advanced searches
CH> : powerquery
CH> : position: 0 1 2 3 4 5
CH> :
CH> : Then use phrase queries with slop=1, to permit a one-token gap when
CH> : someone searches for "use powerquery for advanced searches".
CH> right, but in your example "1" is a magic number that works because we are
CH> only dealing with "multi-word synonyms" of 2 words. in general, this
CH> approach requires that you pick some some N such that you are garunteed no
CH> synonym contains more then N-1 words, and set the token positions to...
CH> token: use powerquery for advanced searches
CH> power query
CH> position: 0 N N+1 2N 3N 4N
CH> -Hoss
CH> ---------------------------------------------------------------------
CH> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
CH> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Hungarian notation analyzer and phrase queries
Posted by Chris Hostetter <ho...@fucit.org>.
: Another approach would be to index this as:
:
: token: use power query for advanced searches
: powerquery
: position: 0 1 2 3 4 5
:
: Then use phrase queries with slop=1, to permit a one-token gap when
: someone searches for "use powerquery for advanced searches".
right, but in your example "1" is a magic number that works because we are
only dealing with "multi-word synonyms" of 2 words. in general, this
approach requires that you pick some some N such that you are garunteed no
synonym contains more then N-1 words, and set the token positions to...
token: use powerquery for advanced searches
power query
position: 0 N N+1 2N 3N 4N
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Hungarian notation analyzer and phrase queries
Posted by Doug Cutting <cu...@apache.org>.
Paul Smith wrote:
> I have written a custom analyzer to tokenize PowerQuery into 'power',
> 'query, and 'powerquery' and change the position increment to 0, but I
> don't quite get the desired behavior. The phrase query "use power query
> for advanced searches" does not match, but "use query for advanced
> searches", and "use power for advanced searches" do.
>
> Any ideas?
Another approach would be to index this as:
token: use power query for advanced searches
powerquery
position: 0 1 2 3 4 5
Then use phrase queries with slop=1, to permit a one-token gap when
someone searches for "use powerquery for advanced searches".
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org