You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "m.harig" <m....@gmail.com> on 2009/08/04 07:32:12 UTC
Searching doubt
hello all
thanks for lucene............ this is my doubt , am searching for
a keyword "about us" from my lucene index , am not getting the results what
i want , since the urls are formed like the below
http://www........./aboutus/.....abc/
http://www........./aboutus/.....xyz/
http://www........./aboutus/.....def/
if i search "aboutus" , the results coming up correctly. Please
any1 suggest me how to handle this situation.
--
View this message in context: http://www.nabble.com/Searching-doubt-tp24802552p24802552.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Phil Whelan <ph...@gmail.com>.
(sorry, tangent. I'll be quick)
On Tue, Aug 4, 2009 at 8:42 AM, Shai Erera<se...@gmail.com> wrote:
> Interesting ... I don't have access to a Japanese dictionary, so I just
> extract bi-grams.
Shai - if you're interested in parsing Japanese, check out Kakasi. It
can split into words and convert Kanji->Katakana/Hirugana/Romaji -
after which I would index them all.
http://kakasi.namazu.org/
http://www.kawao.com/java/kakasi/api/com/kawao/kakasi/Kakasi.html
http://kakasi.namazu.org/stable/kakasi-2.3.4.tar.gz <-- contain GPL
Japanese dictionary
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Shai Erera <se...@gmail.com>.
Interesting ... I don't have access to a Japanese dictionary, so I just
extract bi-grams. But I guess that in this case, if one can access an
English dictionary (are you aware of an "open-source" one, or free one
BTW?), one can use the method you mention.
But still, doing this for every Token you meet is extremely expensive (for
Japanese is all you can do, but this case is rather special), so I'd first
make sure I can pinpoint the very small number of possible tokens I should
process like that.
Shai
On Tue, Aug 4, 2009 at 6:37 PM, Phil Whelan <ph...@gmail.com> wrote:
> On Tue, Aug 4, 2009 at 8:31 AM, Shai Erera<se...@gmail.com> wrote:
> > Hi Darren,
> >
> > The question was, how given a string "aboutus" in a document, you can
> return
> > that document as a result to the query "about us" (note the space). So
> we're
> > mostly discussing how to detect and then break the word "aboutus" to two
> > words.
>
> When traversing Japanese text you have a use a similar algorithm to
> searching a maze (keep left and retrace your steps). It's possible to
> go a long way along sentence before you find the tokens you've already
> picked out are invalid. Rough example...
>
> thereallibrary
> there allibrary
> there all i brary (fail)
> the reallibrary
> the real library
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Searching doubt
Posted by Phil Whelan <ph...@gmail.com>.
On Tue, Aug 4, 2009 at 8:31 AM, Shai Erera<se...@gmail.com> wrote:
> Hi Darren,
>
> The question was, how given a string "aboutus" in a document, you can return
> that document as a result to the query "about us" (note the space). So we're
> mostly discussing how to detect and then break the word "aboutus" to two
> words.
When traversing Japanese text you have a use a similar algorithm to
searching a maze (keep left and retrace your steps). It's possible to
go a long way along sentence before you find the tokens you've already
picked out are invalid. Rough example...
thereallibrary
there allibrary
there all i brary (fail)
the reallibrary
the real library
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Matthew Hall <mh...@informatics.jax.org>.
Well.. search on both anyhow.
"about us" OR "aboutus" should hit the spot I think.
Matt
Ian Lea wrote:
>> The question was, how given a string "aboutus" in a document, you can return
>> that document as a result to the query "about us" (note the space). So we're
>> mostly discussing how to detect and then break the word "aboutus" to two
>> words.
>>
>
> I haven't really been following this thread so apologies if way off
> target, but reading the above makes me wonder if it can simply be
> reversed: remove the space from "about us" and search on that.
>
>
> --
> Ian.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Matthew Hall
Software Engineer
Mouse Genome Informatics
mhall@informatics.jax.org
(207) 288-6012
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Ian Lea <ia...@gmail.com>.
> The question was, how given a string "aboutus" in a document, you can return
> that document as a result to the query "about us" (note the space). So we're
> mostly discussing how to detect and then break the word "aboutus" to two
> words.
I haven't really been following this thread so apologies if way off
target, but reading the above makes me wonder if it can simply be
reversed: remove the space from "about us" and search on that.
--
Ian.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by da...@ontrenet.com.
Ahhhh, ok. Interesting problem there as well.
I'll think on that one some too!
cheers.
> Hi Darren,
>
> The question was, how given a string "aboutus" in a document, you can
> return
> that document as a result to the query "about us" (note the space). So
> we're
> mostly discussing how to detect and then break the word "aboutus" to two
> words.
>
> What you wrote though seems interesting as well, only I think not related
> to
> Harig's original question. Maybe he'll be interested in that too though.
>
> Shai
>
> On Tue, Aug 4, 2009 at 6:27 PM, <da...@ontrenet.com> wrote:
>
>> Just catching this thread, but if I understand what is being asked I can
>> share how I do multi-word phrase matching. If that's not what's wanted,
>> pardons!
>>
>> Ok, I load an entire dictionary into a lucene index, phrases and all.
>>
>> When I'm scanning some text, I do lookups in this dictionary index using
>> one word at a time with the word _at the beginning_ of the indexed field
>> only. This returns all words/phrases beginning with the word I searched
>> for.
>>
>> I then scan the rest of the input text and compare it to the longest
>> matching phrase in my lucene results. That then becomes a meaningful
>> token.
>>
>> Input text:
>> "The President of the United States lives in the White House"
>>
>> Tokens:
>> "The"
>> "President of the United States"
>> "lives"
>> "in"
>> "the"
>> "White House"
>>
>> Term: "President"
>> Result:
>> "President of a Company"
>> "President"
>> "President of the United States"
>>
>> Take the longest match.
>>
>> HTH,
>> Darren
>>
>>
>>
>> > On Tue, Aug 4, 2009 at 3:56 AM, Shai Erera<se...@gmail.com> wrote:
>> >> 2) Use a dictionary (real dictionary), and search it for every
>> >> substring,
>> >> e.g. "a", "ab", "abo" ... "about" etc. If you find a match, split it
>> >> there.
>> >> This needs some fine tuning, like checking if the rest is also a word
>> >> and if
>> >> the full string is also a word, so that you don't break up meaningful
>> >> words.
>> >> You'll need to get a dictionary for that.
>> >
>> > I do not have a solution to this, but it strikes me as very similar to
>> > they way you traverse Japanese to break words, since that has no
>> > spaces. Is there a Japanese tokenizer and, if so, does it handle this?
>> > If so, you could replace the Japanese dictionary with an English
>> > dictionary. Just a random thought had that might / might not help.
>> >
>> > Phil
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by "m.harig" <m....@gmail.com>.
Thanks all,
but how nutch handle this problem? am aware of nutch but not in
depth. If i search the keyword "about us" , nutch gives me exactly what i
want. Is there any scoring techinques? please let me know.
--
View this message in context: http://www.nabble.com/Searching-doubt-tp24802552p24820550.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Shai Erera <se...@gmail.com>.
I had suggested that in my first response, but I think Harig's problem is
that those words are not known in advance. Therefore, facing the query
"about us" and converting it to "aboutus" is simple, but what about queries
like "united states", or "united states of america"? Should they be
'grouped' together always? And if the query contains more than 2 words,
should every couple be grouped? That will lead to n^2 terms to process for
each query, which is very expensive.
The dictionary approach can work in both directions. I.e., query("about us")
--> query("aboutus", "about us") can work as well if someone creates a
mapping from "about us" to "aboutus". But this dictionary need to be built
in advance, or at least add terms to it as you encounter more cases.
Harig - I don't think there's a magic solution here. If you don't know what
to expect, you can only do a "best effort" solution. There were a couple of
suggestions proposed, and I think you should evaluate them and decide what
can work for you. If you prefer any particular solution over the others,
then let us know and we'll try to help further.
>From all the solutions proposed, my personal preference is to use a
dictionary and during indexing try to break those words according to
dictionary terms. If that's not possible (e.g. there isn't a dictionary you
can use), you can try to break the words into sub strings. But I'd
definitely would go for a solution on the indexing side, since usually we
prefer to pay more during indexing than during search.
BTW, another solution that pops into mind is to use an NGram Analyzer, that
will extract 'grams' of words. So the word "aboutus" will be converted
(using N=3) to: "abo", "bou", "out", "utu", "tus". Then the query "about us"
will be converted (using N=3 again) to: "abo", "bou", "out", "us" (assuming
you split words on whitespace) and this document will match. The downside of
this is that other documents will match too (e.g. documents with the word
"out"), but you can improve the scoring by running the following query
"about us \"about\" \"us\"". Notice that the last two words are put in a
phrase. So docs that contain the word "about" will be scored higher than
docs that contain just the word "out". Also, notice that with this approach,
the default search operator is recommended to be OR, as requesting AND on
NGrams may hurt recall severely.
Shai
On Tue, Aug 4, 2009 at 7:34 PM, N Hira <nh...@cognocys.com> wrote:
>
> Good summary, Shai.
>
> I've missed some of this thread as well, but does anyone know what happened
> to the suggestion about query manipulation?
>
> e.g., query (about us) => query("about us", "aboutus")
> query(credit card) => query("credit card", "creditcard")
>
> Regards,
>
> -h
>
>
>
> ----- Original Message ----
> From: Shai Erera <se...@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Tuesday, August 4, 2009 10:31:46 AM
> Subject: Re: Searching doubt
>
> Hi Darren,
>
> The question was, how given a string "aboutus" in a document, you can
> return
> that document as a result to the query "about us" (note the space). So
> we're
> mostly discussing how to detect and then break the word "aboutus" to two
> words.
>
> What you wrote though seems interesting as well, only I think not related
> to
> Harig's original question. Maybe he'll be interested in that too though.
>
> Shai
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Searching doubt
Posted by N Hira <nh...@cognocys.com>.
Good summary, Shai.
I've missed some of this thread as well, but does anyone know what happened to the suggestion about query manipulation?
e.g., query (about us) => query("about us", "aboutus")
query(credit card) => query("credit card", "creditcard")
Regards,
-h
----- Original Message ----
From: Shai Erera <se...@gmail.com>
To: java-user@lucene.apache.org
Sent: Tuesday, August 4, 2009 10:31:46 AM
Subject: Re: Searching doubt
Hi Darren,
The question was, how given a string "aboutus" in a document, you can return
that document as a result to the query "about us" (note the space). So we're
mostly discussing how to detect and then break the word "aboutus" to two
words.
What you wrote though seems interesting as well, only I think not related to
Harig's original question. Maybe he'll be interested in that too though.
Shai
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Shai Erera <se...@gmail.com>.
Hi Darren,
The question was, how given a string "aboutus" in a document, you can return
that document as a result to the query "about us" (note the space). So we're
mostly discussing how to detect and then break the word "aboutus" to two
words.
What you wrote though seems interesting as well, only I think not related to
Harig's original question. Maybe he'll be interested in that too though.
Shai
On Tue, Aug 4, 2009 at 6:27 PM, <da...@ontrenet.com> wrote:
> Just catching this thread, but if I understand what is being asked I can
> share how I do multi-word phrase matching. If that's not what's wanted,
> pardons!
>
> Ok, I load an entire dictionary into a lucene index, phrases and all.
>
> When I'm scanning some text, I do lookups in this dictionary index using
> one word at a time with the word _at the beginning_ of the indexed field
> only. This returns all words/phrases beginning with the word I searched
> for.
>
> I then scan the rest of the input text and compare it to the longest
> matching phrase in my lucene results. That then becomes a meaningful
> token.
>
> Input text:
> "The President of the United States lives in the White House"
>
> Tokens:
> "The"
> "President of the United States"
> "lives"
> "in"
> "the"
> "White House"
>
> Term: "President"
> Result:
> "President of a Company"
> "President"
> "President of the United States"
>
> Take the longest match.
>
> HTH,
> Darren
>
>
>
> > On Tue, Aug 4, 2009 at 3:56 AM, Shai Erera<se...@gmail.com> wrote:
> >> 2) Use a dictionary (real dictionary), and search it for every
> >> substring,
> >> e.g. "a", "ab", "abo" ... "about" etc. If you find a match, split it
> >> there.
> >> This needs some fine tuning, like checking if the rest is also a word
> >> and if
> >> the full string is also a word, so that you don't break up meaningful
> >> words.
> >> You'll need to get a dictionary for that.
> >
> > I do not have a solution to this, but it strikes me as very similar to
> > they way you traverse Japanese to break words, since that has no
> > spaces. Is there a Japanese tokenizer and, if so, does it handle this?
> > If so, you could replace the Japanese dictionary with an English
> > dictionary. Just a random thought had that might / might not help.
> >
> > Phil
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Searching doubt
Posted by da...@ontrenet.com.
Just catching this thread, but if I understand what is being asked I can
share how I do multi-word phrase matching. If that's not what's wanted,
pardons!
Ok, I load an entire dictionary into a lucene index, phrases and all.
When I'm scanning some text, I do lookups in this dictionary index using
one word at a time with the word _at the beginning_ of the indexed field
only. This returns all words/phrases beginning with the word I searched
for.
I then scan the rest of the input text and compare it to the longest
matching phrase in my lucene results. That then becomes a meaningful
token.
Input text:
"The President of the United States lives in the White House"
Tokens:
"The"
"President of the United States"
"lives"
"in"
"the"
"White House"
Term: "President"
Result:
"President of a Company"
"President"
"President of the United States"
Take the longest match.
HTH,
Darren
> On Tue, Aug 4, 2009 at 3:56 AM, Shai Erera<se...@gmail.com> wrote:
>> 2) Use a dictionary (real dictionary), and search it for every
>> substring,
>> e.g. "a", "ab", "abo" ... "about" etc. If you find a match, split it
>> there.
>> This needs some fine tuning, like checking if the rest is also a word
>> and if
>> the full string is also a word, so that you don't break up meaningful
>> words.
>> You'll need to get a dictionary for that.
>
> I do not have a solution to this, but it strikes me as very similar to
> they way you traverse Japanese to break words, since that has no
> spaces. Is there a Japanese tokenizer and, if so, does it handle this?
> If so, you could replace the Japanese dictionary with an English
> dictionary. Just a random thought had that might / might not help.
>
> Phil
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Phil Whelan <ph...@gmail.com>.
On Tue, Aug 4, 2009 at 3:56 AM, Shai Erera<se...@gmail.com> wrote:
> 2) Use a dictionary (real dictionary), and search it for every substring,
> e.g. "a", "ab", "abo" ... "about" etc. If you find a match, split it there.
> This needs some fine tuning, like checking if the rest is also a word and if
> the full string is also a word, so that you don't break up meaningful words.
> You'll need to get a dictionary for that.
I do not have a solution to this, but it strikes me as very similar to
they way you traverse Japanese to break words, since that has no
spaces. Is there a Japanese tokenizer and, if so, does it handle this?
If so, you could replace the Japanese dictionary with an English
dictionary. Just a random thought had that might / might not help.
Phil
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Shai Erera <se...@gmail.com>.
If you don't know which tokens you'll face, then it's really a much harder
problem. If you know where the token is, e.g. it's always in
http://some.example.site/a/b/<here will be the token to break>/index.html,
then it eases the task a bit. Otherwise you'll need to search every single
token produced. I can think of several ways to break "aboutus" to "about
us", or any other sequence for that matter:
1) Break it to "a boutus", "ab outus" ... "about us", "aboutu s", index all
of them in the same position. Expensive though. This I'd recommend only if
you know where this token is located (otherwise it will explode your term
dictionary).
2) Use a dictionary (real dictionary), and search it for every substring,
e.g. "a", "ab", "abo" ... "about" etc. If you find a match, split it there.
This needs some fine tuning, like checking if the rest is also a word and if
the full string is also a word, so that you don't break up meaningful words.
You'll need to get a dictionary for that.
The key though - do you know exactly where this token is? Otherwise, every
solution will be a killer to performance.
Shai
On Tue, Aug 4, 2009 at 12:59 PM, m.harig <m....@gmail.com> wrote:
>
> Thanks ,
>
> i've noticed that , but the code is for known tokens, how do i
> do it for dynamic tokens , meaning , i don't know the urls , someone picked
> up the urls and i'll index it. Is there any technique to use while indexing
> ? am using lucene 2.4.0 version. Please suggest me.
> --
> View this message in context:
> http://www.nabble.com/Searching-doubt-tp24802552p24805609.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Searching doubt
Posted by "m.harig" <m....@gmail.com>.
Thanks ,
i've noticed that , but the code is for known tokens, how do i
do it for dynamic tokens , meaning , i don't know the urls , someone picked
up the urls and i'll index it. Is there any technique to use while indexing
? am using lucene 2.4.0 version. Please suggest me.
--
View this message in context: http://www.nabble.com/Searching-doubt-tp24802552p24805609.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Shai Erera <se...@gmail.com>.
Well, if you have more cases like "aboutus", then I think the TokenFilter
approach will help you. You should create your own Analyzer which receives
another Analyzer as argument, and impl it's tokenStream() like this (it's
the general idea):
public TokenStream tokenStream(String fld, Reader reader) {
TokenStream ts = wrappedAnalyzer.tokenStream(fld, reader);
ts = new SynonymTokenFilter(ts);
return ts;
}
You can look in contrib/highlighter for a SynonymAnalyzer (an inner class of
HighlighterTest.java). Maybe you can copy some of the code from there.
The idea for this SynonymTokenFilter is to get a map like this:
aboutus --> [about us]
creditcards --> [credit cards], [visa] (the latter is just an example for
another synonym you can use)
....
Then when the filter encounters a Token, it looks it up in the map and
replaces it w/ the ones in the map, or returns it and its synonyms. You
should make sure all synonyms are placed in the same position (i.e.,
setPositionIncrement(0)).
If you know these strings appear just in a "url" field, you can apply this
token filter for just the url field, and save a lot of unnecessary token
lookups in the map. To do this, you should extend Analyzer and override
tokenStream and return this filter for just the "url" field (notice
tokenStream receives the field name as arg).
Hope this helps,
Shai
On Tue, Aug 4, 2009 at 10:29 AM, m.harig <m....@gmail.com> wrote:
>
> Thanks for your reply,
>
> my original code snippet is
>
> IndexSearcher searcher = new IndexSearcher(indexDir);
> Analyzer analyzer = new StopAnalyzer();
>
> BooleanClause.Occur[] flags = { BooleanClause.Occur.SHOULD,
> BooleanClause.Occur.SHOULD,
> BooleanClause.Occur.SHOULD,
> BooleanClause.Occur.SHOULD,
> BooleanClause.Occur.SHOULD };
>
> // for multiple keyword search
> Query query = MultiFieldQueryParser.parse(qryStr, new
> String[] {
> "title", "path", "contents", "summary",
> "size" }, flags,
> analyzer);
>
> Hits hits = getHits(qryStr, searcher, query, "path");
>
>
> for (int inc = 0; inc < hits.length(); inc++) {
>
> Document doc = hits.doc(inc);
> ///////
>
> }
>
>
> // private method to get hits by using phrase query
>
> private Hits getHits(String qryStr, Searcher searcher, Query
> queryNorm,
> String phraseQryTerm) throws Exception {
>
> Hits hits = null;
>
> // for single word search
> PhraseQuery queryPhrase = new PhraseQuery();
> queryPhrase.setBoost(4.0f);
> queryPhrase.setSlop(1);
> queryPhrase.add(new Term(phraseQryTerm,
> qryStr.toLowerCase()));
>
> hits = searcher.search(queryPhrase);
>
> return hits;
> }
>
>
> if i use this code for my search , it gives me the unwanted search hits ,
>
> as i mentioned , if i search for "about us" , this is an example ,
> there may be more number of urls like this , for example , "credit cards"
> ,
> "book marks" , how do i handle it ?
> --
> View this message in context:
> http://www.nabble.com/Searching-doubt-tp24802552p24803560.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Searching doubt
Posted by "m.harig" <m....@gmail.com>.
Thanks for your reply,
my original code snippet is
IndexSearcher searcher = new IndexSearcher(indexDir);
Analyzer analyzer = new StopAnalyzer();
BooleanClause.Occur[] flags = { BooleanClause.Occur.SHOULD,
BooleanClause.Occur.SHOULD, BooleanClause.Occur.SHOULD,
BooleanClause.Occur.SHOULD, BooleanClause.Occur.SHOULD };
// for multiple keyword search
Query query = MultiFieldQueryParser.parse(qryStr, new String[] {
"title", "path", "contents", "summary", "size" }, flags,
analyzer);
Hits hits = getHits(qryStr, searcher, query, "path");
for (int inc = 0; inc < hits.length(); inc++) {
Document doc = hits.doc(inc);
///////
}
// private method to get hits by using phrase query
private Hits getHits(String qryStr, Searcher searcher, Query queryNorm,
String phraseQryTerm) throws Exception {
Hits hits = null;
// for single word search
PhraseQuery queryPhrase = new PhraseQuery();
queryPhrase.setBoost(4.0f);
queryPhrase.setSlop(1);
queryPhrase.add(new Term(phraseQryTerm, qryStr.toLowerCase()));
hits = searcher.search(queryPhrase);
return hits;
}
if i use this code for my search , it gives me the unwanted search hits ,
as i mentioned , if i search for "about us" , this is an example ,
there may be more number of urls like this , for example , "credit cards" ,
"book marks" , how do i handle it ?
--
View this message in context: http://www.nabble.com/Searching-doubt-tp24802552p24803560.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Shai Erera <se...@gmail.com>.
I don't see that you use the Analyzer anywhere (i.e. it's created by not
used?).
Also, the wildcard query you create may be very inefficient, as it will
expand all the terms under the DEFAULT_FIELD. If the DEFAULT_FIELD is the
field where all your "default searchable" terms are indexed, there could be
tens of thousands to millions of terms (depends on how many documents you
add).
Also, what exactly do you expect that this query will return? All docs? Is
the word "aboutus" there?
Maybe you should do the following:
WildcardQuery wq = new WildcardQuery(new Term(DEFAULT_FIELD, "about"));
searcher.search(wq);
This should work, if the urls are indexed under the DEFAULT_FIELD.
Shai
On Tue, Aug 4, 2009 at 9:46 AM, m.harig <m....@gmail.com> wrote:
>
> Thanks
>
> This is my codw snippet
>
> IndexSearcher searcher = new IndexSearcher(indexDir);
> Analyzer analyzer = new StopAnalyzer();
>
> WildcardQuery query = new WildcardQuery(new
> Term(DEFAULT_FIELD));
> searcher.search(query);
>
> but its returning 0 hits as results , where am i wrong? please help me out
> of this.
>
> --
> View this message in context:
> http://www.nabble.com/Searching-doubt-tp24802552p24803073.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Searching doubt
Posted by "m.harig" <m....@gmail.com>.
Thanks
This is my codw snippet
IndexSearcher searcher = new IndexSearcher(indexDir);
Analyzer analyzer = new StopAnalyzer();
WildcardQuery query = new WildcardQuery(new Term(DEFAULT_FIELD));
searcher.search(query);
but its returning 0 hits as results , where am i wrong? please help me out
of this.
--
View this message in context: http://www.nabble.com/Searching-doubt-tp24802552p24803073.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Searching doubt
Posted by Shai Erera <se...@gmail.com>.
I can think of another approach - during indexing, capture the word
"aboutus" and index it as "about us" and "aboutus" in the same position.
That way both queries will work. You'd need to write your own TokenFilter,
maybe a SynonymTokenFilter (since this reminds me of "synonyms" usage) that
accept a list of words and their "matching counterparts" ... something along
those lines.
Another thing, which is basically the same, but from the other side -->
check the query string for such words and replace them w/ what you want. So
you can search for "about us" in the query string and replace it w/
"aboutus".
I prefer the first approach since it's much more performing, but is more
complicated than the second one. It will also allow you to handle other
cases like that.
Shai
On Tue, Aug 4, 2009 at 9:20 AM, Anshum <an...@gmail.com> wrote:
> Hi Harig,
> What you are trying to do is search for 2 tokens as one. You'd have to
> index
> the url as you want for the token to be searchable. Else you might try a
> wildcard query .
>
> --
> Anshum Gupta
> Naukri Labs!
> http://ai-cafe.blogspot.com
>
> The facts expressed here belong to everybody, the opinions to me. The
> distinction is yours to draw............
>
>
> On Tue, Aug 4, 2009 at 11:02 AM, m.harig <m....@gmail.com> wrote:
>
> >
> > hello all
> >
> > thanks for lucene............ this is my doubt , am searching
> for
> > a keyword "about us" from my lucene index , am not getting the results
> what
> > i want , since the urls are formed like the below
> >
> > http://www........./aboutus/.....abc/
> > http://www........./aboutus/.....xyz/
> > http://www........./aboutus/.....def/
> >
> > if i search "aboutus" , the results coming up correctly. Please
> > any1 suggest me how to handle this situation.
> > --
> > View this message in context:
> > http://www.nabble.com/Searching-doubt-tp24802552p24802552.html
> > Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
Re: Searching doubt
Posted by Anshum <an...@gmail.com>.
Hi Harig,
What you are trying to do is search for 2 tokens as one. You'd have to index
the url as you want for the token to be searchable. Else you might try a
wildcard query .
--
Anshum Gupta
Naukri Labs!
http://ai-cafe.blogspot.com
The facts expressed here belong to everybody, the opinions to me. The
distinction is yours to draw............
On Tue, Aug 4, 2009 at 11:02 AM, m.harig <m....@gmail.com> wrote:
>
> hello all
>
> thanks for lucene............ this is my doubt , am searching for
> a keyword "about us" from my lucene index , am not getting the results what
> i want , since the urls are formed like the below
>
> http://www........./aboutus/.....abc/
> http://www........./aboutus/.....xyz/
> http://www........./aboutus/.....def/
>
> if i search "aboutus" , the results coming up correctly. Please
> any1 suggest me how to handle this situation.
> --
> View this message in context:
> http://www.nabble.com/Searching-doubt-tp24802552p24802552.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>