You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by teko <te...@gmail.com> on 2014/05/08 15:03:53 UTC
How to locate a Phrase inside text (like a Browser text searcher)
Hi, someone can help me with it??
I need do a search to locate a phrase inside text, but, I need locate this
phrase on texts like that:
'John Mail' <- phrase I want locate
' Good Morning John Mail how are you? ' < I need find this phrase here
' Good MorningJohn Mail how are you? ' < here too
' GoodMorning John Mailhow are you? ' < and here
I tried using with 'WhiteSpaceAnalyzer' and 'QueryParser'... but not work
(locate just in the first sample above... but not the others)
Please, I really need help with it!
Thanks (note: Sorry my english!! xD)
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to locate a Phrase inside text (like a Browser text
searcher)
Posted by teko <te...@gmail.com>.
Wow man!!
Forget what I said before!! I did tries using your method... well, to
generate the index, really, it's still a bit more slow (1/2 minutes more),
but, in query... man, It's work very well, and, fast, very fast!!
Really, here is so fast that what generate the bottleneck, is the write
inside DB... haha man, I think this is the method. I need do more some test
yet, but... I realy think I will use this method (NGram + WildcardQuery)
Emanuel Buzek, thanks man.. thanks very much!! You don't know how you
helped me with it!
I'll do more test now and I'll post here the result.
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075p4136272.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to locate a Phrase inside text (like a Browser text
searcher)
Posted by teko <te...@gmail.com>.
Emanuel Buzek,
Well, I tried using the method 'ShingleFilter' first, and I thought it
worked well, but, at last, it still did not work like I want..
So, I tried use NGram... I created a new analyzer to use it, and, I did a
test... Well, it works, but, I still need do some manually validation to
each result. Now, it works and return exactly all I want, but, is a bit
slow...
Let me explain what I do...
1 - First, I create a personal Analyzer, using NGramTokenizer as tolken...
in this class, contains too a StopFilter and LowercaseFitler... the default.
2 - I do query, but, I didn't use a 'literal' phrase(like that: "\"John
Mayer\"")... I search this way:
-> to locate 'John Mayer' I do: queryParser.parser("John Mayer");
2.1 - It will return to me all documents that have occurrences of this
two words... but, still is not I want yet...
2.2 - I get the Document, read the text inside document, and now, I search
(using regex) the exact combination:
-> to locate 'John Mayer' I do: "John( |)Mayer" -> regex
The problem, is the processe time... I did a test with a thousand phrases to
locate... and well, it takes a bit more than 40 minutes.
Is a time too long man.. Now, I'm trying solve this question... do you have
some tips?
Thanks...
Note: I edited the title and I removed: 'SOLVED'
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLVED-How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075p4136236.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to locate a Phrase inside text (like a Browser text searcher)
Posted by Emanuel Buzek <em...@roke.cz>.
Hi Teko,
sure - I use Lucene though elasticsearch, but I suppose that doesnt make a
difference in this situation. I needed something like what you were trying
to accomplish - basically to search any substring... wildcarded queries
worked but were kind of slow.
This is my analyzer that works for me perfectly:
"settings" : {
"analysis" : {
"analyzer" : {
"default" : {
"type" : "custom",
"filter" : ["icu_folding"],
"tokenizer" : "nGram"
}
},
"tokenizer": {
"nGram" : {
"type" : "nGram",
"min_gram" : 2,
"max_gram" : 20,
"token_chars" : ["letter", "digit", "punctuation", "symbol"]
}
}
}
},
it basically indexes any substrings made of letters, digits, punctuation
and symbols of length 2..20.
but if that previous solution works for you, that's good :)
2014-05-14 20:23 GMT+02:00 teko <te...@gmail.com>:
> *Emanuel Buzek,
>
> Can you explain how you use NGram?? Did you create a Analyzer? is it??
> Sorry, but, I really don't have a great knowledge about Lucene...
>
> Thank advance!
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/SOLVED-How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075p4135607.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Emanuel Buzek
Software Engineer, ROKE.cz <http://www.roke.cz>
tel: +420 776 54 26 26
Re: How to locate a Phrase inside text (like a Browser text
searcher)
Posted by teko <te...@gmail.com>.
*Emanuel Buzek,
Can you explain how you use NGram?? Did you create a Analyzer? is it??
Sorry, but, I really don't have a great knowledge about Lucene...
Thank advance!
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLVED-How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075p4135607.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to locate a Phrase inside text (like a Browser text searcher)
Posted by Emanuel Buzek <em...@roke.cz>.
I was trying to solve pretty much the same thing few weeks back and I ended
up using the NGram tokenizer. Although it made my index much larger (the
index grew 15x), the fulltext queries are pretty fast and I don't have to
use wildcards in queries.
http://lucene.apache.org/core/4_4_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramTokenizer.html
cheers, Ema
2014-05-13 2:39 GMT+02:00 Michael Sokolov <ms...@safaribooksonline.com>:
> ShingleFilter can help with this; it concatenates neighboring tokens. So
> a search for "good morning john" becomes a search for
>
> "goodmorning john" OR
> "good morningjohn" OR
> "good morning john"
>
> it makes your index much bigger because of all the terms, but you may find
> it's worth the cost
>
> -Mike
>
>
> On 5/11/2014 9:46 PM, Jack Krupansky wrote:
>
>> The word delimiter filter can help for "MorningJohn" by setting its
>> option to split on case change.
>>
>> You might be able to handle "Mailhow" using the
>> DictionaryCompoundWordTokenFilter, but that requires that you create a
>> complete dictionary of terms that can split off. That's not very practical.
>> In truth, Lucene/Solr doesn't have a good out of the box solution for this
>> use case.
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: teko
>> Sent: Thursday, May 8, 2014 9:03 AM
>> To: java-user@lucene.apache.org
>> Subject: How to locate a Phrase inside text (like a Browser text searcher)
>>
>> Hi, someone can help me with it??
>> I need do a search to locate a phrase inside text, but, I need locate this
>> phrase on texts like that:
>> 'John Mail' <- phrase I want locate
>> ' Good Morning John Mail how are you? ' < I need find this phrase here
>> ' Good MorningJohn Mail how are you? ' < here too
>> ' GoodMorning John Mailhow are you? ' < and here
>>
>> I tried using with 'WhiteSpaceAnalyzer' and 'QueryParser'... but not work
>> (locate just in the first sample above... but not the others)
>>
>> Please, I really need help with it!
>> Thanks (note: Sorry my english!! xD)
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.
>> nabble.com/How-to-locate-a-Phrase-inside-text-like-a-
>> Browser-text-searcher-tp4135075.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Emanuel Buzek
Software Engineer, ROKE.cz <http://www.roke.cz>
tel: +420 776 54 26 26
Re: How to locate a Phrase inside text (like a Browser text searcher)
Posted by Jack Krupansky <ja...@basetechnology.com>.
True, for the first two use cases, but as I indicated, the third use case is
problematic since the token needs to be split. The n-gram solution does seem
to cover it though, sort of.
The n-gram solution doesn't cover "good morning, john" or "good morning -
john", but that could be handled by having a tokenizer that that simply
ignored punctuation and whitespace and generated one big original token and
then n-grammed it based on some maximal query phrase size. And... the
original requirement spec didn't list that as a use case anyway.
-- Jack Krupansky
-----Original Message-----
From: Michael Sokolov
Sent: Monday, May 12, 2014 8:39 PM
To: java-user@lucene.apache.org
Subject: Re: How to locate a Phrase inside text (like a Browser text
searcher)
ShingleFilter can help with this; it concatenates neighboring tokens.
So a search for "good morning john" becomes a search for
"goodmorning john" OR
"good morningjohn" OR
"good morning john"
it makes your index much bigger because of all the terms, but you may
find it's worth the cost
-Mike
On 5/11/2014 9:46 PM, Jack Krupansky wrote:
> The word delimiter filter can help for "MorningJohn" by setting its option
> to split on case change.
>
> You might be able to handle "Mailhow" using the
> DictionaryCompoundWordTokenFilter, but that requires that you create a
> complete dictionary of terms that can split off. That's not very
> practical. In truth, Lucene/Solr doesn't have a good out of the box
> solution for this use case.
>
> -- Jack Krupansky
>
> -----Original Message----- From: teko
> Sent: Thursday, May 8, 2014 9:03 AM
> To: java-user@lucene.apache.org
> Subject: How to locate a Phrase inside text (like a Browser text searcher)
>
> Hi, someone can help me with it??
> I need do a search to locate a phrase inside text, but, I need locate this
> phrase on texts like that:
> 'John Mail' <- phrase I want locate
> ' Good Morning John Mail how are you? ' < I need find this phrase here
> ' Good MorningJohn Mail how are you? ' < here too
> ' GoodMorning John Mailhow are you? ' < and here
>
> I tried using with 'WhiteSpaceAnalyzer' and 'QueryParser'... but not work
> (locate just in the first sample above... but not the others)
>
> Please, I really need help with it!
> Thanks (note: Sorry my english!! xD)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to locate a Phrase inside text (like a Browser text
searcher)
Posted by teko <te...@gmail.com>.
wow!! Thanks Michael!! It's works perfectly! thanks man!!
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075p4135449.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to locate a Phrase inside text (like a Browser text searcher)
Posted by Michael Sokolov <ms...@safaribooksonline.com>.
ShingleFilter can help with this; it concatenates neighboring tokens.
So a search for "good morning john" becomes a search for
"goodmorning john" OR
"good morningjohn" OR
"good morning john"
it makes your index much bigger because of all the terms, but you may
find it's worth the cost
-Mike
On 5/11/2014 9:46 PM, Jack Krupansky wrote:
> The word delimiter filter can help for "MorningJohn" by setting its
> option to split on case change.
>
> You might be able to handle "Mailhow" using the
> DictionaryCompoundWordTokenFilter, but that requires that you create a
> complete dictionary of terms that can split off. That's not very
> practical. In truth, Lucene/Solr doesn't have a good out of the box
> solution for this use case.
>
> -- Jack Krupansky
>
> -----Original Message----- From: teko
> Sent: Thursday, May 8, 2014 9:03 AM
> To: java-user@lucene.apache.org
> Subject: How to locate a Phrase inside text (like a Browser text
> searcher)
>
> Hi, someone can help me with it??
> I need do a search to locate a phrase inside text, but, I need locate
> this
> phrase on texts like that:
> 'John Mail' <- phrase I want locate
> ' Good Morning John Mail how are you? ' < I need find this phrase here
> ' Good MorningJohn Mail how are you? ' < here too
> ' GoodMorning John Mailhow are you? ' < and here
>
> I tried using with 'WhiteSpaceAnalyzer' and 'QueryParser'... but not work
> (locate just in the first sample above... but not the others)
>
> Please, I really need help with it!
> Thanks (note: Sorry my english!! xD)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to locate a Phrase inside text (like a Browser text searcher)
Posted by Jack Krupansky <ja...@basetechnology.com>.
The word delimiter filter can help for "MorningJohn" by setting its option
to split on case change.
You might be able to handle "Mailhow" using the
DictionaryCompoundWordTokenFilter, but that requires that you create a
complete dictionary of terms that can split off. That's not very practical.
In truth, Lucene/Solr doesn't have a good out of the box solution for this
use case.
-- Jack Krupansky
-----Original Message-----
From: teko
Sent: Thursday, May 8, 2014 9:03 AM
To: java-user@lucene.apache.org
Subject: How to locate a Phrase inside text (like a Browser text searcher)
Hi, someone can help me with it??
I need do a search to locate a phrase inside text, but, I need locate this
phrase on texts like that:
'John Mail' <- phrase I want locate
' Good Morning John Mail how are you? ' < I need find this phrase here
' Good MorningJohn Mail how are you? ' < here too
' GoodMorning John Mailhow are you? ' < and here
I tried using with 'WhiteSpaceAnalyzer' and 'QueryParser'... but not work
(locate just in the first sample above... but not the others)
Please, I really need help with it!
Thanks (note: Sorry my english!! xD)
--
View this message in context:
http://lucene.472066.n3.nabble.com/How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to locate a Phrase inside text (like a Browser text searcher)
Posted by Jose Carlos Canova <jo...@gmail.com>.
try to use the lucene wildcard. *John*Mail*
The analyzer is just how you want the segment terms on your index. the
query parser is how you tokenize the terms that that you want to query
against the index (something like that). But lucene allows you use the wild
card to handle with "other cases" that your tokenizer cannot detect as a
unique token.
Interesting question indeed, hope that this can help you.
http://lucene.apache.org/core/2_9_4/queryparsersyntax.html#Wildcard%20Searches
...
On Thu, May 8, 2014 at 10:03 AM, teko <te...@gmail.com> wrote:
> Hi, someone can help me with it??
> I need do a search to locate a phrase inside text, but, I need locate this
> phrase on texts like that:
> 'John Mail' <- phrase I want locate
> ' Good Morning John Mail how are you? ' < I need find this phrase here
> ' Good MorningJohn Mail how are you? ' < here too
> ' GoodMorning John Mailhow are you? ' < and here
>
> I tried using with 'WhiteSpaceAnalyzer' and 'QueryParser'... but not work
> (locate just in the first sample above... but not the others)
>
> Please, I really need help with it!
> Thanks (note: Sorry my english!! xD)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-locate-a-Phrase-inside-text-like-a-Browser-text-searcher-tp4135075.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>