You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Akanksha Baid <ak...@gmail.com> on 2007/08/09 09:34:55 UTC

frequent phrases

I was wondering if there is a "search based" method to find the top-k
frequent phrases in a set of documents.( I do not have a particular phrase
in mind so PhraseQuery can probably be ruled out).
I have implemented something that works using termvectors and termpositions
but the performance is not great so far since I am basically iterating
multiple times and hacking my way around. I was wondering if an API exists
for finding frequent phrases and/or if someone could point me to some code
for the same.

Thanks.

Re: frequent phrases

Posted by Mathieu Lecarme <ma...@garambrogne.net>.
some tools exist for finding duplicated parts in document.
You split document in phrase, and build ngram with word. If you wont  
complete phrase, work with all words, for a partial, work with 5  
words ngram, for example. ngram list is convert to hash, and hash is  
used as an indexed Field for the document. With this trick, you can  
use phrase as you use to do with word, without using too much space.

I'm not sure to be as clear as I want.

M.

Le 9 août 07 à 09:34, Akanksha Baid a écrit :

> I was wondering if there is a "search based" method to find the top-k
> frequent phrases in a set of documents.( I do not have a particular  
> phrase
> in mind so PhraseQuery can probably be ruled out).
> I have implemented something that works using termvectors and  
> termpositions
> but the performance is not great so far since I am basically iterating
> multiple times and hacking my way around. I was wondering if an API  
> exists
> for finding frequent phrases and/or if someone could point me to  
> some code
> for the same.
>
> Thanks.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: frequent phrases

Posted by karl wettin <ka...@gmail.com>.
9 aug 2007 kl. 09.34 skrev Akanksha Baid:

> I was wondering if there is a "search based" method to find the top-k
> frequent phrases in a set of documents.( I do not have a particular  
> phrase
> in mind so PhraseQuery can probably be ruled out).
> I have implemented something that works using termvectors and  
> termpositions
> but the performance is not great so far since I am basically iterating
> multiple times and hacking my way around. I was wondering if an API  
> exists
> for finding frequent phrases and/or if someone could point me to  
> some code
> for the same.

I think this is the closest thing available in the issue tracker:

https://issues.apache.org/jira/browse/LUCENE-725

-- 
karl

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: frequent phrases

Posted by karl wettin <ka...@gmail.com>.
9 aug 2007 kl. 09.34 skrev Akanksha Baid:

> I was wondering if there is a "search based" method to find the top-k
> frequent phrases in a set of documents.( I do not have a particular  
> phrase
> in mind so PhraseQuery can probably be ruled out).
> I have implemented something that works using termvectors and  
> termpositions
> but the performance is not great so far since I am basically iterating
> multiple times and hacking my way around. I was wondering if an API  
> exists
> for finding frequent phrases and/or if someone could point me to  
> some code
> for the same.

By the way, if you explain what you aim to use it for, then you might  
get
more elaborate responses.


-- 
karl



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org