You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mark Miller <ma...@gmail.com> on 2007/05/02 14:52:36 UTC
Re: Keyphrase Extraction
From what I know you generally have to pay if you want something that
does this really well. Or check out http://www.nzdl.org/Kea/
Unfortunately, the license is GPL. Really too bad; now that it is all
Java, it would make a great combo with Lucene.
- Mark
mark harwood wrote:
> I believe the code Otis is referring to is here: http://issues.apache.org/jira/browse/LUCENE-474
>
> This is index-level analysis but could be adapted to work for just a single document.
> The implementation is optimised for speed rather than being a thorough examination of phrase significance.
>
> Cheers
> Mark
>
> ----- Original Message ----
> From: Otis Gospodnetic <ot...@yahoo.com>
> To: java-user@lucene.apache.org
> Sent: Monday, 30 April, 2007 4:11:36 AM
> Subject: Re: Keyphrase Extraction
>
> Av, look at Lucene's JIRA and search for Mark Harwood. I believe he once contributed something that does this in JIRA. If you are interested in a commercial solution, I can recommend LingPipe.
>
> Otis
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Lucene Consulting - http://lucene-consulting.com/
>
>
> ----- Original Message ----
> From: "av_work@yahoo.com" <av...@yahoo.com>
> To: java-user@lucene.apache.org
> Sent: Sunday, April 29, 2007 5:24:17 PM
> Subject: Keyphrase Extraction
>
> Hi,
>
> I tried using MoreLikeThis contrib feature to extract "interesting terms" from a document. This works very well - but only for SINGLE words.
>
> I am looking for a way to extra "keyPHRASES" from a document. Is there an easy way to achieve this using Lucene index?
>
> Thanks in advance!
> Av
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
>
> ___________________________________________________________
> Yahoo! Answers - Got a question? Someone out there knows the answer. Try it
> now.
> http://uk.answers.yahoo.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Keyphrase Extraction
Posted by José Ramón Pérez Agüera <jo...@gmail.com>.
here you have a very good tool for Keyphrase Extraction. It is GNU and
easy to integrate in Lucene.
http://www.paynter.info/academia/Kea.php
best
jose
On 5/8/07, Bill Janssen <ja...@parc.com> wrote:
> Dawid Weiss wrote:
> > You could also try splitting the document into paragraphs and use Carrot2's
> > Lingo algorithm (www.carrot2.org) on a paragraph-level to extract clusters.
> > Labelling routine in Lingo should extract 'key' phrases; this analysis is
> > heavily frequency-based, but... you know, you may want to try it.
>
> Just to make sure I'm following...
>
> So you're suggesting splitting the document into paragraphs, then
> treating each paragraph as if it were a Carrot2 search result,
> performing the clustering, then looking at the label Lingo chooses for
> each cluster, and treating that label as the "key phrase"?
>
> Would DirectDocumentFeedExample be a good starting point?
>
> Bill
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
José Ramón Pérez Agüera
Dept. de Ingeniería del Software e Inteligencia Artificial
Despacho 411 tlf. 913947599
Facultad de Informática
Universidad Complutense de Madrid
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Keyphrase Extraction (via Lingo)
Posted by Bill Janssen <ja...@parc.com>.
> Dawid Weiss wrote:
> > You could also try splitting the document into paragraphs and use Carrot2's
> > Lingo algorithm (www.carrot2.org) on a paragraph-level to extract clusters.
> > Labelling routine in Lingo should extract 'key' phrases; this analysis is
> > heavily frequency-based, but... you know, you may want to try it.
>
> Just to make sure I'm following...
>
> So you're suggesting splitting the document into paragraphs, then
> treating each paragraph as if it were a Carrot2 search result,
> performing the clustering, then looking at the label Lingo chooses for
> each cluster, and treating that label as the "key phrase"?
I tried it. Not so great results, but perhaps I'm doing it wrong.
Here's my code. The input file is a text file with an ID number and
one paragraph per (long) line -- standard textual paragraphs. I'm
running on a corpus of technical papers.
Bill
-----------------------------------------------------------------
import org.carrot2.filter.lingo.common.*;
import org.carrot2.filter.lingo.lsicluster.*;
import java.io.*;
public class test {
public static void main (String[] argv) {
DefaultClusteringContext context = new DefaultClusteringContext();
try {
BufferedReader r = new BufferedReader(new FileReader(argv[0]));
String line;
while ((line = r.readLine()) != null) {
String[] parts = line.trim().split("\\s");
// there must be an easier way to split off the first token
// of a line...
if (parts.length > 1) {
String id = parts[0];
// and to glue the other parts together again...
String body = parts[1];
for (int i = 2; i < parts.length; i++) {
body = body + " " + parts[i];
}
context.addSnippet(new Snippet(id, "", body));
}
}
} catch (Exception x) {
x.printStackTrace(System.err);
}
context.setQuery("");
Cluster[] clusters = context.cluster();
for (int i = 0; i < clusters.length; i++) {
System.out.println("Cluster --");
String[] labels = clusters[i].getLabels();
for (int j = 0; j < labels.length; j++) {
System.out.println(" Label: " + labels[j]);
}
Snippet[] snippets = clusters[i].getSnippets();
System.out.println(" " + snippets.length + " snippets:");
for (int j = 0; j < snippets.length; j++) {
System.out.println(" " + snippets[j].getSnippetId() +
" -- " + snippets[j].getText());
}
}
}
}
----------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Keyphrase Extraction
Posted by Bill Janssen <ja...@parc.com>.
Dawid Weiss wrote:
> You could also try splitting the document into paragraphs and use Carrot2's
> Lingo algorithm (www.carrot2.org) on a paragraph-level to extract clusters.
> Labelling routine in Lingo should extract 'key' phrases; this analysis is
> heavily frequency-based, but... you know, you may want to try it.
Just to make sure I'm following...
So you're suggesting splitting the document into paragraphs, then
treating each paragraph as if it were a Carrot2 search result,
performing the clustering, then looking at the label Lingo chooses for
each cluster, and treating that label as the "key phrase"?
Would DirectDocumentFeedExample be a good starting point?
Bill
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Keyphrase Extraction
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
You could also try splitting the document into paragraphs and use Carrot2's
Lingo algorithm (www.carrot2.org) on a paragraph-level to extract clusters.
Labelling routine in Lingo should extract 'key' phrases; this analysis is
heavily frequency-based, but... you know, you may want to try it.
Dawid
Mark Miller wrote:
> From what I know you generally have to pay if you want something that
> does this really well. Or check out http://www.nzdl.org/Kea/
> Unfortunately, the license is GPL. Really too bad; now that it is all
> Java, it would make a great combo with Lucene.
>
> - Mark
>
> mark harwood wrote:
>> I believe the code Otis is referring to is here:
>> http://issues.apache.org/jira/browse/LUCENE-474
>>
>> This is index-level analysis but could be adapted to work for just a
>> single document.
>> The implementation is optimised for speed rather than being a thorough
>> examination of phrase significance.
>> Cheers
>> Mark
>>
>> ----- Original Message ----
>> From: Otis Gospodnetic <ot...@yahoo.com>
>> To: java-user@lucene.apache.org
>> Sent: Monday, 30 April, 2007 4:11:36 AM
>> Subject: Re: Keyphrase Extraction
>>
>> Av, look at Lucene's JIRA and search for Mark Harwood. I believe he
>> once contributed something that does this in JIRA. If you are
>> interested in a commercial solution, I can recommend LingPipe.
>>
>> Otis
>> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
>> Lucene Consulting - http://lucene-consulting.com/
>>
>>
>> ----- Original Message ----
>> From: "av_work@yahoo.com" <av...@yahoo.com>
>> To: java-user@lucene.apache.org
>> Sent: Sunday, April 29, 2007 5:24:17 PM
>> Subject: Keyphrase Extraction
>>
>> Hi,
>>
>> I tried using MoreLikeThis contrib feature to extract "interesting
>> terms" from a document. This works very well - but only for SINGLE words.
>>
>> I am looking for a way to extra "keyPHRASES" from a document. Is there
>> an easy way to achieve this using Lucene index?
>>
>> Thanks in advance!
>> Av
>>
>> __________________________________________________
>> Do You Yahoo!?
>> Tired of spam? Yahoo! Mail has the best spam protection around
>> http://mail.yahoo.com
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>> ___________________________________________________________
>> Yahoo! Answers - Got a question? Someone out there knows the answer.
>> Try it
>> now.
>> http://uk.answers.yahoo.com/
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org