You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mark Miller <ma...@gmail.com> on 2007/05/02 14:52:36 UTC

Re: Keyphrase Extraction

 From what I know you generally have to pay if you want something that 
does this really well. Or check out http://www.nzdl.org/Kea/
Unfortunately, the license is GPL. Really too bad; now that it is all 
Java, it would make a great combo with Lucene.

- Mark

mark harwood wrote:
> I believe the code Otis is referring to is here: http://issues.apache.org/jira/browse/LUCENE-474
>
> This is index-level analysis but could be adapted to work for just a single document.
> The implementation is optimised for speed rather than being a thorough examination of phrase significance. 
>
> Cheers
> Mark
>
> ----- Original Message ----
> From: Otis Gospodnetic <ot...@yahoo.com>
> To: java-user@lucene.apache.org
> Sent: Monday, 30 April, 2007 4:11:36 AM
> Subject: Re: Keyphrase Extraction
>
> Av, look at Lucene's JIRA and search for Mark Harwood.  I believe he once contributed something that does this in JIRA.  If you are interested in a commercial solution, I can recommend LingPipe.
>
> Otis
> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
> Lucene Consulting - http://lucene-consulting.com/
>
>
> ----- Original Message ----
> From: "av_work@yahoo.com" <av...@yahoo.com>
> To: java-user@lucene.apache.org
> Sent: Sunday, April 29, 2007 5:24:17 PM
> Subject: Keyphrase Extraction
>
> Hi,
>
> I tried using MoreLikeThis contrib feature to extract "interesting terms" from a document. This works very well - but only for SINGLE words.
>
> I am looking for a way to extra "keyPHRASES" from a document. Is there an easy way to achieve this using Lucene index?
>
> Thanks in advance!
> Av
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
>
>       ___________________________________________________________
> Yahoo! Answers - Got a question? Someone out there knows the answer. Try it
> now.
> http://uk.answers.yahoo.com/ 
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Keyphrase Extraction

Posted by José Ramón Pérez Agüera <jo...@gmail.com>.
here you have a very good tool for Keyphrase Extraction. It is GNU and
easy to integrate in Lucene.

http://www.paynter.info/academia/Kea.php

best

jose

On 5/8/07, Bill Janssen <ja...@parc.com> wrote:
> Dawid Weiss wrote:
> > You could also try splitting the document into paragraphs and use Carrot2's
> > Lingo algorithm (www.carrot2.org) on a paragraph-level to extract clusters.
> > Labelling routine in Lingo should extract 'key' phrases; this analysis is
> > heavily frequency-based, but... you know, you may want to try it.
>
> Just to make sure I'm following...
>
> So you're suggesting splitting the document into paragraphs, then
> treating each paragraph as if it were a Carrot2 search result,
> performing the clustering, then looking at the label Lingo chooses for
> each cluster, and treating that label as the "key phrase"?
>
> Would DirectDocumentFeedExample be a good starting point?
>
> Bill
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
José Ramón Pérez Agüera

Dept. de Ingeniería del Software e Inteligencia Artificial
Despacho 411 tlf. 913947599
Facultad de Informática
Universidad Complutense de Madrid

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Keyphrase Extraction (via Lingo)

Posted by Bill Janssen <ja...@parc.com>.
> Dawid Weiss wrote:
> > You could also try splitting the document into paragraphs and use Carrot2's 
> > Lingo algorithm (www.carrot2.org) on a paragraph-level to extract clusters. 
> > Labelling routine in Lingo should extract 'key' phrases; this analysis is 
> > heavily frequency-based, but... you know, you may want to try it.
> 
> Just to make sure I'm following...
> 
> So you're suggesting splitting the document into paragraphs, then
> treating each paragraph as if it were a Carrot2 search result,
> performing the clustering, then looking at the label Lingo chooses for
> each cluster, and treating that label as the "key phrase"?

I tried it.  Not so great results, but perhaps I'm doing it wrong.
Here's my code.  The input file is a text file with an ID number and
one paragraph per (long) line -- standard textual paragraphs.  I'm
running on a corpus of technical papers.

Bill

-----------------------------------------------------------------
import org.carrot2.filter.lingo.common.*;
import org.carrot2.filter.lingo.lsicluster.*;

import java.io.*;

public class test {

    public static void main (String[] argv) {
        DefaultClusteringContext context = new DefaultClusteringContext();
        try {
            BufferedReader r = new BufferedReader(new FileReader(argv[0]));
            String line;
            while ((line = r.readLine()) != null) {
                String[] parts = line.trim().split("\\s");
		// there must be an easier way to split off the first token
		// of a line...
                if (parts.length > 1) {
                    String id = parts[0];
		    // and to glue the other parts together again...
                    String body = parts[1];
                    for (int i = 2;  i < parts.length;  i++) {
                        body = body + " " + parts[i];
                    }
                    context.addSnippet(new Snippet(id, "", body));
                }
            }
        } catch (Exception x) {
            x.printStackTrace(System.err);
        }
        
        context.setQuery("");
        Cluster[] clusters = context.cluster();
        for (int i = 0;  i < clusters.length;  i++) {
            System.out.println("Cluster --");
            String[] labels = clusters[i].getLabels();
            for (int j = 0;  j < labels.length;  j++) {
                System.out.println("    Label:  " + labels[j]);
            }
            Snippet[] snippets = clusters[i].getSnippets();
            System.out.println("    " + snippets.length + " snippets:");
            for (int j = 0;  j < snippets.length;  j++) {
                System.out.println("         " + snippets[j].getSnippetId() +
		" -- " + snippets[j].getText());
            }
        }
    }
}
----------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Keyphrase Extraction

Posted by Bill Janssen <ja...@parc.com>.
Dawid Weiss wrote:
> You could also try splitting the document into paragraphs and use Carrot2's 
> Lingo algorithm (www.carrot2.org) on a paragraph-level to extract clusters. 
> Labelling routine in Lingo should extract 'key' phrases; this analysis is 
> heavily frequency-based, but... you know, you may want to try it.

Just to make sure I'm following...

So you're suggesting splitting the document into paragraphs, then
treating each paragraph as if it were a Carrot2 search result,
performing the clustering, then looking at the label Lingo chooses for
each cluster, and treating that label as the "key phrase"?

Would DirectDocumentFeedExample be a good starting point?

Bill

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Keyphrase Extraction

Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
You could also try splitting the document into paragraphs and use Carrot2's 
Lingo algorithm (www.carrot2.org) on a paragraph-level to extract clusters. 
Labelling routine in Lingo should extract 'key' phrases; this analysis is 
heavily frequency-based, but... you know, you may want to try it.

Dawid

Mark Miller wrote:
>  From what I know you generally have to pay if you want something that 
> does this really well. Or check out http://www.nzdl.org/Kea/
> Unfortunately, the license is GPL. Really too bad; now that it is all 
> Java, it would make a great combo with Lucene.
> 
> - Mark
> 
> mark harwood wrote:
>> I believe the code Otis is referring to is here: 
>> http://issues.apache.org/jira/browse/LUCENE-474
>>
>> This is index-level analysis but could be adapted to work for just a 
>> single document.
>> The implementation is optimised for speed rather than being a thorough 
>> examination of phrase significance.
>> Cheers
>> Mark
>>
>> ----- Original Message ----
>> From: Otis Gospodnetic <ot...@yahoo.com>
>> To: java-user@lucene.apache.org
>> Sent: Monday, 30 April, 2007 4:11:36 AM
>> Subject: Re: Keyphrase Extraction
>>
>> Av, look at Lucene's JIRA and search for Mark Harwood.  I believe he 
>> once contributed something that does this in JIRA.  If you are 
>> interested in a commercial solution, I can recommend LingPipe.
>>
>> Otis
>> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
>> Lucene Consulting - http://lucene-consulting.com/
>>
>>
>> ----- Original Message ----
>> From: "av_work@yahoo.com" <av...@yahoo.com>
>> To: java-user@lucene.apache.org
>> Sent: Sunday, April 29, 2007 5:24:17 PM
>> Subject: Keyphrase Extraction
>>
>> Hi,
>>
>> I tried using MoreLikeThis contrib feature to extract "interesting 
>> terms" from a document. This works very well - but only for SINGLE words.
>>
>> I am looking for a way to extra "keyPHRASES" from a document. Is there 
>> an easy way to achieve this using Lucene index?
>>
>> Thanks in advance!
>> Av
>>
>> __________________________________________________
>> Do You Yahoo!?
>> Tired of spam?  Yahoo! Mail has the best spam protection around 
>> http://mail.yahoo.com
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>>       ___________________________________________________________
>> Yahoo! Answers - Got a question? Someone out there knows the answer. 
>> Try it
>> now.
>> http://uk.answers.yahoo.com/
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>   
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org