You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Ugo Cei <ug...@apache.org> on 2005/06/15 14:31:07 UTC
[OT] Determining the similarity between a pair of texts
Excuse the Off-Topic, but I'm looking for a Java API for determining
the degree of similarity (based on word frequency or whatever) between
two text strings.
I though of posting here since I know there are some people here expert
in semantic web technologies that could maybe help me.
Thanks in advance and sorry again for the OT.
Ugo
--
Ugo Cei
Tech Blog: http://agylen.com/
Open Source Zone: http://oszone.org/
Wine & Food Blog: http://www.divinocibo.it/
Re: [OT] Determining the similarity between a pair of texts
Posted by Stefano Mazzocchi <st...@apache.org>.
Ugo Cei wrote:
> Il giorno 15/giu/05, alle 18:27, Stefano Mazzocchi ha scritto:
>
>> I've been working on this for the past few months. There is no clearcut
>> solution, but using LSI is probably the best approach for the above
>
>
> LSI == ?
latent semantic indexing
>> As for string distance, you might want to check out secondstring.sf.net.
>
> There are lots of algorith there
> <http://secondstring.sourceforge.net/javadoc/com/wcohen/secondstring/
> package-summary.html>. Which one would you suggest before I start
> trying them all one by one?
it really depends on what you need to do. A simple first order
Levenstein can make you go a long way, if you don't know the
distribution of the frequencies of the characters in advance. Otherwise,
you get better results, but at the price of being less effective if the
statistical properties of your data streams start changing over time.
--
Stefano.
Re: [OT] Determining the similarity between a pair of texts
Posted by Ugo Cei <ug...@apache.org>.
Il giorno 15/giu/05, alle 18:27, Stefano Mazzocchi ha scritto:
> I've been working on this for the past few months. There is no clearcut
> solution, but using LSI is probably the best approach for the above
LSI == ?
>
> As for string distance, you might want to check out
> secondstring.sf.net.
There are lots of algorith there
<http://secondstring.sourceforge.net/javadoc/com/wcohen/secondstring/
package-summary.html>. Which one would you suggest before I start
trying them all one by one?
Ugo
--
Ugo Cei
Tech Blog: http://agylen.com/
Open Source Zone: http://oszone.org/
Wine & Food Blog: http://www.divinocibo.it/
Re: [OT] Determining the similarity between a pair of texts
Posted by Stefano Mazzocchi <st...@apache.org>.
Peter Hunsberger wrote:
> On 6/15/05, Ugo Cei <u....@pronetics.it> wrote:
>
>>Il giorno 15/giu/05, alle 16:32, Tony Collen ha scritto:
>
>
> <snip/>
>
>>Actually, what I am trying to come up is an algorithm for determining
>>whether two texts refer (more or less) about similar subjects.
>>
>
> Eee, then you may have to jump into the NLP stuff and Ontology
> processing in general; non-trivial stuff...
I've been working on this for the past few months. There is no clearcut
solution, but using LSI is probably the best approach for the above
(even if there is tons of literature on variations of approaches).
As for string distance, you might want to check out secondstring.sf.net.
--
Stefano.
Re: [OT] Determining the similarity between a pair of texts
Posted by Peter Hunsberger <pe...@gmail.com>.
On 6/15/05, Ugo Cei <u....@pronetics.it> wrote:
> Il giorno 15/giu/05, alle 16:32, Tony Collen ha scritto:
<snip/>
>
> Actually, what I am trying to come up is an algorithm for determining
> whether two texts refer (more or less) about similar subjects.
>
Eee, then you may have to jump into the NLP stuff and Ontology
processing in general; non-trivial stuff...
--
Peter Hunsberger
Re: [OT] Determining the similarity between a pair of texts
Posted by Ugo Cei <u....@pronetics.it>.
Il giorno 15/giu/05, alle 16:32, Tony Collen ha scritto:
> Ugo,
>
> I think what you're looking for is the Levenshtein Distance Algorithm.
>
> http://www.google.com/search?
> hl=en&q=java+Levenshtein+implementation&btnG=Google+Search
Nice! I also found an implementation nearby:
http://jakarta.apache.org/commons/lang/api/org/apache/commons/lang/
StringUtils.html#getLevenshteinDistance(java.lang.String,%20java.lang.St
ring)
;)
However, this algorithm is useful for finding single-character
differences, whereas I am more interested in word differences. IOW, the
LD between "test" and "tent" is 1 and the LD between "test" and "barf"
is 4, but for my purpose it should be 1 in both cases. And the LD
between "test case" and "tent base" is smaller than the one between
"test case" and "case under test", but I need it to be the reverse.
Actually, what I am trying to come up is an algorithm for determining
whether two texts refer (more or less) about similar subjects.
Ugo
--
Ugo Cei
Tech Blog: http://agylen.com/
Open Source Zone: http://oszone.org/
Wine & Food Blog: http://www.divinocibo.it/
Re: [OT] Determining the similarity between a pair of texts
Posted by Tony Collen <co...@umn.edu>.
Ugo,
I think what you're looking for is the Levenshtein Distance Algorithm.
http://www.google.com/search?hl=en&q=java+Levenshtein+implementation&btnG=Google+Search
HTH,
Tony
Ugo Cei wrote:
> Excuse the Off-Topic, but I'm looking for a Java API for determining the
> degree of similarity (based on word frequency or whatever) between two
> text strings.
>
> I though of posting here since I know there are some people here expert
> in semantic web technologies that could maybe help me.
>
> Thanks in advance and sorry again for the OT.
>
> Ugo
>
Re: [OT] Determining the similarity between a pair of texts
Posted by Torsten Curdt <tc...@apache.org>.
>>Excuse the Off-Topic, but I'm looking for a Java API for determining
>>the degree of similarity (based on word frequency or whatever) between
>>two text strings.
also commons codec has some algorithms
...depends on what you are after exactly
http://jakarta.apache.org/commons/codec/
cheers
--
Torsten
Re: [OT] Determining the similarity between a pair of texts
Posted by Peter Hunsberger <pe...@gmail.com>.
On 6/15/05, Ugo Cei <ug...@apache.org> wrote:
> Excuse the Off-Topic, but I'm looking for a Java API for determining
> the degree of similarity (based on word frequency or whatever) between
> two text strings.
>
> I though of posting here since I know there are some people here expert
> in semantic web technologies that could maybe help me.
>
> Thanks in advance and sorry again for the OT.
>
Probably not exacftly what you want but you can find a collection of
Natural Language Processing (NLP) toools at:
http://opennlp.sourceforge.net/projects.html
If you've got to build your own NLP algorithims this might help...
--
Peter Hunsberger (not an expert in semantic web technologies...)