You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michael Giles <mg...@visionstudio.com> on 2004/03/08 22:37:47 UTC
Filtering out duplicate documents...
I'm looking for a way to filter out duplicate documents from an index
(either while indexing, or after the fact). It seems like there should be
an approach of comparing the terms for two documents, but I'm wondering if
any other folks (i.e. nutch) have come up with a solution to this problem.
Obviously you can compute the Levenstein distance on the text, but that is
way too computationally intensive to scale. So the goal is to find
something that would be workable in a production system. For example, a
given NYT article, and its printer friendly version should be deemed to be
the same.
-Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Filtering out duplicate documents...
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
My impression is the new term vector support should at least make this
type of comparison feasible in some manner. I'd be interested to see
what you come up with if you give this a try. You will need the latest
CVS codebase.
Erik
On Mar 8, 2004, at 4:37 PM, Michael Giles wrote:
> I'm looking for a way to filter out duplicate documents from an index
> (either while indexing, or after the fact). It seems like there
> should be an approach of comparing the terms for two documents, but
> I'm wondering if any other folks (i.e. nutch) have come up with a
> solution to this problem.
>
> Obviously you can compute the Levenstein distance on the text, but
> that is way too computationally intensive to scale. So the goal is to
> find something that would be workable in a production system. For
> example, a given NYT article, and its printer friendly version should
> be deemed to be the same.
>
> -Mike
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org