You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Paul Taylor <pa...@fastmail.fm> on 2007/02/23 16:16:44 UTC
Can I use Lucene to retrieve a list of duplicates
Hi I have Java Swing application with a table, I was considering using
Lucene to index the data in the table. One task Id like to do is for the
user to select 'Find Duplicate records for Column X', then I would
filter the table to show only records where there is more than one with
the same value i.e duplicate for that column. Is there a way to return
all the duplicates from a Lucene index.
thanks paul Taylor
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Paul Taylor <pa...@fastmail.fm>.
yes Ive seen this before thanks, it was an article that referred to this
that pointed me towards lucene in the first place :)
Erik Hatcher wrote:
>
> On Feb 23, 2007, at 10:16 AM, Paul Taylor wrote:
>
>> Hi I have Java Swing application with a table, I was considering
>> using Lucene to index the data in the table. One task Id like to do
>> is for the user to select 'Find Duplicate records for Column X', then
>> I would filter the table to show only records where there is more
>> than one with the same value i.e duplicate for that column. Is there
>> a way to return all the duplicates from a Lucene index.
>
> Check this stuff out:
> <http://svn.apache.org/repos/asf/lucene/java/trunk/contrib/swing/> -
> might be of help.
>
> Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> --Internal Virus Database is out-of-date.
> Checked by AVG Free Edition.
> Version: 7.1.394 / Virus Database: 268.16.5/616 - Release Date:
> 04/01/2007
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Feb 23, 2007, at 10:16 AM, Paul Taylor wrote:
> Hi I have Java Swing application with a table, I was considering
> using Lucene to index the data in the table. One task Id like to do
> is for the user to select 'Find Duplicate records for Column X',
> then I would filter the table to show only records where there is
> more than one with the same value i.e duplicate for that column. Is
> there a way to return all the duplicates from a Lucene index.
Check this stuff out: <http://svn.apache.org/repos/asf/lucene/java/
trunk/contrib/swing/> - might be of help.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Paul Taylor <pa...@fastmail.fm>.
Thanks Eric/Chris for all your help
Yes the missing magic was:
TermDocs termDocs = ir.termDocs(terms.term());
instead of
TermDocs termDocs = ir.termDocs();
I will try and use FieldCache as well.
This is the first time Ive used Lucene, but it certainly looks like it will help me with a number of areas in my application :)
paul
Chris Hostetter wrote:
> what you are doing below is iterating over every term in your index, and
> for each Term, recording if that term appears in more then one doc (using
> IndexReader.document which is a really bad idea ingeneral in a loop like
> this)
>
> your orriginal problem description was " 'Find Duplicate records for
> Column X' then I would filter the table to show only records where there
> is more than one with the same value i.e duplicate for that column." ...
>
> if we equate columns with fields, then what you are doing is certainly
> overkill -- you can request a TermEnum that starts at a particular field
> Or use the full TermEnum and "skipTo" the first possible temr in your
> field) and then stop iterating when you get to a new field -- you can aso
> get a lot better performance when compiling hte list of ROW_NUMBER field
> values if you use a FieldCache instead of fetching each document.
>
>
> : Date: Mon, 26 Feb 2007 16:25:11 +0000
> : From: Paul Taylor <pa...@fastmail.fm>
> : Reply-To: java-user@lucene.apache.org, paul_t100@fastmail.fm
> : To: Erick Erickson <er...@gmail.com>
> : Cc: java-user@lucene.apache.org
> : Subject: Re: Can I use Lucene to retrieve a list of duplicates
> :
> : Hi
> :
> : I got it working before I saw your latest mail, the only problem is
> : that it doesn't look very efficient. This is my duplicate method, the
> : problem is that I have to enumerate through *every* term. This was worse
> : before because I was only interested
> : in terms that matched a particular field (column) but had enumerate
> : through every term whatever field it was part of, so I recreated my
> : index so that each document only contained a row number field, and a
> : second field for the value of the column, however this means I am going
> : to end up with a number of different indexes each solving a particular
> : problem.
> :
> : paul
> :
> : public List<Integer> getDuplicates()
> : {
> : List<Integer> matches = new ArrayList<Integer>();
> : try
> : {
> : IndexReader ir = IndexReader.open(directory);
> : TermEnum terms = ir.terms();
> : while (terms.next())
> : {
> : if (terms.docFreq() > 1)
> : {
> : TermDocs termDocs = ir.termDocs(terms.term());
> : while (termDocs.next())
> : {
> : Document d = ir.document(termDocs.doc());
> : matches.add(new
> : Integer(d.getField(ROW_NUMBER).stringValue()));
> : }
> : }
> : }
> :
> : }
> : catch (IOException ioe)
> : {
> : ioe.printStackTrace();
> : }
> : return matches;
> : }
> :
> : ---------------------------------------------------------------------
> : To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> : For additional commands, e-mail: java-user-help@lucene.apache.org
> :
>
>
>
> -Hoss
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Chris Hostetter <ho...@fucit.org>.
what you are doing below is iterating over every term in your index, and
for each Term, recording if that term appears in more then one doc (using
IndexReader.document which is a really bad idea ingeneral in a loop like
this)
your orriginal problem description was " 'Find Duplicate records for
Column X' then I would filter the table to show only records where there
is more than one with the same value i.e duplicate for that column." ...
if we equate columns with fields, then what you are doing is certainly
overkill -- you can request a TermEnum that starts at a particular field
Or use the full TermEnum and "skipTo" the first possible temr in your
field) and then stop iterating when you get to a new field -- you can aso
get a lot better performance when compiling hte list of ROW_NUMBER field
values if you use a FieldCache instead of fetching each document.
: Date: Mon, 26 Feb 2007 16:25:11 +0000
: From: Paul Taylor <pa...@fastmail.fm>
: Reply-To: java-user@lucene.apache.org, paul_t100@fastmail.fm
: To: Erick Erickson <er...@gmail.com>
: Cc: java-user@lucene.apache.org
: Subject: Re: Can I use Lucene to retrieve a list of duplicates
:
: Hi
:
: I got it working before I saw your latest mail, the only problem is
: that it doesn't look very efficient. This is my duplicate method, the
: problem is that I have to enumerate through *every* term. This was worse
: before because I was only interested
: in terms that matched a particular field (column) but had enumerate
: through every term whatever field it was part of, so I recreated my
: index so that each document only contained a row number field, and a
: second field for the value of the column, however this means I am going
: to end up with a number of different indexes each solving a particular
: problem.
:
: paul
:
: public List<Integer> getDuplicates()
: {
: List<Integer> matches = new ArrayList<Integer>();
: try
: {
: IndexReader ir = IndexReader.open(directory);
: TermEnum terms = ir.terms();
: while (terms.next())
: {
: if (terms.docFreq() > 1)
: {
: TermDocs termDocs = ir.termDocs(terms.term());
: while (termDocs.next())
: {
: Document d = ir.document(termDocs.doc());
: matches.add(new
: Integer(d.getField(ROW_NUMBER).stringValue()));
: }
: }
: }
:
: }
: catch (IOException ioe)
: {
: ioe.printStackTrace();
: }
: return matches;
: }
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Paul Taylor <pa...@fastmail.fm>.
Hi
I got it working before I saw your latest mail, the only problem is
that it doesn't look very efficient. This is my duplicate method, the
problem is that I have to enumerate through *every* term. This was worse
before because I was only interested
in terms that matched a particular field (column) but had enumerate
through every term whatever field it was part of, so I recreated my
index so that each document only contained a row number field, and a
second field for the value of the column, however this means I am going
to end up with a number of different indexes each solving a particular
problem.
paul
public List<Integer> getDuplicates()
{
List<Integer> matches = new ArrayList<Integer>();
try
{
IndexReader ir = IndexReader.open(directory);
TermEnum terms = ir.terms();
while (terms.next())
{
if (terms.docFreq() > 1)
{
TermDocs termDocs = ir.termDocs(terms.term());
while (termDocs.next())
{
Document d = ir.document(termDocs.doc());
matches.add(new
Integer(d.getField(ROW_NUMBER).stringValue()));
}
}
}
}
catch (IOException ioe)
{
ioe.printStackTrace();
}
return matches;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Erick Erickson <er...@gmail.com>.
Here's an excerpt from something I wrote to enumerate all the terms for a
field. I hacked out some of my tracing, so it may not even compile <G>.....
Basically, change the line "if (td.next())" to "while (td.next())" and every
time you stay in that loop for more than one cycle, you'll have duplicates
for that particular term....
private void enumField(String field) throws Exception
{
long start = System.currentTimeMillis();
TermEnum termEnum = this.reader.getIndexReader().terms(new
Term(field, ""));
this.writer.println("");
this.writer.println("");
this.writer.println("");
this.writer.println("Values for term " + field);
TermDocs td = this.reader.getIndexReader().termDocs();
Term term = termEnum.term();
int idx = 0;
int jdx = 0;
while ((term != null) && term.field().equals(field)) {
termEnum.next();
td.seek(termEnum);
if (td.next()) {
++jdx;
}
term = termEnum.term();
++idx;
}
}
Erick
On 2/26/07, Paul Taylor <pa...@fastmail.fm> wrote:
>
> Hi,
>
> Sorry I don't see how I get access to TermEnums. So far Ive created a
> document per row, the first field holds the row id, then i have one
> field per column, and checked the index has been created ok with some
> search querys.
> I now want to pass a column to check, and receive a list of all the
> documents that contain a term in that column which is used by at
> least one other document for that column ( a duplicate term).
>
> thanks paul
>
> Chris Hostetter wrote:
> > : Thanks this might do it, but do I need to know the terms beforehand, I
> > : just want to return any terms with frequency more than one?
> >
> > no, TermEnum will let you iterate over all the terms ... you don't even
> > need TermDocs if you just want the docFreq for each term (which would be
> 1
> > if there are no duplicates)
> >
> > : Erick Erickson wrote:
> > : > Sure, you can use the TermDocs/TermEnum classes. Basically, for a
> term
> > : > (probably column value in your app) these let you quickly answer the
> > : > question "which (and how many) documents does this term appear in".
> > : > What you get is the Lucene doc id, which let's you fetch all the
> > : > information about the documents you want.
> > : >
> > : > Erick
> > : >
> > : > On 2/23/07, *Paul Taylor* <paul_t100@fastmail.fm
> > : > <ma...@fastmail.fm>> wrote:
> > : >
> > : > Hi I have Java Swing application with a table, I was considering
> using
> > : > Lucene to index the data in the table. One task Id like to do is
> > : > for the
> > : > user to select 'Find Duplicate records for Column X', then I
> would
> > : > filter the table to show only records where there is more than
> one
> > : > with
> > : > the same value i.e duplicate for that column. Is there a way to
> return
> > : > all the duplicates from a Lucene index.
> > : >
> > : > thanks paul Taylor
> > : >
> > : >
> ---------------------------------------------------------------------
> > : > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > : > <ma...@lucene.apache.org>
> > : > For additional commands, e-mail:
> java-user-help@lucene.apache.org
> > : > <ma...@lucene.apache.org>
> > : >
> > : >
> > : >
> ------------------------------------------------------------------------
> > : >
> > : > Internal Virus Database is out-of-date.
> > : > Checked by AVG Free Edition.
> > : > Version: 7.1.394 / Virus Database: 268.16.5/616 - Release Date:
> 04/01/2007
> > : >
> > :
> > :
> > : ---------------------------------------------------------------------
> > : To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > : For additional commands, e-mail: java-user-help@lucene.apache.org
> > :
> >
> >
> >
> > -Hoss
> >
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Paul Taylor <pa...@fastmail.fm>.
Hi,
Sorry I don't see how I get access to TermEnums. So far Ive created a
document per row, the first field holds the row id, then i have one
field per column, and checked the index has been created ok with some
search querys.
I now want to pass a column to check, and receive a list of all the
documents that contain a term in that column which is used by at
least one other document for that column ( a duplicate term).
thanks paul
Chris Hostetter wrote:
> : Thanks this might do it, but do I need to know the terms beforehand, I
> : just want to return any terms with frequency more than one?
>
> no, TermEnum will let you iterate over all the terms ... you don't even
> need TermDocs if you just want the docFreq for each term (which would be 1
> if there are no duplicates)
>
> : Erick Erickson wrote:
> : > Sure, you can use the TermDocs/TermEnum classes. Basically, for a term
> : > (probably column value in your app) these let you quickly answer the
> : > question "which (and how many) documents does this term appear in".
> : > What you get is the Lucene doc id, which let's you fetch all the
> : > information about the documents you want.
> : >
> : > Erick
> : >
> : > On 2/23/07, *Paul Taylor* <paul_t100@fastmail.fm
> : > <ma...@fastmail.fm>> wrote:
> : >
> : > Hi I have Java Swing application with a table, I was considering using
> : > Lucene to index the data in the table. One task Id like to do is
> : > for the
> : > user to select 'Find Duplicate records for Column X', then I would
> : > filter the table to show only records where there is more than one
> : > with
> : > the same value i.e duplicate for that column. Is there a way to return
> : > all the duplicates from a Lucene index.
> : >
> : > thanks paul Taylor
> : >
> : > ---------------------------------------------------------------------
> : > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> : > <ma...@lucene.apache.org>
> : > For additional commands, e-mail: java-user-help@lucene.apache.org
> : > <ma...@lucene.apache.org>
> : >
> : >
> : > ------------------------------------------------------------------------
> : >
> : > Internal Virus Database is out-of-date.
> : > Checked by AVG Free Edition.
> : > Version: 7.1.394 / Virus Database: 268.16.5/616 - Release Date: 04/01/2007
> : >
> :
> :
> : ---------------------------------------------------------------------
> : To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> : For additional commands, e-mail: java-user-help@lucene.apache.org
> :
>
>
>
> -Hoss
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Chris Hostetter <ho...@fucit.org>.
: Thanks this might do it, but do I need to know the terms beforehand, I
: just want to return any terms with frequency more than one?
no, TermEnum will let you iterate over all the terms ... you don't even
need TermDocs if you just want the docFreq for each term (which would be 1
if there are no duplicates)
: Erick Erickson wrote:
: > Sure, you can use the TermDocs/TermEnum classes. Basically, for a term
: > (probably column value in your app) these let you quickly answer the
: > question "which (and how many) documents does this term appear in".
: > What you get is the Lucene doc id, which let's you fetch all the
: > information about the documents you want.
: >
: > Erick
: >
: > On 2/23/07, *Paul Taylor* <paul_t100@fastmail.fm
: > <ma...@fastmail.fm>> wrote:
: >
: > Hi I have Java Swing application with a table, I was considering using
: > Lucene to index the data in the table. One task Id like to do is
: > for the
: > user to select 'Find Duplicate records for Column X', then I would
: > filter the table to show only records where there is more than one
: > with
: > the same value i.e duplicate for that column. Is there a way to return
: > all the duplicates from a Lucene index.
: >
: > thanks paul Taylor
: >
: > ---------------------------------------------------------------------
: > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: > <ma...@lucene.apache.org>
: > For additional commands, e-mail: java-user-help@lucene.apache.org
: > <ma...@lucene.apache.org>
: >
: >
: > ------------------------------------------------------------------------
: >
: > Internal Virus Database is out-of-date.
: > Checked by AVG Free Edition.
: > Version: 7.1.394 / Virus Database: 268.16.5/616 - Release Date: 04/01/2007
: >
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Paul Taylor <pa...@fastmail.fm>.
Thanks this might do it, but do I need to know the terms beforehand, I
just want to return any terms with frequency more than one?
Erick Erickson wrote:
> Sure, you can use the TermDocs/TermEnum classes. Basically, for a term
> (probably column value in your app) these let you quickly answer the
> question "which (and how many) documents does this term appear in".
> What you get is the Lucene doc id, which let's you fetch all the
> information about the documents you want.
>
> Erick
>
> On 2/23/07, *Paul Taylor* <paul_t100@fastmail.fm
> <ma...@fastmail.fm>> wrote:
>
> Hi I have Java Swing application with a table, I was considering using
> Lucene to index the data in the table. One task Id like to do is
> for the
> user to select 'Find Duplicate records for Column X', then I would
> filter the table to show only records where there is more than one
> with
> the same value i.e duplicate for that column. Is there a way to return
> all the duplicates from a Lucene index.
>
> thanks paul Taylor
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> <ma...@lucene.apache.org>
> For additional commands, e-mail: java-user-help@lucene.apache.org
> <ma...@lucene.apache.org>
>
>
> ------------------------------------------------------------------------
>
> Internal Virus Database is out-of-date.
> Checked by AVG Free Edition.
> Version: 7.1.394 / Virus Database: 268.16.5/616 - Release Date: 04/01/2007
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can I use Lucene to retrieve a list of duplicates
Posted by Erick Erickson <er...@gmail.com>.
Sure, you can use the TermDocs/TermEnum classes. Basically, for a term
(probably column value in your app) these let you quickly answer the
question "which (and how many) documents does this term appear in". What you
get is the Lucene doc id, which let's you fetch all the information about
the documents you want.
Erick
On 2/23/07, Paul Taylor <pa...@fastmail.fm> wrote:
>
> Hi I have Java Swing application with a table, I was considering using
> Lucene to index the data in the table. One task Id like to do is for the
> user to select 'Find Duplicate records for Column X', then I would
> filter the table to show only records where there is more than one with
> the same value i.e duplicate for that column. Is there a way to return
> all the duplicates from a Lucene index.
>
> thanks paul Taylor
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>