You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by oh...@cox.net on 2009/08/02 10:32:24 UTC
Weird discrepancy with term counts vs. terms (off by 1)
Hi,
I've noticed a kind of strange problem with term counts and actual terms.
Some background: I wrote an app that creates an index, including a "path" field.
I am now working on an app (code was in the previous thread) that, as part of what it does, needs to get a list of all of the "path" fields for documents that were added.
I first noticed the problem that I'm seeing while working on this latter app. Basically, what I noticed was that while I was adding 13 documents to the index, when I listed the "path" terms, there were only 12 of them.
So then, I reviewed the index using Luke, and what I saw with that was that there were indeed only 12 "path" terms (under "Term Count" on the left), but, when I clicked the "Show Top Terms" in Luke, there were 13 terms listed by Luke.
At this point, I'm very puzzled about all of this :(...
Can anyone explain why the difference in Luke, and, more importantly, what I am only getting 12 (i.e., 1 less than the # of documents added) when I try to programmatically list the terms?
Thanks,
Jim
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Weird discrepancy with term counts vs. terms (off by 1)
Posted by Phil Whelan <ph...@gmail.com>.
On Sun, Aug 2, 2009 at 10:58 AM, Andrzej Bialecki<ab...@getopt.org> wrote:
> Thank you Phil for spotting this bug - this fix will be included in the next
> release of Luke.
Glad to help. Thanks for building this great tool!
Phil
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Weird discrepancy with term counts vs. terms (off by 1)
Posted by Andrzej Bialecki <ab...@getopt.org>.
Phil Whelan wrote:
> Hi Jim,
>
> On Sun, Aug 2, 2009 at 9:08 AM, Phil Whelan<ph...@gmail.com> wrote:
>>> So then, I reviewed the index using Luke, and what I saw with that was that there were indeed only 12 "path" terms (under "Term Count" on the left), but, when I clicked the "Show Top Terms" in Luke, there were 13 terms listed by Luke.
>> Yes, I just checked this and this seems to be a bug with Luke. It
>> always shows 1 less than in "Term Count" than it should. Well spotted.
>
> I was able to see why this way happening in the Luke source and I've
> submitted the following patch to Andrzej, the author of Luke.
>
Thank you Phil for spotting this bug - this fix will be included in the
next release of Luke.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Weird discrepancy with term counts vs. terms (off by 1)
Posted by Phil Whelan <ph...@gmail.com>.
Hi Jim,
On Sun, Aug 2, 2009 at 9:08 AM, Phil Whelan<ph...@gmail.com> wrote:
>
>> So then, I reviewed the index using Luke, and what I saw with that was that there were indeed only 12 "path" terms (under "Term Count" on the left), but, when I clicked the "Show Top Terms" in Luke, there were 13 terms listed by Luke.
>
> Yes, I just checked this and this seems to be a bug with Luke. It
> always shows 1 less than in "Term Count" than it should. Well spotted.
I was able to see why this way happening in the Luke source and I've
submitted the following patch to Andrzej, the author of Luke.
Thanks,
Phil
--- luke.orig/src/org/getopt/luke/Luke.java 2009-03-19 22:41:34.000000000 -0700
+++ luke-src-0.9.2/src/org/getopt/luke/Luke.java 2009-08-02
09:33:24.000000000 -0700
@@ -813,23 +813,18 @@
setString(iFields, "text", String.valueOf(idxFields.length));
Object iTerms = find(pOver, "iTerms");
termCounts.clear();
- FieldTermCount ftc = new FieldTermCount();
+ FieldTermCount ftc = null;
TermEnum te = ir.terms();
numTerms = 0;
while (te.next()) {
Term currTerm = te.term();
- if (ftc.fieldname == null) {
+ if (ftc == null || ftc.fieldname == null || ftc.fieldname !=
currTerm.field()) {
// initialize
- ftc.fieldname = currTerm.field();
- termCounts.put(ftc.fieldname, ftc);
- }
- if (ftc.fieldname == currTerm.field()) {
- ftc.termCount++;
- } else {
ftc = new FieldTermCount();
ftc.fieldname = currTerm.field();
termCounts.put(ftc.fieldname, ftc);
}
+ ftc.termCount++;
numTerms++;
}
te.close();
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Weird discrepancy with term counts vs. terms (off by 1)
Posted by Phil Whelan <ph...@gmail.com>.
Hi Jim,
On Sun, Aug 2, 2009 at 12:12 PM, <oh...@cox.net> wrote:
> i.e., I was ignoring the 1st term in the TermEnum (since the .next() bumps the TermEnum to the 2nd term, initially).
Great! Glad you found the problem. I couldn't see it.
Phil
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Weird discrepancy with term counts vs. terms (off by 1)
Posted by oh...@cox.net.
Hi Phil,
For problem with my app, it wasn't what you suggested (about the tokens, etc.).
For some later things, my indexer creates both a "path" field that is analyzed (and thus tokenized, etc.) and another field, "fullpath", which is not analyzed (and thus, not tokenized).
The problem with my app was that I was create a TermEnum:
Term term = new Term("fullpath", "");
termsEnumerator = reader.terms(term);
and then going immediately into a while loop:
while (termsEnumerator.next()) {
.
.
}
i.e., I was ignoring the 1st term in the TermEnum (since the .next() bumps the TermEnum to the 2nd term, initially).
Anyway, so the code that I ended up with is:
try {
System.out.println("Outside while: About to get 1st termsEnumerator.term()...");
currentTerm = termsEnumerator.term();
currentField = currentTerm.field();
termpathcount++;
System.out.println("Outside while: 1st Field = [" + currentField + "] Term = [" + currentTerm.text() + "]");
System.out.println("Outside while: About to drop into while()...");
while (termsEnumerator.next()) {
currentTerm = termsEnumerator.term();
currentField = currentTerm.field();
if (currentField.equalsIgnoreCase("fullpath")) {
termpathcount++;
System.out.println("Count=" + termpathcount + " Field = [" + currentField + "] Term = [" + currentTerm.text() + "]");
}
} // end while()
termsEnumerator.close();
System.out.println("Matching terms count = " + termpathcount);
} catch (Exception e) {
System.out.println("** ERROR **: Exception while stepping through index: [" + e + "]");
e.printStackTrace();
}
and, that seems to be working perfectly.
Also, thanks for following up re. that Luke problem. That was one piece of this "puzzle" that was kind of driving me batty :)!!
Jim
---- Phil Whelan <ph...@gmail.com> wrote:
> Hi Jim,
>
> On Sun, Aug 2, 2009 at 1:32 AM, <oh...@cox.net> wrote:
> > I first noticed the problem that I'm seeing while working on this latter app. Basically, what I noticed was that while I was adding 13 documents to the index, when I listed the "path" terms, there were only 12 of them.
>
> Field text (the whole "path" in your case) and terms (the tokens of
> the field text) are different.
>
> The StandardAnalyzer breaks up words like this...
> Field text = "/a/b/c.txt"
> Tokens = {"a","b","c","txt"}
>
> So this 1 field of 1 document become 4 terms / tokens (not sure if
> there is a difference in this terminology between "terms" and "tokens"
> sorry).
> Therefore, you're going to have more terms than documents initially,
> but as the overlap in term usage increases this changes.
>
> For instance, these 3 paths
> "/a/b/c/d.txt","/b/c/d/a.txt","/c/d/a/b.txt" are still only a total of
> 4 terms, since they share the same terms.
>
> In fact, StandardAnalyzer goes a bit further than that and removes
> "stop-words", such as "a" (or "an", "the") as it's designed for
> general text searching.
>
> That said, I think you have a point with the next part of your question...
>
> > So then, I reviewed the index using Luke, and what I saw with that was that there were indeed only 12 "path" terms (under "Term Count" on the left), but, when I clicked the "Show Top Terms" in Luke, there were 13 terms listed by Luke.
>
> Yes, I just checked this and this seems to be a bug with Luke. It
> always shows 1 less than in "Term Count" than it should. Well spotted.
>
> Cheers,
> Phil
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Weird discrepancy with term counts vs. terms (off by 1)
Posted by Phil Whelan <ph...@gmail.com>.
Hi Jim,
On Sun, Aug 2, 2009 at 1:32 AM, <oh...@cox.net> wrote:
> I first noticed the problem that I'm seeing while working on this latter app. Basically, what I noticed was that while I was adding 13 documents to the index, when I listed the "path" terms, there were only 12 of them.
Field text (the whole "path" in your case) and terms (the tokens of
the field text) are different.
The StandardAnalyzer breaks up words like this...
Field text = "/a/b/c.txt"
Tokens = {"a","b","c","txt"}
So this 1 field of 1 document become 4 terms / tokens (not sure if
there is a difference in this terminology between "terms" and "tokens"
sorry).
Therefore, you're going to have more terms than documents initially,
but as the overlap in term usage increases this changes.
For instance, these 3 paths
"/a/b/c/d.txt","/b/c/d/a.txt","/c/d/a/b.txt" are still only a total of
4 terms, since they share the same terms.
In fact, StandardAnalyzer goes a bit further than that and removes
"stop-words", such as "a" (or "an", "the") as it's designed for
general text searching.
That said, I think you have a point with the next part of your question...
> So then, I reviewed the index using Luke, and what I saw with that was that there were indeed only 12 "path" terms (under "Term Count" on the left), but, when I clicked the "Show Top Terms" in Luke, there were 13 terms listed by Luke.
Yes, I just checked this and this seems to be a bug with Luke. It
always shows 1 less than in "Term Count" than it should. Well spotted.
Cheers,
Phil
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Weird discrepancy with term counts vs. terms (off by 1)
Posted by oh...@cox.net.
Hi,
BTW, my indexer app is basically the same as the demo IndexFiles.java. Here's part of the main:
try {
IndexWriter writer = new IndexWriter(INDEX_DIR, new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
System.out.println("Indexing to directory '" +INDEX_DIR+ "'...");
indexDocs(writer, docDir);
System.out.println("Optimizing...");
writer.optimize();
writer.close();
Date end = new Date();
System.out.println(end.getTime() - start.getTime() + " total milliseconds");
} catch (IOException e) {
System.out.println(" caught a " + e.getClass() +
"\n with message: " + e.getMessage());
}
when I run the indexer, I can see it say it added the document that ends up being "missing" from the terms.
Thanks,
Jim
---- ohaya@cox.net wrote:
> Hi,
>
> I've noticed a kind of strange problem with term counts and actual terms.
>
> Some background: I wrote an app that creates an index, including a "path" field.
>
> I am now working on an app (code was in the previous thread) that, as part of what it does, needs to get a list of all of the "path" fields for documents that were added.
>
> I first noticed the problem that I'm seeing while working on this latter app. Basically, what I noticed was that while I was adding 13 documents to the index, when I listed the "path" terms, there were only 12 of them.
>
> So then, I reviewed the index using Luke, and what I saw with that was that there were indeed only 12 "path" terms (under "Term Count" on the left), but, when I clicked the "Show Top Terms" in Luke, there were 13 terms listed by Luke.
>
> At this point, I'm very puzzled about all of this :(...
>
> Can anyone explain why the difference in Luke, and, more importantly, what I am only getting 12 (i.e., 1 less than the # of documents added) when I try to programmatically list the terms?
>
> Thanks,
> Jim
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org