You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Robert Watkins <rw...@foo-bar.org> on 2005/12/13 13:33:18 UTC
html parsers and numers of terms
I have been experimenting with a couple of HTML parsers, primarily to
compare performance, but have discovered a difference in the index for
which I haven't, with assurance discovered the cause.
The difference is in the number of terms reported by Luke. The indexes
created with the content parsed using JTidy generally have about 30%
fewer terms than those created with content parsed using HTMLParser
(htmlparser.org).
The only difference I can discern (using debug logs and diff) is with
the way entities are handled by the two parsers. Using JTidy, any HTML
entities are converted to the literal character; using HTMLParser they
are left as an entity (named or numeric). In the fields that are
tokenized, entities not already converted are done so in the index, which
leaves only the fields not tokenized. It does not seem likely to me that
this could account for 30% of the terms indexed.
Is it possible to use Luke (or some other tool) to make a more detailed
comparison of the two indexes? I have tried to find a difference in the
top terms indexed, and while the order of the top terms does change, the
numbers do not. Am I missing something obvious?
Thanks,
-- Robert
--------------------
Robert Watkins
rwatkins@foo-bar.org
--------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: html parsers and numers of terms
Posted by "J.J. Larrea" <jj...@panix.com>.
Glad that hint was useful. I was totally bit by that artifact myself. It turns out that there were XML numeric character references within VARCHAR fields in a database I was indexing, so I never suspected that the NCRs I was seeing in Luke had anything to do with the non-XML non-HTML (so I thought) source data.
Also take care that when fields are stored it is quite easy to get confused between the stored values, which aren't analyzed, and the indexed tokens, which obviously are. Asking Luke to reconstruct a source document from its indexed tokens is a great way to see it from an "index-eye" view, which can be very revealing.
- J.J.
At 11:36 AM -0500 12/13/05, Robert Watkins wrote:
>Aha! I had, indeed, been fooled by Luke into thinking that the entities
>had been converted upon analysis, but you have set me straight.
>
>Thanks,
>-- Robert
>
>On Tue, 13 Dec 2005, J.J. Larrea wrote:
>
>>Beware of HTML/XML entities in your input stream! The Lucene analyzers (including StandardAnalyzer) do not interpret these representation-specific encodings, and assume the & and ; delimiters are punctuation. How they deal with punctuation depends on the specific Analyzer logic.
>>
>>[ snipped ]
>>
>>PS: Also note that when using Luke to see what is indexed, it uses NCRs eg. é to display non-ASCII characters, allowing one to be easily confused as to whether the NCRs were indexed or the Unicode characters were indexed.
>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: html parsers and numers of terms
Posted by Robert Watkins <rw...@foo-bar.org>.
Aha! I had, indeed, been fooled by Luke into thinking that the entities
had been converted upon analysis, but you have set me straight.
Thanks,
-- Robert
On Tue, 13 Dec 2005, J.J. Larrea wrote:
> Beware of HTML/XML entities in your input stream! The Lucene analyzers (including StandardAnalyzer) do not interpret these representation-specific encodings, and assume the & and ; delimiters are punctuation. How they deal with punctuation depends on the specific Analyzer logic.
>
> [ snipped ]
>
> PS: Also note that when using Luke to see what is indexed, it uses NCRs eg. é to display non-ASCII characters, allowing one to be easily confused as to whether the NCRs were indexed or the Unicode characters were indexed.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: html parsers and numers of terms
Posted by "J.J. Larrea" <jj...@panix.com>.
Beware of HTML/XML entities in your input stream! The Lucene analyzers (including StandardAnalyzer) do not interpret these representation-specific encodings, and assume the & and ; delimiters are punctuation. How they deal with punctuation depends on the specific Analyzer logic.
For example, here is the output from running the (horribly useful) Lucene In Action class lia.analyzer.AnalyzerDemo on the strings "Nausée", "Nausée", and "Nausée", all of which are equivalent in an HTML encoding:
Analyzing "Naus?e"
WhitespaceAnalyzer:
[Naus?e]
SimpleAnalyzer:
[naus?e]
StopAnalyzer:
[naus?e]
StandardAnalyzer:
[naus?e]
(The ? is an artifact of seeing the Unicode output on a text terminal)
Analyzing "Nausée"
WhitespaceAnalyzer:
[Nausée]
SimpleAnalyzer:
[naus] [eacute] [e]
StopAnalyzer:
[naus] [eacute] [e]
StandardAnalyzer:
[nausé] [e]
Analyzing "Nausée"
WhitespaceAnalyzer:
[Nausée]
SimpleAnalyzer:
[naus] [e]
StopAnalyzer:
[naus] [e]
StandardAnalyzer:
[naus] [233] [e]
Nasty! Obviously indexing these tokens as shown will give a GIGO result.
So you must decode symbolic and numeric character references before they hit the analyzer, either in your XML/HTML parser or externally.
- J.J.
PS: Also note that when using Luke to see what is indexed, it uses NCRs eg. é to display non-ASCII characters, allowing one to be easily confused as to whether the NCRs were indexed or the Unicode characters were indexed.
At 7:33 AM -0500 12/13/05, Robert Watkins wrote:
>I have been experimenting with a couple of HTML parsers, primarily to
>compare performance, but have discovered a difference in the index for
>which I haven't, with assurance discovered the cause.
>
>The difference is in the number of terms reported by Luke. The indexes
>created with the content parsed using JTidy generally have about 30%
>fewer terms than those created with content parsed using HTMLParser
>(htmlparser.org).
>
>The only difference I can discern (using debug logs and diff) is with
>the way entities are handled by the two parsers. Using JTidy, any HTML
>entities are converted to the literal character; using HTMLParser they
>are left as an entity (named or numeric). In the fields that are
>tokenized, entities not already converted are done so in the index, which
>leaves only the fields not tokenized. It does not seem likely to me that
>this could account for 30% of the terms indexed.
>
>Is it possible to use Luke (or some other tool) to make a more detailed
>comparison of the two indexes? I have tried to find a difference in the
>top terms indexed, and while the order of the top terms does change, the
>numbers do not. Am I missing something obvious?
>
>Thanks,
>-- Robert
>
>--------------------
>Robert Watkins
>rwatkins@foo-bar.org
>--------------------
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: html parsers and numers of terms
Posted by Robert Watkins <rw...@foo-bar.org>.
So obvious I missed it (at least that's my excuse). I'm on the road at
the moment and -- can you believe it? -- didn't bring my copy of Lucene
In Action with me! Looks like I'll have to get the source code from
lucenebook.com to crib the analyzer demo code.
Much obliged,
-- Robert
On Tue, 13 Dec 2005, Erik Hatcher wrote:
> How about taking a single simple HTML file, running it through each parser,
> dumping the tokens into separate collections (or output to a single text
> file) and diff them?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: html parsers and numers of terms
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
How about taking a single simple HTML file, running it through each
parser, dumping the tokens into separate collections (or output to a
single text file) and diff them?
Erik
On Dec 13, 2005, at 7:33 AM, Robert Watkins wrote:
> I have been experimenting with a couple of HTML parsers, primarily to
> compare performance, but have discovered a difference in the index for
> which I haven't, with assurance discovered the cause.
>
> The difference is in the number of terms reported by Luke. The indexes
> created with the content parsed using JTidy generally have about 30%
> fewer terms than those created with content parsed using HTMLParser
> (htmlparser.org).
>
> The only difference I can discern (using debug logs and diff) is with
> the way entities are handled by the two parsers. Using JTidy, any HTML
> entities are converted to the literal character; using HTMLParser they
> are left as an entity (named or numeric). In the fields that are
> tokenized, entities not already converted are done so in the index,
> which
> leaves only the fields not tokenized. It does not seem likely to me
> that
> this could account for 30% of the terms indexed.
>
> Is it possible to use Luke (or some other tool) to make a more
> detailed
> comparison of the two indexes? I have tried to find a difference in
> the
> top terms indexed, and while the order of the top terms does
> change, the
> numbers do not. Am I missing something obvious?
>
> Thanks,
> -- Robert
>
> --------------------
> Robert Watkins
> rwatkins@foo-bar.org
> --------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: html parsers and numers of terms
Posted by ohad <oh...@eng.tau.ac.il>.
Hi All
Just wanted to fill you in on an open source project i have recently
published. It is called Java Mozilla Html Parser ( on sourceforge ,
http://sourceforge.net/projects/mozillaparser ).
I had to create a java html parser that works just like browser do , so the
natural selection was using mozilla's parser. I tried to use JavaXPCOM at
first , but i could not access any of the classes i required (nsParser and
nsContentSink ) - somehow they were not included in the javaIDL... so... I
had to build a jni wrapper around mozilla's parser.
The outcome is a very nice parser that does the job and is stable so far of
what i have seen ( no crashes , no memory leaks). it is even compareable in
performacnce with tagsoup and neko , which are the ones that i have checked.
There are , however , some serious drawbacks that i would appreciate if
anyone will care to show me how to solve them. first and most annoying is
the fact that all parsing is done on a single thread - as you know , when
starting initXPCOM() one creates an event queue that can only be accessed
threw the thread that owns it. I could not proxy the parser objects i had no
matter what i did.
I would appreceate it if someone would try it out and tell me what the
parser feels like , as well as download the code from sourceforge and try to
compile it and see why the proxy things doesn't work.
Thanks everyone in advance
Ohad Serfaty
--
View this message in context: http://www.nabble.com/html-parsers-and-numers-of-terms-tf733866.html#a8397645
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org