You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Otis Gospodnetic <ot...@yahoo.com> on 2002/02/15 21:47:10 UTC
Re: HTMLParser
While you are at it, perhaps it would be good to add support for add
other META tags, such as "robots", especially since people are working
on adding a web crawler component to Lucene.
Thanks,
Otis
--- Daniel Calvo <dc...@task.com.br> wrote:
> Hi,
>
> I was playing with HTMLParser.jj and made some changes you might be
> interested in. What I did was start handling <META> tags (added
> new methods: getAuthor, getKeywords and getMetadata and changed
> getSummary to check if there's any metadata item with
> name=="description"). I'm also filtering out any text inside
> <STYLE>...</STYLE> (like <SCRIPT> is being handled).
> I've performed some tests and I belive I didn't break anything ;-)
>
> The patch is as follows
>
> Best regards,
>
> --Daniel
>
> Index: HTMLParser.jj
> ===================================================================
> RCS file:
>
/home/cvspublic/jakarta-lucene/src/demo/org/apache/lucene/demo/html/HTMLParser.jj,v
> retrieving revision 1.1
> diff -u -r1.1 HTMLParser.jj
> --- HTMLParser.jj 26 Jan 2002 15:01:31 -0000 1.1
> +++ HTMLParser.jj 15 Feb 2002 20:39:49 -0000
> @@ -66,6 +66,8 @@
> package org.apache.lucene.demo.html;
>
> import java.io.*;
> +import java.util.Map;
> +import java.util.HashMap;
>
> public class HTMLParser {
> public static int SUMMARY_LENGTH = 200;
> @@ -76,11 +78,13 @@
> boolean titleComplete = false;
> boolean inTitle = false;
> boolean inScript = false;
> + boolean inStyle = false;
> boolean afterTag = false;
> boolean afterSpace = false;
> String eol = System.getProperty("line.separator");
> PipedReader pipeIn = null;
> PipedWriter pipeOut;
> + HashMap metadata = new HashMap(7);
>
> public HTMLParser(File file) throws FileNotFoundException {
> this(new FileInputStream(file));
> @@ -109,15 +113,60 @@
> wait(10);
> }
> }
> - if (summary.length() > SUMMARY_LENGTH)
> - summary.setLength(SUMMARY_LENGTH);
> + // look in metadata
> + String description = (String) metadata.get("description");
> + if (description != null)
> + return description;
> + else {
> + if (summary.length() > SUMMARY_LENGTH)
> + summary.setLength(SUMMARY_LENGTH);
> +
> + String sum = summary.toString().trim();
> + String tit = getTitle();
> + if (sum.startsWith(tit))
> + return sum.substring(tit.length());
> + else
> + return sum;
> + }
> + }
> +
> + public String getAuthor() throws IOException, InterruptedException
> {
> + if (pipeIn == null)
> + getReader(); // spawn parsing thread
> + while (true) {
> + synchronized(this) {
> + if (summary.length() > 0) // assume that all metadata
> + break; // has already been collected
> + wait(10);
> + }
> + }
> + return (String)metadata.get("author");
> + }
> +
> + public String getKeywords() throws IOException,
> InterruptedException {
> + if (pipeIn == null)
> + getReader(); // spawn parsing thread
> + while (true) {
> + synchronized(this) {
> + if (summary.length() > 0) // assume that all metadata
> + break; // has already been collected
> + wait(10);
> + }
> + }
> + return (String)metadata.get("keywords");
> + }
>
> - String sum = summary.toString().trim();
> - String tit = getTitle();
> - if (sum.startsWith(tit))
> - return sum.substring(tit.length());
> - else
> - return sum;
> + public Map getMetadata() throws IOException, InterruptedException
> {
> + if (pipeIn == null)
> + getReader(); // spawn parsing thread
> + while (true) {
> + synchronized(this) {
> + if (summary.length() > 0) // assume that all metadata
> + break; // has already been collected
> + wait(10);
> + }
> + }
> + return metadata;
> }
>
> public Reader getReader() throws IOException {
> @@ -144,7 +193,7 @@
> }
>
> void addText(String text) throws IOException {
> - if (inScript)
> + if (inScript || inStyle)
> return;
> if (inTitle)
> title.append(text);
> @@ -165,7 +214,7 @@
> }
>
> void addSpace() throws IOException {
> - if (inScript)
> + if (inScript || inStyle)
> return;
> if (!afterSpace) {
> if (inTitle)
> @@ -216,23 +265,38 @@
> {
> Token t1, t2;
> boolean inImg = false;
> + boolean inMeta = false;
> + String name = null;
> + String content = null;
> }
> {
> t1=<TagName> {
> - inTitle = t1.image.equalsIgnoreCase("<title"); // keep track if
> in <TITLE>
> - inImg = t1.image.equalsIgnoreCase("<img"); // keep track if in
> <IMG>
> - if (inScript) { // keep track if in <SCRIPT>
> + inTitle = t1.image.equalsIgnoreCase("<title"); // keep track
> if in <TITLE>
> + inImg = t1.image.equalsIgnoreCase("<img"); // keep track
> if in <IMG>
> + inMeta = t1.image.equalsIgnoreCase("<meta"); // keep track
> if in <META>
> + if (inScript) { // keep track
> if in <SCRIPT>
> inScript = !t1.image.equalsIgnoreCase("</script");
> } else {
> inScript = t1.image.equalsIgnoreCase("<script");
> }
> + if (inStyle) { // keep track
> if in <STYLE>
> + inStyle = !t1.image.equalsIgnoreCase("</style");
> + } else {
> + inStyle = t1.image.equalsIgnoreCase("<style");
> + }
> }
> (t1=<ArgName>
> (<ArgEquals>
> - (t2=ArgValue() // save ALT text in IMG tag
> + (t2=ArgValue()
> {
> if (inImg && t1.image.equalsIgnoreCase("alt") && t2 != null)
> - addText("[" + t2.image + "]");
> + addText("[" + t2.image + "]"); // save ALT text in
> IMG tag
> + if (inMeta && t1.image.equalsIgnoreCase("name") && t2 !=
> null)
> + name = t2.image.toLowerCase(); // save name in META
> tag
> + if (inMeta && t1.image.equalsIgnoreCase("content") && t2 !=
> null)
> + content = t2.image; // save content in
> META tag
> + if (inMeta && name != null && content != null)
> + metadata.put(name, content); // save metadata
> }
> )?
> )?
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do You Yahoo!?
Got something to say? Say it better with Yahoo! Video Mail
http://mail.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: HTMLParser
Posted by Erik Hatcher <li...@ehatchersolutions.com>.
I'm wondering how this compares to JTidy. Anyone know?
How does HTMLParser.jj compare to JTidy's capabilities?
Erik
----- Original Message -----
From: "Paulo Gaspar" <pa...@krankikom.de>
To: "Lucene Developers List" <lu...@jakarta.apache.org>
Sent: Friday, February 15, 2002 9:13 PM
Subject: RE: HTMLParser
> Can the following Xerces based HTML parser be interesting for
> your work?
>
> This is just the initial ANNOUNCE but there are further
> developments.
>
>
> Have fun,
> Paulo Gaspar
>
> > -----Original Message-----
> > From: Andy Clark [mailto:andyc@apache.org]
> > Sent: Saturday, February 09, 2002 4:16 AM
> > To: general@xml.apache.org
> > Cc: xerces-j-dev@xml.apache.org
> > Subject: [ANNOUNCE] Xerces HTML Parser
> >
> >
> > For a long time users have asked if Xerces can parse HTML files.
> > But since most HTML documents are not well-formed XML documents,
> > it is generally not possible to use a conforming XML parser to
> > read HTML documents.
> >
> > However, the Xerces Native Interface (XNI) that is the foundation
> > of the Xerces2 implementation defines a framework that allows
> > different kinds of parsers to be constructed by connecting a
> > pipeline of parser components. Therefore, as long as a component
> > can be written that generates the appropriate XNI "events", then
> > it can be used to emit SAX events, build DOM trees, or anything
> > else that you can think of.
> >
> > So, as a fun little exercise, I have written a basic HTML parser
> > using XNI. It consists of an HTML scanner component that can scan
> > HTML files and generate XNI events and a tag balancing component.
> > The tag balancer cleans up the events produced by the scanner,
> > balancing mismatched tags and adding tags where necessary. And
> > it does all of this in a streaming manner to minimize the amount
> > of memory required.
> >
> > Since I wrote the HTML parser as an example of using XNI and
> > because the code is considered alpha quality (but it seems to
> > work quite well, actually!), I am posting the code with a very
> > limited license. Even though it contains the complete source
> > code for the HTML parser, the license only allows the user to
> > experiment but gives no right to actually use the code in a
> > product.
> >
> > If the source isn't "free" or "open", why release it at all?
> > I want to get an idea of what people think of the code first.
> > Then, if there's enough interest, I would like to either donate
> > the code to the Xerces-J project or make it available elsewhere
> > under a true open source license.
> >
> > So, if you've been looking for a way to parse HTML documents
> > please try out the HTML parser and let me know what you think.
> > There should be enough information in the documentation to get
> > you started. Check out the "NekoHTML" project listed on my
> > Apache web site: http://www.apache.org/~andyc/
> >
> > Have fun!
> >
> > --
> > Andy Clark * andyc@apache.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: HTMLParser
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hm, I thought this place would have a PDF parser, but it does not.
It does seem to have a RTF parser:
http://cobase-www.cs.ucla.edu/pub/javacc/
Perhaps some of these things can be adopted by Lucene, people could
contribute Java classes for interacting with specific parsers, and all
that could then be included in Lucene to work together with those
DocumentHandlers mentioned a few days ago.
Otis
--- Daniel Calvo <dc...@task.com.br> wrote:
> Maybe...I'll have to give it a try first
>
> Anyway, I was playing with Lucene's HTMParser in order to understand
> a little better how JavaCC works. My real interest is in PDF
> and RTF parsers. I've tried Websearch PDF parser but it only worked
> well with the examples provided. I wasn't able to parse
> correctly even PDF files distributed by Adobe. I've also had a lot of
> trouble with files converted to PDF (probably via dvi2pdf or
> something like that). Recently I read on this list (or maybe it was
> on the users list) that someone else was having trouble with
> both Websearch and PJ library parsers.
>
> I've just downloaded Adobe's PDF Specification and later I'll try to
> see if there's any room for improvement in Websearch code. I
> know PDF has various features (compression, cryptography, etc.) that
> complicate the parsing and I'm not willing to spend much time
> doing this but I'll probably try something.
>
> --Daniel
>
> > -----Original Message-----
> > From: Paulo Gaspar [mailto:paulo.gaspar@krankikom.de]
> > Sent: sexta-feira, 15 de fevereiro de 2002 23:14
> > To: Lucene Developers List
> > Subject: RE: HTMLParser
> >
> >
> > Can the following Xerces based HTML parser be interesting for
> > your work?
> >
> > This is just the initial ANNOUNCE but there are further
> > developments.
> >
> >
> > Have fun,
> > Paulo Gaspar
> >
> > > -----Original Message-----
> > > From: Andy Clark [mailto:andyc@apache.org]
> > > Sent: Saturday, February 09, 2002 4:16 AM
> > > To: general@xml.apache.org
> > > Cc: xerces-j-dev@xml.apache.org
> > > Subject: [ANNOUNCE] Xerces HTML Parser
> > >
> > >
> > > For a long time users have asked if Xerces can parse HTML files.
> > > But since most HTML documents are not well-formed XML documents,
> > > it is generally not possible to use a conforming XML parser to
> > > read HTML documents.
> > >
> > > However, the Xerces Native Interface (XNI) that is the foundation
> > > of the Xerces2 implementation defines a framework that allows
> > > different kinds of parsers to be constructed by connecting a
> > > pipeline of parser components. Therefore, as long as a component
> > > can be written that generates the appropriate XNI "events", then
> > > it can be used to emit SAX events, build DOM trees, or anything
> > > else that you can think of.
> > >
> > > So, as a fun little exercise, I have written a basic HTML parser
> > > using XNI. It consists of an HTML scanner component that can scan
> > > HTML files and generate XNI events and a tag balancing component.
> > > The tag balancer cleans up the events produced by the scanner,
> > > balancing mismatched tags and adding tags where necessary. And
> > > it does all of this in a streaming manner to minimize the amount
> > > of memory required.
> > >
> > > Since I wrote the HTML parser as an example of using XNI and
> > > because the code is considered alpha quality (but it seems to
> > > work quite well, actually!), I am posting the code with a very
> > > limited license. Even though it contains the complete source
> > > code for the HTML parser, the license only allows the user to
> > > experiment but gives no right to actually use the code in a
> > > product.
> > >
> > > If the source isn't "free" or "open", why release it at all?
> > > I want to get an idea of what people think of the code first.
> > > Then, if there's enough interest, I would like to either donate
> > > the code to the Xerces-J project or make it available elsewhere
> > > under a true open source license.
> > >
> > > So, if you've been looking for a way to parse HTML documents
> > > please try out the HTML parser and let me know what you think.
> > > There should be enough information in the documentation to get
> > > you started. Check out the "NekoHTML" project listed on my
> > > Apache web site: http://www.apache.org/~andyc/
> > >
> > > Have fun!
> > >
> > > --
> > > Andy Clark * andyc@apache.org
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > >
> >
> > --
> > To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> <ma...@jakarta.apache.org>
> >
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: HTMLParser
Posted by Daniel Calvo <dc...@task.com.br>.
Maybe...I'll have to give it a try first
Anyway, I was playing with Lucene's HTMParser in order to understand a little better how JavaCC works. My real interest is in PDF
and RTF parsers. I've tried Websearch PDF parser but it only worked well with the examples provided. I wasn't able to parse
correctly even PDF files distributed by Adobe. I've also had a lot of trouble with files converted to PDF (probably via dvi2pdf or
something like that). Recently I read on this list (or maybe it was on the users list) that someone else was having trouble with
both Websearch and PJ library parsers.
I've just downloaded Adobe's PDF Specification and later I'll try to see if there's any room for improvement in Websearch code. I
know PDF has various features (compression, cryptography, etc.) that complicate the parsing and I'm not willing to spend much time
doing this but I'll probably try something.
--Daniel
> -----Original Message-----
> From: Paulo Gaspar [mailto:paulo.gaspar@krankikom.de]
> Sent: sexta-feira, 15 de fevereiro de 2002 23:14
> To: Lucene Developers List
> Subject: RE: HTMLParser
>
>
> Can the following Xerces based HTML parser be interesting for
> your work?
>
> This is just the initial ANNOUNCE but there are further
> developments.
>
>
> Have fun,
> Paulo Gaspar
>
> > -----Original Message-----
> > From: Andy Clark [mailto:andyc@apache.org]
> > Sent: Saturday, February 09, 2002 4:16 AM
> > To: general@xml.apache.org
> > Cc: xerces-j-dev@xml.apache.org
> > Subject: [ANNOUNCE] Xerces HTML Parser
> >
> >
> > For a long time users have asked if Xerces can parse HTML files.
> > But since most HTML documents are not well-formed XML documents,
> > it is generally not possible to use a conforming XML parser to
> > read HTML documents.
> >
> > However, the Xerces Native Interface (XNI) that is the foundation
> > of the Xerces2 implementation defines a framework that allows
> > different kinds of parsers to be constructed by connecting a
> > pipeline of parser components. Therefore, as long as a component
> > can be written that generates the appropriate XNI "events", then
> > it can be used to emit SAX events, build DOM trees, or anything
> > else that you can think of.
> >
> > So, as a fun little exercise, I have written a basic HTML parser
> > using XNI. It consists of an HTML scanner component that can scan
> > HTML files and generate XNI events and a tag balancing component.
> > The tag balancer cleans up the events produced by the scanner,
> > balancing mismatched tags and adding tags where necessary. And
> > it does all of this in a streaming manner to minimize the amount
> > of memory required.
> >
> > Since I wrote the HTML parser as an example of using XNI and
> > because the code is considered alpha quality (but it seems to
> > work quite well, actually!), I am posting the code with a very
> > limited license. Even though it contains the complete source
> > code for the HTML parser, the license only allows the user to
> > experiment but gives no right to actually use the code in a
> > product.
> >
> > If the source isn't "free" or "open", why release it at all?
> > I want to get an idea of what people think of the code first.
> > Then, if there's enough interest, I would like to either donate
> > the code to the Xerces-J project or make it available elsewhere
> > under a true open source license.
> >
> > So, if you've been looking for a way to parse HTML documents
> > please try out the HTML parser and let me know what you think.
> > There should be enough information in the documentation to get
> > you started. Check out the "NekoHTML" project listed on my
> > Apache web site: http://www.apache.org/~andyc/
> >
> > Have fun!
> >
> > --
> > Andy Clark * andyc@apache.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: HTMLParser
Posted by Paulo Gaspar <pa...@krankikom.de>.
Can the following Xerces based HTML parser be interesting for
your work?
This is just the initial ANNOUNCE but there are further
developments.
Have fun,
Paulo Gaspar
> -----Original Message-----
> From: Andy Clark [mailto:andyc@apache.org]
> Sent: Saturday, February 09, 2002 4:16 AM
> To: general@xml.apache.org
> Cc: xerces-j-dev@xml.apache.org
> Subject: [ANNOUNCE] Xerces HTML Parser
>
>
> For a long time users have asked if Xerces can parse HTML files.
> But since most HTML documents are not well-formed XML documents,
> it is generally not possible to use a conforming XML parser to
> read HTML documents.
>
> However, the Xerces Native Interface (XNI) that is the foundation
> of the Xerces2 implementation defines a framework that allows
> different kinds of parsers to be constructed by connecting a
> pipeline of parser components. Therefore, as long as a component
> can be written that generates the appropriate XNI "events", then
> it can be used to emit SAX events, build DOM trees, or anything
> else that you can think of.
>
> So, as a fun little exercise, I have written a basic HTML parser
> using XNI. It consists of an HTML scanner component that can scan
> HTML files and generate XNI events and a tag balancing component.
> The tag balancer cleans up the events produced by the scanner,
> balancing mismatched tags and adding tags where necessary. And
> it does all of this in a streaming manner to minimize the amount
> of memory required.
>
> Since I wrote the HTML parser as an example of using XNI and
> because the code is considered alpha quality (but it seems to
> work quite well, actually!), I am posting the code with a very
> limited license. Even though it contains the complete source
> code for the HTML parser, the license only allows the user to
> experiment but gives no right to actually use the code in a
> product.
>
> If the source isn't "free" or "open", why release it at all?
> I want to get an idea of what people think of the code first.
> Then, if there's enough interest, I would like to either donate
> the code to the Xerces-J project or make it available elsewhere
> under a true open source license.
>
> So, if you've been looking for a way to parse HTML documents
> please try out the HTML parser and let me know what you think.
> There should be enough information in the documentation to get
> you started. Check out the "NekoHTML" project listed on my
> Apache web site: http://www.apache.org/~andyc/
>
> Have fun!
>
> --
> Andy Clark * andyc@apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>