You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tika.apache.org by Anne Blankert <an...@geodan.nl> on 2009/10/16 19:20:56 UTC
How to insert whitespace when parsing html
Hello List,
According to the changelog, Tika now inserts whitespace when parsing
html documents, but I do not understand how to get it to work.
My HTML document has a fragment:
<td valign="top">09.0025849<br/><br/></td><td valign="top">Priemstraat
4<br/><br/></td><td valign="top">07-10-2009<br/><br/></td></tr><tr><td
valign="top">1<br/><br/></td>
And the below code snippet produces output:
09.0025849Priemstraat 407-10-20091
I was hoping to get output like:
09.0025849 Priemstraat 4 07-10-2009 1
Any tips on handling this?
Code (using tika-core-0.4.jar and tika-parsers-0.4.jar):
URL htmlPage = new URL(htmlPage);
DataInputStream daHTMLfromDaItem = new
DataInputStream(htmlPage.openStream());
ContentHandler bodyContentHandler = new BodyContentHandler();
Metadata metadata = new Metadata();
AutoDetectParser p = new AutoDetectParser();
try {
// get HTML and convert to text in bodyContentHandler
p.parse(daHTMLfromDaItem, bodyContentHandler, metadata);
System.out.println (bodyContentHandler.toString());
Thanks,
Anneb
Re: How to insert whitespace when parsing html
Posted by Benson Margulies <bi...@gmail.com>.
HTMLParser is the class. You need to make your own, and then set up your own
config XML file to reference it.
On Fri, Oct 23, 2009 at 12:30 PM, Anne Blankert <an...@geodan.nl>wrote:
> Thanks for the suggestion. I do not know which HTML class you mean. I
> therefore subclassed 'DefaultHandler' and I think my whitespace problem has
> been solved.
>
> ContentHandler callback functions "startElement" and "endElement" do NOT
> get called for every HTML element. This seems to be true if the HTML is not
> completely valid. However, most browsers display the HTML content correctly,
> so the TIKA extraction should also ignore the fact that the HTML is not
> valid.
> I discovered that ContentHandler callback function "characters" gets called
> separately for each element, even if the corresponding HTML element is not
> valid. This behaviour can be used to insert a space for each call to
> callback function "characters". There is a problem here: if the element has
> a large content, callback function "characters" gets called more than once
> for the same HTML element. I tried to work around this problem by inserting
> spaces only for element content smaller than (arbitrarily chosen) 100
> characters.
>
> For my purposes this undocumented(?) hack seems to work quite well. I have
> included the code for the ContentHandler subclass below. Any comments or
> suggestions would be appreciated.
>
> Usage example for class BodyContentHandlerWhitespaced
> ===
> URL htmlPage = new URL(UrlString);
> DataInputStream htmlStream = new DataInputStream(htmlPage.openStream());
> ContentHandler bodyContentHandler = new BodyContentHandlerWhitespaced();
> Metadata metadata = new Metadata();
> AutoDetectParser p = new AutoDetectParser();
> try {
> // get HTML and convert to text in bodyContentHandler
> p.parse(htmlStream, bodyContentHandler, metadata);
> System.out.println (bodyContentHandler.toString());
> etc.
> ===
>
> Code for class BodyContentHandlerWhitespaced subclassed from
> ContentHandler:
>
> ===
> import org.xml.sax.helpers.DefaultHandler;
>
> public class BodyContentHandlerWhitespaced extends DefaultHandler {
> private StringBuffer m_content, m_bodyContent, m_otherContent;
> BodyContentHandlerWhitespaced ()
> {
> super();
> m_bodyContent = new StringBuffer();
> m_otherContent = new StringBuffer();
> m_content = m_otherContent;
> }
> public void characters (char[] ch, int start, int length) throws
> org.xml.sax.SAXException
> {
> //insert space for every call to this callback function when length <
> 100
> if (length < 100) {
> m_content.append(" ");
> }
> // store the text into the current content member
> m_content.append(new String(ch, start, length));
> }
> public void startElement(java.lang.String uri,
> java.lang.String localName,
> java.lang.String name,
> org.xml.sax.Attributes atts)
> throws org.xml.sax.SAXException
> {
> if (localName.equalsIgnoreCase("body"))
> {
> //set current content member to m_bodyContent
> m_content = m_bodyContent;
> }
> }
> public void endElement(java.lang.String uri,
> java.lang.String localName,
> java.lang.String name)
> throws org.xml.sax.SAXException
> {
> if (localName.equalsIgnoreCase("body"))
> {
> //set current content member to m_otherContent
> m_content = m_otherContent;
> }
> }
> public String toString()
> {
> // return content
> if (m_bodyContent.length() == 0)
> {
> // no body content available, return all content
> return m_otherContent.toString();
> }
> // return only body content
> return m_bodyContent.toString();
> }
> }
>
>
>
> On 2009-10-23 17:34, Benson Margulies wrote:
>
>> If you want to keep <br/>, you need to subclass and customize the HTML
>> class. A recent change makes this simples.
>>
>> On Fri, Oct 23, 2009 at 11:20 AM, Anne Blankert <anne.blankert@geodan.nl<mailto:
>> anne.blankert@geodan.nl>> wrote:
>>
>> Extra information:
>>
>> The source: http://www2.nijmegen.nl/content/869991/kapvergunningen
>> does not validate through http://validator.w3.org because the
>> <table> element is in between <p> </p> like this: <p><table>
>> .....</table></p>
>>
>> Could that be the reason why the ContentHandler callback methods
>> are not called for the elements inside the <table> element?
>>
>> In browsers, this page displays normally.
>>
>>
>>
>>
>
> --
>
> Anne Blankert
>
> Geodan Systems & Research
>
>
>
>
Re: How to insert whitespace when parsing html
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
Your problem was reported earlier as TIKA-268 [1] and is fixed in Tika trunk.
[1] https://issues.apache.org/jira/browse/TIKA-268
BR,
Jukka Zitting
Re: How to insert whitespace when parsing html
Posted by Anne Blankert <an...@geodan.nl>.
Thanks for the suggestion. I do not know which HTML class you mean. I
therefore subclassed 'DefaultHandler' and I think my whitespace problem
has been solved.
ContentHandler callback functions "startElement" and "endElement" do NOT
get called for every HTML element. This seems to be true if the HTML is
not completely valid. However, most browsers display the HTML content
correctly, so the TIKA extraction should also ignore the fact that the
HTML is not valid.
I discovered that ContentHandler callback function "characters" gets
called separately for each element, even if the corresponding HTML
element is not valid. This behaviour can be used to insert a space for
each call to callback function "characters". There is a problem here: if
the element has a large content, callback function "characters" gets
called more than once for the same HTML element. I tried to work around
this problem by inserting spaces only for element content smaller than
(arbitrarily chosen) 100 characters.
For my purposes this undocumented(?) hack seems to work quite well. I
have included the code for the ContentHandler subclass below. Any
comments or suggestions would be appreciated.
Usage example for class BodyContentHandlerWhitespaced
===
URL htmlPage = new URL(UrlString);
DataInputStream htmlStream = new DataInputStream(htmlPage.openStream());
ContentHandler bodyContentHandler = new BodyContentHandlerWhitespaced();
Metadata metadata = new Metadata();
AutoDetectParser p = new AutoDetectParser();
try {
// get HTML and convert to text in bodyContentHandler
p.parse(htmlStream, bodyContentHandler, metadata);
System.out.println (bodyContentHandler.toString());
etc.
===
Code for class BodyContentHandlerWhitespaced subclassed from ContentHandler:
===
import org.xml.sax.helpers.DefaultHandler;
public class BodyContentHandlerWhitespaced extends DefaultHandler {
private StringBuffer m_content, m_bodyContent, m_otherContent;
BodyContentHandlerWhitespaced ()
{
super();
m_bodyContent = new StringBuffer();
m_otherContent = new StringBuffer();
m_content = m_otherContent;
}
public void characters (char[] ch, int start, int length) throws
org.xml.sax.SAXException
{
//insert space for every call to this callback function when
length < 100
if (length < 100)
{
m_content.append(" ");
}
// store the text into the current content member
m_content.append(new String(ch, start, length));
}
public void startElement(java.lang.String uri,
java.lang.String localName,
java.lang.String name,
org.xml.sax.Attributes atts)
throws org.xml.sax.SAXException
{
if (localName.equalsIgnoreCase("body"))
{
//set current content member to m_bodyContent
m_content = m_bodyContent;
}
}
public void endElement(java.lang.String uri,
java.lang.String localName,
java.lang.String name)
throws org.xml.sax.SAXException
{
if (localName.equalsIgnoreCase("body"))
{
//set current content member to m_otherContent
m_content = m_otherContent;
}
}
public String toString()
{
// return content
if (m_bodyContent.length() == 0)
{
// no body content available, return all content
return m_otherContent.toString();
}
// return only body content
return m_bodyContent.toString();
}
}
On 2009-10-23 17:34, Benson Margulies wrote:
> If you want to keep <br/>, you need to subclass and customize the HTML
> class. A recent change makes this simples.
>
> On Fri, Oct 23, 2009 at 11:20 AM, Anne Blankert
> <anne.blankert@geodan.nl <ma...@geodan.nl>> wrote:
>
> Extra information:
>
> The source: http://www2.nijmegen.nl/content/869991/kapvergunningen
> does not validate through http://validator.w3.org because the
> <table> element is in between <p> </p> like this: <p><table>
> .....</table></p>
>
> Could that be the reason why the ContentHandler callback methods
> are not called for the elements inside the <table> element?
>
> In browsers, this page displays normally.
>
>
>
--
Anne Blankert
Geodan Systems & Research
Re: How to insert whitespace when parsing html
Posted by Benson Margulies <bi...@gmail.com>.
If you want to keep <br/>, you need to subclass and customize the HTML
class. A recent change makes this simples.
On Fri, Oct 23, 2009 at 11:20 AM, Anne Blankert <an...@geodan.nl>wrote:
> Extra information:
>
> The source: http://www2.nijmegen.nl/content/869991/kapvergunningen
> does not validate through http://validator.w3.org because the <table>
> element is in between <p> </p> like this: <p><table> .....</table></p>
>
> Could that be the reason why the ContentHandler callback methods are not
> called for the elements inside the <table> element?
>
> In browsers, this page displays normally.
>
>
>
Re: How to insert whitespace when parsing html
Posted by Anne Blankert <an...@geodan.nl>.
Extra information:
The source: http://www2.nijmegen.nl/content/869991/kapvergunningen
does not validate through http://validator.w3.org because the <table>
element is in between <p> </p> like this: <p><table> .....</table></p>
Could that be the reason why the ContentHandler callback methods are not
called for the elements inside the <table> element?
In browsers, this page displays normally.
Re: How to insert whitespace when parsing html
Posted by Anne Blankert <an...@geodan.nl>.
Some new information:
The problem seems to be in the use of <br/>. I created a new class that
extends the Contenthandler and found that callback methods 'characters',
'startElement' and 'endElement' are NOT called for the <td> elements
that contain <br/> elements.
For HTML the <br/> really means something, so I think this should
somehow be converted to whitespace or something else. I think it should
not be ignored. Am I overlooking something?
On 2009-10-16 19:20, Anne Blankert wrote:
> Hello List,
>
> According to the changelog, Tika now inserts whitespace when parsing
> html documents, but I do not understand how to get it to work.
>
> My HTML document has a fragment:
> <td valign="top">09.0025849<br/><br/></td><td valign="top">Priemstraat
> 4<br/><br/></td><td valign="top">07-10-2009<br/><br/></td></tr><tr><td
> valign="top">1<br/><br/></td>
>
> And the below code snippet produces output:
> 09.0025849Priemstraat 407-10-20091
>
> I was hoping to get output like:
> 09.0025849 Priemstraat 4 07-10-2009 1
>
> Any tips on handling this?
>
> Code (using tika-core-0.4.jar and tika-parsers-0.4.jar):
>
> URL htmlPage = new URL(htmlPage);
> DataInputStream daHTMLfromDaItem = new
> DataInputStream(htmlPage.openStream());
> ContentHandler bodyContentHandler = new BodyContentHandler();
> Metadata metadata = new Metadata();
> AutoDetectParser p = new AutoDetectParser();
> try {
> // get HTML and convert to text in bodyContentHandler
> p.parse(daHTMLfromDaItem, bodyContentHandler, metadata);
> System.out.println (bodyContentHandler.toString());
>
>
>
> Thanks,
>
> Anneb
>