You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tika.apache.org by Anne Blankert <an...@geodan.nl> on 2009/10/16 19:20:56 UTC

How to insert whitespace when parsing html

Hello List,

According to the changelog, Tika now inserts whitespace when parsing 
html documents, but I do not understand how to get it to work.

My HTML document has a fragment:
<td valign="top">09.0025849<br/><br/></td><td valign="top">Priemstraat 
4<br/><br/></td><td valign="top">07-10-2009<br/><br/></td></tr><tr><td 
valign="top">1<br/><br/></td>

And the below code snippet produces output:
09.0025849Priemstraat 407-10-20091

I was hoping to get output like:
09.0025849 Priemstraat 4 07-10-2009 1

Any tips on handling this?

Code (using tika-core-0.4.jar and tika-parsers-0.4.jar):

  URL htmlPage = new URL(htmlPage);
  DataInputStream daHTMLfromDaItem = new 
DataInputStream(htmlPage.openStream());
  ContentHandler bodyContentHandler = new BodyContentHandler();        
  Metadata metadata = new Metadata();
  AutoDetectParser p = new AutoDetectParser();
  try {
           // get HTML and convert to text in bodyContentHandler
           p.parse(daHTMLfromDaItem, bodyContentHandler, metadata);
           System.out.println (bodyContentHandler.toString());



Thanks,

Anneb


Re: How to insert whitespace when parsing html

Posted by Benson Margulies <bi...@gmail.com>.
HTMLParser is the class. You need to make your own, and then set up your own
config XML file to reference it.

On Fri, Oct 23, 2009 at 12:30 PM, Anne Blankert <an...@geodan.nl>wrote:

> Thanks for the suggestion. I do not know which HTML class you mean. I
> therefore subclassed 'DefaultHandler' and I think my whitespace problem has
> been solved.
>
> ContentHandler callback functions "startElement" and "endElement" do NOT
> get called for every HTML element. This seems to be true if the HTML is not
> completely valid. However, most browsers display the HTML content correctly,
> so the TIKA extraction should also ignore the fact that the HTML is not
> valid.
> I discovered that ContentHandler callback function "characters" gets called
> separately for each element, even if the corresponding HTML element is not
> valid. This behaviour can be used to insert a space for  each call to
> callback function "characters". There is a problem here: if the element has
> a large content, callback function "characters" gets called more than once
> for the same HTML element. I tried to work around this problem by inserting
> spaces only for element content smaller than (arbitrarily chosen) 100
> characters.
>
> For my purposes this undocumented(?) hack seems to work quite well. I have
> included the code for the ContentHandler subclass below. Any comments or
> suggestions would be appreciated.
>
> Usage example for class BodyContentHandlerWhitespaced
> ===
> URL htmlPage = new URL(UrlString);
> DataInputStream htmlStream = new DataInputStream(htmlPage.openStream());
> ContentHandler bodyContentHandler = new BodyContentHandlerWhitespaced();
> Metadata metadata = new Metadata();
> AutoDetectParser p = new AutoDetectParser();
> try {
>         // get HTML and convert to text in bodyContentHandler
>         p.parse(htmlStream, bodyContentHandler, metadata);
>         System.out.println (bodyContentHandler.toString());
> etc.
> ===
>
> Code for class BodyContentHandlerWhitespaced subclassed from
> ContentHandler:
>
> ===
> import org.xml.sax.helpers.DefaultHandler;
>
> public class BodyContentHandlerWhitespaced extends DefaultHandler {
>     private StringBuffer m_content, m_bodyContent, m_otherContent;
>     BodyContentHandlerWhitespaced ()
>   {
>       super();
>       m_bodyContent = new StringBuffer();
>       m_otherContent = new StringBuffer();
>       m_content = m_otherContent;
>   }
>     public void characters (char[] ch, int start, int length) throws
> org.xml.sax.SAXException
>   {
>       //insert space for every call to this callback function when length <
> 100
>       if (length < 100)         {
>           m_content.append(" ");
>       }
>       // store the text into the current content member
>       m_content.append(new String(ch, start, length));
>   }
>     public void startElement(java.lang.String uri,
>           java.lang.String localName,
>           java.lang.String name,
>           org.xml.sax.Attributes atts)
>    throws org.xml.sax.SAXException
>    {
>       if (localName.equalsIgnoreCase("body"))
>       {
>           //set current content member to m_bodyContent
>           m_content = m_bodyContent;
>       }
>    }
>     public void endElement(java.lang.String uri,
>           java.lang.String localName,
>           java.lang.String name)
>    throws org.xml.sax.SAXException
>    {
>       if (localName.equalsIgnoreCase("body"))
>       {
>           //set current content member to m_otherContent
>           m_content = m_otherContent;
>       }
>    }
>     public String toString()
>   {
>       // return content
>       if (m_bodyContent.length() == 0)
>       {
>           // no body content available, return all content
>           return m_otherContent.toString();
>       }
>       // return only body content
>       return m_bodyContent.toString();
>   }
> }
>
>
>
> On 2009-10-23 17:34, Benson Margulies wrote:
>
>> If you want to keep <br/>, you need to subclass and customize the HTML
>> class. A recent change makes this simples.
>>
>> On Fri, Oct 23, 2009 at 11:20 AM, Anne Blankert <anne.blankert@geodan.nl<mailto:
>> anne.blankert@geodan.nl>> wrote:
>>
>>    Extra information:
>>
>>    The source: http://www2.nijmegen.nl/content/869991/kapvergunningen
>>    does not validate through http://validator.w3.org because the
>>    <table> element is in between <p> </p> like this: <p><table>
>>    .....</table></p>
>>
>>    Could that be the reason why the ContentHandler callback methods
>>    are not called for the elements inside the <table> element?
>>
>>    In browsers, this page displays normally.
>>
>>
>>
>>
>
> --
>
> Anne Blankert
>
> Geodan Systems & Research
>
>
>
>

Re: How to insert whitespace when parsing html

Posted by Jukka Zitting <ju...@gmail.com>.
Hi,

Your problem was reported earlier as TIKA-268 [1] and is fixed in Tika trunk.

[1] https://issues.apache.org/jira/browse/TIKA-268

BR,

Jukka Zitting

Re: How to insert whitespace when parsing html

Posted by Anne Blankert <an...@geodan.nl>.
Thanks for the suggestion. I do not know which HTML class you mean. I 
therefore subclassed 'DefaultHandler' and I think my whitespace problem 
has been solved.

ContentHandler callback functions "startElement" and "endElement" do NOT 
get called for every HTML element. This seems to be true if the HTML is 
not completely valid. However, most browsers display the HTML content 
correctly, so the TIKA extraction should also ignore the fact that the 
HTML is not valid.
I discovered that ContentHandler callback function "characters" gets 
called separately for each element, even if the corresponding HTML 
element is not valid. This behaviour can be used to insert a space for  
each call to callback function "characters". There is a problem here: if 
the element has a large content, callback function "characters" gets 
called more than once for the same HTML element. I tried to work around 
this problem by inserting spaces only for element content smaller than 
(arbitrarily chosen) 100 characters.

For my purposes this undocumented(?) hack seems to work quite well. I 
have included the code for the ContentHandler subclass below. Any 
comments or suggestions would be appreciated.

Usage example for class BodyContentHandlerWhitespaced
===
URL htmlPage = new URL(UrlString);
 DataInputStream htmlStream = new DataInputStream(htmlPage.openStream());
 ContentHandler bodyContentHandler = new BodyContentHandlerWhitespaced();
 Metadata metadata = new Metadata();
 AutoDetectParser p = new AutoDetectParser();
 try {
          // get HTML and convert to text in bodyContentHandler
          p.parse(htmlStream, bodyContentHandler, metadata);
          System.out.println (bodyContentHandler.toString());
etc.
===

Code for class BodyContentHandlerWhitespaced subclassed from ContentHandler:

===
import org.xml.sax.helpers.DefaultHandler;

public class BodyContentHandlerWhitespaced extends DefaultHandler {
   
    private StringBuffer m_content, m_bodyContent, m_otherContent;
   
    BodyContentHandlerWhitespaced ()
    {
        super();
        m_bodyContent = new StringBuffer();
        m_otherContent = new StringBuffer();
        m_content = m_otherContent;
    }
   
    public void characters (char[] ch, int start, int length) throws 
org.xml.sax.SAXException
    {
        //insert space for every call to this callback function when 
length < 100
        if (length < 100)   
        {
            m_content.append(" ");
        }
        // store the text into the current content member
        m_content.append(new String(ch, start, length));
    }
   
    public void startElement(java.lang.String uri,
            java.lang.String localName,
            java.lang.String name,
            org.xml.sax.Attributes atts)
     throws org.xml.sax.SAXException
     {
        if (localName.equalsIgnoreCase("body"))
        {
            //set current content member to m_bodyContent
            m_content = m_bodyContent;
        }
     }
   
    public void endElement(java.lang.String uri,
            java.lang.String localName,
            java.lang.String name)
     throws org.xml.sax.SAXException
     {
        if (localName.equalsIgnoreCase("body"))
        {
            //set current content member to m_otherContent
            m_content = m_otherContent;
        }
     }
   
    public String toString()
    {
        // return content
        if (m_bodyContent.length() == 0)
        {
            // no body content available, return all content
            return m_otherContent.toString();
        }
        // return only body content
        return m_bodyContent.toString();
    }
}



On 2009-10-23 17:34, Benson Margulies wrote:
> If you want to keep <br/>, you need to subclass and customize the HTML 
> class. A recent change makes this simples.
>
> On Fri, Oct 23, 2009 at 11:20 AM, Anne Blankert 
> <anne.blankert@geodan.nl <ma...@geodan.nl>> wrote:
>
>     Extra information:
>
>     The source: http://www2.nijmegen.nl/content/869991/kapvergunningen
>     does not validate through http://validator.w3.org because the
>     <table> element is in between <p> </p> like this: <p><table>
>     .....</table></p>
>
>     Could that be the reason why the ContentHandler callback methods
>     are not called for the elements inside the <table> element?
>
>     In browsers, this page displays normally.
>
>
>


-- 

Anne Blankert

Geodan Systems & Research




Re: How to insert whitespace when parsing html

Posted by Benson Margulies <bi...@gmail.com>.
If you want to keep <br/>, you need to subclass and customize the HTML
class. A recent change makes this simples.

On Fri, Oct 23, 2009 at 11:20 AM, Anne Blankert <an...@geodan.nl>wrote:

> Extra information:
>
> The source: http://www2.nijmegen.nl/content/869991/kapvergunningen
> does not validate through http://validator.w3.org because the <table>
> element is in between <p> </p> like this: <p><table> .....</table></p>
>
> Could that be the reason why the ContentHandler callback methods are not
> called for the elements inside the <table> element?
>
> In browsers, this page displays normally.
>
>
>

Re: How to insert whitespace when parsing html

Posted by Anne Blankert <an...@geodan.nl>.
Extra information:

The source: http://www2.nijmegen.nl/content/869991/kapvergunningen
does not validate through http://validator.w3.org because the <table> 
element is in between <p> </p> like this: <p><table> .....</table></p>

Could that be the reason why the ContentHandler callback methods are not 
called for the elements inside the <table> element?

In browsers, this page displays normally.



Re: How to insert whitespace when parsing html

Posted by Anne Blankert <an...@geodan.nl>.
Some new information:
The problem seems to be in the use of <br/>. I created a new class that 
extends the Contenthandler and found that callback methods 'characters', 
'startElement' and 'endElement' are NOT called for the <td> elements 
that contain <br/> elements.

For HTML the <br/> really means something, so I think this should 
somehow be converted to whitespace or something else. I think it should 
not be ignored. Am I overlooking something?



On 2009-10-16 19:20, Anne Blankert wrote:
> Hello List,
>
> According to the changelog, Tika now inserts whitespace when parsing 
> html documents, but I do not understand how to get it to work.
>
> My HTML document has a fragment:
> <td valign="top">09.0025849<br/><br/></td><td valign="top">Priemstraat 
> 4<br/><br/></td><td valign="top">07-10-2009<br/><br/></td></tr><tr><td 
> valign="top">1<br/><br/></td>
>
> And the below code snippet produces output:
> 09.0025849Priemstraat 407-10-20091
>
> I was hoping to get output like:
> 09.0025849 Priemstraat 4 07-10-2009 1
>
> Any tips on handling this?
>
> Code (using tika-core-0.4.jar and tika-parsers-0.4.jar):
>
>  URL htmlPage = new URL(htmlPage);
>  DataInputStream daHTMLfromDaItem = new 
> DataInputStream(htmlPage.openStream());
>  ContentHandler bodyContentHandler = new BodyContentHandler();        
>  Metadata metadata = new Metadata();
>  AutoDetectParser p = new AutoDetectParser();
>  try {
>           // get HTML and convert to text in bodyContentHandler
>           p.parse(daHTMLfromDaItem, bodyContentHandler, metadata);
>           System.out.println (bodyContentHandler.toString());
>
>
>
> Thanks,
>
> Anneb
>