You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2011/09/25 19:18:26 UTC
[jira] [Commented] (TIKA-730) WriteOutContentHandler concatenates
title tag and body text.
[ https://issues.apache.org/jira/browse/TIKA-730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114295#comment-13114295 ]
Uwe Schindler commented on TIKA-730:
------------------------------------
The content handler should be used after BodyContentHandler. The XHTMLRequestHandler used as input before only inserts ignorableWhitespace between body tags, not in head. The <head> section is not intended to pass through WriteOutContentHandler, therefore use a BodyContentHandler as input to WriteOutContentHandler.
> WriteOutContentHandler concatenates title tag and body text.
> ------------------------------------------------------------
>
> Key: TIKA-730
> URL: https://issues.apache.org/jira/browse/TIKA-730
> Project: Tika
> Issue Type: Bug
> Components: general, parser
> Affects Versions: 0.9
> Reporter: Raimund Merkert
>
> I just noticed that the WriteOutContentHandler concatenates strings that it should not concatenate. I noticed this in case of a title tag which was combined with the first text in a body, e.g.: <head><title>a</title><head><body>b</body>
> results in "ab" and not "a b" (or something else with a break). Interestingly, "<p>a</p><p>b</p>" does get broken into separate words.
> I'm not aware of a better way to extract text only with an out-of-the-box tika.
> I've added a small unit test here:
> {code}
> package tika;
> import java.io.ByteArrayInputStream;
> import java.io.InputStream;
> import java.io.StringWriter;
> import java.nio.charset.Charset;
> import junit.framework.Assert;
> import org.apache.tika.parser.AutoDetectParser;
> import org.apache.tika.parser.ParseContext;
> import org.apache.tika.sax.WriteOutContentHandler;
> import org.junit.Test;
> public class WriteOutContentHandler_JUnit {
> private static final String HTML = "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">"
> + "<html xmlns=\"http://www.w3.org/1999/xhtml\"><head><title>title</title></head> <body>a</body></html>";
> public static String processStream(String str) throws Exception {
> InputStream in = new ByteArrayInputStream(str.getBytes(Charset
> .forName("UTF-8")));
> AutoDetectParser parser = new AutoDetectParser();
> ParseContext context = new ParseContext();
> org.apache.tika.metadata.Metadata m = new org.apache.tika.metadata.Metadata();
> StringWriter out = new StringWriter();
> WriteOutContentHandler ctHandler = new WriteOutContentHandler(out);
> try {
> parser.parse(in, ctHandler, m, context);
> return out.toString();
> } finally {
> out.flush();
> }
> }
> @Test
> public void testParse() throws Exception {
> String data = processStream(HTML);
> data = data.trim();
> System.err.println("Extracted:\n" + data);
> Assert.assertFalse(data.equals("titlea"));
> }
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira