You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "Hudson (JIRA)" <ji...@apache.org> on 2015/12/08 23:48:11 UTC

[jira] [Commented] (NUTCH-2042) parse-html increase chunk size used to detect charset

    [ https://issues.apache.org/jira/browse/NUTCH-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047624#comment-15047624 ] 

Hudson commented on NUTCH-2042:
-------------------------------

SUCCESS: Integrated in Nutch-nutchgora #1542 (See [https://builds.apache.org/job/Nutch-nutchgora/1542/])
NUTCH-2042 parse-html increase chunk size used to detect charset (snagel: [http://svn.apache.org/viewvc/nutch/branches/2.x/?view=rev&rev=1718718])
* 2.x/CHANGES.txt
* 2.x/src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HtmlParser.java


> parse-html increase chunk size used to detect charset
> -----------------------------------------------------
>
>                 Key: NUTCH-2042
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2042
>             Project: Nutch
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 2.3, 1.10
>            Reporter: Sebastian Nagel
>            Assignee: Sebastian Nagel
>            Priority: Minor
>             Fix For: 2.3.1, 1.12
>
>         Attachments: NUTCH-2042-2x-v1.patch, NUTCH-2042-trunk-v1.patch, NUTCH-2042-trunk-v2.patch
>
>
> The chunk used to detect the encoding of a document is set to 2000 bytes. Although it is definitely best practice to "define" the character set on top, 2000 bytes are sometimes not enough: 20 longer <link> elements pointing to javascript and css libs may "hide" the <meta> element containing content type and encoding. Same problem has been observed in TIKA-357 and solved by increasing the buffer size to 8 kB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)