[jira] [Updated] (NUTCH-1643) Unnecessary fetching with http.content.limit when using protocol-http

["Lewis John McGibbney (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lewis John McGibbney updated NUTCH-1643:
----------------------------------------

    Attachment: NUTCH-1643v2.patch

Hi Talat, the attachment merely adds useful Javadoc to many methods which we touch within this patch. When you get time, if you are able to reply to my other post then we can work to get this patch looking better. Thank you

> Unnecessary fetching with http.content.limit when using protocol-http
> ---------------------------------------------------------------------
>
>                 Key: NUTCH-1643
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1643
>             Project: Nutch
>          Issue Type: Bug
>          Components: protocol
>    Affects Versions: 2.1, 2.2, 2.2.1
>            Reporter: Talat UYARER
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: NUTCH-1643.patch, NUTCH-1643v2.patch
>
>
> In protocol-http, Even If I have http.content.limit value set, protocol-http fetches files of all sizes (larger files are fetched until limit allows). 
> But when Parsing, parser skips incomplete files (if parser.skip.truncated configuration is true). It seems like an unnecessary effort to partially fetch contents larger than limit if they are not gonna be parsed.



--
This message was sent by Atlassian JIRA
(v6.1#6144)