[jira] [Commented] (NUTCH-1643) Unnecessary fetching with http.content.limit when using protocol-http

["Talat UYARER (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988706#comment-13988706 ] 

Talat UYARER commented on NUTCH-1643:
-------------------------------------

Hi [~lewismc],

Sorry for late reply. I can generate this patch without prefix. But I wonder when i developed this patch. I didn't know truncate option in parse. What do you think we should control parse.skip.truncate option ? If skip trunacate is ture you can set null for unnecessary fetching. Wdyt ? 

{code:title=src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java|borderStyle=solid}
    ByteArrayOutputStream out = new ByteArrayOutputStream(Http.BUFFER_SIZE);
    if (http.getMaxContent() >= 0
      && contentLength > http.getMaxContent()){   // limit download size
      // If file size bigger then limit size, it writes null content. 
      //contentLength  = http.getMaxContent();
      content = out.toByteArray();
      return;
    }
{code}

> Unnecessary fetching with http.content.limit when using protocol-http
> ---------------------------------------------------------------------
>
>                 Key: NUTCH-1643
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1643
>             Project: Nutch
>          Issue Type: Bug
>          Components: protocol
>    Affects Versions: 2.1, 2.2, 2.2.1
>            Reporter: Talat UYARER
>            Priority: Minor
>             Fix For: 2.4
>
>         Attachments: NUTCH-1643.patch, NUTCH-1643v2.patch, NUTCH-1643v3.patch
>
>
> In protocol-http, Even If I have http.content.limit value set, protocol-http fetches files of all sizes (larger files are fetched until limit allows). 
> But when Parsing, parser skips incomplete files (if parser.skip.truncated configuration is true). It seems like an unnecessary effort to partially fetch contents larger than limit if they are not gonna be parsed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)