You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "Tim Allison (Jira)" <ji...@apache.org> on 2023/11/06 20:08:00 UTC

[jira] [Resolved] (NUTCH-3020) ParseSegment should check for protocol's flags for truncation

     [ https://issues.apache.org/jira/browse/NUTCH-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Allison resolved NUTCH-3020.
--------------------------------
    Fix Version/s: 1.20
       Resolution: Fixed

> ParseSegment should check for protocol's flags for truncation
> -------------------------------------------------------------
>
>                 Key: NUTCH-3020
>                 URL: https://issues.apache.org/jira/browse/NUTCH-3020
>             Project: Nutch
>          Issue Type: Task
>            Reporter: Tim Allison
>            Priority: Minor
>             Fix For: 1.20
>
>
> As discussed on the user list, several protocols can identify when a fetch has been truncated. ParseSegment only checks for the number of bytes fetched vs the http length header (if it exists). We should modify ParseSegment to check for notification of truncation from the protocols.
> I noticed this specifically with okhttp, but other protocols may flag truncation as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)