[jira] [Commented] (NUTCH-2562) protocol-http fails to read large chunked HTTP responses

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499031#comment-16499031 ] 

ASF GitHub Bot commented on NUTCH-2562:
---------------------------------------

sebastian-nagel closed pull request #329: NUTCH-2562 protocol-http fails to read large chunked HTTP responses
URL: https://github.com/apache/nutch/pull/329
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java b/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java
index 591b94298..56ae789f3 100644
--- a/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java
+++ b/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java
@@ -437,9 +437,11 @@ private void readChunkedContent(PushbackInputStream in, StringBuffer line)
         break;
       }
 
-      if (http.getMaxContent() >= 0 && (contentBytesRead + chunkLen) > http
-          .getMaxContent())
+      if (http.getMaxContent() >= 0
+          && (contentBytesRead + chunkLen) > http.getMaxContent()) {
+        // content will be trimmed when processing this chunk
         chunkLen = http.getMaxContent() - contentBytesRead;
+      }
 
       // read one chunk
       int chunkBytesRead = 0;
@@ -465,17 +467,26 @@ private void readChunkedContent(PushbackInputStream in, StringBuffer line)
       }
 
       contentBytesRead += chunkBytesRead;
+      if (http.getMaxContent() >= 0
+          && contentBytesRead >= http.getMaxContent()) {
+        Http.LOG.trace("Http: content limit reached");
+        break;
+      }
+
       readLine(in, line, false);
 
     }
 
+    content = out.toByteArray();
+
     if (!doneChunks) {
+      // content trimmed
       if (contentBytesRead != http.getMaxContent())
         throw new HttpException("chunk eof: !doneChunk && didn't max out");
       return;
     }
 
-    content = out.toByteArray();
+    // read trailing headers
     parseHeaders(in, line, null);
 
   }


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> protocol-http fails to read large chunked HTTP responses
> --------------------------------------------------------
>
>                 Key: NUTCH-2562
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2562
>             Project: Nutch
>          Issue Type: Sub-task
>    Affects Versions: 1.14
>            Reporter: Gerard Bouchar
>            Priority: Major
>             Fix For: 1.15
>
>
> While reading chunked content, if the content size becomes larger than http.getMaxContent(), instead of just stopping and truncate the content, it tries to read a new chunk before having read the previous one completely, resulting in a '{color:#333333}bad chunk length' error.{color}
>  
> {color:#333333}See: https://github.com/apache/nutch/blob/master/src/plugin/protocol-http/src/java/org/apache/nutch/protocol/http/HttpResponse.java#L440-L442{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)