You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Robert Svoboda <sv...@i2s.cz> on 2008/01/15 14:48:46 UTC
httpclient bug in HttpMethodBase.responseBodyConsumed?
Hi,
I'm running HC 3.1 in threaded environment with
MultiThreadedHttpConnectionManager.
I keep getting this exception:
java.lang.IllegalStateException: Connection is not open
at org.apache.commons.httpclient.HttpConnection.assertOpen(HttpConnection.java:1303)
at org.apache.commons.httpclient.HttpConnection.isResponseAvailable(HttpConnection.java:906)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.isResponseAvailable(MultiThreadedHttpConnectionManager.java:1329)
at org.apache.commons.httpclient.HttpMethodBase.responseBodyConsumed(HttpMethodBase.java:2284)
at org.apache.commons.httpclient.HttpMethodBase$1.responseConsumed(HttpMethodBase.java:1767)
at org.apache.commons.httpclient.AutoCloseInputStream.notifyWatcher(AutoCloseInputStream.java:198)
at org.apache.commons.httpclient.AutoCloseInputStream.close(AutoCloseInputStream.java:158)
at org.apache.commons.httpclient.HttpMethodBase.releaseConnection(HttpMethodBase.java:1085)
at com.newstin.dc.util.HttpClientUtil.getBody(HttpClientUtil.java:110)
at com.newstin.dc.util.HttpClientUtil.getBodySilent(HttpClientUtil.java:149)
at com.newstin.dc.task.crawling.feedcrawler.FeedDocumentProcessor.readContent(FeedDocumentProcessor.java:128)
at com.newstin.dc.task.crawling.feedcrawler.FeedDocumentProcessor.processDocument(FeedDocumentProcessor.java:102)
at com.newstin.dc.task.crawling.feedcrawler.FeedProcessor.processDocument(FeedProcessor.java:192)
at com.newstin.dc.task.crawling.feedcrawler.FeedProcessor.call(FeedProcessor.java:137)
at com.newstin.dc.task.crawling.feedcrawler.FeedProcessor.call(FeedProcessor.java:54)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
There is a call to ensureConnectionRelease() in responseBodyConsumed() but if
this Exception happens, ensureConnectionRelease() is never called thereby
connection is never released.
This happens on only one specific site from the pool of many thousands.
What do you think?
Regards,
Robert
--
Robert Svoboda
Developer
NewsTin a.s.
Lomnickeho 9
140 00 Praha 4
+420-739 665 153
http://www.newstin.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: httpclient bug in HttpMethodBase.responseBodyConsumed?
Posted by Robert Svoboda <sv...@i2s.cz>.
* Oleg Kalnichevski <ol...@apache.org> [2008-01-16 12:41]:
> I am unable to reproduce the problem locally. The code snippet below
> seems to work just fine.
I tried to reproduce problem in self-contained code (threads,
multiple urls, ...) but I don't see it happening either.
Confused.
> I still do not quite understand why the connection object is
> in the CLOSED state at the invocation of the
> #releaseConnection() method. It should not be.
I removed that azcentral.com site from my collection and
everything is back to normal since then...
> Anyways, I checked in the patch causing the
> HttpConnection#isResponseAvailable(int) to return false if the
> connection is closed instead of throwing java.lang.IllegalStateException
thanks a lot Oleg
Robert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: httpclient bug in HttpMethodBase.responseBodyConsumed?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2008-01-16 at 10:55 +0100, Robert Svoboda wrote:
> * Oleg Kalnichevski <ol...@apache.org> [2008-01-15 22:25]:
> >
> > On Tue, 2008-01-15 at 14:48 +0100, Robert Svoboda wrote:
>
> [...]
>
> > Robert, could you please try to reproduce the problem with a small
> > (preferably self-contained) test case and run it with the context/wire
> > logging on?
>
> Log file here: http://onyon.net/tmp/r080/log.tar.bz2 [40+kb]
>
> I didn't write self-contained test but if this log is not
> enough I'll do it.
>
> Robert
>
Robert,
I am unable to reproduce the problem locally. The code snippet below
seems to work just fine. I still do not quite understand why the
connection object is in the CLOSED state at the invocation of the
#releaseConnection() method. It should not be.
Anyways, I checked in the patch causing the
HttpConnection#isResponseAvailable(int) to return false if the
connection is closed instead of throwing java.lang.IllegalStateException
Hope this helps
Oleg
==============
HttpClient client = new HttpClient();
client.getParams().setParameter(HttpClientParams.USER_AGENT,
"Mozilla/5.0 (X11; compatible; newstin.com; HTTPClient 3.1)");
client.getParams().setBooleanParameter(
HttpClientParams.ALLOW_CIRCULAR_REDIRECTS, true);
GetMethod httpget =
new GetMethod("http://www.azcentral.com/rsslinks/458818");
try {
client.executeMethod(httpget);
String s = httpget.getResponseBodyAsString();
System.out.println("content len: " + s.length());
} finally {
httpget.releaseConnection();
}
==============
[DEBUG] header - >> "GET /rsslinks/458818 HTTP/1.1[\r][\n]"
[DEBUG] header - >> "User-Agent: Mozilla/5.0 (X11; compatible;
newstin.com; HTTPClient 3.1)[\r][\n]"
[DEBUG] header - >> "Host: www.azcentral.com[\r][\n]"
[DEBUG] header - >> "[\r][\n]"
[DEBUG] header - << "HTTP/1.1 302 Found[\r][\n]"
[DEBUG] header - << "HTTP/1.1 302 Found[\r][\n]"
[DEBUG] header - << "Date: Wed, 16 Jan 2008 11:27:01 GMT[\r][\n]"
[DEBUG] header - << "Server: Apache[\r][\n]"
[DEBUG] header - << "X-Powered-By: PHP/5.2.3[\r][\n]"
[DEBUG] header - << "Location:
http://www.azcentral.com/sports/cheapseats/articles/0116p2main0116.html[\r][\n]"
[DEBUG] header - << "Vary: Accept-Encoding,User-Agent[\r][\n]"
[DEBUG] header - << "P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND
DEM", policyref="/w3c/p3p.xml"[\r][\n]"
[DEBUG] header - << "Content-Length: 0[\r][\n]"
[DEBUG] header - << "Content-Type: text/html;
charset=iso-8859-1[\r][\n]"
[DEBUG] header - << "[\r][\n]"
[DEBUG] header - >> "GET /sports/cheapseats/articles/0116p2main0116.html
HTTP/1.1[\r][\n]"
[DEBUG] header - >> "User-Agent: Mozilla/5.0 (X11; compatible;
newstin.com; HTTPClient 3.1)[\r][\n]"
[DEBUG] header - >> "Host: www.azcentral.com[\r][\n]"
[DEBUG] header - >> "[\r][\n]"
[DEBUG] header - << "HTTP/1.1 302 Found[\r][\n]"
[DEBUG] header - << "HTTP/1.1 302 Found[\r][\n]"
[DEBUG] header - << "Date: Wed, 16 Jan 2008 11:27:01 GMT[\r][\n]"
[DEBUG] header - << "Server: Apache[\r][\n]"
[DEBUG] header - << "Vary: Accept-Encoding,User-Agent[\r][\n]"
[DEBUG] header - << "X-Powered-By: PHP/5.2.3[\r][\n]"
[DEBUG] header - <<
"Location: /sports/cheapseats/articles/0116p2main0116.html?&wired[\r][\n]"
[DEBUG] header - << "P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND
DEM", policyref="/w3c/p3p.xml"[\r][\n]"
[DEBUG] header - << "Content-Length: 0[\r][\n]"
[DEBUG] header - << "Content-Type: text/html;
charset=iso-8859-1[\r][\n]"
[DEBUG] header - << "[\r][\n]"
[DEBUG] header - >>
"GET /sports/cheapseats/articles/0116p2main0116.html?&wired
HTTP/1.1[\r][\n]"
[DEBUG] header - >> "User-Agent: Mozilla/5.0 (X11; compatible;
newstin.com; HTTPClient 3.1)[\r][\n]"
[DEBUG] header - >> "Host: www.azcentral.com[\r][\n]"
[DEBUG] header - >> "[\r][\n]"
[DEBUG] header - << "HTTP/1.1 200 OK[\r][\n]"
[DEBUG] header - << "HTTP/1.1 200 OK[\r][\n]"
[DEBUG] header - << "Date: Wed, 16 Jan 2008 11:27:02 GMT[\r][\n]"
[DEBUG] header - << "Server: Apache[\r][\n]"
[DEBUG] header - << "Accept-Ranges: bytes[\r][\n]"
[DEBUG] header - << "Vary: Accept-Encoding,User-Agent[\r][\n]"
[DEBUG] header - << "P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTR STP IND
DEM", policyref="/w3c/p3p.xml"[\r][\n]"
[DEBUG] header - << "Transfer-Encoding: chunked[\r][\n]"
[DEBUG] header - << "Content-Type: text/html;
charset=iso-8859-1[\r][\n]"
[DEBUG] header - << "[\r][\n]"
[WARN] HttpMethodBase - Going to buffer response body of large or
unknown size. Using getResponseBodyAsStream instead is recommended.
[DEBUG] header - << "[\r][\n]"
content len: 53588
==============
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: httpclient bug in HttpMethodBase.responseBodyConsumed?
Posted by Robert Svoboda <sv...@i2s.cz>.
* Oleg Kalnichevski <ol...@apache.org> [2008-01-15 22:25]:
>
> On Tue, 2008-01-15 at 14:48 +0100, Robert Svoboda wrote:
[...]
> Robert, could you please try to reproduce the problem with a small
> (preferably self-contained) test case and run it with the context/wire
> logging on?
Log file here: http://onyon.net/tmp/r080/log.tar.bz2 [40+kb]
I didn't write self-contained test but if this log is not
enough I'll do it.
Robert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: httpclient bug in HttpMethodBase.responseBodyConsumed?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2008-01-15 at 14:48 +0100, Robert Svoboda wrote:
> Hi,
>
> I'm running HC 3.1 in threaded environment with
> MultiThreadedHttpConnectionManager.
>
> I keep getting this exception:
>
> java.lang.IllegalStateException: Connection is not open
> at org.apache.commons.httpclient.HttpConnection.assertOpen(HttpConnection.java:1303)
> at org.apache.commons.httpclient.HttpConnection.isResponseAvailable(HttpConnection.java:906)
> at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.isResponseAvailable(MultiThreadedHttpConnectionManager.java:1329)
> at org.apache.commons.httpclient.HttpMethodBase.responseBodyConsumed(HttpMethodBase.java:2284)
> at org.apache.commons.httpclient.HttpMethodBase$1.responseConsumed(HttpMethodBase.java:1767)
> at org.apache.commons.httpclient.AutoCloseInputStream.notifyWatcher(AutoCloseInputStream.java:198)
> at org.apache.commons.httpclient.AutoCloseInputStream.close(AutoCloseInputStream.java:158)
> at org.apache.commons.httpclient.HttpMethodBase.releaseConnection(HttpMethodBase.java:1085)
> at com.newstin.dc.util.HttpClientUtil.getBody(HttpClientUtil.java:110)
> at com.newstin.dc.util.HttpClientUtil.getBodySilent(HttpClientUtil.java:149)
> at com.newstin.dc.task.crawling.feedcrawler.FeedDocumentProcessor.readContent(FeedDocumentProcessor.java:128)
> at com.newstin.dc.task.crawling.feedcrawler.FeedDocumentProcessor.processDocument(FeedDocumentProcessor.java:102)
> at com.newstin.dc.task.crawling.feedcrawler.FeedProcessor.processDocument(FeedProcessor.java:192)
> at com.newstin.dc.task.crawling.feedcrawler.FeedProcessor.call(FeedProcessor.java:137)
> at com.newstin.dc.task.crawling.feedcrawler.FeedProcessor.call(FeedProcessor.java:54)
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> There is a call to ensureConnectionRelease() in responseBodyConsumed() but if
> this Exception happens, ensureConnectionRelease() is never called thereby
> connection is never released.
>
> This happens on only one specific site from the pool of many thousands.
>
> What do you think?
>
Here's the culprit.
http://hc.apache.org/httpclient-3.x/xref/org/apache/commons/httpclient/HttpConnection.html#895
However, it is still not clear to me how come the connection has been
closed on the _client_ side before #releaseConnection. This looks fishy
to me.
Robert, could you please try to reproduce the problem with a small
(preferably self-contained) test case and run it with the context/wire
logging on?
Oleg
> Regards,
> Robert
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org