You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Joseph Naegele <jn...@grierforensics.com> on 2016/12/06 16:21:38 UTC
HttpAsyncClient bounded download size
Hi folks,
How can I limit the amount of data downloaded for a request executed by the HttpAsyncClient and still process the response as "completed" in the registered FutureCallback? The use case is a large scale web crawler that truncates resources deemed too large.
I started by limiting the amount of data read from the response entity's InputStream, however this doesn't work with the default BasicAsyncResponseConsumer, because it uses the dynamically expanding SimpleInputBuffer to download the entire response entity.
I implemented my own HttpAsyncResponseConsumer, similar to the BasicAsyncResponseConsumer, and tried using IOControl to signal shutdown once the I've read maximum desired number of bytes, however this triggers a ConnectionClosedException. This is undesirable because I can't distinguish it from other causes of ConnectionClosedExceptions, and I want to treat "truncated" responses as completed in the registered FutureCallback (where I post-process the response).
Is there another method of implementing my desired functionality?
Thanks,
Joe Naegele
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: HttpAsyncClient bounded download size
Posted by Joseph Naegele <jn...@grierforensics.com>.
Thanks Oleg, this works.
I also needed to pass my custom HttpAsyncResponseConsumer to the FutureCallback in order to extract the HttpResponse in the callback's "failure" method, and ensure my HttpAsyncResponseConsumer maintains the HttpResponse even after its "releaseResources" method is called.
I can probably provide an example in the future if anyone ever wants to see how I did this.
Joe
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Wednesday, December 07, 2016 4:36 AM
To: HttpClient User Discussion <ht...@hc.apache.org>
Subject: Re: HttpAsyncClient bounded download size
On Tue, 2016-12-06 at 11:21 -0500, Joseph Naegele wrote:
> Hi folks,
>
> How can I limit the amount of data downloaded for a request executed by the HttpAsyncClient and still process the response as "completed" in the registered FutureCallback? The use case is a large scale web crawler that truncates resources deemed too large.
>
> I started by limiting the amount of data read from the response entity's InputStream, however this doesn't work with the default BasicAsyncResponseConsumer, because it uses the dynamically expanding SimpleInputBuffer to download the entire response entity.
>
> I implemented my own HttpAsyncResponseConsumer, similar to the BasicAsyncResponseConsumer, and tried using IOControl to signal shutdown once the I've read maximum desired number of bytes, however this triggers a ConnectionClosedException. This is undesirable because I can't distinguish it from other causes of ConnectionClosedExceptions, and I want to treat "truncated" responses as completed in the registered FutureCallback (where I post-process the response).
>
> Is there another method of implementing my desired functionality?
>
> Thanks,
> Joe Naegele
>
Hi Joe
Make your custom HttpAsyncResponseConsumer throw a custom exception,
like IHadEnoughException or some such, and see if it gets correctly
propagated to the result callback. If it does not, please raise an issue
in JIRA.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: HttpAsyncClient bounded download size
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2016-12-06 at 11:21 -0500, Joseph Naegele wrote:
> Hi folks,
>
> How can I limit the amount of data downloaded for a request executed by the HttpAsyncClient and still process the response as "completed" in the registered FutureCallback? The use case is a large scale web crawler that truncates resources deemed too large.
>
> I started by limiting the amount of data read from the response entity's InputStream, however this doesn't work with the default BasicAsyncResponseConsumer, because it uses the dynamically expanding SimpleInputBuffer to download the entire response entity.
>
> I implemented my own HttpAsyncResponseConsumer, similar to the BasicAsyncResponseConsumer, and tried using IOControl to signal shutdown once the I've read maximum desired number of bytes, however this triggers a ConnectionClosedException. This is undesirable because I can't distinguish it from other causes of ConnectionClosedExceptions, and I want to treat "truncated" responses as completed in the registered FutureCallback (where I post-process the response).
>
> Is there another method of implementing my desired functionality?
>
> Thanks,
> Joe Naegele
>
Hi Joe
Make your custom HttpAsyncResponseConsumer throw a custom exception,
like IHadEnoughException or some such, and see if it gets correctly
propagated to the result callback. If it does not, please raise an issue
in JIRA.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: HttpAsyncClient bounded download size
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
Hello,
Can't help you with the code but just to make sure you also consider an additional option: in order to be friendly to servers you can request a partial response with a range header in the first place (however not sure about the interop impact).
Gruss
Bernd
--
http://bernd.eckenfels.net
On Tue, Dec 6, 2016 at 5:21 PM +0100, "Joseph Naegele" <jn...@grierforensics.com> wrote:
Hi folks,
How can I limit the amount of data downloaded for a request executed by the HttpAsyncClient and still process the response as "completed" in the registered FutureCallback? The use case is a large scale web crawler that truncates resources deemed too large.
I started by limiting the amount of data read from the response entity's InputStream, however this doesn't work with the default BasicAsyncResponseConsumer, because it uses the dynamically expanding SimpleInputBuffer to download the entire response entity.
I implemented my own HttpAsyncResponseConsumer, similar to the BasicAsyncResponseConsumer, and tried using IOControl to signal shutdown once the I've read maximum desired number of bytes, however this triggers a ConnectionClosedException. This is undesirable because I can't distinguish it from other causes of ConnectionClosedExceptions, and I want to treat "truncated" responses as completed in the registered FutureCallback (where I post-process the response).
Is there another method of implementing my desired functionality?
Thanks,
Joe Naegele
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org