You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Arnold Maderthaner <ar...@maderthaner.me> on 2014/10/13 11:59:07 UTC
Using HTTPClient for mulithreaded HTTP Get download
Hi !
I'm trying to write a simple application that does a multi threaded
HTTP Get download from a server which needs to download approx. 2000
files.
I have a list of URLs that I want to fetch and want the binary content
to be stored either on the filesystem or in memory.
Depending on configuration I want to use 10 threads to download this
from the server.
What is the best method to implement this ? Should I use an
ExecutorService with a fixed thread pool and create an own HTTPClient
for each thread or is there some better (less expensive) approach ?
yours
Arnold
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Using HTTPClient for mulithreaded HTTP Get download
Posted by Jaikit Savla <ja...@yahoo.com.INVALID>.
Yes. As Oleg mentioned, critical thing is to have1. Singleton instance of HTTPClient which will be used for making all requests. This will make sure you reuse the same connections ( connection pooling)
2. While configuring your Httpclient, use Pooling Httpclient connection manager.
Jaikit
On Monday, October 13, 2014 7:23 AM, Arnold Maderthaner <ar...@maderthaner.me> wrote:
Hi !
So I can do something similar like here
(http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/examples/org/apache/http/examples/client/ClientMultiThreadedExecution.java)
using ExecutorService than "normal" threads (just need to use the
PoolingHttpClientConnectionManager) ?
yours
Arnold
On Mon, Oct 13, 2014 at 3:50 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Mon, 2014-10-13 at 11:59 +0200, Arnold Maderthaner wrote:
>> Hi !
>>
>> I'm trying to write a simple application that does a multi threaded
>> HTTP Get download from a server which needs to download approx. 2000
>> files.
>> I have a list of URLs that I want to fetch and want the binary content
>> to be stored either on the filesystem or in memory.
>> Depending on configuration I want to use 10 threads to download this
>> from the server.
>>
>> What is the best method to implement this ? Should I use an
>> ExecutorService with a fixed thread pool and create an own HTTPClient
>> for each thread or is there some better (less expensive) approach ?
>>
>
> You should be using multiple worker threads (by means of ExecutorService
> or otherwise) but make worker threads share the same instance of
> HttpClient.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Using HTTPClient for mulithreaded HTTP Get download
Posted by Arnold Maderthaner <ar...@maderthaner.me>.
Hi !
So I can do something similar like here
(http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/examples/org/apache/http/examples/client/ClientMultiThreadedExecution.java)
using ExecutorService than "normal" threads (just need to use the
PoolingHttpClientConnectionManager) ?
yours
Arnold
On Mon, Oct 13, 2014 at 3:50 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Mon, 2014-10-13 at 11:59 +0200, Arnold Maderthaner wrote:
>> Hi !
>>
>> I'm trying to write a simple application that does a multi threaded
>> HTTP Get download from a server which needs to download approx. 2000
>> files.
>> I have a list of URLs that I want to fetch and want the binary content
>> to be stored either on the filesystem or in memory.
>> Depending on configuration I want to use 10 threads to download this
>> from the server.
>>
>> What is the best method to implement this ? Should I use an
>> ExecutorService with a fixed thread pool and create an own HTTPClient
>> for each thread or is there some better (less expensive) approach ?
>>
>
> You should be using multiple worker threads (by means of ExecutorService
> or otherwise) but make worker threads share the same instance of
> HttpClient.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Using HTTPClient for mulithreaded HTTP Get download
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2014-10-13 at 11:59 +0200, Arnold Maderthaner wrote:
> Hi !
>
> I'm trying to write a simple application that does a multi threaded
> HTTP Get download from a server which needs to download approx. 2000
> files.
> I have a list of URLs that I want to fetch and want the binary content
> to be stored either on the filesystem or in memory.
> Depending on configuration I want to use 10 threads to download this
> from the server.
>
> What is the best method to implement this ? Should I use an
> ExecutorService with a fixed thread pool and create an own HTTPClient
> for each thread or is there some better (less expensive) approach ?
>
You should be using multiple worker threads (by means of ExecutorService
or otherwise) but make worker threads share the same instance of
HttpClient.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org