You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Paul Bear <xp...@gmail.com> on 2015/03/04 10:54:56 UTC
GETs and Responses in different threads ?
I have a large list of URLs (about 1 millon) and I want to run
- one thread that runs through the list and asynchronously sends GET
requests
- several worker threads that process the responses
Is it possible to separate sending GETs and processing responses in
different threads using Apache Client ?
Any ideas are welcome!
Re: GETs and Responses in different threads ?
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
> On 4 Mar 2015, at 10:54, Paul Bear <xp...@gmail.com> wrote:
>
> I have a large list of URLs (about 1 millon) and I want to run
>
> - one thread that runs through the list and asynchronously sends GET
> requests
> - several worker threads that process the responses
>
> Is it possible to separate sending GETs and processing responses in
> different threads using Apache Client ?
>
> Any ideas are welcome!
Well, just use BUbiNG, our crawler. :) It will take care of politeness, etc., and download in parallel from any number of hosts. You just have to implement the HTMLParser interface to do the processing you need.
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: GETs and Responses in different threads ?
Posted by Graeme Wallace <gr...@farecompare.com>.
Hi Paul,
We implement the FutureCallback<HttpResponse> that gets passed into
HttpClient.execute().
In there we create the runnable and toss it on our executor service. And
inside the run() method of the runnable we read the content of the
HttpResponse.
class HttpCallback implements FutureCallback<HttpResponse> {
ContentReader _contentReader;
public HttpCallback(ContentReader contentReader) {
_contentReader = contentReader;
}
@Override
public void completed (HttpResponse result) {
_contentReader.setHttpResponse(result);
getContext().getReaderService().submit(_contentReader);
}
@Override
public void failed (Exception ex) {
ErrorType type = ErrorType.UNKNOWN_ERROR;
if (ex instanceof SocketTimeoutException) {
_log.warn(_contentReader.getEfsRequest().getFareSource().getFareSourceName()
+ ": No response ");
type = ErrorType.TIMEOUT_ABORT;
}
else {
_log.error("Error in asynchronous vendor request: " +
_contentReader.getEfsRequest().getFareSource().getFareSourceName(), ex);
}
_contentReader.getCallback().failure(type);
}
@Override
public void cancelled () {
_contentReader.getCallback().failure(ErrorType.CANCELLED);
}
}
On Fri, Mar 6, 2015 at 11:05 AM, Paul Bear <xp...@gmail.com> wrote:
> >
> > > What we do, is have the http callback handle the completed(), failed(),
> > > cancelled() but immediately wrap the HttpResponse within a Runnable and
> > > throw it over to an executor service to be handled. That way, the 2
> > threads
> > > that async http uses to handle incoming responses are doing the minimum
> > > amount of work.
> >
>
> Graeme, thanks for the details. Do you know in which class the Runnable is
> thrown over to the executor service ? I was looking into most classes of
> the HttpAsyncClient 4.1-beta1 package, but couldn't make out the class.
>
> Paul
>
>
>
> >
> > Our config looks like
> >
> > int timeout =
> > configuration.getAsyncConnectionTimeout();
> > int reactorThreadPoolSize = 16;
> > int maxConnectionsPerVendor = 10;
> > int maxConnections = 1024;
> > int numReaderThreads = 2;
> >
> > if (configuration.getAsyncReactorThreadPoolSize() !=
> null)
> > {
> > reactorThreadPoolSize =
> > configuration.getAsyncReactorThreadPoolSize();
> > }
> >
> > if (configuration.getAsyncMaxConnectionsPerVendor() !=
> > null) {
> > maxConnectionsPerVendor =
> > configuration.getAsyncMaxConnectionsPerVendor();
> > }
> >
> > if
> (configuration.getAsyncMaxConnectionsAcrossAllVendors()
> > != null) {
> > maxConnections =
> > configuration.getAsyncMaxConnectionsAcrossAllVendors();
> > }
> >
> > if (configuration.getAsyncReaderThreadPoolSize() !=
> null) {
> > numReaderThreads =
> > configuration.getAsyncReaderThreadPoolSize();
> > }
> >
> >
> > RequestConfig requestConfig;
> >
> > if (context.isUseProxy()) {
> > requestConfig = RequestConfig.custom()
> > .setSocketTimeout(timeout)
> > .setProxy(new
> HttpHost(context.getProxyHost(),
> > context.getProxyPort()))
> > .setConnectTimeout(timeout)
> > .build();
> >
> > }
> > else {
> > requestConfig = RequestConfig.custom()
> > .setSocketTimeout(timeout)
> > .setConnectTimeout(timeout)
> > .build();
> > }
> >
> > ConnectionConfig connectionConfig =
> > ConnectionConfig.custom()
> >
> .setMalformedInputAction(CodingErrorAction.IGNORE)
> >
> .setUnmappableInputAction(CodingErrorAction.IGNORE)
> > .setCharset(Consts.UTF_8)
> > .build();
> >
> > IOReactorConfig reactorConfig = IOReactorConfig.custom()
> > .setIoThreadCount(reactorThreadPoolSize)
> > .build();
> >
> > CloseableHttpAsyncClient httpClient =
> > HttpAsyncClients.custom()
> > .setDefaultRequestConfig(requestConfig)
> > .setDefaultIOReactorConfig(reactorConfig)
> > .setDefaultConnectionConfig(connectionConfig)
> > .setMaxConnPerRoute(maxConnectionsPerVendor)
> > .setMaxConnTotal(maxConnections)
> > .build();
> >
> > context.setHttpClient(httpClient);
> >
> >
> > context.setReaderService(Executors.newFixedThreadPool(numReaderThreads));
> >
> >
> >
> >
> > On Wed, Mar 4, 2015 at 10:55 AM, Paul Bear <xp...@gmail.com> wrote:
> >
> > > Graeme, thanks for the example. I wonder what the number of threads is
> > that
> > > process the responses ? Is it the number of Processors from Runtime
> > class,
> > > is it configurable ?
> > >
> > > 2015-03-04 11:04 GMT+01:00 Graeme Wallace <
> > graeme.wallace@farecompare.com
> > > >:
> > >
> > > > Async Http package does what you are looking for.
> > > >
> > > > https://hc.apache.org/httpcomponents-asyncclient-dev/examples.html
> > > >
> > > > 3rd example down is where you want to start.
> > > >
> > > >
> > > > Graeme
> > > >
> > > > On Wed, Mar 4, 2015 at 9:54 AM, Paul Bear <xp...@gmail.com> wrote:
> > > >
> > > > > I have a large list of URLs (about 1 millon) and I want to run
> > > > >
> > > > > - one thread that runs through the list and asynchronously sends
> GET
> > > > > requests
> > > > > - several worker threads that process the responses
> > > > >
> > > > > Is it possible to separate sending GETs and processing responses in
> > > > > different threads using Apache Client ?
> > > > >
> > > > > Any ideas are welcome!
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Graeme Wallace
> > > > CTO
> > > > FareCompare.com
> > > > O: 972 588 1414
> > > > M: 214 681 9018
> > > >
> > >
> >
> >
> >
> > --
> > Graeme Wallace
> > CTO
> > FareCompare.com
> > O: 972 588 1414
> > M: 214 681 9018
> >
>
--
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018
Re: GETs and Responses in different threads ?
Posted by Paul Bear <xp...@gmail.com>.
>
> > What we do, is have the http callback handle the completed(), failed(),
> > cancelled() but immediately wrap the HttpResponse within a Runnable and
> > throw it over to an executor service to be handled. That way, the 2
> threads
> > that async http uses to handle incoming responses are doing the minimum
> > amount of work.
>
Graeme, thanks for the details. Do you know in which class the Runnable is
thrown over to the executor service ? I was looking into most classes of
the HttpAsyncClient 4.1-beta1 package, but couldn't make out the class.
Paul
>
> Our config looks like
>
> int timeout =
> configuration.getAsyncConnectionTimeout();
> int reactorThreadPoolSize = 16;
> int maxConnectionsPerVendor = 10;
> int maxConnections = 1024;
> int numReaderThreads = 2;
>
> if (configuration.getAsyncReactorThreadPoolSize() != null)
> {
> reactorThreadPoolSize =
> configuration.getAsyncReactorThreadPoolSize();
> }
>
> if (configuration.getAsyncMaxConnectionsPerVendor() !=
> null) {
> maxConnectionsPerVendor =
> configuration.getAsyncMaxConnectionsPerVendor();
> }
>
> if (configuration.getAsyncMaxConnectionsAcrossAllVendors()
> != null) {
> maxConnections =
> configuration.getAsyncMaxConnectionsAcrossAllVendors();
> }
>
> if (configuration.getAsyncReaderThreadPoolSize() != null) {
> numReaderThreads =
> configuration.getAsyncReaderThreadPoolSize();
> }
>
>
> RequestConfig requestConfig;
>
> if (context.isUseProxy()) {
> requestConfig = RequestConfig.custom()
> .setSocketTimeout(timeout)
> .setProxy(new HttpHost(context.getProxyHost(),
> context.getProxyPort()))
> .setConnectTimeout(timeout)
> .build();
>
> }
> else {
> requestConfig = RequestConfig.custom()
> .setSocketTimeout(timeout)
> .setConnectTimeout(timeout)
> .build();
> }
>
> ConnectionConfig connectionConfig =
> ConnectionConfig.custom()
> .setMalformedInputAction(CodingErrorAction.IGNORE)
> .setUnmappableInputAction(CodingErrorAction.IGNORE)
> .setCharset(Consts.UTF_8)
> .build();
>
> IOReactorConfig reactorConfig = IOReactorConfig.custom()
> .setIoThreadCount(reactorThreadPoolSize)
> .build();
>
> CloseableHttpAsyncClient httpClient =
> HttpAsyncClients.custom()
> .setDefaultRequestConfig(requestConfig)
> .setDefaultIOReactorConfig(reactorConfig)
> .setDefaultConnectionConfig(connectionConfig)
> .setMaxConnPerRoute(maxConnectionsPerVendor)
> .setMaxConnTotal(maxConnections)
> .build();
>
> context.setHttpClient(httpClient);
>
>
> context.setReaderService(Executors.newFixedThreadPool(numReaderThreads));
>
>
>
>
> On Wed, Mar 4, 2015 at 10:55 AM, Paul Bear <xp...@gmail.com> wrote:
>
> > Graeme, thanks for the example. I wonder what the number of threads is
> that
> > process the responses ? Is it the number of Processors from Runtime
> class,
> > is it configurable ?
> >
> > 2015-03-04 11:04 GMT+01:00 Graeme Wallace <
> graeme.wallace@farecompare.com
> > >:
> >
> > > Async Http package does what you are looking for.
> > >
> > > https://hc.apache.org/httpcomponents-asyncclient-dev/examples.html
> > >
> > > 3rd example down is where you want to start.
> > >
> > >
> > > Graeme
> > >
> > > On Wed, Mar 4, 2015 at 9:54 AM, Paul Bear <xp...@gmail.com> wrote:
> > >
> > > > I have a large list of URLs (about 1 millon) and I want to run
> > > >
> > > > - one thread that runs through the list and asynchronously sends GET
> > > > requests
> > > > - several worker threads that process the responses
> > > >
> > > > Is it possible to separate sending GETs and processing responses in
> > > > different threads using Apache Client ?
> > > >
> > > > Any ideas are welcome!
> > > >
> > >
> > >
> > >
> > > --
> > > Graeme Wallace
> > > CTO
> > > FareCompare.com
> > > O: 972 588 1414
> > > M: 214 681 9018
> > >
> >
>
>
>
> --
> Graeme Wallace
> CTO
> FareCompare.com
> O: 972 588 1414
> M: 214 681 9018
>
Re: GETs and Responses in different threads ?
Posted by Graeme Wallace <gr...@farecompare.com>.
What we do, is have the http callback handle the completed(), failed(),
cancelled() but immediately wrap the HttpResponse within a Runnable and
throw it over to an executor service to be handled. That way, the 2 threads
that async http uses to handle incoming responses are doing the minimum
amount of work.
Our config looks like
int timeout =
configuration.getAsyncConnectionTimeout();
int reactorThreadPoolSize = 16;
int maxConnectionsPerVendor = 10;
int maxConnections = 1024;
int numReaderThreads = 2;
if (configuration.getAsyncReactorThreadPoolSize() != null) {
reactorThreadPoolSize =
configuration.getAsyncReactorThreadPoolSize();
}
if (configuration.getAsyncMaxConnectionsPerVendor() !=
null) {
maxConnectionsPerVendor =
configuration.getAsyncMaxConnectionsPerVendor();
}
if (configuration.getAsyncMaxConnectionsAcrossAllVendors()
!= null) {
maxConnections =
configuration.getAsyncMaxConnectionsAcrossAllVendors();
}
if (configuration.getAsyncReaderThreadPoolSize() != null) {
numReaderThreads =
configuration.getAsyncReaderThreadPoolSize();
}
RequestConfig requestConfig;
if (context.isUseProxy()) {
requestConfig = RequestConfig.custom()
.setSocketTimeout(timeout)
.setProxy(new HttpHost(context.getProxyHost(),
context.getProxyPort()))
.setConnectTimeout(timeout)
.build();
}
else {
requestConfig = RequestConfig.custom()
.setSocketTimeout(timeout)
.setConnectTimeout(timeout)
.build();
}
ConnectionConfig connectionConfig =
ConnectionConfig.custom()
.setMalformedInputAction(CodingErrorAction.IGNORE)
.setUnmappableInputAction(CodingErrorAction.IGNORE)
.setCharset(Consts.UTF_8)
.build();
IOReactorConfig reactorConfig = IOReactorConfig.custom()
.setIoThreadCount(reactorThreadPoolSize)
.build();
CloseableHttpAsyncClient httpClient =
HttpAsyncClients.custom()
.setDefaultRequestConfig(requestConfig)
.setDefaultIOReactorConfig(reactorConfig)
.setDefaultConnectionConfig(connectionConfig)
.setMaxConnPerRoute(maxConnectionsPerVendor)
.setMaxConnTotal(maxConnections)
.build();
context.setHttpClient(httpClient);
context.setReaderService(Executors.newFixedThreadPool(numReaderThreads));
On Wed, Mar 4, 2015 at 10:55 AM, Paul Bear <xp...@gmail.com> wrote:
> Graeme, thanks for the example. I wonder what the number of threads is that
> process the responses ? Is it the number of Processors from Runtime class,
> is it configurable ?
>
> 2015-03-04 11:04 GMT+01:00 Graeme Wallace <graeme.wallace@farecompare.com
> >:
>
> > Async Http package does what you are looking for.
> >
> > https://hc.apache.org/httpcomponents-asyncclient-dev/examples.html
> >
> > 3rd example down is where you want to start.
> >
> >
> > Graeme
> >
> > On Wed, Mar 4, 2015 at 9:54 AM, Paul Bear <xp...@gmail.com> wrote:
> >
> > > I have a large list of URLs (about 1 millon) and I want to run
> > >
> > > - one thread that runs through the list and asynchronously sends GET
> > > requests
> > > - several worker threads that process the responses
> > >
> > > Is it possible to separate sending GETs and processing responses in
> > > different threads using Apache Client ?
> > >
> > > Any ideas are welcome!
> > >
> >
> >
> >
> > --
> > Graeme Wallace
> > CTO
> > FareCompare.com
> > O: 972 588 1414
> > M: 214 681 9018
> >
>
--
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018
Re: GETs and Responses in different threads ?
Posted by Paul Bear <xp...@gmail.com>.
Graeme, thanks for the example. I wonder what the number of threads is that
process the responses ? Is it the number of Processors from Runtime class,
is it configurable ?
2015-03-04 11:04 GMT+01:00 Graeme Wallace <gr...@farecompare.com>:
> Async Http package does what you are looking for.
>
> https://hc.apache.org/httpcomponents-asyncclient-dev/examples.html
>
> 3rd example down is where you want to start.
>
>
> Graeme
>
> On Wed, Mar 4, 2015 at 9:54 AM, Paul Bear <xp...@gmail.com> wrote:
>
> > I have a large list of URLs (about 1 millon) and I want to run
> >
> > - one thread that runs through the list and asynchronously sends GET
> > requests
> > - several worker threads that process the responses
> >
> > Is it possible to separate sending GETs and processing responses in
> > different threads using Apache Client ?
> >
> > Any ideas are welcome!
> >
>
>
>
> --
> Graeme Wallace
> CTO
> FareCompare.com
> O: 972 588 1414
> M: 214 681 9018
>
Re: GETs and Responses in different threads ?
Posted by Graeme Wallace <gr...@farecompare.com>.
Async Http package does what you are looking for.
https://hc.apache.org/httpcomponents-asyncclient-dev/examples.html
3rd example down is where you want to start.
Graeme
On Wed, Mar 4, 2015 at 9:54 AM, Paul Bear <xp...@gmail.com> wrote:
> I have a large list of URLs (about 1 millon) and I want to run
>
> - one thread that runs through the list and asynchronously sends GET
> requests
> - several worker threads that process the responses
>
> Is it possible to separate sending GETs and processing responses in
> different threads using Apache Client ?
>
> Any ideas are welcome!
>
--
Graeme Wallace
CTO
FareCompare.com
O: 972 588 1414
M: 214 681 9018