You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Bill Speirs <bi...@gmail.com> on 2011/08/24 16:28:08 UTC
HttpParams appear to be static/global
I'm working on a proxy server using the 4.2-alpha1 httpcore and 4.1.2
httpclient. In the server portion of my code I setup
SyncBasicHttpParams as follows:
HttpParams params = new SyncBasicHttpParams();
params.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 0)
.setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE,
8 * 1024)
.setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK,
false)
.setBooleanParameter(CoreConnectionPNames.TCP_NODELAY, true)
.setParameter(CoreProtocolPNames.ORIGIN_SERVER, "FactSet
Lima Proxy")
.setParameter(CoreProtocolPNames.PROTOCOL_VERSION,
HttpVersion.HTTP_1_1.toString())
.setParameter(CoreProtocolPNames.HTTP_ELEMENT_CHARSET,
HTTP.UTF_8);
I then use them to create an AsyncNHttpServiceHandler and
DefaultServerIOEventDispatch.
In another portion of code I create a new SyncBasicHttpParams object
and use it to configure my client:
final HttpParams params = new SyncBasicHttpParams();
params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, handleRedirects);
params.setBooleanParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS, false);
params.setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.IGNORE_COOKIES);
HttpConnectionParams.setSoTimeout(params, maxTimeout);
HttpConnectionParams.setConnectionTimeout(params, maxTimeout);
this.client = new DefaultHttpClient(this.connManager, params);
However, for some reason the HTTP_ELEMENT_CHARSET parameter is
"bleeding" over from my server into my client, forcing my client
requests to be in UTF-8 as well.
Why is this happening? Are the settings for SyncBasicHttpParams global somehow?
Thanks...
Bill-
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HttpParams appear to be static/global
Posted by Bill Speirs <bi...@gmail.com>.
Agreed... doing all IO through one thread is prob dumb.
Like Synapse, I have a few threads listening, then I put the request through
an "assembly line" to modify the request, then I proxy the request to the
back-end server using another pool of threads, finally modifying the
response before sending it back to the requestor.
Hopefully we will open-source this at some point as it is fairly generic.
Bill-
On Aug 26, 2011 1:01 PM, "Oleg Kalnichevski" <ol...@apache.org> wrote:
> On Fri, 2011-08-26 at 11:55 -0400, Bill Speirs wrote:
>> I'm not trying to be pedantic, just trying to understand this...
>>
>> From the Synapse source code for HttpCoreNIOListener:
>> 250 Thread t = new Thread(new Runnable() {
>> 251 public void run() {
>> 252 try {
>> 253 ioReactor.execute(ioEventDispatch);
>>
>> From the source code for HttpCoreNIOSender:
>> 195 Thread t = new Thread(new Runnable() {
>> 196 public void run() {
>> 197 try {
>> 198 ioReactor.execute(ioEventDispatch);
>>
>> It appears to me that two threads are created for two different
>> IOReactor objects: one for listening to incoming requests, and another
>> for sending requests. While this is 100% acceptable (and close to what
>> I'm currently doing), is it possible to have both running in one
>> thread? How would the IOEventDispatch which would be handling
>> everything know that connected() one time would mean that someone has
>> connected to the server, and another time it meant a connection was
>> made to the back-end service. Maybe through an attribute (and
>> associated getAttribute call) in the IOSession?
>>
>> Thanks...
>>
>> Bill-
>>
>
> You could do that, albeit with a bit of custom coding. You could extend
> BaseIOReactor and add methods to open outgoing connections and listen
> for incoming ones. The question is whether or not that would make a lot
> of sense. Multi-core systems are commonplace nowadays. If all i/o events
> were handled by one thread only one core could be utilized for i/o
> operations, thus making the system less scalable.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
Re: HttpParams appear to be static/global
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2011-08-26 at 11:55 -0400, Bill Speirs wrote:
> I'm not trying to be pedantic, just trying to understand this...
>
> From the Synapse source code for HttpCoreNIOListener:
> 250 Thread t = new Thread(new Runnable() {
> 251 public void run() {
> 252 try {
> 253 ioReactor.execute(ioEventDispatch);
>
> From the source code for HttpCoreNIOSender:
> 195 Thread t = new Thread(new Runnable() {
> 196 public void run() {
> 197 try {
> 198 ioReactor.execute(ioEventDispatch);
>
> It appears to me that two threads are created for two different
> IOReactor objects: one for listening to incoming requests, and another
> for sending requests. While this is 100% acceptable (and close to what
> I'm currently doing), is it possible to have both running in one
> thread? How would the IOEventDispatch which would be handling
> everything know that connected() one time would mean that someone has
> connected to the server, and another time it meant a connection was
> made to the back-end service. Maybe through an attribute (and
> associated getAttribute call) in the IOSession?
>
> Thanks...
>
> Bill-
>
You could do that, albeit with a bit of custom coding. You could extend
BaseIOReactor and add methods to open outgoing connections and listen
for incoming ones. The question is whether or not that would make a lot
of sense. Multi-core systems are commonplace nowadays. If all i/o events
were handled by one thread only one core could be utilized for i/o
operations, thus making the system less scalable.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HttpParams appear to be static/global
Posted by Bill Speirs <bi...@gmail.com>.
I'm not trying to be pedantic, just trying to understand this...
>From the Synapse source code for HttpCoreNIOListener:
250 Thread t = new Thread(new Runnable() {
251 public void run() {
252 try {
253 ioReactor.execute(ioEventDispatch);
>From the source code for HttpCoreNIOSender:
195 Thread t = new Thread(new Runnable() {
196 public void run() {
197 try {
198 ioReactor.execute(ioEventDispatch);
It appears to me that two threads are created for two different
IOReactor objects: one for listening to incoming requests, and another
for sending requests. While this is 100% acceptable (and close to what
I'm currently doing), is it possible to have both running in one
thread? How would the IOEventDispatch which would be handling
everything know that connected() one time would mean that someone has
connected to the server, and another time it meant a connection was
made to the back-end service. Maybe through an attribute (and
associated getAttribute call) in the IOSession?
Thanks...
Bill-
> On Fri, Aug 26, 2011 at 9:29 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>> A very common pattern for proxies (used by Apache Synapse, for instance)
>> is to have a small number of i/o dispatch threads (approx. as many as
>> the CPU cores) handling i/o events for incoming and outgoing connections
>> asynchronously and a larger pool of worker threads that are employed to
>> execute long, potentially blocking operations (such as content
>> transformation). I _suspect_ that is what you want.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HttpParams appear to be static/global
Posted by Bill Speirs <bi...@gmail.com>.
This is very much the type of thing I'm looking for. What I do not
understand how to accomplish is having one (or a few) threads which
handle the i/o events for both the incoming requests and the outgoing
requests to the back-end services, and then their responses.
I'll take a look at the source for Apache Synapse though...
Thanks!
Bill-
On Fri, Aug 26, 2011 at 9:29 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> Bill,
>
> A very common pattern for proxies (used by Apache Synapse, for instance)
> is to have a small number of i/o dispatch threads (approx. as many as
> the CPU cores) handling i/o events for incoming and outgoing connections
> asynchronously and a larger pool of worker threads that are employed to
> execute long, potentially blocking operations (such as content
> transformation). I _suspect_ that is what you want.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HttpParams appear to be static/global
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2011-08-25 at 15:14 -0400, Bill Speirs wrote:
> What would be best, for my proxy, would be if I could have a thread
> (or pool of threads) waiting on read/write events from *either* the
> server socket(s) or the client socket(s). I do not believe that is
> possible though... does anyone know how to do that sort of thing?
> Using an ExecutorCompletionService was the only way I knew to
> accomplish this, but it required two pools of threads.
>
> Bill-
>
Bill,
A very common pattern for proxies (used by Apache Synapse, for instance)
is to have a small number of i/o dispatch threads (approx. as many as
the CPU cores) handling i/o events for incoming and outgoing connections
asynchronously and a larger pool of worker threads that are employed to
execute long, potentially blocking operations (such as content
transformation). I _suspect_ that is what you want.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HttpParams appear to be static/global
Posted by Bill Speirs <bi...@gmail.com>.
What would be best, for my proxy, would be if I could have a thread
(or pool of threads) waiting on read/write events from *either* the
server socket(s) or the client socket(s). I do not believe that is
possible though... does anyone know how to do that sort of thing?
Using an ExecutorCompletionService was the only way I knew to
accomplish this, but it required two pools of threads.
Bill-
On Wed, Aug 24, 2011 at 1:51 PM, Bill Speirs <bi...@gmail.com> wrote:
> That is exactly what is happening... the request that is passed to the
> NHttpRequestHandler.handle() method has the HTTP params set with the
> characters set = UTF-8. Good catch and makes sense as I basically pass
> this request on to the client to execute.
>
> I need to look into the async client and see how it would fit into my
> design. I started with sync for both client & server, and quickly
> out-grew the server side of things. I keep my
> NHttpRequestHandler.handle() method async by wrapping the request in a
> continuation and placing it on a ExecutorCompletionService.
>
> Bill-
>
> On Wed, Aug 24, 2011 at 10:39 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>> On Wed, 2011-08-24 at 10:28 -0400, Bill Speirs wrote:
>>> I'm working on a proxy server using the 4.2-alpha1 httpcore and 4.1.2
>>> httpclient. In the server portion of my code I setup
>>> SyncBasicHttpParams as follows:
>>>
>>>
>>> HttpParams params = new SyncBasicHttpParams();
>>>
>>> params.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 0)
>>> .setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE,
>>> 8 * 1024)
>>> .setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK,
>>> false)
>>> .setBooleanParameter(CoreConnectionPNames.TCP_NODELAY, true)
>>> .setParameter(CoreProtocolPNames.ORIGIN_SERVER, "FactSet
>>> Lima Proxy")
>>> .setParameter(CoreProtocolPNames.PROTOCOL_VERSION,
>>> HttpVersion.HTTP_1_1.toString())
>>> .setParameter(CoreProtocolPNames.HTTP_ELEMENT_CHARSET,
>>> HTTP.UTF_8);
>>>
>>> I then use them to create an AsyncNHttpServiceHandler and
>>> DefaultServerIOEventDispatch.
>>>
>>> In another portion of code I create a new SyncBasicHttpParams object
>>> and use it to configure my client:
>>> final HttpParams params = new SyncBasicHttpParams();
>>>
>>> params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, handleRedirects);
>>> params.setBooleanParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS, false);
>>> params.setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.IGNORE_COOKIES);
>>>
>>> HttpConnectionParams.setSoTimeout(params, maxTimeout);
>>> HttpConnectionParams.setConnectionTimeout(params, maxTimeout);
>>>
>>> this.client = new DefaultHttpClient(this.connManager, params);
>>>
>>> However, for some reason the HTTP_ELEMENT_CHARSET parameter is
>>> "bleeding" over from my server into my client, forcing my client
>>> requests to be in UTF-8 as well.
>>>
>>> Why is this happening? Are the settings for SyncBasicHttpParams global somehow?
>>>
>>
>> No should be no global settings anywhere in HttpCore or HttpClient.
>> Please note, though, that parameters set on the HTTP request level will
>> always take precedence over parameters set on the HTTP client level. I
>> suspect that parameters of individual messages received by the async
>> listener inherit its settings.
>>
>> By the way, since you are using a non-blocking listener for incoming
>> connections, would not HttpAsyncClient be a better match for handling
>> outgoing connections than HttpClient?
>>
>> Cheers
>>
>> Oleg
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HttpParams appear to be static/global
Posted by Bill Speirs <bi...@gmail.com>.
That is exactly what is happening... the request that is passed to the
NHttpRequestHandler.handle() method has the HTTP params set with the
characters set = UTF-8. Good catch and makes sense as I basically pass
this request on to the client to execute.
I need to look into the async client and see how it would fit into my
design. I started with sync for both client & server, and quickly
out-grew the server side of things. I keep my
NHttpRequestHandler.handle() method async by wrapping the request in a
continuation and placing it on a ExecutorCompletionService.
Bill-
On Wed, Aug 24, 2011 at 10:39 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2011-08-24 at 10:28 -0400, Bill Speirs wrote:
>> I'm working on a proxy server using the 4.2-alpha1 httpcore and 4.1.2
>> httpclient. In the server portion of my code I setup
>> SyncBasicHttpParams as follows:
>>
>>
>> HttpParams params = new SyncBasicHttpParams();
>>
>> params.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 0)
>> .setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE,
>> 8 * 1024)
>> .setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK,
>> false)
>> .setBooleanParameter(CoreConnectionPNames.TCP_NODELAY, true)
>> .setParameter(CoreProtocolPNames.ORIGIN_SERVER, "FactSet
>> Lima Proxy")
>> .setParameter(CoreProtocolPNames.PROTOCOL_VERSION,
>> HttpVersion.HTTP_1_1.toString())
>> .setParameter(CoreProtocolPNames.HTTP_ELEMENT_CHARSET,
>> HTTP.UTF_8);
>>
>> I then use them to create an AsyncNHttpServiceHandler and
>> DefaultServerIOEventDispatch.
>>
>> In another portion of code I create a new SyncBasicHttpParams object
>> and use it to configure my client:
>> final HttpParams params = new SyncBasicHttpParams();
>>
>> params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, handleRedirects);
>> params.setBooleanParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS, false);
>> params.setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.IGNORE_COOKIES);
>>
>> HttpConnectionParams.setSoTimeout(params, maxTimeout);
>> HttpConnectionParams.setConnectionTimeout(params, maxTimeout);
>>
>> this.client = new DefaultHttpClient(this.connManager, params);
>>
>> However, for some reason the HTTP_ELEMENT_CHARSET parameter is
>> "bleeding" over from my server into my client, forcing my client
>> requests to be in UTF-8 as well.
>>
>> Why is this happening? Are the settings for SyncBasicHttpParams global somehow?
>>
>
> No should be no global settings anywhere in HttpCore or HttpClient.
> Please note, though, that parameters set on the HTTP request level will
> always take precedence over parameters set on the HTTP client level. I
> suspect that parameters of individual messages received by the async
> listener inherit its settings.
>
> By the way, since you are using a non-blocking listener for incoming
> connections, would not HttpAsyncClient be a better match for handling
> outgoing connections than HttpClient?
>
> Cheers
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HttpParams appear to be static/global
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2011-08-24 at 10:28 -0400, Bill Speirs wrote:
> I'm working on a proxy server using the 4.2-alpha1 httpcore and 4.1.2
> httpclient. In the server portion of my code I setup
> SyncBasicHttpParams as follows:
>
>
> HttpParams params = new SyncBasicHttpParams();
>
> params.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 0)
> .setIntParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE,
> 8 * 1024)
> .setBooleanParameter(CoreConnectionPNames.STALE_CONNECTION_CHECK,
> false)
> .setBooleanParameter(CoreConnectionPNames.TCP_NODELAY, true)
> .setParameter(CoreProtocolPNames.ORIGIN_SERVER, "FactSet
> Lima Proxy")
> .setParameter(CoreProtocolPNames.PROTOCOL_VERSION,
> HttpVersion.HTTP_1_1.toString())
> .setParameter(CoreProtocolPNames.HTTP_ELEMENT_CHARSET,
> HTTP.UTF_8);
>
> I then use them to create an AsyncNHttpServiceHandler and
> DefaultServerIOEventDispatch.
>
> In another portion of code I create a new SyncBasicHttpParams object
> and use it to configure my client:
> final HttpParams params = new SyncBasicHttpParams();
>
> params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, handleRedirects);
> params.setBooleanParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS, false);
> params.setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.IGNORE_COOKIES);
>
> HttpConnectionParams.setSoTimeout(params, maxTimeout);
> HttpConnectionParams.setConnectionTimeout(params, maxTimeout);
>
> this.client = new DefaultHttpClient(this.connManager, params);
>
> However, for some reason the HTTP_ELEMENT_CHARSET parameter is
> "bleeding" over from my server into my client, forcing my client
> requests to be in UTF-8 as well.
>
> Why is this happening? Are the settings for SyncBasicHttpParams global somehow?
>
No should be no global settings anywhere in HttpCore or HttpClient.
Please note, though, that parameters set on the HTTP request level will
always take precedence over parameters set on the HTTP client level. I
suspect that parameters of individual messages received by the async
listener inherit its settings.
By the way, since you are using a non-blocking listener for incoming
connections, would not HttpAsyncClient be a better match for handling
outgoing connections than HttpClient?
Cheers
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org