You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Daniel Mueller <da...@digitalstrider.com> on 2007/06/10 12:44:20 UTC
current state of httpclient 4.0
Hi all,
In what state is the new version of the httpclient? I checked it out from
SVN and it seems to be using the new httpcore underneath. I think you have
refactored the original httpclient into two projects, so the two are fully
functional. Is that correct? And since my main interest lies in the nio part
of the httpcore, what needs to be done to have this part working with the
httpclient (if there's anything which is not working, not sure about that)?
Some pointers would be great, so I can speed up my understanding of the
code.
Cheers,
Daniel
Re: current state of httpclient 4.0
Posted by Roland Weber <os...@dubioso.net>.
Roland Weber wrote:
> I had some ideas about logging,
> but they were not formally written down iirc.
Actually, they were:
http://mail-archives.apache.org/mod_mbox/jakarta-httpcomponents-dev/200606.mbox/%3c44943568.7080102@dubioso.net%3e
cheers,
Roland
---------------------------------------------------------------------
To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpcomponents-dev-help@jakarta.apache.org
Re: current state of httpclient 4.0
Posted by Roland Weber <os...@dubioso.net>.
Hi Daniel,
> I'm now reading the wiki and will then delve into the code. You can
> expect a
> whole host of new questions when I'm through with that ;)
I'll be on a business trip for some time next week. I will
answer questions that where not addressed by others when
I am back.
> A couple of
> things
> I do have already on my mind:
>
> - I'm not too well versed in http to know all the details involved.
> I'd certainly need a guide so that the things I come up with end up being
> too simplistic.
We'll point out what you are missing.
> I actually don't quite see the ouch yet, but I'll
> investigate.
expect-continue requires the client to wait for a response
from the server before sending the message body. Perfect
match for an _asynchronous_ API, isn't it? ;-)
> - What version of jdk would you target with httpDispatch? My first
> shot would be: API maybe 1.3, and then pluggable backends depending on
> needs (don't know yet).
1.4 or 5.0. When we started with the discussions
two years ago, 1.4 was reasonable. When I see where
we are now, it's probably OK to require 5.0 if that
makes niossl available. The API will probably be
compatible with 1.4 anyway.
> - Is there a testsuite how you test httpclient, beside the unit tests.
Nope. We have test servers in the unit tests, there
is little that an extra testsuite could add. It's
not like we can afford to put up dedicated servers
just to test SSL tunnelling through a proxy with
NTLM authentication :-) I'm running some of the
examples against a local Apache proxy. Otherwise,
it's the test servers in the unit tests.
cheers,
Roland
---------------------------------------------------------------------
To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpcomponents-dev-help@jakarta.apache.org
Re: current state of httpclient 4.0
Posted by Daniel Mueller <da...@digitalstrider.com>.
Hi Roland,
On 6/10/07, Roland Weber <os...@dubioso.net> wrote:
>
> Hi Daniel,
>
> my apologies for the state of our website.
no problem, happens with my projects at work all the time...
The current
> status of HttpAsync is whack-puffed. I buried the ashes
> in SVN [2] just after HttpCore-alpha4 was released.
> There are no current plans to resurrect the idea. Since
> most people associate "async" with "NIO", we are now
> referring to the original idea as HttpDispatch. The
> empty HttpAsync component has become a potential home
> for NIO modules.
>
> > Was it intended to be used both with a blocking and a non-blocking
> > implementation (only internally, not from the client side of view, i.e.
> one
> > implementation on httpcore, one on httpcore-nio)?
>
> Yes. The API is intentionally neutral. I meant to develop
> a blocking implementation, since that's what I know. I
> agree with Oleg that a NIO implementation would make
> more sense.
Sounds like a nice API if there's a decent way to pull it off.
> Are there any thoughts around, preliminary designs,
> > starting points to investigate for the design
> > of such a component?
>
> There is a design page in the Wiki:
> http://wiki.apache.org/jakarta-httpclient/HttpAsyncThreadingDesign
>
> The design for the dispatching of requests using
> blocking IO is rather well developed. There is no
> corresponding design for NIO. Questions such as the
> handling of tunnelling, authentication, redirects,
> or expect-continue handshake (Ouch!) had not yet
> been adressed. I had some ideas about logging,
> but they were not formally written down iirc.
> The design was created at a very early point of
> HttpComponents development. In fact, way too early.
> Since it was whack-puffed, we have created a lot of
> pieces for that puzzle in HttpConn and HttpClient.
>
> If you feel like working on a NIO implementation,
> you could start adding your ideas to the Wiki.
> I can dig out the API and update it based on some
> of the newer pieces in HttpClient and/or HttpConn.
> I feel that at the current state of development,
> there would be no point in resurrecting the early
> attempt at a blocking implementation.
>
I'm now reading the wiki and will then delve into the code. You can expect a
whole host of new questions when I'm through with that ;) A couple of things
I do have already on my mind:
- I'm not too well versed in http to know all the details involved.
I'd certainly need a guide so that the things I come up with end up being
too simplistic. I actually don't quite see the ouch yet, but I'll
investigate.
- What version of jdk would you target with httpDispatch? My first
shot would be: API maybe 1.3, and then pluggable backends depending on
needs (don't know yet).
- Is there a testsuite how you test httpclient, beside the unit tests.
I'd like to have some way of telling if I'm on the right track and
regression testing most of the time does the trick with alternative
implementations.
Thanks for the pointers,
Daniel
Re: current state of httpclient 4.0
Posted by Roland Weber <os...@dubioso.net>.
Hi Daniel,
my apologies for the state of our website. The current
status of HttpAsync is whack-puffed. I buried the ashes
in SVN [2] just after HttpCore-alpha4 was released.
There are no current plans to resurrect the idea. Since
most people associate "async" with "NIO", we are now
referring to the original idea as HttpDispatch. The
empty HttpAsync component has become a potential home
for NIO modules.
> This does look more or less like what I want to do. I take
> it that as of yet, not much work has occurred on that front.
Quite some work, but little tangible result.
> What are the plans for this project?
Unless committers come rushing to us, it will remain
whack-puffed for another year, or two, or longer.
> Is this the asynchronous variant of HttpClient
> (with an appropriate API)?
Oleg likes to think of it that way. I guess it would
develop in that direction, if we had cycles to spare.
> Was it intended to be used both with a blocking and a non-blocking
> implementation (only internally, not from the client side of view, i.e. one
> implementation on httpcore, one on httpcore-nio)?
Yes. The API is intentionally neutral. I meant to develop
a blocking implementation, since that's what I know. I
agree with Oleg that a NIO implementation would make
more sense.
> Are there any thoughts around, preliminary designs,
> starting points to investigate for the design
> of such a component?
There is a design page in the Wiki:
http://wiki.apache.org/jakarta-httpclient/HttpAsyncThreadingDesign
The design for the dispatching of requests using
blocking IO is rather well developed. There is no
corresponding design for NIO. Questions such as the
handling of tunnelling, authentication, redirects,
or expect-continue handshake (Ouch!) had not yet
been adressed. I had some ideas about logging,
but they were not formally written down iirc.
The design was created at a very early point of
HttpComponents development. In fact, way too early.
Since it was whack-puffed, we have created a lot of
pieces for that puzzle in HttpConn and HttpClient.
If you feel like working on a NIO implementation,
you could start adding your ideas to the Wiki.
I can dig out the API and update it based on some
of the newer pieces in HttpClient and/or HttpConn.
I feel that at the current state of development,
there would be no point in resurrecting the early
attempt at a blocking implementation.
cheers,
Roland
[1]
http://mail-archives.apache.org/mod_mbox/jakarta-httpcomponents-dev/200609.mbox/%3c450EBD12.6000902@dubioso.net%3e
[2]
http://svn.apache.org/repos/asf/jakarta/httpcomponents/httpasync/branches/suspended-at-HttpCoreAlpha4/
---------------------------------------------------------------------
To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpcomponents-dev-help@jakarta.apache.org
Re: current state of httpclient 4.0
Posted by Daniel Mueller <da...@digitalstrider.com>.
Hi Roland,
Thanks for the pointers, I'll investigate those.
On the website and in the repository there's the HttpAsync Project. It
states in the overview:
The HttpAsync component provides an asynchronous interface for client
side HTTP communication, which separates the sending of a request from the
receiving of the response. This simplifies development of multithreaded HTTP
applications and allows for new features such as pipelining.
This does look more or less like what I want to do. I take it that as of
yet, not much work has occurred on that front. What are the plans for this
project? Is this the asynchronous variant of HttpClient (with an appropriate
API)? Was it intended to be used both with a blocking and a non-blocking
implementation (only internally, not from the client side of view, i.e. one
implementation on httpcore, one on httpcore-nio)? Are there any thoughts
around, preliminary designs, starting points to investigate for the design
of such a component?
Thanks in advance,
Daniel
On 6/10/07, Roland Weber <os...@dubioso.net> wrote:
>
> Hi Daniel,
>
> > And since my main interest lies in the nio part
> > of the httpcore, what needs to be done to have this part working with
> the
> > httpclient (if there's anything which is not working, not sure about
> that)?
> > Some pointers would be great, so I can speed up my understanding of the
> > code.
>
> Although it's not currently split explicitly into build modules,
> the HttpClient component has four parts of code. They are
> recognizable by the package structure. The four parts are:
>
> * HttpCookie: should work with NIO all right
>
> * HttpAuth: should work with NIO all right. The logic for
> performing authentication is partially in HttpClient (see below),
> in particular the (Default)HttpAuthenticationHandler. That one
> should work all right with NIO, too.
>
> * HttpConn: connection management is strictly blocking IO,
> you can't use that API with NIO. There are some classes about
> route building (HttpRoute, RouteTracker, RouteDirector) which
> I believe can be useful with NIO too.
>
> * HttpClient: the default (and only) implementation is based
> on HttpConn and therefore doesn't mix with NIO. There is one
> class HttpState which has ties to both HttpCookie and HttpAuth,
> that should work all right in a NIO context. Splitting that
> class into pieces that can be put directly in HttpCookie and
> HttpAuth, respectively, is on my list post-alpha1.
>
> The rest of available documentation is the JavaDocs, or the
> pieces of information posted to the mailing list. If you
> need HttpClient functionality or parts thereof in a NIO
> context, I recommend that you dive into class
> o.a.h.impl.client.DefaultClientRequestExecutor. That is
> where all the parts are tied together for blocking IO.
> Identify what you need, then do the same with NIO.
>
> hope that helps,
> Roland
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> httpcomponents-dev-help@jakarta.apache.org
>
>
Re: current state of httpclient 4.0
Posted by Roland Weber <os...@dubioso.net>.
Hi Daniel,
> And since my main interest lies in the nio part
> of the httpcore, what needs to be done to have this part working with the
> httpclient (if there's anything which is not working, not sure about that)?
> Some pointers would be great, so I can speed up my understanding of the
> code.
Although it's not currently split explicitly into build modules,
the HttpClient component has four parts of code. They are
recognizable by the package structure. The four parts are:
* HttpCookie: should work with NIO all right
* HttpAuth: should work with NIO all right. The logic for
performing authentication is partially in HttpClient (see below),
in particular the (Default)HttpAuthenticationHandler. That one
should work all right with NIO, too.
* HttpConn: connection management is strictly blocking IO,
you can't use that API with NIO. There are some classes about
route building (HttpRoute, RouteTracker, RouteDirector) which
I believe can be useful with NIO too.
* HttpClient: the default (and only) implementation is based
on HttpConn and therefore doesn't mix with NIO. There is one
class HttpState which has ties to both HttpCookie and HttpAuth,
that should work all right in a NIO context. Splitting that
class into pieces that can be put directly in HttpCookie and
HttpAuth, respectively, is on my list post-alpha1.
The rest of available documentation is the JavaDocs, or the
pieces of information posted to the mailing list. If you
need HttpClient functionality or parts thereof in a NIO
context, I recommend that you dive into class
o.a.h.impl.client.DefaultClientRequestExecutor. That is
where all the parts are tied together for blocking IO.
Identify what you need, then do the same with NIO.
hope that helps,
Roland
---------------------------------------------------------------------
To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpcomponents-dev-help@jakarta.apache.org
Re: current state of httpclient 4.0
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2007-06-10 at 14:10 +0200, Daniel Mueller wrote:
> > > And since my main interest lies in the nio part
> > > of the httpcore, what needs to be done to have this part working with
> > the
> > > httpclient (if there's anything which is not working, not sure about
> > that)?
> > > Some pointers would be great, so I can speed up my understanding of the
> > > code.
> > >
> >
> > This used to be and still is a major point of contention. I believe NIO
> > makes very little sense for 'pure' HTTP servers and clients. To date NIO
> > has always tended to be significantly slower than classic I/O as long as
> > the number of I/O worker threads was below several thousands. I still do
> > not see a valid use case where a pure HTTP client may need more than
> > just a handful of I/O threads (a few hundred at most). That basically
> > eliminates all advantages of NIO for HttpClient.
>
>
>
> Having said all that, I am very, very, very open to helping people build
> > client side components based on HttpCore NIO to be used in some HTTP
> > proxy or gateway services.
> >
>
> I have a quite limited knowledge of NIO and IO in general as a matter of
> fact. Not much beside mostly theoretical stuff and these things often don't
> apply one to one in the field. Having said that my use case is the
> following:
> I want to build an access layer to the amazon S3 service. The S3 service has
> a SOAP and a REST API and I'm currently opting for the REST variant. This
> access layer will be used as one of the back ends of a processing pipeline
> inside a server. The rest of the server is built in SEDA style with event
> driven/asynchronous interfaces between the stages. That's why I need my
> access layer to be event driven/asynchronous. The logical step was then to
> think about implementing the access layer also in a asynchronous manner
> which is where NIO came into view because it encourages that kind of design.
> While investigating S3, REST, NIO and asynchronous (http) clients I stumbled
> accross several interesting things and one of them is the restlet API[1]. It
> looks as if it would solve a good portion of my demands for an asynchronous
> handling of resources, but has no asynchronous client implementation as of
> today. The one they do have is built on httpclient 3.x. While following that
> trail I ended up with your still alpha version of httpcomponents and decided
> to have a look.
> If you have a look at my use case and then think a bit further along that
> line of thought you end up with something that almost behaves like a gateway
> but uses webservices (REST or SOAP) as the backend. WS* and REST is all the
> rage today. While it certainly does not solve all the advertised problems,
> it does solve a couple. And building your servers in a non-blocking fashion
> is also gaining momentum (I have no good pointers for this one, it's simply
> a gut feeling). And the last link in the chain is the call to backend
> systems of any sort (web services, databases, file system, whatever). So
> maybe from this point of view, NIO makes sense after all.
>
I am getting old and grumpy ;-) So, let me turn this around. If the raw
throughput and simplicity are not overriding concerns for your
architecture, I does make sense to take a very close look at the event
driven, non-blocking transport layer based on NIO. In your particular
case NIO does seem like a better choice.
> Currently most of these ideas are still being formulated and I'm simply
> looking for ways to achieve what I have in mind. I've spotted your view on
> NIO http clients already somewhere else in the list-archives and would
> certainly be very interested if you could elaborate a bit.
All I am saying one should base their decisions on the set of
requirements and goals they are striving to achieve, not hype or trends.
NIO solves some problems posed by classic I/O paradigm at the expense of
introducing others (complexity and tendency to be prone to out of memory
conditions). So, as the old saying goes: pick up the right tool for the
job.
I ran enough performance tests to be convinced NIO always tended to be
perform worse than classic I/O in terms of raw throughput. Every new SUN
JRE to date that offered better NIO performance also happened to be more
efficient at switching contexts, so all gains made by NIO were
immediately offset by a more efficient threading support.
Bottom-line, NIO comes at a certain price, see for yourself if it is
worth paying given your particular problem domain.
> Those plans I
> have aren't set in stone and I want a solution which solves my problem,
> nothing more.
>
> But if the architecture makes any sense at all, I do have to have the
> components to achieve it. And I will help to get things rolling where I can
> in the httpcomponents project.
>
Any contribution to the project would be hugely appreciated. I guess all
present project committers would love to see more people on board.
Cheers
Oleg
> Thanks and cheers
> Daniel
>
> [1] http://www.restlet.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpcomponents-dev-help@jakarta.apache.org
Re: current state of httpclient 4.0
Posted by Daniel Mueller <da...@digitalstrider.com>.
> > And since my main interest lies in the nio part
> > of the httpcore, what needs to be done to have this part working with
> the
> > httpclient (if there's anything which is not working, not sure about
> that)?
> > Some pointers would be great, so I can speed up my understanding of the
> > code.
> >
>
> This used to be and still is a major point of contention. I believe NIO
> makes very little sense for 'pure' HTTP servers and clients. To date NIO
> has always tended to be significantly slower than classic I/O as long as
> the number of I/O worker threads was below several thousands. I still do
> not see a valid use case where a pure HTTP client may need more than
> just a handful of I/O threads (a few hundred at most). That basically
> eliminates all advantages of NIO for HttpClient.
Having said all that, I am very, very, very open to helping people build
> client side components based on HttpCore NIO to be used in some HTTP
> proxy or gateway services.
>
I have a quite limited knowledge of NIO and IO in general as a matter of
fact. Not much beside mostly theoretical stuff and these things often don't
apply one to one in the field. Having said that my use case is the
following:
I want to build an access layer to the amazon S3 service. The S3 service has
a SOAP and a REST API and I'm currently opting for the REST variant. This
access layer will be used as one of the back ends of a processing pipeline
inside a server. The rest of the server is built in SEDA style with event
driven/asynchronous interfaces between the stages. That's why I need my
access layer to be event driven/asynchronous. The logical step was then to
think about implementing the access layer also in a asynchronous manner
which is where NIO came into view because it encourages that kind of design.
While investigating S3, REST, NIO and asynchronous (http) clients I stumbled
accross several interesting things and one of them is the restlet API[1]. It
looks as if it would solve a good portion of my demands for an asynchronous
handling of resources, but has no asynchronous client implementation as of
today. The one they do have is built on httpclient 3.x. While following that
trail I ended up with your still alpha version of httpcomponents and decided
to have a look.
If you have a look at my use case and then think a bit further along that
line of thought you end up with something that almost behaves like a gateway
but uses webservices (REST or SOAP) as the backend. WS* and REST is all the
rage today. While it certainly does not solve all the advertised problems,
it does solve a couple. And building your servers in a non-blocking fashion
is also gaining momentum (I have no good pointers for this one, it's simply
a gut feeling). And the last link in the chain is the call to backend
systems of any sort (web services, databases, file system, whatever). So
maybe from this point of view, NIO makes sense after all.
Currently most of these ideas are still being formulated and I'm simply
looking for ways to achieve what I have in mind. I've spotted your view on
NIO http clients already somewhere else in the list-archives and would
certainly be very interested if you could elaborate a bit. Those plans I
have aren't set in stone and I want a solution which solves my problem,
nothing more.
But if the architecture makes any sense at all, I do have to have the
components to achieve it. And I will help to get things rolling where I can
in the httpcomponents project.
Thanks and cheers
Daniel
[1] http://www.restlet.org/
Re: current state of httpclient 4.0
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2007-06-10 at 12:44 +0200, Daniel Mueller wrote:
> Hi all,
>
> In what state is the new version of the httpclient? I checked it out from
> SVN and it seems to be using the new httpcore underneath. I think you have
> refactored the original httpclient into two projects, so the two are fully
> functional. Is that correct?
This is correct. As we leaned from our experience of building and
maintaining HttpClient 3.x there were many cases where monolithic
HttpClient proved not flexible enough, primarily when used in proxies of
some kind or by web crawlers. So, we wanted to make HttpClient more
modular to enable users to assemble their own custom HTTP services in a
piecemeal fashion. Currently we maintain two independently versioned
modules: HttpCore and HttpClient, but we are open to accepting more
components into the fold _provided_ there are enough contributors to
maintain them.
> And since my main interest lies in the nio part
> of the httpcore, what needs to be done to have this part working with the
> httpclient (if there's anything which is not working, not sure about that)?
> Some pointers would be great, so I can speed up my understanding of the
> code.
>
This used to be and still is a major point of contention. I believe NIO
makes very little sense for 'pure' HTTP servers and clients. To date NIO
has always tended to be significantly slower than classic I/O as long as
the number of I/O worker threads was below several thousands. I still do
not see a valid use case where a pure HTTP client may need more than
just a handful of I/O threads (a few hundred at most). That basically
eliminates all advantages of NIO for HttpClient.
Having said all that, I am very, very, very open to helping people build
client side components based on HttpCore NIO to be used in some HTTP
proxy or gateway services.
Hope this clarifies things somewhat
Cheers
Oleg
> Cheers,
> Daniel
---------------------------------------------------------------------
To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpcomponents-dev-help@jakarta.apache.org