You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by Mike Heath <mh...@apache.org> on 2008/02/12 02:27:42 UTC
[AsyncWeb] Ideas for client
I posted some use cases here:
http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
still need some refinement to properly convey what I want but they're a
decent start.
I've also posted a hypothetical AsyncWeb Client API at
http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
further promote discussion and foster more innovative ideas. I would
love to here some feedback on this API. What do you like, dislike, not
understand? Where do you see room for improvement? The API is really
rough in places but for the most part it conveys the ideas I've had over
the past week or so. Any suggestions for name changes to classes and/or
methods are welcome.
-Mike
Re: [AsyncWeb] Ideas for client
Posted by Mike Heath <mh...@apache.org>.
This is very cool to hear Matteo! I will definitely have to take a look
at your code. It would be nice to integrate some of this into MINA.
-Mike
Matteo Merli wrote:
> Hi,
>
> I've started an asyncronous DNS client based on MINA for my HTTP
> caching proxy (also based on MINA:
> http://alijo.googlecode.com )
>
> The client is based on the code from the DNS server in Apache Directory.
>
> The implementations is not complete yes as :
>
> * Only returns first IP address in the DNS reply
> * Only uses first DNS server (primary), as read from /etc/resolv.conf
> * Can cache DNS addresses, but.. caches forever as no expiration is applied
> * Does not use aliases
>
> I plan to work more on the DNS client. If anyone is interested, the
> sources can be found here:
>
> DNS Codec:
> http://alijo.googlecode.com/svn/trunk/protocol-dns/
>
> DNS Client:
> http://alijo.googlecode.com/svn/trunk/alijo/src/main/java/org/mm/proxycache/dns/
>
> Again, the code is not polished , but if there is interest I can work
> onrefining it and extracting it in a decent form.
>
> Regards,
> Matteo
> <ma...@gmail.com>
>
> On Feb 12, 2008 3:59 PM, Eero Nevalainen <ee...@indagon.com> wrote:
>> Btw, what was the status of DNS resolution in MINA? I remember that at
>> some point DNS resolving used the blocking java API calls. I could
>> imagine that an asynchronous HTTP-client would have to perform a lot of
>> DNS queries(or not, depending on use). Could this become a pithole?
>>
>> -Eero
>>
>>
>> Rick McGuire wrote:
>>> The one feature I like about the AHC client that appears to be missing
>>> here is the higher-level abstraction of an HTTP request. The one
>>> drawback of doing everything with URLs is the requirement that the user
>>> of the client needs to be responsible for encoding all of the parameter
>>> information in the URL. In the AHC approach, a request is configured as
>>> an operation to a particular address and additional specifics of the
>>> operation are attached to the request (parameters, credentials needed
>>> for authentication, proxy configuration etc.). The AHC client then uses
>>> that information to handle the URL encoding, authentication challenges,
>>> proxy connection, etc. There are times where a straighforward "fetch me
>>> this URL mode" is sufficient. There are other situations where that
>>> becomes awkward to use.
>>>
>>> Rick
>>>
>>> Mike Heath wrote:
>>>> I posted some use cases here:
>>>> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
>>>> still need some refinement to properly convey what I want but they're a
>>>> decent start.
>>>>
>>>> I've also posted a hypothetical AsyncWeb Client API at
>>>> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
>>>> further promote discussion and foster more innovative ideas. I would
>>>> love to here some feedback on this API. What do you like, dislike, not
>>>> understand? Where do you see room for improvement? The API is really
>>>> rough in places but for the most part it conveys the ideas I've had over
>>>> the past week or so. Any suggestions for name changes to classes and/or
>>>> methods are welcome.
>>>>
>>>> -Mike
>>>>
>>>>
>>
>
>
>
Re: [AsyncWeb] Ideas for client
Posted by Matteo Merli <ma...@gmail.com>.
Hi,
I've started an asyncronous DNS client based on MINA for my HTTP
caching proxy (also based on MINA:
http://alijo.googlecode.com )
The client is based on the code from the DNS server in Apache Directory.
The implementations is not complete yes as :
* Only returns first IP address in the DNS reply
* Only uses first DNS server (primary), as read from /etc/resolv.conf
* Can cache DNS addresses, but.. caches forever as no expiration is applied
* Does not use aliases
I plan to work more on the DNS client. If anyone is interested, the
sources can be found here:
DNS Codec:
http://alijo.googlecode.com/svn/trunk/protocol-dns/
DNS Client:
http://alijo.googlecode.com/svn/trunk/alijo/src/main/java/org/mm/proxycache/dns/
Again, the code is not polished , but if there is interest I can work
onrefining it and extracting it in a decent form.
Regards,
Matteo
<ma...@gmail.com>
On Feb 12, 2008 3:59 PM, Eero Nevalainen <ee...@indagon.com> wrote:
> Btw, what was the status of DNS resolution in MINA? I remember that at
> some point DNS resolving used the blocking java API calls. I could
> imagine that an asynchronous HTTP-client would have to perform a lot of
> DNS queries(or not, depending on use). Could this become a pithole?
>
> -Eero
>
>
> Rick McGuire wrote:
> > The one feature I like about the AHC client that appears to be missing
> > here is the higher-level abstraction of an HTTP request. The one
> > drawback of doing everything with URLs is the requirement that the user
> > of the client needs to be responsible for encoding all of the parameter
> > information in the URL. In the AHC approach, a request is configured as
> > an operation to a particular address and additional specifics of the
> > operation are attached to the request (parameters, credentials needed
> > for authentication, proxy configuration etc.). The AHC client then uses
> > that information to handle the URL encoding, authentication challenges,
> > proxy connection, etc. There are times where a straighforward "fetch me
> > this URL mode" is sufficient. There are other situations where that
> > becomes awkward to use.
> >
> > Rick
> >
> > Mike Heath wrote:
> >> I posted some use cases here:
> >> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
> >> still need some refinement to properly convey what I want but they're a
> >> decent start.
> >>
> >> I've also posted a hypothetical AsyncWeb Client API at
> >> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
> >> further promote discussion and foster more innovative ideas. I would
> >> love to here some feedback on this API. What do you like, dislike, not
> >> understand? Where do you see room for improvement? The API is really
> >> rough in places but for the most part it conveys the ideas I've had over
> >> the past week or so. Any suggestions for name changes to classes and/or
> >> methods are welcome.
> >>
> >> -Mike
> >>
> >>
> >
>
>
--
Matteo Merli
<ma...@gmail.com>
Re: Blocking DNS queries (Was: Re: [AsyncWeb] Ideas for client)
Posted by Maarten Bosteels <mb...@gmail.com>.
On Feb 13, 2008 10:07 AM, 이희승 (Trustin Lee) <tr...@gmail.com> wrote:
> 2008-02-12 (화), 23:57 -0700, Mike Heath 쓰시길:
> > 이희승 (Trustin Lee) wrote:
> > > 2008-02-12 (화), 11:16 -0700, Mike Heath 쓰시길:
> > >> Eero Nevalainen wrote:
> > >>> Btw, what was the status of DNS resolution in MINA? I remember that
> at
> > >>> some point DNS resolving used the blocking java API calls. I could
> > >>> imagine that an asynchronous HTTP-client would have to perform a lot
> of
> > >>> DNS queries(or not, depending on use). Could this become a pithole?
> > >> That's a very good point, Eero. I think blocking DNS lookups could
> > >> definitely be a very big pit hole. Did anything ever come from all
> the
> > >> asynchronous DNS lookup discussions?
> > >>
> > >> And that's not the only DNS issue. If a DNS returns multiple A
> records
> > >> for a given host name, most browsers will try to connect to the IP
> > >> address from the first A record. If that fails, they'll try the next
> IP
> > >> address and so on until the connection succeeds or all of the A
> records
> > >> have been exhausted. We should support this same behavior in
> AsyncWeb.
> > >
> > > I thought it's the operating system who decides which A record to use
> > > and it's often random round-robin style. Is there any related
> > > documentation or is it from your experiment? Just curious.
> >
> > The way Mozilla browsers handle DNS Round Robin is explained here
> > http://www.mozilla.org/docs/netlib/dns.html under the "Round Robin
> > Support" section.
> >
> > > It would be nice to have a built in DNS lookup mechanism in MINA as we
> > > discussed in our previous discussion, but once the domain name is
> > > resolved, it will be cached by the operation system, so it shouldn't
> be
> > > a big problem in most cases. It will of course be a problem if
> someone
> > > is writing a web robot or something similar.
> >
> > I do agree that in most cases it won't be a problem.
>
> Thanks for the link to the documentation.
>
> So, I guess we need some pluggability in domain name resolution. For
> now, we are using InetSocketAddress and as long as we depend on it, we
> don't have any control over how domain name is resolved in MINA. This
> mean that we need to provide our own endpoint representation classes.
> It will probably cause the major API change, but I think it worth to do
> this. How does that sound? Will we need to implement this in 2.0 or...
> in 3.0? :)
>
I would say 2.0-M2
Maarten
>
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
>
Re: Blocking DNS queries (Was: Re: [AsyncWeb] Ideas for client)
Posted by Mike Heath <mh...@apache.org>.
이희승 (Trustin Lee) wrote:
<snip>
> So, I guess we need some pluggability in domain name resolution. For
> now, we are using InetSocketAddress and as long as we depend on it, we
> don't have any control over how domain name is resolved in MINA. This
> mean that we need to provide our own endpoint representation classes.
> It will probably cause the major API change, but I think it worth to do
> this. How does that sound? Will we need to implement this in 2.0 or...
> in 3.0? :)
Unless we already have some DNS client code that we're comfortable with,
I think we should push this off to a 2.1 or 3.0 release.
-Mike
Re: Blocking DNS queries (Was: Re: [AsyncWeb] Ideas for client)
Posted by "이희승 (Trustin Lee)" <tr...@gmail.com>.
2008-02-12 (화), 23:57 -0700, Mike Heath 쓰시길:
> 이희승 (Trustin Lee) wrote:
> > 2008-02-12 (화), 11:16 -0700, Mike Heath 쓰시길:
> >> Eero Nevalainen wrote:
> >>> Btw, what was the status of DNS resolution in MINA? I remember that at
> >>> some point DNS resolving used the blocking java API calls. I could
> >>> imagine that an asynchronous HTTP-client would have to perform a lot of
> >>> DNS queries(or not, depending on use). Could this become a pithole?
> >> That's a very good point, Eero. I think blocking DNS lookups could
> >> definitely be a very big pit hole. Did anything ever come from all the
> >> asynchronous DNS lookup discussions?
> >>
> >> And that's not the only DNS issue. If a DNS returns multiple A records
> >> for a given host name, most browsers will try to connect to the IP
> >> address from the first A record. If that fails, they'll try the next IP
> >> address and so on until the connection succeeds or all of the A records
> >> have been exhausted. We should support this same behavior in AsyncWeb.
> >
> > I thought it's the operating system who decides which A record to use
> > and it's often random round-robin style. Is there any related
> > documentation or is it from your experiment? Just curious.
>
> The way Mozilla browsers handle DNS Round Robin is explained here
> http://www.mozilla.org/docs/netlib/dns.html under the "Round Robin
> Support" section.
>
> > It would be nice to have a built in DNS lookup mechanism in MINA as we
> > discussed in our previous discussion, but once the domain name is
> > resolved, it will be cached by the operation system, so it shouldn't be
> > a big problem in most cases. It will of course be a problem if someone
> > is writing a web robot or something similar.
>
> I do agree that in most cases it won't be a problem.
Thanks for the link to the documentation.
So, I guess we need some pluggability in domain name resolution. For
now, we are using InetSocketAddress and as long as we depend on it, we
don't have any control over how domain name is resolved in MINA. This
mean that we need to provide our own endpoint representation classes.
It will probably cause the major API change, but I think it worth to do
this. How does that sound? Will we need to implement this in 2.0 or...
in 3.0? :)
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: Blocking DNS queries (Was: Re: [AsyncWeb] Ideas for client)
Posted by Mike Heath <mh...@apache.org>.
이희승 (Trustin Lee) wrote:
> 2008-02-12 (화), 11:16 -0700, Mike Heath 쓰시길:
>> Eero Nevalainen wrote:
>>> Btw, what was the status of DNS resolution in MINA? I remember that at
>>> some point DNS resolving used the blocking java API calls. I could
>>> imagine that an asynchronous HTTP-client would have to perform a lot of
>>> DNS queries(or not, depending on use). Could this become a pithole?
>> That's a very good point, Eero. I think blocking DNS lookups could
>> definitely be a very big pit hole. Did anything ever come from all the
>> asynchronous DNS lookup discussions?
>>
>> And that's not the only DNS issue. If a DNS returns multiple A records
>> for a given host name, most browsers will try to connect to the IP
>> address from the first A record. If that fails, they'll try the next IP
>> address and so on until the connection succeeds or all of the A records
>> have been exhausted. We should support this same behavior in AsyncWeb.
>
> I thought it's the operating system who decides which A record to use
> and it's often random round-robin style. Is there any related
> documentation or is it from your experiment? Just curious.
The way Mozilla browsers handle DNS Round Robin is explained here
http://www.mozilla.org/docs/netlib/dns.html under the "Round Robin
Support" section.
> It would be nice to have a built in DNS lookup mechanism in MINA as we
> discussed in our previous discussion, but once the domain name is
> resolved, it will be cached by the operation system, so it shouldn't be
> a big problem in most cases. It will of course be a problem if someone
> is writing a web robot or something similar.
I do agree that in most cases it won't be a problem.
-Mike
Blocking DNS queries (Was: Re: [AsyncWeb] Ideas for client)
Posted by "이희승 (Trustin Lee)" <tr...@gmail.com>.
2008-02-12 (화), 11:16 -0700, Mike Heath 쓰시길:
> Eero Nevalainen wrote:
> > Btw, what was the status of DNS resolution in MINA? I remember that at
> > some point DNS resolving used the blocking java API calls. I could
> > imagine that an asynchronous HTTP-client would have to perform a lot of
> > DNS queries(or not, depending on use). Could this become a pithole?
>
> That's a very good point, Eero. I think blocking DNS lookups could
> definitely be a very big pit hole. Did anything ever come from all the
> asynchronous DNS lookup discussions?
>
> And that's not the only DNS issue. If a DNS returns multiple A records
> for a given host name, most browsers will try to connect to the IP
> address from the first A record. If that fails, they'll try the next IP
> address and so on until the connection succeeds or all of the A records
> have been exhausted. We should support this same behavior in AsyncWeb.
I thought it's the operating system who decides which A record to use
and it's often random round-robin style. Is there any related
documentation or is it from your experiment? Just curious.
It would be nice to have a built in DNS lookup mechanism in MINA as we
discussed in our previous discussion, but once the domain name is
resolved, it will be cached by the operation system, so it shouldn't be
a big problem in most cases. It will of course be a problem if someone
is writing a web robot or something similar.
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: [AsyncWeb] Ideas for client
Posted by Mike Heath <mh...@apache.org>.
Eero Nevalainen wrote:
> Btw, what was the status of DNS resolution in MINA? I remember that at
> some point DNS resolving used the blocking java API calls. I could
> imagine that an asynchronous HTTP-client would have to perform a lot of
> DNS queries(or not, depending on use). Could this become a pithole?
That's a very good point, Eero. I think blocking DNS lookups could
definitely be a very big pit hole. Did anything ever come from all the
asynchronous DNS lookup discussions?
And that's not the only DNS issue. If a DNS returns multiple A records
for a given host name, most browsers will try to connect to the IP
address from the first A record. If that fails, they'll try the next IP
address and so on until the connection succeeds or all of the A records
have been exhausted. We should support this same behavior in AsyncWeb.
-Mike
>
> -Eero
>
> Rick McGuire wrote:
>> The one feature I like about the AHC client that appears to be missing
>> here is the higher-level abstraction of an HTTP request. The one
>> drawback of doing everything with URLs is the requirement that the
>> user of the client needs to be responsible for encoding all of the
>> parameter information in the URL. In the AHC approach, a request is
>> configured as an operation to a particular address and additional
>> specifics of the operation are attached to the request (parameters,
>> credentials needed for authentication, proxy configuration etc.). The
>> AHC client then uses that information to handle the URL encoding,
>> authentication challenges, proxy connection, etc. There are times
>> where a straighforward "fetch me this URL mode" is sufficient. There
>> are other situations where that becomes awkward to use.
>>
>> Rick
>>
>> Mike Heath wrote:
>>> I posted some use cases here:
>>> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
>>> still need some refinement to properly convey what I want but they're a
>>> decent start.
>>>
>>> I've also posted a hypothetical AsyncWeb Client API at
>>> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
>>> further promote discussion and foster more innovative ideas. I would
>>> love to here some feedback on this API. What do you like, dislike, not
>>> understand? Where do you see room for improvement? The API is really
>>> rough in places but for the most part it conveys the ideas I've had over
>>> the past week or so. Any suggestions for name changes to classes and/or
>>> methods are welcome.
>>>
>>> -Mike
>>>
>>>
>>
>
>
Re: [AsyncWeb] Ideas for client
Posted by Eero Nevalainen <ee...@indagon.com>.
Btw, what was the status of DNS resolution in MINA? I remember that at
some point DNS resolving used the blocking java API calls. I could
imagine that an asynchronous HTTP-client would have to perform a lot of
DNS queries(or not, depending on use). Could this become a pithole?
-Eero
Rick McGuire wrote:
> The one feature I like about the AHC client that appears to be missing
> here is the higher-level abstraction of an HTTP request. The one
> drawback of doing everything with URLs is the requirement that the user
> of the client needs to be responsible for encoding all of the parameter
> information in the URL. In the AHC approach, a request is configured as
> an operation to a particular address and additional specifics of the
> operation are attached to the request (parameters, credentials needed
> for authentication, proxy configuration etc.). The AHC client then uses
> that information to handle the URL encoding, authentication challenges,
> proxy connection, etc. There are times where a straighforward "fetch me
> this URL mode" is sufficient. There are other situations where that
> becomes awkward to use.
>
> Rick
>
> Mike Heath wrote:
>> I posted some use cases here:
>> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
>> still need some refinement to properly convey what I want but they're a
>> decent start.
>>
>> I've also posted a hypothetical AsyncWeb Client API at
>> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
>> further promote discussion and foster more innovative ideas. I would
>> love to here some feedback on this API. What do you like, dislike, not
>> understand? Where do you see room for improvement? The API is really
>> rough in places but for the most part it conveys the ideas I've had over
>> the past week or so. Any suggestions for name changes to classes and/or
>> methods are welcome.
>>
>> -Mike
>>
>>
>
Re: [AsyncWeb] Ideas for client
Posted by Tuure Laurinolli <tu...@indagon.com>.
Mike Heath wrote:
<snip>
>> I think in general some amount of knowledge about the request is unavoidable
>> on the response. If the response object has no knowledge of what the
>> request was, there will be certain things that are not possible to do.
>> There may be many examples, but one example I can think of is inspecting
>> the Set-Cookie header to accept or reject cookies. Things like domain and
>> path validation are needed but that requires the request. AHC's
>> HttpResponseMessage also does not know about the request, and it's been a
>> pain point.
>
> I agree. I've added an HttpRequestFuture that extends HttpFuture in my
> proposed API. The HttpRequestFuture has a getHttpRequest() method on
> it. I'm adding a few other tweaks and then I'll republish my proposed API.
Keeping the request around may be expensive, consider POSTs with a large
body for example. However, I think that having the other information
available in the response would be a good idea. However, it's not
strictly necessary as that information (or the request itself, even) can
be passed to the Listener directly, if a new listener is created for
each request.
Re: [AsyncWeb] Ideas for client
Posted by Mike Heath <mh...@apache.org>.
Sangjin Lee wrote:
<snip>
> That sounds like a good approach. Still some necessary normalization (like
> something I see in DefaultHttpRequest.normalize()) would need to happen in
> both cases, no?
normalize() is defined in the MutableHttpRequest interface so in the
case of a mutable request we would definitely have to normalize it. In
the case of a non-mutable request (currently we don't have an
HttpRequest that doesn't also implement MutableHttpRequest), we'll have
to assume the request is already "normalized."
<snip>
> I think in general some amount of knowledge about the request is unavoidable
> on the response. If the response object has no knowledge of what the
> request was, there will be certain things that are not possible to do.
> There may be many examples, but one example I can think of is inspecting
> the Set-Cookie header to accept or reject cookies. Things like domain and
> path validation are needed but that requires the request. AHC's
> HttpResponseMessage also does not know about the request, and it's been a
> pain point.
I agree. I've added an HttpRequestFuture that extends HttpFuture in my
proposed API. The HttpRequestFuture has a getHttpRequest() method on
it. I'm adding a few other tweaks and then I'll republish my proposed API.
-Mike
Re: [AsyncWeb] Ideas for client
Posted by Sangjin Lee <sj...@gmail.com>.
On Wed, Feb 13, 2008 at 3:04 PM, Mike Heath <mh...@apache.org> wrote:
> Sangjin Lee wrote:
> > What I've seen with AHC is that the configuration is often the most
> > challenging aspect. One metaphor I used is that HttpClient is more like
> a
> > browser. Things like keep-alive, user-agent, accept-encoding, etc.
> normally
> > belong to the browser and not at the individual request level. I'm sure
> > there are many delicate decisions to make that don't get solved easily
> by
> > this metaphor, but I think it's certainly useful.
>
> This is the metaphor I would like to follow is well which is why in my
> API proposal I didn't provide a mechanism for sending a raw HttpRequest
> object through the HttpClient.
>
> After reviewing all the feedback and thinking about the problem more,
> I'm thinking that if the user submits a MutableHttpRequest, then the
> HttpClient will modify that request as appropriate. If the HttpRequest
> does not implement MutableHttpRequest, then the request will be sent
> unmodified. I think this should solve the problem adequately. WDT?
That sounds like a good approach. Still some necessary normalization (like
something I see in DefaultHttpRequest.normalize()) would need to happen in
both cases, no?
>
> > I've mentioned this before, but one thing I like with AHC is handling
> > multiple requests with a completion queue (not unlike
> CompletionService).
> > This addresses a use case which is a variation of one of Mike's use
> cases.
> >
> > Suppose you want to send requests to N URLs concurrently. You want to
> limit
> > the overall duration to a certain time, and you want to place a standard
> > error result for the URL for which the response did not get in time.
> This
> > is a pretty typical situation with a "scatter-and-gather" scenario (e.g.
> > portal with multiple server-side content aggregation, etc.).
> >
> > With a completion queue, one can do things like the following:
> >
> > CompletionQueue q;
> > client.send(request1, q);
> > client.send(request2, q);
> > ...
> > client.send(requestN, q);
> >
> > for (int i = 0; i < N; i++) {
> > Future f = q.take();
> > Response r = f.get();
> > }
> >
> > This can be done by the user in terms of a listener/callback, but it
> would
> > certainly be nice to provide support for this... My 2 cents.
>
> I too really like the idea of using a Queue for handling futures. It
> opens up a lot of interesting possibilities.
>
> The problem I see is for each method in HttpClient, do we provide an
> overloaded version that accepts a Queue? This would make the API very
> cluttered IMO.
I suspect this would be an HttpListener in your class proposal. We could
provide a completion queue as an HttpListener. Callers can then simply add
the queue to their future to handle it. Yeah, I'm not too enthused about
adding the queue to the APIs, although that's what we ended up doing. :)
One nice thing about the completion queue is that it can be used with
multiple HttpClients as well.
>
>
> We use an HttpListener to add the completed future to the queue. See
> the following example.
>
> BlockingQueue q;
> queueListener = new QueueListener(q);
>
> client.send(request1).addListener(queueListener);
> client.send(request2).addListener(queueListener);
> client.send(request3).addListener(queueListener);
> client.send(request4).addListener(queueListener);
>
> for (int i = 0; i < N; i++) {
> Future f = q.take();
> HttpResponse r = f.get();
> }
>
> The problem with this approach is that with the AsyncWeb client API, as
> I've proposed it, there would be no way to know which request the future
> object represents because we don't know the order in which each
> HttpListener will be invoked.
I think in general some amount of knowledge about the request is unavoidable
on the response. If the response object has no knowledge of what the
request was, there will be certain things that are not possible to do.
There may be many examples, but one example I can think of is inspecting
the Set-Cookie header to accept or reject cookies. Things like domain and
path validation are needed but that requires the request. AHC's
HttpResponseMessage also does not know about the request, and it's been a
pain point.
>
>
Re: [AsyncWeb] Ideas for client
Posted by Sangjin Lee <sj...@gmail.com>.
On Fri, Feb 29, 2008 at 12:03 AM, Alan D. Cabrera <li...@toolazydogs.com>
wrote:
>
> On Feb 13, 2008, at 3:04 PM, Mike Heath wrote:
>
> > Sangjin Lee wrote:
> >> What I've seen with AHC is that the configuration is often the most
> >> challenging aspect. One metaphor I used is that HttpClient is
> >> more like a
> >> browser. Things like keep-alive, user-agent, accept-encoding,
> >> etc. normally
> >> belong to the browser and not at the individual request level.
> >> I'm sure
> >> there are many delicate decisions to make that don't get solved
> >> easily by
> >> this metaphor, but I think it's certainly useful.
> >
> > This is the metaphor I would like to follow is well which is why in my
> > API proposal I didn't provide a mechanism for sending a raw
> > HttpRequest
> > object through the HttpClient.
> >
> > After reviewing all the feedback and thinking about the problem more,
> > I'm thinking that if the user submits a MutableHttpRequest, then the
> > HttpClient will modify that request as appropriate. If the
> > HttpRequest
> > does not implement MutableHttpRequest, then the request will be sent
> > unmodified. I think this should solve the problem adequately. WDT?
>
> I think we should keep things crazy simple until we have a problem we
> need to solve. Maybe you are and I don't see it. Can you explain
> why we need to use HttpRequest in the HttpClient?
>
> >
> >> I've mentioned this before, but one thing I like with AHC is handling
> >> multiple requests with a completion queue (not unlike
> >> CompletionService).
> >> This addresses a use case which is a variation of one of Mike's
> >> use cases.
> >>
> >> Suppose you want to send requests to N URLs concurrently. You
> >> want to limit
> >> the overall duration to a certain time, and you want to place a
> >> standard
> >> error result for the URL for which the response did not get in
> >> time. This
> >> is a pretty typical situation with a "scatter-and-gather" scenario
> >> (e.g.
> >> portal with multiple server-side content aggregation, etc.).
> >>
> >> With a completion queue, one can do things like the following:
> >>
> >> CompletionQueue q;
> >> client.send(request1, q);
> >> client.send(request2, q);
> >> ...
> >> client.send(requestN, q);
> >>
> >> for (int i = 0; i < N; i++) {
> >> Future f = q.take();
> >> Response r = f.get();
> >> }
> >>
> >> This can be done by the user in terms of a listener/callback, but
> >> it would
> >> certainly be nice to provide support for this... My 2 cents.
> >
> > I too really like the idea of using a Queue for handling futures. It
> > opens up a lot of interesting possibilities.
> >
> > The problem I see is for each method in HttpClient, do we provide an
> > overloaded version that accepts a Queue? This would make the API very
> > cluttered IMO.
> >
> > We use an HttpListener to add the completed future to the queue. See
> > the following example.
> >
> > BlockingQueue q;
> > queueListener = new QueueListener(q);
> >
> > client.send(request1).addListener(queueListener);
> > client.send(request2).addListener(queueListener);
> > client.send(request3).addListener(queueListener);
> > client.send(request4).addListener(queueListener);
> >
> > for (int i = 0; i < N; i++) {
> > Future f = q.take();
> > HttpResponse r = f.get();
> > }
> >
> > The problem with this approach is that with the AsyncWeb client
> > API, as
> > I've proposed it, there would be no way to know which request the
> > future
> > object represents because we don't know the order in which each
> > HttpListener will be invoked.
>
> I'm not sure how we keep track of requests in Sangjin's proposal
> either. If we can't, then your solution is an elegant equivalent.
>
>
> Regards,
> Alan
>
>
Keeping track of the requests is a concern that exists in both proposals.
Thanks,
Sangjin
Re: [AsyncWeb] Ideas for client
Posted by "Alan D. Cabrera" <li...@toolazydogs.com>.
On Feb 13, 2008, at 3:04 PM, Mike Heath wrote:
> Sangjin Lee wrote:
>> What I've seen with AHC is that the configuration is often the most
>> challenging aspect. One metaphor I used is that HttpClient is
>> more like a
>> browser. Things like keep-alive, user-agent, accept-encoding,
>> etc. normally
>> belong to the browser and not at the individual request level.
>> I'm sure
>> there are many delicate decisions to make that don't get solved
>> easily by
>> this metaphor, but I think it's certainly useful.
>
> This is the metaphor I would like to follow is well which is why in my
> API proposal I didn't provide a mechanism for sending a raw
> HttpRequest
> object through the HttpClient.
>
> After reviewing all the feedback and thinking about the problem more,
> I'm thinking that if the user submits a MutableHttpRequest, then the
> HttpClient will modify that request as appropriate. If the
> HttpRequest
> does not implement MutableHttpRequest, then the request will be sent
> unmodified. I think this should solve the problem adequately. WDT?
I think we should keep things crazy simple until we have a problem we
need to solve. Maybe you are and I don't see it. Can you explain
why we need to use HttpRequest in the HttpClient?
>
>> I've mentioned this before, but one thing I like with AHC is handling
>> multiple requests with a completion queue (not unlike
>> CompletionService).
>> This addresses a use case which is a variation of one of Mike's
>> use cases.
>>
>> Suppose you want to send requests to N URLs concurrently. You
>> want to limit
>> the overall duration to a certain time, and you want to place a
>> standard
>> error result for the URL for which the response did not get in
>> time. This
>> is a pretty typical situation with a "scatter-and-gather" scenario
>> (e.g.
>> portal with multiple server-side content aggregation, etc.).
>>
>> With a completion queue, one can do things like the following:
>>
>> CompletionQueue q;
>> client.send(request1, q);
>> client.send(request2, q);
>> ...
>> client.send(requestN, q);
>>
>> for (int i = 0; i < N; i++) {
>> Future f = q.take();
>> Response r = f.get();
>> }
>>
>> This can be done by the user in terms of a listener/callback, but
>> it would
>> certainly be nice to provide support for this... My 2 cents.
>
> I too really like the idea of using a Queue for handling futures. It
> opens up a lot of interesting possibilities.
>
> The problem I see is for each method in HttpClient, do we provide an
> overloaded version that accepts a Queue? This would make the API very
> cluttered IMO.
>
> We use an HttpListener to add the completed future to the queue. See
> the following example.
>
> BlockingQueue q;
> queueListener = new QueueListener(q);
>
> client.send(request1).addListener(queueListener);
> client.send(request2).addListener(queueListener);
> client.send(request3).addListener(queueListener);
> client.send(request4).addListener(queueListener);
>
> for (int i = 0; i < N; i++) {
> Future f = q.take();
> HttpResponse r = f.get();
> }
>
> The problem with this approach is that with the AsyncWeb client
> API, as
> I've proposed it, there would be no way to know which request the
> future
> object represents because we don't know the order in which each
> HttpListener will be invoked.
I'm not sure how we keep track of requests in Sangjin's proposal
either. If we can't, then your solution is an elegant equivalent.
Regards,
Alan
Re: [AsyncWeb] Ideas for client
Posted by Mike Heath <mh...@apache.org>.
Sangjin Lee wrote:
> What I've seen with AHC is that the configuration is often the most
> challenging aspect. One metaphor I used is that HttpClient is more like a
> browser. Things like keep-alive, user-agent, accept-encoding, etc. normally
> belong to the browser and not at the individual request level. I'm sure
> there are many delicate decisions to make that don't get solved easily by
> this metaphor, but I think it's certainly useful.
This is the metaphor I would like to follow is well which is why in my
API proposal I didn't provide a mechanism for sending a raw HttpRequest
object through the HttpClient.
After reviewing all the feedback and thinking about the problem more,
I'm thinking that if the user submits a MutableHttpRequest, then the
HttpClient will modify that request as appropriate. If the HttpRequest
does not implement MutableHttpRequest, then the request will be sent
unmodified. I think this should solve the problem adequately. WDT?
> I've mentioned this before, but one thing I like with AHC is handling
> multiple requests with a completion queue (not unlike CompletionService).
> This addresses a use case which is a variation of one of Mike's use cases.
>
> Suppose you want to send requests to N URLs concurrently. You want to limit
> the overall duration to a certain time, and you want to place a standard
> error result for the URL for which the response did not get in time. This
> is a pretty typical situation with a "scatter-and-gather" scenario (e.g.
> portal with multiple server-side content aggregation, etc.).
>
> With a completion queue, one can do things like the following:
>
> CompletionQueue q;
> client.send(request1, q);
> client.send(request2, q);
> ...
> client.send(requestN, q);
>
> for (int i = 0; i < N; i++) {
> Future f = q.take();
> Response r = f.get();
> }
>
> This can be done by the user in terms of a listener/callback, but it would
> certainly be nice to provide support for this... My 2 cents.
I too really like the idea of using a Queue for handling futures. It
opens up a lot of interesting possibilities.
The problem I see is for each method in HttpClient, do we provide an
overloaded version that accepts a Queue? This would make the API very
cluttered IMO.
We use an HttpListener to add the completed future to the queue. See
the following example.
BlockingQueue q;
queueListener = new QueueListener(q);
client.send(request1).addListener(queueListener);
client.send(request2).addListener(queueListener);
client.send(request3).addListener(queueListener);
client.send(request4).addListener(queueListener);
for (int i = 0; i < N; i++) {
Future f = q.take();
HttpResponse r = f.get();
}
The problem with this approach is that with the AsyncWeb client API, as
I've proposed it, there would be no way to know which request the future
object represents because we don't know the order in which each
HttpListener will be invoked.
I will try to refine this in the API that I'm proposing.
-Mike
> On Feb 12, 2008 10:53 AM, Mike Heath <mh...@apache.org> wrote:
>
>> Rick McGuire wrote:
>>> Mike Heath wrote:
>>>> Rick McGuire wrote:
>>>>
>>>>> The one feature I like about the AHC client that appears to be missing
>>>>> here is the higher-level abstraction of an HTTP request. The one
>>>>> drawback of doing everything with URLs is the requirement that the
>> user
>>>>> of the client needs to be responsible for encoding all of the
>> parameter
>>>>> information in the URL. In the AHC approach, a request is configured
>> as
>>>>> an operation to a particular address and additional specifics of the
>>>>> operation are attached to the request (parameters, credentials needed
>>>>> for authentication, proxy configuration etc.). The AHC client then
>> uses
>>>>> that information to handle the URL encoding, authentication
>> challenges,
>>>>> proxy connection, etc. There are times where a straighforward "fetch
>> me
>>>>> this URL mode" is sufficient. There are other situations where that
>>>>> becomes awkward to use.
>>>>>
>>>> Thanks for the feedback Rick. That's one of the issues I've been going
>>>> back and forth on. In the HttpConnection interface, you have no choice
>>>> but to build and send an HttpRequest manually. So you have some
>>>> flexibility there. I've been debating whether or not to put support
>> for
>>>> sending a raw HttpRequest in the 'simplified' HttpClient. I've been
>>>> leaning toward add that functionality to the HttpClient your reply is
>>>> making me lean even further in that direction.
>>>>
>>>> -Mike
>>>>
>>> Yes, it is a bit of a balancing act. After a certain point, it becomes
>>> difficult to continue adding all of the possibilities to the method
>>> signatures. I think a good balance point is to support having a request
>>> object that's got all of the bells and whistles and a few methods that
>>> handle the most typical use scenarios (but in the end, are just creating
>>> the same type of request object on the caller's behalf).
>>> Rick
>> Yes I totally agree. A lot of the methods I put in HttpClient were
>> brainstorming ideas. I was trying to support the most common cases as
>> well as some of the more complex cases (like doing a PUT with a file,
>> for example.)
>>
>> We need to find a balance of simplicity and function and I'm not sure
>> where that point is at. Some of the issues that need to be addressed
>> with supporting sending raw HttpRequest objects using the HttpClient
>> (the way I've modeled it, that is) are:
>> - If the HttpClient has connection pooling enabled but a request is
>> sent without KeepAlive and Connection headers, do we automatically
>> change the request to turn keepAlive on and use connection pooling or do
>> we create a new connection?
>> - If the HttpClient has a cookie manager, do we automatically add
>> cookies to a manually created HttpRequest?
>> - Are there other headers that we would want to automatically populate
>> if their absent (User-Agent, Accept*, etc.) ?
>> - Do we even want to modify a manually created HttpRequest? Should
>> this be configurable? If so how?
>>
>> I don't have answers to these questions which is why I decided to just
>> put support for sending raw HttpRequest messages in the HttpConnection
>> interface because that's the interface I intended to make available for
>> complex use cases (COMET, dealing with large data streams, etc.)
>> However, I think we need to find some middle ground. I'm just not sure
>> where that middle ground is.
>>
>> Thanks again for your feedback, Rick.
>>
>> -Mike
>>
>>>
>>>>
>>>>
>>>>> Rick
>>>>>
>>>>> Mike Heath wrote:
>>>>>
>>>>>> I posted some use cases here:
>>>>>> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
>>>>>> still need some refinement to properly convey what I want but they're
>> a
>>>>>> decent start.
>>>>>>
>>>>>> I've also posted a hypothetical AsyncWeb Client API at
>>>>>> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent
>> to
>>>>>> further promote discussion and foster more innovative ideas. I would
>>>>>> love to here some feedback on this API. What do you like, dislike,
>> not
>>>>>> understand? Where do you see room for improvement? The API is
>> really
>>>>>> rough in places but for the most part it conveys the ideas I've had
>>>>>> over
>>>>>> the past week or so. Any suggestions for name changes to classes
>>>>>> and/or
>>>>>> methods are welcome.
>>>>>>
>>>>>> -Mike
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
>
Re: [AsyncWeb] Ideas for client
Posted by Sangjin Lee <sj...@gmail.com>.
What I've seen with AHC is that the configuration is often the most
challenging aspect. One metaphor I used is that HttpClient is more like a
browser. Things like keep-alive, user-agent, accept-encoding, etc. normally
belong to the browser and not at the individual request level. I'm sure
there are many delicate decisions to make that don't get solved easily by
this metaphor, but I think it's certainly useful.
I've mentioned this before, but one thing I like with AHC is handling
multiple requests with a completion queue (not unlike CompletionService).
This addresses a use case which is a variation of one of Mike's use cases.
Suppose you want to send requests to N URLs concurrently. You want to limit
the overall duration to a certain time, and you want to place a standard
error result for the URL for which the response did not get in time. This
is a pretty typical situation with a "scatter-and-gather" scenario (e.g.
portal with multiple server-side content aggregation, etc.).
With a completion queue, one can do things like the following:
CompletionQueue q;
client.send(request1, q);
client.send(request2, q);
...
client.send(requestN, q);
for (int i = 0; i < N; i++) {
Future f = q.take();
Response r = f.get();
}
This can be done by the user in terms of a listener/callback, but it would
certainly be nice to provide support for this... My 2 cents.
Thanks,
Sangjin
On Feb 12, 2008 10:53 AM, Mike Heath <mh...@apache.org> wrote:
> Rick McGuire wrote:
> > Mike Heath wrote:
> >> Rick McGuire wrote:
> >>
> >>> The one feature I like about the AHC client that appears to be missing
> >>> here is the higher-level abstraction of an HTTP request. The one
> >>> drawback of doing everything with URLs is the requirement that the
> user
> >>> of the client needs to be responsible for encoding all of the
> parameter
> >>> information in the URL. In the AHC approach, a request is configured
> as
> >>> an operation to a particular address and additional specifics of the
> >>> operation are attached to the request (parameters, credentials needed
> >>> for authentication, proxy configuration etc.). The AHC client then
> uses
> >>> that information to handle the URL encoding, authentication
> challenges,
> >>> proxy connection, etc. There are times where a straighforward "fetch
> me
> >>> this URL mode" is sufficient. There are other situations where that
> >>> becomes awkward to use.
> >>>
> >>
> >> Thanks for the feedback Rick. That's one of the issues I've been going
> >> back and forth on. In the HttpConnection interface, you have no choice
> >> but to build and send an HttpRequest manually. So you have some
> >> flexibility there. I've been debating whether or not to put support
> for
> >> sending a raw HttpRequest in the 'simplified' HttpClient. I've been
> >> leaning toward add that functionality to the HttpClient your reply is
> >> making me lean even further in that direction.
> >>
> >> -Mike
> >>
> > Yes, it is a bit of a balancing act. After a certain point, it becomes
> > difficult to continue adding all of the possibilities to the method
> > signatures. I think a good balance point is to support having a request
> > object that's got all of the bells and whistles and a few methods that
> > handle the most typical use scenarios (but in the end, are just creating
> > the same type of request object on the caller's behalf).
> > Rick
>
> Yes I totally agree. A lot of the methods I put in HttpClient were
> brainstorming ideas. I was trying to support the most common cases as
> well as some of the more complex cases (like doing a PUT with a file,
> for example.)
>
> We need to find a balance of simplicity and function and I'm not sure
> where that point is at. Some of the issues that need to be addressed
> with supporting sending raw HttpRequest objects using the HttpClient
> (the way I've modeled it, that is) are:
> - If the HttpClient has connection pooling enabled but a request is
> sent without KeepAlive and Connection headers, do we automatically
> change the request to turn keepAlive on and use connection pooling or do
> we create a new connection?
> - If the HttpClient has a cookie manager, do we automatically add
> cookies to a manually created HttpRequest?
> - Are there other headers that we would want to automatically populate
> if their absent (User-Agent, Accept*, etc.) ?
> - Do we even want to modify a manually created HttpRequest? Should
> this be configurable? If so how?
>
> I don't have answers to these questions which is why I decided to just
> put support for sending raw HttpRequest messages in the HttpConnection
> interface because that's the interface I intended to make available for
> complex use cases (COMET, dealing with large data streams, etc.)
> However, I think we need to find some middle ground. I'm just not sure
> where that middle ground is.
>
> Thanks again for your feedback, Rick.
>
> -Mike
>
> >
> >
> >>
> >>
> >>
> >>> Rick
> >>>
> >>> Mike Heath wrote:
> >>>
> >>>> I posted some use cases here:
> >>>> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
> >>>> still need some refinement to properly convey what I want but they're
> a
> >>>> decent start.
> >>>>
> >>>> I've also posted a hypothetical AsyncWeb Client API at
> >>>> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent
> to
> >>>> further promote discussion and foster more innovative ideas. I would
> >>>> love to here some feedback on this API. What do you like, dislike,
> not
> >>>> understand? Where do you see room for improvement? The API is
> really
> >>>> rough in places but for the most part it conveys the ideas I've had
> >>>> over
> >>>> the past week or so. Any suggestions for name changes to classes
> >>>> and/or
> >>>> methods are welcome.
> >>>>
> >>>> -Mike
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >
> >
>
>
Re: [AsyncWeb] Ideas for client
Posted by Mike Heath <mh...@apache.org>.
Rick McGuire wrote:
> Mike Heath wrote:
>> Rick McGuire wrote:
>>
>>> The one feature I like about the AHC client that appears to be missing
>>> here is the higher-level abstraction of an HTTP request. The one
>>> drawback of doing everything with URLs is the requirement that the user
>>> of the client needs to be responsible for encoding all of the parameter
>>> information in the URL. In the AHC approach, a request is configured as
>>> an operation to a particular address and additional specifics of the
>>> operation are attached to the request (parameters, credentials needed
>>> for authentication, proxy configuration etc.). The AHC client then uses
>>> that information to handle the URL encoding, authentication challenges,
>>> proxy connection, etc. There are times where a straighforward "fetch me
>>> this URL mode" is sufficient. There are other situations where that
>>> becomes awkward to use.
>>>
>>
>> Thanks for the feedback Rick. That's one of the issues I've been going
>> back and forth on. In the HttpConnection interface, you have no choice
>> but to build and send an HttpRequest manually. So you have some
>> flexibility there. I've been debating whether or not to put support for
>> sending a raw HttpRequest in the 'simplified' HttpClient. I've been
>> leaning toward add that functionality to the HttpClient your reply is
>> making me lean even further in that direction.
>>
>> -Mike
>>
> Yes, it is a bit of a balancing act. After a certain point, it becomes
> difficult to continue adding all of the possibilities to the method
> signatures. I think a good balance point is to support having a request
> object that's got all of the bells and whistles and a few methods that
> handle the most typical use scenarios (but in the end, are just creating
> the same type of request object on the caller's behalf).
> Rick
Yes I totally agree. A lot of the methods I put in HttpClient were
brainstorming ideas. I was trying to support the most common cases as
well as some of the more complex cases (like doing a PUT with a file,
for example.)
We need to find a balance of simplicity and function and I'm not sure
where that point is at. Some of the issues that need to be addressed
with supporting sending raw HttpRequest objects using the HttpClient
(the way I've modeled it, that is) are:
- If the HttpClient has connection pooling enabled but a request is
sent without KeepAlive and Connection headers, do we automatically
change the request to turn keepAlive on and use connection pooling or do
we create a new connection?
- If the HttpClient has a cookie manager, do we automatically add
cookies to a manually created HttpRequest?
- Are there other headers that we would want to automatically populate
if their absent (User-Agent, Accept*, etc.) ?
- Do we even want to modify a manually created HttpRequest? Should
this be configurable? If so how?
I don't have answers to these questions which is why I decided to just
put support for sending raw HttpRequest messages in the HttpConnection
interface because that's the interface I intended to make available for
complex use cases (COMET, dealing with large data streams, etc.)
However, I think we need to find some middle ground. I'm just not sure
where that middle ground is.
Thanks again for your feedback, Rick.
-Mike
>
>
>>
>>
>>
>>> Rick
>>>
>>> Mike Heath wrote:
>>>
>>>> I posted some use cases here:
>>>> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
>>>> still need some refinement to properly convey what I want but they're a
>>>> decent start.
>>>>
>>>> I've also posted a hypothetical AsyncWeb Client API at
>>>> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
>>>> further promote discussion and foster more innovative ideas. I would
>>>> love to here some feedback on this API. What do you like, dislike, not
>>>> understand? Where do you see room for improvement? The API is really
>>>> rough in places but for the most part it conveys the ideas I've had
>>>> over
>>>> the past week or so. Any suggestions for name changes to classes
>>>> and/or
>>>> methods are welcome.
>>>>
>>>> -Mike
>>>>
>>>>
>>>
>>
>>
>>
>
>
Re: [AsyncWeb] Ideas for client
Posted by Rick McGuire <ri...@gmail.com>.
Mike Heath wrote:
> Rick McGuire wrote:
>
>> The one feature I like about the AHC client that appears to be missing
>> here is the higher-level abstraction of an HTTP request. The one
>> drawback of doing everything with URLs is the requirement that the user
>> of the client needs to be responsible for encoding all of the parameter
>> information in the URL. In the AHC approach, a request is configured as
>> an operation to a particular address and additional specifics of the
>> operation are attached to the request (parameters, credentials needed
>> for authentication, proxy configuration etc.). The AHC client then uses
>> that information to handle the URL encoding, authentication challenges,
>> proxy connection, etc. There are times where a straighforward "fetch me
>> this URL mode" is sufficient. There are other situations where that
>> becomes awkward to use.
>>
>
> Thanks for the feedback Rick. That's one of the issues I've been going
> back and forth on. In the HttpConnection interface, you have no choice
> but to build and send an HttpRequest manually. So you have some
> flexibility there. I've been debating whether or not to put support for
> sending a raw HttpRequest in the 'simplified' HttpClient. I've been
> leaning toward add that functionality to the HttpClient your reply is
> making me lean even further in that direction.
>
> -Mike
>
Yes, it is a bit of a balancing act. After a certain point, it becomes
difficult to continue adding all of the possibilities to the method
signatures. I think a good balance point is to support having a request
object that's got all of the bells and whistles and a few methods that
handle the most typical use scenarios (but in the end, are just creating
the same type of request object on the caller's behalf).
Rick
>
>
>
>> Rick
>>
>> Mike Heath wrote:
>>
>>> I posted some use cases here:
>>> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
>>> still need some refinement to properly convey what I want but they're a
>>> decent start.
>>>
>>> I've also posted a hypothetical AsyncWeb Client API at
>>> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
>>> further promote discussion and foster more innovative ideas. I would
>>> love to here some feedback on this API. What do you like, dislike, not
>>> understand? Where do you see room for improvement? The API is really
>>> rough in places but for the most part it conveys the ideas I've had over
>>> the past week or so. Any suggestions for name changes to classes and/or
>>> methods are welcome.
>>>
>>> -Mike
>>>
>>>
>>>
>>
>
>
>
Re: [AsyncWeb] Ideas for client
Posted by Mike Heath <mh...@apache.org>.
Rick McGuire wrote:
> The one feature I like about the AHC client that appears to be missing
> here is the higher-level abstraction of an HTTP request. The one
> drawback of doing everything with URLs is the requirement that the user
> of the client needs to be responsible for encoding all of the parameter
> information in the URL. In the AHC approach, a request is configured as
> an operation to a particular address and additional specifics of the
> operation are attached to the request (parameters, credentials needed
> for authentication, proxy configuration etc.). The AHC client then uses
> that information to handle the URL encoding, authentication challenges,
> proxy connection, etc. There are times where a straighforward "fetch me
> this URL mode" is sufficient. There are other situations where that
> becomes awkward to use.
Thanks for the feedback Rick. That's one of the issues I've been going
back and forth on. In the HttpConnection interface, you have no choice
but to build and send an HttpRequest manually. So you have some
flexibility there. I've been debating whether or not to put support for
sending a raw HttpRequest in the 'simplified' HttpClient. I've been
leaning toward add that functionality to the HttpClient your reply is
making me lean even further in that direction.
-Mike
>
> Rick
>
> Mike Heath wrote:
>> I posted some use cases here:
>> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
>> still need some refinement to properly convey what I want but they're a
>> decent start.
>>
>> I've also posted a hypothetical AsyncWeb Client API at
>> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
>> further promote discussion and foster more innovative ideas. I would
>> love to here some feedback on this API. What do you like, dislike, not
>> understand? Where do you see room for improvement? The API is really
>> rough in places but for the most part it conveys the ideas I've had over
>> the past week or so. Any suggestions for name changes to classes and/or
>> methods are welcome.
>>
>> -Mike
>>
>>
>
>
Re: [AsyncWeb] Ideas for client
Posted by Rick McGuire <ri...@gmail.com>.
The one feature I like about the AHC client that appears to be missing
here is the higher-level abstraction of an HTTP request. The one
drawback of doing everything with URLs is the requirement that the user
of the client needs to be responsible for encoding all of the parameter
information in the URL. In the AHC approach, a request is configured as
an operation to a particular address and additional specifics of the
operation are attached to the request (parameters, credentials needed
for authentication, proxy configuration etc.). The AHC client then uses
that information to handle the URL encoding, authentication challenges,
proxy connection, etc. There are times where a straighforward "fetch me
this URL mode" is sufficient. There are other situations where that
becomes awkward to use.
Rick
Mike Heath wrote:
> I posted some use cases here:
> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
> still need some refinement to properly convey what I want but they're a
> decent start.
>
> I've also posted a hypothetical AsyncWeb Client API at
> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
> further promote discussion and foster more innovative ideas. I would
> love to here some feedback on this API. What do you like, dislike, not
> understand? Where do you see room for improvement? The API is really
> rough in places but for the most part it conveys the ideas I've had over
> the past week or so. Any suggestions for name changes to classes and/or
> methods are welcome.
>
> -Mike
>
>
Re: [AsyncWeb] Ideas for client
Posted by "Alan D. Cabrera" <li...@toolazydogs.com>.
On Mar 18, 2008, at 11:13 AM, Mike Heath wrote:
> Alan Cabrera wrote:
>>
>> On Mar 1, 2008, at 10:43 PM, Alan D. Cabrera wrote:
>>
>>>
>>> On Feb 29, 2008, at 2:55 PM, Mike Heath wrote:
>>>
>>>> Alan D. Cabrera wrote:
>>>>>
>>>>> So, HCF can be optional. If you create HCs directly any cookies
>>>>> that
>>>>> are created are scoped w/ that client. The HCF would take an
>>>>> implementation of a CookieManager (CM) interface. I guess HC
>>>>> could as
>>>>> well.
>>>>
>>>> That's how I've designed my API. The HCF has a configure CM and
>>>> the HC
>>>> can have it's own CM if that's desired.
>>>>
>>>> I don't have explicit support for CookieManager chaining though.
>>>> I'll
>>>> have to add that.
>>>
>>> What is CM chaining?
>>
>> Still curious.
>
> We discussed the idea of having a parent CM and a child CM would store
> its own cookies but if id didn't have a cookie it would look to its
> parent for the cookie. This is what I was referring to when I said
> "CM
> chaining". I was indicating that it may be a good idea to add
> explicit
> support to the client API for this type of parent/child relationship.
>
> I'm going to move the API we've been discussing to my sandbox so
> everyone can peruse the code and we can better discuss these ideas and
> start implementing them. I've made a number of changes that I'm
> anxious
> to get feedback on.
What use case does this support? It really strikes me as odd.
But, I guess it doesn't matter so long as the parent paradigm doesn't
sneak into the CM interface.
Regards,
Alan
Re: [AsyncWeb] Ideas for client
Posted by Mike Heath <mh...@apache.org>.
Alan Cabrera wrote:
>
> On Mar 1, 2008, at 10:43 PM, Alan D. Cabrera wrote:
>
>>
>> On Feb 29, 2008, at 2:55 PM, Mike Heath wrote:
>>
>>> Alan D. Cabrera wrote:
>>>>
>>>> So, HCF can be optional. If you create HCs directly any cookies that
>>>> are created are scoped w/ that client. The HCF would take an
>>>> implementation of a CookieManager (CM) interface. I guess HC could as
>>>> well.
>>>
>>> That's how I've designed my API. The HCF has a configure CM and the HC
>>> can have it's own CM if that's desired.
>>>
>>> I don't have explicit support for CookieManager chaining though. I'll
>>> have to add that.
>>
>> What is CM chaining?
>
> Still curious.
We discussed the idea of having a parent CM and a child CM would store
its own cookies but if id didn't have a cookie it would look to its
parent for the cookie. This is what I was referring to when I said "CM
chaining". I was indicating that it may be a good idea to add explicit
support to the client API for this type of parent/child relationship.
I'm going to move the API we've been discussing to my sandbox so
everyone can peruse the code and we can better discuss these ideas and
start implementing them. I've made a number of changes that I'm anxious
to get feedback on.
-Mike
Re: [AsyncWeb] Ideas for client
Posted by Alan Cabrera <ad...@toolazydogs.com>.
On Mar 1, 2008, at 10:43 PM, Alan D. Cabrera wrote:
>
> On Feb 29, 2008, at 2:55 PM, Mike Heath wrote:
>
>> Alan D. Cabrera wrote:
>>>
>>> So, HCF can be optional. If you create HCs directly any cookies
>>> that
>>> are created are scoped w/ that client. The HCF would take an
>>> implementation of a CookieManager (CM) interface. I guess HC
>>> could as
>>> well.
>>
>> That's how I've designed my API. The HCF has a configure CM and
>> the HC
>> can have it's own CM if that's desired.
>>
>> I don't have explicit support for CookieManager chaining though.
>> I'll
>> have to add that.
>
> What is CM chaining?
Still curious.
Regards,
Alan
Re: [AsyncWeb] Ideas for client
Posted by "Alan D. Cabrera" <li...@toolazydogs.com>.
On Mar 3, 2008, at 12:22 PM, Sangjin Lee wrote:
> I haven't had a chance to add my use case: the "scatter-and-gather"
> situation where one uses a completion queue to handle the result.
> Hopefully
> I'll do that some time this week...
> I also would like to discuss how to get back to the request while
> processing
> the response. It is often desirable and sometimes even necessary to
> find
> out the request from the response. How should we address this?
>
> One could obviously have a direct reference to the Request from the
> Response, but there are drawbacks. It basically means that the
> requests
> will stick around until the response goes out of scope. This may
> have some
> negative consequences especially if requests are large objects and/or
> mutable.
>
> Another approach might be to refer only to certain parts of the
> request from
> the response; e.g. the request URL. However, it's not clear if it's
> only
> the request URL callers would be interested in, and then the list of
> these
> attributes may grow pretty quickly.
>
> What do others think?
I like the idea of being able to attach a user object, or handle if
you will, to the request. This handle will be passed back in the
response.
Regards,
Alan
Re: [AsyncWeb] Ideas for client
Posted by Sangjin Lee <sj...@gmail.com>.
I haven't had a chance to add my use case: the "scatter-and-gather"
situation where one uses a completion queue to handle the result. Hopefully
I'll do that some time this week...
I also would like to discuss how to get back to the request while processing
the response. It is often desirable and sometimes even necessary to find
out the request from the response. How should we address this?
One could obviously have a direct reference to the Request from the
Response, but there are drawbacks. It basically means that the requests
will stick around until the response goes out of scope. This may have some
negative consequences especially if requests are large objects and/or
mutable.
Another approach might be to refer only to certain parts of the request from
the response; e.g. the request URL. However, it's not clear if it's only
the request URL callers would be interested in, and then the list of these
attributes may grow pretty quickly.
What do others think?
Thanks,
Sangjin
Re: [AsyncWeb] Ideas for client
Posted by "Alan D. Cabrera" <li...@toolazydogs.com>.
On Feb 29, 2008, at 2:55 PM, Mike Heath wrote:
> Alan D. Cabrera wrote:
>>
>> So, HCF can be optional. If you create HCs directly any cookies that
>> are created are scoped w/ that client. The HCF would take an
>> implementation of a CookieManager (CM) interface. I guess HC could
>> as
>> well.
>
> That's how I've designed my API. The HCF has a configure CM and the
> HC
> can have it's own CM if that's desired.
>
> I don't have explicit support for CookieManager chaining though. I'll
> have to add that.
What is CM chaining?
>> I see similarities w/ what Mike has proposed; geniuses think
>> alike. :)
>> So here are the issues with what I have with that design. Most are
>> probably bike shed issues.
>>
>> - The hierarchy is a bit confusing. HttpConnector and HttpClient
>> inherit off of HttpClientConfigurable but, HttpConnector can generate
>> HttpClients; seems odd to me in that it appears to be mixing
>> metaphors.
>
> The idea that I'm trying to convey here is that the HCF (or
> HttpConnector in the case of my API) can hold a configuration that
> will
> be the default configuration of the HCs the HCF creates.
Ok, so I guess the cause of the confusion for me would be that
HttpConnector can generate HttpClients and HttpConnecttions. We
discuss this below.
>> - I think that HttpClientFactory and HttpClient seems more concrete
>> than
>> HttpConnector and HttpClient/HttpConnection. The relationship
>> seems a
>> bit obtuse until one reads the javadocs and thinks on it a bit. I
>> suggest that we split your metaphor and have a
>> HttpClientFactory/HttpClient and HttpConnectionFactory/
>> HttpConnection.
>> I really doubt that someone will be generating *both* connections and
>> clients at the same time. The factories would extend HttpFactory
>> which
>> would be like HttpClientConfigurable but without a lot of the
>> parameters
>> like user agent. I really like name factory, it's an industry
>> accepted
>> pattern. Implementations of the factories would have getters/setters
>> for timeouts and what ever the factory implementer thought was cool,
>> e.g. pooling mumbo jumbo.
>
> I have to agree that I like the name HttpClientFactory too. I keep
> going back and forth on it though. I went with HttpConnector in my
> API
> because it's what would hold the MINA IoConnector instance and because
> it's creating both HttpConnections and HttpClients.
>
> The reason that I have HttpConnector creating both HttpClient and
> HttpConnections is because I was thinking that the HC implementation
> would just use an HttpConnection under the hood. This way we could
> create a generic HttpClient implementation that would just use a
> HttpConnection. We could also create a generic HttpConnection pooler
> that simply pool HttpConnection instances. This way we would only
> have
> to implement different HCF and HttpConnection instances to support
> different transports. This should make it easier for us to support
> different versions of MINA as well as make room for non MINA based
> implementations. Of course, why would anyone NOT want to use MINA? ;)
>
> As I think about it though, there's still no reason we can't separate
> the HttpClientFactory and the HttpConnectionFactory and still do
> what I
> just described. :)
Yeah, I think that while we probably will have an HTTPConnection under
the hood of an HTTPClient, there's no reason to stress that
implementation detail by having a connector generate both. I think we
should ask how does that strengthen the metaphor? I think it
detracts. When we split the two we get two nice clean and easy
metaphors for users to swallow.
We would only merge the two if it were commonplace for one to use both
to accomplish a single task; a situation that I am stretched to
contrive.
>> - HttpConnection, I don't see the point in setting an attribute w/
>> out a
>> value just to emulate a boolean. I'm sure that I'm being dense but I
>> think it's important to keep it crazy simple.
>> - setAttributeIfAbsent can be implemented by locking on the
>> connection,
>> no? I'm probably being dense here too.
>
> All the attribute methods were copied verbatim from MINA's IoSession.
>
> From an implementation perspective, using an attribute w/o a value to
> emulate a boolean keeps things crazy simple IMO. If I need a flag I
> can
> easily see if the flag is set by calling containsAttribute. I don't
> have to do something like
> ((Boolean)connection.getAttribute(key)).booleanvalue(). We use this
> patter in MINA frequently. It seams to work fine and is well
> understood
> by MINA users.
There's great merit in sticking w/ MINA's paradigm even if, IMO, this
bit is flawed.
I am not against checking to see if a key exists. I'm against setting
a key w/ no value which, IMO, smacks of an ad hoc addition. But, in
this case my opinion is moot and, like I said, a bike shed issue.
> Synchronizing on the HttpConnection would only work of all the
> attribute
> methods were synchronized which we don't guarantee. That is why we
> have
> setAttributeIfAbsent. Besides, if you're using a ConcurrentMap to
> implement the attribute support, you'll want to be able to use
> ConcurrentMap#putIfAbsent().
The default implementation should sync on this. Exotic
implementations would use the IoSessionDataStructureFactory and can
make the appropriate wrapper. I think that it smacks of an ad hoc
addition; JMHO. I think that there is great value in following the
paradigms that maps in Java use, i.e. wrap behavior. But, in this
additional case my opinion is moot and another bike shed issue.
>> Thoughts?
>
> Thanks for the feedback! I'm really excited to move forward with
> this.
> I've had a number of discussions with people outside of Apache WRT
> building an asynchronous web services client framework. The people I
> spoke to are specifically very interested in an asynchronous Amazon
> Web
> Services client.
Yeah, I'm writing a Xen API JCA Connector and want to use this puppy,
among other things.
Regards,
Alan
Re: [AsyncWeb] Ideas for client
Posted by Mike Heath <mh...@apache.org>.
Alan D. Cabrera wrote:
>
> On Feb 11, 2008, at 5:27 PM, Mike Heath wrote:
>
>> I posted some use cases here:
>> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
>> still need some refinement to properly convey what I want but they're a
>> decent start.
>
> I've added some the use cases that I need to support.
I saw those. They look good. I've got a few more use cases that I
would like to add as well. I hope to get around to doing that this weekend.
>> I've also posted a hypothetical AsyncWeb Client API at
>> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the intent to
>> further promote discussion and foster more innovative ideas. I would
>> love to here some feedback on this API. What do you like, dislike, not
>> understand? Where do you see room for improvement? The API is really
>> rough in places but for the most part it conveys the ideas I've had over
>> the past week or so. Any suggestions for name changes to classes and/or
>> methods are welcome.
>
>
> Nice. This was along the same lines as I was thinking.
>
> I like going back to my browser metaphor. I think that it fulfills the
> use cases in the wiki and Rick's concerns. I realize that what follows
> is a huge overlap to what you've proposed and what was already discussed.
>
> Let's say that we have an HttpClientFactory (HCF). This factory would
> contain the default configuration that could be used to generate any
> number of HttpClients (HC). I mention on the wiki that it would be a
> great place to hold "permanent" cookies across HCs. HCs can be thought
> of as instances of a browser. Temporary cookies would have the same
> lifetime as the HCs. Configuration information can also be set inside
> the HC and it would be shared across multiple gets/puts; which addresses
> some of Rick's and Sangjin's concerns.
I like this approach. You can still share cookies across the HC's but
session cookies are specific to the HC that received the session cookie.
Nice.
> So, HCF can be optional. If you create HCs directly any cookies that
> are created are scoped w/ that client. The HCF would take an
> implementation of a CookieManager (CM) interface. I guess HC could as
> well.
That's how I've designed my API. The HCF has a configure CM and the HC
can have it's own CM if that's desired.
I don't have explicit support for CookieManager chaining though. I'll
have to add that.
> I see similarities w/ what Mike has proposed; geniuses think alike. :)
> So here are the issues with what I have with that design. Most are
> probably bike shed issues.
>
> - The hierarchy is a bit confusing. HttpConnector and HttpClient
> inherit off of HttpClientConfigurable but, HttpConnector can generate
> HttpClients; seems odd to me in that it appears to be mixing metaphors.
The idea that I'm trying to convey here is that the HCF (or
HttpConnector in the case of my API) can hold a configuration that will
be the default configuration of the HCs the HCF creates.
> - I would have left some configuration specific information out of the
> HttpClientConfigurable interface and left it up to implementations to
> decide what is relevant, e.g. timeouts and user agents.
A lot of the configuration options are brain storming ideas. I haven't
decided what I like and what I don't like yet. :) I just wanted to put
ideas out there.
I do agree though, I think a lot of the configuration options should be
implementation specific. And as far as the user agent option goes, I
was thinking that should be in the configuration in case a user wants
spoof the user agent so the web server thinks the HC is IE or Firefox.
> - I'm guessing that HttpConnection is used for large read/writes. Can
> you put an example on the wiki on how this would be used?
Yes, the idea is far either large read/write or for long lived
connections (COMET clients). I'll try to put together or more concrete
example of what I'm trying accomplish.
> - I like the HttpFuture and how you can chain things.
This is copied out of the MINA API.
> - I think that HttpClientFactory and HttpClient seems more concrete than
> HttpConnector and HttpClient/HttpConnection. The relationship seems a
> bit obtuse until one reads the javadocs and thinks on it a bit. I
> suggest that we split your metaphor and have a
> HttpClientFactory/HttpClient and HttpConnectionFactory/HttpConnection.
> I really doubt that someone will be generating *both* connections and
> clients at the same time. The factories would extend HttpFactory which
> would be like HttpClientConfigurable but without a lot of the parameters
> like user agent. I really like name factory, it's an industry accepted
> pattern. Implementations of the factories would have getters/setters
> for timeouts and what ever the factory implementer thought was cool,
> e.g. pooling mumbo jumbo.
I have to agree that I like the name HttpClientFactory too. I keep
going back and forth on it though. I went with HttpConnector in my API
because it's what would hold the MINA IoConnector instance and because
it's creating both HttpConnections and HttpClients.
The reason that I have HttpConnector creating both HttpClient and
HttpConnections is because I was thinking that the HC implementation
would just use an HttpConnection under the hood. This way we could
create a generic HttpClient implementation that would just use a
HttpConnection. We could also create a generic HttpConnection pooler
that simply pool HttpConnection instances. This way we would only have
to implement different HCF and HttpConnection instances to support
different transports. This should make it easier for us to support
different versions of MINA as well as make room for non MINA based
implementations. Of course, why would anyone NOT want to use MINA? ;)
As I think about it though, there's still no reason we can't separate
the HttpClientFactory and the HttpConnectionFactory and still do what I
just described. :)
/me mulls it over
> - HttpConnection, I don't see the point in setting an attribute w/out a
> value just to emulate a boolean. I'm sure that I'm being dense but I
> think it's important to keep it crazy simple.
> - setAttributeIfAbsent can be implemented by locking on the connection,
> no? I'm probably being dense here too.
All the attribute methods were copied verbatim from MINA's IoSession.
>From an implementation perspective, using an attribute w/o a value to
emulate a boolean keeps things crazy simple IMO. If I need a flag I can
easily see if the flag is set by calling containsAttribute. I don't
have to do something like
((Boolean)connection.getAttribute(key)).booleanvalue(). We use this
patter in MINA frequently. It seams to work fine and is well understood
by MINA users.
Synchronizing on the HttpConnection would only work of all the attribute
methods were synchronized which we don't guarantee. That is why we have
setAttributeIfAbsent. Besides, if you're using a ConcurrentMap to
implement the attribute support, you'll want to be able to use
ConcurrentMap#putIfAbsent().
>
> Thoughts?
Thanks for the feedback! I'm really excited to move forward with this.
I've had a number of discussions with people outside of Apache WRT
building an asynchronous web services client framework. The people I
spoke to are specifically very interested in an asynchronous Amazon Web
Services client.
There's so much potential for this.
-Mike
Re: [AsyncWeb] Ideas for client
Posted by "Alan D. Cabrera" <li...@toolazydogs.com>.
On Feb 11, 2008, at 5:27 PM, Mike Heath wrote:
> I posted some use cases here:
> http://cwiki.apache.org/confluence/display/AWEB/ClientUseCases They
> still need some refinement to properly convey what I want but
> they're a
> decent start.
I've added some the use cases that I need to support.
> I've also posted a hypothetical AsyncWeb Client API at
> http://swamp.homelinux.net/mina/asyncweb/client/api/ with the
> intent to
> further promote discussion and foster more innovative ideas. I would
> love to here some feedback on this API. What do you like, dislike,
> not
> understand? Where do you see room for improvement? The API is really
> rough in places but for the most part it conveys the ideas I've had
> over
> the past week or so. Any suggestions for name changes to classes
> and/or
> methods are welcome.
Nice. This was along the same lines as I was thinking.
I like going back to my browser metaphor. I think that it fulfills
the use cases in the wiki and Rick's concerns. I realize that what
follows is a huge overlap to what you've proposed and what was
already discussed.
Let's say that we have an HttpClientFactory (HCF). This factory
would contain the default configuration that could be used to
generate any number of HttpClients (HC). I mention on the wiki that
it would be a great place to hold "permanent" cookies across HCs.
HCs can be thought of as instances of a browser. Temporary cookies
would have the same lifetime as the HCs. Configuration information
can also be set inside the HC and it would be shared across multiple
gets/puts; which addresses some of Rick's and Sangjin's concerns.
So, HCF can be optional. If you create HCs directly any cookies that
are created are scoped w/ that client. The HCF would take an
implementation of a CookieManager (CM) interface. I guess HC could
as well.
I see similarities w/ what Mike has proposed; geniuses think
alike. :) So here are the issues with what I have with that design.
Most are probably bike shed issues.
- The hierarchy is a bit confusing. HttpConnector and HttpClient
inherit off of HttpClientConfigurable but, HttpConnector can generate
HttpClients; seems odd to me in that it appears to be mixing metaphors.
- I would have left some configuration specific information out of
the HttpClientConfigurable interface and left it up to
implementations to decide what is relevant, e.g. timeouts and user
agents.
- I'm guessing that HttpConnection is used for large read/writes.
Can you put an example on the wiki on how this would be used?
- I like the HttpFuture and how you can chain things.
- I think that HttpClientFactory and HttpClient seems more concrete
than HttpConnector and HttpClient/HttpConnection. The relationship
seems a bit obtuse until one reads the javadocs and thinks on it a
bit. I suggest that we split your metaphor and have a
HttpClientFactory/HttpClient and HttpConnectionFactory/
HttpConnection. I really doubt that someone will be generating
*both* connections and clients at the same time. The factories would
extend HttpFactory which would be like HttpClientConfigurable but
without a lot of the parameters like user agent. I really like name
factory, it's an industry accepted pattern. Implementations of the
factories would have getters/setters for timeouts and what ever the
factory implementer thought was cool, e.g. pooling mumbo jumbo.
- HttpConnection, I don't see the point in setting an attribute w/out
a value just to emulate a boolean. I'm sure that I'm being dense but
I think it's important to keep it crazy simple.
- setAttributeIfAbsent can be implemented by locking on the
connection, no? I'm probably being dense here too.
Thoughts?
Regards,
Alan