Improving out of the box http facilities to deal with changing demands.

[Kevin Brown <et...@google.com>]

If you've been following the discussions on the spec discussion group, you
probably don't need much more context. If you haven't, I'd suggest heading
over there for a bit and reading through the templating discussion threads,
as this will potentially become a large issue for OpenSocial as a whole.

In order to effectively support an app model resembling what I've proposed
(which seems to be the direction the wind is blowing), we're highly likely
to see situations where proxied http requests are even more important than
they are now. Many of us have already dealt with apps that immediately
execute a makeRequest at load time already. Preload has been an OK
workaround, but it's becoming more and more obvious that this doesn't scale
so well for a lot of situations.

The first step here is going to be ensuring that our http facilities are
extremely robust. The out of the box PHP experience makes good use of curl,
which is probably good enough, but the base Java fetcher just plain sucks.
Paul Lindner had a great implementation of HttpFetcher using HttpClient 4.
At the time, I wanted to use it but we got a version conflict with Abdera
and gave up (we have our own proprietary solution for use at Google).

What I'd like to do is make sure that we have a standard, well-documented
http policy. Ultimately, that policy would be fed back into / aligned with
eventual standardization in the specification. This includes simple things
like adhering to HTTP caching rules as well as more esoteric issues, such as
persistent connections and pipelining.

This is really a time for brainstorming, since I don't expect any "real"
action to be taken on the specification for at least a few months, but I
want to be sure that we're getting things right.

Re: Improving out of the box http facilities to deal with changing demands.

[Leonardo Foderaro <st...@gmail.com>]

I don't know if it can be helpful / interesting, but my experiment
with the c-based lib for gadget handling seems to work fine with http
fetches (especially now that it uses libevent and curl in multi mode)
I've been busy these days with my job.. now I'm writing a simple
apache module to wrap it, just to see how it could looks like.
I'll let you know how it runs, maybe with some stress test.

thanks
leo


On Mon, Jun 30, 2008 at 7:07 PM, Louis Ryan <lr...@google.com> wrote:
> I think we can use HttpClient 4 if we really want BTW. Either by creating
> Classloaders (yuk!), repackaging HTTP client to avoid name conflicts
> (hopefully much simpler but would require some Maven fiddling) or just use
> the Abdera version which is still better than anything in java.*
>
> On Mon, Jun 30, 2008 at 9:20 AM, Brian Eaton <be...@google.com> wrote:
>
>> On Sat, Jun 28, 2008 at 3:00 PM, Kevin Brown <et...@google.com> wrote:
>> > The first step here is going to be ensuring that our http facilities are
>> > extremely robust. The out of the box PHP experience makes good use of
>> curl,
>> > which is probably good enough, but the base Java fetcher just plain
>> sucks.
>> > Paul Lindner had a great implementation of HttpFetcher using HttpClient
>> 4.
>> > At the time, I wanted to use it but we got a version conflict with Abdera
>> > and gave up (we have our own proprietary solution for use at Google).
>>
>> I suspect the normal mode of operation for the Shindig HTTP fetcher
>> (both PHP and java) will be to use an HTTP proxy server.  We should
>> make it easy to do that, which means good testing and documentation.
>>
>> Using a proxy is a good idea because Shindig will be running on your
>> internal network, but you don't want the Shindig HTTP fetcher to be
>> able to make requests to your internal network.  The easiest way to
>> avoid that is to set up an HTTP proxy in a DMZ somewhere.
>>
>

Re: Improving out of the box http facilities to deal with changing demands.

[Kevin Brown <et...@google.com>]

On Mon, Jun 30, 2008 at 10:07 AM, Louis Ryan <lr...@google.com> wrote:

> I think we can use HttpClient 4 if we really want BTW. Either by creating
> Classloaders (yuk!), repackaging HTTP client to avoid name conflicts
> (hopefully much simpler but would require some Maven fiddling) or just use
> the Abdera version which is still better than anything in java.*


I still have a patch file (woefully out of date) to use HttpClient 3.x. We
could drop that in the short term.


>
>
> On Mon, Jun 30, 2008 at 9:20 AM, Brian Eaton <be...@google.com> wrote:
>
> > On Sat, Jun 28, 2008 at 3:00 PM, Kevin Brown <et...@google.com> wrote:
> > > The first step here is going to be ensuring that our http facilities
> are
> > > extremely robust. The out of the box PHP experience makes good use of
> > curl,
> > > which is probably good enough, but the base Java fetcher just plain
> > sucks.
> > > Paul Lindner had a great implementation of HttpFetcher using HttpClient
> > 4.
> > > At the time, I wanted to use it but we got a version conflict with
> Abdera
> > > and gave up (we have our own proprietary solution for use at Google).
> >
> > I suspect the normal mode of operation for the Shindig HTTP fetcher
> > (both PHP and java) will be to use an HTTP proxy server.  We should
> > make it easy to do that, which means good testing and documentation.
> >
> > Using a proxy is a good idea because Shindig will be running on your
> > internal network, but you don't want the Shindig HTTP fetcher to be
> > able to make requests to your internal network.  The easiest way to
> > avoid that is to set up an HTTP proxy in a DMZ somewhere.
> >
>

Re: Improving out of the box http facilities to deal with changing demands.

[Louis Ryan <lr...@google.com>]

I think we can use HttpClient 4 if we really want BTW. Either by creating
Classloaders (yuk!), repackaging HTTP client to avoid name conflicts
(hopefully much simpler but would require some Maven fiddling) or just use
the Abdera version which is still better than anything in java.*

On Mon, Jun 30, 2008 at 9:20 AM, Brian Eaton <be...@google.com> wrote:

> On Sat, Jun 28, 2008 at 3:00 PM, Kevin Brown <et...@google.com> wrote:
> > The first step here is going to be ensuring that our http facilities are
> > extremely robust. The out of the box PHP experience makes good use of
> curl,
> > which is probably good enough, but the base Java fetcher just plain
> sucks.
> > Paul Lindner had a great implementation of HttpFetcher using HttpClient
> 4.
> > At the time, I wanted to use it but we got a version conflict with Abdera
> > and gave up (we have our own proprietary solution for use at Google).
>
> I suspect the normal mode of operation for the Shindig HTTP fetcher
> (both PHP and java) will be to use an HTTP proxy server.  We should
> make it easy to do that, which means good testing and documentation.
>
> Using a proxy is a good idea because Shindig will be running on your
> internal network, but you don't want the Shindig HTTP fetcher to be
> able to make requests to your internal network.  The easiest way to
> avoid that is to set up an HTTP proxy in a DMZ somewhere.
>

Re: Improving out of the box http facilities to deal with changing demands.

[Chris Chabot <ch...@xs4all.nl>]

The PHP version already supports using a proxy, however one thing to  
keep in mind is that an external proxy server has no clue about the  
the refreshInterval=<time> param in makeRequest :)

	-- Chris

On Jun 30, 2008, at 6:20 PM, Brian Eaton wrote:

> I suspect the normal mode of operation for the Shindig HTTP fetcher
> (both PHP and java) will be to use an HTTP proxy server.  We should
> make it easy to do that, which means good testing and documentation.
>
> Using a proxy is a good idea because Shindig will be running on your
> internal network, but you don't want the Shindig HTTP fetcher to be
> able to make requests to your internal network.  The easiest way to
> avoid that is to set up an HTTP proxy in a DMZ somewhere.

Re: Improving out of the box http facilities to deal with changing demands.

[Brian Eaton <be...@google.com>]

On Sat, Jun 28, 2008 at 3:00 PM, Kevin Brown <et...@google.com> wrote:
> The first step here is going to be ensuring that our http facilities are
> extremely robust. The out of the box PHP experience makes good use of curl,
> which is probably good enough, but the base Java fetcher just plain sucks.
> Paul Lindner had a great implementation of HttpFetcher using HttpClient 4.
> At the time, I wanted to use it but we got a version conflict with Abdera
> and gave up (we have our own proprietary solution for use at Google).

I suspect the normal mode of operation for the Shindig HTTP fetcher
(both PHP and java) will be to use an HTTP proxy server.  We should
make it easy to do that, which means good testing and documentation.

Using a proxy is a good idea because Shindig will be running on your
internal network, but you don't want the Shindig HTTP fetcher to be
able to make requests to your internal network.  The easiest way to
avoid that is to set up an HTTP proxy in a DMZ somewhere.