[ANNOUNCE] Apache::AppCluster 0.2

[Mark Maunder <ma...@swiftcamel.com>]

Hi all,

I'm about to post this module to CPAN. Please take a look and let me
know if you think this is appropriate for the Apache::* namespace and if
you have any problems with it ('make test' is quite comprehensive).

The module is available from:
http://www.swiftcamel.com/modules/Apache-AppCluster-0.02.tar.gz

~mark

Here is the readme:

Apache::AppCluster is a lightweight mod_perl RPC mechanism that allows
you to
use your mod_perl web servers as distributed application servers that
serve
multiple concurrent RPC requests to remote clients across a network. The
client
component has the ability to fire off multiple simultaneous requests to
multiple remote application servers and collect the responses
simultaneously.

This is similar to SOAP::Lite in that it is a web based RPC mechanism,
but
it has the advantage of being able to send/receive multiple concurrent
requests
to the same or different remote application servers and the
methods/functions
called on the remote servers may receive and return Perl data structures
of
arbritary complexity - entire objects can be flung back and forth with
ease.

Please see Apache::AppCluster::Client and Apache::AppCluster::Server
documentation for full details on server configuration (very easy) and
Client
usage (OO interface).

INSTALLATION:

Untar the distribution into a directory that will be readable by the
user
nobody. (i.e. dont use /root for installation). The test suite runs a
web server on port 8228 and this runs as user nobody.

As per usual do the following:

perl Makefile.PL
make
make test
make install


If you run into problems during the 'make test' stage, please email me
the
error log which is at: Server/t/error_log. Also include the last few
lines of 'make test' output.

APACHE CONFIG:

The documentation for Apache::AppCluster::Server contains everything
you'll
need to set up the server component. The only thing to keep in mind is
that
if you are going to be sending multiple concurrent requests from the
client
to an apache server, make sure the server is set up to handle the load.
Do this by setting MaxClients, StartServers, MinSpareServers and
MaxSpareServers.
If you're going to be hitting it with 20 concurrent requests, make sure
there
are 20 child servers standing by to handle your requests.

Re: [ANNOUNCE] Apache::AppCluster 0.2

[Mark Maunder <ma...@swiftcamel.com>]

brian moseley wrote:

> On Wed, 9 Jan 2002, Mark Maunder wrote:
>
> > Well, I guess two methods could be added to the client
> > object. One to add a concurrent request to be called
> > (register_request()) and one to send all registered
> > requests concurrently. I'm not the author though, so
> > you'll have to chat to Jochen about that.
>
> couldn't you just subclass RPC::PlClient?

The transport is different (HTTP/POST) and I dont think I can easily
just drop in another (alternative) transport - I may as well rewrite.

> > The server and transport would have to be rewritten
> > pretty much from scratch I think. The transport needs to
> > be HTTP POST requests and responses. The server needs to
> > be set up as a mod_perl handler that takes advantage of
> > everything mod_perl has to offer.
>
> why "needs"? i'm sure lots of people would rather run a very
> lightweight non-http/apache-based server.
>

Agreed. Are there any more besides a standalone pure perl daemon and
mod_perl/apache?

>
> > I dont really mind whether we incorporate this stuff
> > into PlClient or AppCluster or both, but I do think that
> > both the concurrency in AppCluster and tied object API
> > in PlRPC are really useful and would be even better with
> > the remote app server being mod_perl.
>
> seems like the ideal api gives you the best functionality
> from both original apis and abstracts away the choice of
> transport and server.
>

yeah - agreed.

>
> > An idea might be to incorporate both the AppCluster
> > concurrency and PlRPC style api into an Apache::* module
> > that gives us the best of both worlds with mod_perl
> > performance (etc.) on the server side. (and then get rid
> > of AppCluster since it will be redundant)
>
> perhaps i misunderstand, but you're suggesting making the
> client an Apache module? why?

Well, the server component (at present) is a mod_perl handler, and I
wanted to bundle both together so I stuck in in the Apache namespace
(pending any objections of course). Seems like RPC might make more sense
if it becomes platform/server neutral, since Apache::* binds the server
platform to mod_perl.

> i like the idea of being able to write client code that uses
> the same rpc api no matter whether i choose to use soap,
> xml-rpc, a more specific http post, plrpc's transport
> (whatever that might be), or whatever as the transport. not
> all of the "drivers" would have to support the same feature
> set (i think your mechanism supports arbitrarily deep data
> structures?).
>
> that rpc api is one of the things p5ee is looking for, i
> believe.

It seems like you're asking a bit much here (the holy grail of RPC?).
SOAP and xml-rpc are their own client/server system. Are we going to
integrate this with both of them and support standalone perl daemons
etc.? I considered writing a soap client that allows for concurrent
requests, but I found SOAP::Lite to be slow under mod_perl so opted for
rolling my own instead. Also SOAP is platform neutral and I'm not sure,
but I think it wont allow for perl data structures as complex as
Storable does.

I think you should probably distinguish between transport and encoding.
Transports could be http, https or a plain old socket (if we're talking
to a perl daemon) and an encrypted socket connection using CBC.
Encodings we've chatted about are Storable's freeze/thaw and SOAP's XML,
and then xml-rpc (which I assume has it's own encoding of method name
and params etc in xml - although I dont know it at all). I think having
various transports is fine. But I'm not sure what the motivation is for
varying the encodings, unless we want our client to talk to SOAP and
xml-rpc servers, or vice versa.

Perhaps integrating PlRPC and AppCluster's client API's and allowing for
either standalone daemon or mod_perl server is a good start? We can use
HTTP, HTTPS, direct sockets and encrypted sockets as the first
transports. The client can have two modes - concurrent remote procedure
calls, or creating a copy of the remote object PlRPC style.

~mark.

Re: [ANNOUNCE] Apache::AppCluster 0.2

[brian moseley <bc...@maz.org>]

On Wed, 9 Jan 2002, Mark Maunder wrote:

> Well, I guess two methods could be added to the client
> object. One to add a concurrent request to be called
> (register_request()) and one to send all registered
> requests concurrently. I'm not the author though, so
> you'll have to chat to Jochen about that.

couldn't you just subclass RPC::PlClient?

> The server and transport would have to be rewritten
> pretty much from scratch I think. The transport needs to
> be HTTP POST requests and responses. The server needs to
> be set up as a mod_perl handler that takes advantage of
> everything mod_perl has to offer.

why "needs"? i'm sure lots of people would rather run a very
lightweight non-http/apache-based server.

> I dont really mind whether we incorporate this stuff
> into PlClient or AppCluster or both, but I do think that
> both the concurrency in AppCluster and tied object API
> in PlRPC are really useful and would be even better with
> the remote app server being mod_perl.

seems like the ideal api gives you the best functionality
from both original apis and abstracts away the choice of
transport and server.

> An idea might be to incorporate both the AppCluster
> concurrency and PlRPC style api into an Apache::* module
> that gives us the best of both worlds with mod_perl
> performance (etc.) on the server side. (and then get rid
> of AppCluster since it will be redundant)

perhaps i misunderstand, but you're suggesting making the
client an Apache module? why?

i like the idea of being able to write client code that uses
the same rpc api no matter whether i choose to use soap,
xml-rpc, a more specific http post, plrpc's transport
(whatever that might be), or whatever as the transport. not
all of the "drivers" would have to support the same feature
set (i think your mechanism supports arbitrarily deep data
structures?).

that rpc api is one of the things p5ee is looking for, i
believe.

Re: [ANNOUNCE] Apache::AppCluster 0.2

[Mark Maunder <ma...@swiftcamel.com>]

brian moseley wrote:

> On Wed, 9 Jan 2002, Mark Maunder wrote:
>
> > The difference is that AppCluster allows you to call
> > multiple remote methods on the server (or on multiple
> > distributed servers) simultaneously. However, I wasn't
> > aware of PlRPC and I really like the interface i.e. the
> > way it creates a copy of the remote object locally and
> > allows you to call methods on it as if it were just
> > another object.
>
> would it require too much surgery and api change if you
> added the concurrency support to PlRPC?

Well, I guess two methods could be added to the client object. One to
add a concurrent request to be called (register_request()) and one to
send all registered requests concurrently. I'm not the author though, so
you'll have to chat to Jochen about that.

The server and transport would have to be rewritten pretty much from
scratch I think. The transport needs to be HTTP POST requests and
responses. The server needs to be set up as a mod_perl handler that
takes advantage of everything mod_perl has to offer.

>From my point of view, it's easier to duplicate Jochen's work in
AppCluster by adding the same type of interface i.e. creating a copy of
a remote object locally and calling methods on that object as per normal
while the actual method call is submitted via a POST to a remote
mod_perl app server.

I dont really mind whether we incorporate this stuff into PlClient or
AppCluster or both, but I do think that both the concurrency in
AppCluster and tied object API in PlRPC are really useful and would be
even better with the remote app server being mod_perl.

An idea might be to incorporate both the AppCluster concurrency and
PlRPC style api  into an Apache::* module that gives us the best of both
worlds with mod_perl performance (etc.) on the server side. (and then
get rid of AppCluster since it will be redundant)

Let me know if that sounds like a good idea and I'll start work on it.
Perhaps we could call it Apache::PlRPC (now with added concurrency!)

~mark

Re: [ANNOUNCE] Apache::AppCluster 0.2

[brian moseley <bc...@maz.org>]

On Wed, 9 Jan 2002, Mark Maunder wrote:

> The difference is that AppCluster allows you to call
> multiple remote methods on the server (or on multiple
> distributed servers) simultaneously. However, I wasn't
> aware of PlRPC and I really like the interface i.e. the
> way it creates a copy of the remote object locally and
> allows you to call methods on it as if it were just
> another object.

would it require too much surgery and api change if you
added the concurrency support to PlRPC?

Re: [ANNOUNCE] Apache::AppCluster 0.2

[Mark Maunder <ma...@swiftcamel.com>]

Gunther Birznieks wrote:

> Is this a lot different from PlRPC?
>
> Perhaps this should be a transport layer that is added to PlRPC rather than
> creating a brand-new service?
>
> Ideally it would be nice if it were architected with mod_perl and HTTP
> being the default mechanism of transport but that if you wanted to, you
> could make it into a standalone Perl daemon that forked by itself.

The difference is that AppCluster allows you to call multiple remote methods on
the server (or on multiple distributed servers) simultaneously. However, I wasn't
aware of PlRPC and I really like the interface i.e. the way it creates a copy of
the remote object locally and allows you to call methods on it as if it were just
another object.

I could not do this with AppCluster without sacrificing the concurrency it
offers. At present you create a single appcluster object and then register the
method calls you would like it to call, one by one. You then call a single method
(with a timeout) that calls all registered remote methods simultaneously. (It
uses non-blocking IO and so provides concurrency in a single thread of execution)

re: allowing the server to be a standalone daemon that forks by itself.
This project actually started exactly like that. I grabbed the pre-forking server
example from the Cookbook and used that as the basis for the server. I found that
performance was horrible though, because forking additional servers took too long
and I also was using DBI and I missed Apache::DBI (and all the other great
apache::* mods) too much. So I used a good solid server that offers a great Perl
persistence engine. I'm not sure why anyone would want to roll their own server
in Perl. If there is a reason, then I could change the server class to create a
server abstraction layer of some kind.

~mark.

Re: [ANNOUNCE] Apache::AppCluster 0.2

[Gunther Birznieks <gu...@extropia.com>]

Is this a lot different from PlRPC?

Perhaps this should be a transport layer that is added to PlRPC rather than 
creating a brand-new service?

Ideally it would be nice if it were architected with mod_perl and HTTP 
being the default mechanism of transport but that if you wanted to, you 
could make it into a standalone Perl daemon that forked by itself.

Later,
    Gunther

At 09:23 AM 1/9/2002, Mark Maunder wrote:
>Apache::AppCluster is now in CPAN and can be accessed at:
>http://search.cpan.org/search?dist=Apache-AppCluster
>
>This consists of a client and server module that can be used to develop
>mod_perl clustered web services. A client application can make multiple
>simultaneous API calls to a mod_perl server (or multiple servers) passing
>and receiving any perl reference or object as parameters. The server module
>runs the called module::method in an eval loop and handles errors gracefully
>while taking advantage of the persistence offered by mod_perl. The client
>has a bunch of useful methods and includes a timeout for the total time
>allowed for all remote calls to complete.
>
>Let me know if you have any suggestions or find this useful. (or find any
>bugs)
>
>~mark
>
>*snip*
>The uploaded file
>
>     Apache-AppCluster-0.02.tar.gz
>
>has entered CPAN as
>
>   file: $CPAN/authors/id/M/MA/MAUNDER/Apache-AppCluster-0.02.tar.gz
>   size: 23995 bytes
>    md5: 0abff0a4a2aa053c9f8ae3c00dd86434
>
>No action is required on your part
>Request entered by: MAUNDER (Mark D. Maunder)
>Request entered on: Sun, 06 Jan 2002 16:31:57 GMT
>Request completed:  Sun, 06 Jan 2002 16:33:33 GMT

__________________________________________________
Gunther Birznieks (gunther.birznieks@eXtropia.com)
eXtropia - The Open Web Technology Company
http://www.eXtropia.com/

Re: [ANNOUNCE] Apache::AppCluster 0.2

[Mark Maunder <ma...@swiftcamel.com>]

Apache::AppCluster is now in CPAN and can be accessed at:
http://search.cpan.org/search?dist=Apache-AppCluster

This consists of a client and server module that can be used to develop
mod_perl clustered web services. A client application can make multiple
simultaneous API calls to a mod_perl server (or multiple servers) passing
and receiving any perl reference or object as parameters. The server module
runs the called module::method in an eval loop and handles errors gracefully
while taking advantage of the persistence offered by mod_perl. The client
has a bunch of useful methods and includes a timeout for the total time
allowed for all remote calls to complete.

Let me know if you have any suggestions or find this useful. (or find any
bugs)

~mark

*snip*
The uploaded file

    Apache-AppCluster-0.02.tar.gz

has entered CPAN as

  file: $CPAN/authors/id/M/MA/MAUNDER/Apache-AppCluster-0.02.tar.gz
  size: 23995 bytes
   md5: 0abff0a4a2aa053c9f8ae3c00dd86434

No action is required on your part
Request entered by: MAUNDER (Mark D. Maunder)
Request entered on: Sun, 06 Jan 2002 16:31:57 GMT
Request completed:  Sun, 06 Jan 2002 16:33:33 GMT