200 vhosts, 5 sets of libs, 2GB RAM, 1 httpd (seeking advice)

[Chris Hagglund <ch...@shrinkingplanet.ca>]

Hi all,

I'm having trouble figuring out the right mix of configuration settings 
to do what I want, which is to have 5 separate sets libraries (DEV, 
TEST, PROD, developer1_personal_dev, developer2_personal_dev) get shared 
by a large number of vhosts. The way I am doing it now is each vhost 
Include's the appropriate mod_perl config file for whatever set of libs 
it needs to be on. The Include'd files each will do PerlOptions +Parent 
and PerlSwitches -Mlib=/path/to/specific/libs and then load a bunch of 
PerlModules from those libs.

The trouble is, as each vhost loads this config file for its plain and 
SSL vhost directives, the amount of memory being used seems to be going 
up, and restarting apache is taking longer and longer. Intead of loading 
it over and over again (twice for vhosts with ssl!) I am hoping to find 
a way to tell apache to load PROD mp configuration, then load 100 PROD 
vhosts that will all share it, then load TEST mp configuration, then 
load 20 vhosts that use those libs, etc, but I am getting error like: 
'mod_perl is already running, too late for PerlSwitches' when I am 
trying to start setting this up.

Am I doing something wrong, or is there an esablished best-practice for 
doing this kind of thing?

Thanks for any information.

Regards,
Chris Hagglund.

Re: 200 vhosts, 5 sets of libs, 2GB RAM, 1 httpd (seeking advice)

[Robert Landrum <rl...@aol.net>]

Chris Hagglund wrote:
> The trouble is, as each vhost loads this config file for its plain and 
> SSL vhost directives, the amount of memory being used seems to be going 
> up, and restarting apache is taking longer and longer. Intead of loading 
> it over and over again (twice for vhosts with ssl!) I am hoping to find 
> a way to tell apache to load PROD mp configuration, then load 100 PROD 
> vhosts that will all share it, then load TEST mp configuration, then 
> load 20 vhosts that use those libs, etc, but I am getting error like: 
> 'mod_perl is already running, too late for PerlSwitches' when I am 
> trying to start setting this up.

It sounds like you want a two tier architecture.

The front end will proxy a set of mod_perl servers running behind it. 
The front end will be configured to point to either your TEST, DEV, 
PROD, dev1, or dev2 apache instances.

Each apache instance will only load one set of modules.

It should reduce the amount of memory being used substantially and speed 
up apache restarts.

Rob

Re: 200 vhosts, 5 sets of libs, 2GB RAM, 1 httpd (seeking advice)

[Chris Hagglund <ch...@shrinkingplanet.ca>]

Thanks everyone for the suggestions. I appreciate it. Will ponder the 
options.

Regards,
Chris Hagglund.


Perrin Harkins wrote:
> On 4/11/07, Jonathan Vanasco <jv...@2xlp.com> wrote:
>> Are you aware of the dangers in namespace collision this can cause?
>
> I'd guess that he is, and that's why uses PerlOptions +Parent.  See
> http://perl.apache.org/docs/2.0/user/config/config.html#C_Parent_
>
> The problem with that strategy is that it means your 100 PROD vhosts
> don't share anything.  The idea to set up the interpreter once for all
> the PROD vhosts is a good one, but I'm not aware of any context level
> in between vhost and global, so I don't think it will work.
>
> The proxy idea will work, and is not all that hard really.  You could
> probably even rig the servers to all share the same config file and
> just have them choose which block to follow by passing -D options on
> the command line.
>
> - Perrin
>

Re: 200 vhosts, 5 sets of libs, 2GB RAM, 1 httpd (seeking advice)

[Perrin Harkins <pe...@elem.com>]

On 4/11/07, Jonathan Vanasco <jv...@2xlp.com> wrote:
> Are you aware of the dangers in namespace collision this can cause?

I'd guess that he is, and that's why uses PerlOptions +Parent.  See
http://perl.apache.org/docs/2.0/user/config/config.html#C_Parent_

The problem with that strategy is that it means your 100 PROD vhosts
don't share anything.  The idea to set up the interpreter once for all
the PROD vhosts is a good one, but I'm not aware of any context level
in between vhost and global, so I don't think it will work.

The proxy idea will work, and is not all that hard really.  You could
probably even rig the servers to all share the same config file and
just have them choose which block to follow by passing -D options on
the command line.

- Perrin

Re: 200 vhosts, 5 sets of libs, 2GB RAM, 1 httpd (seeking advice)

[Jonathan Vanasco <jv...@2xlp.com>]

On Apr 11, 2007, at 11:23 AM, Chris Hagglund wrote:

> I'm having trouble figuring out the right mix of configuration  
> settings to do what I want, which is to have 5 separate sets  
> libraries (DEV, TEST, PROD, developer1_personal_dev,  
> developer2_personal_dev) get shared by a large number of vhosts.  
> The way I am doing it now is each vhost Include's the appropriate  
> mod_perl config file for whatever set of libs it needs to be on.  
> The Include'd files each will do PerlOptions +Parent and  
> PerlSwitches -Mlib=/path/to/specific/libs and then load a bunch of  
> PerlModules from those libs.

Are you aware of the dangers in namespace collision this can cause?   
You're going to run into problems when DEV & PROD have different  
versions of the same module -- only one will load.

> The trouble is, as each vhost loads this config file for its plain  
> and SSL vhost directives, the amount of memory being used seems to  
> be going up, and restarting apache is taking longer and longer.
never restart apache.  stop it, then start it.  but never restart it  
( ie : apachectl graceful || apachectl restart )


> Intead of loading it over and over again (twice for vhosts with  
> ssl!) I am hoping to find a way to tell apache to load PROD mp  
> configuration, then load 100 PROD vhosts that will all share it,  
> then load TEST mp configuration, then load 20 vhosts that use those  
> libs, etc, but I am getting error like: 'mod_perl is already  
> running, too late for PerlSwitches' when I am trying to start  
> setting this up.
>
> Am I doing something wrong, or is there an esablished best-practice  
> for doing this kind of thing?

I think you're best off with running something on port 80 -- either  
apache / squid / nginx  that handles proxying to multiple apache 
+mod_perl backends

1 for prod, 1 for test, 1 for dev etc

Con:
	you're going to use more memory doing that -- each apache  will   
load its own version of the common libraries.
Pro:
	a- right now your entire system can fail because of a bug in 1  
module.  if someone has a serious bug in dev or or personal_dev, they  
can bring down everything
	b- you can better balance your resources:  give the testing/dev  
stuff a short max memory and run fewer instances -- free up memory  
for the production system.  right now your system is blocking  
production resources for 4 versions of testing systems.  thats kind  
of insane.
	c- for dev stuff, you're often better using 1 or two httpds MAX.   
its damn crazy trying to debug otherwise.


// Jonathan Vanasco

| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - -