forking/subprocess/etc

[Jonathan <mo...@2xlp.com>]

Right now in a modperl app, I have this functionality:

	mod-perl request 1:
		User requests a contact list to be imported from the internet,
		mp logs a 'request' to a database, redirects to status page

	python daemon
		continually checks database for import requests, performs import,  
cleans up requests

	mod-perl request 2+
		checks for import request status, reload every 20s

That  works perfect for most of my stuff...

however, a new function I've just worked out has a rather lengthy  
logic system and series of Regex calls that are already written in Perl.
Its not like a lot of the admin/management stuff in python, and i  
dont feel like converting stuff today

so i need to get this handled within mod_perl somehow, and am a bit  
uneasy on how to handle this

Possible Solutions:
	handle import within mod_perl
		No.
			Import can take 5-200s, as it requires downloading multiple URLs.   
That blocks apache.
	fork() within mod_perl
		Probable No.
  			online docs suggest this will fork apache+modperl, which would  
take up too much memory
			child processes will persist as zombies when complete

i've got to be missing something

anyone have a clue?  the archives on the list aren't very helpful.



// Jonathan Vanasco

| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  
- - - - - - - - - - - - - - - -

Re: forking/subprocess/etc

[Perrin Harkins <pe...@elem.com>]

On Tue, 2006-09-05 at 17:50 -0400, Jonathan Vanasco wrote:
> > Incidentally, I don't see why your Python job queue can't just  
> > execute a Perl script to do this job.
> 
> didn't really think of that :)
> 
> although if i did that, i'd have to do some crazy PID watching thing,  
> to make sure I don't have too many jobs running at once...

In my Perl job queue system, we just limit the number of processes it
will fork using Parallel::ForkManager.

- Perrin

Re: forking/subprocess/etc

[Jonathan Vanasco <mo...@2xlp.com>]

On Sep 5, 2006, at 4:14 PM, Perrin Harkins wrote:

> The memory would be shared by CoW and I haven't seen that zombie
> problem.

yeah, but apache instances seem to take up 5-10mb of memory per- 
instance.  i think 3 is for apache itself, and the rest is various  
stuff.  10 requests to this page within 1 minute of each other could  
wipe out 100mb of ram.


> I usually handle this kind of thing with a job queue system
> though, i.e. the same way you did with your Python thing.

i ended up just converting / writing to python. but just as good-  
flipping between the two languages helps me refactor and optimize  
code  -- and clarity -- as i have to thing about solutions slightly  
differently.

> Incidentally, I don't see why your Python job queue can't just  
> execute a Perl script to do this job.

didn't really think of that :)

although if i did that, i'd have to do some crazy PID watching thing,  
to make sure I don't have too many jobs running at once...

i ended up just hacking together a quick daemon using Twisted Python-  
took about 4 hours to get running (including perl conversion, which  
netted me a bunch of regex optimizations while i was at it!)   
( granted I had another one running so it was easy to transition this )

	startup-
		clear locks ,
		create job queue
	every 5 seconds-
		poll db for requests, create locks on requests, transfer to the job  
queue
		run the job queue

there's some stuff in there to manage processes , but its working as  
I expected.

Re: forking/subprocess/etc

[Perrin Harkins <pe...@elem.com>]

On Mon, 2006-09-04 at 18:55 -0400, Jonathan wrote:
> 	fork() within mod_perl
> 		Probable No.
>   			online docs suggest this will fork apache+modperl, which would  
> take up too much memory
> 			child processes will persist as zombies when complete
> 
> i've got to be missing something

The memory would be shared by CoW and I haven't seen that zombie
problem.  I usually handle this kind of thing with a job queue system
though, i.e. the same way you did with your Python thing.  Incidentally,
I don't see why your Python job queue can't just execute a Perl script
to do this job.

- Perrin