You are viewing a plain text version of this content. The canonical link for it is here.
Posted to rivet-dev@tcl.apache.org by Massimo Manghi <ma...@unipr.it> on 2011/01/24 01:09:21 UTC
master interpreter
The code in branches/master-interp is now holding the code I modified with the
master interpreter creation and initialization through the undocumented
ServerInitScript directive. Children interps are supposed to be clones of the
master interp and shouldn't need further explicit additional initialization
(didn't test it in a complex environment though)
-- Massimo
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: master interpreter
Posted by Damon Courtney <da...@tclhome.com>.
Freakin' awesome! Good work, Massimo, and thank you. Looks like the giant may awaken for a bit after all. 0-]
D
On Jan 25, 2011, at 12:40 PM, Karl Lehenbauer wrote:
> You are correct, Massimo. We have eight cores, 200 children, and before
> the ServerInitScript patch the machine had a very high load average for
> multiple minutes after a graceful. With the patch, startup time for 200
> children to a calm, quiet system is five seconds. So yeah, it is a 98%
> reduction or, said another way, more than 50X faster.
>
> Also since the packages all get loaded into the parent, the forked
> children share all the VM pages containing procs, arrays, long strings,
> etc, with copy-on-write keeping any changes from being shared. This has
> reduced the memory footprint by tens of megs per child, another big win
> for ServerInitScript.
>
> Massimo's patch is in production at FlightAware and has successfully
> served more than a million pages across several webservers, with no
> errors. I consider the patch fully tested at this point, and it can be
> merged into the mainline any time -- it's good to go.
>
> Karl
>
> On 1/25/11 11:54 AM, "Massimo Manghi" <ma...@unipr.it> wrote:
>
>> Wow, if I understand your estimation on the reduction of time needed to
>> start up the server was correct.
>>
>> How many processors do you have on this machine? The time needed to
>> merely launch the 150 children had to be >(12*150)/(num of processors)
>> seconds, since each of them took at least 12 secs of CPU time (not
>> counting other time spent in waiting state). On a machine hosting 4
>> processors it means around 4-5 minutes which is in agreement with what
>> you reported. Perhaps my reckoning is over simplified, it would mean the
>> time was cut by a 98%....
>>
>> -- Massimo
>>
>> On 01/25/2011 08:45 AM, Karl Lehenbauer wrote:
>>> Massimo et al,
>>>
>>> I got the master-interp branch of Rivet built and running on a
>>> near-production next-generation webserver running FreeBSD
>>> 8.2-PRERELEASE. Moving the loading of the hundreds of packages we are
>>> doing in the ChildInitScript into the ServerInitScript cut the apachectl
>>> graceful time to launch 150 children to about five seconds of realtime
>>> from start to the system being stable and idle and ready to handle
>>> requests. And it works. Each child process shows 0 CPU seconds, as
>>> opposed to about 12 CPU seconds having each child do all the
>>> initialization separately (and mostly system CPU, too).
>>>
>>> We had some hacks in there to randomly delay Apache child startup, too,
>>> because it made the whole thing faster by reducing contention. That's
>>> off in test and we won't need it anymore.
>>>
>>> If all goes well we'll try it tomorrow in production and if it works on
>>> one of the webservers, we'll push it to all of them.
>>>
>>> Karl
>>>
>>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
> For additional commands, e-mail: rivet-dev-help@tcl.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: master interpreter
Posted by Karl Lehenbauer <ka...@gmail.com>.
You are correct, Massimo. We have eight cores, 200 children, and before
the ServerInitScript patch the machine had a very high load average for
multiple minutes after a graceful. With the patch, startup time for 200
children to a calm, quiet system is five seconds. So yeah, it is a 98%
reduction or, said another way, more than 50X faster.
Also since the packages all get loaded into the parent, the forked
children share all the VM pages containing procs, arrays, long strings,
etc, with copy-on-write keeping any changes from being shared. This has
reduced the memory footprint by tens of megs per child, another big win
for ServerInitScript.
Massimo's patch is in production at FlightAware and has successfully
served more than a million pages across several webservers, with no
errors. I consider the patch fully tested at this point, and it can be
merged into the mainline any time -- it's good to go.
Karl
On 1/25/11 11:54 AM, "Massimo Manghi" <ma...@unipr.it> wrote:
>Wow, if I understand your estimation on the reduction of time needed to
>start up the server was correct.
>
>How many processors do you have on this machine? The time needed to
>merely launch the 150 children had to be >(12*150)/(num of processors)
>seconds, since each of them took at least 12 secs of CPU time (not
>counting other time spent in waiting state). On a machine hosting 4
>processors it means around 4-5 minutes which is in agreement with what
>you reported. Perhaps my reckoning is over simplified, it would mean the
>time was cut by a 98%....
>
> -- Massimo
>
>On 01/25/2011 08:45 AM, Karl Lehenbauer wrote:
>> Massimo et al,
>>
>> I got the master-interp branch of Rivet built and running on a
>>near-production next-generation webserver running FreeBSD
>>8.2-PRERELEASE. Moving the loading of the hundreds of packages we are
>>doing in the ChildInitScript into the ServerInitScript cut the apachectl
>>graceful time to launch 150 children to about five seconds of realtime
>>from start to the system being stable and idle and ready to handle
>>requests. And it works. Each child process shows 0 CPU seconds, as
>>opposed to about 12 CPU seconds having each child do all the
>>initialization separately (and mostly system CPU, too).
>>
>> We had some hacks in there to randomly delay Apache child startup, too,
>>because it made the whole thing faster by reducing contention. That's
>>off in test and we won't need it anymore.
>>
>> If all goes well we'll try it tomorrow in production and if it works on
>>one of the webservers, we'll push it to all of them.
>>
>> Karl
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: master interpreter
Posted by Massimo Manghi <ma...@unipr.it>.
Wow, if I understand your estimation on the reduction of time needed to
start up the server was correct.
How many processors do you have on this machine? The time needed to
merely launch the 150 children had to be >(12*150)/(num of processors)
seconds, since each of them took at least 12 secs of CPU time (not
counting other time spent in waiting state). On a machine hosting 4
processors it means around 4-5 minutes which is in agreement with what
you reported. Perhaps my reckoning is over simplified, it would mean the
time was cut by a 98%....
-- Massimo
On 01/25/2011 08:45 AM, Karl Lehenbauer wrote:
> Massimo et al,
>
> I got the master-interp branch of Rivet built and running on a near-production next-generation webserver running FreeBSD 8.2-PRERELEASE. Moving the loading of the hundreds of packages we are doing in the ChildInitScript into the ServerInitScript cut the apachectl graceful time to launch 150 children to about five seconds of realtime from start to the system being stable and idle and ready to handle requests. And it works. Each child process shows 0 CPU seconds, as opposed to about 12 CPU seconds having each child do all the initialization separately (and mostly system CPU, too).
>
> We had some hacks in there to randomly delay Apache child startup, too, because it made the whole thing faster by reducing contention. That's off in test and we won't need it anymore.
>
> If all goes well we'll try it tomorrow in production and if it works on one of the webservers, we'll push it to all of them.
>
> Karl
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: master interpreter
Posted by Karl Lehenbauer <ka...@gmail.com>.
Massimo et al,
I got the master-interp branch of Rivet built and running on a near-production next-generation webserver running FreeBSD 8.2-PRERELEASE. Moving the loading of the hundreds of packages we are doing in the ChildInitScript into the ServerInitScript cut the apachectl graceful time to launch 150 children to about five seconds of realtime from start to the system being stable and idle and ready to handle requests. And it works. Each child process shows 0 CPU seconds, as opposed to about 12 CPU seconds having each child do all the initialization separately (and mostly system CPU, too).
We had some hacks in there to randomly delay Apache child startup, too, because it made the whole thing faster by reducing contention. That's off in test and we won't need it anymore.
If all goes well we'll try it tomorrow in production and if it works on one of the webservers, we'll push it to all of them.
Karl
On Jan 23, 2011, at 6:09 PM, Massimo Manghi wrote:
> The code in branches/master-interp is now holding the code I modified with the
> master interpreter creation and initialization through the undocumented
> ServerInitScript directive. Children interps are supposed to be clones of the
> master interp and shouldn't need further explicit additional initialization
> (didn't test it in a complex environment though)
>
> -- Massimo
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
> For additional commands, e-mail: rivet-dev-help@tcl.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: master interpreter
Posted by Karl Lehenbauer <ka...@gmail.com>.
On Jan 23, 2011, at 6:09 PM, Massimo Manghi wrote:
> The code in branches/master-interp is now holding the code I modified with the
> master interpreter creation and initialization through the undocumented
> ServerInitScript directive. Children interps are supposed to be clones of the
> master interp and shouldn't need further explicit additional initialization
> (didn't test it in a complex environment though)
Hello Massimo,
This is fantastic news. We will test it, starting tomorrow. I am really excited about this.
Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org