You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by James Smith <js...@sanger.ac.uk> on 2021/02/07 12:49:46 UTC
RE: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
As welsey said – try Registry, that was the standard way of using mod_perl to cache perl in the server – but your problem might be due to the note in PerlRun…
https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description
META: document that for now we don't chdir() into the script's dir, because it affects the whole process under threads. ModPerl::PerlRunPrefork<https://perl.apache.org/docs/2.0/api/ModPerl/PerlRunPrefork.html> should be used by those who run only under prefork MPM.
{tbh most people don’t use mod perl under threads anyway as there isn’t really a gain from using them}
It suggests you use ModPerl/PerlRunPrefork – as this does an additional step to cd to the script directory – which might be your issue….
From: Steven Haigh <ne...@crc.id.au>
Sent: 07 February 2021 01:00
To: modperl@perl.apache.org
Subject: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
Hi all,
So for many years I've been slack and writing perl scripts to do various things - but never needed more than the normal apache +ExecCGI and Template Toolkit.
One of my sites has become a bit more popular, so I'd like to spend a bit of time on performance. Currently, I'm seeing ~300-400ms of what I believe to be execution time of the script loading, running, and then blatting its output to STDOUT and the browser can go do its thing.
I believe most of the delay would be to do with loading perl, its modules etc etc
I know that the current trend would be to re-write the entire site in a more modern, daemon based solution - and I started down the Mojolicious path - but the amount of re-writing to save 1/3rd of a second seems to be excessive
Would I be correct in thinking that mod_perl would help in this case?
I did try a basic test, but I have a 'use functions' in all my scripts that loads a .pm with some global vars and a lot of common subs - and for whatever reason (can't find anything on Google as to why), none of the subs are recognised in the main script when loaded via ModPerl::PerlRun.
So throwing it out to the list - am I on the right track? wasting my time? or just a simple mistake?
--
Steven Haigh 📧 netwiz@crc.id.au<ma...@crc.id.au> 💻 https://www.crc.id.au [crc.id.au]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=bosoTbkecbnrPukObNK-5Duc1p3JTllIM7_FHhBYKW4&s=vQDi0ezyEZDOz86GVraerPdT76UjN2in3UdPh8fglRM&e=>
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
RE: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by James Smith <js...@sanger.ac.uk>.
The advantage of the web proxy is not from securing your app - although there are things you can do on the reverse proxy to secure less secure apps
It's main advantage is that it doesn't run a large software stack - and so it makes it harder for people to compromise your front end and then compromise your internal networks, and even then they have to get from that DMZ into your main infrastructure.
We go a step further at work. We have a DMZ <- a web zone <- internal zone - so even if you can compromise the DMZ and the web servers you still don't have direct access to the other machines - taking servers + desktop machines - something like 30-50K cores.
-----Original Message-----
From: Clive Eisen <cl...@hildebrand.co.uk>
Sent: 09 February 2021 19:23
To: Rafael Caceres <rc...@aasa.com.pe>
Cc: James Smith <js...@sanger.ac.uk>; Vincent Veyron <vv...@wanadoo.fr>; modperl@perl.apache.org
Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
> On 9 Feb 2021, at 19:16, Rafael Caceres <rc...@aasa.com.pe> wrote:
>
> Another thing that can be done is keep the app server + DB inside your LAN and place a reverse proxy on your DMZ, that adds some level of protection.
Not really - the only protection is if all your apis or web pages are secure - the reverse proxy does not help or hinder that.
—
C
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by Clive Eisen <cl...@hildebrand.co.uk>.
> On 9 Feb 2021, at 19:16, Rafael Caceres <rc...@aasa.com.pe> wrote:
>
> Another thing that can be done is keep the app server + DB inside your LAN and place a reverse proxy on your DMZ, that adds some level of protection.
Not really - the only protection is if all your apis or web pages are secure - the reverse proxy does not help or hinder that.
—
C
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by Rafael Caceres <rc...@aasa.com.pe>.
Another thing that can be done is keep the app server + DB inside your LAN and place a reverse proxy on your DMZ, that adds some level of protection.
Rafael
On Feb 9, 2021, 2:08 PM -0500, Clive Eisen <cl...@hildebrand.co.uk>, wrote:
On 9 Feb 2021, at 18:45, James Smith <js...@sanger.ac.uk> wrote:
It doesn't matter what db - and whether you wrap it in eval it is a problem (postgres has a similar problem - the one with least problems is MySQL) - if you have a secure environment where your databases are in a firewalled zone it will happen to all of them... It's a nasty bit of networking - it does mean our meant to be secure enterprise level apps running against Oracle and less secure and less stable than the other apps we have (go figure!)…
20 years ago I had exactly this argument with Amex when we wanted to use it for payment on the site I then worked for.
They said put a firewall between the app and db layerx
I said it's a dedicated nic/vlan on both sides and the ONLY port that is open is the db - what is a firewall going to add to that.
Eventually they agreed.
Security people who say firewall firewall firewall will solve all your security issues (or even some of them) are useless.
Most of them do it by the book - which should in all case just be the starting point.
Just my 2p
—
C
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by Clive Eisen <cl...@hildebrand.co.uk>.
> On 9 Feb 2021, at 18:45, James Smith <js...@sanger.ac.uk> wrote:
>
> It doesn't matter what db - and whether you wrap it in eval it is a problem (postgres has a similar problem - the one with least problems is MySQL) - if you have a secure environment where your databases are in a firewalled zone it will happen to all of them... It's a nasty bit of networking - it does mean our meant to be secure enterprise level apps running against Oracle and less secure and less stable than the other apps we have (go figure!)…
20 years ago I had exactly this argument with Amex when we wanted to use it for payment on the site I then worked for.
They said put a firewall between the app and db layerx
I said it's a dedicated nic/vlan on both sides and the ONLY port that is open is the db - what is a firewall going to add to that.
Eventually they agreed.
Security people who say firewall firewall firewall will solve all your security issues (or even some of them) are useless.
Most of them do it by the book - which should in all case just be the starting point.
Just my 2p
—
C
RE: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by James Smith <js...@sanger.ac.uk>.
It doesn't matter what db - and whether you wrap it in eval it is a problem (postgres has a similar problem - the one with least problems is MySQL) - if you have a secure environment where your databases are in a firewalled zone it will happen to all of them... It's a nasty bit of networking - it does mean our meant to be secure enterprise level apps running against Oracle and less secure and less stable than the other apps we have (go figure!)...
-----Original Message-----
From: Vincent Veyron <vv...@wanadoo.fr>
Sent: 09 February 2021 17:47
To: modperl@perl.apache.org
Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
On Sun, 7 Feb 2021 20:21:34 +0000
James Smith <js...@sanger.ac.uk> wrote:
Hi James,
> DBI sharing doesn't really gain you much - and can actually lead you into a whole world of pain. It isn't actually worth turning it on at all.
>
Never had a problem with it myself in years of using it, but I wrap my queries in an eval { } and check $@, so that the scripts are not left hanging; also I have a postgresql db ;-).
I ran some tests with ab, I do see an improvement in response speed :
my $dbh = DBI->connect()
Concurrency Level: 5
Time taken for tests: 22.198 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 8435000 bytes
HTML transferred: 8176000 bytes
Requests per second: 45.05 [#/sec] (mean)
Time per request: 110.990 [ms] (mean)
Time per request: 22.198 [ms] (mean, across all concurrent requests)
Transfer rate: 371.08 [Kbytes/sec] received
my $dbh = DBI->connect_cached()
Concurrency Level: 5
Time taken for tests: 15.133 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 8435000 bytes
HTML transferred: 8176000 bytes
Requests per second: 66.08 [#/sec] (mean)
Time per request: 75.664 [ms] (mean)
Time per request: 15.133 [ms] (mean, across all concurrent requests)
Transfer rate: 544.33 [Kbytes/sec] received
--
Bien à vous, Vincent Veyron
https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwIFAw&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=u0vYr2KXDAvFiif8YsX-7Uho_gsySe2x9Z3OHcD_Br4&s=7Hyp0l39Edk8cAZK0idIxVxKi3OQXhkR96T0T42b2tM&e=
Logiciel libre de comptabilité générale en partie double
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by Chris <cp...@bennettconstruction.us>.
On Fri, Feb 12, 2021 at 05:15:29PM +0100, André Warnier (tomcat/perl) wrote:
> My comment was just basically so as to avoid the case where someone else
> would later be searching the archives of this mailing list for information
> about DBI, and never find these (useful for DBI) posts, because DBI is not
> in the subject.
I can't disagree with that. Put up a short thread for the archives with a good
subject and content about what this thread references with some kind of link
to this thread?
Some of the good topics I really found helpful are also long threads
that wander around getting off-topic of the subject. I've saved quite a
few useful posts from those threads. Good archiving is very important.
I have found using regular web search engines to be an exercise in
useless frustration.
I'm reading up on several of the topics here. I plan to ask some more
questions about those after a bit. What is a good way to reference back
to this thread's sections that will get into the archive in a useful way?
Chris
>
> On 12.02.2021 00:51, Chris wrote:
> > On Thu, Feb 11, 2021 at 09:52:16AM +0100, André Warnier (tomcat/perl) wrote:
> > > Isn't this discussion about connection pools and firewalls etc getting a bit
> > > far from the initial subject of the thread ?
> >
> > Perhaps. But this has become a pretty low volume mailing list.
> > This "thread" has moved me to spend hours looking at changing and/or
> > better understanding the work I have done (pretty old code) and the
> > work I am now starting.
> >
> > For me, I'm re-reading the manual pages for the DBI modules,
> > etc. I've also added another mailing list to follow about DBI.
> >
> > And I will now have some threads to add in the near future.
> > Threads I wouldn't have thought of.
> > But this isn't my mailing list, so breaking these topics into new
> > threads is just fine. Not a problem at all. 8-)
> >
> > Recently, something "clicked on" for me about mod_perl.
> > Which is pretty thrilling for me. ;-}
> >
> > Chris
> >
> >
> > >
> > > On 09.02.2021 23:03, Mithun Bhattacharya wrote:
> > > > I would consider mine a small setup on an internal network and I have
> > > > used both Sybase and SQL Server. In our case the DBA's preferred us to
> > > > remain connected rather than make too many connections - we need DB
> > > > access in bursts - it could be quiet for more than an hour and then
> > > > suddenly we might need hundreds of connections within few minutes (if we
> > > > didnt cache it). Another thing was we were connecting from forked
> > > > processes so at some point everything gets reaped including the
> > > > connections. Our style of coding has been to connect to the DB wherever
> > > > we actually need to fire one or more SQLs and do connect_cached in the
> > > > actual implementation (it is a separate library since we had to place a
> > > > wrapper to acquire credentials)
> > > >
> > > > On Tue, Feb 9, 2021 at 2:34 PM James Smith <js5@sanger.ac.uk <ma...@sanger.ac.uk>> wrote:
> > > >
> > > > Mithun,
> > > >
> > > > I’m not sure on what scale you work – but these are from experience in sites with
> > > > small to medium load – and we rarely see an appreciable gain in using cached or pooled
> > > > connections, just the occasional heartache they cause.
> > > > If you are working on small applications with a minimal number of databases on the DB
> > > > server then you may see some performance improvement (but tbh not as much as you used
> > > > to – as the servers have changed) Unfortunately I don’t in both my main and secondary
> > > > roles, and I know many others who come across these limitations as well.
> > > >
> > > > I’m not saying don’t use persistent or cached connections – but leaving it to some
> > > > hidden layers is not necessarily a good thing to do – it can have unforeseen side
> > > > effects {and Apache::DBI & PHP pconnect have both shown these up}
> > > >
> > > > If you are working with e.g. with MySQL the overhead of the (socket) connection is
> > > > very small, but having more connections open to cope with persistent connections
> > > > {memory wise} often needs specifying a much large database server – or not being able
> > > > to do all the nice tricks to in memory indexes and queries [to increase query
> > > > performance]. Being able to chose which connections you keep open and which you
> > > > open/close on a per request basis gives you the benefits of caching without the risks
> > > > involved [other than the “lock table” issue].____
> > > >
> > > > __ __
> > > >
> > > > __ __
> > > >
> > > > *From:*Mithun Bhattacharya <mithnb@gmail.com <ma...@gmail.com>>
> > > > *Sent:* 09 February 2021 18:34
> > > > *To:* mod_perl list <modperl@perl.apache.org <ma...@perl.apache.org>>
> > > > *Subject:* Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]____
> > > >
> > > > __ __
> > > >
> > > > Connection caching does work for most use cases - we have to accept James works in
> > > > scenarios most developers can't fathom :) ____
> > > >
> > > > __ __
> > > >
> > > > If you are just firing off simple SQL's without any triggers or named temporary tables
> > > > involved you should be good. The only times we recall tripping on cached connection is
> > > > when two different code snippets tried to create the same temporary table. Another
> > > > time the code was expecting the disconnect to complete the connection cleanup.____
> > > >
> > > > __ __
> > > >
> > > > On Tue, Feb 9, 2021 at 11:47 AM Vincent Veyron <vv.lists@wanadoo.fr
> > > > <ma...@wanadoo.fr>> wrote:____
> > > >
> > > > On Sun, 7 Feb 2021 20:21:34 +0000
> > > > James Smith <js5@sanger.ac.uk <ma...@sanger.ac.uk>> wrote:
> > > >
> > > > Hi James,
> > > >
> > > > > DBI sharing doesn't really gain you much - and can actually lead you into a
> > > > whole world of pain. It isn't actually worth turning it on at all.
> > > > >
> > > >
> > > > Never had a problem with it myself in years of using it, but I wrap my queries in
> > > > an eval { } and check $@, so that the scripts are not left hanging; also I have a
> > > > postgresql db ;-).
> > > >
> > > > I ran some tests with ab, I do see an improvement in response speed :
> > > >
> > > > my $dbh = DBI->connect()
> > > > Concurrency Level: 5
> > > > Time taken for tests: 22.198 seconds
> > > > Complete requests: 1000
> > > > Failed requests: 0
> > > > Total transferred: 8435000 bytes
> > > > HTML transferred: 8176000 bytes
> > > > Requests per second: 45.05 [#/sec] (mean)
> > > > Time per request: 110.990 [ms] (mean)
> > > > Time per request: 22.198 [ms] (mean, across all concurrent requests)
> > > > Transfer rate: 371.08 [Kbytes/sec] received
> > > >
> > > > my $dbh = DBI->connect_cached()
> > > > Concurrency Level: 5
> > > > Time taken for tests: 15.133 seconds
> > > > Complete requests: 1000
> > > > Failed requests: 0
> > > > Total transferred: 8435000 bytes
> > > > HTML transferred: 8176000 bytes
> > > > Requests per second: 66.08 [#/sec] (mean)
> > > > Time per request: 75.664 [ms] (mean)
> > > > Time per request: 15.133 [ms] (mean, across all concurrent requests)
> > > > Transfer rate: 544.33 [Kbytes/sec] received
> > > >
> > > >
> > > > --
> > > >
> > > > Bien à vous, Vincent Veyron
> > > >
> > > > https://compta.libremen.com [compta.libremen.com]
> > > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=CnIW-j3Bw_IfohZCciiwtkoqvr6nV2hHrNYMPpEOe8E&s=uf6Qi4tnTPryVuPvOKwfZQcFOksecWyn-LYPDVj44lY&e=>
> > > > Logiciel libre de comptabilité générale en partie double
> > > >
> > > >
> > > > ____
> > > >
> > > > -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity
> > > > registered in England with number 1021457 and a company registered in England with
> > > > number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
> > > >
> > >
>
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
My comment was just basically so as to avoid the case where someone else would later be
searching the archives of this mailing list for information about DBI, and never find
these (useful for DBI) posts, because DBI is not in the subject.
On 12.02.2021 00:51, Chris wrote:
> On Thu, Feb 11, 2021 at 09:52:16AM +0100, André Warnier (tomcat/perl) wrote:
>> Isn't this discussion about connection pools and firewalls etc getting a bit
>> far from the initial subject of the thread ?
>
> Perhaps. But this has become a pretty low volume mailing list.
> This "thread" has moved me to spend hours looking at changing and/or
> better understanding the work I have done (pretty old code) and the
> work I am now starting.
>
> For me, I'm re-reading the manual pages for the DBI modules,
> etc. I've also added another mailing list to follow about DBI.
>
> And I will now have some threads to add in the near future.
> Threads I wouldn't have thought of.
> But this isn't my mailing list, so breaking these topics into new
> threads is just fine. Not a problem at all. 8-)
>
> Recently, something "clicked on" for me about mod_perl.
> Which is pretty thrilling for me. ;-}
>
> Chris
>
>
>>
>> On 09.02.2021 23:03, Mithun Bhattacharya wrote:
>>> I would consider mine a small setup on an internal network and I have
>>> used both Sybase and SQL Server. In our case the DBA's preferred us to
>>> remain connected rather than make too many connections - we need DB
>>> access in bursts - it could be quiet for more than an hour and then
>>> suddenly we might need hundreds of connections within few minutes (if we
>>> didnt cache it). Another thing was we were connecting from forked
>>> processes so at some point everything gets reaped including the
>>> connections. Our style of coding has been to connect to the DB wherever
>>> we actually need to fire one or more SQLs and do connect_cached in the
>>> actual implementation (it is a separate library since we had to place a
>>> wrapper to acquire credentials)
>>>
>>> On Tue, Feb 9, 2021 at 2:34 PM James Smith <js5@sanger.ac.uk <ma...@sanger.ac.uk>> wrote:
>>>
>>> Mithun,
>>>
>>> I’m not sure on what scale you work – but these are from experience in sites with
>>> small to medium load – and we rarely see an appreciable gain in using cached or pooled
>>> connections, just the occasional heartache they cause.
>>> If you are working on small applications with a minimal number of databases on the DB
>>> server then you may see some performance improvement (but tbh not as much as you used
>>> to – as the servers have changed) Unfortunately I don’t in both my main and secondary
>>> roles, and I know many others who come across these limitations as well.
>>>
>>> I’m not saying don’t use persistent or cached connections – but leaving it to some
>>> hidden layers is not necessarily a good thing to do – it can have unforeseen side
>>> effects {and Apache::DBI & PHP pconnect have both shown these up}
>>>
>>> If you are working with e.g. with MySQL the overhead of the (socket) connection is
>>> very small, but having more connections open to cope with persistent connections
>>> {memory wise} often needs specifying a much large database server – or not being able
>>> to do all the nice tricks to in memory indexes and queries [to increase query
>>> performance]. Being able to chose which connections you keep open and which you
>>> open/close on a per request basis gives you the benefits of caching without the risks
>>> involved [other than the “lock table” issue].____
>>>
>>> __ __
>>>
>>> __ __
>>>
>>> *From:*Mithun Bhattacharya <mithnb@gmail.com <ma...@gmail.com>>
>>> *Sent:* 09 February 2021 18:34
>>> *To:* mod_perl list <modperl@perl.apache.org <ma...@perl.apache.org>>
>>> *Subject:* Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]____
>>>
>>> __ __
>>>
>>> Connection caching does work for most use cases - we have to accept James works in
>>> scenarios most developers can't fathom :) ____
>>>
>>> __ __
>>>
>>> If you are just firing off simple SQL's without any triggers or named temporary tables
>>> involved you should be good. The only times we recall tripping on cached connection is
>>> when two different code snippets tried to create the same temporary table. Another
>>> time the code was expecting the disconnect to complete the connection cleanup.____
>>>
>>> __ __
>>>
>>> On Tue, Feb 9, 2021 at 11:47 AM Vincent Veyron <vv.lists@wanadoo.fr
>>> <ma...@wanadoo.fr>> wrote:____
>>>
>>> On Sun, 7 Feb 2021 20:21:34 +0000
>>> James Smith <js5@sanger.ac.uk <ma...@sanger.ac.uk>> wrote:
>>>
>>> Hi James,
>>>
>>> > DBI sharing doesn't really gain you much - and can actually lead you into a
>>> whole world of pain. It isn't actually worth turning it on at all.
>>> >
>>>
>>> Never had a problem with it myself in years of using it, but I wrap my queries in
>>> an eval { } and check $@, so that the scripts are not left hanging; also I have a
>>> postgresql db ;-).
>>>
>>> I ran some tests with ab, I do see an improvement in response speed :
>>>
>>> my $dbh = DBI->connect()
>>> Concurrency Level: 5
>>> Time taken for tests: 22.198 seconds
>>> Complete requests: 1000
>>> Failed requests: 0
>>> Total transferred: 8435000 bytes
>>> HTML transferred: 8176000 bytes
>>> Requests per second: 45.05 [#/sec] (mean)
>>> Time per request: 110.990 [ms] (mean)
>>> Time per request: 22.198 [ms] (mean, across all concurrent requests)
>>> Transfer rate: 371.08 [Kbytes/sec] received
>>>
>>> my $dbh = DBI->connect_cached()
>>> Concurrency Level: 5
>>> Time taken for tests: 15.133 seconds
>>> Complete requests: 1000
>>> Failed requests: 0
>>> Total transferred: 8435000 bytes
>>> HTML transferred: 8176000 bytes
>>> Requests per second: 66.08 [#/sec] (mean)
>>> Time per request: 75.664 [ms] (mean)
>>> Time per request: 15.133 [ms] (mean, across all concurrent requests)
>>> Transfer rate: 544.33 [Kbytes/sec] received
>>>
>>>
>>> --
>>>
>>> Bien à vous, Vincent Veyron
>>>
>>> https://compta.libremen.com [compta.libremen.com]
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=CnIW-j3Bw_IfohZCciiwtkoqvr6nV2hHrNYMPpEOe8E&s=uf6Qi4tnTPryVuPvOKwfZQcFOksecWyn-LYPDVj44lY&e=>
>>> Logiciel libre de comptabilité générale en partie double
>>>
>>>
>>> ____
>>>
>>> -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity
>>> registered in England with number 1021457 and a company registered in England with
>>> number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
>>>
>>
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by Chris <cp...@bennettconstruction.us>.
On Thu, Feb 11, 2021 at 09:52:16AM +0100, André Warnier (tomcat/perl) wrote:
> Isn't this discussion about connection pools and firewalls etc getting a bit
> far from the initial subject of the thread ?
Perhaps. But this has become a pretty low volume mailing list.
This "thread" has moved me to spend hours looking at changing and/or
better understanding the work I have done (pretty old code) and the
work I am now starting.
For me, I'm re-reading the manual pages for the DBI modules,
etc. I've also added another mailing list to follow about DBI.
And I will now have some threads to add in the near future.
Threads I wouldn't have thought of.
But this isn't my mailing list, so breaking these topics into new
threads is just fine. Not a problem at all. 8-)
Recently, something "clicked on" for me about mod_perl.
Which is pretty thrilling for me. ;-}
Chris
>
> On 09.02.2021 23:03, Mithun Bhattacharya wrote:
> > I would consider mine a small setup on an internal network and I have
> > used both Sybase and SQL Server. In our case the DBA's preferred us to
> > remain connected rather than make too many connections - we need DB
> > access in bursts - it could be quiet for more than an hour and then
> > suddenly we might need hundreds of connections within few minutes (if we
> > didnt cache it). Another thing was we were connecting from forked
> > processes so at some point everything gets reaped including the
> > connections. Our style of coding has been to connect to the DB wherever
> > we actually need to fire one or more SQLs and do connect_cached in the
> > actual implementation (it is a separate library since we had to place a
> > wrapper to acquire credentials)
> >
> > On Tue, Feb 9, 2021 at 2:34 PM James Smith <js5@sanger.ac.uk <ma...@sanger.ac.uk>> wrote:
> >
> > Mithun,
> >
> > I’m not sure on what scale you work – but these are from experience in sites with
> > small to medium load – and we rarely see an appreciable gain in using cached or pooled
> > connections, just the occasional heartache they cause.
> > If you are working on small applications with a minimal number of databases on the DB
> > server then you may see some performance improvement (but tbh not as much as you used
> > to – as the servers have changed) Unfortunately I don’t in both my main and secondary
> > roles, and I know many others who come across these limitations as well.
> >
> > I’m not saying don’t use persistent or cached connections – but leaving it to some
> > hidden layers is not necessarily a good thing to do – it can have unforeseen side
> > effects {and Apache::DBI & PHP pconnect have both shown these up}
> >
> > If you are working with e.g. with MySQL the overhead of the (socket) connection is
> > very small, but having more connections open to cope with persistent connections
> > {memory wise} often needs specifying a much large database server – or not being able
> > to do all the nice tricks to in memory indexes and queries [to increase query
> > performance]. Being able to chose which connections you keep open and which you
> > open/close on a per request basis gives you the benefits of caching without the risks
> > involved [other than the “lock table” issue].____
> >
> > __ __
> >
> > __ __
> >
> > *From:*Mithun Bhattacharya <mithnb@gmail.com <ma...@gmail.com>>
> > *Sent:* 09 February 2021 18:34
> > *To:* mod_perl list <modperl@perl.apache.org <ma...@perl.apache.org>>
> > *Subject:* Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]____
> >
> > __ __
> >
> > Connection caching does work for most use cases - we have to accept James works in
> > scenarios most developers can't fathom :) ____
> >
> > __ __
> >
> > If you are just firing off simple SQL's without any triggers or named temporary tables
> > involved you should be good. The only times we recall tripping on cached connection is
> > when two different code snippets tried to create the same temporary table. Another
> > time the code was expecting the disconnect to complete the connection cleanup.____
> >
> > __ __
> >
> > On Tue, Feb 9, 2021 at 11:47 AM Vincent Veyron <vv.lists@wanadoo.fr
> > <ma...@wanadoo.fr>> wrote:____
> >
> > On Sun, 7 Feb 2021 20:21:34 +0000
> > James Smith <js5@sanger.ac.uk <ma...@sanger.ac.uk>> wrote:
> >
> > Hi James,
> >
> > > DBI sharing doesn't really gain you much - and can actually lead you into a
> > whole world of pain. It isn't actually worth turning it on at all.
> > >
> >
> > Never had a problem with it myself in years of using it, but I wrap my queries in
> > an eval { } and check $@, so that the scripts are not left hanging; also I have a
> > postgresql db ;-).
> >
> > I ran some tests with ab, I do see an improvement in response speed :
> >
> > my $dbh = DBI->connect()
> > Concurrency Level: 5
> > Time taken for tests: 22.198 seconds
> > Complete requests: 1000
> > Failed requests: 0
> > Total transferred: 8435000 bytes
> > HTML transferred: 8176000 bytes
> > Requests per second: 45.05 [#/sec] (mean)
> > Time per request: 110.990 [ms] (mean)
> > Time per request: 22.198 [ms] (mean, across all concurrent requests)
> > Transfer rate: 371.08 [Kbytes/sec] received
> >
> > my $dbh = DBI->connect_cached()
> > Concurrency Level: 5
> > Time taken for tests: 15.133 seconds
> > Complete requests: 1000
> > Failed requests: 0
> > Total transferred: 8435000 bytes
> > HTML transferred: 8176000 bytes
> > Requests per second: 66.08 [#/sec] (mean)
> > Time per request: 75.664 [ms] (mean)
> > Time per request: 15.133 [ms] (mean, across all concurrent requests)
> > Transfer rate: 544.33 [Kbytes/sec] received
> >
> >
> > --
> >
> > Bien à vous, Vincent Veyron
> >
> > https://compta.libremen.com [compta.libremen.com]
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=CnIW-j3Bw_IfohZCciiwtkoqvr6nV2hHrNYMPpEOe8E&s=uf6Qi4tnTPryVuPvOKwfZQcFOksecWyn-LYPDVj44lY&e=>
> > Logiciel libre de comptabilité générale en partie double
> >
> >
> > ____
> >
> > -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity
> > registered in England with number 1021457 and a company registered in England with
> > number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
> >
>
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
Isn't this discussion about connection pools and firewalls etc getting a bit far from the
initial subject of the thread ?
On 09.02.2021 23:03, Mithun Bhattacharya wrote:
> I would consider mine a small setup on an internal network and I have used both Sybase and
> SQL Server. In our case the DBA's preferred us to remain connected rather than make too
> many connections - we need DB access in bursts - it could be quiet for more than an hour
> and then suddenly we might need hundreds of connections within few minutes (if we didnt
> cache it). Another thing was we were connecting from forked processes so at some point
> everything gets reaped including the connections. Our style of coding has been to connect
> to the DB wherever we actually need to fire one or more SQLs and do connect_cached in the
> actual implementation (it is a separate library since we had to place a wrapper to
> acquire credentials)
>
> On Tue, Feb 9, 2021 at 2:34 PM James Smith <js5@sanger.ac.uk <ma...@sanger.ac.uk>> wrote:
>
> Mithun,
>
> I’m not sure on what scale you work – but these are from experience in sites with
> small to medium load – and we rarely see an appreciable gain in using cached or pooled
> connections, just the occasional heartache they cause.
> If you are working on small applications with a minimal number of databases on the DB
> server then you may see some performance improvement (but tbh not as much as you used
> to – as the servers have changed) Unfortunately I don’t in both my main and secondary
> roles, and I know many others who come across these limitations as well.
>
> I’m not saying don’t use persistent or cached connections – but leaving it to some
> hidden layers is not necessarily a good thing to do – it can have unforeseen side
> effects {and Apache::DBI & PHP pconnect have both shown these up}
>
> If you are working with e.g. with MySQL the overhead of the (socket) connection is
> very small, but having more connections open to cope with persistent connections
> {memory wise} often needs specifying a much large database server – or not being able
> to do all the nice tricks to in memory indexes and queries [to increase query
> performance]. Being able to chose which connections you keep open and which you
> open/close on a per request basis gives you the benefits of caching without the risks
> involved [other than the “lock table” issue].____
>
> __ __
>
> __ __
>
> *From:*Mithun Bhattacharya <mithnb@gmail.com <ma...@gmail.com>>
> *Sent:* 09 February 2021 18:34
> *To:* mod_perl list <modperl@perl.apache.org <ma...@perl.apache.org>>
> *Subject:* Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]____
>
> __ __
>
> Connection caching does work for most use cases - we have to accept James works in
> scenarios most developers can't fathom :) ____
>
> __ __
>
> If you are just firing off simple SQL's without any triggers or named temporary tables
> involved you should be good. The only times we recall tripping on cached connection is
> when two different code snippets tried to create the same temporary table. Another
> time the code was expecting the disconnect to complete the connection cleanup.____
>
> __ __
>
> On Tue, Feb 9, 2021 at 11:47 AM Vincent Veyron <vv.lists@wanadoo.fr
> <ma...@wanadoo.fr>> wrote:____
>
> On Sun, 7 Feb 2021 20:21:34 +0000
> James Smith <js5@sanger.ac.uk <ma...@sanger.ac.uk>> wrote:
>
> Hi James,
>
> > DBI sharing doesn't really gain you much - and can actually lead you into a
> whole world of pain. It isn't actually worth turning it on at all.
> >
>
> Never had a problem with it myself in years of using it, but I wrap my queries in
> an eval { } and check $@, so that the scripts are not left hanging; also I have a
> postgresql db ;-).
>
> I ran some tests with ab, I do see an improvement in response speed :
>
> my $dbh = DBI->connect()
> Concurrency Level: 5
> Time taken for tests: 22.198 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 8435000 bytes
> HTML transferred: 8176000 bytes
> Requests per second: 45.05 [#/sec] (mean)
> Time per request: 110.990 [ms] (mean)
> Time per request: 22.198 [ms] (mean, across all concurrent requests)
> Transfer rate: 371.08 [Kbytes/sec] received
>
> my $dbh = DBI->connect_cached()
> Concurrency Level: 5
> Time taken for tests: 15.133 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 8435000 bytes
> HTML transferred: 8176000 bytes
> Requests per second: 66.08 [#/sec] (mean)
> Time per request: 75.664 [ms] (mean)
> Time per request: 15.133 [ms] (mean, across all concurrent requests)
> Transfer rate: 544.33 [Kbytes/sec] received
>
>
> --
>
> Bien à vous, Vincent Veyron
>
> https://compta.libremen.com [compta.libremen.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=CnIW-j3Bw_IfohZCciiwtkoqvr6nV2hHrNYMPpEOe8E&s=uf6Qi4tnTPryVuPvOKwfZQcFOksecWyn-LYPDVj44lY&e=>
> Logiciel libre de comptabilité générale en partie double
>
>
> ____
>
> -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity
> registered in England with number 1021457 and a company registered in England with
> number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
>
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by Dave Morgan <da...@1001111.com>.
As a long time Oracle DBA and perl mangler I have refrained from joining
this conversation until now.
Connection caching is a configuration option in DBI, not a coding
requirement. For web servers it is a single line edit in startup.pl
import DBICache versus import DBI if I recall incorrectly.
NO CODE CHANGES REQUIRED
Now caching connections will require clean code. Code that was
acceptable on a short term connection
causes problems on long term connections. Cursors left open, virtual
connections not closed, etc
are non-issues using DBI but huge problems with DBICache
Applications doing heavy sql or batch work should not use DBICache. Just
from an administrative perspective.
Session less applications like web servers SHALL use it. And I use the
legal meaning for shall
I don't care what database is on the end of a DBI connection, I mangle
em all, poorly
YMMV
Dave
--
Dave Morgan
Senior Consultant, 1001111 Alberta Limited
dave.morgan@1001111.com
403 399 2442
Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
Posted by Mithun Bhattacharya <mi...@gmail.com>.
I would consider mine a small setup on an internal network and I have used
both Sybase and SQL Server. In our case the DBA's preferred us to remain
connected rather than make too many connections - we need DB access in
bursts - it could be quiet for more than an hour and then suddenly we might
need hundreds of connections within few minutes (if we didnt cache it).
Another thing was we were connecting from forked processes so at some point
everything gets reaped including the connections. Our style of coding has
been to connect to the DB wherever we actually need to fire one or more
SQLs and do connect_cached in the actual implementation (it is a separate
library since we had to place a wrapper to acquire credentials)
On Tue, Feb 9, 2021 at 2:34 PM James Smith <js...@sanger.ac.uk> wrote:
> Mithun,
>
> I’m not sure on what scale you work – but these are from experience in
> sites with small to medium load – and we rarely see an appreciable gain in
> using cached or pooled connections, just the occasional heartache they
> cause.
> If you are working on small applications with a minimal number of
> databases on the DB server then you may see some performance improvement
> (but tbh not as much as you used to – as the servers have changed)
> Unfortunately I don’t in both my main and secondary roles, and I know many
> others who come across these limitations as well.
>
> I’m not saying don’t use persistent or cached connections – but leaving it
> to some hidden layers is not necessarily a good thing to do – it can have
> unforeseen side effects {and Apache::DBI & PHP pconnect have both shown
> these up}
>
> If you are working with e.g. with MySQL the overhead of the (socket)
> connection is very small, but having more connections open to cope with
> persistent connections {memory wise} often needs specifying a much large
> database server – or not being able to do all the nice tricks to in memory
> indexes and queries [to increase query performance]. Being able to chose
> which connections you keep open and which you open/close on a per request
> basis gives you the benefits of caching without the risks involved [other
> than the “lock table” issue].
>
>
>
>
>
> *From:* Mithun Bhattacharya <mi...@gmail.com>
> *Sent:* 09 February 2021 18:34
> *To:* mod_perl list <mo...@perl.apache.org>
> *Subject:* Re: Moving ExecCGI to mod_perl - performance and custom
> 'modules' [EXT]
>
>
>
> Connection caching does work for most use cases - we have to accept James
> works in scenarios most developers can't fathom :)
>
>
>
> If you are just firing off simple SQL's without any triggers or named
> temporary tables involved you should be good. The only times we recall
> tripping on cached connection is when two different code snippets tried to
> create the same temporary table. Another time the code was expecting the
> disconnect to complete the connection cleanup.
>
>
>
> On Tue, Feb 9, 2021 at 11:47 AM Vincent Veyron <vv...@wanadoo.fr>
> wrote:
>
> On Sun, 7 Feb 2021 20:21:34 +0000
> James Smith <js...@sanger.ac.uk> wrote:
>
> Hi James,
>
> > DBI sharing doesn't really gain you much - and can actually lead you
> into a whole world of pain. It isn't actually worth turning it on at all.
> >
>
> Never had a problem with it myself in years of using it, but I wrap my
> queries in an eval { } and check $@, so that the scripts are not left
> hanging; also I have a postgresql db ;-).
>
> I ran some tests with ab, I do see an improvement in response speed :
>
> my $dbh = DBI->connect()
> Concurrency Level: 5
> Time taken for tests: 22.198 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 8435000 bytes
> HTML transferred: 8176000 bytes
> Requests per second: 45.05 [#/sec] (mean)
> Time per request: 110.990 [ms] (mean)
> Time per request: 22.198 [ms] (mean, across all concurrent requests)
> Transfer rate: 371.08 [Kbytes/sec] received
>
> my $dbh = DBI->connect_cached()
> Concurrency Level: 5
> Time taken for tests: 15.133 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 8435000 bytes
> HTML transferred: 8176000 bytes
> Requests per second: 66.08 [#/sec] (mean)
> Time per request: 75.664 [ms] (mean)
> Time per request: 15.133 [ms] (mean, across all concurrent requests)
> Transfer rate: 544.33 [Kbytes/sec] received
>
>
> --
>
> Bien à vous, Vincent Veyron
>
> https://compta.libremen.com [compta.libremen.com]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=CnIW-j3Bw_IfohZCciiwtkoqvr6nV2hHrNYMPpEOe8E&s=uf6Qi4tnTPryVuPvOKwfZQcFOksecWyn-LYPDVj44lY&e=>
> Logiciel libre de comptabilité générale en partie double
>
>
> -- The Wellcome Sanger Institute is operated by Genome Research Limited, a
> charity registered in England with number 1021457 and a company registered
> in England with number 2742969, whose registered office is 215 Euston Road,
> London, NW1 2BE.
>
RE: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by James Smith <js...@sanger.ac.uk>.
Mithun,
I’m not sure on what scale you work – but these are from experience in sites with small to medium load – and we rarely see an appreciable gain in using cached or pooled connections, just the occasional heartache they cause.
If you are working on small applications with a minimal number of databases on the DB server then you may see some performance improvement (but tbh not as much as you used to – as the servers have changed) Unfortunately I don’t in both my main and secondary roles, and I know many others who come across these limitations as well.
I’m not saying don’t use persistent or cached connections – but leaving it to some hidden layers is not necessarily a good thing to do – it can have unforeseen side effects {and Apache::DBI & PHP pconnect have both shown these up}
If you are working with e.g. with MySQL the overhead of the (socket) connection is very small, but having more connections open to cope with persistent connections {memory wise} often needs specifying a much large database server – or not being able to do all the nice tricks to in memory indexes and queries [to increase query performance]. Being able to chose which connections you keep open and which you open/close on a per request basis gives you the benefits of caching without the risks involved [other than the “lock table” issue].
From: Mithun Bhattacharya <mi...@gmail.com>
Sent: 09 February 2021 18:34
To: mod_perl list <mo...@perl.apache.org>
Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
Connection caching does work for most use cases - we have to accept James works in scenarios most developers can't fathom :)
If you are just firing off simple SQL's without any triggers or named temporary tables involved you should be good. The only times we recall tripping on cached connection is when two different code snippets tried to create the same temporary table. Another time the code was expecting the disconnect to complete the connection cleanup.
On Tue, Feb 9, 2021 at 11:47 AM Vincent Veyron <vv...@wanadoo.fr>> wrote:
On Sun, 7 Feb 2021 20:21:34 +0000
James Smith <js...@sanger.ac.uk>> wrote:
Hi James,
> DBI sharing doesn't really gain you much - and can actually lead you into a whole world of pain. It isn't actually worth turning it on at all.
>
Never had a problem with it myself in years of using it, but I wrap my queries in an eval { } and check $@, so that the scripts are not left hanging; also I have a postgresql db ;-).
I ran some tests with ab, I do see an improvement in response speed :
my $dbh = DBI->connect()
Concurrency Level: 5
Time taken for tests: 22.198 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 8435000 bytes
HTML transferred: 8176000 bytes
Requests per second: 45.05 [#/sec] (mean)
Time per request: 110.990 [ms] (mean)
Time per request: 22.198 [ms] (mean, across all concurrent requests)
Transfer rate: 371.08 [Kbytes/sec] received
my $dbh = DBI->connect_cached()
Concurrency Level: 5
Time taken for tests: 15.133 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 8435000 bytes
HTML transferred: 8176000 bytes
Requests per second: 66.08 [#/sec] (mean)
Time per request: 75.664 [ms] (mean)
Time per request: 15.133 [ms] (mean, across all concurrent requests)
Transfer rate: 544.33 [Kbytes/sec] received
--
Bien à vous, Vincent Veyron
https://compta.libremen.com [compta.libremen.com]<https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=CnIW-j3Bw_IfohZCciiwtkoqvr6nV2hHrNYMPpEOe8E&s=uf6Qi4tnTPryVuPvOKwfZQcFOksecWyn-LYPDVj44lY&e=>
Logiciel libre de comptabilité générale en partie double
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
Posted by Mithun Bhattacharya <mi...@gmail.com>.
Connection caching does work for most use cases - we have to accept James
works in scenarios most developers can't fathom :)
If you are just firing off simple SQL's without any triggers or named
temporary tables involved you should be good. The only times we recall
tripping on cached connection is when two different code snippets tried to
create the same temporary table. Another time the code was expecting the
disconnect to complete the connection cleanup.
On Tue, Feb 9, 2021 at 11:47 AM Vincent Veyron <vv...@wanadoo.fr> wrote:
> On Sun, 7 Feb 2021 20:21:34 +0000
> James Smith <js...@sanger.ac.uk> wrote:
>
> Hi James,
>
> > DBI sharing doesn't really gain you much - and can actually lead you
> into a whole world of pain. It isn't actually worth turning it on at all.
> >
>
> Never had a problem with it myself in years of using it, but I wrap my
> queries in an eval { } and check $@, so that the scripts are not left
> hanging; also I have a postgresql db ;-).
>
> I ran some tests with ab, I do see an improvement in response speed :
>
> my $dbh = DBI->connect()
> Concurrency Level: 5
> Time taken for tests: 22.198 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 8435000 bytes
> HTML transferred: 8176000 bytes
> Requests per second: 45.05 [#/sec] (mean)
> Time per request: 110.990 [ms] (mean)
> Time per request: 22.198 [ms] (mean, across all concurrent requests)
> Transfer rate: 371.08 [Kbytes/sec] received
>
> my $dbh = DBI->connect_cached()
> Concurrency Level: 5
> Time taken for tests: 15.133 seconds
> Complete requests: 1000
> Failed requests: 0
> Total transferred: 8435000 bytes
> HTML transferred: 8176000 bytes
> Requests per second: 66.08 [#/sec] (mean)
> Time per request: 75.664 [ms] (mean)
> Time per request: 15.133 [ms] (mean, across all concurrent requests)
> Transfer rate: 544.33 [Kbytes/sec] received
>
>
> --
>
> Bien à vous, Vincent Veyron
>
> https://compta.libremen.com
> Logiciel libre de comptabilité générale en partie double
>
>
>
>
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by Vincent Veyron <vv...@wanadoo.fr>.
On Sun, 7 Feb 2021 20:21:34 +0000
James Smith <js...@sanger.ac.uk> wrote:
Hi James,
> DBI sharing doesn't really gain you much - and can actually lead you into a whole world of pain. It isn't actually worth turning it on at all.
>
Never had a problem with it myself in years of using it, but I wrap my queries in an eval { } and check $@, so that the scripts are not left hanging; also I have a postgresql db ;-).
I ran some tests with ab, I do see an improvement in response speed :
my $dbh = DBI->connect()
Concurrency Level: 5
Time taken for tests: 22.198 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 8435000 bytes
HTML transferred: 8176000 bytes
Requests per second: 45.05 [#/sec] (mean)
Time per request: 110.990 [ms] (mean)
Time per request: 22.198 [ms] (mean, across all concurrent requests)
Transfer rate: 371.08 [Kbytes/sec] received
my $dbh = DBI->connect_cached()
Concurrency Level: 5
Time taken for tests: 15.133 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 8435000 bytes
HTML transferred: 8176000 bytes
Requests per second: 66.08 [#/sec] (mean)
Time per request: 75.664 [ms] (mean)
Time per request: 15.133 [ms] (mean, across all concurrent requests)
Transfer rate: 544.33 [Kbytes/sec] received
--
Bien à vous, Vincent Veyron
https://compta.libremen.com
Logiciel libre de comptabilité générale en partie double
Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
Posted by John Dunlap <Jo...@lariat.co>.
We're in a similar situation with 20-40 databases per server. We use
PGBouncer for connection pooling.
On Sun, Feb 7, 2021 at 2:23 PM James Smith <js...@sanger.ac.uk> wrote:
> DBI sharing doesn't really gain you much - and can actually lead you into
> a whole world of pain. It isn't actually worth turning it on at all.
>
> We use dedicated DB caching in the cases where we benefit from it as and
> when you need it (low level caching database)...
>
> Although milage may vary - our problem was DB servers with 30 or 40
> databases on them being connected from a number of approximately 50-100
> child apaches, meant we ended up blowing up the connections to the database
> server really quickly...
>
>
> -----Original Message-----
> From: Vincent Veyron <vv...@wanadoo.fr>
> Sent: 07 February 2021 19:06
> To: Steven Haigh <ne...@crc.id.au>
> Cc: James Smith <js...@sanger.ac.uk>; modperl@perl.apache.org
> Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
> [EXT]
>
> On Sun, 07 Feb 2021 23:58:17 +1100
> Steven Haigh <ne...@crc.id.au> wrote:
> >
> > I haven't gotten into the preload or DBI sharing yet - as that'll end
> > up needing a bit of a rewrite of code to take advantage of. I'd be
> > open to suggestions here from those who have done it in the past to
> > save me going down some dead ends :D
>
> I use mod_perl handlers, so not sure how it mixes with PerlRegistry, but
> you probably want to have a look at connect_cached
>
>
> --
>
> Bien à vous, Vincent Veyron
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwIFAw&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=C0OcuGbNbfxaSa8ASgV3uFXzejn7MpjIUH1aP1RbiyU&s=GPr8VuKQ3rZCzCPwggyAHdCOojK6ZThmShKk0Jb3maI&e=
> Logiciel libre de comptabilité générale en partie double
>
>
>
> --
> The Wellcome Sanger Institute is operated by Genome Research
> Limited, a charity registered in England with number 1021457 and a
> company registered in England with number 2742969, whose registered
> office is 215 Euston Road, London, NW1 2BE.
>
--
John Dunlap
*CTO | Lariat *
*Direct:*
*john@lariat.co <jo...@lariat.co>*
*Customer Service:*
877.268.6667
support@lariat.co
RE: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by James Smith <js...@sanger.ac.uk>.
No it was one of our DBAs who told us to take if off…! It causes them all sorts of problems – the main one being it keeps connections open, which is the best way to completely foobar the database servers! We make them happy by turning it off. We’ve even had Oracle developers (sorry not people developing SQL in oracle but people developing Oracle) scratching their heads over things like this and not being able to come up with a solution…..
We’ve discovered lots of interesting side effects with connection pooling when using over a secure network – e.g. the Oracle client doesn’t like running through a firewall – as it can’t cope when the firewall drops the connection – both sides of the system the “pool” thinks the connection is open, the database thinks the connection is open, and the client just hangs….. We have seen similar things with other applications which use connection pooling and long duration database handles. We get round it with permanent MySQL connections by closing/re-opening them after 5 minutes of inactivity – hence the need to develop our own cache/pool….
From: Mithun Bhattacharya <mi...@gmail.com>
Sent: 07 February 2021 20:36
To: James Smith <js...@sanger.ac.uk>
Cc: Vincent Veyron <vv...@wanadoo.fr>; Steven Haigh <ne...@crc.id.au>; modperl@perl.apache.org
Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
DBI sharing has it's own issues but in most use cases it does a pretty good job and keeps the DBA's happy - that is very important ;)
On Sun, Feb 7, 2021 at 2:23 PM James Smith <js...@sanger.ac.uk>> wrote:
DBI sharing doesn't really gain you much - and can actually lead you into a whole world of pain. It isn't actually worth turning it on at all.
We use dedicated DB caching in the cases where we benefit from it as and when you need it (low level caching database)...
Although milage may vary - our problem was DB servers with 30 or 40 databases on them being connected from a number of approximately 50-100 child apaches, meant we ended up blowing up the connections to the database server really quickly...
-----Original Message-----
From: Vincent Veyron <vv...@wanadoo.fr>>
Sent: 07 February 2021 19:06
To: Steven Haigh <ne...@crc.id.au>>
Cc: James Smith <js...@sanger.ac.uk>>; modperl@perl.apache.org<ma...@perl.apache.org>
Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
On Sun, 07 Feb 2021 23:58:17 +1100
Steven Haigh <ne...@crc.id.au>> wrote:
>
> I haven't gotten into the preload or DBI sharing yet - as that'll end
> up needing a bit of a rewrite of code to take advantage of. I'd be
> open to suggestions here from those who have done it in the past to
> save me going down some dead ends :D
I use mod_perl handlers, so not sure how it mixes with PerlRegistry, but you probably want to have a look at connect_cached
--
Bien à vous, Vincent Veyron
https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwIFAw&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=C0OcuGbNbfxaSa8ASgV3uFXzejn7MpjIUH1aP1RbiyU&s=GPr8VuKQ3rZCzCPwggyAHdCOojK6ZThmShKk0Jb3maI&e=
Logiciel libre de comptabilité générale en partie double
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
Posted by Mithun Bhattacharya <mi...@gmail.com>.
DBI sharing has it's own issues but in most use cases it does a pretty good
job and keeps the DBA's happy - that is very important ;)
On Sun, Feb 7, 2021 at 2:23 PM James Smith <js...@sanger.ac.uk> wrote:
> DBI sharing doesn't really gain you much - and can actually lead you into
> a whole world of pain. It isn't actually worth turning it on at all.
>
> We use dedicated DB caching in the cases where we benefit from it as and
> when you need it (low level caching database)...
>
> Although milage may vary - our problem was DB servers with 30 or 40
> databases on them being connected from a number of approximately 50-100
> child apaches, meant we ended up blowing up the connections to the database
> server really quickly...
>
>
> -----Original Message-----
> From: Vincent Veyron <vv...@wanadoo.fr>
> Sent: 07 February 2021 19:06
> To: Steven Haigh <ne...@crc.id.au>
> Cc: James Smith <js...@sanger.ac.uk>; modperl@perl.apache.org
> Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
> [EXT]
>
> On Sun, 07 Feb 2021 23:58:17 +1100
> Steven Haigh <ne...@crc.id.au> wrote:
> >
> > I haven't gotten into the preload or DBI sharing yet - as that'll end
> > up needing a bit of a rewrite of code to take advantage of. I'd be
> > open to suggestions here from those who have done it in the past to
> > save me going down some dead ends :D
>
> I use mod_perl handlers, so not sure how it mixes with PerlRegistry, but
> you probably want to have a look at connect_cached
>
>
> --
>
> Bien à vous, Vincent Veyron
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwIFAw&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=C0OcuGbNbfxaSa8ASgV3uFXzejn7MpjIUH1aP1RbiyU&s=GPr8VuKQ3rZCzCPwggyAHdCOojK6ZThmShKk0Jb3maI&e=
> Logiciel libre de comptabilité générale en partie double
>
>
>
> --
> The Wellcome Sanger Institute is operated by Genome Research
> Limited, a charity registered in England with number 1021457 and a
> company registered in England with number 2742969, whose registered
> office is 215 Euston Road, London, NW1 2BE.
>
RE: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by James Smith <js...@sanger.ac.uk>.
I meant between requests in the same thread – it uses persistent connections which are bad in a number of ways {e.g. fun with locking if you use it and your script dies halfway through, keeps too many connections open and can block the database, has issues with secure set ups {firewall between web and servers for instance}
From: Wesley Peng <we...@pengfamily.de>
Sent: 08 February 2021 00:29
To: modperl@perl.apache.org
Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
what's DBI sharing? do you mean Apache::DBI?
Does perl has Java similar DB connection pool?
Thanks.
On Mon, Feb 8, 2021, at 4:21 AM, James Smith wrote:
DBI sharing doesn't really gain you much - and can actually lead you into a whole world of pain. It isn't actually worth turning it on at all.
We use dedicated DB caching in the cases where we benefit from it as and when you need it (low level caching database)...
Although milage may vary - our problem was DB servers with 30 or 40 databases on them being connected from a number of approximately 50-100 child apaches, meant we ended up blowing up the connections to the database server really quickly...
-----Original Message-----
From: Vincent Veyron <vv...@wanadoo.fr>>
Sent: 07 February 2021 19:06
To: Steven Haigh <ne...@crc.id.au>>
Cc: James Smith <js...@sanger.ac.uk>>; modperl@perl.apache.org<ma...@perl.apache.org>
Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
On Sun, 07 Feb 2021 23:58:17 +1100
Steven Haigh <ne...@crc.id.au>> wrote:
>
> I haven't gotten into the preload or DBI sharing yet - as that'll end
> up needing a bit of a rewrite of code to take advantage of. I'd be
> open to suggestions here from those who have done it in the past to
> save me going down some dead ends :D
I use mod_perl handlers, so not sure how it mixes with PerlRegistry, but you probably want to have a look at connect_cached
--
Bien à vous, Vincent Veyron
https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwIFAw&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=C0OcuGbNbfxaSa8ASgV3uFXzejn7MpjIUH1aP1RbiyU&s=GPr8VuKQ3rZCzCPwggyAHdCOojK6ZThmShKk0Jb3maI&e=
Logiciel libre de comptabilité générale en partie double
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
Posted by Wesley Peng <we...@pengfamily.de>.
what's DBI sharing? do you mean Apache::DBI?
Does perl has Java similar DB connection pool?
Thanks.
On Mon, Feb 8, 2021, at 4:21 AM, James Smith wrote:
> DBI sharing doesn't really gain you much - and can actually lead you into a whole world of pain. It isn't actually worth turning it on at all.
>
> We use dedicated DB caching in the cases where we benefit from it as and when you need it (low level caching database)...
>
> Although milage may vary - our problem was DB servers with 30 or 40 databases on them being connected from a number of approximately 50-100 child apaches, meant we ended up blowing up the connections to the database server really quickly...
>
>
> -----Original Message-----
> From: Vincent Veyron <vv...@wanadoo.fr>
> Sent: 07 February 2021 19:06
> To: Steven Haigh <ne...@crc.id.au>
> Cc: James Smith <js...@sanger.ac.uk>; modperl@perl.apache.org
> Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
>
> On Sun, 07 Feb 2021 23:58:17 +1100
> Steven Haigh <ne...@crc.id.au> wrote:
> >
> > I haven't gotten into the preload or DBI sharing yet - as that'll end
> > up needing a bit of a rewrite of code to take advantage of. I'd be
> > open to suggestions here from those who have done it in the past to
> > save me going down some dead ends :D
>
> I use mod_perl handlers, so not sure how it mixes with PerlRegistry, but you probably want to have a look at connect_cached
>
>
> --
>
> Bien à vous, Vincent Veyron
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwIFAw&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=C0OcuGbNbfxaSa8ASgV3uFXzejn7MpjIUH1aP1RbiyU&s=GPr8VuKQ3rZCzCPwggyAHdCOojK6ZThmShKk0Jb3maI&e=
> Logiciel libre de comptabilité générale en partie double
>
>
>
> --
> The Wellcome Sanger Institute is operated by Genome Research
> Limited, a charity registered in England with number 1021457 and a
> company registered in England with number 2742969, whose registered
> office is 215 Euston Road, London, NW1 2BE.
>
RE: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by James Smith <js...@sanger.ac.uk>.
DBI sharing doesn't really gain you much - and can actually lead you into a whole world of pain. It isn't actually worth turning it on at all.
We use dedicated DB caching in the cases where we benefit from it as and when you need it (low level caching database)...
Although milage may vary - our problem was DB servers with 30 or 40 databases on them being connected from a number of approximately 50-100 child apaches, meant we ended up blowing up the connections to the database server really quickly...
-----Original Message-----
From: Vincent Veyron <vv...@wanadoo.fr>
Sent: 07 February 2021 19:06
To: Steven Haigh <ne...@crc.id.au>
Cc: James Smith <js...@sanger.ac.uk>; modperl@perl.apache.org
Subject: Re: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
On Sun, 07 Feb 2021 23:58:17 +1100
Steven Haigh <ne...@crc.id.au> wrote:
>
> I haven't gotten into the preload or DBI sharing yet - as that'll end
> up needing a bit of a rewrite of code to take advantage of. I'd be
> open to suggestions here from those who have done it in the past to
> save me going down some dead ends :D
I use mod_perl handlers, so not sure how it mixes with PerlRegistry, but you probably want to have a look at connect_cached
--
Bien à vous, Vincent Veyron
https://urldefense.proofpoint.com/v2/url?u=https-3A__compta.libremen.com&d=DwIFAw&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=C0OcuGbNbfxaSa8ASgV3uFXzejn7MpjIUH1aP1RbiyU&s=GPr8VuKQ3rZCzCPwggyAHdCOojK6ZThmShKk0Jb3maI&e=
Logiciel libre de comptabilité générale en partie double
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Re: Moving ExecCGI to mod_perl - performance and custom 'modules'
[EXT]
Posted by Vincent Veyron <vv...@wanadoo.fr>.
On Sun, 07 Feb 2021 23:58:17 +1100
Steven Haigh <ne...@crc.id.au> wrote:
>
> I haven't gotten into the preload or DBI sharing yet - as that'll end
> up needing a bit of a rewrite of code to take advantage of. I'd be open
> to suggestions here from those who have done it in the past to save me
> going down some dead ends :D
I use mod_perl handlers, so not sure how it mixes with PerlRegistry, but you probably want to have a look at connect_cached
--
Bien à vous, Vincent Veyron
https://compta.libremen.com
Logiciel libre de comptabilité générale en partie double
RE: [EXT]
Posted by James Smith <js...@sanger.ac.uk>.
This can be a bigger gain if you are limited with memory as multiple processes will share the same block of physical memory { so limiting swap } – this is one of the advantages of the mod_perl approach over the app approach in things like dancer. You have the flexibility to include what you want – if you have a single webpage or service which needs a large number of modules – but is perhaps used once in every 100k requests you don’t have to load it in at the start of the process, and you have the flexibility to choose how much or how little you want of the framework preloaded…
From: Adam Prime <ad...@utoronto.ca>
Sent: 07 February 2021 13:45
To: Steven Haigh <ne...@crc.id.au>
Cc: James Smith <js...@sanger.ac.uk>; modperl@perl.apache.org
Subject: Re: [EXT]
There is one other thing you can do relatively easily that may get you a marginal gain when Apache spins up new children. Load some or all of your Perl dependencies before Apache forks. There is also an opportunity to load other static resources at this time, but that can get a little more involved and/or confusing. There is some documentation here:
https://perl.apache.org/docs/2.0/user/handlers/server.html#Startup_File [perl.apache.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__perl.apache.org_docs_2.0_user_handlers_server.html-23Startup-5FFile&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=kwRJ9OU2GaHG3eTqlSv3NGFCM8bbDVVw6VWYGivfszw&s=sT5APUQZ6MWHqhwLVLrTEK4xPf6OEANKV0EePFJ1L6Y&e=>
Adam
On Feb 7, 2021, at 8:15 AM, Steven Haigh <ne...@crc.id.au>> wrote:
In fact, I just realised that 'ab' test is rather restrictive.... So here's a bit more of an extended test:
# ab -k -n 1000 -c 32
Apache + ExecCGI:
Requests per second: 14.26 [#/sec] (mean)
Time per request: 2244.181 [ms] (mean)
Time per request: 70.131 [ms] (mean, across all concurrent requests)
Apache + mod_perl (ModPerl::PerlRegistry):
Requests per second: 132.14 [#/sec] (mean)
Time per request: 242.175 [ms] (mean)
Time per request: 7.568 [ms] (mean, across all concurrent requests)
Interestingly, without Keepalives, the story is much the same:
# ab -n 1000 -c 32
Apache + ExecCGI:
Requests per second: 14.15 [#/sec] (mean)
Time per request: 2260.875 [ms] (mean)
Time per request: 70.652 [ms] (mean, across all concurrent requests)
Apache + mod_perl (ModPerl::PerlRegistry):
Requests per second: 154.48 [#/sec] (mean)
Time per request: 207.140 [ms] (mean)
Time per request: 6.473 [ms] (mean, across all concurrent requests)
Running some benchmarks across various parts of my site made me realise I also had some RewriteRules in the apache config that still had H=cgi-script - changed those to H=perl-script and saw similar improvements:
ExecCGI - Requests per second: 11.84 [#/sec] (mean)
mod_perl - Requests per second: 130.97 [#/sec] (mean)
That's quite some gains for a days work.
--
Steven Haigh 📧 netwiz@crc.id.au<ma...@crc.id.au> 💻 https://www.crc.id.au [crc.id.au]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=kwRJ9OU2GaHG3eTqlSv3NGFCM8bbDVVw6VWYGivfszw&s=xsyidls-ez8g3gIshJPO4FIGzGjiIp_qnzQ1yujeNK4&e=>
On Sun, Feb 7, 2021 at 23:58, Steven Haigh <ne...@crc.id.au>> wrote:
Interestingly, I did get things working with ModPerl::PerlRegistry.
What I couldn't find *anywhere* is that the data I was loading in Template Toolkit was included in the file in the __DATA__ area - which causes mod_perl to fall over!
The only way I managed to find this was the following error in the *system* /var/log/httpd/error_log (didn't show up in the vhost error_log!):
readline() on unopened filehandle DATA at /usr/lib64/perl5/vendor_perl/Template/Provider.pm line 638.
Took me a LONG time to find a vague post that reading in lines from <DATA> kills mod_perl. Not sure why - but I stripped all the templates out and put them in a file instead and re-wrote that bit of code, and things started working.
I had to fix a few lib path issues, but after getting my head around that, most things seem to work as before - however I don't notice much of an improvement in execution times, I do see this improvement using 'ab -n 100 -c32':
Apache + ExecCGI: Requests per second: 13.50 [#/sec] (mean)
Apache + mod_perl: Requests per second: 59.81 [#/sec] (mean)
This is obviously a good thing.
I haven't gotten into the preload or DBI sharing yet - as that'll end up needing a bit of a rewrite of code to take advantage of. I'd be open to suggestions here from those who have done it in the past to save me going down some dead ends :D
--
Steven Haigh 📧 netwiz@crc.id.au<ma...@crc.id.au> 💻 https://www.crc.id.au [crc.id.au]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=kwRJ9OU2GaHG3eTqlSv3NGFCM8bbDVVw6VWYGivfszw&s=xsyidls-ez8g3gIshJPO4FIGzGjiIp_qnzQ1yujeNK4&e=>
On Sun, Feb 7, 2021 at 12:49, James Smith <js...@sanger.acuk>> wrote:
As welsey said – try Registry, that was the standard way of using mod_perl to cache perl in the server – but your problem might be due to the note in PerlRun…
https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description [perl.apache.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__perl.apache.org_docs_2.0_api_ModPerl_PerlRun.html-23Description&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=kwRJ9OU2GaHG3eTqlSv3NGFCM8bbDVVw6VWYGivfszw&s=Q2A6j7uCIWTtGhG1FpeG0VpNOpux6GBFmS0kpJslum4&e=>
META: document that for now we don't chdir() into the script's dir, because it affects the whole process under threads. ModPerl::PerlRunPrefork [perl.apache.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__perl.apache.org_docs_20_api_ModPerl_PerlRunPrefork.html&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=kwRJ9OU2GaHG3eTqlSv3NGFCM8bbDVVw6VWYGivfszw&s=G7IOIdWoxLg27VmZ1YKmv0HfFWgABanbrrJUbQ1OvwQ&e=> should be used by those who run only under prefork MPM.
{tbh most people don’t use mod perl under threads anyway as there isn’t really a gain from using them}
It suggests you use ModPerl/PerlRunPrefork – as this does an additional step to cd to the script directory – which might be your issue….
From: Steven Haigh <ne...@crc.id.au>>
Sent: 07 February 2021 01:00
To: modperl@perl.apache.org<ma...@perl.apache.org>
Subject: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
Hi all,
So for many years I've been slack and writing perl scripts to do various things - but never needed more than the normal apache +ExecCGI and Template Toolkit.
One of my sites has become a bit more popular, so I'd like to spend a bit of time on performance. Currently, I'm seeing ~300-400ms of what I believe to be execution time of the script loading, running, and then blatting its output to STDOUT and the browser can go do its thing.
I believe most of the delay would be to do with loading perl, its modules etc etc
I know that the current trend would be to re-write the entire site in a more modern, daemon based solution - and I started down the Mojolicious path - but the amount of re-writing to save 1/3rd of a second seems to be excessive
Would I be correct in thinking that mod_perl would help in this case?
I did try a basic test, but I have a 'use functions' in all my scripts that loads a .pm with some global vars and a lot of common subs - and for whatever reason (can't find anything on Google as to why), none of the subs are recognised in the main script when loaded via ModPerl::PerlRun.
So throwing it out to the list - am I on the right track? wasting my time? or just a simple mistake?
--
Steven Haigh 📧 netwiz@crc.id.au<ma...@crc.id.au> 💻 https://www.crc.id.au [crc.id.au]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=bosoTbkecbnrPukObNK-5Duc1p3JTllIM7_FHhBYKW4&s=vQDi0ezyEZDOz86GVraerPdT76UjN2in3UdPh8fglRM&e=>
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Re:
Posted by Adam Prime <ad...@utoronto.ca>.
There is one other thing you can do relatively easily that may get you a marginal gain when Apache spins up new children. Load some or all of your Perl dependencies before Apache forks. There is also an opportunity to load other static resources at this time, but that can get a little more involved and/or confusing. There is some documentation here:
https://perl.apache.org/docs/2.0/user/handlers/server.html#Startup_File
Adam
> On Feb 7, 2021, at 8:15 AM, Steven Haigh <ne...@crc.id.au> wrote:
>
>
> In fact, I just realised that 'ab' test is rather restrictive.... So here's a bit more of an extended test:
>
> # ab -k -n 1000 -c 32
>
> Apache + ExecCGI:
> Requests per second: 14.26 [#/sec] (mean)
> Time per request: 2244.181 [ms] (mean)
> Time per request: 70.131 [ms] (mean, across all concurrent requests)
>
> Apache + mod_perl (ModPerl::PerlRegistry):
> Requests per second: 132.14 [#/sec] (mean)
> Time per request: 242.175 [ms] (mean)
> Time per request: 7.568 [ms] (mean, across all concurrent requests)
>
> Interestingly, without Keepalives, the story is much the same:
>
> # ab -n 1000 -c 32
>
> Apache + ExecCGI:
> Requests per second: 14.15 [#/sec] (mean)
> Time per request: 2260.875 [ms] (mean)
> Time per request: 70.652 [ms] (mean, across all concurrent requests)
>
> Apache + mod_perl (ModPerl::PerlRegistry):
> Requests per second: 154.48 [#/sec] (mean)
> Time per request: 207.140 [ms] (mean)
> Time per request: 6.473 [ms] (mean, across all concurrent requests)
>
> Running some benchmarks across various parts of my site made me realise I also had some RewriteRules in the apache config that still had H=cgi-script - changed those to H=perl-script and saw similar improvements:
>
> ExecCGI - Requests per second: 11.84 [#/sec] (mean)
> mod_perl - Requests per second: 130.97 [#/sec] (mean)
>
> That's quite some gains for a days work.
>
> --
> Steven Haigh
>
> 📧 netwiz@crc.id.au
> 💻 https://www.crc.id.au
>
>> On Sun, Feb 7, 2021 at 23:58, Steven Haigh <ne...@crc.id.au> wrote:
>> Interestingly, I did get things working with ModPerl::PerlRegistry.
>>
>> What I couldn't find *anywhere* is that the data I was loading in Template Toolkit was included in the file in the __DATA__ area - which causes mod_perl to fall over!
>>
>> The only way I managed to find this was the following error in the *system* /var/log/httpd/error_log (didn't show up in the vhost error_log!):
>> readline() on unopened filehandle DATA at /usr/lib64/perl5/vendor_perl/Template/Provider.pm line 638.
>>
>> Took me a LONG time to find a vague post that reading in lines from <DATA> kills mod_perl. Not sure why - but I stripped all the templates out and put them in a file instead and re-wrote that bit of code, and things started working.
>>
>> I had to fix a few lib path issues, but after getting my head around that, most things seem to work as before - however I don't notice much of an improvement in execution times, I do see this improvement using 'ab -n 100 -c32':
>>
>> Apache + ExecCGI: Requests per second: 13.50 [#/sec] (mean)
>> Apache + mod_perl: Requests per second: 59.81 [#/sec] (mean)
>>
>> This is obviously a good thing.
>>
>> I haven't gotten into the preload or DBI sharing yet - as that'll end up needing a bit of a rewrite of code to take advantage of. I'd be open to suggestions here from those who have done it in the past to save me going down some dead ends :D
>>
>> --
>> Steven Haigh
>>
>> 📧 netwiz@crc.id.au
>> 💻 https://www.crc.id.au
>>
>>> On Sun, Feb 7, 2021 at 12:49, James Smith <js...@sanger.acuk> wrote:
>>> As welsey said – try Registry, that was the standard way of using mod_perl to cache perl in the server – but your problem might be due to the note in PerlRun…
>>>
>>> https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description
>>> META: document that for now we don't chdir() into the script's dir, because it affects the whole process under threads. ModPerl::PerlRunPrefork should be used by those who run only under prefork MPM.
>>> {tbh most people don’t use mod perl under threads anyway as there isn’t really a gain from using them}
>>>
>>> It suggests you use ModPerl/PerlRunPrefork – as this does an additional step to cd to the script directory – which might be your issue….
>>>
>>> From: Steven Haigh <ne...@crc.id.au>
>>> Sent: 07 February 2021 01:00
>>> To: modperl@perl.apache.org
>>> Subject: Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
>>>
>>> Hi all,
>>>
>>> So for many years I've been slack and writing perl scripts to do various things - but never needed more than the normal apache +ExecCGI and Template Toolkit.
>>>
>>> One of my sites has become a bit more popular, so I'd like to spend a bit of time on performance. Currently, I'm seeing ~300-400ms of what I believe to be execution time of the script loading, running, and then blatting its output to STDOUT and the browser can go do its thing.
>>>
>>> I believe most of the delay would be to do with loading perl, its modules etc etc
>>>
>>> I know that the current trend would be to re-write the entire site in a more modern, daemon based solution - and I started down the Mojolicious path - but the amount of re-writing to save 1/3rd of a second seems to be excessive
>>>
>>> Would I be correct in thinking that mod_perl would help in this case?
>>>
>>> I did try a basic test, but I have a 'use functions' in all my scripts that loads a .pm with some global vars and a lot of common subs - and for whatever reason (can't find anything on Google as to why), none of the subs are recognised in the main script when loaded via ModPerl::PerlRun.
>>>
>>> So throwing it out to the list - am I on the right track? wasting my time? or just a simple mistake?
>>>
>>> --
>>> Steven Haigh 📧 netwiz@crc.id.au 💻 https://www.crc.id.au [crc.id.au]
>>> -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
RE: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
Posted by James Smith <js...@sanger.ac.uk>.
That is a good sign – I would run with brutal at least once and see what it throws up
We tend to ignore a couple of the warnings – one is postfix if/unless and the other is multiline strings {we embed a lot of simple HTML templates in code and it means I can make the HTML readable when rendered rather than being one long string, plus SQL queries are more readable and heredocs are messy if you want to do concatenation or lots of printf/sprintf calls}
We have it as part of our svn commit pre-hooks so that people can’t push **** code to our repos and break things {one of the reasons we haven’t moved to git! as hooks are a bit messier… and people may not have the right software on the machines they have their repos on}
From: Steven Haigh <ne...@crc.id.au>
Sent: 08 February 2021 09:54
To: modperl@perl.apache.org
Subject: RE: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
On Mon, Feb 8, 2021 at 09:13, James Smith <js...@sanger.ac.uk>> wrote:
Use perl-critic this will find most of the nasties that you have the classic is:
Thanks for the tip! I have no idea how long I've been writing stuff in perl - and I never knew of this!
I ran it with the -3 option - which I figure is a good middle ground...
The good news, I just ran it over a lot of my code and it seems the only real things it picks up are not having a /x on the end of regex matches, using hard tabs, and multiline strings. I'd say that's a good sign.
It did pick up a couple of open statements that I didn't have a close for (*slaps wrist*), but I haven't seen much in the way of what looks to be major issues.
I was trying to find the PBP references - and was amazed that the Perl Best Practices *ebook* s $56.20 AUD hahahah
Amazon has a few copies listed second hand, with 3 weeks shipping.... The joys of being on an island a long way from anything ;)
--
Steven Haigh 📧 netwiz@crc.id.au<ma...@crc.id.au> 💻 https://www.crc.id.au [crc.id.au]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=e5xV9ANT9Dnmf1ept9VmObBssOhOe3Cci6goE99x62c&s=bK6vilo7Ud3M2_JhyA9RucR6k68i7T0pLK49RbBQbjY&e=>
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
RE: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
Posted by Steven Haigh <ne...@crc.id.au>.
On Mon, Feb 8, 2021 at 09:13, James Smith <js...@sanger.ac.uk> wrote:
> Use perl-critic this will find most of the nasties that you have the
> classic is:
Thanks for the tip! I have no idea how long I've been writing stuff in
perl - and I never knew of this!
I ran it with the -3 option - which I figure is a good middle ground...
The good news, I just ran it over a lot of my code and it seems the
only real things it picks up are not having a /x on the end of regex
matches, using hard tabs, and multiline strings. I'd say that's a good
sign.
It did pick up a couple of open statements that I didn't have a close
for (*slaps wrist*), but I haven't seen much in the way of what looks
to be major issues.
I was trying to find the PBP references - and was amazed that the Perl
Best Practices *ebook* s $56.20 AUD hahahah
Amazon has a few copies listed second hand, with 3 weeks shipping....
The joys of being on an island a long way from anything ;)
--
Steven Haigh
📧 netwiz@crc.id.au <ma...@crc.id.au>
💻 https://www.crc.id.au <https://www.crc.id.au/>
RE: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
Posted by James Smith <js...@sanger.ac.uk>.
Use perl-critic this will find most of the nasties that you have the classic is:
my $var = {code} if {condition};
The my gets round perl strict, but $var doesn’t get updated if {condition} isn’t met, so holds the variable from the last time round..
Better is
my $var = ‘’;
$var = {code} if {condition};
or
my $var = {condition} ? {code} : ‘’;
From: Steven Haigh <ne...@crc.id.au>
Sent: 08 February 2021 09:09
To: modperl@perl.apache.org
Subject: Re: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
On Sun, Feb 7, 2021 at 15:17, Chris <cp...@bennettconstruction.us>> wrote:
Just remember to always write clean code that resets variables after doing tasks.
I'm a bit curious about this - whilst I'm still testing all this on a staging environment, how can I tell if things can leak between runs?
Is coding to normal 'use strict; use warnings;' standards good enough?
Are there other ways to confirm correct operations?
--
Steven Haigh 📧 netwiz@crc.id.au<ma...@crc.id.au> 💻 https://www.crc.id.au [crc.id.au]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=4WgSuD9BTWr09e71eanf57nyBNHOPXE7hNOgLlKJcA4&s=UVVXHyoyL1iyknEcMp0sf6Nm1yO69SF8h4h4XDhcGeo&e=>
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Re: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
Posted by "André Warnier (tomcat/perl)" <aw...@ice-sa.com>.
On 08.02.2021 10:09, Steven Haigh wrote:
> On Sun, Feb 7, 2021 at 15:17, Chris <cp...@bennettconstruction.us> wrote:
>> Just remember to always write clean code that resets variables after doing tasks.
>
> I'm a bit curious about this - whilst I'm still testing all this on a staging environment,
> how can I tell if things can leak between runs?
>
> Is coding to normal 'use strict; use warnings;' standards good enough?
Hi. Read this, carefully :
http://perl.apache.org/docs/2.0/user/troubleshooting/troubleshooting.html#Variable__x_will_not_stay_shared_at
In particular the example below "An Easy Break-in".
(and replace
use vars ($authenticated);
by
our $authenticated;
)
("use vars" is deprecated, see https://perldoc.perl.org/vars)
The point is :
- imagine an Apache Prefork starting 5 children
- each child, when it starts, contains its own fresh copy of the perl interpreter
- requests which come in, are directed by the main Apache, to any child that is free at
the time.
- when a child runs your script/module for the first time (with its particular perl
interpreter), the script/module gets compiled, and then run, and the compiled code is
cached by the corresponding perl interpreter
- global variables (such as $authenticated here) get defined during compilation, so are
part of the cached script/module code.
- so the first time an Apache child runs your script, $authenticated is defined, but
"empty" (undef). Then when the script runs, it assigns a value to it, so it is no longer
undef.
- the next time your script is run *by the same Apache child*, the cached compiled version
of the script is used (*), which already has $authenticated defined, and the previous
value (set by the previous run) is still in it.
However, if it happens that the next request is run by another "fresh" Apache child, that
one (its own instance of the perl interpreter) does not yet have a pre-compiled version of
your script, so it gets compiled again, and in that instance $authenticated gets defined
again, empty.
Since you cannot control which Apache child runs your script the next time you issue a
request, the result may appear random (as far as $authenticated is concerned).
This "feature" can sometimes be very useful as an optimisation (for example, if you want
to initialise a complex read-only structure only once per Apache child life), but in the
general case, it will lead to strange things happening if you are not careful.
So to answer your question : "> how can I tell if things can leak between runs?",
a quick answer would be : just *assume* that everything "leaks" between runs, and make
sure that you initialise every variable in a known way, before using it.
mod_perl is great fun, and the ability to run perl scripts much faster is only the tip of
the iceberg. But like every fun thing, it has some minor quirks like that. Do not let
them discourage you.
(*) that's the point, and that's why it is much faster
>
> Are there other ways to confirm correct operations?
>
> --
> Steven Haigh 📧 netwiz@crc.id.au <ma...@crc.id.au>💻 https://www.crc.id.au
> <https://www.crc.id.au/>
Re: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
Posted by Steven Haigh <ne...@crc.id.au>.
On Sun, Feb 7, 2021 at 15:17, Chris
<cp...@bennettconstruction.us> wrote:
> Just remember to always write clean code that resets variables after
> doing tasks.
I'm a bit curious about this - whilst I'm still testing all this on a
staging environment, how can I tell if things can leak between runs?
Is coding to normal 'use strict; use warnings;' standards good enough?
Are there other ways to confirm correct operations?
--
Steven Haigh
📧 netwiz@crc.id.au <ma...@crc.id.au>
💻 https://www.crc.id.au <https://www.crc.id.au/>
RE: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
Posted by James Smith <js...@sanger.ac.uk>.
Agree in this - you need to always think that a mod_perl app is running in a loop where each loop is an iteration, so if you don't initialise something at the start of the script - it can have the value at the end of it's last call.... Use Perl critic it is a good one to find your gotchas...
Try
my $admin = 1 if $user_is_logged_in and $user eq 'admin';
may seem innocent in as script and compiles even under use strict... but now put this in a loop
1st request user is logged in and is admin - this gets set to true.
2nd request user isn't logged in - but this doesn't get reset to "false"....
-----Original Message-----
From: Chris <cp...@bennettconstruction.us>
Sent: 07 February 2021 21:18
To: modperl@perl.apache.org
Subject: Re: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
On Sun, Feb 07, 2021 at 09:21:41PM +0800, Wesley Peng wrote:
> If you can take time to rewrite all codes with modPerl handlers, that will improve performance a lot.
I've never used Template, but I just eventually wrote all handlers.
I moved from Registry to all handlers, bit by bit.
You can mix Registry and handlers together without any problems.
Just remember to always write clean code that resets variables after doing tasks. The same code runs multiple times. Variables might retain old values. It's a good habit to keep for all perl code, even outside of mod_perl.
I considered changing all of my mod_perl code to something newer, but I decided to just keep it. No regrets.
Chris
>
> On Sun, Feb 7, 2021, at 9:14 PM, Steven Haigh wrote:
> > In fact, I just realised that 'ab' test is rather restrictive.... So here's a bit more of an extended test:
> >
> > # ab -k -n 1000 -c 32
> >
> > Apache + ExecCGI:
> > Requests per second: 14.26 [#/sec] (mean)
> > Time per request: 2244.181 [ms] (mean)
> > Time per request: 70.131 [ms] (mean, across all concurrent requests)
> >
> > Apache + mod_perl (ModPerl::PerlRegistry):
> > Requests per second: 132.14 [#/sec] (mean)
> > Time per request: 242.175 [ms] (mean)
> > Time per request: 7.568 [ms] (mean, across all concurrent requests)
> >
> > Interestingly, without Keepalives, the story is much the same:
> >
> > # ab -n 1000 -c 32
> >
> > Apache + ExecCGI:
> > Requests per second: 14.15 [#/sec] (mean)
> > Time per request: 2260.875 [ms] (mean)
> > Time per request: 70.652 [ms] (mean, across all concurrent requests)
> >
> > Apache + mod_perl (ModPerl::PerlRegistry):
> > Requests per second: 154.48 [#/sec] (mean)
> > Time per request: 207.140 [ms] (mean)
> > Time per request: 6.473 [ms] (mean, across all concurrent requests)
> >
> > Running some benchmarks across various parts of my site made me realise I also had some RewriteRules in the apache config that still had H=cgi-script - changed those to H=perl-script and saw similar improvements:
> >
> > ExecCGI - Requests per second: 11.84 [#/sec] (mean)
> > mod_perl - Requests per second: 130.97 [#/sec] (mean)
> >
> > That's quite some gains for a days work.
> >
> > --
> > Steven Haigh 📧 netwiz@crc.id.au 💻
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au&d
> > =DwIDaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj
> > 4oDX0XM7vQ&m=jUH_UYtcqlwO076IYfxVYrdow4cIntukmuDhs07wzzE&s=EheaEdaj-
> > 0DWuhH62PQjVTVfPw5WRAazyGkJFaYUC8E&e=
> >
> > On Sun, Feb 7, 2021 at 23:58, Steven Haigh <ne...@crc.id.au> wrote:
> >> Interestingly, I did get things working with ModPerl::PerlRegistry.
> >>
> >> What I couldn't find *anywhere* is that the data I was loading in Template Toolkit was included in the file in the __DATA__ area - which causes mod_perl to fall over!
> >>
> >> The only way I managed to find this was the following error in the *system* /var/log/httpd/error_log (didn't show up in the vhost error_log!):
> >> readline() on unopened filehandle DATA at /usr/lib64/perl5/vendor_perl/Template/Provider.pm line 638.
> >>
> >> Took me a LONG time to find a vague post that reading in lines from <DATA> kills mod_perl. Not sure why - but I stripped all the templates out and put them in a file instead and re-wrote that bit of code, and things started working.
> >>
> >> I had to fix a few lib path issues, but after getting my head around that, most things seem to work as before - however I don't notice much of an improvement in execution times, I do see this improvement using 'ab -n 100 -c32':
> >>
> >> Apache + ExecCGI: Requests per second: 13.50 [#/sec] (mean)
> >> Apache + mod_perl: Requests per second: 59.81 [#/sec] (mean)
> >>
> >> This is obviously a good thing.
> >>
> >> I haven't gotten into the preload or DBI sharing yet - as that'll
> >> end up needing a bit of a rewrite of code to take advantage of. I'd
> >> be open to suggestions here from those who have done it in the past
> >> to save me going down some dead ends :D
> >>
> >> --
> >> Steven Haigh 📧 netwiz@crc.id.au 💻
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au&
> >> d=DwIDaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1e
> >> cj4oDX0XM7vQ&m=jUH_UYtcqlwO076IYfxVYrdow4cIntukmuDhs07wzzE&s=EheaEd
> >> aj-0DWuhH62PQjVTVfPw5WRAazyGkJFaYUC8E&e=
> >>
> >> On Sun, Feb 7, 2021 at 12:49, James Smith <js...@sanger.acuk> wrote:
> >>> As welsey said – try Registry, that was the standard way of using
> >>> mod_perl to cache perl in the server – but your problem might be
> >>> due to the note in PerlRun…
> >>>
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__perl.apache.o
> >>> rg_docs_2.0_api_ModPerl_PerlRun.html-23Description&d=DwIDaQ&c=D7By
> >>> GjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m
> >>> =jUH_UYtcqlwO076IYfxVYrdow4cIntukmuDhs07wzzE&s=X28W3EsAPwLXjwa4Im6
> >>> 8Ak7wzU4ss-EYYegNaEoii18&e=
> >>> META: document that for now we don't chdir() into the script's dir, because it affects the whole process under threads. `ModPerl::PerlRunPrefork <https://urldefense.proofpoint.com/v2/url?u=https-3A__perl.apache.org_docs_20_api_ModPerl_PerlRunPrefork.html&d=DwIDaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=jUH_UYtcqlwO076IYfxVYrdow4cIntukmuDhs07wzzE&s=UxzvtZHkq8JTl8nKRcp4326um33tXkdKZsEU6YlmFhQ&e= >` should be used by those who run only under prefork MPM.
> >>> {tbh most people don’t use mod perl under threads anyway as there
> >>> isn’t really a gain from using them}
> >>>
> >>> It suggests you use ModPerl/PerlRunPrefork – as this does an additional step to cd to the script directory – which might be your issue….
>
> >>>
>
> >>> *From:* Steven Haigh <ne...@crc.id.au>
> >>> *Sent:* 07 February 2021 01:00
> >>> *To:* modperl@perl.apache.org
> >>> *Subject:* Moving ExecCGI to mod_perl - performance and custom
> >>> 'modules' [EXT]
>
> >>>
>
> >>> Hi all,
>
> >>>
>
> >>> So for many years I've been slack and writing perl scripts to do various things - but never needed more than the normal apache +ExecCGI and Template Toolkit.
>
> >>>
>
> >>> One of my sites has become a bit more popular, so I'd like to spend a bit of time on performance. Currently, I'm seeing ~300-400ms of what I believe to be execution time of the script loading, running, and then blatting its output to STDOUT and the browser can go do its thing.
>
> >>>
>
> >>> I believe most of the delay would be to do with loading perl, its
> >>> modules etc etc
>
> >>>
>
> >>> I know that the current trend would be to re-write the entire site
> >>> in a more modern, daemon based solution - and I started down the
> >>> Mojolicious path - but the amount of re-writing to save 1/3rd of a
> >>> second seems to be excessive
>
> >>>
>
> >>> Would I be correct in thinking that mod_perl would help in this case?
>
> >>>
>
> >>> I did try a basic test, but I have a 'use functions' in all my scripts that loads a .pm with some global vars and a lot of common subs - and for whatever reason (can't find anything on Google as to why), none of the subs are recognised in the main script when loaded via ModPerl::PerlRun.
>
> >>>
>
> >>> So throwing it out to the list - am I on the right track? wasting my time? or just a simple mistake?
>
> >>>
>
> >>> --
>
> >>> Steven Haigh 📧 netwiz@crc.id.au 💻
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au
> >>> &d=DwIDaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge
> >>> 1ecj4oDX0XM7vQ&m=jUH_UYtcqlwO076IYfxVYrdow4cIntukmuDhs07wzzE&s=Ehe
> >>> aEdaj-0DWuhH62PQjVTVfPw5WRAazyGkJFaYUC8E&e= [crc.id.au]
> >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.a
> >>> u_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0
> >>> ge1ecj4oDX0XM7vQ&m=bosoTbkecbnrPukObNK-5Duc1p3JTllIM7_FHhBYKW4&s=v
> >>> QDi0ezyEZDOz86GVraerPdT76UjN2in3UdPh8fglRM&e=>
>
> >>> -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
--
The Wellcome Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
Re: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
Posted by Chris <cp...@bennettconstruction.us>.
On Sun, Feb 07, 2021 at 09:21:41PM +0800, Wesley Peng wrote:
> If you can take time to rewrite all codes with modPerl handlers, that will improve performance a lot.
I've never used Template, but I just eventually wrote all handlers.
I moved from Registry to all handlers, bit by bit.
You can mix Registry and handlers together without any problems.
Just remember to always write clean code that resets variables after
doing tasks. The same code runs multiple times. Variables might retain
old values. It's a good habit to keep for all perl code, even outside of
mod_perl.
I considered changing all of my mod_perl code to something newer, but I
decided to just keep it. No regrets.
Chris
>
> On Sun, Feb 7, 2021, at 9:14 PM, Steven Haigh wrote:
> > In fact, I just realised that 'ab' test is rather restrictive.... So here's a bit more of an extended test:
> >
> > # ab -k -n 1000 -c 32
> >
> > Apache + ExecCGI:
> > Requests per second: 14.26 [#/sec] (mean)
> > Time per request: 2244.181 [ms] (mean)
> > Time per request: 70.131 [ms] (mean, across all concurrent requests)
> >
> > Apache + mod_perl (ModPerl::PerlRegistry):
> > Requests per second: 132.14 [#/sec] (mean)
> > Time per request: 242.175 [ms] (mean)
> > Time per request: 7.568 [ms] (mean, across all concurrent requests)
> >
> > Interestingly, without Keepalives, the story is much the same:
> >
> > # ab -n 1000 -c 32
> >
> > Apache + ExecCGI:
> > Requests per second: 14.15 [#/sec] (mean)
> > Time per request: 2260.875 [ms] (mean)
> > Time per request: 70.652 [ms] (mean, across all concurrent requests)
> >
> > Apache + mod_perl (ModPerl::PerlRegistry):
> > Requests per second: 154.48 [#/sec] (mean)
> > Time per request: 207.140 [ms] (mean)
> > Time per request: 6.473 [ms] (mean, across all concurrent requests)
> >
> > Running some benchmarks across various parts of my site made me realise I also had some RewriteRules in the apache config that still had H=cgi-script - changed those to H=perl-script and saw similar improvements:
> >
> > ExecCGI - Requests per second: 11.84 [#/sec] (mean)
> > mod_perl - Requests per second: 130.97 [#/sec] (mean)
> >
> > That's quite some gains for a days work.
> >
> > --
> > Steven Haigh 📧 netwiz@crc.id.au 💻 https://www.crc.id.au
> >
> > On Sun, Feb 7, 2021 at 23:58, Steven Haigh <ne...@crc.id.au> wrote:
> >> Interestingly, I did get things working with ModPerl::PerlRegistry.
> >>
> >> What I couldn't find *anywhere* is that the data I was loading in Template Toolkit was included in the file in the __DATA__ area - which causes mod_perl to fall over!
> >>
> >> The only way I managed to find this was the following error in the *system* /var/log/httpd/error_log (didn't show up in the vhost error_log!):
> >> readline() on unopened filehandle DATA at /usr/lib64/perl5/vendor_perl/Template/Provider.pm line 638.
> >>
> >> Took me a LONG time to find a vague post that reading in lines from <DATA> kills mod_perl. Not sure why - but I stripped all the templates out and put them in a file instead and re-wrote that bit of code, and things started working.
> >>
> >> I had to fix a few lib path issues, but after getting my head around that, most things seem to work as before - however I don't notice much of an improvement in execution times, I do see this improvement using 'ab -n 100 -c32':
> >>
> >> Apache + ExecCGI: Requests per second: 13.50 [#/sec] (mean)
> >> Apache + mod_perl: Requests per second: 59.81 [#/sec] (mean)
> >>
> >> This is obviously a good thing.
> >>
> >> I haven't gotten into the preload or DBI sharing yet - as that'll end up needing a bit of a rewrite of code to take advantage of. I'd be open to suggestions here from those who have done it in the past to save me going down some dead ends :D
> >>
> >> --
> >> Steven Haigh 📧 netwiz@crc.id.au 💻 https://www.crc.id.au
> >>
> >> On Sun, Feb 7, 2021 at 12:49, James Smith <js...@sanger.acuk> wrote:
> >>> As welsey said – try Registry, that was the standard way of using mod_perl to cache perl in the server – but your problem might be due to the note in PerlRun…
> >>>
> >>> https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description
> >>> META: document that for now we don't chdir() into the script's dir, because it affects the whole process under threads. `ModPerl::PerlRunPrefork <https://perl.apache.org/docs/20/api/ModPerl/PerlRunPrefork.html>` should be used by those who run only under prefork MPM.
> >>> {tbh most people don’t use mod perl under threads anyway as there isn’t really a gain from using them}
> >>>
> >>> It suggests you use ModPerl/PerlRunPrefork – as this does an additional step to cd to the script directory – which might be your issue….
>
> >>>
>
> >>> *From:* Steven Haigh <ne...@crc.id.au>
> >>> *Sent:* 07 February 2021 01:00
> >>> *To:* modperl@perl.apache.org
> >>> *Subject:* Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
>
> >>>
>
> >>> Hi all,
>
> >>>
>
> >>> So for many years I've been slack and writing perl scripts to do various things - but never needed more than the normal apache +ExecCGI and Template Toolkit.
>
> >>>
>
> >>> One of my sites has become a bit more popular, so I'd like to spend a bit of time on performance. Currently, I'm seeing ~300-400ms of what I believe to be execution time of the script loading, running, and then blatting its output to STDOUT and the browser can go do its thing.
>
> >>>
>
> >>> I believe most of the delay would be to do with loading perl, its modules etc etc
>
> >>>
>
> >>> I know that the current trend would be to re-write the entire site in a more modern, daemon based solution - and I started down the Mojolicious path - but the amount of re-writing to save 1/3rd of a second seems to be excessive
>
> >>>
>
> >>> Would I be correct in thinking that mod_perl would help in this case?
>
> >>>
>
> >>> I did try a basic test, but I have a 'use functions' in all my scripts that loads a .pm with some global vars and a lot of common subs - and for whatever reason (can't find anything on Google as to why), none of the subs are recognised in the main script when loaded via ModPerl::PerlRun.
>
> >>>
>
> >>> So throwing it out to the list - am I on the right track? wasting my time? or just a simple mistake?
>
> >>>
>
> >>> --
>
> >>> Steven Haigh 📧 netwiz@crc.id.au 💻 https://www.crc.id.au [crc.id.au] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=bosoTbkecbnrPukObNK-5Duc1p3JTllIM7_FHhBYKW4&s=vQDi0ezyEZDOz86GVraerPdT76UjN2in3UdPh8fglRM&e=>
>
> >>> -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
Re: Moving ExecCGI to mod_perl - performance and custom'modules'[EXT]
Posted by Wesley Peng <we...@pengfamily.de>.
If you can take time to rewrite all codes with modPerl handlers, that will improve performance a lot.
On Sun, Feb 7, 2021, at 9:14 PM, Steven Haigh wrote:
> In fact, I just realised that 'ab' test is rather restrictive.... So here's a bit more of an extended test:
>
> # ab -k -n 1000 -c 32
>
> Apache + ExecCGI:
> Requests per second: 14.26 [#/sec] (mean)
> Time per request: 2244.181 [ms] (mean)
> Time per request: 70.131 [ms] (mean, across all concurrent requests)
>
> Apache + mod_perl (ModPerl::PerlRegistry):
> Requests per second: 132.14 [#/sec] (mean)
> Time per request: 242.175 [ms] (mean)
> Time per request: 7.568 [ms] (mean, across all concurrent requests)
>
> Interestingly, without Keepalives, the story is much the same:
>
> # ab -n 1000 -c 32
>
> Apache + ExecCGI:
> Requests per second: 14.15 [#/sec] (mean)
> Time per request: 2260.875 [ms] (mean)
> Time per request: 70.652 [ms] (mean, across all concurrent requests)
>
> Apache + mod_perl (ModPerl::PerlRegistry):
> Requests per second: 154.48 [#/sec] (mean)
> Time per request: 207.140 [ms] (mean)
> Time per request: 6.473 [ms] (mean, across all concurrent requests)
>
> Running some benchmarks across various parts of my site made me realise I also had some RewriteRules in the apache config that still had H=cgi-script - changed those to H=perl-script and saw similar improvements:
>
> ExecCGI - Requests per second: 11.84 [#/sec] (mean)
> mod_perl - Requests per second: 130.97 [#/sec] (mean)
>
> That's quite some gains for a days work.
>
> --
> Steven Haigh 📧 netwiz@crc.id.au 💻 https://www.crc.id.au
>
> On Sun, Feb 7, 2021 at 23:58, Steven Haigh <ne...@crc.id.au> wrote:
>> Interestingly, I did get things working with ModPerl::PerlRegistry.
>>
>> What I couldn't find *anywhere* is that the data I was loading in Template Toolkit was included in the file in the __DATA__ area - which causes mod_perl to fall over!
>>
>> The only way I managed to find this was the following error in the *system* /var/log/httpd/error_log (didn't show up in the vhost error_log!):
>> readline() on unopened filehandle DATA at /usr/lib64/perl5/vendor_perl/Template/Provider.pm line 638.
>>
>> Took me a LONG time to find a vague post that reading in lines from <DATA> kills mod_perl. Not sure why - but I stripped all the templates out and put them in a file instead and re-wrote that bit of code, and things started working.
>>
>> I had to fix a few lib path issues, but after getting my head around that, most things seem to work as before - however I don't notice much of an improvement in execution times, I do see this improvement using 'ab -n 100 -c32':
>>
>> Apache + ExecCGI: Requests per second: 13.50 [#/sec] (mean)
>> Apache + mod_perl: Requests per second: 59.81 [#/sec] (mean)
>>
>> This is obviously a good thing.
>>
>> I haven't gotten into the preload or DBI sharing yet - as that'll end up needing a bit of a rewrite of code to take advantage of. I'd be open to suggestions here from those who have done it in the past to save me going down some dead ends :D
>>
>> --
>> Steven Haigh 📧 netwiz@crc.id.au 💻 https://www.crc.id.au
>>
>> On Sun, Feb 7, 2021 at 12:49, James Smith <js...@sanger.acuk> wrote:
>>> As welsey said – try Registry, that was the standard way of using mod_perl to cache perl in the server – but your problem might be due to the note in PerlRun…
>>>
>>> https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description
>>> META: document that for now we don't chdir() into the script's dir, because it affects the whole process under threads. `ModPerl::PerlRunPrefork <https://perl.apache.org/docs/20/api/ModPerl/PerlRunPrefork.html>` should be used by those who run only under prefork MPM.
>>> {tbh most people don’t use mod perl under threads anyway as there isn’t really a gain from using them}
>>>
>>> It suggests you use ModPerl/PerlRunPrefork – as this does an additional step to cd to the script directory – which might be your issue….
>>>
>>> *From:* Steven Haigh <ne...@crc.id.au>
>>> *Sent:* 07 February 2021 01:00
>>> *To:* modperl@perl.apache.org
>>> *Subject:* Moving ExecCGI to mod_perl - performance and custom 'modules' [EXT]
>>>
>>> Hi all,
>>>
>>> So for many years I've been slack and writing perl scripts to do various things - but never needed more than the normal apache +ExecCGI and Template Toolkit.
>>>
>>> One of my sites has become a bit more popular, so I'd like to spend a bit of time on performance. Currently, I'm seeing ~300-400ms of what I believe to be execution time of the script loading, running, and then blatting its output to STDOUT and the browser can go do its thing.
>>>
>>> I believe most of the delay would be to do with loading perl, its modules etc etc
>>>
>>> I know that the current trend would be to re-write the entire site in a more modern, daemon based solution - and I started down the Mojolicious path - but the amount of re-writing to save 1/3rd of a second seems to be excessive
>>>
>>> Would I be correct in thinking that mod_perl would help in this case?
>>>
>>> I did try a basic test, but I have a 'use functions' in all my scripts that loads a .pm with some global vars and a lot of common subs - and for whatever reason (can't find anything on Google as to why), none of the subs are recognised in the main script when loaded via ModPerl::PerlRun.
>>>
>>> So throwing it out to the list - am I on the right track? wasting my time? or just a simple mistake?
>>>
>>> --
>>> Steven Haigh 📧 netwiz@crc.id.au 💻 https://www.crc.id.au [crc.id.au] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=bosoTbkecbnrPukObNK-5Duc1p3JTllIM7_FHhBYKW4&s=vQDi0ezyEZDOz86GVraerPdT76UjN2in3UdPh8fglRM&e=>
>>> -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
RE: Moving ExecCGI to mod_perl - performance andcustom
'modules'
[EXT]
Posted by Steven Haigh <ne...@crc.id.au>.
In fact, I just realised that 'ab' test is rather restrictive.... So
here's a bit more of an extended test:
# ab -k -n 1000 -c 32
Apache + ExecCGI:
Requests per second: 14.26 [#/sec] (mean)
Time per request: 2244.181 [ms] (mean)
Time per request: 70.131 [ms] (mean, across all concurrent
requests)
Apache + mod_perl (ModPerl::PerlRegistry):
Requests per second: 132.14 [#/sec] (mean)
Time per request: 242.175 [ms] (mean)
Time per request: 7.568 [ms] (mean, across all concurrent
requests)
Interestingly, without Keepalives, the story is much the same:
# ab -n 1000 -c 32
Apache + ExecCGI:
Requests per second: 14.15 [#/sec] (mean)
Time per request: 2260.875 [ms] (mean)
Time per request: 70.652 [ms] (mean, across all concurrent
requests)
Apache + mod_perl (ModPerl::PerlRegistry):
Requests per second: 154.48 [#/sec] (mean)
Time per request: 207.140 [ms] (mean)
Time per request: 6.473 [ms] (mean, across all concurrent
requests)
Running some benchmarks across various parts of my site made me realise
I also had some RewriteRules in the apache config that still had
H=cgi-script - changed those to H=perl-script and saw similar
improvements:
ExecCGI - Requests per second: 11.84 [#/sec] (mean)
mod_perl - Requests per second: 130.97 [#/sec] (mean)
That's quite some gains for a days work.
--
Steven Haigh
📧 netwiz@crc.id.au <ma...@crc.id.au>
💻 https://www.crc.id.au <https://www.crc.id.au/>
On Sun, Feb 7, 2021 at 23:58, Steven Haigh <ne...@crc.id.au> wrote:
> Interestingly, I did get things working with ModPerl::PerlRegistry.
>
> What I couldn't find *anywhere* is that the data I was loading in
> Template Toolkit was included in the file in the __DATA__ area -
> which causes mod_perl to fall over!
>
> The only way I managed to find this was the following error in the
> *system* /var/log/httpd/error_log (didn't show up in the vhost
> error_log!):
> readline() on unopened filehandle DATA at
> /usr/lib64/perl5/vendor_perl/Template/Provider.pm line 638.
>
> Took me a LONG time to find a vague post that reading in lines from
> <DATA> kills mod_perl. Not sure why - but I stripped all the
> templates out and put them in a file instead and re-wrote that bit of
> code, and things started working.
>
> I had to fix a few lib path issues, but after getting my head around
> that, most things seem to work as before - however I don't notice
> much of an improvement in execution times, I do see this improvement
> using 'ab -n 100 -c32':
>
> Apache + ExecCGI: Requests per second: 13.50 [#/sec] (mean)
> Apache + mod_perl: Requests per second: 59.81 [#/sec] (mean)
>
> This is obviously a good thing.
>
> I haven't gotten into the preload or DBI sharing yet - as that'll end
> up needing a bit of a rewrite of code to take advantage of. I'd be
> open to suggestions here from those who have done it in the past to
> save me going down some dead ends :D
>
> --
> Steven Haigh
>
> 📧 netwiz@crc.id.au <ma...@crc.id.au>
> 💻 https://www.crc.id.au <https://www.crc.id.au/>
>
> On Sun, Feb 7, 2021 at 12:49, James Smith <js...@sanger.acuk> wrote:
>> As welsey said – try Registry, that was the standard way of using
>> mod_perl to cache perl in the server – but your problem might be
>> due to the note in PerlRun…
>>
>> <https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description>
>> META: document that for now we don't chdir() into the script's dir,
>> because it affects the whole process under threads.
>> ModPerl::PerlRunPrefork
>> <https://perl.apache.org/docs/20/api/ModPerl/PerlRunPrefork.html>
>> should be used by those who run only under prefork MPM.
>> {tbh most people don’t use mod perl under threads anyway as there
>> isn’t really a gain from using them}
>>
>> It suggests you use ModPerl/PerlRunPrefork – as this does an
>> additional step to cd to the script directory – which might be
>> your issue….
>>
>>
>>
>> *From:*Steven Haigh <ne...@crc.id.au>
>> *Sent:* 07 February 2021 01:00
>> *To:* modperl@perl.apache.org
>> *Subject:* Moving ExecCGI to mod_perl - performance and custom
>> 'modules' [EXT]
>>
>>
>>
>> Hi all,
>>
>>
>>
>> So for many years I've been slack and writing perl scripts to do
>> various things - but never needed more than the normal apache
>> +ExecCGI and Template Toolkit.
>>
>>
>>
>> One of my sites has become a bit more popular, so I'd like to spend
>> a bit of time on performance. Currently, I'm seeing ~300-400ms of
>> what I believe to be execution time of the script loading, running,
>> and then blatting its output to STDOUT and the browser can go do its
>> thing.
>>
>>
>>
>> I believe most of the delay would be to do with loading perl, its
>> modules etc etc
>>
>>
>>
>> I know that the current trend would be to re-write the entire site
>> in a more modern, daemon based solution - and I started down the
>> Mojolicious path - but the amount of re-writing to save 1/3rd of a
>> second seems to be excessive
>>
>>
>>
>> Would I be correct in thinking that mod_perl would help in this case?
>>
>>
>>
>> I did try a basic test, but I have a 'use functions' in all my
>> scripts that loads a .pm with some global vars and a lot of common
>> subs - and for whatever reason (can't find anything on Google as to
>> why), none of the subs are recognised in the main script when loaded
>> via ModPerl::PerlRun.
>>
>>
>>
>> So throwing it out to the list - am I on the right track? wasting my
>> time? or just a simple mistake?
>>
>>
>>
>> --
>>
>> Steven Haigh 📧netwiz@crc.id.au
>> <ma...@crc.id.au>💻https://www.crc.id.au [crc.id.au]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=bosoTbkecbnrPukObNK-5Duc1p3JTllIM7_FHhBYKW4&s=vQDi0ezyEZDOz86GVraerPdT76UjN2in3UdPh8fglRM&e=>
>>
>> -- The Wellcome Sanger Institute is operated by Genome Research
>> Limited, a charity registered in England with number 1021457 and a
>> company registered in England with number 2742969, whose registered
>> office is 215 Euston Road, London, NW1 2BE.
RE: Moving ExecCGI to mod_perl - performance and custom'modules'
[EXT]
Posted by Steven Haigh <ne...@crc.id.au>.
Interestingly, I did get things working with ModPerl::PerlRegistry.
What I couldn't find *anywhere* is that the data I was loading in
Template Toolkit was included in the file in the __DATA__ area - which
causes mod_perl to fall over!
The only way I managed to find this was the following error in the
*system* /var/log/httpd/error_log (didn't show up in the vhost
error_log!):
readline() on unopened filehandle DATA at
/usr/lib64/perl5/vendor_perl/Template/Provider.pm line 638.
Took me a LONG time to find a vague post that reading in lines from
<DATA> kills mod_perl. Not sure why - but I stripped all the templates
out and put them in a file instead and re-wrote that bit of code, and
things started working.
I had to fix a few lib path issues, but after getting my head around
that, most things seem to work as before - however I don't notice much
of an improvement in execution times, I do see this improvement using
'ab -n 100 -c32':
Apache + ExecCGI: Requests per second: 13.50 [#/sec] (mean)
Apache + mod_perl: Requests per second: 59.81 [#/sec] (mean)
This is obviously a good thing.
I haven't gotten into the preload or DBI sharing yet - as that'll end
up needing a bit of a rewrite of code to take advantage of. I'd be open
to suggestions here from those who have done it in the past to save me
going down some dead ends :D
--
Steven Haigh
📧 netwiz@crc.id.au <ma...@crc.id.au>
💻 https://www.crc.id.au <https://www.crc.id.au/>
On Sun, Feb 7, 2021 at 12:49, James Smith <js...@sanger.ac.uk> wrote:
> As welsey said – try Registry, that was the standard way of using
> mod_perl to cache perl in the server – but your problem might be
> due to the note in PerlRun…
>
> <https://perl.apache.org/docs/2.0/api/ModPerl/PerlRun.html#Description>
> META: document that for now we don't chdir() into the script's dir,
> because it affects the whole process under threads.
> ModPerl::PerlRunPrefork
> <https://perl.apache.org/docs/2.0/api/ModPerl/PerlRunPrefork.html>
> should be used by those who run only under prefork MPM.
> {tbh most people don’t use mod perl under threads anyway as there
> isn’t really a gain from using them}
>
> It suggests you use ModPerl/PerlRunPrefork – as this does an
> additional step to cd to the script directory – which might be your
> issue….
>
>
>
> *From:*Steven Haigh <ne...@crc.id.au>
> *Sent:* 07 February 2021 01:00
> *To:* modperl@perl.apache.org
> *Subject:* Moving ExecCGI to mod_perl - performance and custom
> 'modules' [EXT]
>
>
>
> Hi all,
>
>
>
> So for many years I've been slack and writing perl scripts to do
> various things - but never needed more than the normal apache
> +ExecCGI and Template Toolkit.
>
>
>
> One of my sites has become a bit more popular, so I'd like to spend a
> bit of time on performance. Currently, I'm seeing ~300-400ms of what
> I believe to be execution time of the script loading, running, and
> then blatting its output to STDOUT and the browser can go do its
> thing.
>
>
>
> I believe most of the delay would be to do with loading perl, its
> modules etc etc
>
>
>
> I know that the current trend would be to re-write the entire site in
> a more modern, daemon based solution - and I started down the
> Mojolicious path - but the amount of re-writing to save 1/3rd of a
> second seems to be excessive
>
>
>
> Would I be correct in thinking that mod_perl would help in this case?
>
>
>
> I did try a basic test, but I have a 'use functions' in all my
> scripts that loads a .pm with some global vars and a lot of common
> subs - and for whatever reason (can't find anything on Google as to
> why), none of the subs are recognised in the main script when loaded
> via ModPerl::PerlRun.
>
>
>
> So throwing it out to the list - am I on the right track? wasting my
> time? or just a simple mistake?
>
>
>
> --
>
> Steven Haigh 📧netwiz@crc.id.au
> <ma...@crc.id.au>💻https://www.crc.id.au [crc.id.au]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.crc.id.au_&d=DwMFaQ&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=oH2yp0ge1ecj4oDX0XM7vQ&m=bosoTbkecbnrPukObNK-5Duc1p3JTllIM7_FHhBYKW4&s=vQDi0ezyEZDOz86GVraerPdT76UjN2in3UdPh8fglRM&e=>
>
> -- The Wellcome Sanger Institute is operated by Genome Research
> Limited, a charity registered in England with number 1021457 and a
> company registered in England with number 2742969, whose registered
> office is 215 Euston Road, London, NW1 2BE.