You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@perl.apache.org by Heiko Jansen <ja...@hbz-nrw.de> on 2007/04/20 12:11:38 UTC
offtopic: Embedding Perl in C++: Help needed
Dear *.
I´m aware that the following is not really a mod_perl related mail but I
already posted this question elsewhere (e.g. on perlmonks.org) without
luck and I didn't find another community with equal knowledge about this
topic.
Feel free to answer me directly to keep this list clean.
I have to maintain a multithreaded (150+) C++ application (running as
64Bit App on Solaris 10, SPARC, SunCC) which uses embedded perl to allow
for easy data modification (utf-8 encoded strings) in certain places.
This does work in general, but every now and then the application
crashes although the same perl code has been run for a few days.
The cores usually look similar to this stack:
-- snip --
----------------- lwp# 114 / thread# 114 --------------------
ffffffff723b0e00 Perl_sv_clear
ffffffff723b16f4 Perl_sv_free
ffffffff723a7a94 S_visit
ffffffff723a7cd8 Perl_sv_clean_objs
ffffffff7232e414 perl_destruct
ffffffff7510568c __1cHrun_sub6Fpvpc_1_
ffffffff75106434 __1cTcEmbdPerlFilterListJrunFilter6MrknKipsCString_3r1_b_
...
-- / snip --
Currently the invocation of the perl filters is implemented as follows:
When a new (previously unencountered) perl script is to be loaded the
following code is executed, allocating a new perl interpreter used
solely for this script:
-- snip --
void * new_perlFilter( char * pcScriptName )
{
int _iArgc = 2;
char* _ppcArgv[] = { "", NULL, NULL };
_ppcArgv[1] = pcScriptName;
pthread_mutex_lock(&cloneMux);
PerlInterpreter * ppi = perl_alloc();
PERL_SET_CONTEXT( ppi );
perl_construct( ppi );
if( perl_parse( ppi,
xs_init,
_iArgc,
_ppcArgv,
(char **)NULL )
)
{
ppi = NULL;
}
pthread_mutex_unlock(&cloneMux);
return ppi;
}
-- / snip --
There is a global, mutex-protected list of these instances returned by
the above function (keyed with the scripts name) and when the
respective script is needed, the following code is executed, calling a sub
"ips_handler" in that script (with pPi being the interpreter instance
from the list):
char * run_sub( void * pPi, tPerlCallMap* pCallMap)
{ char* _pcRet = NULL;
try {
if(pPi) {
STRLEN _lLength = 0;
SV* _pSV = NULL;
char* _pcErg = NULL;
int _iStat = 0;
pthread_mutex_lock(&cloneMux);
PERL_SET_CONTEXT( (PerlInterpreter*)pPi );
PerlInterpreter* ppicl =
perl_clone((PerlInterpreter*)pPi,
CLONEf_COPY_STACKS);
pthread_mutex_unlock(&cloneMux);
PerlInterpreter* my_perl = ppicl;// nec. for dSP
PERL_SET_CONTEXT( my_perl );
dSP ;
ENTER ;
SAVETMPS ;
PUSHMARK(SP) ;
tPerlCallMap::iterator _oItr;
for(_oItr = pCallMap->begin();
_oItr != pCallMap->end();
_oItr++)
{
SV* _pSVIn = newSVpv(_oItr->second.c_str(), 0);
SvUTF8_on(_pSVIn);
XPUSHs(sv_2mortal(_pSVIn));
}
PUTBACK ;
_iStat = call_pv("ips_handler",
G_SCALAR | G_EVAL | G_KEEPERR);
if(_iStat) {
SPAGAIN;
_pSV = POPs;
SvUTF8_on(_pSV);
_pcErg = SvPV(_pSV, _lLength);
long _lErgLength = strlen(_pcErg);
_pcRet = new char[_lErgLength + 1];
if( _pcRet != NULL ) {
_pcRet[_lErgLength] = 0;
strcpy( _pcRet, _pcErg);
}
PUTBACK;
}
FREETMPS ;
LEAVE ;
PERL_SET_CONTEXT( my_perl );
perl_destruct( my_perl );
perl_free( my_perl );
}
}
catch(int nWhy) {}
catch(...) {}
return _pcRet;
}
I don´t see where the problem comes in.
The scripts run ok for a large number of invocations, there is no "exit"
in it and a "die" should be catched by the G_EVAL flag...
It even happens with perl-only scripts (i.e. no XS, only our script,
warnings, strict, URI::Escape, and (inderectly) Carp).
Maybe it´s some concurrency problem, but I don´t get it.
Besides my missing insight in the problem I´m wondering if there isn´t a
better approach to embedding perl:
* Using one interpreter per thread,
* with the same interpreter parsing every script the respective
thread is told to invoke and
* invoking the scripts using a modification of the "package
Embed::Persistent" example in `perldoc perlembed`
This should eliminate the "perl_destruct" calls which may trigger the
cores (although it´d increase memory consumption)?!
Any hints are welcome!
Best regards and many thanks for your help and patience
Heiko
Cologne, Germany
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: offtopic: Embedding Perl in C++: Help needed
Posted by Heiko Jansen <ja...@hbz-nrw.de>.
Hi all,
Am Freitag, den 20.04.2007, 19:24 +0200 schrieb Heiko Jansen:
> Am Freitag, den 20.04.2007, 09:25 -0700 schrieb Philip M. Gollucci:
> > > I have to maintain a multithreaded (150+) C++ application (running as
> > > 64Bit App on Solaris 10, SPARC, SunCC) which uses embedded perl to allow
> > > for easy data modification (utf-8 encoded strings) in certain places.
> > > This does work in general, but every now and then the application
> > > crashes although the same perl code has been run for a few days.
> >
> > I only briefly read this, but you're not hitting a system limit of some
> > kind are you ? Max open files, sockets, processes ?
just to let you know:
Some minor tweaks in the CPP-Code may have helped but it looks like the
most successful action I could take was switching to the latest code in
the 5.8.x maintenance tree
(http://public.activestate.com/pub/apc/perl-5.8.x/).
Especially the recent thread safety fix for "tr///" seems to have helped
a lot.
I still see senseless error messages like
-- snip --
(in cleanup) Can't find string terminator '"' anywhere before EOF
at /digibib/tools/lib/perl5/5.8.8/unicore/To/Fold.pl line 108
-- / snip --
although the respective files are perfectly fine, but at least the
processes don´t die anymore.
I can´t see a reason why perl should be unable to read and parse the
entire file but maybe I can fix that by preloading these files/modules
in an interpreter instance which is then cloned for every new instance.
heiko
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: offtopic: Embedding Perl in C++: Help needed
Posted by Heiko Jansen <ja...@hbz-nrw.de>.
Am Freitag, den 20.04.2007, 09:25 -0700 schrieb Philip M. Gollucci:
> > I have to maintain a multithreaded (150+) C++ application (running as
> > 64Bit App on Solaris 10, SPARC, SunCC) which uses embedded perl to allow
> > for easy data modification (utf-8 encoded strings) in certain places.
> > This does work in general, but every now and then the application
> > crashes although the same perl code has been run for a few days.
>
> I only briefly read this, but you're not hitting a system limit of some
> kind are you ? Max open files, sockets, processes ?
>
> sockstat, systat, fstat, lsof ... etc ?
The proc has a limit of 4096 file handles and I have never seen it come
even close to that limit.
Inside the perl scripts I do not open any files, databases handles or
the like (neither reading nor writing).
The server has 16GB of real memory and I have never seen it run out of
memory.
The data exchange with the scripts is done via the stack, as seen in the
initial post.
> Are you sure everything is cleaning up after itself correctly and
> completely ?
I see quite a number of "Scalars leaked" message on STDERR but have
never been able to track that down.
The scripts are fairly simple so I´m quite sure there are no leftovers
(also the interpreter is cloned, used and thrown away - there should be
nothing left?!)
The only thing I didn´t mention so far is, that I remember seeing output
like this sometimes before the app crashed:
-- snip --
Use of uninitialized value in pattern match (m//) at /tools/lib/perl5/5.8.8/utf8_heavy.pl line 211.
Use of uninitialized value in scalar assignment at /tools/lib/perl5/5.8.8/utf8_heavy.pl line 227.
Use of uninitialized value in pattern match (m//) at /tools/lib/perl5/5.8.8/utf8_heavy.pl line 228.
-- / snip --
The text above is from today. This is my utf8_heavy.pl:
-- snip --
188 my $extras;
189 my $bits = 0;
190
191 my $ORIG = $list;
192 if ($list) {
193 my @tmp = split(/^/m, $list);
194 my %seen;
195 no warnings;
196 $extras = join '', grep /^[^0-9a-fA-F]/, @tmp;
197 $list = join '',
198 map { $_->[1] }
199 sort { $a->[0] <=> $b->[0] }
200 map { /^([0-9a-fA-F]+)/; [ hex($1), $_ ] }
201 grep { /^([0-9a-fA-F]+)/ and not $seen{$1}++ } @tmp; # XXX doesn't do ranges right
202 }
203
204 if ($none) {
205 my $hextra = sprintf "%04x", $none + 1;
206 $list =~ s/\tXXXX$/\t$hextra/mg;
207 }
208
209 if ($minbits < 32) {
210 my $top = 0;
211 while ($list =~ /^([0-9a-fA-F]+)(?:[\t]([0-9a-fA-F]+)?)(?:[ \t]([0-9a-fA-F]+))?/mg) {
212 my $min = hex $1;
213 my $max = defined $2 ? hex $2 : $min;
214 my $val = defined $3 ? hex $3 : 0;
215 $val += $max - $min if defined $3;
216 $top = $val if $val > $top;
217 }
218 $bits =
219 $top > 0xffff ? 32 :
220 $top > 0xff ? 16 :
221 $top > 1 ? 8 : 1
222 }
223 $bits = $minbits if $bits < $minbits;
224
225 my @extras;
226 for my $x ($extras) {
227 pos $x = 0;
228 while ($x =~ /^([^0-9a-fA-F\n])(.*)/mg) {
-- / snip --
I don`t understand how the values can be undefined but maybe my perl is
even worse than my english.
The scripts should always get some data to work on. I´m not sure though,
that it may never be invalid utf-8 - but since the data is parsed from
and to XML I should have seen log entries on that (unless the scripts
always die on invalid utf-8...)
If there is anything else I could check, please tell me.
Many thanks
heiko
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org
Re: offtopic: Embedding Perl in C++: Help needed
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
> I have to maintain a multithreaded (150+) C++ application (running as
> 64Bit App on Solaris 10, SPARC, SunCC) which uses embedded perl to allow
> for easy data modification (utf-8 encoded strings) in certain places.
> This does work in general, but every now and then the application
> crashes although the same perl code has been run for a few days.
I only briefly read this, but you're not hitting a system limit of some
kind are you ? Max open files, sockets, processes ?
sockstat, systat, fstat, lsof ... etc ?
Are you sure everything is cleaning up after itself correctly and
completely ?
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/EC88A0BF 0DE5 C55C 6BF3 B235 2DAB B89E 1324 9B4F EC88 A0BF
Work like you don't need the money,
love like you'll never get hurt,
and dance like nobody's watching.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org