FreeBSD, pthreads unresoved symbols and LD_PRELOAD revisited

[Eric Brunson <br...@brunson.com>]

I apologize if this goes out twice, I think I wrote it on my laptop at
work, then shut it down without hitting send.

While upgrading to 3.3 on Thursday I ran into the previously discussed
"pthread_*" unresolved symbol problem and used the
LD_PRELOAD=/lib/libc_r.so fix, which worked and seemed to get us most of
the way through our regression testing (such as it is). 

We spent several hours tracking down why certain subprocesses were
failing when using Popen and after much trial and error, came to realize
that *any* module we compiled ourselves was failing, even the 3.1.4
version that we were originally running from ports.  We started coming
to the conclusion that the threaded libc was causing problems with
subprocesses spawning subprocess ("spawn" in our expect scripts was the
culprit that pointed out the problem, then I was able to produce failing
test cases in bash and ksh scripts).

Comparing build environments between our vanilla source builds and the
ports tree, I found a patch to configure that changes PyLIBS to use the
same libraries that are linked when python is built in the ports tree. 
The patch for 3.1.4 applies cleanly with a 'fuzz' warning to both the
3.2.10 source and the 3.3-svn source.  The crux of the patch its to
change one line:

-PyLIBS=`grep "^LIB[SMC]=" ${PyLIBPL}/Makefile | cut -f2 -d= | tr
'\011\012\015' '   '`
+PyLIBS=`ldd $PYTHON_BIN | sed -n 's,^.* => [^ ]*/lib\(.*\)\.so[^ ]*
\((.*)\),-l\1,p' | grep -v '^-lc$' | xargs echo`

this enables any version we tried (3.1.4, 3.2.10 or 3.3) to be compiled
then run without modifying the LD_PRELOAD environment variable and also
fixed our problems with subprocess spawning.

Should we try to implement this change in the build environment?  Or is
it enough to have it referenced in the documentation?  I'll probably add
a wiki article on the problem.

Thanks,
e.

Re: FreeBSD, pthreads unresoved symbols and LD_PRELOAD revisited

[Eric Brunson <br...@brunson.com>]

Graham Dumpleton wrote:
>
> On 03/12/2006, at 1:41 PM, Eric Brunson wrote:
>
>>> Can you explain properly what that one line is achieving.
>>
>> The change is causing mod_python to be linked with the same libs that
>> python was when it was built, in the absence of the Makefile.
>
> If you have the python-dev package installed which does include the
> Makefile,
> does it work without requiring your fiddle?

First of all, apologies for a mistake in my last email, it got it
backwards, the change in the configure script causes the addition of
-lpthread, not the removal of it.

There doesn't seem to be a "python-dev" package in FreeBSD, as there is
in linux.  The python-2.4.2 package include headers and config.

Looking at the Makefile that is distributed in the package, it appears
to disagree with the link options of the binary in the same package:

$ grep "^LIB[SMC]=" /usr/local/lib/python2.4/config/Makefile | cut -f2
-d= | tr '\011\012\015' '   '
   -lutil   -lm   

vs:

$ ldd /usr/local/bin/python2.4 | sed -n 's,^.* => [^ ]*/lib\(.*\)\.so[^
]* \((.*)\),-l\1,p' | grep -v '^-lc' | xargs echo
-lutil -lstdc++ -lm -lpthread

At first I thought this may be a flaw in the Makefile in the package,
but I think it's still our problem...

grep "^LIB[SMC]=" /usr/local/lib/python2.4/config/Makefile
LIBS=            -lutil
LIBM=           -lm
LIBC=

But:
grep "^LDFLAGS=" /usr/local/lib/python2.4/config/Makefile
LDFLAGS=        -pthread

we're clearly missing that important library.

So, maybe our change just needs to be:

grep -e "^LIB[SMC]=" -e "^LDFLAGS="
/usr/local/lib/python2.4/config/Makefile | cut -f2 -d= | tr
'\011\012\015' '   '

which yields:
 -pthread     -lutil   -lm

What do you think of that?

>
> I am not even sure why the Makefile would be missing because to even
> compile
> mod_python requires the Python header files be available which generally
> implies that the 'config' directory in the Python library directory
> would be
> present, including the Makefile.
>
> It seems to me a bit like this is a fiddle to workaround that your
> Python installation
> is incomplete or broken. Note that the list of libraries to link is
> potentially not the
> only thing that is obtained from the Makefile so not having it could
> cause other
> problems. So doing a fiddle to workaround it not being there seems
> wrong to me.
>
> Why is the Makefile not present in the first place?
>
> Graham

Re: FreeBSD, pthreads unresoved symbols and LD_PRELOAD revisited

[Graham Dumpleton <gr...@dscpl.com.au>]

On 03/12/2006, at 1:41 PM, Eric Brunson wrote:

>> Can you explain properly what that one line is achieving.
>
> The change is causing mod_python to be linked with the same libs that
> python was when it was built, in the absence of the Makefile.

If you have the python-dev package installed which does include the  
Makefile,
does it work without requiring your fiddle?

I am not even sure why the Makefile would be missing because to even  
compile
mod_python requires the Python header files be available which generally
implies that the 'config' directory in the Python library directory  
would be
present, including the Makefile.

It seems to me a bit like this is a fiddle to workaround that your  
Python installation
is incomplete or broken. Note that the list of libraries to link is  
potentially not the
only thing that is obtained from the Makefile so not having it could  
cause other
problems. So doing a fiddle to workaround it not being there seems  
wrong to me.

Why is the Makefile not present in the first place?

Graham

Re: FreeBSD, pthreads unresoved symbols and LD_PRELOAD revisited

[Eric Brunson <br...@brunson.com>]

Graham Dumpleton wrote:
>
> On 03/12/2006, at 7:34 AM, Jim Gallacher wrote:
>
>> Eric Brunson wrote:
>>> I apologize if this goes out twice, I think I wrote it on my laptop at
>>> work, then shut it down without hitting send.
>>> While upgrading to 3.3 on Thursday I ran into the previously discussed
>>> "pthread_*" unresolved symbol problem and used the
>>> LD_PRELOAD=/lib/libc_r.so fix, which worked and seemed to get us
>>> most of
>>> the way through our regression testing (such as it is). We spent
>>> several hours tracking down why certain subprocesses were
>>> failing when using Popen and after much trial and error, came to
>>> realize
>>> that *any* module we compiled ourselves was failing, even the 3.1.4
>>> version that we were originally running from ports.  We started coming
>>> to the conclusion that the threaded libc was causing problems with
>>> subprocesses spawning subprocess ("spawn" in our expect scripts was the
>>> culprit that pointed out the problem, then I was able to produce
>>> failing
>>> test cases in bash and ksh scripts).
>>> Comparing build environments between our vanilla source builds and the
>>> ports tree, I found a patch to configure that changes PyLIBS to use the
>>> same libraries that are linked when python is built in the ports
>>> tree. The patch for 3.1.4 applies cleanly with a 'fuzz' warning to
>>> both the
>>> 3.2.10 source and the 3.3-svn source.  The crux of the patch its to
>>> change one line:
>>> -PyLIBS=`grep "^LIB[SMC]=" ${PyLIBPL}/Makefile | cut -f2 -d= | tr
>>> '\011\012\015' '   '`
>>> +PyLIBS=`ldd $PYTHON_BIN | sed -n 's,^.* => [^ ]*/lib\(.*\)\.so[^ ]*
>>> \((.*)\),-l\1,p' | grep -v '^-lc$' | xargs echo`
>>> this enables any version we tried (3.1.4, 3.2.10 or 3.3) to be compiled
>>> then run without modifying the LD_PRELOAD environment variable and also
>>> fixed our problems with subprocess spawning.
>>> Should we try to implement this change in the build environment?  Or is
>>> it enough to have it referenced in the documentation?  I'll probably
>>> add
>>> a wiki article on the problem.
>>
>> The wiki is fine if people want to backport to 3.1.4 or 3.2.10, but
>> it should be fixed in the 3.3 source. Could you open a JIRA issue and
>> attach the patch so we don't loose track of it? I'd definitely like
>> to see a trouble-free build on FreeBSD.
>
> The 'ldd' program doesn't exist everywhere and so that change can't be
> used.
>
> Can you explain properly what that one line is achieving.

The change is causing mod_python to be linked with the same libs that
python was when it was built, in the absence of the Makefile.

>
> If you are trying to force the linking in of additional libraries, eg,
> -lc_r, what happens
> when you set LIBS when running configure. Eg.
>
>   LIBS=-lc_r ./configure ...
>
> Even if this doesn't work but it is what you are hoping to achieve, it
> would be better
> to still require it to be a manual override rather than automatic. 

I don't feel the defaults being chosen for the platform are acceptable,
if others using FreeBSD agree, then I think we should change the
defaults.  The simple presence of libpthreads doesn't mean it should be
used in all cases.  Unfortunately I'm not familiar enough with autoconf
to know what's available to be changed.

> That could mean
> making LIBS be used, or have a special configure option to enable
> linking in of
> other libraries.
>
> Anyway, please explain what the change is doing.

I can tell you what it is doing in this specific case of FreeBSD 6.1 and
the python 2.4.2 ports package, which is to exclude libpthread.  Can we
just make configure automatically exclude pthreads for FreeBSD using any
of the methods you describe (allowing the --with-pthreads to be
specified if the user desires)?  If we're compiling for python 2.5 or
greater, we can simply use the output of "python2.5-config --ldflags",
but for versions before 2.5 that isn't an option. 

I feel like I'm repeating myself, but... the goal is to link against the
same libraries as python, no matter what they are, and this is obviously
a problem since work arounds for our bad link decisions have already
been discussed here and on the developers list.

>
> Thanks.
>
> Graham

Re: FreeBSD, pthreads unresoved symbols and LD_PRELOAD revisited

[Graham Dumpleton <gr...@dscpl.com.au>]

On 03/12/2006, at 7:34 AM, Jim Gallacher wrote:

> Eric Brunson wrote:
>> I apologize if this goes out twice, I think I wrote it on my  
>> laptop at
>> work, then shut it down without hitting send.
>> While upgrading to 3.3 on Thursday I ran into the previously  
>> discussed
>> "pthread_*" unresolved symbol problem and used the
>> LD_PRELOAD=/lib/libc_r.so fix, which worked and seemed to get us  
>> most of
>> the way through our regression testing (such as it is). We spent  
>> several hours tracking down why certain subprocesses were
>> failing when using Popen and after much trial and error, came to  
>> realize
>> that *any* module we compiled ourselves was failing, even the 3.1.4
>> version that we were originally running from ports.  We started  
>> coming
>> to the conclusion that the threaded libc was causing problems with
>> subprocesses spawning subprocess ("spawn" in our expect scripts  
>> was the
>> culprit that pointed out the problem, then I was able to produce  
>> failing
>> test cases in bash and ksh scripts).
>> Comparing build environments between our vanilla source builds and  
>> the
>> ports tree, I found a patch to configure that changes PyLIBS to  
>> use the
>> same libraries that are linked when python is built in the ports  
>> tree. The patch for 3.1.4 applies cleanly with a 'fuzz' warning to  
>> both the
>> 3.2.10 source and the 3.3-svn source.  The crux of the patch its to
>> change one line:
>> -PyLIBS=`grep "^LIB[SMC]=" ${PyLIBPL}/Makefile | cut -f2 -d= | tr
>> '\011\012\015' '   '`
>> +PyLIBS=`ldd $PYTHON_BIN | sed -n 's,^.* => [^ ]*/lib\(.*\)\.so[^ ]*
>> \((.*)\),-l\1,p' | grep -v '^-lc$' | xargs echo`
>> this enables any version we tried (3.1.4, 3.2.10 or 3.3) to be  
>> compiled
>> then run without modifying the LD_PRELOAD environment variable and  
>> also
>> fixed our problems with subprocess spawning.
>> Should we try to implement this change in the build environment?   
>> Or is
>> it enough to have it referenced in the documentation?  I'll  
>> probably add
>> a wiki article on the problem.
>
> The wiki is fine if people want to backport to 3.1.4 or 3.2.10, but  
> it should be fixed in the 3.3 source. Could you open a JIRA issue  
> and attach the patch so we don't loose track of it? I'd definitely  
> like to see a trouble-free build on FreeBSD.

The 'ldd' program doesn't exist everywhere and so that change can't  
be used.

Can you explain properly what that one line is achieving.

If you are trying to force the linking in of additional libraries,  
eg, -lc_r, what happens
when you set LIBS when running configure. Eg.

   LIBS=-lc_r ./configure ...

Even if this doesn't work but it is what you are hoping to achieve,  
it would be better
to still require it to be a manual override rather than automatic.  
That could mean
making LIBS be used, or have a special configure option to enable  
linking in of
other libraries.

Anyway, please explain what the change is doing.

Thanks.

Graham

Re: FreeBSD, pthreads unresoved symbols and LD_PRELOAD revisited

[Jim Gallacher <jp...@jgassociates.ca>]

Eric Brunson wrote:
> I apologize if this goes out twice, I think I wrote it on my laptop at
> work, then shut it down without hitting send.
> 
> While upgrading to 3.3 on Thursday I ran into the previously discussed
> "pthread_*" unresolved symbol problem and used the
> LD_PRELOAD=/lib/libc_r.so fix, which worked and seemed to get us most of
> the way through our regression testing (such as it is). 
> 
> We spent several hours tracking down why certain subprocesses were
> failing when using Popen and after much trial and error, came to realize
> that *any* module we compiled ourselves was failing, even the 3.1.4
> version that we were originally running from ports.  We started coming
> to the conclusion that the threaded libc was causing problems with
> subprocesses spawning subprocess ("spawn" in our expect scripts was the
> culprit that pointed out the problem, then I was able to produce failing
> test cases in bash and ksh scripts).
> 
> Comparing build environments between our vanilla source builds and the
> ports tree, I found a patch to configure that changes PyLIBS to use the
> same libraries that are linked when python is built in the ports tree. 
> The patch for 3.1.4 applies cleanly with a 'fuzz' warning to both the
> 3.2.10 source and the 3.3-svn source.  The crux of the patch its to
> change one line:
> 
> -PyLIBS=`grep "^LIB[SMC]=" ${PyLIBPL}/Makefile | cut -f2 -d= | tr
> '\011\012\015' '   '`
> +PyLIBS=`ldd $PYTHON_BIN | sed -n 's,^.* => [^ ]*/lib\(.*\)\.so[^ ]*
> \((.*)\),-l\1,p' | grep -v '^-lc$' | xargs echo`
> 
> this enables any version we tried (3.1.4, 3.2.10 or 3.3) to be compiled
> then run without modifying the LD_PRELOAD environment variable and also
> fixed our problems with subprocess spawning.
> 
> Should we try to implement this change in the build environment?  Or is
> it enough to have it referenced in the documentation?  I'll probably add
> a wiki article on the problem.

The wiki is fine if people want to backport to 3.1.4 or 3.2.10, but it 
should be fixed in the 3.3 source. Could you open a JIRA issue and 
attach the patch so we don't loose track of it? I'd definitely like to 
see a trouble-free build on FreeBSD.

Jim

Re: FreeBSD, pthreads unresoved symbols and LD_PRELOAD revisited

[Eric Brunson <br...@brunson.com>]

Nice, I managed to misspell "unresolved" in the subject of this mail
*and* the title of the Wiki article.  Any way to change that without
deleting the article and recreating it with the correct title?  Luckily
it's spelled right in the body so the search engines will pick it up.  :-/


Eric Brunson wrote:
> I apologize if this goes out twice, I think I wrote it on my laptop at
> work, then shut it down without hitting send.
>
> While upgrading to 3.3 on Thursday I ran into the previously discussed
> "pthread_*" unresolved symbol problem and used the
> LD_PRELOAD=/lib/libc_r.so fix, which worked and seemed to get us most of
> the way through our regression testing (such as it is). 
>
> We spent several hours tracking down why certain subprocesses were
> failing when using Popen and after much trial and error, came to realize
> that *any* module we compiled ourselves was failing, even the 3.1.4
> version that we were originally running from ports.  We started coming
> to the conclusion that the threaded libc was causing problems with
> subprocesses spawning subprocess ("spawn" in our expect scripts was the
> culprit that pointed out the problem, then I was able to produce failing
> test cases in bash and ksh scripts).
>
> Comparing build environments between our vanilla source builds and the
> ports tree, I found a patch to configure that changes PyLIBS to use the
> same libraries that are linked when python is built in the ports tree. 
> The patch for 3.1.4 applies cleanly with a 'fuzz' warning to both the
> 3.2.10 source and the 3.3-svn source.  The crux of the patch its to
> change one line:
>
> -PyLIBS=`grep "^LIB[SMC]=" ${PyLIBPL}/Makefile | cut -f2 -d= | tr
> '\011\012\015' '   '`
> +PyLIBS=`ldd $PYTHON_BIN | sed -n 's,^.* => [^ ]*/lib\(.*\)\.so[^ ]*
> \((.*)\),-l\1,p' | grep -v '^-lc$' | xargs echo`
>
> this enables any version we tried (3.1.4, 3.2.10 or 3.3) to be compiled
> then run without modifying the LD_PRELOAD environment variable and also
> fixed our problems with subprocess spawning.
>
> Should we try to implement this change in the build environment?  Or is
> it enough to have it referenced in the documentation?  I'll probably add
> a wiki article on the problem.
>
> Thanks,
> e.
>
>