You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2022/10/06 15:03:15 UTC

[Bug 66297] New: Undefined symbol: RAND_egd modules/ssl/.libs/libmod_ssl.a(ssl_engine_rand.o)

https://bz.apache.org/bugzilla/show_bug.cgi?id=66297

            Bug ID: 66297
           Summary: Undefined symbol: RAND_egd
                    modules/ssl/.libs/libmod_ssl.a(ssl_engine_rand.o)
           Product: Apache httpd-2
           Version: 2.5-HEAD
          Hardware: Sun
                OS: Solaris
            Status: NEW
          Severity: regression
          Priority: P2
         Component: Build
          Assignee: bugs@httpd.apache.org
          Reporter: ran.mozes@oracle.com
  Target Milestone: ---

While trying to compile with the following configuration:

CFLAGS="-m32"; export CFLAGS
"./configure" \
"--prefix=/var/opt/sun/xvm/uce/opt/server" \
"--disable-proxy_express" \
"--disable-proxy_balancer" \
"--enable-cgi=static" \
"--enable-dir=static" \
"--enable-ssl=static" \
"--enable-pcre=static" \
"--enable-unixd=static" \
"--enable-alias=static" \
"--enable-setenvif=static" \
"--enable-log-config=static" \
"--enable-deflate=static" \
"--enable-filter=static" \
"--enable-socache-dbm=static" \
"--enable-mime=static" \
"--enable-authz-core=static" \
"--enable-authz-host=static" \
"--enable-proxy=static" \
"--enable-proxy-http=static" \
"--enable-rewrite=static" \
"--enable-headers=static" \
"--enable-cache=static" \
"--enable-cache-disk=static" \
"--enable-dav=static" \
"--enable-autoindex=static" \
"--enable-reqtimeout=static" \
"--enable-mime_magic=static" \
"--with-mpm=prefork" \
"--with-ssl=/var/opt/sun/xvm/uce/opt/server/" \
"--with-module=sigvalidate:/scratch/rmozes/apache_upgrade_2.4.54/mod_sigvalidate.c"
\
"--with-included-apr" \
"--with-pcre=/var/opt/sun/xvm/uce/opt/server/bin/pcre-config" \
"--with-zlib=/var/opt/sun/xvm/uce/opt/server" \
"--with-expat=/usr/sfw/" \
"CFLAGS=-m32" \
"$@"

using modules: 
apr-1.7.0
apr-util-1.6.1
openssl1.1.1o (same error with openssl1.1.1q)

the make process will fail with undefined symbol RAND_egd: 

bash-3.2$ /scratch/rmozes/apache_upgrade_2.4.54/httpd-2.4.54/srclib/apr/libtool
--silent --mode=link gcc -std=gnu99    -m32   
-L/var/opt/sun/xvm/uce/opt/server/lib   -o httpd  modules.lo buildmark.o
-export-dynamic server/libmain.la  modules/aaa/libmod_authz_host.la
modules/aaa/libmod_authz_core.la modules/cache/libmod_cache.la
modules/cache/libmod_cache_disk.la modules/cache/libmod_socache_dbm.la
modules/core/libmod_so.la modules/filters/libmod_reqtimeout.la
modules/filters/libmod_filter.la modules/filters/libmod_deflate.la
modules/http/libmod_http.la modules/http/libmod_mime.la
modules/loggers/libmod_log_config.la modules/metadata/libmod_mime_magic.la
modules/metadata/libmod_headers.la modules/metadata/libmod_setenvif.la
modules/proxy/libmod_proxy.la modules/proxy/libmod_proxy_connect.la
modules/proxy/libmod_proxy_ftp.la modules/proxy/libmod_proxy_http.la
modules/proxy/libmod_proxy_fcgi.la modules/proxy/libmod_proxy_scgi.la
modules/proxy/libmod_proxy_uwsgi.la modules/proxy/libmod_proxy_wstunnel.la
modules/proxy/libmod_proxy_ajp.la modules/ssl/libmod_ssl.la
modules/arch/unix/libmod_unixd.la modules/dav/main/libmod_dav.la
modules/generators/libmod_autoindex.la modules/generators/libmod_cgi.la
modules/dav/fs/libmod_dav_fs.la modules/sigvalidate/libmod_sigvalidate.la
modules/mappers/libmod_dir.la modules/mappers/libmod_alias.la
modules/mappers/libmod_rewrite.la  server/mpm/prefork/libprefork.la 
os/unix/libos.la -L/var/opt/sun/xvm/uce/opt/server/lib
-R/var/opt/sun/xvm/uce/opt/server/lib -lpcre          -lz                      
 
/scratch/rmozes/apache_upgrade_2.4.54/httpd-2.4.54/srclib/apr-util/libaprutil-1.la
-lexpat
/scratch/rmozes/apache_upgrade_2.4.54/httpd-2.4.54/srclib/apr/libapr-1.la
-luuid -lsendfile -lrt -lsocket -lnsl -lpthread
Undefined                       first referenced
 symbol                             in file
RAND_egd                           
modules/ssl/.libs/libmod_ssl.a(ssl_engine_rand.o)
ld: fatal: symbol referencing errors. No output written to .libs/httpd
collect2: ld returned 1 exit status
bash-3.2$

the symbold RAND_egd is truly not there  
bash-3.2$ nm -gDC /var/opt/sun/xvm/uce/opt/server/lib/libcrypto.so | grep -i
"RAND_egd"
bash-3.2$ nm -gDC /var/opt/sun/xvm/uce/opt/server/lib/libssl.so | grep -i
"RAND_egd"
bash-3.2$

I'm not sure if this is an Apache bug or openssl or rather something I should
have done while compiling the openssl. 
openssl was compiled with:
./Configure solaris-x86-gcc shared threads 386 no-ssl3
--prefix=/var/opt/sun/xvm/uce/opt/server
--openssldir=/var/opt/sun/xvm/uce/opt/server
CNF_CFLAGS=-pthreads -Wa,--noexecstack
CNF_EX_LIBS=-lsocket -lnsl -ldl -pthreads -lrt

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Re: [Bug 66297] Undefined symbol: RAND_egd modules/ssl/.libs/libmod_ssl.a(ssl_engine_rand.o)

Posted by Rainer Jung <ra...@kippdata.de>.
I think you are judging to fast. For instance on my Solaris 11 Sparc 
system, there is a local OpenSSL installation under 
/lib/64/libcrypto.so.1.0.0 which does have the RAND_egd symbol. The 
local file /usr/include/openssl/opensslconf.h does not define 
OPENSSL_NO_EGD and /usr/include/openssl/rand.h declares RAND_egd.

So in my case the system installation of openssl does support RAND_egd.

To make sure, that httpd finds all info for your custom openssl - header 
files and libs - before the ones from the operating system, the include 
and linker paths need to be set up correctly during configure and build.

Since during the configure check on your system the gcc flags are:

gcc -std=gnu99 -o conftest -m32  -DSOLARIS2=10
-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D_LARGEFILE64_SOURCE
-I/var/opt/sun/xvm/uce/opt/server/include   -L/usr/sfw//lib
-L/var/opt/sun/xvm/uce/opt/server/lib conftest.c -lssl -lcrypto    -luuid
-lsendfile -lrt -lsocket -lnsl  -lpthread

we can see, that it would eg. use libs from /usr/sfw/lib before your own 
openssl libs.

Therefore I think iit makes sense to do the litlle investigation that I 
suggested.

In addition: what are your configure flags plus additional env vars you set?

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


[Bug 66297] Undefined symbol: RAND_egd modules/ssl/.libs/libmod_ssl.a(ssl_engine_rand.o)

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=66297

--- Comment #3 from ran.mozes@oracle.com ---
exchanging information also with openssl folks, they said that the
configuration used in opensslconf.h and rand.h is disabling the RAND_egd by
default unless openssl is compiled with "enable-egd" option (which it wasn't).

It seems likely to me that a bug in the Apache build process is incorrectly
detecting that EGD is enabled when it is not. 

The other option with 2 sets of OpenSSL headers seems unlikely to me because on
my build env the Apache config process should have taken whatever is there from
one location "--with-ssl=/var/opt/sun/xvm/uce/opt/server/"
This location was cleaned before the process and then openssl was built with
the same prefix as mentioned before. 

bash-3.2$ sudo find /var/opt/ -name opensslconf.h
Password:
/var/opt/sun/xvm/uce/opt/server/include/openssl/opensslconf.h

only one instance found

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


[Bug 66297] Undefined symbol: RAND_egd modules/ssl/.libs/libmod_ssl.a(ssl_engine_rand.o)

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=66297

--- Comment #2 from ran.mozes@oracle.com ---
Thanks for looking into it Rainer. 

I double checked that and I don’t think to have conflicting versions of the
openssl in the included paths. I cleaned the path
/var/opt/sun/xvm/uce/opt/server/ before compilation,
then compiled the current openssl version into it. 

configure:27443: checking for OpenSSL
configure:27456: checking for user-provided OpenSSL base directory
configure:27472: result: /var/opt/sun/xvm/uce/opt/server
configure:27809: checking for OpenSSL version >= 0.9.8a
configure:27828: gcc -std=gnu99 -c -m32  -DSOLARIS2=10
-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D_LARGEFILE64_SOURCE
-I/var/opt/sun/xvm/uce/opt/server/include conftest.c >&5
configure:27828: $? = 0
configure:27829: result: OK

Looks the same in (my) opensslconf.h file 
#ifndef OPENSSL_NO_EGD
# define OPENSSL_NO_EGD
#endif


in (my) ap_config_auto.h it says:
/* Define to 1 if you have the `RAND_egd' function. */
#define HAVE_RAND_EGD 1

This file differs from yours. 

in config.log I see:
configure:27935: checking for RAND_egd
configure:27935: gcc -std=gnu99 -o conftest -m32  -DSOLARIS2=10
-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D_LARGEFILE64_SOURCE
-I/var/opt/sun/xvm/uce/opt/server/include   -L/usr/sfw//lib
-L/var/opt/sun/xvm/uce/opt/server/lib conftest.c -lssl -lcrypto    -luuid
-lsendfile -lrt -lsocket -lnsl  -lpthread >&5
configure:27935: $? = 0
configure:27935: result: yes
configure:27961: result: yes

Not sure if related but the file also includes this line:
ac_cv_func_RAND_egd=yes

and also found this:
include/ap_config_auto.h.in:/* Define to 1 if you have the `RAND_egd' function.
*/
include/ap_config_auto.h.in:#undef HAVE_RAND_EGD


So it seems the OpenSSL which is found and used by configure supports RAND_egd
(resp. OPENSSL_NO_EGD is undefined) but the one used for the compilation does
not.

Looks to me like openssl claims to support the functionality but doesn’t expose
it in the '.so‘ file as it should have. 
Do you think the same? Can you find the symbols of the RAND_egd* functions in
libcrypto.so or libssl.so? 

Also Apache decides to use this functionality (#define HAVE_RAND_EGD 1)
although the Solaris OS does include  (see openssl doc for this:
https://www.openssl.org/docs/manmaster/man3/RAND_egd.html) 
ls /dev/urandom
/dev/urandom

I wonder why in your case the compilation has decided to use /* #undef
HAVE_RAND_EGD */ 

As a workaround I have manually changed ap_config_auto.h
   include/ap_config_auto.h.in:#undef HAVE_RAND_EGD
then ran the make process from start and the compilation ended successfully.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


[Bug 66297] Undefined symbol: RAND_egd modules/ssl/.libs/libmod_ssl.a(ssl_engine_rand.o)

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=66297

--- Comment #4 from Rainer Jung <ra...@kippdata.de> ---
> configure:27443: checking for OpenSSL
> configure:27456: checking for user-provided OpenSSL base directory
> configure:27472: result: /var/opt/sun/xvm/uce/opt/server
> configure:27809: checking for OpenSSL version >= 0.9.8a
> configure:27828: gcc -std=gnu99 -c -m32  -DSOLARIS2=10
> -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D_LARGEFILE64_SOURCE
> -I/var/opt/sun/xvm/uce/opt/server/include conftest.c >&5
> configure:27828: $? = 0
> configure:27829: result: OK

Good.

> Looks the same in (my) opensslconf.h file
> #ifndef OPENSSL_NO_EGD
> # define OPENSSL_NO_EGD
> #endif

Good.

> in (my) ap_config_auto.h it says:
> /* Define to 1 if you have the `RAND_egd' function. */
> #define HAVE_RAND_EGD 1

Bad, but it comes from ...

> in config.log I see:
> configure:27935: checking for RAND_egd
> configure:27935: gcc -std=gnu99 -o conftest -m32  -DSOLARIS2=10
> -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D_LARGEFILE64_SOURCE
> -I/var/opt/sun/xvm/uce/opt/server/include   -L/usr/sfw//lib
> -L/var/opt/sun/xvm/uce/opt/server/lib conftest.c -lssl -lcrypto    -luuid
> -lsendfile -lrt -lsocket -lnsl  -lpthread >&5
> configure:27935: $? = 0
> configure:27935: result: yes
> configure:27961: result: yes

So this is the culprit and what we ned to understand.

> Not sure if related but the file also includes this line:
> ac_cv_func_RAND_egd=yes

That is also a consequence of configure detecting RAND_egd.

> and also found this:
> include/ap_config_auto.h.in:/* Define to 1 if you have the `RAND_egd' function.
> */
> include/ap_config_auto.h.in:#undef HAVE_RAND_EGD

That is also a consequence of configure detecting RAND_egd.

> So it seems the OpenSSL which is found and used by configure supports RAND_egd
> (resp. OPENSSL_NO_EGD is undefined) but the one used for the compilation does
> not.

Yes.

> Looks to me like openssl claims to support the functionality but doesn’t expose
> it in the '.so‘ file as it should have.
> Do you think the same? Can you find the symbols of the RAND_egd* functions in
> libcrypto.so or libssl.so?

No it doesn't have the symbol either.

BUT: For instance on my Solaris 11 Sparc system, there is a local OpenSSL
installation under /lib/64/libcrypto.so.1.0.0 which *does* have the RAND_egd
symbol. The local file /usr/include/openssl/opensslconf.h does not define
OPENSSL_NO_EGD and /usr/include/openssl/rand.h declares RAND_egd. So in my case
the system installation of openssl does support RAND_egd, but my custom
OpenSSL, which I use to build httpd, does not.

To make sure, that httpd finds all info for your custom openssl - header files
and libs - before the ones from the operating system, the include and linker
paths need to be set up correctly during configure and build.

Since during the configure check on your system the gcc flags are:

gcc -std=gnu99 -o conftest -m32  -DSOLARIS2=10
-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D_LARGEFILE64_SOURCE
-I/var/opt/sun/xvm/uce/opt/server/include   -L/usr/sfw//lib
-L/var/opt/sun/xvm/uce/opt/server/lib conftest.c -lssl -lcrypto    -luuid
-lsendfile -lrt -lsocket -lnsl  -lpthread

we can see, that it would eg. use libs from /usr/sfw/lib before your own
openssl libs.

Therefore I think it makes sense to do the litle investigation that I suggest
further below.

In addition: what are your configure flags plus additional env vars you set? 

> Also Apache decides to use this functionality (#define HAVE_RAND_EGD 1)
> although the Solaris OS does include  (see openssl doc for this:
> https://www.openssl.org/docs/manmaster/man3/RAND_egd.html)
> ls /dev/urandom
> /dev/urandom

Using egd - if detected as existing - seems to have higher priority than just
/dev/urandom.

> I wonder why in your case the compilation has decided to use /* #undef
> HAVE_RAND_EGD */
> 
> As a workaround I have manually changed ap_config_auto.h
>     include/ap_config_auto.h.in:#undef HAVE_RAND_EGD
> then ran the make process from start and the compilation ended successfully.

OK, the compilation result will be fine, the workaround is OK, but of course it
would be better to understand why it goes wrong without woraround.

So if you are still interested in fixing it really, we can have a look at what
the above

 > configure:27935:

gcc -std=gnu99 -o conftest -m32  -DSOLARIS2=10 -D_POSIX_PTHREAD_SEMANTICS
-D_REENTRANT -D_LARGEFILE64_SOURCE -I/var/opt/sun/xvm/uce/opt/server/include  
-L/usr/sfw//lib -L/var/opt/sun/xvm/uce/opt/server/lib conftest.c -lssl -lcrypto
-luuid -lsendfile -lrt -lsocket -lnsl  -lpthread >&5

compiles and why it succeeeds.

There are several ways of doing it, but the following should be easy:

- edit your configure script, replace "RAND_egd" by "RAND_egd_unknown"

There should be only one line in configure that contains the function name:

       for ac_func in ENGINE_init ENGINE_load_builtin_engines RAND_egd

After you replace it with the name of the function that definitely does not
exist, rerun configure. Since this time the detection fails, configure will
also output the full text of the C program it tried to compile. That is
helpful, because we can pick that C source code, put it into a file conftest.c
and run the compilation ourselfs with the above

gcc -std=gnu99 -o conftest -m32  -DSOLARIS2=10 -D_POSIX_PTHREAD_SEMANTICS
-D_REENTRANT -D_LARGEFILE64_SOURCE -I/var/opt/sun/xvm/uce/opt/server/include  
-L/usr/sfw//lib -L/var/opt/sun/xvm/uce/opt/server/lib conftest.c -lssl -lcrypto
-luuid -lsendfile -lrt -lsocket -lnsl  -lpthread

Of course we would replace RAND_egd_unknown by the original RAND_egd in
conftest.c.

For example in my case, configure outputs:

...
checking for RAND_egd
gcc ...
Undefined                       first referenced
  symbol                             in file
RAND_egd                            /var/tmp//ccIg85bK.o
ld: fatal: Symbol referencing errors. No output written to conftest
collect2: error: ld returned 1 exit status
$? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME ""
| #define PACKAGE_TARNAME ""
| #define PACKAGE_VERSION ""
| #define PACKAGE_STRING ""
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE_URL ""
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define __EXTENSIONS__ 1
| #define _ALL_SOURCE 1
| #define _GNU_SOURCE 1
| #define _POSIX_PTHREAD_SEMANTICS 1
| #define _TANDEM_SOURCE 1
| #define STDC_HEADERS 1
| #define HAVE_STRING_H 1
| #define HAVE_LIMITS_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_SYS_SOCKET_H 1
| #define HAVE_PWD_H 1
| #define HAVE_GRP_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_SYS_PROCESSOR_H 1
| #define HAVE_SYS_SEM_H 1
| #define HAVE_SYS_SDT_H 1
| #define HAVE_SYS_LOADAVG_H 1
| #define HAVE_SYS_WAIT_H 1
| #define HAVE_GETPWNAM 1
| #define HAVE_GETGRNAM 1
| #define HAVE_INITGROUPS 1
| #define HAVE_GETPGID 1
| #define HAVE_FOPEN64 1
| #define HAVE_GETLOADAVG 1
| #define HAVE_SYS_TIMES_H 1
| #define HAVE_TIMES 1
| #define HAVE_OPENSSL_ENGINE_H 1
| #define HAVE_SSL_CTX_NEW 1
| #define HAVE_OPENSSL_INIT_SSL 1
| #define HAVE_ENGINE_INIT 1
| #define HAVE_ENGINE_LOAD_BUILTIN_ENGINES 1
| /* end confdefs.h.  */
| /* Define RAND_egd to an innocuous variant, in case <limits.h> declares
RAND_egd.
|    For example, HP-UX 11i <limits.h> declares gettimeofday.  */
| #define RAND_egd innocuous_RAND_egd
|
| /* System header to define __stub macros and hopefully few prototypes,
|     which can conflict with char RAND_egd (); below.
|     Prefer <limits.h> to <assert.h> if __STDC__ is defined, since
|     <limits.h> exists even on freestanding compilers.  */
|
| #ifdef __STDC__
| # include <limits.h>
| #else
| # include <assert.h>
| #endif
|
| #undef RAND_egd
|
| /* Override any GCC internal prototype to avoid an error.
|    Use char because int might match the return type of a GCC
|    builtin and then its argument prototype would still apply.  */
| #ifdef __cplusplus
| extern "C"
| #endif
| char RAND_egd ();
| /* The GNU C library defines this for functions which it implements
|     to always fail with ENOSYS.  Some functions are actually named
|     something starting with __ and the normal name is an alias.  */
| #if defined __stub_RAND_egd || defined __stub___RAND_egd
| choke me
| #endif
|
| int
| main ()
| {
| return RAND_egd ();
|   ;
|   return 0;
| }
...

The text block with the "| " in front of the lines is the conftest.c contents.
But it is better you create it on your system and not just take the one from
mine.

Once you have your conftest.c and run gcc - which in your case should succeed
although we do not know yet why, you can do some checks:

Check with "nm" and "ldd" whether the symbol is referenced by the created a.out
and where it might come from? If the symbol gets used, you should be able to
find the library that provided it. If the symbol is not used, it might be
defined as a macro somewhere (although the above test code tries to rule that
out). Nevertheless you can then use "gcc -E" to only resolve all header files
and definitions in conftest.c and look in the result for RAND_egd.

I hope that's understandable!

Best regards,

Rainer

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


[Bug 66297] Undefined symbol: RAND_egd modules/ssl/.libs/libmod_ssl.a(ssl_engine_rand.o)

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=66297

--- Comment #1 from Rainer Jung <ra...@kippdata.de> ---
There seems to be an OpenSSL header file mixup.

When OpenSSL is compiled, it detects egd support. The definition of RAND_egd()
in openssl/rand.h is protected by an

# ifndef OPENSSL_NO_EGD

and in (my) openssl/opensslconf.h I find:

#ifndef OPENSSL_NO_EGD
# define OPENSSL_NO_EGD
#endif

When httpd gets compiled, its configure script checks for RAND_egd via

      AC_CHECK_FUNCS([ENGINE_init ENGINE_load_builtin_engines RAND_egd])

and sets HAVE_RAND_EGD in include/ap_config_auto.h. For my build there is:

/* #undef HAVE_RAND_EGD */

Configure output and config.log also show

checking for RAND_egd... no

So it seems the OpenSSL which is found and used by configure supports RAND_egd
(resp. OPENSSL_NO_EGD is undefined) but the one used for the compilation does
not.

Please check configure output and the definitions in the quoted header file, to
find, where things start to go wrong.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org