You are viewing a plain text version of this content. The canonical link for it is here.
Posted to cvs@httpd.apache.org by sf...@apache.org on 2011/02/03 21:20:02 UTC
svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in
server/main.c
Author: sf
Date: Thu Feb 3 20:20:02 2011
New Revision: 1066944
URL: http://svn.apache.org/viewvc?rev=1066944&view=rev
Log:
Reload resolv.conf on graceful restarts
PR: 50619
Submitted by: Matt Miller <m miller f5 com>, Stefan Fritsch
Modified:
httpd/httpd/trunk/CHANGES
httpd/httpd/trunk/configure.in
httpd/httpd/trunk/server/main.c
Modified: httpd/httpd/trunk/CHANGES
URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/CHANGES?rev=1066944&r1=1066943&r2=1066944&view=diff
==============================================================================
--- httpd/httpd/trunk/CHANGES [utf-8] (original)
+++ httpd/httpd/trunk/CHANGES [utf-8] Thu Feb 3 20:20:02 2011
@@ -2,6 +2,9 @@
Changes with Apache 2.3.11
+ *) core: Reload resolv.conf on graceful restarts. PR 50619.
+ [Matt Miller <m miller f5 com>, Stefan Fritsch]
+
*) mod_mime: Ignore leading dots when looking for mime extensions.
PR 50434 [Stefan Fritsch]
Modified: httpd/httpd/trunk/configure.in
URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/configure.in?rev=1066944&r1=1066943&r2=1066944&view=diff
==============================================================================
--- httpd/httpd/trunk/configure.in (original)
+++ httpd/httpd/trunk/configure.in Thu Feb 3 20:20:02 2011
@@ -400,6 +400,7 @@ sys/sem.h \
sys/sdt.h
)
AC_HEADER_SYS_WAIT
+AC_HEADER_RESOLV
dnl ## Check for typedefs, structures, and compiler characteristics.
@@ -416,6 +417,7 @@ LIBS="$saved_LIBS"
dnl See Comment #Spoon
+AC_SEARCH_LIBS(res_init, resolv, , [AC_SEARCH_LIBS(__res_init,resolv)])
AC_CHECK_FUNCS( \
getpwnam \
getgrnam \
@@ -424,7 +426,9 @@ bindprocessor \
prctl \
timegm \
getpgid \
-fopen64
+fopen64 \
+res_init \
+__res_init
)
dnl confirm that a void pointer is large enough to store a long integer
Modified: httpd/httpd/trunk/server/main.c
URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/server/main.c?rev=1066944&r1=1066943&r2=1066944&view=diff
==============================================================================
--- httpd/httpd/trunk/server/main.c (original)
+++ httpd/httpd/trunk/server/main.c Thu Feb 3 20:20:02 2011
@@ -44,6 +44,16 @@
#include <unistd.h>
#endif
+#if HAVE_NETINET_IN_H
+#include <netinet/in.h>
+#endif
+#if HAVE_ARPA_NAMESER_H
+#include <arpa/nameser.h>
+#endif
+#if HAVE_RESOLV_H
+#include <resolv.h>
+#endif
+
/* WARNING: Win32 binds http_main.c dynamically to the server. Please place
* extern functions and global data in another appropriate module.
*
@@ -679,6 +689,17 @@ int main(int argc, const char * const ar
apr_pool_clear(pconf);
ap_clear_auth_internal();
+/* glibc has __res_init that is #defined to res_init */
+#if HAVE_RES_INIT || HAVE___RES_INIT
+ /*
+ * resolv.conf may have changed, so this will read/reread it.
+ */
+ if (res_init() == -1) {
+ ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL,
+ "Resolver initialization failed.");
+ }
+#endif
+
for (mod = ap_prelinked_modules; *mod != NULL; mod++) {
ap_register_hooks(*mod, pconf);
}
Re: reducing memory fragmentation
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Sunday 06 February 2011, Stefan Fritsch wrote:
> Therefore I propose to use mmap/mumap in the apr_allocator instead
> of malloc, either as a compile time option or whenever
> max_mem_free is set for the allocator. This would then completely
> separate the allocations from apr_allocator and normal malloc and
> reduce fragmentation. And it would make apr_allocator_max_free_set
> actually give memory back to the OS.
>
> This would be a relatively simple change (compared to replacing the
> whole allocator). Sounds ok?
I have made some measurements for this, doing some relatively complex
requests (a dir index of a dir with 10000 files) with apr 1.4.2/httpd
trunk/mpm_event under Linux/glibc:
reqs/s mem usage
after tests
malloc MaxMemFree unlimited 42 92M
malloc MaxMemFree 128 41 93M
malloc MaxMemFree 4 43 70M
mmap MaxMemFree unlimited 45 85M
mmap MaxMemFree 512 43 66M
mmap MaxMemFree 128 35 31M
mmap MaxMemFree 4 32 21M
The accuracy is not too high, I guess all the numbers are taken to be
+/- 1.
As you see, the the current allocator_alloc version which uses malloc
does not really allow to give back significant ammounts of RAM to the
OS, regardless of MaxMemFree. But with mmap, MaxMemFree has a
significant effect. And with the same ammount of RAM used, the mmap
version is not slower than the malloc version. Therefore with Linux
glibc, the mmap version seems superior in all respects.
I will commit this alternative allocator selectable with a configure
option. For now, it will need to be manually selected. Later, we can
think about changing the default for some platforms.
Cheers,
Stefan
Re: reducing memory fragmentation
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Sunday 20 February 2011, Jim Jagielski wrote:
> On Feb 19, 2011, at 2:57 PM, Stefan Fritsch wrote:
> > I am sure that apr_allocator with mmap is not an advantage on all
> > platforms. Actually, apr allocating only multiples of 4k should
> > make it easier for malloc to avoid fragmentation. But glibc's
> > malloc fails miserably in that regard.
>
> Is there a way during configuration time that we can do
> some rough testing on "best" multiple to use...?
I don't know about the "best" multiple. One would probably have to do
benchmarks for that. But it is possible to check for mallocs that
behave similar to glibc's malloc. If several allocated chunks are not
allocated with distances that are a multiples of the page size, the
malloc is inefficient.
But before activating such a configure check, I would wait for some
feedback from some people using the mmap option in real-world.
For glibc/i386, the attached test program gives:
$ ./malloc_test ; echo $?
pagesize: 4096
minalloc: 8192
malloc offset: 8
1
If you force glibc to use mmap, the result is different. But i don't
know how efficient glibc is in this mode:
$ MALLOC_MMAP_THRESHOLD_=8192 ./malloc_test ; echo $?
pagesize: 4096
minalloc: 8192
malloc offset: 0
0
On FreeBSD amd64:
$ ./malloc_test ; echo $?
pagesize: 4096
minalloc: 8192
malloc offset: 0
0
On non-mainstream hardware (Linux glibc/ia64)
$ ./malloc_test ; echo $?
pagesize: 16384
minalloc: 16384
malloc offset: 16
1
On Linux i386 with tcmalloc:
$ LD_PRELOAD=/usr/lib/libtcmalloc_minimal.so.0.0.0 ./malloc_test ;
echo $?
pagesize: 4096
minalloc: 8192
malloc offset: 0
0
But AFAIK tcmalloc is not a good general choice either, because it
will never give memory back to the OS.
Re: reducing memory fragmentation
Posted by Jim Jagielski <ji...@apache.org>.
On Feb 19, 2011, at 2:57 PM, Stefan Fritsch wrote:
>
> I am sure that apr_allocator with mmap is not an advantage on all
> platforms. Actually, apr allocating only multiples of 4k should make
> it easier for malloc to avoid fragmentation. But glibc's malloc fails
> miserably in that regard.
>
Is there a way during configuration time that we can do
some rough testing on "best" multiple to use...?
Re: reducing memory fragmentation
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Saturday 19 February 2011, Greg Stein wrote:
> On Fri, Feb 18, 2011 at 16:55, Stefan Fritsch <sf...@sfritsch.de>
wrote:
> > On Thursday 17 February 2011, Jim Jagielski wrote:
> >> Please also look at Greg's pocore memory allocation stuff...
> >> his hash stuff is also quite nice. Would be useful to be
> >> able to use that as well....
> >
> > That looks like a much bigger change than what I have just
> > commited. But I agree that before trying to optimize apr's
> > allocator, one should try pocore's.
> >
> > Have you thought about how to do this? Probably every apr pool
> > would be a wrapper for a pocore pool. But what about the other
> > users of apr_allocator, like bucket allocators?
>
> There is a branch in apr for wrapping pocore's pools and hash
> tables[1].
Nice, I didn't know about that.
> Obviously, the indirection slows it down, but it does
> demonstrate how it would work. (and it does: I've run the entire
> svn test suite using this wrapping)
Have you made any measurements how much the slow down is?
> My experiments show that mmap/munmap are NOT speed-advantageous on
> MacOS. But if you're looking at long-term memory usage and avoiding
> fragmentation... I don't have a good way to test that. That said,
> pocore should not be subject to fragmentation like apr. Its
> coalescing feature (designed w/ the APIs present, but not yet
> coded) will avoid much fragmentation.
I am sure that apr_allocator with mmap is not an advantage on all
platforms. Actually, apr allocating only multiples of 4k should make
it easier for malloc to avoid fragmentation. But glibc's malloc fails
miserably in that regard.
For the purpose of httpd giving unused memory back to the OS, your
current apr/pocore branch won't be an improvement because the bucket
allocators still use apr_allocator, which will hold on to some free
memory.
Re: reducing memory fragmentation
Posted by Greg Stein <gs...@gmail.com>.
On Fri, Feb 18, 2011 at 16:55, Stefan Fritsch <sf...@sfritsch.de> wrote:
> On Thursday 17 February 2011, Jim Jagielski wrote:
>> Please also look at Greg's pocore memory allocation stuff...
>> his hash stuff is also quite nice. Would be useful to be
>> able to use that as well....
>
> That looks like a much bigger change than what I have just commited.
> But I agree that before trying to optimize apr's allocator, one should
> try pocore's.
>
> Have you thought about how to do this? Probably every apr pool would
> be a wrapper for a pocore pool. But what about the other users of
> apr_allocator, like bucket allocators?
There is a branch in apr for wrapping pocore's pools and hash
tables[1]. Obviously, the indirection slows it down, but it does
demonstrate how it would work. (and it does: I've run the entire svn
test suite using this wrapping)
Just one comment: use pocore's child pools rather than its "post"
notion. I need to get rid of the posts and simply speed up child
pools.
The basic problem with the apr allocator is its granularity. The reuse
of memory is also a problem. pocore will store arbitrary-sized
segments and unused fragments and whatnot into a red-black tree to
quickly grab it and reuse it.
My experiments show that mmap/munmap are NOT speed-advantageous on
MacOS. But if you're looking at long-term memory usage and avoiding
fragmentation... I don't have a good way to test that. That said,
pocore should not be subject to fragmentation like apr. Its coalescing
feature (designed w/ the APIs present, but not yet coded) will avoid
much fragmentation.
Cheers,
-g
[1] http://svn.apache.org/repos/asf/apr/apr/branches/gstein-pocore/
... see memory/unix/apr_pools.c and tables/apr_hash.c
Re: reducing memory fragmentation
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Thursday 17 February 2011, Jim Jagielski wrote:
> Please also look at Greg's pocore memory allocation stuff...
> his hash stuff is also quite nice. Would be useful to be
> able to use that as well....
That looks like a much bigger change than what I have just commited.
But I agree that before trying to optimize apr's allocator, one should
try pocore's.
Have you thought about how to do this? Probably every apr pool would
be a wrapper for a pocore pool. But what about the other users of
apr_allocator, like bucket allocators?
Re: reducing memory fragmentation
Posted by Jim Jagielski <ji...@jaguNET.com>.
Please also look at Greg's pocore memory allocation stuff...
his hash stuff is also quite nice. Would be useful to be
able to use that as well....
On Feb 6, 2011, at 5:34 PM, Stefan Fritsch wrote:
> On Friday 04 February 2011, William A. Rowe Jr. wrote:
>>>> My thought is that this might not be the only thing to 'refresh'
>>>> when you have an app which needs current state. It would be a
>>>> chance to flush out all caches and set the state basically to
>>>> apr_initialize(), modulo any open resources or currently
>>>> allocated pools. It could even go so far as to free() all the
>>>> unused pool memory, giving us a really fresh start on heap
>>>> pages that have been fragmented to heck.
>>>
>>> ++1 for some way of handling the fragmentation... of course,
>>> we could just switch to pocore for the memory stuff in APR. :)
>>
>> Which helps for apr_allocator but not for the internal
>> fragmentation within individual pools. An optional pocore feature
>> for APR 2 sounds great, patches welcome :)
>
> The pools give their memory back to the allocator when they are
> cleared. Therefore at least the transaction pools and their children
> should not be a major problem for fragmentation.
>
> However, one major problem are the allocators not giving back memory,
> or if configured with max_mem_free, giving it back in random order so
> that it is very unlikely that the malloc implementation can give
> memory back to the OS.
>
> A second problem I have noticed with glibc: Its malloc allocates APR's
> typical multiple-of-4k chunks in a normal linked list. This means that
> there is some N bytes header allocated before each chunk. Now if APR
> has allocated say 16k, and frees it again, the hole is 16k+N. However
> this is not big enough to allocate two 8k chunks, which would need
> (8k+N)*2 == 16k+2N bytes. I think this may contribute to the
> fragmentation seen under Linux. In addition to that, glibc's malloc
> allocates smaller chunks in the same linked list, so in practice we
> have something like this:
>
> 8K pool chunk
> 8K pool chunk
> some bytes from a third party lib
> 12K pool chunk
> some more bytes from normal malloc
> 8K pool chunk
>
> This doesn't help with fragmentation either.
>
> Therefore I propose to use mmap/mumap in the apr_allocator instead of
> malloc, either as a compile time option or whenever max_mem_free is
> set for the allocator. This would then completely separate the
> allocations from apr_allocator and normal malloc and reduce
> fragmentation. And it would make apr_allocator_max_free_set actually
> give memory back to the OS.
>
> This would be a relatively simple change (compared to replacing the
> whole allocator). Sounds ok?
>
reducing memory fragmentation
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Friday 04 February 2011, William A. Rowe Jr. wrote:
> >> My thought is that this might not be the only thing to 'refresh'
> >> when you have an app which needs current state. It would be a
> >> chance to flush out all caches and set the state basically to
> >> apr_initialize(), modulo any open resources or currently
> >> allocated pools. It could even go so far as to free() all the
> >> unused pool memory, giving us a really fresh start on heap
> >> pages that have been fragmented to heck.
> >
> > ++1 for some way of handling the fragmentation... of course,
> > we could just switch to pocore for the memory stuff in APR. :)
>
> Which helps for apr_allocator but not for the internal
> fragmentation within individual pools. An optional pocore feature
> for APR 2 sounds great, patches welcome :)
The pools give their memory back to the allocator when they are
cleared. Therefore at least the transaction pools and their children
should not be a major problem for fragmentation.
However, one major problem are the allocators not giving back memory,
or if configured with max_mem_free, giving it back in random order so
that it is very unlikely that the malloc implementation can give
memory back to the OS.
A second problem I have noticed with glibc: Its malloc allocates APR's
typical multiple-of-4k chunks in a normal linked list. This means that
there is some N bytes header allocated before each chunk. Now if APR
has allocated say 16k, and frees it again, the hole is 16k+N. However
this is not big enough to allocate two 8k chunks, which would need
(8k+N)*2 == 16k+2N bytes. I think this may contribute to the
fragmentation seen under Linux. In addition to that, glibc's malloc
allocates smaller chunks in the same linked list, so in practice we
have something like this:
8K pool chunk
8K pool chunk
some bytes from a third party lib
12K pool chunk
some more bytes from normal malloc
8K pool chunk
This doesn't help with fragmentation either.
Therefore I propose to use mmap/mumap in the apr_allocator instead of
malloc, either as a compile time option or whenever max_mem_free is
set for the allocator. This would then completely separate the
allocations from apr_allocator and normal malloc and reduce
fragmentation. And it would make apr_allocator_max_free_set actually
give memory back to the OS.
This would be a relatively simple change (compared to replacing the
whole allocator). Sounds ok?
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in
server/main.c
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 2/4/2011 3:47 PM, Jim Jagielski wrote:
>
> On Feb 4, 2011, at 4:29 PM, William A. Rowe Jr. wrote:
>
>> On 2/4/2011 3:26 PM, Jim Jagielski wrote:
>>>
>>> On Feb 4, 2011, at 4:14 PM, William A. Rowe Jr. wrote:
>>>
>>>> On 2/4/2011 2:54 PM, Stefan Fritsch wrote:
>>>>>>
>>>>>> +1 here as well, if this is to be addressed at all, the portability
>>>>>> layer seems like the correct place to do it, if desired by the
>>>>>> app.
>>>>>
>>>>> Would you prefer os/unix or APR? I am also happy to simply revert if
>>>>> that is the majority opinion.
>>>>
>>>> I suspect that a generic apr '_refresh' function would be most useful
>>>> across the board, others might disagree.
>>>>
>>>
>>> Isn't APR for "portability"? All I see is a single function
>>> call...
>>>
>>> If we have to pollute something with this res_init(), then httpd
>>> is likely the better place than expanding APR even more beyond
>>> being a "simple" portable runtime ;)
>>>
>>> But I'm fine either way... as long as it builds and links ;)
>>
>> My thought is that this might not be the only thing to 'refresh' when
>> you have an app which needs current state. It would be a chance to
>> flush out all caches and set the state basically to apr_initialize(),
>> modulo any open resources or currently allocated pools. It could even
>> go so far as to free() all the unused pool memory, giving us a really
>> fresh start on heap pages that have been fragmented to heck.
>
> ++1 for some way of handling the fragmentation... of course,
> we could just switch to pocore for the memory stuff in APR. :)
Which helps for apr_allocator but not for the internal fragmentation within
individual pools. An optional pocore feature for APR 2 sounds great, patches
welcome :)
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in server/main.c
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 4, 2011, at 4:29 PM, William A. Rowe Jr. wrote:
> On 2/4/2011 3:26 PM, Jim Jagielski wrote:
>>
>> On Feb 4, 2011, at 4:14 PM, William A. Rowe Jr. wrote:
>>
>>> On 2/4/2011 2:54 PM, Stefan Fritsch wrote:
>>>>>
>>>>> +1 here as well, if this is to be addressed at all, the portability
>>>>> layer seems like the correct place to do it, if desired by the
>>>>> app.
>>>>
>>>> Would you prefer os/unix or APR? I am also happy to simply revert if
>>>> that is the majority opinion.
>>>
>>> I suspect that a generic apr '_refresh' function would be most useful
>>> across the board, others might disagree.
>>>
>>
>> Isn't APR for "portability"? All I see is a single function
>> call...
>>
>> If we have to pollute something with this res_init(), then httpd
>> is likely the better place than expanding APR even more beyond
>> being a "simple" portable runtime ;)
>>
>> But I'm fine either way... as long as it builds and links ;)
>
> My thought is that this might not be the only thing to 'refresh' when
> you have an app which needs current state. It would be a chance to
> flush out all caches and set the state basically to apr_initialize(),
> modulo any open resources or currently allocated pools. It could even
> go so far as to free() all the unused pool memory, giving us a really
> fresh start on heap pages that have been fragmented to heck.
>
++1 for some way of handling the fragmentation... of course,
we could just switch to pocore for the memory stuff in APR. :)
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in server/main.c
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 4, 2011, at 4:29 PM, William A. Rowe Jr. wrote:
> On 2/4/2011 3:26 PM, Jim Jagielski wrote:
>>
>> On Feb 4, 2011, at 4:14 PM, William A. Rowe Jr. wrote:
>>
>>> On 2/4/2011 2:54 PM, Stefan Fritsch wrote:
>>>>>
>>>>> +1 here as well, if this is to be addressed at all, the portability
>>>>> layer seems like the correct place to do it, if desired by the
>>>>> app.
>>>>
>>>> Would you prefer os/unix or APR? I am also happy to simply revert if
>>>> that is the majority opinion.
>>>
>>> I suspect that a generic apr '_refresh' function would be most useful
>>> across the board, others might disagree.
>>>
>>
>> Isn't APR for "portability"? All I see is a single function
>> call...
>>
>> If we have to pollute something with this res_init(), then httpd
>> is likely the better place than expanding APR even more beyond
>> being a "simple" portable runtime ;)
>>
>> But I'm fine either way... as long as it builds and links ;)
>
> My thought is that this might not be the only thing to 'refresh' when
> you have an app which needs current state. It would be a chance to
> flush out all caches and set the state basically to apr_initialize(),
> modulo any open resources or currently allocated pools. It could even
> go so far as to free() all the unused pool memory, giving us a really
> fresh start on heap pages that have been fragmented to heck.
>
++1 for some way of handling the fragmentation... of course,
we could just switch to pocore for the memory stuff in APR. :)
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in
server/main.c
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 2/4/2011 3:26 PM, Jim Jagielski wrote:
>
> On Feb 4, 2011, at 4:14 PM, William A. Rowe Jr. wrote:
>
>> On 2/4/2011 2:54 PM, Stefan Fritsch wrote:
>>>>
>>>> +1 here as well, if this is to be addressed at all, the portability
>>>> layer seems like the correct place to do it, if desired by the
>>>> app.
>>>
>>> Would you prefer os/unix or APR? I am also happy to simply revert if
>>> that is the majority opinion.
>>
>> I suspect that a generic apr '_refresh' function would be most useful
>> across the board, others might disagree.
>>
>
> Isn't APR for "portability"? All I see is a single function
> call...
>
> If we have to pollute something with this res_init(), then httpd
> is likely the better place than expanding APR even more beyond
> being a "simple" portable runtime ;)
>
> But I'm fine either way... as long as it builds and links ;)
My thought is that this might not be the only thing to 'refresh' when
you have an app which needs current state. It would be a chance to
flush out all caches and set the state basically to apr_initialize(),
modulo any open resources or currently allocated pools. It could even
go so far as to free() all the unused pool memory, giving us a really
fresh start on heap pages that have been fragmented to heck.
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in server/main.c
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 4, 2011, at 4:14 PM, William A. Rowe Jr. wrote:
> On 2/4/2011 2:54 PM, Stefan Fritsch wrote:
>>>
>>> +1 here as well, if this is to be addressed at all, the portability
>>> layer seems like the correct place to do it, if desired by the
>>> app.
>>
>> Would you prefer os/unix or APR? I am also happy to simply revert if
>> that is the majority opinion.
>
> I suspect that a generic apr '_refresh' function would be most useful
> across the board, others might disagree.
>
Isn't APR for "portability"? All I see is a single function
call...
If we have to pollute something with this res_init(), then httpd
is likely the better place than expanding APR even more beyond
being a "simple" portable runtime ;)
But I'm fine either way... as long as it builds and links ;)
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in
server/main.c
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 2/4/2011 2:54 PM, Stefan Fritsch wrote:
>>
>> +1 here as well, if this is to be addressed at all, the portability
>> layer seems like the correct place to do it, if desired by the
>> app.
>
> Would you prefer os/unix or APR? I am also happy to simply revert if
> that is the majority opinion.
I suspect that a generic apr '_refresh' function would be most useful
across the board, others might disagree.
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in server/main.c
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Friday 04 February 2011, William A. Rowe Jr. wrote:
> On 2/4/2011 7:18 AM, Jim Jagielski wrote:
> > On Feb 4, 2011, at 4:03 AM, Joe Orton wrote:
> >> On Thu, Feb 03, 2011 at 08:20:02PM -0000, sf@apache.org wrote:
> >>> Author: sf
> >>> Date: Thu Feb 3 20:20:02 2011
> >>> New Revision: 1066944
> >>>
> >>> URL: http://svn.apache.org/viewvc?rev=1066944&view=rev
> >>> Log:
> >>> Reload resolv.conf on graceful restarts
> >>>
> >>> PR: 50619
> >>> Submitted by: Matt Miller <m miller f5 com>, Stefan Fritsch
> >>
> >> Why is it necessary to push out management of the libc resolver
> >> to the daemon? This looks like creeping featurism. You can
> >> and should use nscd if you need running daemons to be able to
> >> cope with resolver config changes on the fly.
I thought to follow the principle of least surprise and reload
resolv.conf during restart. And when the need to change the
nameservers arises, it is usually too late to switch to nscd. Besides,
nscd it a PITA and I wouldn't use it if there was no compelling reason
(like LDAP).
> >
> > ++1.
>
> +1 here as well, if this is to be addressed at all, the portability
> layer seems like the correct place to do it, if desired by the
> app.
Would you prefer os/unix or APR? I am also happy to simply revert if
that is the majority opinion.
Cheers,
Stefan
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in
server/main.c
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 2/4/2011 7:18 AM, Jim Jagielski wrote:
>
> On Feb 4, 2011, at 4:03 AM, Joe Orton wrote:
>
>> On Thu, Feb 03, 2011 at 08:20:02PM -0000, sf@apache.org wrote:
>>> Author: sf
>>> Date: Thu Feb 3 20:20:02 2011
>>> New Revision: 1066944
>>>
>>> URL: http://svn.apache.org/viewvc?rev=1066944&view=rev
>>> Log:
>>> Reload resolv.conf on graceful restarts
>>>
>>> PR: 50619
>>> Submitted by: Matt Miller <m miller f5 com>, Stefan Fritsch
>>
>> Why is it necessary to push out management of the libc resolver to the
>> daemon? This looks like creeping featurism. You can and should use
>> nscd if you need running daemons to be able to cope with resolver config
>> changes on the fly.
>
> ++1.
+1 here as well, if this is to be addressed at all, the portability layer
seems like the correct place to do it, if desired by the app.
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in server/main.c
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 4, 2011, at 4:03 AM, Joe Orton wrote:
> On Thu, Feb 03, 2011 at 08:20:02PM -0000, sf@apache.org wrote:
>> Author: sf
>> Date: Thu Feb 3 20:20:02 2011
>> New Revision: 1066944
>>
>> URL: http://svn.apache.org/viewvc?rev=1066944&view=rev
>> Log:
>> Reload resolv.conf on graceful restarts
>>
>> PR: 50619
>> Submitted by: Matt Miller <m miller f5 com>, Stefan Fritsch
>
> Why is it necessary to push out management of the libc resolver to the
> daemon? This looks like creeping featurism. You can and should use
> nscd if you need running daemons to be able to cope with resolver config
> changes on the fly.
>
++1.
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES
configure.in server/main.c
Posted by Joe Orton <jo...@redhat.com>.
On Thu, Feb 03, 2011 at 08:20:02PM -0000, sf@apache.org wrote:
> Author: sf
> Date: Thu Feb 3 20:20:02 2011
> New Revision: 1066944
>
> URL: http://svn.apache.org/viewvc?rev=1066944&view=rev
> Log:
> Reload resolv.conf on graceful restarts
>
> PR: 50619
> Submitted by: Matt Miller <m miller f5 com>, Stefan Fritsch
Why is it necessary to push out management of the libc resolver to the
daemon? This looks like creeping featurism. You can and should use
nscd if you need running daemons to be able to cope with resolver config
changes on the fly.
Regards, Joe
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in server/main.c
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Feb 4, 2011, at 1:42 PM, William A. Rowe Jr. wrote:
> On 2/4/2011 7:20 AM, Jim Jagielski wrote:
>>
>> On Feb 3, 2011, at 6:03 PM, William A. Rowe Jr. wrote:
>>
>>> On 2/3/2011 5:01 PM, Stefan Fritsch wrote:
>>>> On Thursday 03 February 2011, Jim Jagielski wrote:
>>>>> Don't we also need to now start linking against lresolv??
>>>>>
>>>>> ..../dev/httpd-git-trunk/srclib/apr-util/libaprutil-1.la -lexpat
>>>>> -liconv
>>>>> /Users/jim/src/asf/code/dev/httpd-git-trunk/srclib/apr/libapr-1.la
>>>>> -lpthread Undefined symbols:
>>>>> "_res_9_init", referenced from:
>>>>> _main in libmain.a(main.o)
>>>>> ld: symbol(s) not found
>>>>> collect2: ld returned 1 exit status
>>>>> make[1]: *** [httpd] Error 1
>>>>> make: *** [install-recursive] Error 1
>>>>>
>>>>> after this patch, build now breaks (at least os OS X).
>>>>
>>>> I have read the AC_SEARCH_LIBS docs [1] to mean that it adds -lresolv
>>>> automatically to LIBS if needed. On my system, the test result is
>>>> "none needed", so I can't really test this.
>>>
>>> It's probably not needed because that platform picks up the global namespace,
>>> and resolv was already snagged by loading libapr-1.so from httpd.
>>
>> Was it? From what I can see, only httpd uses res_init which in
>> many systems is *just* in lresolv...
>
> Did Stefan's patch help? We throw away LIBS/LDFLAGS etc fairly often, and
> they may need to be explicitly added.
>
Haven't tried yet...
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in
server/main.c
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 2/4/2011 7:20 AM, Jim Jagielski wrote:
>
> On Feb 3, 2011, at 6:03 PM, William A. Rowe Jr. wrote:
>
>> On 2/3/2011 5:01 PM, Stefan Fritsch wrote:
>>> On Thursday 03 February 2011, Jim Jagielski wrote:
>>>> Don't we also need to now start linking against lresolv??
>>>>
>>>> ..../dev/httpd-git-trunk/srclib/apr-util/libaprutil-1.la -lexpat
>>>> -liconv
>>>> /Users/jim/src/asf/code/dev/httpd-git-trunk/srclib/apr/libapr-1.la
>>>> -lpthread Undefined symbols:
>>>> "_res_9_init", referenced from:
>>>> _main in libmain.a(main.o)
>>>> ld: symbol(s) not found
>>>> collect2: ld returned 1 exit status
>>>> make[1]: *** [httpd] Error 1
>>>> make: *** [install-recursive] Error 1
>>>>
>>>> after this patch, build now breaks (at least os OS X).
>>>
>>> I have read the AC_SEARCH_LIBS docs [1] to mean that it adds -lresolv
>>> automatically to LIBS if needed. On my system, the test result is
>>> "none needed", so I can't really test this.
>>
>> It's probably not needed because that platform picks up the global namespace,
>> and resolv was already snagged by loading libapr-1.so from httpd.
>
> Was it? From what I can see, only httpd uses res_init which in
> many systems is *just* in lresolv...
Did Stefan's patch help? We throw away LIBS/LDFLAGS etc fairly often, and
they may need to be explicitly added.
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in server/main.c
Posted by Jim Jagielski <ji...@apache.org>.
On Feb 3, 2011, at 6:03 PM, William A. Rowe Jr. wrote:
> On 2/3/2011 5:01 PM, Stefan Fritsch wrote:
>> On Thursday 03 February 2011, Jim Jagielski wrote:
>>> Don't we also need to now start linking against lresolv??
>>>
>>> ..../dev/httpd-git-trunk/srclib/apr-util/libaprutil-1.la -lexpat
>>> -liconv
>>> /Users/jim/src/asf/code/dev/httpd-git-trunk/srclib/apr/libapr-1.la
>>> -lpthread Undefined symbols:
>>> "_res_9_init", referenced from:
>>> _main in libmain.a(main.o)
>>> ld: symbol(s) not found
>>> collect2: ld returned 1 exit status
>>> make[1]: *** [httpd] Error 1
>>> make: *** [install-recursive] Error 1
>>>
>>> after this patch, build now breaks (at least os OS X).
>>
>> I have read the AC_SEARCH_LIBS docs [1] to mean that it adds -lresolv
>> automatically to LIBS if needed. On my system, the test result is
>> "none needed", so I can't really test this.
>
> It's probably not needed because that platform picks up the global namespace,
> and resolv was already snagged by loading libapr-1.so from httpd.
>
Was it? From what I can see, only httpd uses res_init which in
many systems is *just* in lresolv...
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in
server/main.c
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 2/3/2011 5:01 PM, Stefan Fritsch wrote:
> On Thursday 03 February 2011, Jim Jagielski wrote:
>> Don't we also need to now start linking against lresolv??
>>
>> ..../dev/httpd-git-trunk/srclib/apr-util/libaprutil-1.la -lexpat
>> -liconv
>> /Users/jim/src/asf/code/dev/httpd-git-trunk/srclib/apr/libapr-1.la
>> -lpthread Undefined symbols:
>> "_res_9_init", referenced from:
>> _main in libmain.a(main.o)
>> ld: symbol(s) not found
>> collect2: ld returned 1 exit status
>> make[1]: *** [httpd] Error 1
>> make: *** [install-recursive] Error 1
>>
>> after this patch, build now breaks (at least os OS X).
>
> I have read the AC_SEARCH_LIBS docs [1] to mean that it adds -lresolv
> automatically to LIBS if needed. On my system, the test result is
> "none needed", so I can't really test this.
It's probably not needed because that platform picks up the global namespace,
and resolv was already snagged by loading libapr-1.so from httpd.
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in server/main.c
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Thursday 03 February 2011, Jim Jagielski wrote:
> Don't we also need to now start linking against lresolv??
>
> ..../dev/httpd-git-trunk/srclib/apr-util/libaprutil-1.la -lexpat
> -liconv
> /Users/jim/src/asf/code/dev/httpd-git-trunk/srclib/apr/libapr-1.la
> -lpthread Undefined symbols:
> "_res_9_init", referenced from:
> _main in libmain.a(main.o)
> ld: symbol(s) not found
> collect2: ld returned 1 exit status
> make[1]: *** [httpd] Error 1
> make: *** [install-recursive] Error 1
>
> after this patch, build now breaks (at least os OS X).
I have read the AC_SEARCH_LIBS docs [1] to mean that it adds -lresolv
automatically to LIBS if needed. On my system, the test result is
"none needed", so I can't really test this.
Maybe we need an explicit APR_ADDTO? Does the attached patch help?
[1] http://www.gnu.org/software/hello/manual/autoconf/Libraries.html
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in
server/main.c
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 2/3/2011 3:48 PM, Jim Jagielski wrote:
> Don't we also need to now start linking against lresolv??
>
> ..../dev/httpd-git-trunk/srclib/apr-util/libaprutil-1.la -lexpat -liconv /Users/jim/src/asf/code/dev/httpd-git-trunk/srclib/apr/libapr-1.la -lpthread
> Undefined symbols:
> "_res_9_init", referenced from:
> _main in libmain.a(main.o)
> ld: symbol(s) not found
> collect2: ld returned 1 exit status
> make[1]: *** [httpd] Error 1
> make: *** [install-recursive] Error 1
>
> after this patch, build now breaks (at least os OS X).
>
> On Feb 3, 2011, at 3:20 PM, sf@apache.org wrote:
>
>> Author: sf
>> Date: Thu Feb 3 20:20:02 2011
>> New Revision: 1066944
>>
>> URL: http://svn.apache.org/viewvc?rev=1066944&view=rev
>> Log:
>> Reload resolv.conf on graceful restarts
>>
>> PR: 50619
>> Submitted by: Matt Miller <m miller f5 com>, Stefan Fritsch
>>
>> Modified:
>> httpd/httpd/trunk/CHANGES
>> httpd/httpd/trunk/configure.in
>> httpd/httpd/trunk/server/main.c
>>
>> Modified: httpd/httpd/trunk/CHANGES
>> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/CHANGES?rev=1066944&r1=1066943&r2=1066944&view=diff
>> ==============================================================================
>> --- httpd/httpd/trunk/CHANGES [utf-8] (original)
>> +++ httpd/httpd/trunk/CHANGES [utf-8] Thu Feb 3 20:20:02 2011
>> @@ -2,6 +2,9 @@
>>
>> Changes with Apache 2.3.11
>>
>> + *) core: Reload resolv.conf on graceful restarts. PR 50619.
>> + [Matt Miller <m miller f5 com>, Stefan Fritsch]
>> +
>> *) mod_mime: Ignore leading dots when looking for mime extensions.
>> PR 50434 [Stefan Fritsch]
>>
>>
>> Modified: httpd/httpd/trunk/configure.in
>> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/configure.in?rev=1066944&r1=1066943&r2=1066944&view=diff
>> ==============================================================================
>> --- httpd/httpd/trunk/configure.in (original)
>> +++ httpd/httpd/trunk/configure.in Thu Feb 3 20:20:02 2011
>> @@ -400,6 +400,7 @@ sys/sem.h \
>> sys/sdt.h
>> )
>> AC_HEADER_SYS_WAIT
>> +AC_HEADER_RESOLV
>>
>> dnl ## Check for typedefs, structures, and compiler characteristics.
>>
>> @@ -416,6 +417,7 @@ LIBS="$saved_LIBS"
>>
>> dnl See Comment #Spoon
>>
>> +AC_SEARCH_LIBS(res_init, resolv, , [AC_SEARCH_LIBS(__res_init,resolv)])
>> AC_CHECK_FUNCS( \
>> getpwnam \
>> getgrnam \
>> @@ -424,7 +426,9 @@ bindprocessor \
>> prctl \
>> timegm \
>> getpgid \
>> -fopen64
>> +fopen64 \
>> +res_init \
>> +__res_init
>> )
>>
>> dnl confirm that a void pointer is large enough to store a long integer
>>
>> Modified: httpd/httpd/trunk/server/main.c
>> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/server/main.c?rev=1066944&r1=1066943&r2=1066944&view=diff
>> ==============================================================================
>> --- httpd/httpd/trunk/server/main.c (original)
>> +++ httpd/httpd/trunk/server/main.c Thu Feb 3 20:20:02 2011
>> @@ -44,6 +44,16 @@
>> #include <unistd.h>
>> #endif
>>
>> +#if HAVE_NETINET_IN_H
>> +#include <netinet/in.h>
>> +#endif
>> +#if HAVE_ARPA_NAMESER_H
>> +#include <arpa/nameser.h>
>> +#endif
>> +#if HAVE_RESOLV_H
>> +#include <resolv.h>
>> +#endif
>> +
>> /* WARNING: Win32 binds http_main.c dynamically to the server. Please place
>> * extern functions and global data in another appropriate module.
>> *
>> @@ -679,6 +689,17 @@ int main(int argc, const char * const ar
>> apr_pool_clear(pconf);
>> ap_clear_auth_internal();
>>
>> +/* glibc has __res_init that is #defined to res_init */
>> +#if HAVE_RES_INIT || HAVE___RES_INIT
>> + /*
>> + * resolv.conf may have changed, so this will read/reread it.
>> + */
>> + if (res_init() == -1) {
>> + ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL,
>> + "Resolver initialization failed.");
>> + }
>> +#endif
>> +
>> for (mod = ap_prelinked_modules; *mod != NULL; mod++) {
>> ap_register_hooks(*mod, pconf);
>> }
This mess absolutely screams for a portable solution in apr.
Re: svn commit: r1066944 - in /httpd/httpd/trunk: CHANGES configure.in server/main.c
Posted by Jim Jagielski <ji...@jaguNET.com>.
Don't we also need to now start linking against lresolv??
..../dev/httpd-git-trunk/srclib/apr-util/libaprutil-1.la -lexpat -liconv /Users/jim/src/asf/code/dev/httpd-git-trunk/srclib/apr/libapr-1.la -lpthread
Undefined symbols:
"_res_9_init", referenced from:
_main in libmain.a(main.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
make[1]: *** [httpd] Error 1
make: *** [install-recursive] Error 1
after this patch, build now breaks (at least os OS X).
On Feb 3, 2011, at 3:20 PM, sf@apache.org wrote:
> Author: sf
> Date: Thu Feb 3 20:20:02 2011
> New Revision: 1066944
>
> URL: http://svn.apache.org/viewvc?rev=1066944&view=rev
> Log:
> Reload resolv.conf on graceful restarts
>
> PR: 50619
> Submitted by: Matt Miller <m miller f5 com>, Stefan Fritsch
>
> Modified:
> httpd/httpd/trunk/CHANGES
> httpd/httpd/trunk/configure.in
> httpd/httpd/trunk/server/main.c
>
> Modified: httpd/httpd/trunk/CHANGES
> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/CHANGES?rev=1066944&r1=1066943&r2=1066944&view=diff
> ==============================================================================
> --- httpd/httpd/trunk/CHANGES [utf-8] (original)
> +++ httpd/httpd/trunk/CHANGES [utf-8] Thu Feb 3 20:20:02 2011
> @@ -2,6 +2,9 @@
>
> Changes with Apache 2.3.11
>
> + *) core: Reload resolv.conf on graceful restarts. PR 50619.
> + [Matt Miller <m miller f5 com>, Stefan Fritsch]
> +
> *) mod_mime: Ignore leading dots when looking for mime extensions.
> PR 50434 [Stefan Fritsch]
>
>
> Modified: httpd/httpd/trunk/configure.in
> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/configure.in?rev=1066944&r1=1066943&r2=1066944&view=diff
> ==============================================================================
> --- httpd/httpd/trunk/configure.in (original)
> +++ httpd/httpd/trunk/configure.in Thu Feb 3 20:20:02 2011
> @@ -400,6 +400,7 @@ sys/sem.h \
> sys/sdt.h
> )
> AC_HEADER_SYS_WAIT
> +AC_HEADER_RESOLV
>
> dnl ## Check for typedefs, structures, and compiler characteristics.
>
> @@ -416,6 +417,7 @@ LIBS="$saved_LIBS"
>
> dnl See Comment #Spoon
>
> +AC_SEARCH_LIBS(res_init, resolv, , [AC_SEARCH_LIBS(__res_init,resolv)])
> AC_CHECK_FUNCS( \
> getpwnam \
> getgrnam \
> @@ -424,7 +426,9 @@ bindprocessor \
> prctl \
> timegm \
> getpgid \
> -fopen64
> +fopen64 \
> +res_init \
> +__res_init
> )
>
> dnl confirm that a void pointer is large enough to store a long integer
>
> Modified: httpd/httpd/trunk/server/main.c
> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/server/main.c?rev=1066944&r1=1066943&r2=1066944&view=diff
> ==============================================================================
> --- httpd/httpd/trunk/server/main.c (original)
> +++ httpd/httpd/trunk/server/main.c Thu Feb 3 20:20:02 2011
> @@ -44,6 +44,16 @@
> #include <unistd.h>
> #endif
>
> +#if HAVE_NETINET_IN_H
> +#include <netinet/in.h>
> +#endif
> +#if HAVE_ARPA_NAMESER_H
> +#include <arpa/nameser.h>
> +#endif
> +#if HAVE_RESOLV_H
> +#include <resolv.h>
> +#endif
> +
> /* WARNING: Win32 binds http_main.c dynamically to the server. Please place
> * extern functions and global data in another appropriate module.
> *
> @@ -679,6 +689,17 @@ int main(int argc, const char * const ar
> apr_pool_clear(pconf);
> ap_clear_auth_internal();
>
> +/* glibc has __res_init that is #defined to res_init */
> +#if HAVE_RES_INIT || HAVE___RES_INIT
> + /*
> + * resolv.conf may have changed, so this will read/reread it.
> + */
> + if (res_init() == -1) {
> + ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL,
> + "Resolver initialization failed.");
> + }
> +#endif
> +
> for (mod = ap_prelinked_modules; *mod != NULL; mod++) {
> ap_register_hooks(*mod, pconf);
> }
>
>