You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modules-dev@httpd.apache.org by Jason Funk <ja...@gmail.com> on 2011/06/20 20:39:48 UTC
Module External Configuration
Hello,
The module that I am writing has an external configuration file that it
parses and loads into configuration when the server loads. Before every
request it checks to see if the configuration file has been updated and if
it has it reloads the configuration. The configuration should be shared over
the entire server. The problem I am running into is that after the
configuration file is updated the new configuration gets reloaded and stored
in memory but after a little while, the configuration reverts back to it's
previous version. I assume that this is because a new process was spawned to
handle a new request and the updated memory didn't get carried over (even
though the pointer address didn't change...)
My question is this: What mechanism should I be using in order to
store persistent, mutable, configuration data that is shared between every
child process?
Jason
Re: Module External Configuration
Posted by Ben Noordhuis <in...@bnoordhuis.nl>.
On Tue, Jun 21, 2011 at 23:26, Jason Funk <ja...@gmail.com> wrote:
> One last question about shared memory...
>
> I have my configuration now being loaded successfully into a shared memory
> segment.. now my problem is that someone could change the config so that the
> resulting structure wouldn't fit in the shared memory segment. Is it
> possible to in the child replace my current shared memory segment with a
> bigger one? I tried destroy()ing and then create()ing but that resulted in a
> segfault. Should it have worked? Is there a different way?
As I've said before, no, you cannot portably resize a shared memory
segment. APR doesn't even expose that functionality.
If you're targeting Unices only, you can use Sys V or POSIX IPC: you
open() or shm_open() a memory segment, then ftruncate() it to the
desired size. Make sure to wrap the call to ftruncate() in an
exclusive lock or bad things will happen.
Re: Module External Configuration
Posted by Joe Lewis <jl...@silverhawk.net>.
On Tue, 2011-06-21 at 16:26 -0500, Jason Funk wrote:
> One last question about shared memory...
>
> I have my configuration now being loaded successfully into a shared memory
> segment.. now my problem is that someone could change the config so that the
> resulting structure wouldn't fit in the shared memory segment. Is it
> possible to in the child replace my current shared memory segment with a
> bigger one? I tried destroy()ing and then create()ing but that resulted in a
> segfault. Should it have worked? Is there a different way?
>
> Jason
segfaults are good indicators of pointers pointing to the wrong things.
It will work if you do it right. Is your pointer to the shared memory
in shared memory? Or do the clients have their own pointer that might
change if the data in shm changes location?
You might have to shm the pointer to a separate shm segment, and then
after reading in a new config, change the shm pointer to point to the
new one and destroy the old config. Don't touch the pointer aside from
changing where it points to.
Joe Lewis
--
http://www.silverhawk.net/
Re: Module External Configuration
Posted by Sorin Manolache <so...@gmail.com>.
On Tue, Jun 21, 2011 at 23:26, Jason Funk <ja...@gmail.com> wrote:
> One last question about shared memory...
>
> I have my configuration now being loaded successfully into a shared memory
> segment.. now my problem is that someone could change the config so that the
> resulting structure wouldn't fit in the shared memory segment. Is it
> possible to in the child replace my current shared memory segment with a
> bigger one? I tried destroy()ing and then create()ing but that resulted in a
> segfault. Should it have worked? Is there a different way?
Destroying the shared memory segment with the conf could be dangerous
as other children could serve requests that are using the conf in the
segment that you are destroying.
Protecting with interprocess locks significantly slows down your
server. Maybe a shared-exclusive lock is more efficient, I don't know.
You could use a double buffer approach and reference counters. For
example you load the new conf in buffer 2, then you update the pointer
that points to the active conf, setting it to buffer 2. Then you check
the reference counter of buffer 1 and when it reaches zero, you
destroy buffer 1. The pointer that points to the active buffer, as
well as the reference counters have to be mapped in shared memory too.
S
>
> Jason
>
> On Mon, Jun 20, 2011 at 5:20 PM, Nick Kew <ni...@apache.org> wrote:
>
>> On Mon, 20 Jun 2011 16:10:12 -0400
>> Mike Meyer <mw...@mired.org> wrote:
>>
>>
>> > I assume that this is because a new process was spawned to
>> > > handle a new request and the updated memory didn't get carried over
>> (even
>> > > though the pointer address didn't change...)
>>
>> A new process may be spawned from time to time (though not directly
>> to handle a new request unless you have a very non-standard MPM).
>> You could check config data in a child_init hook.
>>
>> However, if you have this problem, it's probably not your only one.
>> The data reverting suggests that your module's config may be held
>> on the server/config pool. If that's being updated (and if the
>> updates involve any allocation at all) it's a memory leak. Your
>> module should create its own config pool at child_init. Then it
>> can read its own config data immediately, and - crucially -
>> whenever it re-reads the data it can first clean that pool to
>> reclaim memory.
>>
>> > Put the configuration data into shared memory. Create and load the
>> > shared memory in the post config hook. Map the shared memory and
>> > potentially reload it during the child init hook. You'll need to use
>> > appropriate locking if you decide to reload it. Details on the locking
>> > will depend on the data structure details.
>>
>> If you use shared memory, note that 2.3/2.4 have much better support for
>> it than earlier versions, with mod_socache and mod_slotmem.
>>
>> However, this may not be the best thing to do. You need to weigh up the
>> cost of maintaining a per-process copy of the config data against that
>> of using shared memory. Both are valid solutions.
>>
>> Also, do you want the overhead of a stat() every request to see if
>> anything has changed, or could it work on a time basis, so it only
>> performs a stat() if a certain time has elapsed since the last time?
>>
>> --
>> Nick Kew
>>
>> Available for work, contract or permanent.
>> http://www.webthing.com/~nick/cv.html
>>
>
Re: Module External Configuration
Posted by Jason Funk <ja...@gmail.com>.
One last question about shared memory...
I have my configuration now being loaded successfully into a shared memory
segment.. now my problem is that someone could change the config so that the
resulting structure wouldn't fit in the shared memory segment. Is it
possible to in the child replace my current shared memory segment with a
bigger one? I tried destroy()ing and then create()ing but that resulted in a
segfault. Should it have worked? Is there a different way?
Jason
On Mon, Jun 20, 2011 at 5:20 PM, Nick Kew <ni...@apache.org> wrote:
> On Mon, 20 Jun 2011 16:10:12 -0400
> Mike Meyer <mw...@mired.org> wrote:
>
>
> > I assume that this is because a new process was spawned to
> > > handle a new request and the updated memory didn't get carried over
> (even
> > > though the pointer address didn't change...)
>
> A new process may be spawned from time to time (though not directly
> to handle a new request unless you have a very non-standard MPM).
> You could check config data in a child_init hook.
>
> However, if you have this problem, it's probably not your only one.
> The data reverting suggests that your module's config may be held
> on the server/config pool. If that's being updated (and if the
> updates involve any allocation at all) it's a memory leak. Your
> module should create its own config pool at child_init. Then it
> can read its own config data immediately, and - crucially -
> whenever it re-reads the data it can first clean that pool to
> reclaim memory.
>
> > Put the configuration data into shared memory. Create and load the
> > shared memory in the post config hook. Map the shared memory and
> > potentially reload it during the child init hook. You'll need to use
> > appropriate locking if you decide to reload it. Details on the locking
> > will depend on the data structure details.
>
> If you use shared memory, note that 2.3/2.4 have much better support for
> it than earlier versions, with mod_socache and mod_slotmem.
>
> However, this may not be the best thing to do. You need to weigh up the
> cost of maintaining a per-process copy of the config data against that
> of using shared memory. Both are valid solutions.
>
> Also, do you want the overhead of a stat() every request to see if
> anything has changed, or could it work on a time basis, so it only
> performs a stat() if a certain time has elapsed since the last time?
>
> --
> Nick Kew
>
> Available for work, contract or permanent.
> http://www.webthing.com/~nick/cv.html
>
Re: Module External Configuration
Posted by Nick Kew <ni...@apache.org>.
On Mon, 20 Jun 2011 16:10:12 -0400
Mike Meyer <mw...@mired.org> wrote:
> I assume that this is because a new process was spawned to
> > handle a new request and the updated memory didn't get carried over (even
> > though the pointer address didn't change...)
A new process may be spawned from time to time (though not directly
to handle a new request unless you have a very non-standard MPM).
You could check config data in a child_init hook.
However, if you have this problem, it's probably not your only one.
The data reverting suggests that your module's config may be held
on the server/config pool. If that's being updated (and if the
updates involve any allocation at all) it's a memory leak. Your
module should create its own config pool at child_init. Then it
can read its own config data immediately, and - crucially -
whenever it re-reads the data it can first clean that pool to
reclaim memory.
> Put the configuration data into shared memory. Create and load the
> shared memory in the post config hook. Map the shared memory and
> potentially reload it during the child init hook. You'll need to use
> appropriate locking if you decide to reload it. Details on the locking
> will depend on the data structure details.
If you use shared memory, note that 2.3/2.4 have much better support for
it than earlier versions, with mod_socache and mod_slotmem.
However, this may not be the best thing to do. You need to weigh up the
cost of maintaining a per-process copy of the config data against that
of using shared memory. Both are valid solutions.
Also, do you want the overhead of a stat() every request to see if
anything has changed, or could it work on a time basis, so it only
performs a stat() if a certain time has elapsed since the last time?
--
Nick Kew
Available for work, contract or permanent.
http://www.webthing.com/~nick/cv.html
Re: Module External Configuration
Posted by Mike Meyer <mw...@mired.org>.
On Mon, 20 Jun 2011 13:39:48 -0500
Jason Funk <ja...@gmail.com> wrote:
> Hello,
>
> The module that I am writing has an external configuration file that it
> parses and loads into configuration when the server loads. Before every
> request it checks to see if the configuration file has been updated and if
> it has it reloads the configuration. The configuration should be shared over
> the entire server. The problem I am running into is that after the
> configuration file is updated the new configuration gets reloaded and stored
> in memory but after a little while, the configuration reverts back to it's
> previous version. I assume that this is because a new process was spawned to
> handle a new request and the updated memory didn't get carried over (even
> though the pointer address didn't change...)
>
> My question is this: What mechanism should I be using in order to
> store persistent, mutable, configuration data that is shared between every
> child process?
Put the configuration data into shared memory. Create and load the
shared memory in the post config hook. Map the shared memory and
potentially reload it during the child init hook. You'll need to use
appropriate locking if you decide to reload it. Details on the locking
will depend on the data structure details.
<mike
--
Mike Meyer <mw...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Re: Module External Configuration
Posted by Ben Noordhuis <in...@bnoordhuis.nl>.
On Mon, Jun 20, 2011 at 22:46, Jason Funk <ja...@gmail.com> wrote:
> I have moved my configuration over to shared memory (following
> mod_shm_counter as an example) and it conceptually seems to be working. I am
> storing a struct in the memory and members that share it's memory (such as
> the last mod time of the configuration file) persist over multiple children.
> However, when I am loading the configuration file I have to allocate
> additional memory. This apparently doesn't get stored in the same shared
> memory as it isn't sticking around. How do I dynamically allocate new memory
> to the struct that lives in shared memory?
You cannot (portably), you'll have to reserve the extra space when you
the allocate the shared memory.
Re: Module External Configuration
Posted by Sorin Manolache <so...@gmail.com>.
On Mon, Jun 20, 2011 at 22:46, Jason Funk <ja...@gmail.com> wrote:
> I have moved my configuration over to shared memory (following
> mod_shm_counter as an example) and it conceptually seems to be working. I am
> storing a struct in the memory and members that share it's memory (such as
> the last mod time of the configuration file) persist over multiple children.
> However, when I am loading the configuration file I have to allocate
> additional memory. This apparently doesn't get stored in the same shared
> memory as it isn't sticking around. How do I dynamically allocate new memory
> to the struct that lives in shared memory?
Try C++ and boost::interprocess::managed_shared_memory and
boost::interprocess::allocator.
Sorin
>
> Jason
>
> On Mon, Jun 20, 2011 at 3:29 PM, Neil McKee <ne...@inmon.com> wrote:
>
>> In the mod-sflow implementation I have one thread responsible for reading
>> in new configuration as it changes and writing it to a shared-memory area
>> where the worker-processes/threads can pick it up whenever it changes. I
>> don't know if that is the best way or not, but it's one data point for you.
>> Source code is here:
>>
>> http://mod-sflow.googlecode.com
>>
>> Neil
>>
>>
>> On Jun 20, 2011, at 11:39 AM, Jason Funk wrote:
>>
>> > Hello,
>> >
>> > The module that I am writing has an external configuration file that it
>> > parses and loads into configuration when the server loads. Before every
>> > request it checks to see if the configuration file has been updated and
>> if
>> > it has it reloads the configuration. The configuration should be shared
>> over
>> > the entire server. The problem I am running into is that after the
>> > configuration file is updated the new configuration gets reloaded and
>> stored
>> > in memory but after a little while, the configuration reverts back to
>> it's
>> > previous version. I assume that this is because a new process was spawned
>> to
>> > handle a new request and the updated memory didn't get carried over (even
>> > though the pointer address didn't change...)
>> >
>> > My question is this: What mechanism should I be using in order to
>> > store persistent, mutable, configuration data that is shared between
>> every
>> > child process?
>> >
>> >
>> > Jason
>>
>>
>
Re: Module External Configuration
Posted by Jason Funk <ja...@gmail.com>.
I have moved my configuration over to shared memory (following
mod_shm_counter as an example) and it conceptually seems to be working. I am
storing a struct in the memory and members that share it's memory (such as
the last mod time of the configuration file) persist over multiple children.
However, when I am loading the configuration file I have to allocate
additional memory. This apparently doesn't get stored in the same shared
memory as it isn't sticking around. How do I dynamically allocate new memory
to the struct that lives in shared memory?
Jason
On Mon, Jun 20, 2011 at 3:29 PM, Neil McKee <ne...@inmon.com> wrote:
> In the mod-sflow implementation I have one thread responsible for reading
> in new configuration as it changes and writing it to a shared-memory area
> where the worker-processes/threads can pick it up whenever it changes. I
> don't know if that is the best way or not, but it's one data point for you.
> Source code is here:
>
> http://mod-sflow.googlecode.com
>
> Neil
>
>
> On Jun 20, 2011, at 11:39 AM, Jason Funk wrote:
>
> > Hello,
> >
> > The module that I am writing has an external configuration file that it
> > parses and loads into configuration when the server loads. Before every
> > request it checks to see if the configuration file has been updated and
> if
> > it has it reloads the configuration. The configuration should be shared
> over
> > the entire server. The problem I am running into is that after the
> > configuration file is updated the new configuration gets reloaded and
> stored
> > in memory but after a little while, the configuration reverts back to
> it's
> > previous version. I assume that this is because a new process was spawned
> to
> > handle a new request and the updated memory didn't get carried over (even
> > though the pointer address didn't change...)
> >
> > My question is this: What mechanism should I be using in order to
> > store persistent, mutable, configuration data that is shared between
> every
> > child process?
> >
> >
> > Jason
>
>
Re: Module External Configuration
Posted by Neil McKee <ne...@inmon.com>.
In the mod-sflow implementation I have one thread responsible for reading in new configuration as it changes and writing it to a shared-memory area where the worker-processes/threads can pick it up whenever it changes. I don't know if that is the best way or not, but it's one data point for you. Source code is here:
http://mod-sflow.googlecode.com
Neil
On Jun 20, 2011, at 11:39 AM, Jason Funk wrote:
> Hello,
>
> The module that I am writing has an external configuration file that it
> parses and loads into configuration when the server loads. Before every
> request it checks to see if the configuration file has been updated and if
> it has it reloads the configuration. The configuration should be shared over
> the entire server. The problem I am running into is that after the
> configuration file is updated the new configuration gets reloaded and stored
> in memory but after a little while, the configuration reverts back to it's
> previous version. I assume that this is because a new process was spawned to
> handle a new request and the updated memory didn't get carried over (even
> though the pointer address didn't change...)
>
> My question is this: What mechanism should I be using in order to
> store persistent, mutable, configuration data that is shared between every
> child process?
>
>
> Jason