You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Ian Holsman <ia...@cnet.com> on 2001/07/05 06:01:35 UTC
[PATCH/CONTRIB] Shared Mem Hash Table
This implements a hash table using shared memory.
Limitations:
* doesn't have a freelist, so deletes aren't reclaimed
* and can't expand it's size (dont know how to implement this with
current shared memory system)
* no locking (yet)
It is intended for use in modules having a large config requirements
(Patches inline, extra files attached)
It is intented that apr_shm_hash.c will live in apr/tables
apr_shm_hash.h in apr/include
&
testhash.c in apr/test
Index: tables/Makefile.in
===================================================================
RCS file: /home/cvspublic/apr/tables/Makefile.in,v
retrieving revision 1.6
diff -u -u -r1.6 Makefile.in
--- tables/Makefile.in 2001/01/09 11:06:18 1.6
+++ tables/Makefile.in 2001/07/05 03:53:02
@@ -1,5 +1,5 @@
-TARGETS = apr_tables.lo apr_hash.lo
+TARGETS = apr_tables.lo apr_hash.lo apr_shm_hash.lo
# bring in rules.mk for standard functionality
@INCLUDE_RULES@
Index: test/Makefile.in
===================================================================
RCS file: /home/cvspublic/apr/test/Makefile.in,v
retrieving revision 1.57
diff -u -u -r1.57 Makefile.in
--- test/Makefile.in 2001/06/15 20:04:39 1.57
+++ test/Makefile.in 2001/07/05 03:53:02
@@ -25,7 +25,8 @@
testpoll@EXEEXT@ \
testmem@EXEEXT@ \
occhild@EXEEXT@ \
- teststr@EXEEXT@
+ teststr@EXEEXT@ \
+ testhash@EXEEXT@
TARGETS = $(PROGRAMS)
@@ -125,5 +126,8 @@
teststr@EXEEXT@: teststr.lo $(LOCAL_LIBS)
$(LINK) teststr.lo $(LOCAL_LIBS) $(ALL_LIBS)
+
+testhash@EXEEXT@: testhash.lo $(LOCAL_LIBS)
+ $(LINK) testhash.lo $(LOCAL_LIBS) $(ALL_LIBS)
# DO NOT REMOVE
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
On Sat, Jul 07, 2001 at 08:33:54AM +0100, David Reid wrote:
> I'd be interested to know of other tools that people use that are available
> for others here * on the list to use improving the code. So far when pools
insure-lite may help _if_ you use the following trick.
insure-lite is availabe in the non-free section of
redhat 6 cds. insure itself is about $4,000 USD and
no, they have never heard of open source projects
so they are not interested in providing licenses as
a promotional loss-leader [in complete contrast to
vmware who are very supportive of open source].
these code fragments come from samba source (GPL):
/*******************************************************************
close the low 3 fd's and open /dev/null in their place
********************************************************************/
void close_low_fds(void)
{
int fd;
int i;
close(0);
close(1);
#ifndef __INSURE__
close(2);
#endif
/* try and use up these file descriptors, so silly
library routines writing to stdout etc won't cause havoc */
for (i = 0; i < 3; i++)
{
fd = open("/dev/null", O_RDWR, 0);
...
...
[except that insure writes out to stderr, so you can't close
that one]
#ifdef __INSURE__
/*******************************************************************
This routine is a trick to immediately catch errors when debugging
with insure. A xterm with a gdb is popped up when insure catches
a error. It is Linux specific.
********************************************************************/
int _Insure_trapr_error(int a1, int a2, int a3, int a4, int a5, int a6)
{
static int (*fn) ();
int ret;
char pidstr[10];
pstring cmd =
"/usr/X11R6/bin/xterm -display :0 -T Panic -n Panic -e /bin/sh -c 'cat /tmp/ierrs.*.%d ; gdb /proc/%d/exe %d'";
slprintf(pidstr, sizeof(pidstr), "%d", sys_getpid());
pstring_sub(cmd, "%d", pidstr);
if (!fn)
{
static void *h;
h =
dlopen
("/usr/local/parasoft/insure++lite/lib.linux2/libinsure.so",
RTLD_LAZY);
fn = dlsym(h, "_Insure_trapr_error");
}
ret = fn(a1, a2, a3, a4, a5, a6);
system(cmd);
return ret;
}
#endif
[this little trick _is_ actually documented in the insure
documentation. insure-lite strips out stack-debugging
info, making it almost completely useless... _unless_
you use the above little trick.]
insure is *extremely* good. if you have used purify, you will
be laughing at it.
insure detects at *run-time*:
- pointer overwriting:
a = malloc(20);
a = b;
return;
(but it doesn,t detect this:
struct foo
{
void *a;
}
f->a = malloc(20);
memset(f, 0, sizeof(struct foo));
or, it _does_, it detects it as a loss of memory f->a
but not _why_.
but _does_ detect this:
struct foo fa;
struct foo fb;
fa.a = malloc(20);
fa = fb;
cool, huh? :)
- underwriting and overwriting of memory regions and strings
- not freed, freed more than once memory
so, when you do an apr_pool_destroy, it will let you
know immediately if there are any problems AND, get this:
it tells you the stack at the point where the memory
was ALLOCATED!
etc. there's lots more. it does a little bit of
compile-time checking, too.
the down-side: a normal 3mb executable compiles up at
80mb. a 30-times increment in binary size....
> One of the things we really need to improve is our test suite.
yespleasetestsuitesmakeforeasycuttingandpastingtogetrealliveworkingcoderealquick :)
luke
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by David Reid <dr...@jetnet.co.uk>.
Guys,
I'm going to try and add a configure option to allow us to add Electric Fence
support to apache, but will it be useful for APR as well? It's a simple
enough patch as I'm not planning on much autodetection, simply having the user
identify where it is and have a couple of the more common locations checked if
they don't. (I thought --with-ef=/path - thoughts??)
I'd be interested to know of other tools that people use that are available
for others here * on the list to use improving the code. So far when pools
are sms the memory usage is higher than with pools alone (frighteningly so)
but this seems crazy given in testmem the same test with the same setup as per
the pools == sms code isn't anywhere near as hungry. Probably some simplt
optimisation and it'll be fixed :) Tools to let us know this and find the
bottlenecks are incredibly useful.
One of the things we really need to improve is our test suite. Ian, are you
planning on adding your recent test app for pools? We also need a better
buckets test app. The one I quickly wrote in an hour or so at IBM isn't much
use :(
* By which I mean not costing $$$$
david
Cliff Woolley wrote:
> On Fri, 6 Jul 2001, Brian Pane wrote:
>
> > This morning, I instrumented the latest httpd-2.0 souce and found that
> > it's spending 41% of its CPU time in usr mode and 59% in sys when
> > delivering a 50KB, non-server-parsed file (on Linux, using sendfile).
> > For server-parsed requests, I've seen ratios as high as 70% usr, 30% sys.
> >
> > The bad news: the httpd is wasting a huge amount of CPU time in
> > user-space code (for an optimal httpd serving static files, the usr CPU
> > should approach zero; 41% is surprisingly high). The good news: there's
> > an opportunity to cut the httpd's CPU utilization nearly in half by
> > tuning just the user-space code.
>
> The flip side of that, of course, is that we could probably also get rid
> of some useless/redundant system calls, thereby also reducing the time
> spent in sys CPU.
>
> --Cliff
>
> --------------------------------------------------------------
> Cliff Woolley
> cliffwoolley@yahoo.com
> Charlottesville, VA
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Cliff Woolley <cl...@yahoo.com>.
On Fri, 6 Jul 2001, Brian Pane wrote:
> This morning, I instrumented the latest httpd-2.0 souce and found that
> it's spending 41% of its CPU time in usr mode and 59% in sys when
> delivering a 50KB, non-server-parsed file (on Linux, using sendfile).
> For server-parsed requests, I've seen ratios as high as 70% usr, 30% sys.
>
> The bad news: the httpd is wasting a huge amount of CPU time in
> user-space code (for an optimal httpd serving static files, the usr CPU
> should approach zero; 41% is surprisingly high). The good news: there's
> an opportunity to cut the httpd's CPU utilization nearly in half by
> tuning just the user-space code.
The flip side of that, of course, is that we could probably also get rid
of some useless/redundant system calls, thereby also reducing the time
spent in sys CPU.
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Brian Pane <bp...@pacbell.net>.
Ian Holsman wrote:
>On 05 Jul 2001 22:59:25 +0100, David Reid wrote:
>
>>OK, to comment on this prior to the patch being posted...
>>
>>I have SMS running as pools and the implementation I have manages to run the
>>server at 80% of the speed of pools. This is with no changes to the httpd
>>code except using --enable-sms on the configure line. Now, this isn't
>>amazing, but does present a starting point. As I said we haven't started
>>optimising or altering.
>>
[...]
>If you post your patch, I'll run it through quantify which should
>highlight where the bottlenecks are.
>
Detailed profile data from Quantify would be extremely useful--not
just for the new SMS-based pools, but also for APR and the httpd in
general.
This morning, I instrumented the latest httpd-2.0 souce and found that
it's spending 41% of its CPU time in usr mode and 59% in sys when
delivering a 50KB, non-server-parsed file (on Linux, using sendfile).
For server-parsed requests, I've seen ratios as high as 70% usr, 30% sys.
The bad news: the httpd is wasting a huge amount of CPU time in
user-space code (for an optimal httpd serving static files, the usr CPU
should approach zero; 41% is surprisingly high). The good news: there's
an opportunity to cut the httpd's CPU utilization nearly in half by
tuning just the user-space code.
--Brian
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Ian Holsman <ia...@cnet.com>.
On 05 Jul 2001 22:59:25 +0100, David Reid wrote:
> OK, to comment on this prior to the patch being posted...
>
> I have SMS running as pools and the implementation I have manages to run the
> server at 80% of the speed of pools. This is with no changes to the httpd
> code except using --enable-sms on the configure line. Now, this isn't
> amazing, but does present a starting point. As I said we haven't started
> optimising or altering.
>
> We've found a load of problems and I'd like to Thank ben L for his support
> and IRC server while Sander and I discussed/debated/debugged these problems
> online this week. We've been using Electric Fence to try and find problems
> and it's worked well for us, so maybe we'll add it as an option to
> configure? Fixes have been committed for some of these and more will likely
> be on the way. Thoughts?
>
> However, the fact remains that SMS after only a couple of months is ready
> for use in httpd albeit with a performance penalty. Platform known to work
> was FreeBSD 4.3 and prefork MPM. The code used was less tha ideal and
> faster code is on the way :) By the time the patch gets to you all on
> saturday I hope it'll be a bit quicker, but as a starting point it throws up
> some issues :)
>
> I'm in Dubai at the moment and after the frenetic pace of the last couple of
> days I plan to take some time off in the morning round the pool :)
>
> Does that whet your appetites enough?
>
> Basically, we need more eyes looking at the code/implementations as soon as
> possible, and this patch may enable that :)
>
> Sorry Justin, have to be patient! (Not one of your strong points I know)
> Small matter of getting connectivity without it costing me a fortune before
> I commit :)
If you post your patch, I'll run it through quantify which should
highlight where the bottlenecks are.
>
> david
>
> > On Thu, Jul 05, 2001 at 08:36:24AM -0700, Ian Holsman wrote:
> > > In the conversations I have seen SMS was always talked about post 2.0
> > > Release.
> > > Is this still the case?
> >
> > Nah. I think SMS is getting very close to being ready for more people
> > to play with. In order to solve some problems, we're going to have to
> > use SMS. David and Sander have made lots of progress with this. (As
> > I said, I spent yesterday going over the SMS code...)
> >
> > David just got an httpd up that uses no pools, but just SMS. I believe
> > that the API is identical from pools to SMS, but I'm not sure. He has
> > all of the details. I'd bet the pool creation API might be slightly
> > tweaked.
> >
> > Now that we can get an httpd up and serving pages, that means it now
> > needs more people playing with it - i.e. starting to enable this in the
> > mainline code. It might be a configure time option in the short term
> > so we don't screw everyone, but I think it'll be happening fairly soon.
> > -- justin
> >
> >
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by David Reid <dr...@jetnet.co.uk>.
OK, to comment on this prior to the patch being posted...
I have SMS running as pools and the implementation I have manages to run the
server at 80% of the speed of pools. This is with no changes to the httpd
code except using --enable-sms on the configure line. Now, this isn't
amazing, but does present a starting point. As I said we haven't started
optimising or altering.
We've found a load of problems and I'd like to Thank ben L for his support
and IRC server while Sander and I discussed/debated/debugged these problems
online this week. We've been using Electric Fence to try and find problems
and it's worked well for us, so maybe we'll add it as an option to
configure? Fixes have been committed for some of these and more will likely
be on the way. Thoughts?
However, the fact remains that SMS after only a couple of months is ready
for use in httpd albeit with a performance penalty. Platform known to work
was FreeBSD 4.3 and prefork MPM. The code used was less tha ideal and
faster code is on the way :) By the time the patch gets to you all on
saturday I hope it'll be a bit quicker, but as a starting point it throws up
some issues :)
I'm in Dubai at the moment and after the frenetic pace of the last couple of
days I plan to take some time off in the morning round the pool :)
Does that whet your appetites enough?
Basically, we need more eyes looking at the code/implementations as soon as
possible, and this patch may enable that :)
Sorry Justin, have to be patient! (Not one of your strong points I know)
Small matter of getting connectivity without it costing me a fortune before
I commit :)
david
> On Thu, Jul 05, 2001 at 08:36:24AM -0700, Ian Holsman wrote:
> > In the conversations I have seen SMS was always talked about post 2.0
> > Release.
> > Is this still the case?
>
> Nah. I think SMS is getting very close to being ready for more people
> to play with. In order to solve some problems, we're going to have to
> use SMS. David and Sander have made lots of progress with this. (As
> I said, I spent yesterday going over the SMS code...)
>
> David just got an httpd up that uses no pools, but just SMS. I believe
> that the API is identical from pools to SMS, but I'm not sure. He has
> all of the details. I'd bet the pool creation API might be slightly
> tweaked.
>
> Now that we can get an httpd up and serving pages, that means it now
> needs more people playing with it - i.e. starting to enable this in the
> mainline code. It might be a configure time option in the short term
> so we don't screw everyone, but I think it'll be happening fairly soon.
> -- justin
>
>
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Thu, Jul 05, 2001 at 08:36:24AM -0700, Ian Holsman wrote:
> In the conversations I have seen SMS was always talked about post 2.0
> Release.
> Is this still the case?
Nah. I think SMS is getting very close to being ready for more people
to play with. In order to solve some problems, we're going to have to
use SMS. David and Sander have made lots of progress with this. (As
I said, I spent yesterday going over the SMS code...)
David just got an httpd up that uses no pools, but just SMS. I believe
that the API is identical from pools to SMS, but I'm not sure. He has
all of the details. I'd bet the pool creation API might be slightly
tweaked.
Now that we can get an httpd up and serving pages, that means it now
needs more people playing with it - i.e. starting to enable this in the
mainline code. It might be a configure time option in the short term
so we don't screw everyone, but I think it'll be happening fairly soon.
-- justin
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by rb...@covalent.net.
> On 05 Jul 2001 09:14:03 -0700, Ian Holsman wrote:
> > On 05 Jul 2001 08:46:05 -0700, rbb@covalent.net wrote:
> > > On 5 Jul 2001, Ian Holsman wrote:
> > >
> > > > On 04 Jul 2001 21:16:56 -0700, Justin Erenkrantz wrote:
> > > > > On Wed, Jul 04, 2001 at 09:01:35PM -0700, Ian Holsman wrote:
> > > > > > This implements a hash table using shared memory.
> > > > > > Limitations:
> > > > > > * doesn't have a freelist, so deletes aren't reclaimed
> > > > > > * and can't expand it's size (dont know how to implement this with
> > > > > > current shared memory system)
> > > > > > * no locking (yet)
> > > > > > It is intended for use in modules having a large config requirements
> > > > > > (Patches inline, extra files attached)
> > > > > >
> > > > > > It is intented that apr_shm_hash.c will live in apr/tables
> > > > > > apr_shm_hash.h in apr/include
> > > > > > &
> > > > > > testhash.c in apr/test
> > > > >
> > > > > Cool. Can we revisit this once we have reworked the apr_shm interface?
> > > > > I guess someone could commit this now, but it might make sense to hold
> > > > > off until we have a decent SHM interface. And, ideally SHM could/should
> > > > > be implemented around SMS. If so, that might change how this is
> > > > > implemented. -- justin
> > > >
> > > > In the conversations I have seen SMS was always talked about post 2.0
> > > > Release.
> > > > Is this still the case?
> > >
> > > Nobody knows. There are a lot of people focusing on the SMS code, so that
> > > means it is likely to be ready soon. Regardless, adding a shared memory
> > > hash is a dangerous thing for us to do. We need to support that API
> > > forever. We know already that we want to solve this problem with a shared
> > > memory SMS, and our current hash code. I would much rather not litter our
> > > API with functions that we know we don't want to support for very long.
> > >
> >
> > Cool... I'll just implement the patch privately.
> >
> > as a side note;
> > I don't see how you plan on implementing any kind of dynamic structure
> > via a shared-memory SMS which will look to the rest of the code just the
> > same as a non-shared-memory SMS.
> >
> > the basic problem I see is that you will not be able to 'point' to
> > memory in a Shared SMS, just offsets.
You can handle that with offsets. All of that should be hidden behind the
SMS interface. If it isn't, then the SMS interface is broken.
> > so your code which would call the SMS would have to have 2 different
> > implementations.
> >
> > the other problem is that you can't just allocated another block when
> > you run out of space in your first one, as the memory allocated for the
> > second block may not be the at the same offset from your original one
> > cross process.
Everything in Computer Science can be done with one more level of
indirection. :-)
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Ian Holsman <ia...@cnet.com>.
oops...forgot the list.
On 05 Jul 2001 09:14:03 -0700, Ian Holsman wrote:
> On 05 Jul 2001 08:46:05 -0700, rbb@covalent.net wrote:
> > On 5 Jul 2001, Ian Holsman wrote:
> >
> > > On 04 Jul 2001 21:16:56 -0700, Justin Erenkrantz wrote:
> > > > On Wed, Jul 04, 2001 at 09:01:35PM -0700, Ian Holsman wrote:
> > > > > This implements a hash table using shared memory.
> > > > > Limitations:
> > > > > * doesn't have a freelist, so deletes aren't reclaimed
> > > > > * and can't expand it's size (dont know how to implement this with
> > > > > current shared memory system)
> > > > > * no locking (yet)
> > > > > It is intended for use in modules having a large config requirements
> > > > > (Patches inline, extra files attached)
> > > > >
> > > > > It is intented that apr_shm_hash.c will live in apr/tables
> > > > > apr_shm_hash.h in apr/include
> > > > > &
> > > > > testhash.c in apr/test
> > > >
> > > > Cool. Can we revisit this once we have reworked the apr_shm interface?
> > > > I guess someone could commit this now, but it might make sense to hold
> > > > off until we have a decent SHM interface. And, ideally SHM could/should
> > > > be implemented around SMS. If so, that might change how this is
> > > > implemented. -- justin
> > >
> > > In the conversations I have seen SMS was always talked about post 2.0
> > > Release.
> > > Is this still the case?
> >
> > Nobody knows. There are a lot of people focusing on the SMS code, so that
> > means it is likely to be ready soon. Regardless, adding a shared memory
> > hash is a dangerous thing for us to do. We need to support that API
> > forever. We know already that we want to solve this problem with a shared
> > memory SMS, and our current hash code. I would much rather not litter our
> > API with functions that we know we don't want to support for very long.
> >
>
> Cool... I'll just implement the patch privately.
>
> as a side note;
> I don't see how you plan on implementing any kind of dynamic structure
> via a shared-memory SMS which will look to the rest of the code just the
> same as a non-shared-memory SMS.
>
> the basic problem I see is that you will not be able to 'point' to
> memory in a Shared SMS, just offsets.
>
> so your code which would call the SMS would have to have 2 different
> implementations.
>
> the other problem is that you can't just allocated another block when
> you run out of space in your first one, as the memory allocated for the
> second block may not be the at the same offset from your original one
> cross process.
>
> so I guess what i am rambling about is that the type of memory returned
> would be completly different than the current SMS allocators, and you
> would need different data structures to handle these different types of
> memory, so I am still unsure on how you would have set of 'hash'
> functions which would work on both types unless you implement the
> structures without pointers and only use offsets within one memory
> block.
>
> ..ian
>
> > Ryan
> >
> > _____________________________________________________________________________
> > Ryan Bloom rbb@apache.org
> > Covalent Technologies rbb@covalent.net
> > -----------------------------------------------------------------------------
>
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by rb...@covalent.net.
On 5 Jul 2001, Ian Holsman wrote:
> On 04 Jul 2001 21:16:56 -0700, Justin Erenkrantz wrote:
> > On Wed, Jul 04, 2001 at 09:01:35PM -0700, Ian Holsman wrote:
> > > This implements a hash table using shared memory.
> > > Limitations:
> > > * doesn't have a freelist, so deletes aren't reclaimed
> > > * and can't expand it's size (dont know how to implement this with
> > > current shared memory system)
> > > * no locking (yet)
> > > It is intended for use in modules having a large config requirements
> > > (Patches inline, extra files attached)
> > >
> > > It is intented that apr_shm_hash.c will live in apr/tables
> > > apr_shm_hash.h in apr/include
> > > &
> > > testhash.c in apr/test
> >
> > Cool. Can we revisit this once we have reworked the apr_shm interface?
> > I guess someone could commit this now, but it might make sense to hold
> > off until we have a decent SHM interface. And, ideally SHM could/should
> > be implemented around SMS. If so, that might change how this is
> > implemented. -- justin
>
> In the conversations I have seen SMS was always talked about post 2.0
> Release.
> Is this still the case?
Nobody knows. There are a lot of people focusing on the SMS code, so that
means it is likely to be ready soon. Regardless, adding a shared memory
hash is a dangerous thing for us to do. We need to support that API
forever. We know already that we want to solve this problem with a shared
memory SMS, and our current hash code. I would much rather not litter our
API with functions that we know we don't want to support for very long.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Ian Holsman <ia...@cnet.com>.
On 04 Jul 2001 21:16:56 -0700, Justin Erenkrantz wrote:
> On Wed, Jul 04, 2001 at 09:01:35PM -0700, Ian Holsman wrote:
> > This implements a hash table using shared memory.
> > Limitations:
> > * doesn't have a freelist, so deletes aren't reclaimed
> > * and can't expand it's size (dont know how to implement this with
> > current shared memory system)
> > * no locking (yet)
> > It is intended for use in modules having a large config requirements
> > (Patches inline, extra files attached)
> >
> > It is intented that apr_shm_hash.c will live in apr/tables
> > apr_shm_hash.h in apr/include
> > &
> > testhash.c in apr/test
>
> Cool. Can we revisit this once we have reworked the apr_shm interface?
> I guess someone could commit this now, but it might make sense to hold
> off until we have a decent SHM interface. And, ideally SHM could/should
> be implemented around SMS. If so, that might change how this is
> implemented. -- justin
In the conversations I have seen SMS was always talked about post 2.0
Release.
Is this still the case?
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Wed, Jul 04, 2001 at 09:01:35PM -0700, Ian Holsman wrote:
> This implements a hash table using shared memory.
> Limitations:
> * doesn't have a freelist, so deletes aren't reclaimed
> * and can't expand it's size (dont know how to implement this with
> current shared memory system)
> * no locking (yet)
> It is intended for use in modules having a large config requirements
> (Patches inline, extra files attached)
>
> It is intented that apr_shm_hash.c will live in apr/tables
> apr_shm_hash.h in apr/include
> &
> testhash.c in apr/test
Cool. Can we revisit this once we have reworked the apr_shm interface?
I guess someone could commit this now, but it might make sense to hold
off until we have a decent SHM interface. And, ideally SHM could/should
be implemented around SMS. If so, that might change how this is
implemented. -- justin
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Ian Holsman <ia...@cnet.com>.
On 05 Jul 2001 09:07:26 -0400, Bill Stoddard wrote:
> > This implements a hash table using shared memory.
> > Limitations:
> > * doesn't have a freelist, so deletes aren't reclaimed
> > * and can't expand it's size (dont know how to implement this with current shared
> memory system)
>
> To the best of my knowledge, you cannot expand a shared memory segment (after fork) and
> ensure that all processes use the exact same addresses to access that memory. If one of
> your processes increases the size of the hash table, you have forever screwed up the hash
> table for all the other processes because you now have addresses in the table that may
> point to completely different storage in all the other processes.
>
yep... thats true.
what I was planning to do was to have a 'generation' in the header
section (which never changes size), and the non-shared memory section,
and we would increment this when we expanded. the get/set functions
would check the generation flag before anything and would re-open the
shared memory segments NAME+GEN. ...but with the current shared mem
implementation only the parent process can create shared memory, so I
left it as a limitation, as the future SMS might fix this
> Bill
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by Bill Stoddard <bi...@wstoddard.com>.
> This implements a hash table using shared memory.
> Limitations:
> * doesn't have a freelist, so deletes aren't reclaimed
> * and can't expand it's size (dont know how to implement this with current shared
memory system)
To the best of my knowledge, you cannot expand a shared memory segment (after fork) and
ensure that all processes use the exact same addresses to access that memory. If one of
your processes increases the size of the hash table, you have forever screwed up the hash
table for all the other processes because you now have addresses in the table that may
point to completely different storage in all the other processes.
Bill
Re: [PATCH/CONTRIB] Shared Mem Hash Table
Posted by rb...@covalent.net.
I dislike this. This duplicates a lot of logic that we already have in
regular hash files. The whole point of the SMS logic is to make this sort
of thing not necessary anymore. With SMS, we just pass a shared pool into
the apr_make_hash function, and we get a hash table in shared memory.
I would much prefer to finish the SMS work, then to put these API's into
APR. Remember, any API we add to APR we need to keep forever. That is
the nature of writing a library. Since we are well on our way to not
needing separate functions for this, I am -0.5 for it.
Ryan
On Wed, 4 Jul 2001, Ian Holsman wrote:
> This implements a hash table using shared memory.
> Limitations:
> * doesn't have a freelist, so deletes aren't reclaimed
> * and can't expand it's size (dont know how to implement this with
> current shared memory system)
> * no locking (yet)
> It is intended for use in modules having a large config requirements
> (Patches inline, extra files attached)
>
> It is intented that apr_shm_hash.c will live in apr/tables
> apr_shm_hash.h in apr/include
> &
> testhash.c in apr/test
>
> Index: tables/Makefile.in
> ===================================================================
> RCS file: /home/cvspublic/apr/tables/Makefile.in,v
> retrieving revision 1.6
> diff -u -u -r1.6 Makefile.in
> --- tables/Makefile.in 2001/01/09 11:06:18 1.6
> +++ tables/Makefile.in 2001/07/05 03:53:02
> @@ -1,5 +1,5 @@
>
> -TARGETS = apr_tables.lo apr_hash.lo
> +TARGETS = apr_tables.lo apr_hash.lo apr_shm_hash.lo
>
> # bring in rules.mk for standard functionality
> @INCLUDE_RULES@
> Index: test/Makefile.in
> ===================================================================
> RCS file: /home/cvspublic/apr/test/Makefile.in,v
> retrieving revision 1.57
> diff -u -u -r1.57 Makefile.in
> --- test/Makefile.in 2001/06/15 20:04:39 1.57
> +++ test/Makefile.in 2001/07/05 03:53:02
> @@ -25,7 +25,8 @@
> testpoll@EXEEXT@ \
> testmem@EXEEXT@ \
> occhild@EXEEXT@ \
> - teststr@EXEEXT@
> + teststr@EXEEXT@ \
> + testhash@EXEEXT@
>
> TARGETS = $(PROGRAMS)
>
> @@ -125,5 +126,8 @@
>
> teststr@EXEEXT@: teststr.lo $(LOCAL_LIBS)
> $(LINK) teststr.lo $(LOCAL_LIBS) $(ALL_LIBS)
> +
> +testhash@EXEEXT@: testhash.lo $(LOCAL_LIBS)
> + $(LINK) testhash.lo $(LOCAL_LIBS) $(ALL_LIBS)
>
> # DO NOT REMOVE
>
>
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------