You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Tom Harrington <tp...@crosstor.com> on 1999/12/13 18:38:10 UTC
Memory management in 1.3.x
I'm porting 1.3.x to an embedded OS with some rather non-Unixish (and
non-NTish, for that matter) restrictions on memory allocation. Despite
the differences I think it's possible.
What would make this a whole lot easier would be if I could eliminate
all calls to malloc() and the like outside of src/main/alloc.c, forcing
all memory requests to go through the memory API. These malloc() calls
will need to be rewritten in one way or another if this port is to
succeed, and it looks like the best solution is to replace them with
calls like ap_palloc wherever possible.
So I'd like some feedback on the wisdom of changes like the following:
* When a malloc() appears, replace it with:
pool *ptmp;
ptemp = ap_make_sub_pool(NULL);
foo = ap_palloc(ptmp, ....);
* If a corresponding free() appears, replace it with
ap_destroy_pool(ptmp);
It appears that this may not work in all cases, but every little bit
helps.
--
Tom Harrington
CrosStor Software, Inc.
tph@crosstor.com
Re: Memory management in 1.3.x
Posted by Tom Harrington <tp...@crosstor.com>.
Scott Hess wrote:
>
> My suggestion would be to write a malloc()/free() replacement, and
> use -Dmalloc=my_malloc -Dfree=my_free to replace the current uses.
I have considered this possibility, and not yet rejected it. However,
Apache's memory subsystem already implements most of what I would need.
As a result there's a strong incentive to try and use this existing
code rather than reinventing it.
The main problem that I face is the lack of any sort of virtual addressing.
As a result it's impossible to reliably allocate more than a single
memory page (larger allocations fail unless the system can locate
enough adjacent free pages to meet the request). So any malloc() over
4kB is likely to fail. The only real exception to this is that, if you
try to allocate a larger chunk at boot time, there are still lots of
adjacent free pages, and you'll probably get it. It's only later that
memory fragmentation makes allocation a problem. I think I can allocate
a large block in this manner, and modify src/main/alloc.c to use it.
But the malloc()s that don't use the memory API are still a potential
problem.
This solution has a number of obvious limitations, but it's the only
one that seems to exist right now. I am lobbying for a saner memory
management system (since we make the OS, we can enhance it) but it's
not clear if I'll succeed. In the meantime, management here is aware
of the limitations and still thinks the port is a good idea.
> The use
> of pools you suggest seems like it would have rather high overhead for
> those platforms which already have a suitable malloc library.
I would not try to force this solution on platforms where it's not
necessary. A few strategic #ifdefs would enable me to have this change
on my platform without altering the code for anyone else. I'm just
not certain that it'd work the way I want it to.
I hadn't really considered the performance implications, though, I had
mainly focused on "will it work at all?". This may be a useful point
in my effort to improve our memory management...
> Later,
> scott
>
> ----- Original Message -----
> From: Tom Harrington <tp...@crosstor.com>
> To: <ne...@apache.org>
> Sent: Monday, December 13, 1999 9:38 AM
> Subject: Memory management in 1.3.x
>
> > I'm porting 1.3.x to an embedded OS with some rather non-Unixish (and
> > non-NTish, for that matter) restrictions on memory allocation. Despite
> > the differences I think it's possible.
> >
> > What would make this a whole lot easier would be if I could eliminate
> > all calls to malloc() and the like outside of src/main/alloc.c, forcing
> > all memory requests to go through the memory API. These malloc() calls
> > will need to be rewritten in one way or another if this port is to
> > succeed, and it looks like the best solution is to replace them with
> > calls like ap_palloc wherever possible.
> >
> > So I'd like some feedback on the wisdom of changes like the following:
> >
> > * When a malloc() appears, replace it with:
> > pool *ptmp;
> > ptemp = ap_make_sub_pool(NULL);
> > foo = ap_palloc(ptmp, ....);
> >
> > * If a corresponding free() appears, replace it with
> > ap_destroy_pool(ptmp);
> >
> > It appears that this may not work in all cases, but every little bit
> > helps.
> >
> > --
> > Tom Harrington
> > CrosStor Software, Inc.
> > tph@crosstor.com
--
Tom Harrington
CrosStor Software, Inc.
tph@crosstor.com
Re: Memory management in 1.3.x
Posted by Scott Hess <sc...@avantgo.com>.
My suggestion would be to write a malloc()/free() replacement, and
use -Dmalloc=my_malloc -Dfree=my_free to replace the current uses. The use
of pools you suggest seems like it would have rather high overhead for
those platforms which already have a suitable malloc library.
Later,
scott
----- Original Message -----
From: Tom Harrington <tp...@crosstor.com>
To: <ne...@apache.org>
Sent: Monday, December 13, 1999 9:38 AM
Subject: Memory management in 1.3.x
> I'm porting 1.3.x to an embedded OS with some rather non-Unixish (and
> non-NTish, for that matter) restrictions on memory allocation. Despite
> the differences I think it's possible.
>
> What would make this a whole lot easier would be if I could eliminate
> all calls to malloc() and the like outside of src/main/alloc.c, forcing
> all memory requests to go through the memory API. These malloc() calls
> will need to be rewritten in one way or another if this port is to
> succeed, and it looks like the best solution is to replace them with
> calls like ap_palloc wherever possible.
>
> So I'd like some feedback on the wisdom of changes like the following:
>
> * When a malloc() appears, replace it with:
> pool *ptmp;
> ptemp = ap_make_sub_pool(NULL);
> foo = ap_palloc(ptmp, ....);
>
> * If a corresponding free() appears, replace it with
> ap_destroy_pool(ptmp);
>
> It appears that this may not work in all cases, but every little bit
> helps.
>
> --
> Tom Harrington
> CrosStor Software, Inc.
> tph@crosstor.com
Re: Memory management in 1.3.x
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
> > I think you are looking at the wrong code base for work like this. I
> > would suggest looking at Apache 2.0. A lot of the malloc's outside of APR
> > have been done away with, and most of the memory management is being done
> > in a platform specific manner. Take a look at APR and Apache 2.0, and let
> > me know if this will work for you. We are looking to release a beta of
> > 2.0 in a few months.
>
> Thanks for the tip-- 2.0 is already more appealing than 1.3.9 for other
> reasons, from a technical perspective at least (mostly due to
> multithreading). But I don't think our schedule will permit a few months'
> delay for a beta unless it's simply impossible to pull off a 1.3.9 port.
> And while there are some obvious issues I don't think it's impossible.
> Later on I hope to move to 2.0, but I don't think waiting for it is
> an option that's open to me right now.
Sorry to hear that.
> OK, in the meantime, does my malloc --> ap_palloc conversion look like
> it'll work? I don't claim it's ideal, but I think it'll get the job
> done for now.
I haven't given it much thought, and to be honest, I haven't played with
the 1.3 tree in a few months, so I can't give a good opinion of your
suggestion. I suggest trying it and supplying a patch.
Ryan
Re: Memory management in 1.3.x
Posted by Tom Harrington <tp...@crosstor.com>.
Ryan Bloom wrote:
>
> I think you are looking at the wrong code base for work like this. I
> would suggest looking at Apache 2.0. A lot of the malloc's outside of APR
> have been done away with, and most of the memory management is being done
> in a platform specific manner. Take a look at APR and Apache 2.0, and let
> me know if this will work for you. We are looking to release a beta of
> 2.0 in a few months.
Thanks for the tip-- 2.0 is already more appealing than 1.3.9 for other
reasons, from a technical perspective at least (mostly due to
multithreading). But I don't think our schedule will permit a few months'
delay for a beta unless it's simply impossible to pull off a 1.3.9 port.
And while there are some obvious issues I don't think it's impossible.
Later on I hope to move to 2.0, but I don't think waiting for it is
an option that's open to me right now.
> Let me know if there is anything I can help you with.
OK, in the meantime, does my malloc --> ap_palloc conversion look like
it'll work? I don't claim it's ideal, but I think it'll get the job
done for now.
> Ryan
>
> On Mon, 13 Dec 1999, Tom Harrington wrote:
>
> > I'm porting 1.3.x to an embedded OS with some rather non-Unixish (and
> > non-NTish, for that matter) restrictions on memory allocation. Despite
> > the differences I think it's possible.
> >
> > What would make this a whole lot easier would be if I could eliminate
> > all calls to malloc() and the like outside of src/main/alloc.c, forcing
> > all memory requests to go through the memory API. These malloc() calls
> > will need to be rewritten in one way or another if this port is to
> > succeed, and it looks like the best solution is to replace them with
> > calls like ap_palloc wherever possible.
> >
> > So I'd like some feedback on the wisdom of changes like the following:
> >
> > * When a malloc() appears, replace it with:
> > pool *ptmp;
> > ptemp = ap_make_sub_pool(NULL);
> > foo = ap_palloc(ptmp, ....);
> >
> > * If a corresponding free() appears, replace it with
> > ap_destroy_pool(ptmp);
> >
> > It appears that this may not work in all cases, but every little bit
> > helps.
> >
> > --
> > Tom Harrington
> > CrosStor Software, Inc.
> > tph@crosstor.com
> >
>
> _______________________________________________________________________
> Ryan Bloom rbb@raleigh.ibm.com
> 4205 S Miami Blvd
> RTP, NC 27709 It's a beautiful sight to see good dancers
> doing simple steps. It's a painful sight to
> see beginners doing complicated patterns.
--
Tom Harrington
CrosStor Software, Inc.
tph@crosstor.com
Re: Memory management in 1.3.x
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
I think you are looking at the wrong code base for work like this. I
would suggest looking at Apache 2.0. A lot of the malloc's outside of APR
have been done away with, and most of the memory management is being done
in a platform specific manner. Take a look at APR and Apache 2.0, and let
me know if this will work for you. We are looking to release a beta of
2.0 in a few months.
Let me know if there is anything I can help you with.
Ryan
On Mon, 13 Dec 1999, Tom Harrington wrote:
> I'm porting 1.3.x to an embedded OS with some rather non-Unixish (and
> non-NTish, for that matter) restrictions on memory allocation. Despite
> the differences I think it's possible.
>
> What would make this a whole lot easier would be if I could eliminate
> all calls to malloc() and the like outside of src/main/alloc.c, forcing
> all memory requests to go through the memory API. These malloc() calls
> will need to be rewritten in one way or another if this port is to
> succeed, and it looks like the best solution is to replace them with
> calls like ap_palloc wherever possible.
>
> So I'd like some feedback on the wisdom of changes like the following:
>
> * When a malloc() appears, replace it with:
> pool *ptmp;
> ptemp = ap_make_sub_pool(NULL);
> foo = ap_palloc(ptmp, ....);
>
> * If a corresponding free() appears, replace it with
> ap_destroy_pool(ptmp);
>
> It appears that this may not work in all cases, but every little bit
> helps.
>
> --
> Tom Harrington
> CrosStor Software, Inc.
> tph@crosstor.com
>
_______________________________________________________________________
Ryan Bloom rbb@raleigh.ibm.com
4205 S Miami Blvd
RTP, NC 27709 It's a beautiful sight to see good dancers
doing simple steps. It's a painful sight to
see beginners doing complicated patterns.