Re: SMS stuff

[David Reid <dr...@jetnet.co.uk>]

Greg,

> On Wed, Jun 06, 2001 at 05:35:43PM -0700, Justin Erenkrantz wrote:
> > On Thu, Jun 07, 2001 at 01:19:27AM +0100, David Reid wrote:
> > > This isn't possible with pools.  They limit you to a single way of
getting
> > > at your memory regardless of how it was obtained.
> >
> > What I basically said in my email to David was that I could see having a
>
> All right. This has to stop.
>
> David, Sander: bring your conversations onto THIS list. You cannot
continue
> to have conversations in private, develop code in the public APR(UTIL)
tree,
> and then expect us to /not/ participate (review, vote, tweak, patch).

To be honest the "issues" that have come up have been on the list before
solutions were committed :(  we are al busy people and reading every email
in total detail takes a large amount of time.

The basic reason for holding discussions in private is that they have been
extremely theoretical and don't have any code associated with them, nor will
they ever.  They also tended to be long and drawn out - not ideal list
material :(

However, point taken.

>
> The SMS stuff is in APR now, so we are all involved. Keeping conversations
> private is simply going to keep these disconnects happening in the future.

Yes.

>
> At the moment, we have a big group of people who see SMS as a substrate
for
> pools. However, there is a smaller group (David and Sander) who see SMS as
> the end-goal, replacing pools, and that pools will eventually go away.
>
> How did this disconnect happen? Because you guys talked in private, rather
> than with the rest of the group.

Don't believe that was how it happenned at all.  I was very much a cynic
when I started looking at the sms stuff.  The way that  afew people started
raving on about how cool it all was made me shudder and that was why I
looked at it in such detail.  The reviewing of the code an opening my mind
to the possibilities made me see where it could go.  To be perfectly blunt
Sanders constant evangelising made me more anti sms than pro it.  It was the
code that convinced me it could do a lot for APR :(  But you weren't to know
that as it's not something I've said to people before.

> </rant>
>
> > pool take in an option as to which SMS it should use.  I've said this
> > before a few weeks ago (probably when I first looked at the SMS code),
> > but let me reiterate that suggestion.  Obviously, there would be a
> > default SMS that would be used, if one isn't provided when the pool is
> > created.
>
> Exactly. This was my thought, too, so seeing a pool go *inside* of an SMS
> system is whacky.

Why?  sms and pools allocate memory.  A pool is just a specific system for
allocating that at present APR uses throughout, hence we're just adding more
capability to sms.  Don't really see why that is whacky.

>
> >...
> > IMHO, using APR without something that implements cleanup routines (like
> > what the pool does now) is going to be troublesome to implement without
> > rearchitecting the guts of APR.

Sms has more options for cleanup's than pools.  Sander added a scope
variable so we can do selective cleanup's of children or even within
children.  Given that sms also has a well defined parent/child structure
then this should make our memory easier to cleanup, not harder.  I will
admit that not a lot has been to develop that side more, but the code is
still only a month or so old...

There has never been any notion that sms wouldn't have cleanup's so I'm not
sure where this came from.  Perhaps because there is an apr_sms_free
function?

>
> Right on the money. APR is built around pools. There is no reason to even
> attempt to believe that it will be built any other way for a while. APR
2.0
> could have a different system, but for the next year... we're using pools.

This isn't/has never been in dispute.  I wouldn't consider replacing pools
in the near future and am -1 on replacing them before we have a release of
apache 2.0.  sms is a longer term plan.

>
> ----
>
> David said that SMS gives us flexibility that pools don't have. He
suggests
> a few things:
>
> > You want a single block of long lived memory, great use the standard
> > allocator.
> > You want high performance, low footprint memory, use a pool like system.
> > You want simple shared memory, use a shared meory sms.
> > You want higher performance shared meory, use a tracking sms as a child
of a
> > shared sms.
> > etc etc
>
> Single block of long-lived memory: the global pool (or some other
long-lived
> pool)
>
> High perf: use a pool
>
> Simple shared memory: a pool which uses a shared memory SMS for allocating
> its memory.
>
> High perf: already have it
>

Well, we'll agree to differ on these.  The basic problem is that they're all
the same method for allocating memory, all that varies is where the memory
comes from.  Hell, if that's all we want to do then why are we bothering
with sms?  sms adds a lot of baggage if all we really want is to be able to
slot in a backend!

>
> For the few times when we need to alloc/free at a granular level, then
> apr_pool_create() / apr_pool_destroy() are more than sufficient.

So we create the lists, destroy the lists, all just to free a few bytes.
Seems clumbsy  and overkill to me.  A pool is a whole management system, not
just a piece of memory.  That system has overhead which isn't always needed.

>
> I am not seeing a large justification for transforming the entire APR
> library (and bubbling that up into Apache, Subversion, etc). The SMS
> alternative systems seem to be able to fit under the pool abstraction.
>

Well, if that's all we'll ever use sms for then let's ditch it now as it's a
distraction. sms offers much more than that and we don't need it.

>
> Now, it is abundantly clear that a few of us are seeing it this way. What
we
> don't understand is what forces it to go the other way? Why does SMS need
to
> be the top-level abstraction? A simple explanation here would go a long
ways.
>
> For example, "you simple cannot do <X> with a pool, but you do it <THIS>
way
> with SMS."

Well, I'll try, but to be honest your views seem to be pretty well
entrenched :(  However, I'm in "glass half full" mode so we'll see :)

- every time we create a pool we have to go through and lock the
alloc_mutex.  the code is designed for genral usage and this is required.
With the lock changes this will get faster, but it'll still be there.
Justin pointed out that during connections we really are in a single thread,
so no locking is required.  Also at the end of the connection we'll ditch
the whole thing and start over.  Why not have a simple sms module that
doesn't have any locking and simply allocates memory directly from a single
large lump?  We don't even nee to track it as at the end of the connection
we'll just call reset and memset the whole block to 0.  It should be faster
than pools.
- Cliff says that for the bucket code they want a different way of getting
memory, using thread id's.  In pools we have to allocate a pool and then use
a table or some such to associate blocks from that pool with thread ids,
whereas with sms we simply write a sms module that has the management of
thread ids built in, vastly simplifying things for the bucket code.

2 examples that have come up in the past 24 hours...  They may not be the
best ones, but they are ones that have been suggested/implied by people
other than Sander/myself.

I gues what I see as the big advantage of sms is that it gives you a blank
sheet of paper for your memory.  You can write a very specific sms module
(as will be done for the buckets) with all sorts of optimisations/features
that pools just don't let you.  Pools always control the memory in a
particular way.  They have a set structure and that can't be easily changed.
You want to have refcounting for each allocation - implement it on top of
your pools.  In sms you want refcounting, write the code and integrate it
entirely.  You want specific functions for getting at it, not a problem just
add them for your sms module.

That's why it's important that the locking is done at sms level.  If you
don't need locking you'll uses a non locking sms module.  Why go passing in
variables when we don't really need to?

As the author of a piece of code you should know what sort of memory
management you need.  We'll hopefully end up with an array of modules that
you can choose from, an if the one you want isn't there, write a new one!
Whilst this is possible with pools you're adding another layer on top, with
sms you're not.


> Point out that requirement, and you can change a lot of minds here. But
> until then, I think you'll continue to see confused/concerned people, not
> understanding why you are suggesting we toss all of the memory management
in
> APR in favor of SMS.

Well, I'll try, but I guess if this horse is dead then we may as well bury
it :(  I'd hate to think that we'd do that at this stage as more work should
be done to try and justify the potential that some of us see in sms.  That
said if the door has already been bolted then the effort that would be
wasted in developing sms further would be just that, wasted, so we should
move on...  Oops, was this really a "glass half full" statement... :=)

david

Re: SMS stuff

[Luke Kenneth Casson Leighton <lk...@samba-tng.org>]

> Well, we'll agree to differ on these.  The basic problem is that they're all
> the same method for allocating memory, all that varies is where the memory
> comes from.  Hell, if that's all we want to do then why are we bothering
> with sms?  sms adds a lot of baggage if all we really want is to be able to
> slot in a backend!

remember that one of the suggestions i made was to create
a mem-locked SMS, for security of encryption / storage of
private keys (using techniques / code already in GPG).

the basic premise is that pretty much all code uses pools.

therefore, in order to fulfil the requirements,
you back-end pools with the appropriate sms as it's
a self-contained location in the code dependency-order.

luke

Re: SMS stuff

[Luke Kenneth Casson Leighton <lk...@samba-tng.org>]

> Here is why I thought that:
> 
> David Reid wrote:
> > When we're done we'll have
> > 
> > locks -> apr_sms_t
> > sms -> locks -> apr_sms_t
> > 
> > So personally I don't see the problem and thus I made the change!  I guess
> > maybe it's because people keep saying that we're going to change the pools
> > to use sms.  Why?  To get the maximum flexibility we'll need to use sms
> > throughout so while we may have
> > 
> > apr_pstrdup(apr_pool_t *pool, char *str)
> > 
> > we'll end up with
> > 
> > apr_pstrdup(apr_sms_t *mem_sys, char *str)
> 
> 
> So no wonder I'm a bit concerned.
 
ack. 

... okay.  doing a half-way-house isn't going to satisfy you,
me, or anybody.  david clarified in a later post,
on the above.

sander's written up a roadmap that outlines what me, sander 
and david agree on, and it takes into consideration
everyone's concerns: that's my take on it, anyway.


> I haven't complained about the code one bit. Yes, it is simple, but reading
> it isn't change what I'm talking about... I have an issue with what we're
> going to *do* with it. And given that I believe SMS with providing storage
> for a pool, 

ack.

> then I question why we have a pool underneath an SMS.

me, too, don't worry! :)


okay, there are two approaches to do
"SMS with providing storage for a pool"

1) implement apr_pool.c to call functions in apr_sms

2) provide exactly the same functionality and then #define.

if 1) just becomes a thin wrapper, then it's pointless function
overhead and you might as well do 2) _anyway_!

luke

Re: SMS stuff

[Greg Stein <gs...@lyra.org>]

On Thu, Jun 07, 2001 at 03:35:14PM +0200, Luke Kenneth Casson Leighton wrote:
> > > Point out that requirement, and you can change a lot of minds here. But
> > > until then, I think you'll continue to see confused/concerned people, not
> > > understanding why you are suggesting we toss all of the memory management
> > in
> > > APR in favor of SMS.
> 
> uh, sorry to have to point this out like this, but your understanding
> of the original plan for SMS usage is plain wrong.
> 
> i have no idea what makes you think that anyone is suggesting that
> all mman in APR is 'tossed' in favour of sms, and if anyone
> else recommends it i will bitch at them persistently until they give
> a decent justification, or give up. :)

Here is why I thought that:

David Reid wrote:
> When we're done we'll have
> 
> locks -> apr_sms_t
> sms -> locks -> apr_sms_t
> 
> So personally I don't see the problem and thus I made the change!  I guess
> maybe it's because people keep saying that we're going to change the pools
> to use sms.  Why?  To get the maximum flexibility we'll need to use sms
> throughout so while we may have
> 
> apr_pstrdup(apr_pool_t *pool, char *str)
> 
> we'll end up with
> 
> apr_pstrdup(apr_sms_t *mem_sys, char *str)


So no wonder I'm a bit concerned.

>...
> please, like david did, if you don't like or don't understand
> the explanations, please read the code.  it's really short,
> it's really simple - and it's short and simple _because_ we
> [collectively - all of us] have enough experience to realise
> that anything else will cause us to have nightmares until the
> code's ripped out and burnt.  ritually.

I haven't complained about the code one bit. Yes, it is simple, but reading
it isn't change what I'm talking about... I have an issue with what we're
going to *do* with it. And given that I believe SMS with providing storage
for a pool, then I question why we have a pool underneath an SMS.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

Re: SMS stuff

[Luke Kenneth Casson Leighton <lk...@samba-tng.org>]

> > Point out that requirement, and you can change a lot of minds here. But
> > until then, I think you'll continue to see confused/concerned people, not
> > understanding why you are suggesting we toss all of the memory management
> in
> > APR in favor of SMS.

uh, sorry to have to point this out like this, but your understanding
of the original plan for SMS usage is plain wrong.

i have no idea what makes you think that anyone is suggesting that
all mman in APR is 'tossed' in favour of sms, and if anyone
else recommends it i will bitch at them persistently until they give
a decent justification, or give up. :)

_one_ of the suggestions was to create functions in sms that
are identical to those of pools except with different names and
different types.  _one_ [then you add into apr_compat.h
#defines to move them to the new types]

you don't have to take or do that suggestion, esp. as this is
an easy-does-it approach.

please, like david did, if you don't like or don't understand
the explanations, please read the code.  it's really short,
it's really simple - and it's short and simple _because_ we
[collectively - all of us] have enough experience to realise
that anything else will cause us to have nightmares until the
code's ripped out and burnt.  ritually.

:)

luke

RE: SMS stuff

[Sander Striker <st...@samba-tng.org>]

[...]
> Sanders constant evangelising made me more anti sms than pro it.  

*whoops* I tend to do that sometimes when I really have a good
gut feeling about something. I is not supposed to have a negative
effect though. [I must say that I am a bit blind for scepticism,
I see it more as questions that need to be answered and it is
just that that seems to tick people off]

> It was the code that convinced me it could do a lot for APR :(
> But you weren't to know that as it's not something I've said
> to people before.

Not even me. People, if I'm in evangelist mode, please drop me a
line.

Thx,

Sander